Learning method and device for specializing an artificial intelligence model for a user organization

Abstract

To provide a training method for specializing a preliminarily trained artificial intelligence model in a deployed institution, and an apparatus performing the training method.SOLUTION: There are provided a training method for specializing an artificial intelligence model in a deployed institution and an apparatus performing the training method. The present invention specifically is an operation method for a training apparatus operated by at least one processor, the method includes the steps of: extracting dataset to be used for specialized learning from among data processed by medical institution, by taking into account the characteristics of the data processed by the medical institution; and supervising the pre-learned artificial intelligence model by using the dataset.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to artificial intelligence techniques. [Background technology]

[0002] Machine learning technology, represented by deep learning, is providing results that surpass the performance of existing methods in analyzing various types of data, such as images, voice, and text. In addition, machine learning technology is being applied to various fields due to its inherent scalability and flexibility, and various types of neural networks have been made public.

[0003] As such, machine learning-based artificial intelligence (AI) technology is being actively introduced into the medical field. Previously, computer-aided detection (CAD) devices detected lesions based on rules or from candidate areas set in medical images, but recent AI-based medical image reading technology can analyze the entire medical image using AI algorithms and visually display abnormal lesions.

[0004] Medical professionals can receive information on abnormal lesions contained in medical images from diagnostic assistant devices equipped with AI-based medical image reading technology and use this information to make diagnoses.

[0005] Meanwhile, medical institutions still use the same AI model despite domain differences such as imaging equipment, imaging methods, severity, and race. In this case, there is a difference between the data each institution is trying to analyze with the AI ​​model and the AI ​​model's training data, resulting in the AI ​​model's performance being lower than expected in the medical field. Although pre-trained AI models can be fine-tuned using data from each institution to optimize them for each institution's data, the AI ​​model loses the prior knowledge it has learned, which affects its generalization performance and ultimately makes it impossible to guarantee stable operation of the AI ​​model. Summary of the Invention [Problem to be solved by the invention]

[0006] The problem to be solved is to provide a learning method for making a pre-trained artificial intelligence model specific to the institution in which it is used, and an apparatus for performing this method.

[0007] The problem to be solved is to provide a method for collecting data from user institutions to train a pre-trained artificial intelligence model, specifically, to provide a method for selecting data for training an artificial intelligence model from the data from the user institutions and selecting data that requires annotation.

[0008] The problem to be solved is to provide a method for training an artificial intelligence model using data from the user institution while preserving the prior knowledge of the artificial intelligence model. [Means for solving the problem]

[0009] In one embodiment, a method for operating a learning device operated by at least one processor includes the steps of extracting a dataset to be used for specialized learning from data held by a specific institution, selecting annotation targets that require annotation from the dataset using a pre-trained artificial intelligence model, and teaching and learning the pre-trained artificial intelligence model using data in which labels are annotated for the annotation targets.

[0010] The step of selecting the annotation target may use the prediction results of the pre-trained artificial intelligence model for at least a portion of the data in the dataset to select data that is uncertain for the pre-trained artificial intelligence model as the annotation target.

[0011] The step of selecting the annotation target may select the annotation target based on an uncertainty score measured using a prediction result of the pre-trained artificial intelligence model.

[0012] The uncertainty score is measured using at least one of the confidence of the lesion-specific score predicted by the pre-trained artificial intelligence model, the entropy of the lesion-specific heat map predicted by the pre-trained artificial intelligence model, and the co-occurrence of lesions predicted by the pre-trained artificial intelligence model.

[0013] The step of selecting the annotation target may select data representative of the distribution of the dataset from a feature space of the pre-trained artificial intelligence model as the annotation target.

[0014] The operating method may further include a step of annotating information extracted from a radiologist report to the annotation target, or providing a prediction result of the pre-trained artificial intelligence model for the annotation target to an annotator to support the annotation work.

[0015] The step of extracting a data set to be used for specialized learning may determine the amount of data to be used for specialized learning based on the amount of data held by the specific institution and data characteristics.

[0016] The step of training the pre-trained artificial intelligence model may provide information that maintains the prior knowledge of the pre-trained artificial intelligence model to the artificial intelligence model undergoing supervision and learning.

[0017] The step of training the pre-trained artificial intelligence model may include calculating a distillation loss between the artificial intelligence model undergoing supervision learning and a teacher model, and providing the distillation loss to the artificial intelligence model undergoing supervision learning, and the teacher model may be the same model as the pre-trained artificial intelligence model.

[0018] The distillation loss may be a loss that allows the artificial intelligence model during supervision to keep up with the intermediate features and / or final output of the teacher model.

[0019] Another embodiment of a method for operating a learning device operated by at least one processor includes the steps of collecting a first dataset for pre-learning, outputting a first artificial intelligence model that has been pre-learned for at least one task using the first dataset, and outputting a second artificial intelligence model that has been specialized and trained using a second dataset collected from a specific institution while maintaining the prior knowledge acquired in the pre-learning.

[0020] The first artificial intelligence model can be trained using preprocessed data so that the domain of the input data cannot be distinguished, or can be trained adversarially so that the domain of the input data cannot be distinguished from extracted intermediate features.

[0021] The step of outputting the second artificial intelligence model may include calculating a distillation loss between the artificial intelligence model undergoing specialized learning and a teacher model, and providing the distillation loss to the artificial intelligence model undergoing specialized learning so that the second artificial intelligence model maintains the prior knowledge, and the teacher model may be the same model as the pre-trained first artificial intelligence model.

[0022] The step of outputting the second AI model may include performing supervised learning of the first AI model using at least a portion of annotation data annotated with labels from the second dataset, and providing information maintaining prior knowledge of the first AI model to the AI ​​model undergoing supervised learning. The information maintaining prior knowledge of the first AI model may be a distillation loss between the AI ​​model undergoing supervised learning and a teacher model. The teacher model may be the same model as the first AI model.

[0023] The operating method may further include the steps of extracting the second dataset to be used for the specialized learning from data held by the specific institution, selecting annotation targets requiring annotation from the second dataset using the first artificial intelligence model, and obtaining data annotated with labels for the annotation targets.

[0024] The step of selecting the annotation target may use the prediction results of the first artificial intelligence model for at least some of the data in the second dataset to select data that is uncertain for the first artificial intelligence model as the annotation target.

[0025] The step of selecting the annotation target may select data representative of the distribution of the second dataset from a feature space of the first artificial intelligence model as the annotation target.

[0026] A learning device according to yet another embodiment includes a memory for storing instructions and a processor for executing the instructions, wherein the processor extracts a certain amount of medical institution data from a medical institution's data repository, and performs specialized learning of the pre-trained artificial intelligence model using the medical institution data while maintaining prior knowledge of the pre-trained artificial intelligence model.

[0027] The processor may extract uncertain data for the pre-trained artificial intelligence model from the medical institution data using a prediction result of the pre-trained artificial intelligence model for the medical institution data, select the uncertain data as annotation targets requiring annotation, and perform supervision and learning of the pre-trained artificial intelligence model using data annotated with labels for the annotation targets. The processor may provide information for maintaining the prior knowledge to the artificial intelligence model during supervision and learning, thereby maintaining the prior knowledge.

[0028] The processor can select a certain number of representative data representative of the distribution of the medical institution data, and select data for which the prediction of the pre-trained artificial intelligence model is uncertain from the representative data. The uncertain data is selected using at least one of the confidence of the lesion score predicted by the pre-trained artificial intelligence model, the entropy of the lesion heat map predicted by the pre-trained artificial intelligence model, and the co-occurrence of lesions predicted by the pre-trained artificial intelligence model. [Effects of the Invention]

[0029] According to the embodiment, various institutions can use artificial intelligence models specialized for their respective domain characteristics, so there is no need to worry about performance degradation of the artificial intelligence model due to domain disparities such as imaging equipment, imaging method, severity, and race.

[0030] According to the embodiment, the AI ​​model can learn data specific to each institution while maintaining prior knowledge for stable operation. Therefore, according to the embodiment, the AI ​​model specialized for each institution can provide analysis results that reflect the unique characteristics of each institution while providing generalization performance.

[0031] According to the embodiment, it is possible to select representative yet uncertain data from among the data held by each institution as the annotation target, so that annotation can be requested only for the selected data without the need to annotate the entire data. Therefore, according to the embodiment, it is possible to improve the convenience of learning, reduce learning time, and save learning costs.

[0032] According to the embodiment, a provider company that provides artificial intelligence models to institutions can differentiate the artificial intelligence models by varying the amount of data collected and the amount of data requiring annotation depending on the terms of the contract with each institution. [Brief explanation of the drawings]

[0033] [Figure 1] FIG. 1 is a configuration diagram of a learning device according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating a specialized learning method according to an embodiment. [Figure 3] FIG. 10 is a diagram illustrating a specialized learning method according to another embodiment. [Figure 4] FIG. 10 is a diagram illustrating a specialized learning method according to another embodiment. [Figure 5] FIG. 10 is a diagram illustrating a specialized learning method according to yet another embodiment. [Figure 6]FIG. 1 is a diagram illustrating specialized learning according to one embodiment. [Figure 7] 1 is a flowchart of a pre-learning method according to an embodiment. [Figure 8] 1 is a flowchart of a method for collecting data for specialized learning according to one embodiment. [Figure 9] 1 is a flowchart of a specialized learning method according to one embodiment. [Figure 10] FIG. 1 is a block diagram of a computing device according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0034] Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be realized in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts unnecessary for the explanation are omitted in order to clearly explain the present disclosure, and similar parts are designated by similar reference numerals throughout the specification.

[0035] In the description, when a part "comprises" a certain element, this means that it can further include other elements, but not excluding other elements, unless otherwise specified.

[0036] In the description, "transmitting or providing" may include not only direct transmitting or providing, but also indirect transmitting or providing via other devices or using a roundabout route. In the description, expressions in the singular are to be construed as singular or plural unless expressly stated otherwise, such as "one" or "single".

[0037] In the description, the order of operations depicted in the flowcharts may be changed, various operations may be combined, certain operations may be separated, or certain operations may not be performed.

[0038] In the description, terms such as "... unit", "... device", and "... module" refer to a unit that processes at least one function or operation, which can be realized by hardware, software, or a combination of hardware and software.

[0039] In the description, the apparatus is configured and coupled to perform the operations of the present disclosure by executing instructions on at least one processor. The computer program includes instructions written to cause a processor to perform the operations of the present disclosure and is stored on a non-transitory computer-readable storage medium. The computer program is downloaded over a network or sold in the form of a product.

[0040] The artificial intelligence model (AI model) disclosed herein is a machine learning model that learns at least one task and can be implemented as a computer program executed by a processor. The task learned by the AI ​​model can refer to a problem to be solved by machine learning or a task to be performed by machine learning. For example, when performing recognition, classification, prediction, etc. on medical images, recognition, classification, and prediction can each correspond to a separate task.

[0041] The artificial intelligence model of the present disclosure is configured using various neural network-based machine learning models according to input data, task type, learning method, etc. For example, when the artificial intelligence model receives medical images, a convolutional neural network (CNN) model can be used.

[0042] The AI ​​model of the present disclosure can receive various types of data. In the description, an AI model that uses medical images as training data and analysis data can be used as an example, and the AI ​​model that receives the medical images and performs at least one task can be configured in various structures.

[0043] The present disclosure is applicable to medical images of various parts taken using various modalities. For example, the modalities of medical images may be various, such as X-ray, MRI (magnetic resonance imaging), ultrasound, CT (computed tomography), MMG (mammography), DBT (digital breast tomosynthesis), etc.

[0044] In the explanation, the dataset and the data can be explained without distinction.

[0045] In the description, "use institution" refers to an entity that uses an AI model or a device including the same (e.g., a diagnostic assistant device) or a place where an AI model or a device including the same is used, and can include, for example, a hospital, a health checkup center, a company, a school, a public institution, etc. A "use institution" can be simply called an "institution," or can be called a "target institution," "medical institution," "target hospital," "user," etc.

[0046] In the description, the user institution data is data held by the user institution, and may be, for example, medical images taken by imaging equipment installed in the user institution or medical images externally requested by the user institution. The user institution data may be, for example, medical images stored in a hospital's Picture Archiving and Communication System (PACS).

[0047] In the description, "specialization" refers to a process or operation of enabling a pre-trained AI model to output good performance on data from a user institution (e.g., medical images), and may include an operation of optimizing a pre-trained AI model for a user institution, an operation of fine-tuning a pre-trained AI model for a user institution, an operation of customizing a pre-trained AI model for a user institution, etc. Here, "good performance" may refer to a case where the results output from an AI model for the user institution's data are similar to or better than "baseline performance." The "baseline performance" may be set in various ways and may be, for example, a performance index of a pre-trained AI model evaluated using validation data.

[0048] In the description, the learning to specialize a pre-trained artificial intelligence model for a user organization can be referred to as "specialized learning" or "additional learning."

[0049] In the description, a pre-trained artificial intelligence model may be an artificial intelligence model that has been fully trained so that it can be used by a user institution without additional specialized training.

[0050] In the description, data for pre-training an AI model can be referred to as pre-training data, and can also be referred to as in-house data of the company developing the AI ​​model, basic data, source data, etc.

[0051] In the description, the use institution data may be referred to as target-hospital data, target data, or the like.

[0052] Generally, an AI model learns a task using training data and ends the training when the results evaluated using validation data reach a specified performance. While such training data and validation data may be acquired under various conditions and environments, it is difficult to reflect all conditions and environments. Therefore, even if an AI model has completed training, it may produce results that fall short of the validation performance when actual hospital images are input.

[0053] Such problems can arise due to differences in the domains corresponding to the environments and conditions in which the data was collected / generated. Domain disparities can arise due to, for example, variations in imaging equipment, imaging methods, severity, and race.

[0054] For example, if the equipment used to capture the pre-training data differs from the equipment used to capture the data at the facility, differences in the image texture may occur. If the imaging method used to capture the pre-training data differs from the imaging method used at the facility, differences in the information contained in the images may occur, and performance may be reduced due to image differences. For example, imaging methods differ between anterior posterior (AP) and posterior anterior (PA), allowing for images in which the hands are visible or hidden.

[0055] There may be a difference between the severity distribution of the pre-training data and the severity distribution of the data from the user institution. For example, the pre-training data may be collected from a hospital with a relatively large number of patients with high severity, while the user institution may be a hospital with a relatively large number of patients with low severity.

[0056] There may be differences between the racial distribution of the pre-training data and the racial distribution of the user institution data. For example, the pre-training data may be collected from a hospital that serves Asian patients, but the user institution may serve a hospital that serves patients with a different racial distribution.

[0057] In this way, when each user organization analyzes data using an AI model that has completed training, it is necessary to solve the problem of the AI ​​model's performance being lower than expected due to domain disparity. In this case, if the pre-trained AI model is simply fine-tuned using the user organization's data, the AI ​​model will forget the prior knowledge it has learned, which will affect its generalization performance, and ultimately, the stable operation of the AI ​​model cannot be guaranteed.

[0058] Therefore, below we will explain in detail how to train an AI model using the data of the user institution while maintaining the prior knowledge of the AI ​​model. Then, below we will explain in detail how to select data for training the AI ​​model from the data of the user institution and select data that requires annotation.

[0059] FIG. 1 is a diagram illustrating the configuration of a learning device according to an embodiment.

[0060] 1, a training apparatus 100 may include a basic training apparatus 110, a data manager for specialized learning (also simply referred to as a "data manager") 130, and a specialized training apparatus 150 that performs specialized learning on an AI model 200 pre-trained by the basic learning apparatus 110 using data from a user institution. For convenience of explanation, the user institution may be a target hospital, and the data from the user institution may be referred to as target-hospital data.

[0061] The basic learner 110 works in conjunction with a database 120 in which pre-trained data is stored, and outputs an artificial intelligence model 200 that has learned at least one task using the pre-trained data in the database 120. The artificial intelligence model 200 can be called a basic artificial intelligence model, a pre-trained artificial intelligence model, a general artificial intelligence model, etc.

[0062] The data manager 130 can cull user data for specialized learning from the user data repository 10 and store the at least partially annotated user data in the database 140 .

[0063] The specialized learning device 150 works in conjunction with the database 140 in which the data of the user institution is stored, and uses the data of the user institution in the database 140 to perform specialized learning of the pre-trained artificial intelligence model 200 using the data of the user institution, thereby outputting an artificial intelligence model 300 for the user institution. The specialized learning device 150 can use the pre-trained data in the database 120 for specialized learning. The artificial intelligence model 300 specialized and trained by the specialized learning device 150 is provided to the user institution. The specialized and trained artificial intelligence model 300 can be installed in, for example, a data analysis apparatus (e.g., an image reading device) 20 of the institution.

[0064] The data manager 130 and the specialized learning device 150 may be located centrally, such as on a cloud server, and may link with multiple user institutions, perform specialized learning requested by the multiple user institutions, and then provide the AI ​​model to the institution. Alternatively, the data manager 130 and the specialized learning device 150 may be located at each user institution, and perform specialized learning individually.

[0065] For the sake of explanation, the base learner 110, the data manager 130, and the specialized learner 150 are referred to as such, but they may be computing devices operated by at least one processor. Here, the base learner 110, the data manager 130, and the specialized learner 150 may be implemented in a single computing device or distributed across separate computing devices. When distributed across separate computing devices, the base learner 110, the data manager 130, and the specialized learner 150 may communicate with each other via a communication interface.

[0066] Meanwhile, the basic learner 110, the data manager 130, and the specialized learner 150 can be realized by machine learning models required to train the artificial intelligence model. In the description, the artificial intelligence model 200 and the artificial intelligence model 300 can be referred to as target models to be constructed by machine learning.

[0067] The basic learner 110 outputs an AI model 200 that has learned at least one task using pre-training data. The pre-training data may include data acquired from various institutions and / or various equipment. The pre-training data may also include data acquired using various imaging methods. When collecting as much data as possible as pre-training data, it is inevitable that domain disparity will exist. Therefore, it is necessary to reduce the domain disparity of input data through domain generalization.

[0068] The base learner 110 can preprocess the training data to reduce the domain disparity of the input data, and then use the preprocessed training data to train the artificial intelligence model 200.

[0069] For example, images acquired in different domains may have differences in texture, etc. The base learner 110 may remove image features specific to the domain so that the artificial intelligence model 200 cannot distinguish which institution or equipment the input image was acquired from. The base learner 110 may perform preprocessing to remove domain features possessed by images acquired in different domains through image-to-image translation. For example, the base learner 110 may use a Generative Adversarial Network (GAN) as an image translation model, and may perform image transformation using a discriminator and adversarial loss to prevent the discriminator from distinguishing between image domains.

[0070] In addition to domain generalization at the image level, the base learner 110 can perform domain generalization using intermediate features extracted from the intermediate layer of the AI ​​model 200. The base learner 110 can train the AI ​​model through adversarial learning so that a classifier cannot distinguish domains from the intermediate features of the input image.

[0071] The data manager 130 extracts a certain amount of user institution data from the user institution's data repository 10 for specialized learning of the pre-trained AI model 200. The data manager 130 may determine at least a portion of the retrieved user institution data as learning data for specialized learning and provide the learning data to the specialized learner 150. In this case, the data manager 130 may determine at least a portion of the learning data as an annotation target and provide the at least partially annotated learning data to the specialized learner 150. The learning data may include abnormal data and normal data.

[0072] The data manager 130 may determine validation data for evaluating the specialized trained artificial intelligence model 300 from the use institution data and provide the validation data to the specialized learner 150. The validation data is collected so as not to overlap with the training data. For example, in the case of validation data, N abnormal data may be collected for each of C lesions, and N*C normal data may be collected.

[0073] The data repository 10 of the user institution may be, for example, medical images stored in a PACS (Picture Archiving and Communication System) of the target hospital. The data manager 130 can directly extract data by being granted access to the data repository of the user institution, or can obtain the required information from an intermediate device that interfaces with the data repository of the user institution.

[0074] The data manager 130 can determine the amount of data to be used for specialized learning from the data repository of the user institution. The data manager 130 can determine the amount of data to be used for learning by taking into consideration the amount of data held by the user institution and data characteristics. Data characteristics can include the ratio of severe data, age distribution, gender distribution, and race distribution. For example, if the user institution is a university hospital or a health checkup center, the ratio of abnormal data to the total data will differ. Therefore, the amount of data to be brought by each user institution will differ by taking into consideration the data characteristics of each institution.

[0075] The data manager 130 can determine the amount of annotation data required for learning. Because annotation requires time and money, the amount of annotation data can be determined at the request of or in accordance with a contract with the user institution. To reduce costs, specialized learning can be performed without annotating the user institution data. Alternatively, specialized learning can be performed using data in which at least a portion of the user institution data is annotated, taking into account the performance of the specialized AI model 300. In this case, the amount of annotation data can be determined based on the user institution's willingness to pay the costs.

[0076] Annotations can be image-level labels or pixel-level labels. For example, image-level labels annotating the presence or absence of malignant lesions, or pixel-level labels showing the contours of lesions, are possible. The type and level of annotation can be determined in various ways depending on the level of labels available at the user institution and the annotation costs.

[0077] The annotation method can be varied: a human can manually annotate the data with labels, or the images can be automatically annotated with labels extracted from a report prepared by a radiologist.

[0078] The data manager 130 can select data that requires annotation (simply, annotation targets) from the institutional data. At this time, the data manager 130 can select data that is uncertain for the pre-trained AI model 200 as the annotation target, or data that represents the distribution of institutional data as the annotation target. Alternatively, the data manager 130 can select data that represents the distribution of institutional data but is uncertain for the pre-trained AI model 200 as the annotation target. If there is data for which a radiologist report exists, the data manager 130 can select annotation targets from the data for which the report exists.

[0079] The method by which the data manager 130 selects annotation targets may be any of the examples or a combination of examples described below.

[0080] For example, the data manager 130 may measure the uncertainty and / or diversity of the extracted institutional data to select annotation targets. The data manager 130 may select data for which the prediction of the pre-trained artificial intelligence model is uncertain from a certain number of representative data representative of the institutional data.

[0081] The data manager 130 can measure the uncertainty of at least some of the data from the institutional data and select annotation targets. The data manager 130 can define an uncertainty score using the predicted value of the pre-trained artificial intelligence model 200, and select data having an uncertainty score equal to or above a standard or the top k data having the highest uncertainty scores as annotation targets.

[0082] The uncertainty score is defined as follows:

[0083] According to one embodiment, uncertainty is measured using the confidence of the lesion-specific score predicted by the pre-trained AI model 200. The lesion-specific uncertainty score predicted for data by the pre-trained AI model 200 is defined as in Equation 1, and the uncertainty score of the data can be set to the maximum or average value of the uncertainty scores of the lesions. Here, the lesion score is a probability value between 0 and 1, and the uncertainty score is defined such that the uncertainty increases as the lesion score approaches an intermediate value that is neither definitely positive nor definitely negative.

[0084] [Number 1] Lesion uncertainty score = 1 - |lesion score - 0.5|

[0085] According to another embodiment, uncertainty is measured using the entropy of the lesion-specific heat map predicted by the pre-trained AI model 200. The uncertainty score for each lesion is defined as the entropy measured by treating the 2D heat map for each lesion as a single 1D vector. The uncertainty score for the data can be set to the maximum or average value of the uncertainty scores for the lesions. Here, the more similar the values ​​in the vector, the higher the entropy, which means that the pre-trained AI model 200 cannot reliably classify the lesions. Therefore, the higher the entropy, the higher the uncertainty score.

[0086] According to yet another embodiment, uncertainty is measured by the co-occurrence of lesions. The more similar the lesion scores predicted by the pre-trained AI model 200, the more difficult it is for the AI ​​model 200 to predict a lesion from the data, and therefore the higher the uncertainty score of the data. The data manager 130 can compile the scores of C lesions into a vector of length C and measure the entropy of the vector to obtain the uncertainty score of the data. As explained above, the more similar the values ​​in the vector, the higher the uncertainty score.

[0087] Alternatively, the data manager 130 can measure the uncertainty score using the difference between the top two lesion scores among the C lesion scores, as in Equation 2. In other words, the smaller the difference between the top two lesion scores, the more difficult it is for the artificial intelligence model 200 to reliably classify the lesions, and therefore the higher the uncertainty score.

[0088] [Number 2] Data uncertainty score = 1 - |largest lesion score - second largest lesion score|

[0089] An example of how to calculate the uncertainty score based on the co-occurrence of lesions is given below.

[0090] Referring to Table 1, assume that the artificial intelligence model 200 outputs scores (probabilities) for five lesions or classes for data 1 and data 2. Comparing the predicted results for data 1 and data 2, the difference between the five values ​​constituting the vector (0.8, 0.7, 0.2, 0.4, 0.3) for data 2 is more similar than that for data 1, and data 2 is measured to have a higher uncertainty score than data 1. Alternatively, the difference between the top two lesion scores (a=0.8, b=0.7) appearing in data 2 is 0.1, and the difference between the top two lesion scores (a=0.8, b=0.2) appearing in data 1 is 0.6, and data 2 is measured to have a higher uncertainty score than data 1. Therefore, the data manager 130 selects data 2 as an annotation target and can perform or request annotation for data 2.

[0091] [Table 1]

[0092] The data manager 130 can select data representative of the distribution of the institutional data from the feature space of the pre-trained artificial intelligence model 200 as the annotation target. The data manager 130 can measure diversity, select k representative data, and determine them as the annotation target. There are various methods for selecting the k representative data. For example, k data is randomly sampled from the institutional data. After k-means clustering of the institutional data, k data closest to each cluster centroid is selected. Using a k-center greedy algorithm, k data that can cover the entire distribution of the institutional data with a delta (δ) radius is selected. Alternatively, using a robust k-center algorithm, k data that can cover the entire distribution of the institutional data with a delta (δ) radius is selected.

[0093] If there is data for which a radiologist report exists, the data manager 130 can select annotation targets from the data for which the radiologist report exists. In this case, the data manager 130 can select annotation targets taking into account lesions for specialized learning and the positive / negative ratio. The data manager 130 can use a separate language processing model to extract information such as the presence or absence of a lesion from the report. The language processing model can be, for example, a natural language processing (NLP) model or a deep language model.

[0094] The data manager 130 can select annotation targets using the predicted values ​​of the pre-trained artificial intelligence model 200 for data for which a doctor's report is available. Data for which the information in the doctor's report does not match the predictions of the artificial intelligence model 200 is selected as annotation targets.

[0095] The annotation of the annotation targets selected by the data manager 130 can be performed in various ways.

[0096] For example, the data manager 130 can provide annotation targets to the annotation device and can receive data annotated with labels from the annotation device. An annotator can write labels on data using the annotation device, or the annotation device can write labels on data automatically / semi-automatically.

[0097] If a doctor's report is available for the data, the data manager 130 can annotate the data with information extracted from the doctor's report.

[0098] The data manager 130 provides the annotator with the prediction results of the AI ​​model 200 for the annotation target, allowing the annotation work to be performed semi-automatically. The prediction results of the AI ​​model 200 include lesion scores, contours, and abnormality scores of data, and the annotator can refer to the prediction results to perform accurate and quick annotation work.

[0099] On the other hand, if specialized learning is performed without annotating the user data to reduce costs, the data manager 130 can provide all of the user data it has brought as training data. Alternatively, the data manager 130 can select some of the data from the user data it has brought as training data using metrics such as lesion-specific scores, lesion-specific uncertainty scores, and abnormality scores. Metrics such as lesion-specific scores, lesion-specific uncertainty scores, and abnormality scores can be extracted from a pre-trained artificial intelligence model. For example, the top N% of lesion-specific predicted scores are selected as training data. Alternatively, a certain proportion of training data is selected for each score interval, and for example, a specified proportion of data for each of multiple intervals dividing [0, 1] is selected as training data.

[0100] The specialized learner 150 trains the pre-trained AI model 200 using the training data provided by the data manager 130 to generate an AI model 300 for the user organization. At this time, the user organization data and annotated data may not be sufficient for training. Furthermore, when the already trained AI model 200 additionally trains using insufficient new data, catastrophic forgetting, in which the pre-trained prior knowledge is forgotten, may become a problem. Therefore, the specialized learner 150 uses a learning method for memorizing the prior knowledge of the AI ​​model 200, which will be described in detail below. The learning method for memorizing prior knowledge may be called learning without forgetting prior knowledge.

[0101] The specialized learning device 150 can determine the specialized learning method by taking into consideration the amount of training data, the amount of annotated data, the characteristics of the training data, and the like.

[0102] The training data for specialized learning can include at least some annotated data. If the training data includes annotated data, supervised training can be used. If the training data does not include annotated data, semi-supervised learning or unsupervised domain adaptation can be used.

[0103] The specialized learner 150 can define a loss that prevents the prior knowledge of the pre-trained artificial intelligence model 200 from changing, and can use the loss for specialized learning. For example, a supervised loss can be used for annotated user institution data. An unsupervised loss can be used for unannotated user institution data. Also, a loss can be used that prevents the prediction of some data used in pre-training the artificial intelligence model 200 from changing. The specialized learner 150 can use at least a portion of the loss defined for specialized learning to train the artificial intelligence model using the user institution data.

[0104] FIG. 2 is a diagram illustrating a specialized learning method according to an embodiment.

[0105] 2, the specialized learning device 150 stores prior knowledge acquired through prior learning and generates an AI model 300 specialized for the institution data used. To this end, the specialized learning device 150 may utilize knowledge distillation, in which a student model learns by imitating a teacher model.

[0106] The using institution may be a targeted hospital, and the using institution data may be targeted-hospital data.

[0107] For knowledge distillation-based learning, the teacher model 400 is a pre-trained AI model 200 that is frozen as the pre-trained AI model 200. The student model 420 before learning is a pre-trained AI model 200 that undergoes specialized learning without losing its prior knowledge, becoming the specialized AI model 300. Generally, knowledge distillation is used to generate a small student model by imitating a large teacher model. In contrast, in the present disclosure, the initial student model 420 may already be the same AI model 200 as the teacher model 400 before learning, and may eventually become the specialized AI model 300 through specialized learning using institutional data. In other words, the student model 420 of the present disclosure receives help from the teacher model 400 to maintain its original prior knowledge during specialized learning using institutional data. In the explanation, it is explained that the initial model of the student model 420 is a pre-trained artificial intelligence model 200, but the initial model of the student model 420 does not necessarily have to be a pre-trained artificial intelligence model 200, and can also be a model that can learn prior knowledge transmitted from the teacher model 400.

[0108] The specialized learning device 150 assigns the pre-trained AI model 200 to the teacher model 400 and assigns the pre-trained AI model 200 to the student model 420, and then performs prior knowledge retention learning and specialized learning using institutional data on the student model 420. By simultaneously performing prior knowledge retention learning during specialized learning of the student model 420, the student model 420 can learn so that it does not forget its prior knowledge through specialized learning. The specialized learning device 150 weights and sums the loss calculated in the prior knowledge retention learning process (distillation loss) and the loss calculated in the specialized learning process (supervised loss), and backpropagates the weighted and summed loss to the student model 420 to train the student model 420. The weighting values ​​of the distillation loss and the supervised loss can be determined in various ways.

[0109] For prior knowledge retention learning, pre-training data used in pre-learning of the artificial intelligence model 200 is used, or used institution data is used. The data for prior knowledge retention learning can be determined based on the data range accessible by the specialized learner 150. In this case, the data for prior knowledge retention learning does not need to be labeled. The data to be used for prior knowledge retention learning can be selected from the available pre-training data or used institution data using metrics such as lesion-specific scores, lesion-specific uncertainty scores, and abnormality scores. For example, the top N% of lesion-specific scores are selected as learning data. Alternatively, a certain proportion of learning data is selected for each score interval; for example, a specified proportion of data for each of multiple intervals dividing [0, 1] is selected as learning data. Prior knowledge retention learning is performed as follows.

[0110] The specialized learner 150 inputs the same data to the teacher model 400 and the student model 420, calculates a distillation loss that makes the intermediate features and / or outputs obtained from the teacher model 400 and the student model 420 similar, and provides the distillation loss to the student model 420 to train the student model 420 to output values ​​similar to those of the teacher model 400. This prevents the student model 420 from forgetting prior knowledge. As a distillation loss using intermediate features, L1 / L2 loss, cosine similarity, and L1 / L2 loss between the Gram matrices of two intermediate features can be used. As a distillation loss using the final output, cross-entropy between the predicted values ​​of the two models can be used.

[0111] For specialized learning, label-annotated user data is used. On the other hand, for specialized learning, pre-training data used for pre-training the artificial intelligence model 200 can also be used.

[0112] The specialized learner 150 can supervise and learn the student model 420 using the annotated data. The specialized learner 150 calculates a supervised loss, which is the difference between the value predicted by the student model 420 for the input data and the label of the input data, and provides the supervised loss to the student model 420 to supervise and learn the student model 420. Cross-entropy, binary cross-entropy, etc. can be used as the supervised loss.

[0113] The specialized learning device 150 can use validation data extracted from the institution data to validate the student model 420 that has completed the prior knowledge maintenance learning and specialized learning. The validation data may be data different from the data used for learning.

[0114] If the performance verification is met, the specialized learner 150 can finish training the student model 420 and provide it to the using institution.

[0115] If the performance verification is not satisfied, the specialized learning device 150 can repeatedly train the student model 420 until the performance verification is satisfied. To this end, the specialized learning device 150 can request the data manager 130 to reselect annotation targets from the data of the institution of use, and repeat learning using the reselected data.

[0116] 3 and 4 are diagrams illustrating a specialized learning method according to another embodiment.

[0117] 3 and 4, a method in which the specialized learner 150 uses unannotated user data to train the target artificial intelligence models 500a and 500b will be described.

[0118] First, referring to FIG. 3, the specialized learner 150 can learn to adapt the target hospital data to pre-trained data of a different domain through unsupervised domain adaptation. Here, it is assumed that the pre-trained data is data annotated with labels for a task, and the target hospital data is data without labels. The initial model of the artificial intelligence model 500a may be the pre-trained artificial intelligence model 200. Alternatively, the initial model of the artificial intelligence model 500a may not be the pre-trained artificial intelligence model 200, but may be a model capable of learning the pre-trained data and the target hospital data.

[0119] The artificial intelligence model 500a uses pre-training data with labels to learn a task, while also learning any task that can be performed without a label (Task Free).The artificial intelligence model 500a then uses user data that is not annotated with labels to learn any task that can be performed without a label.The artificial intelligence model 500a can learn task-free loss / unsupervised loss while learning task loss / supervised loss.

[0120] Examples of learning that is possible without labels include domain classification learning, domain adversarial learning, and self-supervised learning. Domain classification learning is learning to distinguish which institution the input data was collected from. Domain adversarial learning is learning to generate features that make it impossible to distinguish which institution the input data was collected from. Self-supervised learning is learning by creating labels from the data available, such as rotation prediction.

[0121] The specialized learning device 150 can verify the artificial intelligence model 500a using verification data extracted from the user institution data. The verification data may be data different from the data used for training. If the performance verification is met, the training of the artificial intelligence model 500a is completed and the model is provided to the user institution.

[0122] 4, the specialized learner 150 can obtain a pseudo-label of the use institution data using the result predicted from the use institution data by the pre-trained artificial intelligence model 200. Here, it is assumed that the pre-trained data is data annotated with a label for a task, and the use institution data is data without a label annotated.

[0123] The specialized learner 150 can train the artificial intelligence model 500b using the user institution data annotated with pseudo-labels and the pre-trained data annotated with labels. The initial model of the artificial intelligence model 500b may be the pre-trained artificial intelligence model 200. Alternatively, the initial model of the artificial intelligence model 500b may not be the pre-trained artificial intelligence model 200, but may be a model capable of learning the pre-trained data and the user institution data.

[0124] The specialized learning device 150 can verify the artificial intelligence model 500b using verification data extracted from the user institution data. The verification data may be data different from the data used for training. If the performance verification is met, the training of the artificial intelligence model 500b is completed and the model is provided to the user institution.

[0125] FIG. 5 is a diagram illustrating a specialized learning method according to yet another embodiment.

[0126] Referring to FIG. 5, for learning using unannotated usage data, the specialized learner 150 can train a style shift predictor 600.

[0127] The style transfer predictor 600 receives the source data and the pre-trained data, and performs training to obtain a style transfer function (f) 620 that reduces the difference in style distribution between the source data and the pre-trained data. In this case, the style transfer function (f) may be an invertible function that does not change the amount of information in the image.

[0128] The style distribution of data can be defined in various ways. For example, if the brightness of an image is defined as the style, the specialized learner 150 can use the style conversion predictor 600 to calculate a brightness conversion function that adjusts the average brightness of the image used in pre-training and the average brightness of the image obtained from the source. Alternatively, the style can be defined by the mean and variance of the features included in the data. The style can be defined by various scalars or vectors that can represent the image style.

[0129] The specialized learner 150 can convert the new usage data into a style transfer function (f) and input the style-transformed data into the pre-trained artificial intelligence model 200 to verify the style transfer function.

[0130] Thereafter, new user data generated by the user is converted into a style conversion function (f) and then input to the pre-trained artificial intelligence model 200.

[0131] FIG. 6 is a diagram illustrating specialized learning according to one embodiment.

[0132] Referring to FIG. 6, a basic AI model 200 is pre-trained using pre-training data (1). The AI ​​model 200 may receive pre-processed pre-training data to reduce domain disparity of input data. The pre-training data may be subjected to image-to-image translation to remove domain features of images acquired in different domains. Alternatively, the AI ​​model 200 may be subjected to learning to remove domain features of intermediate features. Domain generalization is performed to reduce domain disparity of the pre-training data through adversarial learning.

[0133] Abnormal training data, normal training data, and validation data are extracted from the data repository 10 of the user institution (2, 3, 4). The learning device 100 can determine the amount of data to be used for specialized training, taking into account the amount of data held in the data repository 10 and the characteristics of the data.

[0134] Annotation targets requiring annotation are selected from the abnormality training data (5). Data that is uncertain to the pre-trained AI model 200 or data that represents the distribution of the institutional data is selected as the annotation target. Alternatively, data that represents the distribution of the institutional data but is uncertain to the pre-trained AI model 200 is selected as the annotation target. Annotation targets are selected based on the uncertainty and / or diversity of the institutional data. Uncertainty is measured using the confidence of the lesion scores predicted by the pre-trained AI model 200, the entropy of the lesion heatmap predicted by the pre-trained AI model 200, or by taking into account the co-occurrence of lesions. To achieve diversity that represents the distribution of the institutional data, annotation targets are randomly sampled or k representative data are selected. Annotation targets are selected from data for which radiologist reports exist.

[0135] Annotated labels are provided to the annotation target (6). The learning device 100 can provide the annotation target to the annotation device and can receive data annotated with labels from the annotation device. An annotator can write labels on the data using the annotation device, or the annotation device can write labels on the data automatically / semi-automatically. The learning device 100 can annotate information extracted from the medical report with the labels of the annotation target. The learning device 100 can provide the annotator with the prediction results of the artificial intelligence model 200 for the annotation target and operate so that the annotation work is performed semi-automatically.

[0136] The artificial intelligence model 300 performs specialized learning using the learning data (7). The artificial intelligence model 300 performs specialized learning using the usage data without forgetting its prior knowledge. For this reason, as described in FIG. 2, the artificial intelligence model 300 can perform learning (prior knowledge maintenance learning) while learning the distillation loss provided from the basic artificial intelligence model 200, so as not to forget the prior knowledge possessed by the basic artificial intelligence model 200. The artificial intelligence model 300 can perform specialized learning using the basic artificial intelligence model 200 as an initial model.

[0137] The specialized trained artificial intelligence model 300 is validated with validation data (8).

[0138] If the performance verification does not meet the standards, abnormal training data is reselected or annotation targets are reselected from the data repository 10 of the user institution (9). Using the reselected training data or reselected annotation targets, the artificial intelligence model 300 again proceeds with specialized learning.

[0139] If the performance verification is met, the artificial intelligence model 300 finishes learning and is provided to the user institution (10).

[0140] FIG. 7 is a flowchart of a pre-learning method according to one embodiment.

[0141] 7, the learning device 100 collects a data set for pre-learning of an artificial intelligence model (S110). At this time, the data set may include data acquired from various institutions, data acquired from various equipment, data acquired using various imaging methods, etc., and domain disparities may exist.

[0142] The learning device 100 trains an artificial intelligence model using the dataset while reducing the domain disparity of the input data (S120). To achieve this, the domain generalization method can be a method of removing domain features from the input data by preprocessing the input data, or a method of removing domain features from features extracted by the artificial intelligence model.

[0143] The learning device 100 can perform preprocessing to remove domain features of input data and train an artificial intelligence model using the preprocessed input data. For example, the learning device 100 can perform preprocessing to remove domain features of images acquired in different domains through image-to-image translation. The learning device 100 can train an image translation model that transforms input data using a discriminator and adversarial loss so that the discriminator cannot distinguish between image domains.

[0144] Alternatively, the learning device 100 can use a classifier to train an artificial intelligence model so that a domain cannot be distinguished from an intermediate feature extracted in the middle of the artificial intelligence model.

[0145] FIG. 8 is a flowchart of a method for collecting data for specialized learning according to one embodiment.

[0146] Referring to FIG. 8, the learning device 100 extracts a dataset to be used for specialized learning from the entire data held by a user institution (e.g., a target hospital) (S210). The dataset to be used for specialized learning may include abnormal learning data, normal learning data, and verification data, and may be determined based on the number of lesions and the number of lesions collected. In this case, the learning device 100 may determine the amount of data to be used for learning in consideration of the amount of data held by the user institution and data characteristics. The amount of data to be used for specialized learning varies depending on the user institution. Data characteristics may include the ratio of abnormal data (ratio of severe data), age distribution, gender distribution, race distribution, etc.

[0147] The learning device 100 selects data representing the distribution of the institutional data from the extracted dataset and uncertain for the pre-trained AI model as an annotation target (S220). The learning device 100 can select data representing the distribution of the institutional data and uncertain for the pre-trained AI model. The learning device 100 can use the uncertainty and / or diversity of the institutional data to select annotation targets. The uncertainty can be measured using the confidence of the lesion scores predicted by the pre-trained AI model, the entropy of the lesion heatmap predicted by the pre-trained AI model 200, or by taking into account the co-occurrence of lesions. For diversity representing the distribution of the institutional data, annotation targets can be randomly sampled or k representative data can be selected. Annotation targets are selected from data for which radiologist reports exist. The amount of annotation data is determined variably according to the performance of the specialized trained AI model, or a fixed amount is determined according to the request of the institution using the data. Meanwhile, as explained with examples in Figures 3 to 5, it is also possible to train the AI ​​model without performing a separate annotation process on the data of the institution using the data.

[0148] The learning device 100 performs annotation on the selected annotation target or provides the prediction result of the artificial intelligence model for the annotation target to support the annotation work of the annotator (S230). The learning device 100 can annotate information extracted from the medical report with the label of the annotation target. The learning device 100 can provide the prediction result of the artificial intelligence model 200 for the annotation target for the annotator and operate so that the annotation work is performed semi-automatically. The annotator of the learning device 100 can be provided with the annotation target and receive the label of the annotation target.

[0149] The learning device 100 provides a data set including the annotated data as learning data for the artificial intelligence model (S240).

[0150] The learning device 100 determines whether data for re-learning the artificial intelligence model is necessary, based on the verification result of the artificial intelligence model trained with the training data (S250).

[0151] When the specialized learning of the artificial intelligence model is completed, the learning device 100 ends the collection of data from the user institution (S260). When data for re-learning is required, the learning device 100 selects unannotated data as new annotation targets (S270). The learning device 100 can select new data that has not been extracted from the entire data held by the user institution. In this way, the learning device 100 can repeatedly reselect annotation targets and re-extract usage data until learning is completed, and provide a new data set including the annotated data as specialized learning data for the artificial intelligence model.

[0152] FIG. 9 is a flowchart of a specialized learning method according to one embodiment.

[0153] 9, the learning device 100 trains an AI model using data collected from a user institution while preventing the AI ​​model from forgetting prior knowledge acquired through prior learning. To this end, a distillation loss is used to enable the AI ​​model undergoing specialized learning (student model) to follow the intermediate features and / or final output of the pre-trained AI model (teacher model), allowing the AI ​​model to learn new data without forgetting prior knowledge.

[0154] The learning device 100 performs supervised learning on an AI model using annotated data from a dataset collected from a user institution (S310). For specialized learning, user institution data annotated with labels can be used, and pre-training data annotated with labels can also be used. The learning device 100 can calculate a supervised loss by comparing the output of the AI ​​model for the input data with the labels and back-propagate the supervised loss to the AI ​​model. The learning device 100 provides the AI ​​model undergoing supervised learning with a distillation loss that enables the AI ​​model to keep up with the intermediate features and / or final output output by the teacher model for the input data, thereby performing prior knowledge retention learning (S320). The teacher model may be an AI model pre-trained with the pre-training data. The initial model of the AI ​​model undergoing supervised learning may also be a pre-trained AI model. For prior knowledge retention learning, user institution data and pre-training data can be used, but unlabeled data is also acceptable. On the other hand, the learning device 100 can select data to be used for prior knowledge maintenance learning from the pre-learning data and the institution data using metrics such as lesion scores, lesion uncertainty scores, and abnormality scores.

[0155] The learning device 100 provides the trained AI model to the user institution (S330). That is, if the specialized trained AI model outputs good performance for the user institution data, it is provided for use by the user institution.

[0156] The learning device 100 weights and sums the training loss and distillation loss using a weighting value, and backpropagates the weighted summed loss to the artificial intelligence model under training, thereby simultaneously progressing prior knowledge retention learning and specialized learning.

[0157] Meanwhile, the learning device 100 can verify the trained AI model using verification data collected from the user institution. Verification using the verification data is performed as needed. Depending on the verification result, the learning device 100 reselects the user institution data for re-learning and proceeds with re-learning of the AI ​​model using the re-selected data.

[0158] In this way, the AI ​​model retains the prior knowledge learned from the pre-training data while learning the task using the annotated user data. Therefore, according to the present disclosure, the AI ​​model operates stably without catastrophic forgetting, which is the forgetting of the prior knowledge pre-trained by the user data, even while being localized to the domain of the user using the user data.

[0159] On the other hand, the artificial intelligence model can be specialized and trained using annotated institutional data, and can also be specialized and trained using unannotated institutional data, as described in Figures 3 to 5.

[0160] FIG. 10 is a block diagram of a computing device according to one embodiment.

[0161] Referring to FIG. 10, the learning device 100, or its constituent components, the base learner 110, the data manager 130, and the specialized learner 150, can be implemented in a computing device 700 operated by at least one processor.

[0162] The computing device 700 may include one or more processors 710, a memory 730 into which computer programs executed by the processor 710 are loaded, a storage device 750 for storing computer programs and various data, a communication interface 770, and a bus 790 connecting these components. The computing device 700 may further include various other components. The processor 710 is a device that controls the operation of the computing device 700 and may be one of various types of processors that process instructions included in a computer program, such as at least one of a central processing unit (CPU), a microprocessor unit (MPU), a microcontroller unit (MCU), a graphic processing unit (GPU), or any other type of processor well known in the technical field of the present disclosure.

[0163] The memory 730 stores various data, instructions, and / or information. The memory 730 can load computer programs from the storage device 750 so that instructions written to perform the operations of the present disclosure are processed by the processor 710. The memory 730 may be, for example, a read only memory (ROM), a random access memory (RAM), etc.

[0164] The storage device 750 can non-temporarily store computer programs and various data, and may include a non-volatile memory such as a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a hard disk, a removable disk, or any other computer-readable recording medium well known in the art to which the present disclosure pertains.

[0165] The communication interface 770 may be a wired / wireless communication module that supports wired / wireless communication.

[0166] Bus 790 provides communication between the components of computing device 700 .

[0167] The computer program contains instructions that are executed by the processor 710 and are stored in a non-transitory computer-readable storage medium, and the instructions cause the processor 710 to perform the operations of the present disclosure. The computer program may be downloaded over a network or sold in the form of a product.

[0168] According to an embodiment, a computer program may include instructions for collecting a dataset for pre-training an artificial intelligence model and training the artificial intelligence model using the collected dataset according to a generalization method for a specified domain. The computer program may include instructions for performing preprocessing to remove domain features of input data and training the artificial intelligence model using the preprocessed input data. The computer program may include instructions for performing preprocessing to remove domain features possessed by images acquired in different domains through image transformation, and training an image transformation model using a discriminator and adversarial loss to transform the input data so that the discriminator cannot distinguish between image domains. The computer program may include instructions for training the artificial intelligence model so that the domain cannot be distinguished from intermediate features extracted from the intermediate portion of the artificial intelligence model by the discriminator.

[0169] In another embodiment, the computer program may include a command to extract a dataset to be used for specialized learning from the entire data held by the institution, a command to select uncertain data from the extracted dataset as an annotation target for the pre-trained artificial intelligence model while representing the distribution of the institution's data, a command to annotate the selected annotation target or to support the annotation work, a command to provide a dataset including the annotated data as training data for the artificial intelligence model, and a command to select new data not to be extracted from the entire data held by the institution or to select unannotated data as a new annotation target to generate a new dataset when data for re-training the artificial intelligence model is needed according to the verification results of the artificial intelligence model trained with the training data.

[0170] According to another embodiment, a computer program may include instructions for teaching and learning an AI model using annotated data from a dataset collected from a user institution. The computer program may include instructions for performing prior knowledge retention learning by providing the AI ​​model during teaching and learning with a distillation loss that allows the teacher model to track intermediate features and / or final output for input data. The computer program may include instructions for selecting data to be used for prior knowledge retention learning from the pre-learning data and the user institution data using metrics such as a lesion-specific prediction score, a lesion-specific uncertainty score, and an anomaly score. The computer program may include instructions for verifying the trained AI model using validation data collected from the user institution, instructions for re-screening the user institution data for re-learning if the performance verification does not meet the criteria, and instructions for re-training the AI ​​model using the re-screened data, and instructions for the learning device 100 to terminate learning and provide the trained AI model to the user institution if the performance verification is met.

[0171] According to the embodiment, various institutions can use artificial intelligence models specialized for their respective domain characteristics, so there is no need to worry about performance degradation of the artificial intelligence model due to domain disparities such as imaging equipment, imaging method, severity, and race.

[0172] According to the embodiment, the AI ​​model can learn data specific to each institution while maintaining prior knowledge for stable operation. Therefore, according to the embodiment, the AI ​​model specialized for each institution can provide analysis results that reflect the unique characteristics of each institution while providing generalization performance.

[0173] According to the embodiment, it is possible to select representative yet uncertain data from among the data held by each institution as an annotation target, and therefore, annotation can be requested only for the selected data without the need to annotate the entire data. Therefore, according to the embodiment, it is possible to improve the convenience of learning, reduce learning time, and save learning costs.

[0174] According to the embodiment, a provider company that provides an artificial intelligence model to an institution can differentiate the artificial intelligence model by varying the amount of data that requires annotation depending on the terms of the contract with each institution.

[0175] The embodiments of the present disclosure described above may be realized not only by an apparatus and a method, but also by a program that realizes functions corresponding to the configuration of the embodiments of the present disclosure, or a recording medium on which the program is recorded.

[0176] Although the embodiments of the present disclosure have been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concepts of the present disclosure as defined in the claims below also fall within the scope of the present disclosure.

Claims

1. A method of operating a learning device operated by at least one processor, comprising: A step of extracting a dataset to be used for specialized learning from data held by a medical institution based on data characteristics held by the medical institution; selecting, from the dataset, annotation targets that require annotation for the specialized learning using a pre-trained artificial intelligence model; and performing supervised learning of the pre-trained artificial intelligence model using the dataset and the labels of the annotation objects.

2. The method of claim 1 , wherein the data characteristics include at least one of a proportion of abnormal data, an age distribution, a gender distribution, or a racial distribution.

3. The step of extracting the data set comprises: The method of claim 1 , further comprising acquiring medical images for the medical institution-specific learning based on data characteristics of medical images stored in a picture archiving and communication system (PACS) of the medical institution.

4. The step of extracting the data set comprises: The method of claim 3 , further comprising acquiring medical images for the medical institution-specific learning based on the amount of image data stored in a PACS of the medical institution.

5. The method further includes using the pre-trained artificial intelligence model to obtain a prediction result for at least a portion of data stored in a data repository of the medical institution; The method of claim 1 , wherein the selecting the annotation target selects an annotation target requiring annotation for the specialized learning using a prediction result for the at least some of the data.

6. The step of selecting an annotation target includes: The method of claim 5 , wherein a certain amount of annotation targets is determined at the request of the medical institution or at the request of a contract with the medical institution.

7. A specialized learning device that performs specialized learning of a pre-trained artificial intelligence model for a specific medical institution based on a specialized learning dataset related to the specific medical institution among a plurality of medical institutions, and provides the specialized trained artificial intelligence model to the specific medical institution; a data manager that is linked to the data repositories of the plurality of medical institutions and generates a specialized learning dataset for the specialized learning based on at least a portion of the data stored in the data repository of the specific medical institution; The data manager In the data repository of each medical institution, data for specialized learning for that medical institution is acquired based on the stored data characteristics; A learning device that uses the pre-trained artificial intelligence model to select annotation targets that require annotation for the specialized learning for the medical institution from the data for the specialized learning for the medical institution.

8. The learning device according to claim 7 , wherein the data characteristics include at least one of a ratio of abnormal data, an age distribution, a gender distribution, and a race distribution.

9. The data manager The learning device according to claim 7 , wherein medical images for specialized learning for the medical institution are acquired based on data characteristics of medical images stored in a PACS (Picture Archiving and Communication System) of the medical institution.

10. The data manager The learning device according to claim 9 , wherein medical images for specialized learning for each medical institution are acquired based on the amount of image data stored in the PACS of each medical institution.

11. The data manager The learning device described in claim 7, wherein the pre-trained artificial intelligence model is used to obtain prediction results for at least a portion of the data stored in the data repository of the specific medical institution, and the prediction results for the at least a portion of the data are used to select annotation targets that require annotation for the specialized learning.

12. The data manager The learning device according to claim 11 , wherein a certain amount of annotation targets is determined at the request of the specific medical institution or in accordance with a contract with the specific medical institution.

13. 1. A method of operating a data manager operated by at least one processor, comprising: The stage of linking with data repositories of multiple medical institutions. A step of extracting a dataset to be used for specializing training of a pre-trained artificial intelligence model to an artificial intelligence model for the specific medical institution based on data extracted based on data characteristics stored in a data repository of a specific medical institution among the plurality of medical institutions; and using the pre-trained artificial intelligence model to select from the dataset annotation targets that require annotation for the specialized learning.

14. 14. The method of claim 13, wherein the data characteristics include at least one of a proportion of abnormal data, an age distribution, a gender distribution, and a race distribution.

15. The step of extracting the data set comprises: The method of claim 13 , further comprising acquiring medical images for specialized learning for the specific medical institution based on data characteristics of medical images stored in a Picture Archiving and Communication System (PACS) of the specific medical institution.

16. The step of extracting the data set comprises: The method of claim 15 , further comprising acquiring medical images for the specific medical institution-specific learning based on the amount of image data stored in a PACS of the specific medical institution.

17. The method further includes using the pre-trained artificial intelligence model to obtain a prediction result for at least a portion of data stored in a data repository of the specific medical institution; The method of claim 13 , wherein the selecting the annotation target selects an annotation target requiring annotation for the specialized learning using a prediction result for the at least some of the data.

18. The step of selecting an annotation target includes: The method of claim 17 , wherein a certain amount of annotation targets is determined at the request of the particular medical institution or at the contract with the medical institution.

Images