Method and device for generating synthetic data

Abstract

A device for generating synthetic data according to one embodiment may perform operations of: acquiring a pathology images for generation of synthetic data; generating patches by dividing the pathology images into a preset size; extracting pattern information of each patch on the basis of a classification model trained to extract pattern information including information on the compositional ratio of biological tissue included in a patch; and training a generative model designed to generate synthetic data by sampling a compositional ratio of biological tissue for a pathology image on the basis of the pattern information.

Description

Method and device for generating synthetic data

[0001] The present invention relates to a method and device for generating synthetic data of pathological images. More specifically, the present invention relates to a technique for extracting pattern information reflecting the composition ratio of biological tissue included in a pathological image and generating synthetic data based on the extracted pattern information, a technique for generating synthetic data by augmenting a pathological image based on the similarity between pattern information reflecting the composition ratio of biological tissue included in a patch specifying a portion of biological tissue, and / or a technique for extracting pattern information reflecting a time-series change aspect of biological tissue from a plurality of pathological images and generating synthetic data based on the extracted pattern information.

[0002]

[0003] Synthetic data refers to virtual data created using generative AI technology that mimics real data. This type of synthetic data generation technology can significantly increase the volume and diversity of data, and is playing a significant role in the field of medical image analysis.

[0004] Synthetic data is essential for improving the performance of machine learning and deep learning algorithms, enabling model training across a variety of scenarios. In particular, in medical image analysis, synthetic data can be used to generate diverse pathological images, increasing the accuracy of medical diagnoses and maximizing research efficiency.

[0005] Existing synthetic data generation technologies are mainly implemented using deep learning models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs).

[0006] For example, GANs can be used to learn patterns from different pathological images and generate new, realistic pathological images. This is useful for supplementing data shortages in pathological research and diagnosis, and for simulating various clinical situations.

[0007] However, existing synthetic data generation technologies have several problems.

[0008] Existing synthetic data technologies primarily focus on the transformation of single images, limiting their ability to adequately reflect the compositional ratios of various biological tissues or temporal changes. Consequently, synthetic data often fails to accurately reflect the detailed patterns and composition of actual pathological images, making the generated images difficult to use for practical diagnosis and research.

[0009] Furthermore, privacy concerns and the risk of data re-identification are being raised as issues during the process of generating synthetic data. Medical data often contains sensitive personal information, such as treatment information, which can raise privacy issues during the synthetic data generation process. In particular, there is a risk of personal information being re-identified during synthetic data generation, which can lead to compliance issues with personal information protection laws. For example, in the US, there have been reports of hospital admission records being re-identified, exposing sensitive personal information. These issues limit the scope of synthetic data utilization and hinder its efficiency.

[0010] Accordingly, a new technique is required that can generate more sophisticated and accurate pathological images while overcoming the limitations of existing synthetic data generation technologies as described above.

[0011]

[0012] The present invention aims to overcome the existing limitations of synthetic data generation technology and improve the accuracy of pathological research and diagnosis by generating more sophisticated and accurate pathological images.

[0013] Specifically, the present invention seeks to generate high-quality synthetic data reflecting various pathological patterns and composition ratios. To this end, the present invention provides a technique for dividing a pathological image into preset sizes to create multiple patches and extracting pattern information regarding the composition ratio of biological tissue contained in each patch. Furthermore, the present invention provides a generative model that combines patches of a pathological image based on the pattern information in each patch to generate new synthetic data.

[0014] Furthermore, the present invention aims to minimize the risk of personal information protection and data re-identification during the synthetic data generation process, thereby enabling secure data utilization. Medical data contains sensitive personal information, such as clinical information, and thus privacy issues may arise during the synthetic data generation process. The present invention aims to address compliance with the Personal Information Protection Act by minimizing the risk of personal information being re-identified during the data combination process.

[0015] Meanwhile, the technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below.

[0016]

[0017] A method for generating synthetic data, performed by a processor according to one embodiment, comprises: acquiring a pathological image for generating synthetic data; generating patches by dividing the pathological image into preset sizes; extracting pattern information of each patch based on a classification model trained to extract pattern information including information on a composition ratio of biological tissue included in the patch; and training a generation model designed to generate synthetic data by sampling the composition ratio of biological tissue for the pathological image based on the pattern information, wherein the generation model, for which the training is completed, can generate synthetic data reflecting the composition ratio of biological tissue for target pattern information.

[0018] Additionally, the pattern information may include information on at least one of the composition ratio of normal tissue, the composition ratio of tumor tissue, and the composition ratio of moisture contained in the patch.

[0019] In addition, the above classification model is configured as a convolutional neural network, and learning can be performed through supervised learning that specifies pattern information including information on the composition ratio of biological tissue included in each patch based on the correct values ​​of the pattern information for the patches and the patches to be used for learning according to a deep learning algorithm.

[0020] In addition, the classification model may include an input layer configured to receive patches during the learning process; an output layer configured to receive correct values ​​of pattern information for each patch during the learning process; and a hidden layer configured with parameters for learning the correlation between patches input to the input layer during the learning process and pattern information for each patch input to the output layer.

[0021] Additionally, the generative model may include an encoder that compresses learning data and converts it into latent variables based on parameters in a direction that minimizes the error of a loss function; and a decoder that generates synthetic data from the latent variables based on parameters in a direction that minimizes the error of the loss function.

[0022] In addition, the learning data used for learning the generative model includes first learning data including a first image including a pathological image and pattern information mapped to the first image; and second learning data including a second image obtained by masking the pathological image according to a predetermined transparency and pattern information mapped to the second image, and the synthetic data output by the generative model may include a synthetic image restored according to the input first or second learning data and the first pattern information of the first or second learning data, and second pattern information for the synthetic image.

[0023] Additionally, the loss function may include a first loss between the pathological image and the synthetic image or a second loss between the first pattern information and the second pattern information.

[0024] In addition, the operation of training the generative model may include a first training operation of inputting the first training data up to a first number of times to train to minimize the first loss and the second loss simultaneously; a second training operation of inputting the second training data up to a second number of times after the first time to train to minimize the first loss and the second loss simultaneously; and a third training operation of inputting the second training data up to a third number of times after the second time to train to remove the first loss from the loss function and minimize the second loss.

[0025] In addition, the second learning operation may include an operation of gradually increasing the transparency of the masking for the pathological image of the second learning data based on a preset ratio as the learning round progresses; an operation of lowering the reflection ratio of the first loss among the loss functions in proportion to the transparency of the masking; and an operation of setting the reflection ratio of the second loss among the loss functions in proportion to the transparency of the masking.

[0026] Additionally, the third learning operation may include an operation of performing learning based on a second image having a transparency of masking for a pathological image of the second learning data of 100%.

[0027] A method performed by a device for generating synthetic data operated by a processor according to one embodiment comprises: acquiring a pathological image for generating synthetic data; generating patches by dividing the pathological image into preset sizes; extracting pattern information including information on the composition ratio of biological tissue included in each patch; augmenting a pathological image by combining patches of a plurality of pathological images based on the pattern information of each patch; and training a generation model designed to generate synthetic data by sampling the composition ratio of biological tissue for the pathological image based on the pattern information of the original and augmented pathological images, wherein the generation model, when trained, can generate synthetic data reflecting the composition ratio of biological tissue for target pattern information.

[0028] Additionally, the pattern information may include information on at least one of the composition ratio of normal tissue, the composition ratio of tumor tissue, and the composition ratio of moisture contained in the patch.

[0029] In addition, the operation of enhancing the pathological image may include an operation of enhancing the pathological image by combining patches having a similarity within a preset range based on the similarity of the ratio constituting the pattern information of each patch.

[0030] In addition, the operation of enhancing the pathological image may include an operation of generating vector information by vectorizing the values ​​of each category constituting the pattern information; an operation of selecting a patch whose Euclidean distance based on the vector information is less than or equal to a preset distance; and an operation of enhancing the pathological image by combining the selected patches.

[0031] In addition, the operation of augmenting the pathological image may include an operation of generating vector information by vectorizing the values ​​of each category constituting the pattern information; an operation of mapping the vector information to a Euclidean space; an operation of selecting a preset number of patches existing within a preset range in the Euclidean space; and an operation of augmenting the pathological image by combining the selected patches.

[0032] Additionally, the generative model may include an encoder that compresses learning data and converts it into latent variables based on parameters in a direction that minimizes the error of a loss function; and a decoder that generates synthetic data from the latent variables based on parameters in a direction that minimizes the error of the loss function.

[0033] In addition, the learning data used for learning the generative model includes first learning data including a first image including a pathological image and pattern information mapped to the first image; and second learning data including a second image obtained by masking the pathological image according to a predetermined transparency and pattern information mapped to the second image, and the synthetic data output by the generative model may include a synthetic image restored according to the input first or second learning data and the first pattern information of the first or second learning data, and second pattern information for the synthetic image.

[0034] Additionally, the loss function may include a first loss between the pathological image and the synthetic image or a second loss between the first pattern information and the second pattern information.

[0035] In addition, the operation of training the generative model may include a first training operation of inputting the first training data up to a first number of times to train to minimize the first loss and the second loss simultaneously; a second training operation of inputting the second training data up to a second number of times after the first time to train to minimize the first loss and the second loss simultaneously; and a third training operation of inputting the second training data up to a third number of times after the second time to train to remove the first loss from the loss function and minimize the second loss.

[0036] In addition, the second learning operation may include an operation of gradually increasing the transparency of the masking for the pathological image of the second learning data based on a preset ratio as the learning round progresses; an operation of lowering the reflection ratio of the first loss among the loss functions in proportion to the transparency of the masking; and an operation of setting the reflection ratio of the second loss among the loss functions in proportion to the transparency of the masking.

[0037] Additionally, the third learning operation may include an operation of performing learning based on a second image having a transparency of masking for a pathological image of the second learning data of 100%.

[0038] A method performed by a synthetic data generating device operated by a processor according to one embodiment comprises: acquiring a plurality of pathological images including a time-series change pattern of a biological tissue; generating patches by dividing the pathological images into preset sizes; extracting pattern information including time-series information and information on a composition ratio of biological tissue included in each patch using a classification model; and training a generation model designed to generate synthetic data by sampling a change pattern according to the time-series information of the composition ratio of biological tissue for a pathological image based on the pattern information, wherein the generation model, when trained, can generate a plurality of synthetic data reflecting the time-series change pattern of the composition ratio of biological tissue based on target pattern information including the time-series information.

[0039] Additionally, the plurality of pathology images may include multiple pathology images acquired at different time points depending on the progression of the tumor tissue.

[0040] Additionally, the pattern information may include information on the composition ratio of normal tissue, the composition ratio of tumor tissue, and the composition ratio of moisture according to the time-series change pattern of tumor tissue included in the patch.

[0041] In addition, the above classification model is composed of a convolutional neural network, and learning can be performed through supervised learning that specifies pattern information including information on the composition ratio of biological tissue included in each patch and the progression of tumor tissue included in each patch based on the correct values ​​of the pattern information for the patches and the patches to be used for learning according to a deep learning algorithm.

[0042] In addition, the generative model may include an encoder that compresses learning data and converts it into latent variables based on parameters in a direction that minimizes the error of a loss function, including a Long Short-Term Memory (LSTM) network; and a decoder that generates synthetic data from the latent variables based on parameters in a direction that minimizes the error of the loss function.

[0043] In addition, the learning data used for learning the generative model includes first learning data including a first image including a pathological image and pattern information mapped to the first image; and second learning data including a second image obtained by masking the pathological image according to a predetermined transparency and pattern information mapped to the second image, and the synthetic data output by the generative model may include a synthetic image restored according to the input first or second learning data and the first pattern information of the first or second learning data, and second pattern information for the synthetic image.

[0044] Additionally, the loss function may include a first loss between the pathological image and the synthetic image or a second loss between the first pattern information and the second pattern information.

[0045] In addition, the operation of training the generative model may include a first training operation of inputting the first training data up to a first number of times to train to minimize the first loss and the second loss simultaneously; a second training operation of inputting the second training data up to a second number of times after the first time to train to minimize the first loss and the second loss simultaneously; and a third training operation of inputting the second training data up to a third number of times after the second time to train to remove the first loss from the loss function and minimize the second loss.

[0046] In addition, the second learning operation may include an operation of gradually increasing the transparency of the masking for the pathological image of the second learning data based on a preset ratio as the learning round progresses; an operation of lowering the reflection ratio of the first loss among the loss functions in proportion to the transparency of the masking; and an operation of setting the reflection ratio of the second loss among the loss functions in proportion to the transparency of the masking.

[0047] Additionally, the third learning operation may include an operation of performing learning based on a second image having a transparency of masking for a pathological image of the second learning data of 100%.

[0048]

[0049] The present invention provides a technology for generating patches by dividing a pathological image into preset sizes, extracting pattern information on the composition ratio of biological tissue included in each patch, and generating new synthetic data based thereon, thereby achieving various important effects.

[0050] First, the present invention segments a pathological image into small patches and extracts pattern information from each patch, thereby generating synthetic data that precisely reflects the compositional ratios and patterns of various biological tissues. This significantly increases the similarity of the synthesized data to real data, making it useful for training machine learning and deep learning models.

[0051] Furthermore, the present invention can significantly improve the efficiency of pathological image analysis by generating high-quality synthetic data containing diverse pattern information. This can reduce subjective judgments and errors that can arise during pathological image analysis, thereby enhancing the reliability of medical diagnoses and maximizing the effectiveness of patient treatment.

[0052] Furthermore, the present invention can more accurately simulate various pathological conditions by combining pattern information to generate enhanced pathological data. This can significantly improve the accuracy of research and diagnosis by supplementing data deficiencies in pathological research and diagnosis and enabling model learning that reflects diverse clinical scenarios.

[0053] Furthermore, the generative model created according to the present invention utilizes only target pattern information when generating synthetic data, thereby minimizing the risk of personal information being re-identified, thereby enabling secure data utilization. This ensures the safe utilization of medical data and addresses compliance with personal information protection laws. Furthermore, it can contribute to expanding the scope of medical data utilization and enhancing the efficiency of data-driven research.

[0054] Therefore, the present invention is expected to be useful in various medical and research fields by providing a new technology that can significantly improve the accuracy and efficiency of pathological image analysis, and to contribute to the advancement of pathological research and diagnosis.

[0055] Meanwhile, the effects of the present invention are not limited to those mentioned above, and other technical effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0056]

[0057] Figure 1 is a configuration diagram of a synthetic data generation device according to one embodiment.

[0058] FIG. 2 is a flowchart illustrating steps of operations performed by a synthetic data generation device according to one embodiment.

[0059] FIG. 3 is an exemplary diagram of an operation of segmenting a pathological image to generate multiple patches according to one embodiment.

[0060] FIG. 4 is an example diagram of an operation for extracting pattern information of each patch based on a classification model according to one embodiment.

[0061] FIG. 5 is an exemplary diagram of an operation of augmenting a pathological image by combining patches segmented from multiple pathological images according to one embodiment.

[0062] FIG. 6 is an exemplary diagram illustrating a first learning operation for training a generative model that generates synthetic data that follows pattern information of learning data according to one embodiment.

[0063] FIG. 7 is an exemplary diagram illustrating a second learning operation for training a generative model to generate synthetic data that follows pattern information of learning data according to one embodiment.

[0064] FIG. 8 is an example diagram of an operation for increasing the transparency of masking for a pathological image as learning rounds progress according to one embodiment.

[0065] FIG. 9 is an exemplary diagram illustrating a third learning operation for training a generative model to generate synthetic data that follows pattern information of learning data according to one embodiment.

[0066] FIG. 10 is an exemplary diagram showing an operation of a generative model that has completed learning according to one embodiment to generate synthetic data reflecting the composition ratio of biological tissue.

[0067] FIG. 11 is an exemplary diagram showing an operation of a generative model that has completed learning according to one embodiment to generate a plurality of synthetic data reflecting the time-series change pattern of the composition ratio of biological tissue.

[0068]

[0069] The purpose, technical configuration, and resulting operational effects of the present invention will be more clearly understood through the following detailed description based on the drawings attached to the specification of the present invention. Reference will now be made to the accompanying drawings, which will further describe embodiments of the present invention.

[0070] The embodiments disclosed herein should not be construed or used to limit the scope of the present invention. Those skilled in the art will readily appreciate that the descriptions herein, including the embodiments, have a wide range of applications. Therefore, any embodiments described in the detailed description of the present invention are intended to serve as illustrative examples to better illustrate the present invention and are not intended to limit the scope of the present invention to the embodiments.

[0071] The functional blocks depicted in the drawings and described below are merely examples of possible implementations. Other implementations may utilize other functional blocks without departing from the spirit and scope of the detailed description. Furthermore, while one or more functional blocks of the present invention are depicted as individual blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

[0072] Additionally, the expression “including certain components” is an “open” expression, simply indicating the presence of those components, and should not be understood as excluding additional components.

[0073] Furthermore, when it is said that a component is “connected” or “connected” to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0074] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that various modifications, equivalents, and / or alternatives of the embodiments of the present invention are included.

[0075] FIG. 1 is a configuration diagram of a synthetic data generation device (100) (hereinafter referred to as 'device (100)') according to one embodiment.

[0076] Referring to FIG. 1, a device (100) according to one embodiment may each include a memory (110), a processor (120), an input / output interface (130), and a communication interface (140).

[0077] The memory (110) can store data acquired from an external device or data generated by the device itself. The memory (110) can store commands that can perform operations of the processor (120). For example, the memory (110) can store pathological images, pattern information, classification models, and generation models, which will be described later.

[0078] The processor (120) is a computing device that controls overall operations. The processor (120) can execute instructions stored in the memory (110). The operation of the device (100) according to the embodiment of this document can be understood as an operation performed by the processor (120).

[0079] The input / output interface (130) may include a hardware interface or software interface for inputting or outputting information.

[0080] The communication interface (140) enables information to be transmitted and received via a communication network. To this end, the communication interface (140) may include a wireless communication module or a wired communication module.

[0081] The device (100) may be implemented in various forms of devices capable of performing operations through a processor (120) and transmitting and receiving information through a network. For example, the device may be implemented in the form of a server, a computer device, a portable communication device, a smart phone, a portable multimedia device, a laptop, a tablet PC, etc., but is not limited to these examples.

[0082] FIG. 2 is a flowchart of operations performed by a device (100) according to one embodiment. The operations of the device (100) according to the embodiment of FIG. 2 can be understood as operations performed by a processor (120).

[0083] Each step disclosed in FIG. 2 is merely a preferred embodiment for achieving the purpose of the present invention, and some steps may be added or deleted as needed, and one step may be included and performed within another step. The order of each operation disclosed in FIG. 2 is merely arranged for convenience of understanding, and this order is not limited to a chronological order, and the order may be changed and performed differently depending on the designer's choice.

[0084] Referring to FIG. 2, at step S1010, the device (100) can acquire pathological images to be used for learning to generate synthetic data. Synthetic data is virtual data created based on generative artificial intelligence technology that mimics real data. Synthetic data is used to have similar properties to real data but not to contain the same information, or to increase the quantity of data with similar properties to real data.

[0085] For example, the device (100) can acquire pathology images from a pathology laboratory, hospital, or public database. For example, the pathology images can include information about various organs (breast, stomach, liver, etc.) and tissues (normal tissue, tumor tissue, etc.).

[0086] In step S1020, the device (100) can generate patches by dividing the pathological image into preset sizes. For example, the device (100) can divide the pathological image into square or rectangular patches of a predetermined size.

[0087] FIG. 3 is an exemplary diagram of an operation of segmenting a pathological image to generate multiple patches according to one embodiment.

[0088] Referring to FIG. 3, the device (100) can assign windows of a predetermined size to an original pathological image. In the example of FIG. 3, in order to divide a pathological image of 512 x 512 pixels into four, four windows of 256 x 256 pixels are assigned to the upper left, upper right, lower left, and lower right of the pathological image of 512 x 512 pixels. Accordingly, the device (100) can capture each window and generate four patches corresponding to the assigned windows. These generated patches become a reference for extracting pattern information specifying the characteristics included in each patch in the analysis step described below.

[0089] At step S1030, the device (100) can extract pattern information including information on the composition ratio of the biological tissue included in each patch.

[0090] Pattern information refers to information that specifies the individual characteristics of tissues contained within a patch area. For example, pattern information may include information on the composition ratio of normal tissue, the composition ratio of tumor tissue, and the composition ratio of moisture contained within the patch.

[0091] Additionally, the pattern information may include information on the composition ratio of normal tissue, the composition ratio of tumor tissue, and the composition ratio of moisture according to the time-series change pattern of the tumor tissue included in the patch. When information on the time-series change pattern is added to the pattern information, this is to enable the synthetic data described later to be generated to reflect the time-series change pattern. In this embodiment, the original pathology data acquired in step S1010 may include multiple pathology data acquired by observing a single person at multiple time intervals (e.g., observing the progression of tumor tissue at multiple time points). Accordingly, the pattern information extracted from each of the pathology data measured at multiple time intervals based on a single person may further include information on the time at which the composition ratio of the biological tissue was measured in addition to the composition ratio of the biological tissue.

[0092] Meanwhile, at step S1030, the device (100) can extract pattern information of each patch using a classification model. The classification model is a machine learning model that outputs classification values ​​for several predefined categories based on given input data.

[0093] FIG. 4 is an example diagram of an operation for extracting pattern information of each patch based on a classification model according to one embodiment.

[0094] Referring to FIG. 4, the device (100) can configure a classification model with a convolutional neural network including an input layer, an output layer, and a hidden layer. The input layer can be set to input patches during the learning process. The output layer can be set to input correct values ​​of pattern information for each patch during the learning process. The hidden layer can learn the correlation between the patches input to the input layer and the pattern information for each patch input to the output layer during the learning process, and parameterize information about the correlation. Such a convolutional neural network can perform supervised learning to specify pattern information including information about the composition ratio of biological tissue included in each patch based on the correct values ​​of the pattern information for the patches and the patches to be used for learning according to a deep learning algorithm.

[0095] Accordingly, the classification model of FIG. 4 can classify the composition ratio of biological tissues (e.g., normal tissue, tumor tissue, water, etc.) included in each patch. For example, when input data called a patch is given, the device (100) can determine classification values ​​for categories such as the composition ratio of normal tissue, the composition ratio of tumor tissue, the progression information of tumor tissue, and the composition ratio of water based on the classification model. In addition, the device (100) can convert the classification values ​​for each category constituting the output pattern information into vector information expressed in a preset vector space.

[0096] Meanwhile, in an embodiment where the original pathology data includes multiple pathology data acquired by observing a single person at various time intervals, the classification model may be trained to extract pattern information that further includes information on the temporal aspect of the biological tissue included in each patch (e.g., progression of tumor tissue).

[0097] At step S1040, the device (100) can enhance the pathological image by combining patches of multiple pathological images based on pattern information of each patch.

[0098] FIG. 5 is an exemplary diagram of an operation of augmenting a pathological image by combining patches segmented from multiple pathological images according to one embodiment.

[0099] Referring to FIG. 5, the device (100) can generate a new pathological image by combining patches having similar patterns based on pattern information extracted from patches of multiple pathological data. For example, the device (100) can augment a pathological image by combining patches having a similarity within a preset range based on the similarity of the ratio constituting the pattern information of each patch. The reason for combining patches having similar pattern information is to generate an augmented image having natural tissue connectivity and to use the augmented image for learning a generation model to generate synthetic data.

[0100] For example, the device (100) can generate vector information by vectorizing the values ​​of each category constituting the pattern information, select patches whose Euclidean distance based on the vector information is less than or equal to a preset distance, and combine the selected patches to enhance the pathological image.

[0101] As another example, the device (100) can map vector information to a Euclidean space, select a preset number of patches existing within a preset range in the Euclidean space, and enhance a pathological image by combining the selected patches.

[0102] With this data augmentation, not only can the original pathological image and the pattern information of the original pathological image of step S1010 be used as learning data for the generation model of step S1050, which will be described later, but also, the augmented pathological image and the pattern information of the augmented pathological image of step S1040 can be used as learning data for the generation model of step S1050, which will be described later.

[0103] At step S1050, the device (100) can train a generative model designed to generate new synthetic data by sampling the composition ratio of biological tissue based on pattern information of a pathological image (e.g., an original pathological image or an enhanced pathological image).

[0104] Meanwhile, if the pattern information includes information on the time-series change pattern of the tissue, the generative model can train a generative model designed to generate multiple synthetic data reflecting the time-series change pattern by sampling the change pattern according to the time-series information of the composition ratio of the biological tissue for the pathological image based on the pattern information of the pathological image.

[0105] A generative model is a model that learns from given training data and generates synthetic data, which is similar data that follows the distribution of the training data. For example, a generative model may be trained to minimize the loss between the pattern information of an input image and the pattern information of an output image. Generative models can be designed based on various algorithms. For example, a generative model may include structures such as a Long Short-Term Memory (LSTM) autoencoder, a Variational Autoencoder (VAE), a Vision Transformer (ViT), or a Generative Adversarial Network (GAN).

[0106] For the convenience of understanding, the description and drawings according to the embodiments of this document will describe the generation model based on the LSTM autoencoder.

[0107] FIG. 6 is an exemplary diagram illustrating a first learning operation for training a generative model that generates synthetic data that follows pattern information of learning data according to one embodiment.

[0108] Before explaining the first learning operation, the structure of the generation model according to the embodiment of this document, which is commonly applied to FIGS. 6 to 11, will be explained.

[0109] The LSTM autoencoder of FIGS. 6 to 11 is a neural network structure that can effectively process time-series data. The LSTM autoencoder is useful when data contains temporal information, and it enables compression and restoration of latent variables by reflecting the time-series information. The LSTM autoencoder encodes the time-series sequence of input pattern information, converts it into a compressed latent vector, and decodes it to restore the original time-series sequence. Since both the encoder and decoder of the LSTM autoencoder are composed of LSTM cells, it can learn long-term dependencies and patterns of time-series data. For example, the LSTM encoder can compress pattern information containing texture information and time information into a latent variable Z by passing it through an LSTM layer. At this time, the LSTM encoder can learn the dependency of time information on texture information, and encode the long-term dependencies and patterns of time-series data into latent variables. Through this, the LSTM autoencoder can extract important features of data containing time series information included in the pattern information of the patch and generate synthetic data that reflects the pattern information.

[0110] An LSTM autoencoder may include an encoder that compresses training data and converts it into latent variables based on parameters in a direction that minimizes the error of a loss function, including an LSTM (Long Short-Term Memory) network, and a decoder that generates synthetic data from the latent variables based on parameters in a direction that minimizes the error of the loss function.

[0111] The learning data used for learning the generative model according to the embodiment of this document can be of two types: first learning data and second learning data.

[0112] The first learning data may include a first image including a pathological image without masking and pattern information of the pathological image.

[0113] The second learning data may include a second image in which the pathological image is masked according to a predetermined transparency and pattern information of the pathological image.

[0114] The synthetic data output by the decoder of the generation model based on the first learning data or the second learning data input to the encoder of the generation model may include a restored synthetic image and pattern information for the restored synthetic image.

[0115] Meanwhile, the training images input to the encoder of the generative model and the pattern information for the training images are referred to as "first pattern information." The synthetic images output by the decoder of the generative model and the pattern information for the synthetic images are referred to as "second pattern information."

[0116] Here, in one embodiment, the second pattern information may be set to "pattern information" directly extracted by the generation model from the synthetic data of the "synthetic image + pattern information" pair directly output by the generation model. In another embodiment, the second pattern information may be set to "pattern information" output by utilizing the classification model of S1030 for the "synthetic image" directly extracted by the generation model, ignoring the "pattern information" directly extracted from the "synthetic image + pattern information" pair directly output by the generation model.

[0117] That is, according to the embodiments of this document, the second pattern information may mean one of the two types described above. The embodiments of this document may configure the loss function described below by specifying the second pattern information based on either of these two types, and this may be selected by the designer designing the generative model.

[0118] The loss function of the generative model may include a first loss reflecting the difference between the pathological image and the synthetic image, and a second loss reflecting the difference between the first pattern information and the second pattern information.

[0119] In the embodiment of this document, the ratio of the first loss and the second loss reflected among the loss functions can be adjusted according to the learning process divided into the first learning operation to the third learning operation.

[0120] Additionally, in the embodiment of the document, the transparency of the masking for the learning image input to the generation model can be adjusted according to the learning process divided into the first learning operation to the third learning operation.

[0121] Referring back to FIG. 6, in the first learning operation, the device (100) can train the generative model so that the first loss and the second loss of the loss function are simultaneously minimized based on the first learning data up to a preset number of first iterations. At this time, various techniques such as the gradient descent method can be used to minimize the loss function.

[0122] Accordingly, in the first learning operation of FIG. 6, the generative model learns how to generate a synthetic image by reflecting the learning image and the pattern information of the learning image.

[0123] FIG. 7 is an exemplary diagram illustrating a second learning operation for training a generative model to generate synthetic data that follows pattern information of learning data according to one embodiment.

[0124] Referring to FIG. 7, as a second learning operation following the first round of the first learning operation process, the device (100) can train a generative model to simultaneously minimize the first loss and the second loss of the loss function based on the second learning data up to a preset number of second rounds. At this time, the device (100) can control the masking transparency of the second image among the second learning data as in the example of FIG. 8.

[0125] FIG. 8 is an example diagram of an operation for controlling the transparency of masking for a pathological image as learning rounds progress according to one embodiment.

[0126] Referring to FIG. 8, the device (100) may be configured to gradually increase the transparency of the masking for the pathological image of the second learning data based on a preset ratio as the learning rounds progress in the second learning operation. In addition, the device (100) may be configured to gradually increase the transparency of the pathological image, and at the same time, to lower the reflection ratio of the first loss among the loss functions of the generation model in the second learning operation in proportion to the transparency of the masking, and to increase the reflection ratio of the second loss in proportion to the transparency of the masking.

[0127] Accordingly, in the second learning operation of FIGS. 7 and 8, the generation model generates a synthetic image by reflecting the learning image and the pattern information of the learning image, but as the learning image is gradually masked and the reflection ratio of the second loss increases, the model learns by increasing the proportion of generating a synthetic image by referencing more pattern information than the learning image.

[0128] FIG. 9 is an exemplary diagram illustrating a third learning operation for training a generative model to generate synthetic data that follows pattern information of learning data according to one embodiment.

[0129] Referring to FIG. 9, as a third learning operation after the second round of the second learning operation process, the device (100) can train a generative model to eliminate the first loss in the loss function and minimize only the second loss based on the second learning data up to a preset number of third rounds. At this time, the device (100) can perform learning based on a second image in which the transparency of the masking for the pathological image of the second learning data is 100%.

[0130] Accordingly, in the third learning operation of Fig. 9, the generation model learns by reflecting only the reflection ratio of the second loss while the learning image is completely masked, so that it learns to generate a synthetic image by purely referring to pattern information. At this time, since the encoder and decoder of the generation model have been trained by the first loss according to the previous first and second learning operations, it can output a synthetic image using only pattern information even when the learning image is completely masked.

[0131] According to the above-described FIGS. 6 to 9, in the first learning operation, the generation model learns to understand the correlation between the learning image and the first pattern information to generate a synthetic image, in the second learning operation, the generation model understands the correlation between the learning image and the first pattern information by inferring a partially masked learning image from the first pattern information to generate a synthetic image, and in the third learning operation, the generation model learns the process of generating a synthetic image using only the first pattern information.

[0132] FIG. 10 is an exemplary diagram showing an operation of a generative model that has completed learning according to one embodiment to generate synthetic data reflecting the composition ratio of biological tissue.

[0133] Referring to FIG. 10, the generative model can generate synthetic data that reflects the composition ratio of biological tissues for a given target pattern information. The target pattern information is an input value for generating synthetic data using the generative model for which learning has been completed, and may include a target value set by the user of the generative model for a category corresponding to the pattern information.

[0134] Accordingly, the generation model of Fig. 10, which has completed learning, can generate a new synthetic image that reflects the composition ratio of the biological tissue desired by the user, using only target pattern information including the composition ratio of the biological tissue desired by the user, without any image information.

[0135] FIG. 11 is an exemplary diagram showing an operation of a generative model that has completed learning according to one embodiment to generate a plurality of synthetic data reflecting the time-series change pattern of the composition ratio of biological tissue.

[0136] Referring to Fig. 11, the generative model can generate synthetic data reflecting the time-series change aspect of the composition ratio of biological tissue based on target pattern information including time-series information. For example, if the generative model has an LSTM autoencoder structure, the LSTM encoder has learned the dependency of time information on tissue information, so it can encode the time-series sequence of target pattern information including tissue information and time information, convert it into a compressed latent vector, and decode it again to generate multiple synthetic data reflecting the time-series sequence included in the latent vector. Through this, the LSTM autoencoder can generate multiple synthetic data reflecting the time-series change aspect of biological tissue by reflecting the target pattern information including time-series information.

[0137] According to the above-described embodiment, the present invention provides a technology for generating patches by dividing a pathological image into preset sizes, extracting pattern information on the composition ratio of biological tissue included in each patch, and generating new synthetic data based thereon, thereby obtaining various important effects.

[0138] First, the present invention segments a pathological image into small patches and extracts pattern information from each patch, thereby generating synthetic data that precisely reflects the compositional ratios and patterns of various biological tissues. This significantly increases the similarity of the synthesized data to real data, making it useful for training machine learning and deep learning models.

[0139] Furthermore, the present invention can significantly improve the efficiency of pathological image analysis by generating high-quality synthetic data containing diverse pattern information. This can reduce subjective judgments and errors that can arise during pathological image analysis, thereby enhancing the reliability of medical diagnoses and maximizing the effectiveness of patient treatment.

[0140] Furthermore, the present invention can more accurately simulate various pathological conditions by combining pattern information to generate enhanced pathological data. This can significantly improve the accuracy of research and diagnosis by supplementing data deficiencies in pathological research and diagnosis and enabling model learning that reflects diverse clinical scenarios.

[0141] Furthermore, the generative model created according to the present invention utilizes only target pattern information when generating synthetic data, thereby minimizing the risk of personal information being re-identified, thereby enabling secure data utilization. This ensures the safe utilization of medical data and addresses compliance with personal information protection laws. Furthermore, it can contribute to expanding the scope of medical data utilization and enhancing the efficiency of data-driven research.

[0142] Therefore, the present invention is expected to be useful in various medical and research fields by providing a new technology that can significantly improve the accuracy and efficiency of pathological image analysis, and to contribute to the advancement of pathological research and diagnosis.

[0143] The various embodiments and terms used in this document are not intended to limit the technical features described in this document to specific embodiments, but should be understood to encompass various modifications, equivalents, or alternatives of the embodiments. In connection with the description of the drawings, similar reference numerals may be used to refer to similar or related components. The singular form of a noun corresponding to an item may include one or more items, unless the context clearly indicates otherwise.

[0144] In this document, the phrases "A or B", "at least one of A and B", "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can each include all possible combinations of the items listed together in that phrase. Terms such as "1", "2", or "first" or "second" may be used merely to distinguish the corresponding component from other corresponding components and do not limit the corresponding components in any other respect (e.g., importance or order). When a (e.g., a first) component is referred to as "coupled" or "connected" to another (e.g., a second) component, with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0145] The term "module" as used herein may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0146] Various embodiments of the present document may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium (e.g., a memory) readable by a device (e.g., an electronic device). The storage medium may include random access memory (RAM), a memory buffer, a hard drive, a database, erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), read-only memory (ROM), and / or the like.

[0147] Additionally, the processor of the embodiments of the present document can call at least one command among one or more commands stored from a storage medium and execute it. This enables the device to operate to perform at least one function according to the at least one command called. These one or more commands may include code generated by a compiler or code executable by an interpreter. The processor may be a general-purpose processor, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), and / or the like.

[0148] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0149] The methods according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated on a machine-readable storage medium, such as a manufacturer's server, an application store's server, or a server's memory.

[0150] According to various embodiments, each component (e.g., a module or a program) of the described components may include one or more entities. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. A method performed by a synthetic data generation device operated by a processor, The act of acquiring pathological images for generating synthetic data; An operation of generating a patch by dividing the above pathological image into preset sizes; An operation of extracting pattern information of each patch based on a classification model learned to extract pattern information including information on the composition ratio of biological tissue included in the patch; and Includes an operation of training a generative model designed to generate synthetic data by sampling the composition ratio of biological tissue for a pathological image based on the above pattern information, The generated model after completing the above learning is Generating synthetic data that reflects the composition ratio of biological tissues with respect to target pattern information. method.

2. In paragraph 1, The above pattern information is Contains information on at least one of the composition ratio of normal tissue, the composition ratio of tumor tissue and the composition ratio of moisture contained in the patch. method.

3. In paragraph 2, The above classification model is Learning is performed through supervised learning that specifies pattern information that includes information on the composition ratio of biological tissues included in each patch based on the correct values ​​of pattern information for the patches and patches to be used for learning according to the deep learning algorithm, which consists of a convolutional neural network. method.

4. In paragraph 3, The above classification model is Input layer set to receive patches during the learning process; An output layer set to input the correct value of the pattern information for each patch of the learning process; and In the learning process, a hidden layer is included, which consists of parameters that learn the correlation between the patches input to the input layer and the pattern information for each patch input to the output layer. method.

5. In paragraph 1, The above generation model is An encoder that compresses training data and converts it into latent variables based on parameters that minimize the error of the loss function; and A decoder that generates synthetic data from the latent variable based on parameters in a direction that minimizes the error of the loss function, method.

6. In paragraph 5, The training data used to train the above generative model is: A first image including a pathological image and first learning data including pattern information mapped to the first image; and A second image comprising a pathological image masked according to a predetermined transparency and second learning data including pattern information mapped to the second image, The synthetic data output by the above generation model is A synthetic image restored according to input first or second learning data and first pattern information of the first or second learning data, and second pattern information for the synthetic image, method.

7. In paragraph 6, The above loss function is Including a first loss between the pathological image and the synthetic image or a second loss between the first pattern information and the second pattern information, method.

8. In paragraph 7, The operation of training the above generative model is A first learning operation that inputs the first learning data up to a preset number of first rounds to learn to simultaneously minimize the first loss and the second loss; After the first round, a second learning operation is performed to input the second learning data up to a preset number of second rounds to simultaneously minimize the first loss and the second loss; and After the second round, a third learning operation is included that inputs the second learning data up to a preset number of third rounds to learn to remove the first loss from the loss function and minimize the second loss. method.

9. In paragraph 8, The above second learning action is An operation for gradually increasing the transparency of the masking for the pathological image of the second learning data based on a preset ratio as the learning round progresses; An operation of lowering the reflection ratio of the first loss among the above loss functions in proportion to the transparency of the masking; and An operation including setting the reflection ratio of the second loss among the above loss functions to be increased in proportion to the transparency of the masking, method.

10. In paragraph 9, The third learning action above is Including an operation of performing learning based on a second image having a transparency of 100% for masking the pathological image of the second learning data. method.

11. In a method performed by a synthetic data generation device operated by a processor, The act of acquiring original pathological images for generating synthetic data; An operation of generating a patch by dividing the above pathological image into preset sizes; An operation of extracting pattern information containing information about the composition ratio of biological tissue contained in each patch; An operation of augmenting a pathological image by combining patches of multiple pathological images based on pattern information of each patch; and Includes an operation of training a generative model designed to generate synthetic data by sampling the composition ratio of biological tissues for the pathological images based on pattern information of the original and augmented pathological images, The generated model after completing the above learning is Generating synthetic data that reflects the composition ratio of biological tissues with respect to target pattern information. method.

12. In paragraph 11, The above pattern information is Contains information on at least one of the composition ratio of normal tissue, the composition ratio of tumor tissue and the composition ratio of moisture contained in the patch. method.

13. In paragraph 12, The action of enhancing the above pathological image is An operation of enhancing a pathological image by combining patches having a similarity in a preset range based on the similarity of the ratio constituting the pattern information of each patch, method.

14. In paragraph 13, The action of enhancing the above pathological image is An operation of generating vector information by vectorizing the values ​​of each category constituting the above pattern information; An operation of selecting a patch whose Euclidean distance based on the above vector information is less than or equal to a preset distance; and An operation that includes enhancing a pathological image by combining the above-mentioned selected patches. method.

15. In paragraph 13, The action of enhancing the above pathological image is An operation of generating vector information by vectorizing the values ​​of each category constituting the above pattern information; An operation of mapping the above vector information to a Euclidean space; An operation of selecting a preset number of patches existing within a preset range in the above Euclidean space; and An operation that includes enhancing a pathological image by combining the above-mentioned selected patches. method.

16. In paragraph 11, The above generation model is An encoder that compresses training data and converts it into latent variables based on parameters that minimize the error of the loss function; and A decoder that generates synthetic data from the latent variable based on parameters in the direction of minimizing the error of the loss function, The training data used to train the above generative model is: A first image including a pathological image and first learning data including pattern information mapped to the first image; and A second image comprising a pathological image masked according to a predetermined transparency and second learning data including pattern information mapped to the second image, The synthetic data output by the above generation model is A synthetic image restored according to input first or second learning data and first pattern information of the first or second learning data, and second pattern information for the synthetic image, The above loss function is Including a first loss between the pathological image and the synthetic image or a second loss between the first pattern information and the second pattern information, method.

17. In paragraph 16, The operation of training the above generative model is A first learning operation that inputs the first learning data up to a preset number of first rounds to learn to simultaneously minimize the first loss and the second loss; After the first round, a second learning operation is performed to input the second learning data up to a preset number of second rounds to simultaneously minimize the first loss and the second loss; and After the second round, a third learning operation is included that inputs the second learning data up to a preset number of third rounds to learn to remove the first loss from the loss function and minimize the second loss. method.

18. In paragraph 17, The above second learning action is An operation for gradually increasing the transparency of the masking for the pathological image of the second learning data based on a preset ratio as the learning round progresses; An operation of lowering the reflection ratio of the first loss among the above loss functions in proportion to the transparency of the masking; and An operation including setting the reflection ratio of the second loss among the above loss functions to be increased in proportion to the transparency of the masking, method.

19. In paragraph 18, The third learning action above is Including an operation of performing learning based on a second image having a transparency of 100% for masking the pathological image of the second learning data. method.

20. A method for performing a synthetic data generation device operated by a processor, An operation of acquiring multiple pathological images including time-series changes in biological tissue; An operation of generating a patch by dividing the above pathological image into preset sizes; An operation of extracting pattern information including information on the composition ratio of biological tissues included in each patch and time-series information using a classification model; and Includes an operation of training a generative model designed to generate synthetic data by sampling the change pattern according to the time-series information of the composition ratio of biological tissue for the pathological image based on the above pattern information, The generated model after completing the above learning is Generating multiple synthetic data reflecting the time-series change pattern of the composition ratio of biological tissue based on target pattern information including time-series information. method.

21. In paragraph 20, The above multiple pathological images are Contains multiple pathological images acquired at different time points depending on the progression of the tumor tissue, method.

22. In paragraph 21, The above pattern information is Contains information on the composition ratio of normal tissue, composition ratio of tumor tissue and composition ratio of moisture according to the time-series change pattern of tumor tissue included in the patch. method.

23. In paragraph 22, The above classification model is Learning is performed through supervised learning that specifies pattern information including information on the composition ratio of biological tissue included in each patch and the progression of tumor tissue included in each patch based on the correct values ​​of pattern information for the patches and the patches to be used for learning according to the deep learning algorithm, which consists of a convolutional neural network. method.

24. In paragraph 20, The above generation model is An encoder that compresses learning data and converts it into latent variables based on parameters that minimize the error of the loss function, including an LSTM (Long Short-Term Memory) network; and A decoder that generates synthetic data from the latent variable based on parameters in the direction of minimizing the error of the loss function, The training data used to train the above generative model is: A first image including a pathological image and first learning data including pattern information mapped to the first image; and A second image comprising a pathological image masked according to a predetermined transparency and second learning data including pattern information mapped to the second image, The synthetic data output by the above generation model is A synthetic image restored according to input first or second learning data and first pattern information of the first or second learning data, and second pattern information for the synthetic image, The above loss function is Including a first loss between the pathological image and the synthetic image or a second loss between the first pattern information and the second pattern information, method.

25. In paragraph 24, The operation of training the above generative model is A first learning operation that inputs the first learning data up to a preset number of first rounds to learn to simultaneously minimize the first loss and the second loss; After the first round, a second learning operation is performed to input the second learning data up to a preset number of second rounds to simultaneously minimize the first loss and the second loss; and After the second round, a third learning operation is included that inputs the second learning data up to a preset number of third rounds to learn to remove the first loss from the loss function and minimize the second loss. method.

26. In paragraph 25, The above second learning action is An operation for gradually increasing the transparency of the masking for the pathological image of the second learning data based on a preset ratio as the learning round progresses; An operation of lowering the reflection ratio of the first loss among the above loss functions in proportion to the transparency of the masking; and An operation including setting the reflection ratio of the second loss among the above loss functions to be increased in proportion to the transparency of the masking, method.

27. In paragraph 26, The third learning action above is Including an operation of performing learning based on a second image having a transparency of 100% for masking the pathological image of the second learning data. method.

Images