Method and apparatus for analyzing pathology slide images

Abstract

A computing device according to one embodiment includes at least one memory and at least one processor, wherein the processor acquires a first pathology slide image depicting at least one first object and biological information of at least one of the first objects, generates learning data using at least one first patch and the biological information included in the first pathology slide image, learns a first machine learning model using the learning data, and analyzes a second pathology slide image using the learned first machine learning model.

Description

[Technical field]

[0001] The present disclosure relates to methods and apparatus for analyzing pathology slide images. [Background technology]

[0002] The field of digital pathology is a field in which whole slide images generated by scanning pathological slide images are used to obtain histological information of a subject or to predict the prognosis.

[0003] Pathology slide images can be obtained from stained tissue samples from a subject. For example, tissue samples can be stained with a variety of staining techniques, including hematoxylin and eosin, trichrome, periodic acid Schiff, autoradiography, enzyme histochemistry, immunofluorescence, and immunohistochemistry. Stained tissue samples can be used for histology and biopsy evaluation to provide the basis for deciding whether to proceed with molecular profiling to understand disease states.

[0004] Recognizing and detecting biological elements from pathology slide images has an important impact on the histological diagnosis, prognosis prediction, and treatment direction of certain diseases. However, if the performance of a machine learning model to detect or segment biological elements from pathology slide images is low, it can be an obstacle to accurate treatment planning for subjects. On the other hand, in order to improve the performance of a machine learning model, it is necessary to prepare a large amount of annotation data, but the preparation process is costly. Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention provides a method and device for analyzing a pathology slide image. It also provides a computer-readable recording medium having a program for executing the method on a computer. The technical problem to be solved is not limited to the above technical problem, and other technical problems may exist. [Means for solving the problem]

[0006] A computing device according to one embodiment includes at least one memory and at least one processor, wherein the processor acquires a first pathology slide image depicting at least one first object and biological information of at least one of the first objects, generates learning data using at least one first patch and the biological information included in the first pathology slide image, learns a first machine learning model using the learning data, and analyzes a second pathology slide image using the learned first machine learning model.

[0007] A method for analyzing pathology slide images according to another aspect includes the steps of obtaining a first pathology slide image depicting at least one first object and biological information of at least one of the first objects, generating training data using at least one first patch and the biological information contained in the first pathology slide image, training a first machine learning model using the training data, and analyzing a second pathology slide image using the trained first machine learning model.

[0008] A computer-readable recording medium according to yet another aspect includes a recording medium having a program recorded thereon for executing the above-described method on a computer. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a system for analyzing a pathology slide image according to an embodiment; [Diagram 2] FIG. 1 is a block diagram of a system and network for preparing, processing, and reviewing slide images of tissue specimens using machine learning models according to one embodiment; [Figure 3a] FIG. 1 is a block diagram illustrating an example of a user terminal according to an embodiment; [Figure 3b] FIG. 1 is a block diagram illustrating an example of a server according to an embodiment. [Figure 4] 1 is a flowchart illustrating an example of a method for processing pathology slide images according to one embodiment; [Diagram 5] FIG. 1 is a diagram for explaining an example of biological information according to an embodiment; [Figure 6] 1 is a flowchart illustrating an example of a processor obtaining spatial transcriptomics information according to an embodiment; [Figure 7] FIG. 1 is a diagram for explaining an example of learning data according to an embodiment; [Figure 8] 10 is a flowchart illustrating another example of a method for processing a pathology slide image according to an embodiment. [Figure 9] FIG. 1 is a diagram illustrating an example in which a processor predicts a subject's treatment response according to an embodiment. [Figure 10] FIG. 1 is a diagram for explaining an example in which a processor learns a first machine learning model according to an embodiment. [Figure 11] FIG. 13 is a diagram for explaining another example in which a processor learns a first machine learning model according to an embodiment. [Figure 12] FIG. 1 is a diagram for explaining an example in which the operation of a processor according to an embodiment is realized; [Figure 13a] FIG. 1 is a diagram for explaining an example in which annotations are generated based on user input according to an embodiment; [Figure 13b] FIG. 1 is a diagram for explaining an example in which annotations are generated based on user input according to an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (Best Mode for Carrying Out the Invention) A computing device according to one embodiment includes at least one memory and at least one processor, wherein the processor acquires a first pathology slide image depicting at least one first object and biological information of at least one of the first objects, generates learning data using at least one first patch and the biological information included in the first pathology slide image, learns a first machine learning model using the learning data, and analyzes a second pathology slide image using the learned first machine learning model.

[0011] The terms used in the embodiments are generally used as widely as possible, but may change depending on the intentions of the engineers in the relevant technical field, legal precedents, the emergence of new technologies, etc. In addition, in certain cases, the applicant may arbitrarily select terms, and in such cases, the meanings of such terms are described in detail in the relevant description. Therefore, the terms used in the specification should be defined based on the meanings of the terms and the overall content of the specification, rather than simply the names of the terms.

[0012] Throughout the specification, when a part "includes" a certain component, it means that it can further include other components, not excluding other components, unless otherwise specified. Furthermore, the terms "unit", "module", etc. described in the specification mean a unit that processes at least one function or operation, which may be realized by hardware or software, or a combination of hardware and software.

[0013] Furthermore, terms including ordinal numbers such as "first" or "second" may be used in the specification to describe various components, but the components should not be limited by the terms. The terms may be used to distinguish one component from another.

[0014] According to one embodiment, a "pathology slide image" may refer to an image of a pathology slide that has been fixed and stained by subjecting tissue, etc., taken from a human body to a series of chemical processing steps. Also, a pathology slide image may refer to a whole slide image (WSI) including a high-resolution image of the entire slide, or may refer to a portion of the whole slide image, for example, one or more patches. For example, a pathology slide image may refer to a digital image taken or scanned by a scanning device (e.g., a digital scanner, etc.), and may include information about a particular protein, cell, tissue, and / or structure in the human body. Furthermore, a pathology slide image may include one or more patches, and one or more patches may have histological information applied (e.g., tagged) by an annotation operation.

[0015] According to an embodiment, the "medical information" may refer to any medically meaningful information that can be extracted from a medical image, and may include, but is not limited to, the area, position, size, cancer diagnosis information, information regarding the possibility of cancer development in a subject, and / or medical conclusions related to cancer treatment of a specific tissue (e.g., cancer tissue, cancer stromal tissue, etc.) and / or a specific cell (e.g., tumor cell, lymphocyte cell, macrophage cell, endothelial cell, fibroblast cell, etc.) in the medical image. In addition, the medical information may include not only quantified values ​​obtained from the medical image, but also visualized information of the values, prediction information according to the values, image information, statistical information, etc. The medical information generated in this manner may be provided to a user terminal, or output or transmitted to a display device and displayed.

[0016] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings. However, the embodiments may be embodied in various different forms and should not be construed as limited to the examples set forth herein.

[0017] FIG. 1 is a diagram illustrating an example of a system for analyzing a pathology slide image according to an embodiment.

[0018] 1, a system 1 includes a user terminal 10 and a server 20. For example, the user terminal 10 and the server 20 are connected by wired or wireless communication and can transmit and receive data (e.g., video data, etc.) to and from each other.

[0019] 1 illustrates the system 1 including a user terminal 10 and a server 20, but is not limited thereto. For example, the system 1 may include other external devices (not shown), and the operations of the user terminal 10 and the server 20 described below may be realized by a single device (e.g., the user terminal 10 or the server 20) or more devices.

[0020] The user terminal 10 may be a computing device including a display device and a device for receiving user input (e.g., a keyboard, a mouse, etc.), and including a memory and a processor. For example, the user terminal 10 may be, but is not limited to, a notebook PC, a desktop PC, a laptop computer, a tablet computer, a smartphone, etc.

[0021] The server 20 may be a device that communicates with an external device (not shown) including the user terminal 10. As an example, the server 20 may be a device that stores various data including a pathology slide image, a bitmap image corresponding to the pathology slide image, information generated by analyzing the pathology slide image (including, for example, information on at least one tissue and cell represented in the pathology slide image, biomarker expression information, etc.), and information on a machine learning model used for analyzing the pathology slide image. Alternatively, the server 20 may be a computing device including a memory and a processor and having its own computing power. When the server 20 is a computing device, the server 20 can perform at least a part of the operation of the user terminal 10 described later with reference to Figs. 1 to 13b. For example, the server 20 may be a cloud server, but is not limited thereto.

[0022] The user terminal 10 outputs an image showing a pathology slide image and / or information generated by analysis of the pathology slide. For example, the image may show various information regarding at least one tissue and cell represented in the pathology slide image. The image may also show biomarker expression information. Furthermore, the image may be a report including medical information regarding at least a portion of an area included in the pathology slide image.

[0023] The pathology slide image may refer to an image of a pathology slide that has been fixed and stained by a series of chemical processing steps in order to observe tissues, etc., taken from a human body under a microscope. As one example, the pathology slide image may refer to a whole slide image including a high-resolution image of the entire slide. As another example, the pathology slide image may refer to a portion of such a high-resolution whole slide image.

[0024] On the other hand, the pathology slide image may show patch regions in the entire slide image divided into patch units. For example, the patch may have a size of a predetermined area. Alternatively, the patch may show an area including each of the objects included in the entire slide.

[0025] Additionally, a pathology slide image may refer to a digital image taken using a microscope and may contain information about cells, tissues, and / or structures within the human body.

[0026] By analyzing the pathology slide images, biological elements (e.g., cancer cells, immune cells, cancer regions, etc.) shown in the pathology slide images can be identified. Such biological elements can be used for histological diagnosis of diseases, prediction of prognosis of diseases, and determination of the direction of treatment for diseases.

[0027] Meanwhile, machine learning models can be used in the analysis of pathology slide images. Here, the machine learning models must be trained to recognize biological elements from pathology slide images. The training data often relies on annotation work performed by experts (e.g., pathologists, etc.) on pathology slide images. Here, the annotation work includes the expert marking the positions and types of cells and / or tissues depicted in the pathology slide images one by one.

[0028] However, because the standards of each expert are different, the annotation results are difficult to provide unified information. Also, since the improvement of the performance of a machine learning model is proportional to the amount of annotation work, a large cost must be allocated to the annotation work in order to generate a high-performance machine learning model.

[0029] According to an embodiment, a user terminal 10 analyzes a pathology slide image using a machine learning model. Here, the user terminal 10 generates training data using a pathology slide image showing an object and biological information of the object, and trains a machine learning model using the training data.

[0030] Therefore, unlike conventional machine learning model learning that relies on annotation work by experts, the user terminal 10 can improve the performance of the machine learning model without annotation work (or even with a small amount of annotation results). This improves the accuracy of the analysis results of the pathology slide images by the machine learning model. In addition, since the user terminal 10 can predict the therapeutic reaction of the subject using the analysis results of the pathology slide images, the accuracy of the predicted results of the therapeutic reaction is also guaranteed.

[0031] As an example, the user terminal 10 can generate learning data by utilizing spatial transcriptomics information of a subject. Thus, unlike conventional learning data that relies on annotation work by experts, the problem of the performance of a machine learning model deteriorating due to differences in the standards of experts is eliminated. In addition, by utilizing the spatial transcriptomics information, spatial gene expression information can be obtained from a pathology slide image. Furthermore, in the case of the spatial transcriptomics process, a single spot may be set to include several cells. Thus, gene expression information obtained within a single spot can be more objective information than judgment based on the visual perception ability of an expert.

[0032] As another example, the user terminal 10 can generate training data using pathology slide images in which the subject is stained in different ways. Depending on the staining method, different biological elements (e.g., proteins located in cell membranes or cell nuclei) are expressed in specific colors in the pathology slide images. Therefore, different biological elements can be identified from the pathology slide images stained in different ways. As a result, when the pathology slide images stained in different ways are used as training data, the performance of the machine learning model is improved.

[0033] Hereinafter, with reference to Figs. 2 to 13b, an example will be described in which the user terminal 10 learns a machine learning model, analyzes a pathology slide image using the learned machine learning model, and predicts a subject's treatment response using the analysis result.

[0034] Meanwhile, for convenience of explanation, the description will be made throughout the specification of the user terminal 10 learning a machine learning model, analyzing a pathology slide image using the learned machine learning model, and predicting a subject's treatment response using the analysis result, but is not limited thereto. For example, at least a part of the operations performed by the user terminal 10 may be performed by the server 20.

[0035] In other words, at least a part of the operation of the user terminal 10 described with reference to Figs. 1 to 13b may be performed by the server 20. For example, the server 20 can generate learning data using a pathology slide image showing a target object and biological information of the target object. The server 20 can also learn a machine learning model using the learning data. Furthermore, the server 20 can analyze the pathology slide image using the learned machine learning model and transmit the analysis result to the user terminal 10. Furthermore, the server 20 can predict the treatment response of the subject using the analysis result and transmit the prediction result to the user terminal 10. However, the operation of the server 20 is not limited to the above.

[0036] FIG. 2 is a block diagram of a system and network for preparing, processing, and reviewing tissue specimen slide images using machine learning models according to one embodiment.

[0037] 2, the system 2 includes user terminals 11 and 12, a scanner 50, an image management system 61, an AI-based biomarker analysis system 62, a laboratory information management system 63, and a server 70. The components (reference numerals 11, 12, 50, 61, 62, 63, and 70) included in the system 2 can be connected to each other via a network 80. For example, the network 80 may be a network that can connect the components (reference numerals 11, 12, 50, 61, 62, 63, and 70) to each other by wired or wireless communication. For example, the system 2 shown in FIG. 2 may include a network that can be connected to a server in a hospital, a laboratory, a laboratory, or the like, and / or a user terminal of a doctor or researcher.

[0038] According to various embodiments of the present disclosure, the methods described below with reference to Figures 3a to 13b can be performed in a user terminal 11, 12, an image management system 61, an AI-based biomarker analysis system 62, a laboratory information management system 63, and / or a hospital or laboratory server 70.

[0039] The scanner 50 can acquire digitized images from tissue sample slides generated with tissue samples from the subject 90. For example, the scanner 50, the user terminals 11, 12, the image management system 61, the AI-based biomarker analysis system 62, the laboratory information management system 63, and / or the hospital or laboratory server 70 can each connect to a network 80, such as the Internet, via one or more computers, servers, and / or mobile devices, or communicate with the user 30 and / or the subject 90 via one or more computers and / or mobile devices.

[0040] The user terminals 11, 12, the image management system 61, the AI-based biomarker analysis system 62, the laboratory information management system 63, and / or the hospital or laboratory server 70 can generate or otherwise acquire from other devices tissue samples, tissue sample slides, digitized images of tissue sample slides, or any combination thereof, of one or more subjects 90. Additionally, the user terminals 11, 12, the image management system 61, the AI-based biomarker analysis system 62, and the laboratory information management system 63 can acquire any combination of subject-specific information, such as the subject's 90 age, disease severity, cancer treatment history, family history, past biopsy records, disease information of the subject 90, etc.

[0041] The scanner 50, the user terminals 11, 12, the image management system 61, the laboratory information management system 63, and / or the hospital or laboratory server 70 can transmit the digitized slide images and / or the subject-specific information to the AI-based biomarker analysis system 62 via the network 80. The AI-based biomarker analysis system 62 can include one or more storage devices (not shown) for storing images and data received from at least one of the scanner 50, the user terminals 11, 12, the image management system 61, the laboratory information management system 63, and / or the hospital or laboratory server 70. The AI-based biomarker analysis system 62 can also include a machine learning model repository for storing machine learning models trained to process the received images and data. For example, the AI-based biomarker analysis system 62 can include a machine learning model that has been learned and trained to predict at least one of information about at least one cell, information about at least one region, information about a biomarker, medical diagnosis information, and / or medical treatment information from a pathology slide image of the subject 90.

[0042] The scanner 50, user terminals 11, 12, AI-based biomarker analysis system 62, laboratory information management system 63, and / or hospital or laboratory server 70 can transmit digitized slide images, subject identification information, and / or results of analysis of the digitized slide images to the image management system 61 via network 80. The image management system 61 can include a repository for storing received images and a repository for storing results of the analysis.

[0043] Additionally, according to various embodiments of the present disclosure, machine learning models learned and trained to predict at least one of information regarding at least one cell, information regarding at least one region, information regarding biomarkers, medical diagnosis information and / or medical treatment information from slide images of subject 90 can be stored and operated in user terminal 11, 12 and / or image management system 61.

[0044] According to various embodiments of the present disclosure, the methods for analyzing pathology slide images, processing subject information, selecting subject groups, designing clinical trials, generating biomarker expression information, and / or setting reference values ​​for specific biomarkers can be performed not only in the AI-based biomarker analysis system 62, but also in the user terminals 11, 12, the image management system 61, the laboratory information management system 63, and / or the hospital or laboratory server 70.

[0045] FIG. 3a is a block diagram illustrating an example of a user terminal according to an embodiment.

[0046] Referring to Fig. 3a, the user terminal 100 includes a processor 110, a memory 120, an input / output interface 130, and a communication module 140. For convenience of explanation, only components related to the present invention are shown in Fig. 3a. Therefore, the user terminal 100 may further include other general-purpose components in addition to the components shown in Fig. 3a. It is obvious to a person skilled in the art related to the present invention that the processor 110, the memory 120, the input / output interface 130, and the communication module 140 shown in Fig. 3a may be realized as independent devices.

[0047] It should be noted that the operation of the user terminal 100 can be performed by the user terminals 11, 12, the image management system 61, the AI-based biomarker analysis system 62, the laboratory information management system 63, and / or the hospital or laboratory server 70 in FIG.

[0048] The processor 110 can process instructions of a computer program by performing basic arithmetic, logic, and input / output operations, where the instructions can be provided from the memory 120 or an external device (e.g., the server 20, etc.), and can generally control the operation of other components included in the user terminal 100.

[0049] The processor 110 may obtain a first pathology slide image depicting at least one first object and biological information of the at least one first object. For example, the biological information may include at least one of information identified from a third pathology slide image and spatial transcriptomics information of the first object. Here, the third pathology slide image may include an image stained in a manner distinct from the first pathology slide image.

[0050] The processor 110 can also generate training data using at least one first patch and biological information included in the first pathology slide image. For example, the training data can include at least one of gene expression information corresponding to the first patch and at least one of the types of cells represented in the first patch.

[0051] Also, the processor 110 may train a first machine learning model using the training data and analyze a second pathology slide image using the trained first machine learning model. As an example, the processor 110 may train the first machine learning model using the training data as ground truth data. As another example, the processor 110 may train the first machine learning model using at least one annotation generated based on a user input as ground truth data. As yet another example, the processor 110 may train the first machine learning model using the training data and at least one annotation as ground truth data.

[0052] Meanwhile, the processor 110 can generate a second machine learning model by adding or removing at least one layer included in the trained first machine learning model, where the second machine learning model can be used to identify at least one cell type represented in the second pathology slide image.

[0053] The processor 110 may also predict a treatment response of a subject corresponding to the second pathology slide image using spatial transcriptomics information of the second subject represented in the second pathology slide image, where the spatial transcriptomics information of the second subject may include at least one of spatial transcriptomics information obtained by the trained first machine learning model and spatial transcriptomics information obtained separately.

[0054] For example, the prediction of the treatment response may be performed by a third machine learning model. As an example, the third machine learning model may be trained using a feature vector extracted from at least one layer included in the trained first machine learning model. As another example, the third machine learning model may be trained using gene expression information included in the spatial transcriptomics information and location information corresponding to the gene expression information.

[0055] Furthermore, the processor 110 can use the first patch and the second patch included in the third pathology slide image as learning data for training the first machine learning model. Alternatively, the processor 110 can use a third patch obtained by image processing the first patch and the second patch as learning data for training the first machine learning model. Here, the second patch can include a patch indicating a position corresponding to the first patch. Alternatively, the processor 110 can use the first patch and at least one annotation generated based on a user input as learning data for training the first machine learning model. Here, the at least one annotation may be generated based on the third pathology slide image.

[0056] Here, the machine learning model means, in machine learning technology and cognitive science, a statistical learning algorithm realized based on the structure of a biological neural network, or a structure that executes the algorithm.

[0057] For example, the machine learning model can show a model having a problem-solving ability by learning so that a node, which is an artificial neuron that forms a network by synaptic connections, repeatedly adjusts the weight value of the synapse and reduces the error between the correct output corresponding to a specific input and the inferred output, like a biological neural network. For example, the machine learning model can include any probability model, neural network model, etc. used in artificial intelligence learning methods such as machine learning and deep learning.

[0058] For example, the machine learning model may be realized by a multilayer perceptron (MLP) consisting of multiple nodes and connections between them. The machine learning model according to the present embodiment may be realized using one of various artificial neural network model structures including MLP. For example, the machine learning model may be composed of an input layer that receives an input signal or data from the outside, an output layer that outputs an output signal or data corresponding to the input data, and at least one hidden layer that is located between the input layer and the output layer and receives a signal from the input layer, extracts characteristics, and sends them to the output layer. The output layer receives a signal or data from the hidden layer and outputs it to the outside.

[0059] Thus, the machine learning model may be trained to receive one or more pathology slide images and extract information about at least one object (e.g., a cell, tissue, structure, etc.) contained in the pathology slide image.

[0060] The processor 110 may be implemented with an array of logic gates or may be implemented with a general-purpose microprocessor and a memory having a program executable by the microprocessor stored therein. For example, the processor 110 may include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the processor 110 may also include an application specific semiconductor (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, the processor 110 may represent a combination of processing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors coupled to a digital signal processor (DSP) core, or any other such configuration.

[0061] The memory 120 may include any non-transitory computer-readable recording medium. For example, the memory 120 may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), or a flash memory. For another example, the permanent mass storage device such as a ROM, an SSD, a flash memory, or a disk drive may be a separate permanent storage device distinct from the memory. In addition, the memory 210 may store an operating system (OS) and at least one program code (for example, code for the processor 110 to perform the operations described below with reference to FIGS. 4 to 13b).

[0062] Such software components may be loaded from a computer-readable recording medium other than the memory 120. Such a separate computer-readable recording medium may be a recording medium that can be directly connected to the user terminal 100, and may include, for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card. Alternatively, the software components may be loaded into the memory 120 via the communication module 140, rather than a computer-readable recording medium. For example, at least one program may be loaded into the memory 120 based on a computer program (e.g., a computer program for the processor 110 to perform the operations described below with reference to Figs. 4 to 13b) installed by a file provided via the communication module 140 by a developer or a file distribution system that distributes an installation file for an application.

[0063] The I / O interface 130 may be a means for interfacing with devices for input or output (e.g., a keyboard, a mouse, etc.) connected to or included in the user terminal 100. Although the I / O interface 130 is shown in Fig. 3a as an element configured separately from the processor 110, this is not limiting and the I / O interface 130 may be configured to be included in the processor 110.

[0064] The communication module 140 can provide a configuration or function for the server 20 and the user terminal 100 to communicate with each other via a network. The communication module 140 can also provide a configuration or function for the user terminal 100 to communicate with other external devices. For example, control signals, instructions, data, etc. provided under the control of the processor 110 can be transmitted to the server 20 and / or the external devices via the communication module 140 and the network.

[0065] Meanwhile, although not shown in Fig. 3a, the user terminal 100 may further include a display device. Alternatively, the user terminal 100 may be connected to an independent display device via a wired or wireless communication method, and may transmit and receive data to and from the display device. For example, the display device may provide the user 30 with pathology slide images, analysis information of the pathology slide images, and prediction information of treatment response.

[0066] FIG. 3b is a block diagram illustrating an example of a server according to an embodiment.

[0067] Referring to Fig. 3b, the server 20 includes a processor 210, a memory 220, and a communication module 230. For convenience of explanation, only components related to the present invention are shown in Fig. 3b. Therefore, the server 200 may further include other general-purpose components in addition to the components shown in Fig. 3b. It is obvious to a person skilled in the art related to the present invention that the processor 210, the memory 220, and the communication module 230 shown in Fig. 3b may be realized as independent devices.

[0068] The processor 210 can acquire pathology slide images from at least one of the internal memory 220, the external memory (not shown), the user terminal 10, and an external device. The processor 210 can acquire a first pathology slide image in which at least one first object is represented and biological information of the at least one first object, generate training data using at least one first patch and biological information included in the first pathology slide image, train a first machine learning model using the training data, or analyze a second pathology slide image using the trained first machine learning model. The processor 210 can also predict a treatment response of a subject corresponding to the second pathology slide image using spatial transcriptomics information of the second object represented in the second pathology slide image.

[0069] In other words, at least one of the operations of the processor 110 described above with reference to Fig. 3a may be performed by the processor 210. In that case, the user terminal 100 can output the information transmitted from the server 20 via a display device.

[0070] Meanwhile, the implementation of the processor 210 is similar to the implementation of the processor 110 described above with reference to FIG. 3a, so a detailed description will be omitted.

[0071] The memory 220 can store various data, such as pathology slide images and data generated by the operation of the processor 210. The memory 220 can also store an operating system (OS) and at least one program (e.g., a program necessary for the operation of the processor 210).

[0072] Meanwhile, the implementation of the memory 220 is similar to the implementation of the memory 220 described above with reference to FIG. 3a, so a detailed description will be omitted.

[0073] The communication module 230 can provide a configuration or function for the server 200 and the user terminal 100 to communicate with each other via a network. The communication module 140 can also provide a configuration or function for the server 200 to communicate with other external devices. For example, control signals, instructions, data, etc. provided under the control of the processor 210 can be transmitted to the user terminal 100 and / or the external devices via the communication module 230 and the network.

[0074] FIG. 4 is a flowchart illustrating an example of a method for processing a pathology slide image according to an embodiment.

[0075] 4, the method for processing a pathology slide image is composed of steps that are processed in time series by the user terminal 10, 100 or the processor 110 shown in Figures 1 to 3a. Therefore, even if the content is omitted below, the content described above regarding the user terminal 10, 100 or the processor 110 shown in Figures 1 to 3a can also be applied to the method for processing a pathology slide image in Figure 4.

[0076] 1 to 3b, at least one of the steps of the flowchart shown in FIG.

[0077] In step 410, the processor 110 obtains a first pathology slide image depicting at least one first object and biological information of the at least one first object, for example, the first object may represent a cell, tissue, and / or structure within a human body.

[0078] For example, the biological information may include spatial transcriptomics information of the first subject and information identified from a third pathology slide image, where the third pathology slide image refers to an image stained in a manner distinct from the first pathology slide image.

[0079] Hereinafter, the biological information will be specifically described with reference to FIG.

[0080] FIG. 5 is a diagram for explaining an example of biological information according to an embodiment.

[0081] Referring to FIG. 5, a subject 90 and an object 91 contained within the body of the subject 90 are shown.

[0082] As an example, the biological information of the subject 91 may include spatial transcriptomics information 511 of the subject 91. The spatial transcriptomics information 511 refers to information obtained by a spatial transcriptomics process. For example, the spatial transcriptomics information 511 may include sequence data obtained by the spatial transcriptomics process, gene expression information confirmed by performing data processing on the sequence data, etc.

[0083] The spatial transcriptomics process is a molecular profiling method that measures gene expression in tissue samples and allows mapping of the locations where genes are expressed. The relative locations of cells and tissues are important for understanding the normal development of cells or tissues and the pathology of diseases. However, traditional bulk-RNAseq analyzes a mixture of different tissues and cells at once, so the details of the spatial patterns of gene expression are not known. The spatial transcriptomics process allows us to ascertain the spatial patterns of gene expression, thus improving not only our understanding of diseases but also the accuracy of disease diagnosis and treatment.

[0084] The spatial transcriptomics information includes genetic information corresponding to the pathology slide image and / or at least one grid included in the pathology slide image. For example, the pathology slide image is divided into multiple grids, and a single grid may be, but is not limited to, a 1 mm×1 mm area.

[0085] The processor 110 can obtain spatial transcriptomics information by processing the sequence data to extract some regions (e.g., a single grid or multiple grids) of the pathology slide image and obtaining genetic information corresponding to the extracted regions.

[0086] An example of how the processor 110 obtains spatial transcriptomics information 511 of the subject 91 will now be described with reference to FIG.

[0087] FIG. 6 is a flow chart illustrating an example of a processor obtaining spatial transcriptomics information according to an embodiment.

[0088] In step 610, the processor 110 obtains sequence data from a spatial transcriptomics process.

[0089] For example, the spatial transcriptomics process can include the steps of Sample Prep, Imaging, Barcoding & Library Construction, and Sequencing.

[0090] In step 620, the processor 110 performs data processing on the sequence data to obtain gene expression information corresponding to the spots.

[0091] The processor 110 processes the sequence data to obtain gene information corresponding to the positions of the spots on the pathology slide image. For example, the pathology slide image is divided into a plurality of spots, and a single spot may be a circular area with a diameter of 55 μm, but is not limited thereto.

[0092] For example, the processor 110 can confirm at which position in the pathology slide image the genetic information contained in the sequence data is expressed based on the barcode information contained in the sequence data. Here, the barcode is the position coordinate value of a specific spot on the pathology slide image, and may be determined in advance. In other words, the barcode and the coordinates on the pathology slide image may be matched.

[0093] For example, 30,000 CDNA sequence reads are required per single spot, but are not limited thereto. Here, a sequence read means a portion sequenced from a DNA fragment. Specifically, read 1 of pair-end sequence data may include a barcode that matches coordinates (i.e., position coordinates of a pathology slide image), and read 2 may include transcript sequence information. That is, one end of a DNA fragment may include a barcode value corresponding to the coordinates of the spot from which the DNA fragment was obtained, and the other end may include sequence information.

[0094] The processor 110 can confirm gene expression information by aligning the fastq file containing sequence information to a reference genome. The processor 110 can also obtain a large number (e.g., about 5,000) of gene expression information for each spot of the pathology slide image using spatial information confirmed from the barcode.

[0095] On the other hand, although not shown in Fig. 6, the processor 110 can confirm what type of cells are present in the spot by using gene expression information corresponding to the spot. Generally, immune cells, cancer cells, etc. have genes that are expressed in a high amount in a cell-specific manner. Therefore, by the processor 110 analyzing the gene expression information corresponding to the spot, it is possible to confirm what types of cells are distributed in the spot area, or what percentage of what types of cells are included.

[0096] Meanwhile, the processor 110 can further use the single cell RNAseq data to confirm the number and type of cells distributed in the spot area. In general, single cell RNAseq data does not include spatial information, but only RNA information of each cell. Thus, the processor 110 can mathematically analyze the sequence data and single cell RNAseq data related to multiple cells (e.g., about 10 cells) included in each spot to confirm how many types of cells are included in each spot, or in what proportion.

[0097] Meanwhile, the processor 110 can use the machine learning model to confirm what type of cells are present in the spot from the sequence data. To this end, the processor 110 can train the machine learning model using the training data. As an example, the training data may include the sequence data acquired in step 610 as input data, and the type of cells as output data. As another example, the training data may include the sequence data acquired in step 610 and a patch of a pathology slide image corresponding to the sequence data as input data, and the type of cells as output data. That is, the machine learning model may be trained to identify the type of cells by considering not only the sequence data but also morphological characteristics included in the pathology slide image.

[0098] As described above with reference to FIG. 6, the processor 110 can generate a plurality of pairs including [patch of a pathology slide image-gene expression information corresponding to the patch]. The processor 110 can also generate a plurality of pairs including [patch of a pathology slide image-information on at least one cell type corresponding to the patch]. The processor 110 can also generate a plurality of pairs including [patch of a pathology slide image-gene expression information corresponding to the patch-information on at least one cell type corresponding to the patch]. The pairs thus generated can be used as training data for the first machine learning model. The training of the first machine learning model will be described later with reference to steps 420 and 430.

[0099] 5, as another example, the biological information of the subject 91 may include information 512 related to biological elements (e.g., cancer cells, immune cells, cancer regions, etc.) of the subject 91. Here, the information 512 related to the biological elements may be confirmed from a pathology slide image of the subject 91.

[0100] Depending on the manner in which the pathology slide image is stained, various biological information about the subject 91 can be ascertained. Therefore, different biological information can be ascertained from the same subject 91 when stained with different manners.

[0101] For example, in HE staining, hematoxylin mainly stains the nuclear region blue-purple, and eosin stains the cytoplasm and extracellular matrix pink. Therefore, with HE staining, the morphology of cells and tissues contained in the subject can be easily confirmed.

[0102] However, HE staining has limitations in identifying specific biological elements expressed in cells, etc. Therefore, immunohistochemistry staining, special staining, immunofluorescence staining, etc. can be used to confirm the degree of expression of specific biological elements.

[0103] For example, immunohistochemical staining methods include PD-L1 (programmed cell death-ligand 1) staining, HER2 (human epidermal growth factor receptor 2) staining, ER (estrogen receptor) staining, PR (progesterone receptor) staining, Ki-67 staining, CD68 staining, etc. Special staining methods include Van Gieson staining, Toluidine blue staining, Giemsa staining, Masson's trichrome staining, PAS (Periodic acid Schiff) staining, etc. Immunofluorescence staining methods include FISH (Fluorescence in situ hybridization), etc.

[0104] The various staining methods described above allow the identification of various biological elements.

[0105] As an example, the expression level of a specific cell signal that has not been confirmed from a pathology slide image by HE staining can be confirmed. For example, PD-L1 or HER2 is a protein or receptor expressed in a malignant tumor cell membrane, and the expression level in tumor cell tissue can be evaluated by PD-L1 staining or HER2 staining. Therefore, if the expression level is high, it can be predicted from the pathology slide image by HE staining that the therapeutic response of an anticancer therapeutic agent targeting the corresponding protein or receptor is high.

[0106] As another example, it is possible to accurately identify tissue components that are not clearly observed in HE-stained pathology slide images. For example, Van Gieson staining specifically stains only collagen, so it is possible to confirm only the expression of collagen in tissue.

[0107] As yet another example, the presence and / or amount of specific cells not identified in the pathology slide images stained by HE can be confirmed. For example, since CD68 specifically stains macrophages, the amount of macrophages, which is difficult to distinguish from other inflammatory cells in the pathology slide images stained by HE, can be easily confirmed in the pathology slide images stained by CD68.

[0108] The processor 110 can use the spatial transcriptomics information 511 and / or the information on biological elements 512 as training data for the machine learning model. An example in which the information on biological elements 512 is used as training data will be described later with reference to Figures 10 and 11.

[0109] Referring again to FIG. 4, in step 420, the processor 110 generates training data using at least one first patch and biological information contained in the first pathology slide image.

[0110] For example, the training data may include at least one of gene expression information corresponding to the patch and at least one cell type represented in the patch. The information on "at least one cell type represented in the patch" included in the training data may be information obtained by processing the gene expression information as described above with reference to FIG.

[0111] Hereinafter, an example of training data for training the first machine learning model will be described with reference to FIG.

[0112] FIG. 7 is a diagram for explaining an example of learning data according to an embodiment.

[0113] 7, a patch 711 is shown in a pathology slide image 710. As described above with reference to FIG. 6, the processor 110 can generate a plurality of pairs to be used as training data. For example, the pairs may be [patch 711-gene expression information 721 corresponding to patch 711], [patch 711-information 722 relating to at least one cell type corresponding to patch 711], or [patch 711-gene expression information 721 corresponding to patch 711-information 722 relating to at least one cell type corresponding to patch 711].

[0114] In other words, the learning data may include gene expression information 721 of the subject represented in the patch 711 and / or information 722 about at least one cell type of the subject represented in the patch 711. Here, the information about at least one cell type represented in the patch 711 may be information obtained by processing the gene expression information 721.

[0115] Referring again to FIG. 4, in step 430, the processor 110 trains a first machine learning model with the training data.

[0116] For example, the processor 110 can train the first machine learning model using the training data generated by step 420 as ground truth data. In other words, for training the first machine learning model, a patch of a pathology slide image can be used as input data, and a pair of [patch of a pathology slide image-gene expression information corresponding to the patch], [patch of a pathology slide image-information on at least one cell type corresponding to the patch], or [patch of a pathology slide image-gene expression information corresponding to the patch-information on at least one cell type corresponding to the patch] can be used as output data.

[0117] As an example, when a pair of a patch of a pathology slide image and gene expression information corresponding to the patch is used as output data, the first machine learning model may be trained to receive an input of a patch and predict gene expression information at the position of the patch.

[0118] As another example, if a pair of [patch of a pathology slide image - information about at least one cell type corresponding to the patch] is used as output data, the first machine learning model may be trained to receive an input of a patch and predict what type of cell is present at the location of that patch.

[0119] As yet another example, when a pair of [patch of pathology slide image-gene expression information corresponding to the patch-information on at least one cell type corresponding to the patch] is used as output data, the first machine learning model may be trained to receive a patch as input and predict the gene expression information and cell type corresponding to the position of the patch.

[0120] Meanwhile, the processor 110 may train the first machine learning model using at least one annotation generated based on a user input. For example, the training of the first machine learning model using the annotation may be additionally performed when the performance of the training using the training data generated in step 420 as ground truth data is not sufficient, but is not limited thereto.

[0121] For example, the user 30 can make annotations by referring to patches of a pathology slide image, and the annotations can include position information within the patches. Meanwhile, the number of users who can make annotations is not limited.

[0122] On the other hand, the processor 110 can also generate a second machine learning model that identifies at least one cell type contained in the subject by adding, removing, or removing and then adding at least one layer contained in the trained first machine learning model.

[0123] As an example, when a pair of [patch of a pathology slide image - gene expression information corresponding to the patch] is used to train a first machine learning model, the processor 110 can generate a second machine learning model by adding at least one layer that predicts cell type to the trained first machine learning model.

[0124] As another example, when a pair of [patch of pathology slide image-gene expression information corresponding to the patch-information on at least one cell type corresponding to the patch] is used to train the first machine learning model, the processor 110 can generate a second machine learning model by removing at least one layer that predicts gene expression information from the trained first machine learning model and adding a new layer.

[0125] In step 440, the processor 110 analyzes the second pathology slide image using the trained first machine learning model.

[0126] Although not shown in FIG. 4, if the processor 110 generates a second machine learning model, the processor 110 can analyze the second pathology slide image using the second machine learning model.

[0127] As described above with reference to Figures 4 to 7, unlike conventional learning of machine learning models that rely on annotation work by experts, the processor 110 can improve the performance of the machine learning model even without annotation work (or even with a small amount of annotation results), thereby improving the accuracy of the analysis results of pathology slide images by the machine learning model.

[0128] FIG. 8 is a flowchart illustrating another example of a method for processing a pathology slide image according to an embodiment.

[0129] 8, the method for processing a pathology slide image is composed of steps that are processed in time series by the user terminal 10, 100 or the processor 110 shown in Figures 1 to 3a. Therefore, even if the content is omitted below, the content described above regarding the user terminal 10, 100 or the processor 110 shown in Figures 1 to 3a can also be applied to the method for processing a pathology slide image in Figure 8.

[0130] 1 to 3b, at least one of the steps of the flowchart shown in FIG.

[0131] On the other hand, steps 810 to 840 respectively correspond to steps 410 to 440. Therefore, in the following, a detailed description of steps 810 to 840 will be omitted.

[0132] In step 850, the processor 110 predicts a therapeutic reaction of the subject 90 corresponding to the second pathology slide image using the spatial transcriptomics information of the second object represented in the second pathology slide image.

[0133] For example, the processor 110 can predict the treatment response of the subject 90 using the third machine learning model. Here, the spatial transcriptomics information of the second subject can include at least one of the spatial transcriptomics information (e.g., gene expression information) acquired by the trained first machine learning model and / or the spatial transcriptomics information acquired separately. Hereinafter, an example in which the processor 110 predicts the treatment response of the subject 90 will be described with reference to FIG. 9.

[0134] FIG. 9 is a diagram for explaining an example in which a processor according to an embodiment predicts a subject's treatment response.

[0135] 9, spatial transcriptomics information 921 can be generated by a trained first machine learning model 911. Spatial transcriptomics information 922 can be generated by another spatial transcriptomics process 912. As described above with reference to step 610, the spatial transcriptomics process 912 can obtain gene expression information corresponding to each of the pathology slide images and the grids contained in the images.

[0136] The processor 110 generates a treatment response prediction result 940 using the third machine learning model 930. For example, the spatial transcriptomics information 921 and / or the spatial transcriptomics information 922 may be input to the third machine learning model 930, and a treatment response prediction result 940 for the subject 90 may be generated.

[0137] As an example, the third machine learning model 930 may be trained using gene expression information contained in the spatial transcriptomics information and location information corresponding to the gene expression information.

[0138] Generally, when a machine learning model (e.g., a convolutional neural network) is trained based on a 2D image, a filter of a predetermined size (e.g., 3×3 pixels) is applied to check the image pattern, and this operation is performed for each channel (e.g., three RGB channels). The filtered value is then passed through a multi-layer neural network, and backpropagation is performed based on the difference between the output result value and the actual result value (e.g., ground truth), thereby training the machine learning model.

[0139] Similar to the above process, the processor 110 can replace gene expression information corresponding to each spot with a channel of a two-dimensional image, and replace position information corresponding to the gene expression information with pixels of the two-dimensional image. The processor 110 can also learn the third machine learning model 930 by performing back propagation based on the difference between the result value output after passing through the multi-layer neural network of the third machine learning model 930 and the result value of the actual treatment response or prognosis of the patient.

[0140] In order to convert the gene expression information corresponding to each spot into a channel, the gene expression information must be spatially divided. Therefore, the processor 110 can obtain gene information corresponding to each spot position on the pathology slide image by performing the above-mentioned process with reference to step 610.

[0141] As another example, the third machine learning model 930 may be trained using a feature vector extracted from at least one layer included in the trained first machine learning model.

[0142] As described above with reference to step 430, the first machine learning model may be trained as a model that predicts gene expression information at the position of a patch based on the patch, a model that predicts what type of cell is present at the position of the patch based on the patch, or a model that predicts gene expression information and cell type corresponding to the position of the patch based on the patch.

[0143] First, the processor 110 can input a pathology slide image to the trained first machine learning model and extract a feature vector from at least one layer included in the trained first machine learning model. For example, the layer to be extracted may be a layer experimentally determined and selected by the user 30, or may be a layer that appropriately predicts the treatment response or prognosis of the subject 90. That is, assuming that the first machine learning model is trained to correctly extract genetically and / or histologically important information (e.g., gene expression information that serves as the basis for predicting the treatment response, or cell types, characteristics, etc.) from the pathology slide image, it can be expected that the feature vector extracted from any intermediate layer of the trained first machine learning model also includes genetically and / or histologically important information.

[0144] The processor 110 can perform the process of extracting feature vectors from at least one layer included in the trained first machine learning model for multiple patches included in a single pathology slide image.

[0145] The processor 110 can then pool the feature vectors to form a vector with a single length. For example, the processor 110 can pool the feature vectors using the average value, pool the feature vectors using the maximum value in each dimension, perform dictionary-based pooling such as Bag-of-Words (BoW) or Fisher Vector, or perform attention-based pooling using an artificial neural network. By such pooling, a single vector corresponding to the pathology slide image of a single subject 90 can be defined.

[0146] The processor 110 can then train a third machine learning model 930 that uses the defined vector to predict responsiveness to a particular immunological anti-cancer agent or responsiveness to a particular treatment.

[0147] As described above with reference to Figures 8 and 9, the processor 110 can improve the accuracy of prediction by training the third machine learning model 930 and predicting the treatment response of the subject 90 using the third machine learning model 930, compared to predicting responsiveness to treatment using only morphological characteristics of the pathology slide image.

[0148] As described above with reference to Figures 6 and 7, the processor 110 can train the first machine learning model using the spatial transcriptomics information 511. Meanwhile, the processor 110 can also train the first machine learning model using information about biological elements 512. Hereinafter, with reference to Figures 10 and 11, an example in which the processor 110 trains the first machine learning model using information about biological elements 512 will be described.

[0149] FIG. 10 is a diagram illustrating an example of a processor according to an embodiment learning a first machine learning model.

[0150] 10, a subject 1010 and pathology slide images 1031 and 1041 representing the subject are shown. Here, it is assumed that a first staining scheme 1021 of the pathology slide image 1031 is different from a second staining scheme 1022 of the pathology slide image 1041. For example, the first staining scheme 1021 may include not only a staining scheme that selectively stains a specific biological element, but also a staining scheme (e.g., HE staining) that can easily confirm the morphology of the nuclei, cytoplasm, and extracellular matrix of all cells contained in the subject.

[0151] The processor 110 can generate training data for training the first machine learning model 1050. Here, the training data can include a patch 1032 included in the pathology slide image 1031 and a patch 1042 included in the pathology slide image 1041. Here, the patch 1032 and the patch 1042 may indicate the same position of the object 1010. In other words, the patch 1042 may indicate a position corresponding to the patch 1032.

[0152] In Fig. 10, it is assumed that a first staining scheme 1021 is a scheme capable of selectively staining a biological element A, and a second staining scheme 1022 is a scheme capable of selectively staining a biological element B. The method of selectively staining various biological elements is as described above with reference to Fig. 5. Also, Fig. 10 shows pathology slide images 1031 and 1041 obtained by two types of staining schemes 1021 and 1022, but is not limited thereto.

[0153] The processor 110 performs image processing so that the object 1010 on the image 1031 and the object 1010 on the image 1041 are perfectly overlapped. For example, the processor 110 can accurately align the object 1010 on the image 1031 and the object 1010 on the image 1041 by performing geometric transformation (e.g., enlargement, reduction, rotation, etc.) on the images 1031 and 1041. The processor 110 also extracts patches 1032 and 1042 at corresponding positions of the images 1031 and 1041, respectively. In this manner, the processor 110 can generate multiple pairs of patches extracted from the image 1031 and the image 1041.

[0154] The processor 110 then trains the first machine learning model 1050 using the patches 1032, 1042. For example, the processor 110 can train the first machine learning model 1050 using the patches 1032 as input data and the patches 1042 as output data. In that case, the patches 1042 can be used as ground truth data.

[0155] FIG. 11 is a diagram for explaining another example in which a processor according to one embodiment learns a first machine learning model.

[0156] 11, there is shown an object 1110 and pathology slide images 1131 and 1141 representing the object. Here, the detailed description of the staining methods 1121 and 1122, the pathology slide images 1131 and 1141, and the patches 1132 and 1142 is the same as that described above with reference to FIG.

[0157] The processor 110 may generate training data for training the first machine learning model 1160. Here, the training data may include a patch 1143 in which image processing 1150 is performed on a patch 1132 and a patch 1142.

[0158] The processor 110 may perform one or more image processes on the patch 1142 to generate a patch 1143 .

[0159] For example, the processor 110 can perform image filtering to leave only parts of the patch 1142 that are stained darker than a specific density, or to leave only parts that express a specific color and remove the rest. However, the image processing technique performed by the processor 110 is not limited to the above-mentioned techniques.

[0160] As another example, the processor 110 may apply more complex image processing techniques or another machine learning model to the patch 1142 to extract semantic information, and use the extracted information as learning data corresponding to the patch 1143. For example, the extracted information may be information displaying the positions of specific cells (e.g., cancer cells, immune cells, etc.) as dots, information displaying the type or class of cells based on the degree of staining and / or the morphology of staining, and the like.

[0161] Here, in the case of an image processing technique, an algorithm may be used that quantifies the amount of dye expression for each pixel included in the image 1141 and utilizes pixel position information. In this case, the extracted information may include information regarding the type and position of a specific cell.

[0162] Meanwhile, the separate machine learning model may be a model that recognizes the position and type of biological element targeted by the staining scheme 1122 of the image 1141. For example, the separate machine learning model may be trained to detect B, which is a biological element expressed by the second staining scheme 1122, when a patch stained with the second staining scheme 1122 is input. Here, if the second staining scheme 1122 is a stain that is expressed in cancer cells, the separate machine learning model may be trained to receive a patch stained with the second staining scheme 1122 and detect the cancer cells. The detection result may be a point indicating the position of each cancer cell, or may be a result of segmenting the cancer cells at the pixel level.

[0163] Although not shown in Fig. 11, the processor 110 may also train the first machine learning model 1160 using at least one annotation generated based on the patch 1132 and a user input. Here, the annotation may be generated based on the image 1141. For example, the training of the first machine learning model utilizing the annotation may be additionally performed when the performance of the training using the training data described above with reference to Figs. 10 and 11 as ground truth data is not sufficient, but is not limited thereto.

[0164] For example, the user 30 can make annotations by referring to the image 1141, and the annotations can include position information within the patch 1142. On the other hand, the number of users who can make annotations is not limited.

[0165] Meanwhile, the processor 110 can also generate another machine learning model by adding, removing, or removing and then adding at least one layer included in the trained first machine learning model 1160. For example, the processor 110 can generate another machine learning model by removing a layer that plays a role in drawing an image in the trained first machine learning model 1160 and adding a new layer that performs a final target task. Here, the final target task may mean a task of further recognizing biological elements that need to be identified separately in addition to biological elements that can be identified from the images 1131 and 1141. Alternatively, the final target task may mean a task that can derive medical information such as a prediction of the degree of expression of a biomarker or a treatment response.

[0166] As described above with reference to Figures 10 and 11, pathology slide images of the same tissue stained with different types of substances are used to train the machine learning model, which eliminates the problems of inaccuracy and increased costs caused by human annotation and enables a large amount of training data to be secured.

[0167] FIG. 12 is a diagram for explaining an example in which the operation of the processor according to an embodiment is realized.

[0168] The example described below with reference to Figure 12 may be the operation of the processor 110 described above with reference to Figures 10 and 11. For example, according to the example shown in Figure 12, the processor 110 may train the first machine learning models 1050, 1160.

[0169] 12, a screen 1210 for selecting pathology slide images stained in different ways is shown. However, the configuration of the screen 1210 is merely an example and can be changed in various ways.

[0170] Display 1210 may display a list of target slide images 1220 and a list of reference slide images 1230. For example, the target slide images may be images stained with a first staining scheme 1021, 1121 and the reference slide images may be images stained with a second staining scheme 1022, 1122.

[0171] User 30 may select image 1221 and image 1231 and select execute button 1240, causing processor 110 to perform the operations described above with reference to Figures 10 and 11. For example, processor 110 may train first machine learning model 1050, 1160 to predict the location and / or type of biological element (e.g., cell, protein, and / or tissue) depicted in image 1221 based on image 1231.

[0172] By the above-mentioned operation of the processor 110, it is possible to output a screen 1250 in which the position and / or type of the biological element shown in the image 1221 is predicted. However, the configuration of the screen 1250 shown in FIG. 12 is merely an example and can be changed in various ways.

[0173] For example, a mini map 1251 may be output on the screen 1250, which displays a portion of the image 1221 that is currently output on the screen 1250. Also, a window 1252 may be set on the screen 1250, which displays a portion that the user 30 focuses on among the portions displayed on the current screen 1250. Here, the position and size of the window 1252 may be set in advance, or may be adjusted by the user 30.

[0174] Meanwhile, an example of the annotation described above with reference to FIGS. 4 and 11 will be described with reference to FIGS. 13a and 13b.

[0175] 13a and 13b are diagrams for explaining an example in which annotations are generated based on user input according to one embodiment.

[0176] If it is determined that the machine learning model is not capable of recognizing biological elements (e.g., tissues, cells, structures, etc.) from the target image, the user 30 can directly correct the annotations.

[0177] Referring to FIG. 13a, assuming that the type and / or location of cells 1321, 1322, 1323 represented in region 1311 of pathology slide image 1310 is mispredicted, user 30 can directly correct the labeling of cells 1321, 1322, 1323.

[0178] Referring to FIG. 13b, a user 30 can select a grid 1340 that includes multiple cells represented in an area 1331 of a pathology slide image 1330, and can also modify the labeling for the cells or tissues included in the grid 1340 all at once.

[0179] As described above, unlike conventional machine learning model learning that relies on annotation work by experts, the processor 110 can improve the performance of the machine learning model without annotation work (or even with a small amount of annotation results). This improves the accuracy of the analysis results of the pathology slide images by the machine learning model. In addition, the processor 110 can predict the treatment response of the subject using the analysis results of the pathology slide images, so the accuracy of the predicted results of the treatment response is also guaranteed.

[0180] Meanwhile, the above-mentioned method can be created by a computer-executable program, and can be realized by a general-purpose digital computer that operates the program using a computer-readable recording medium. Also, the data structure used in the above-mentioned method can be recorded in a computer-readable recording medium by various means. The computer-readable recording medium includes storage media such as magnetic recording media (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.), optically readable media (e.g., CD-ROM, DVD, etc.), etc.

[0181] It will be understood by those skilled in the art that the present embodiment may be realized in modified forms without departing from the essential characteristics of the above description. Therefore, the disclosed method should be considered from an explanatory perspective, not a limiting one, and the scope of the rights should be interpreted as including all differences within the scope of the claims, not the above description, and equivalent thereto.

Claims

1. at least one memory; at least one processor; Including, The at least one processor a first pathology slide image showing a first object and spatial transcriptomics information of the first object; generating training data including information on at least one biological element shown in the first patch using at least one first patch included in the first pathology slide image and the spatial transcriptomics information; training a first machine learning model to predict information on the at least one biological element based on the training data; and analyzing a second pathology slide image showing a second object using the trained first machine learning model; Computing equipment.

2. the biological component comprises at least one of a cell, a tissue, and a structure; The computing device of claim 1 , wherein the information about the at least one biological element includes information about the type, characteristics, gene expression, or location of the at least one biological element.

3. The computing device described in claim 1, wherein information regarding the at least one biological element is obtained based on gene expression information corresponding to the at least one first patch.

4. The gene expression information includes gene expression information for each region of at least one cell; The at least one processor The computing device of claim 3 , wherein the training data is used as ground truth data to train the first machine learning model.

5. The at least one processor The computing device of claim 1 , further comprising: a processor configured to generate a second machine learning model that identifies at least one cell type contained in the first subject by adding or removing at least one layer contained in the trained first machine learning model.

6. The at least one processor 10. The computing device of claim 1, wherein spatial transcriptomics information of a second subject represented in the second pathology slide image is used to predict a therapeutic reaction of the subject corresponding to the second pathology slide image.

7. the prediction of treatment response is made by a third machine learning model; 7. The computing device of claim 6, wherein the spatial transcriptomic information of the second subject comprises at least one of spatial transcriptomic information obtained by the trained first machine learning model and spatial transcriptomic information obtained separately.

8. 8. The computing device of claim 7, wherein the third machine learning model is trained to predict a subject's treatment response using a feature vector extracted from at least one layer included in the trained first machine learning model.

9. 8. The computing device of claim 7, wherein the third machine learning model is trained to predict a subject's treatment response using gene expression information included in the spatial transcriptomics information and location information corresponding to the gene expression information.

10. A step of acquiring a first pathology slide image depicting a first object and spatial transcriptomics information of the first object; generating training data including information about at least one biological element represented in the first patch using at least one first patch included in the first pathology slide image and the spatial transcriptomics information; training a first machine learning model to predict information about the at least one biological component based on the training data; A method for analyzing pathology slide images, comprising: analyzing a second pathology slide image representing a second object using the trained first machine learning model.

11. the biological component comprises at least one of a cell, a tissue, and a structure; The method of claim 10 , wherein the information about the at least one biological element includes information about the type, characteristics, gene expression, or location of the at least one biological element.

12. The method described in claim 10, wherein the information regarding the at least one biological element is obtained based on gene expression information corresponding to the at least one first patch.

13. The gene expression information includes gene expression information for each region of at least one cell; The learning step includes: The method of claim 12 , wherein the training data is used as ground truth data to train the first machine learning model.

14. The method described in claim 10, further comprising a step of generating a second machine learning model that identifies at least one cell type contained in the first subject by adding or removing at least one layer contained in the learned first machine learning model.

15. The method described in claim 10, further comprising a step of predicting the treatment response of a subject corresponding to the second pathology slide image using spatial transcriptomics information of a second subject represented in the second pathology slide image.

16. The prediction of treatment response is performed by a third machine learning model; 16. The method of claim 15, wherein the spatial transcriptomic information of the second subject comprises at least one of spatial transcriptomic information obtained by the trained first machine learning model and spatial transcriptomic information obtained separately.

17. The method described in claim 16, wherein the third machine learning model is trained to predict the treatment response using feature vectors extracted from at least one layer included in the trained first machine learning model.

18. The method described in claim 16, wherein the third machine learning model is trained to predict a subject's treatment response using gene expression information contained in the spatial transcriptomics information and location information corresponding to the gene expression information.

19. A computer-readable recording medium having a program recorded thereon for executing the method according to claim 10 on a computer.

Images