Method and apparatus for providing immunophenotypic and related information for pathology slide images

Abstract

A method is provided for providing immunophenotype-related information for pathology slide images, performed by at least one computer device. [Solution] This method may include steps of acquiring information associated with an immunophenotype for one or more regions of interest in a pathology slide image, generating an image showing the information associated with the immunophenotype based on the information associated with the immunophenotype for the one or more regions of interest, and outputting the image showing the information associated with the immunophenotype.

Description

[Technical Field]

[0001] The present disclosure relates to methods and apparatus for providing information associated with immune phenotypes for pathology slide images, and more particularly to methods and apparatus for generating and outputting images showing information associated with immune phenotypes for one or more regions of interest in a pathology slide image. [Background technology]

[0002] Recently, there has been growing interest in immunotherapy, a third-generation anticancer drug that utilizes the patient's immune system. An immunotherapy can be any drug that prevents cancer cells from evading the body's immune system or enables immune cells to recognize and attack cancer cells. Because it acts through the body's immune system, it has minimal side effects and can extend the survival time of cancer patients compared to other anticancer drugs. However, these immunotherapy drugs are not effective for all cancer patients. Therefore, predicting the reactivity of immunotherapy is important to predict the effectiveness of immunotherapy in current cancer patients.

[0003] Meanwhile, a user (e.g., a doctor or a patient) can be provided with immune response information generated through a pathology slide image of the patient's tissue in order to predict the reactivity of an immune anticancer drug. According to conventional technology, a user (e.g., a doctor or a patient) can be provided with immune response information (e.g., immune cell expression information) for each of a plurality of patches included in a pathology slide image in order to predict the reactivity of an immune anticancer drug. In this case, the user may have difficulty intuitively recognizing the immune response information for each of the numerous patches included in the pathology slide image. Furthermore, immune response information may be generated for patches that are not substantially necessary for predicting the reactivity of an immune anticancer drug. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a method and apparatus for providing information associated with immunophenotypes for pathology slide images to solve the above-mentioned problems. [Means for solving the problem]

[0005] The present disclosure may be embodied in numerous ways, including as a method, an apparatus (system), a computer-readable storage medium storing instructions, or a computer program.

[0006] A method for providing information associated with an immune phenotype for a pathology slide image, performed by at least one computer device according to one embodiment of the present disclosure, includes steps of acquiring information associated with an immune phenotype for one or more regions of interest in the pathology slide image, generating an image indicating the information associated with the immune phenotype based on the information associated with the immune phenotype for the one or more regions of interest, and outputting the image indicating the information associated with the immune phenotype.

[0007] In one embodiment of the present disclosure, one or more regions of interest are determined based on the detection of one or more items of interest in the pathology slide image.

[0008] In one embodiment of the present disclosure, the one or more regions of interest include at least a portion of the pathology slide image that satisfies a condition associated with one or more items of interest.

[0009] In one embodiment of the present disclosure, the one or more regions of interest are regions output by inputting the detection results of one or more target items in a pathology slide image or a pathology slide image into a region of interest extraction model, and the region of interest extraction model is trained to output a reference region of interest by inputting the detection results of one or more target items in a reference pathology slide image or a reference pathology slide image.

[0010] In one embodiment of the present disclosure, the acquiring step includes acquiring an immunophenotype of one or more regions of interest, and the generating step includes generating an image including visual indicators corresponding to the immunophenotype of the one or more regions of interest, wherein the immunophenotype includes at least one of immune activity, immune exclusion, or immune deficiency.

[0011] In one embodiment of the present disclosure, the obtaining step includes obtaining one or more immunophenotype scores for one or more regions of interest, and the generating step includes generating an image including visual indicators corresponding to the one or more immunophenotype scores, wherein the one or more immunophenotype scores include at least one of a score for immune activity, a score for immune exclusion, or a score for immune deficiency.

[0012] In one embodiment of the present disclosure, the acquiring step includes acquiring features associated with one or more immunophenotypes for one or more regions of interest, and the generating step includes generating an image including visual indicators corresponding to the features associated with the one or more immunophenotypes, wherein the features associated with the one or more immunophenotypes include at least one statistical value or vector associated with the immunophenotype.

[0013] In one embodiment of the present disclosure, the outputting step includes outputting an image including one or more regions of interest and visual indicators in the pathology slide image together.

[0014] In one embodiment of the present disclosure, the outputting step includes overlaying an image including visual indicators over one or more regions of interest on the pathology slide image.

[0015] In one embodiment of the present disclosure, the method further includes steps of obtaining detection results of one or more target items from a pathology slide image, generating an image indicating the detection results of the one or more target items, and outputting the image indicating the detection results of the one or more target items.

[0016] A computer program stored on a computer-readable recording medium is provided for executing the method for providing information associated with an immunophenotype for a pathology slide image described above according to one embodiment of the present disclosure.

[0017] A computer device according to one embodiment of the present disclosure includes a memory for storing one or more instructions; and a processor configured to execute the one or more stored instructions to acquire information associated with an immunophenotype for one or more regions of interest in a pathology slide image, generate an image indicating the information associated with the immunophenotype based on the information associated with the immunophenotype for the one or more regions of interest, and output the image indicating the information associated with the immunophenotype. [Effects of the Invention]

[0018] According to some embodiments of the present disclosure, by providing a user with an image visually indicating information associated with an immunophenotype, the user can intuitively recognize the information associated with the immunophenotype for each region. In addition, by providing a visual indicator indicating the information associated with the immunophenotype on a corresponding region of interest in a pathology slide image, the user can easily see at a glance which region of the pathology slide image the information indicated by the visual indicator corresponds to.

[0019] According to some embodiments of the present disclosure, it is possible to determine regions of interest in a pathology slide image that essentially require analysis for immunophenotyping and / or determining the response of an immunological anticancer drug. That is, an information processing system and / or a user terminal can perform processing (e.g., immunophenotyping and / or immunophenotyping score calculation) only on the regions of interest excluding unnecessary regions, rather than on the entire pathology slide image, thereby minimizing computer resources and processing costs.

[0020] According to some embodiments of the present disclosure, more accurate results can be obtained by performing processing (e.g., immunophenotyping and / or immunophenotyping score calculation) only on meaningful regions in immunophenotyping and / or determining whether or not an immunological anticancer drug is responsive.

[0021] The effects of the present disclosure are not limited to these, and other effects not mentioned should be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains (hereinafter referred to as "ordinary engineer") from the description in the claims. [Brief explanation of the drawings]

[0022] BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments of the present disclosure will now be described, without limitation, with reference to the accompanying drawings, in which like reference numerals indicate like elements and in which:

[0014] FIG. [Figure 1] FIG. 1 is an exemplary block diagram showing a system in which an information processing system according to an embodiment of the present disclosure provides information associated with an immunophenotype for a pathology slide image. [Figure 2] 1 is a block diagram showing an internal configuration of an information processing system according to an embodiment of the present disclosure. [Figure 3] FIG. 2 is a block diagram illustrating an internal configuration of a user terminal according to an embodiment of the present disclosure. [Figure 4] 1 is a flowchart illustrating a method for providing information associated with an immunophenotype for a pathology slide image according to one embodiment of the present disclosure. [Figure 5]FIG. 10 illustrates an example of determining one or more regions of interest within a pathology slide image according to one embodiment of the present disclosure. [Figure 6] FIG. 10 illustrates an example of generating immunophenotyping results according to an embodiment of the present disclosure. [Figure 7] FIG. 10 is a diagram showing an example of outputting an immunophenotyping result according to an embodiment of the present disclosure. [Figure 8] FIG. 10 is a diagram showing an example of outputting an immunophenotyping result according to another embodiment of the present disclosure. [Figure 9] FIG. 1 illustrates an example of an artificial neural network model according to an embodiment of the present disclosure. [Figure 10] FIG. 1 is a block diagram of an exemplary computer device (e.g., a user terminal) that provides information associated with immunophenotypes for pathology slide images according to one embodiment of the present disclosure. [Figure 11] FIG. 10 is a diagram illustrating an example of outputting a target item detection result according to an embodiment of the present disclosure. [Figure 12] 10A and 10B are diagrams illustrating an example of outputting a target item detection result and an immune phenotype determination result according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, specific implementations of the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, specific descriptions of well-known functions and configurations will be omitted if they may unnecessarily obscure the gist of the present disclosure.

[0024] In the accompanying drawings, the same or corresponding components are denoted by the same reference numerals. In addition, in the following description of the embodiments, repeated descriptions of the same or corresponding components may be omitted. However, the omission of a description of a component does not mean that such a component is not included in a certain embodiment.

[0025] The advantages and features of the disclosed embodiments, and methods for achieving them, will become clearer with reference to the following examples in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, and may be embodied in various different forms. However, the present embodiments are provided solely for the purpose of completeness of the disclosure and to enable those skilled in the art to accurately recognize the scope of the invention.

[0026] The terms used in this specification will be briefly explained, and the disclosed embodiments will be specifically described. The terms used in this specification are currently commonly used and generally used terms, taking into consideration the function of the present disclosure. However, these terms may change depending on the intentions of engineers in the relevant field, legal precedents, the emergence of new technologies, etc. In addition, in specific cases, the applicant may arbitrarily select terms, and their meanings will be described in detail in the description of the invention. Therefore, the terms used in this disclosure should be defined based on the meanings of the terms and the overall content of the present disclosure, rather than simply by the names of the terms.

[0027] In this specification, unless otherwise clearly specified in the context, singular expressions can include plural expressions, and plural expressions can include singular expressions. Throughout this specification, when a part "comprises" a certain element, this does not mean that other elements are excluded, and that other elements can also be included, unless otherwise specified.

[0028] Additionally, the terms "module" and "module" as used herein refer to software or hardware components, each of which performs a specific function. However, the terms "module" and "module" are not limited to software or hardware. A "module" or "module" may reside on an addressable storage medium or execute on one or more processors. Thus, by way of example, a "module" or "module" may include components such as software components, object-oriented software components, class components, and task components, as well as at least one of processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The components and "modules" or "modules" may be combined into fewer components and "modules" or "modules," or the functionality provided therein may be further separated into additional components and "modules" or "modules."

[0029] According to one embodiment of the present disclosure, a "module" or "unit" may be embodied with a processor and memory. "Processor" should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, "processor" may also refer to an application-specific semiconductor (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), etc. "Processor" may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such configuration. Also, "memory" should be broadly interpreted to include any electronic component capable of storing electronic information. "Memory" can refer to various types of processor-readable media, such as RAM (Random Access Memory), ROM (Read Only Memory), NVRAM (Non-Volatile Random Access Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic or optical data storage devices, registers, etc. Memory is said to be in electronic communication with a processor when the processor can read / write information from or write information to the memory. Memory that is integrated into a processor is in electronic communication with the processor.

[0030] In the present disclosure, a "system" may include at least one of a server device and a cloud device, but is not limited to this. For example, a system may be composed of one or more server devices. As another example, a system may be composed of one or more cloud devices. As yet another example, a system may be operated by both a server device and a cloud device.

[0031] In the present disclosure, "target data" may refer to any data or data item that can be used to train a machine learning model, including, but not limited to, data representing an image, data representing audio or audio features, etc. In the present disclosure, the target data is described as an entire pathology slide image and / or at least one patch (or region) contained in the pathology slide image, but is not limited thereto, and any data that can be used to train a machine learning model may be considered target data. In addition, the target data is tagged with label information through annotation work.

[0032] In this disclosure, the term "pathology slide image" refers to a photographic image of a pathology slide in which tissues, etc., removed from a human body have been fixed and stained through a series of chemical processes for microscopic observation. For example, a pathology slide image may refer to a digital image captured with a microscope and may include information about cells, tissues, and / or structures within the human body. A pathology slide image may also include one or more patches, and one or more patches may be tagged with label information (e.g., information about immunophenotypes) through annotation. For example, a "pathology slide image" may include, but is not limited to, an H&E-stained tissue slide and / or an IHC-stained tissue slide, and may also include tissue slides stained with various staining methods (e.g., chromogenic in situ hybridization (CISH), fluorescent in situ hybridization (FISH), multiplex IHC, etc.) or unstained tissue slides. As another example, a "pathology slide image" may be a patient tissue slide generated to predict immune anticancer drug response, and may include a patient tissue slide before immune anticancer drug treatment and / or a patient tissue slide after immune anticancer drug treatment.

[0033] In the present disclosure, a "biomarker" can be defined as an indicator that can objectively measure a normal or pathological state, the degree of response to a drug, etc. For example, the biomarker can include, but is not limited to, an immune anticancer drug and PD-L1 as a biomarker, and can include various biomarkers associated with immune cells, such as TMB (Tumor Mutation Burden) values, MSI (Microsatellite Instability) values, HRD (Homologous Recombination Deficiency) values, CD3, CD8, CD68, FOXP3, CD20, CD4, CD45, and CD163.

[0034] In the present disclosure, a "patch" may refer to a small region within a pathology slide image. For example, a patch may include a region corresponding to a semantic object extracted by performing segmentation on the pathology slide image. As another example, a patch may refer to a combination of pixels associated with label information generated by analyzing the pathology slide image.

[0035] In the present disclosure, a "region of interest" may refer to at least a portion of a pathology slide image that is to be analyzed. For example, a region of interest may refer to at least a portion of a pathology slide image that includes a target item. As another example, a region of interest may refer to at least a portion of a plurality of patches generated by dividing a pathology slide image.

[0036] In the present disclosure, the terms "machine learning model" and / or "artificial neural network model" may include any model used to infer an answer to a given input. According to one embodiment, the machine learning model may include an artificial neural network model including an input layer, multiple hidden layers, and an output layer. Here, each layer may include multiple nodes. For example, the machine learning model may be trained to infer label information for at least one patch included in a pathology slide image and / or a pathology slide. In this case, the label information generated by the annotation operation is used to train the machine learning model. The machine learning model may also include weights associated with multiple nodes included in the machine learning model. Here, the weights may include any parameters associated with the machine learning model.

[0037] In this disclosure, "training" may refer to any process of modifying weights associated with a machine learning model using at least one patch and label information. According to one embodiment, training may refer to a process of modifying or updating weights associated with a machine learning model using at least one patch and label information through one or more rounds of forward propagation and backward propagation.

[0038] In the present disclosure, "label information" refers to correct answer information for a data sample, and is information obtained as a result of an annotation operation. The terms "label" and "label information" may be used interchangeably with terms such as "annotation" and "tag" in the art. In the present disclosure, "annotation" may refer to an annotation operation and / or annotation information (e.g., label information, etc.) determined by performing the annotation operation. In the present disclosure, "annotation information" may refer to information for the annotation operation and / or information (e.g., label information) generated by the annotation operation.

[0039] In the present disclosure, a "target item" may refer to data / information, image regions, objects, etc. that are to be detected in a pathology slide image. According to one embodiment, a target item may include an object to be detected from a pathology slide image for the diagnosis, treatment, and prevention of a disease (e.g., cancer). For example, a "target item" may include a cell-based target item and a region-based target item.

[0040] In the present disclosure, "each of a plurality of A's" and / or "each of a plurality of A's" can refer to each of all the components included in the plurality of A's, or each of some of the components included in the plurality of A's. For example, each of a plurality of regions of interest can refer to each of all the regions of interest included in the plurality of regions of interest, or each of some of the regions of interest included in the plurality of regions of interest.

[0041] In this disclosure, "instructions" may refer to one or more instruction words that are combined based on functionality, are components of a computer program, and are executed by a processor.

[0042] In the present disclosure, a "user" may refer to a person who uses a user terminal. For example, a user may include an annotator who performs annotation work. As another example, a user may include a doctor or patient who is provided with information associated with an immune phenotype and / or a response prediction result to an immune anticancer drug (e.g., a prediction result as to whether a patient will respond to an immune anticancer drug). Furthermore, a user may refer to a user terminal, and conversely, a user terminal may refer to a user. That is, the terms "user" and "user terminal" may be used interchangeably in this specification.

[0043] FIG. 1 is an exemplary configuration diagram illustrating a system in which an information processing system 100 according to an embodiment of the present disclosure provides information associated with an immune phenotype for a pathology slide image. As shown in the figure, the system for providing information associated with an immune phenotype for a pathology slide image may include an information processing system 100, a user terminal 110, and a storage system 120. Here, the information processing system 100 may be configured to be connected to and communicate with each of the user terminal 110 and the storage system 120. While FIG. 1 illustrates a single user terminal 110, the present invention is not limited thereto, and multiple user terminals 110 may be connected to and communicate with the information processing system 100. Also, while FIG. 1 illustrates the information processing system 100 as a single computer device, the present invention is not limited thereto, and the information processing system 100 may be configured to perform distributed processing of information and / or data via multiple computer devices. Also, while FIG. 1 illustrates the storage system 120 as a single device, the present invention is not limited thereto, and the information processing system 100 may be configured as multiple storage devices or as a cloud-based system. Furthermore, in Figure 1, each component of the system for providing information associated with immunophenotypes for pathology slide images represents a functionally divided functional element, and multiple components may be embodied in a form that is integrated with each other in an actual physical environment.

[0044] The information processing system 100 and the user terminal 110 are any computer devices used to generate and provide information associated with immunophenotypes for pathology slide images. Here, the computer device may refer to any type of device equipped with computer functions, such as, but not limited to, a notebook, desktop, laptop, server, cloud system, etc.

[0045] The information processing system 100 can receive a pathology slide image. For example, the information processing system 100 can receive the pathology slide image from the storage system 120 and / or the user terminal 110. The information processing system 100 can generate information associated with an immunophenotype for the pathology slide image and provide the information to the user 130 via the user terminal 110. In one embodiment, the information processing system 100 can determine one or more regions of interest within the pathology slide image and generate information associated with an immunophenotype for the one or more regions of interest. Here, the information associated with the immunophenotype can include at least one of an immunophenotype of the one or more regions of interest, an immunophenotype score of the one or more regions of interest, and a feature associated with the immunophenotype of the one or more regions of interest.

[0046] In one embodiment, the information processing system 100 may determine one or more regions of interest based on detection results of one or more target items in a pathology slide image. For example, the information processing system 100 may determine one or more regions of interest including at least a partial region of the pathology slide image that satisfies a condition associated with one or more target items. Additionally or alternatively, the information processing system 100 may input detection results of one or more target items in a pathology slide image (e.g., a pathology slide image including the target item detection results) into a region of interest extraction model to determine output regions as one or more regions of interest. Here, the region of interest extraction model may correspond to a model trained to output reference regions of interest when input detection results of one or more target items in a reference pathology slide image (e.g., a reference pathology slide image including the target item detection results).

[0047] The user terminal 110 can acquire information associated with an immunophenotype for one or more regions of interest in a pathology slide image from the information processing system 100. For example, the information associated with the immunophenotype for one or more regions of interest can include an immunophenotype for the one or more regions of interest (e.g., at least one of immune inflamed, immune excluded, or immune desert). Additionally or alternatively, the information associated with the immunophenotype for one or more regions of interest can include one or more immunophenotype scores for the one or more regions of interest (e.g., at least one of an immune inflamed score, an immune excluded score, or an immune desert score). Additionally or alternatively, the information associated with the immunophenotype for one or more regions of interest can include features associated with one or more immunophenotypes for the one or more regions of interest (e.g., at least one statistical value or vector associated with an immunophenotype).

[0048] The user terminal 110 can then generate an image showing the information associated with the immunophenotype based on the information associated with the immunophenotype for one or more regions of interest. In one embodiment, the user terminal 110 can generate an image including visual indicators corresponding to the immunophenotype of one or more regions of interest. In another embodiment, the user terminal 110 can generate an image including visual indicators corresponding to one or more immunophenotype scores. For example, the visual indicators can include color (e.g., color, brightness, saturation, etc.), text, images, marks, graphics, etc.

[0049] The user terminal 110 can output an image showing information associated with the generated immunophenotype. In one embodiment, the user terminal 110 can output both an image including one or more regions of interest in a pathology slide image and a visual marker. That is, the image including one or more regions of interest in a pathology slide image and a visual marker can be displayed together on a display device associated with the user terminal 110. In another embodiment, the user terminal 110 can overlay an image including a visual marker on one or more regions of interest in a pathology slide image. That is, one or more regions of interest in a pathology slide image overlaid with an image including a visual marker can be displayed on a display device associated with the user terminal 110. In this way, an image showing information associated with an immunophenotype can be provided to the user 130 (e.g., a doctor, a patient, etc.) via the user terminal 110.

[0050] The storage system 120 is a device or cloud system that stores and manages pathology slide images associated with a subject patient and various data associated with a machine learning model to provide information associated with the immunophenotype of the pathology slide image. To efficiently manage data, the storage system 120 may store and manage various data using a database. Here, the various data may include any data associated with a machine learning model, such as, but not limited to, a file of the target data, meta information of the target data, label information related to the target data resulting from the annotation work, data related to the annotation work, and a machine learning model (e.g., an artificial neural network model). While the information processing system 100 and the storage system 120 are shown as separate systems in FIG. 1 , they are not limited thereto and may be integrated into a single system.

[0051] According to some embodiments of the present disclosure, by providing a user 130 with an image visually showing information associated with an immunophenotype, the user 130 can intuitively recognize information associated with the immunophenotype for each region. Furthermore, according to some embodiments of the present disclosure, it is possible to determine regions of interest within a pathology slide image that actually require analysis for immunophenotyping and / or determining whether or not a patient responds to an immunotherapy. That is, the information processing system 100 and / or the user terminal 110 can perform processing (e.g., immunophenotyping and / or immunophenotyping score calculation) only on regions of interest excluding unnecessary regions, rather than on the entire pathology slide image, thereby minimizing computer resources and processing costs. Furthermore, by performing processing (e.g., immunophenotyping and / or immunophenotyping score calculation) only on meaningful regions for immunophenotyping and / or determining whether or not a patient responds to an immunotherapy, more accurate prediction results can be provided.

[0052] FIG. 2 is a block diagram showing the internal configuration of an information processing system 100 according to an embodiment of the present disclosure. In order to provide information associated with an immunophenotype for a pathology slide image, the information processing system 100 can generate information associated with an immunophenotype for the pathology slide image. According to an embodiment, as shown in the figure, the information processing system 100 can include a target item detection unit 210, a region of interest determination unit 220, and an immunophenotype determination unit 230. In FIG. 2, each component of the information processing system 100 represents a functionally separated functional element, and multiple components can be embodied in a form in which they are integrated with each other in an actual physical environment.

[0053] The target item detection unit 210 receives a pathology slide image (e.g., an H&E-stained pathology slide image, an IHC-stained pathology slide image, etc.) and detects one or more target items in the received pathology slide image. In one embodiment, the target item detection unit 210 detects one or more target items in the pathology slide image using an artificial neural network target item detection model. Here, the artificial neural network target item detection model may correspond to a model trained to detect one or more reference target items from a reference pathology slide image. For example, the target item detection unit 210 may detect cell-based target items and / or region-based target items in the pathology slide image. That is, the target item detection unit 210 can detect, as target items within a pathology slide image, tumor cells, lymphocytes, macrophages, dendritic cells, fibroblasts, endothelial cells, blood vessels, cancer stroma, cancer epithelium, cancer areas, normal areas (e.g., normal lymph node architecture areas), etc.

[0054] The region of interest determination unit 220 may determine one or more regions of interest within the pathology slide image. Here, the region of interest may include a region in which one or more target items are detected within the pathology slide image. For example, the region of interest determination unit 220 may determine, as the region of interest, a patch including one or more target items among multiple patches constituting the pathology slide image. In one embodiment, the region of interest determination unit 220 may determine one or more regions of interest based on the detection results of one or more target items within the pathology slide image. For example, the region of interest determination unit 220 may determine one or more regions of interest including at least a partial region of the pathology slide image that satisfies a condition associated with one or more target items. Additionally or alternatively, the region of interest determination unit 220 may determine, as the one or more regions of interest, a region output by inputting the detection results of one or more target items within the pathology slide image and / or the pathology slide image into a region of interest extraction model.

[0055] The immunophenotyping unit 230 can generate information associated with the immunophenotype of one or more regions of interest in a pathology slide image. In one embodiment, the immunophenotyping unit 230 can determine the immunophenotype of one or more regions of interest based on the detection results for one or more target items. For example, the immunophenotyping unit 230 can determine whether the immunophenotype of one or more regions of interest is immunoactive, immunorejective, or immunodeficient based on the detection results for one or more target items. In another embodiment, the immunophenotyping unit 230 can calculate an immunophenotype score for one or more regions of interest based on the detection results for one or more target items. For example, the immunophenotyping unit 230 can calculate an immunoactivity score, an immunorejection score, and / or an immunodeficiency score for one or more regions of interest based on the detection results for one or more target items. To this end, the immunophenotyping unit 230 can calculate a score indicating the probability that the immunophenotype of one or more regions of interest is immunoactive, immunorejective, or immunodeficient.

[0056] In another embodiment, the immunophenotyping unit 230 may generate features associated with one or more immunophenotypes for one or more regions of interest. Here, the features associated with one or more immunophenotypes may include at least one of a statistical value or a vector associated with the immunophenotype. For example, the features associated with one or more immunophenotypes may include a score value associated with the immunophenotype output from an artificial neural network model or a machine learning model. That is, the features may include a score value output in the process of determining the immunophenotype for one or more regions of interest. As another example, the features associated with one or more immunophenotypes may include a density value, number, various statistical values, or a vector value indicating the distribution of immune cells of immune cells corresponding to a threshold (or cut-off) for the immune phenotype.

[0057] As another example, features associated with one or more immune phenotypes may include scalar values ​​or vector values ​​including a relative relationship (e.g., a histogram vector or a graph representation vector taking into account direction and distance) between immune cells or cancer cells and specific cells (e.g., cancer cells, immune cells, fibroblasts, lymphocytes, plasma cells, macrophages, endothelial cells, etc.), or a relative statistical value (e.g., a ratio of the number of specific cells to the number of immune cells, etc.). As another example, features associated with one or more immune phenotypes may include scalar values ​​or vector values ​​including a statistical value (e.g., a ratio of the number of immune cells to the number of cancer stroma regions) or a distribution (e.g., a histogram vector or a graph representation vector, etc.) of immune cells or cancer cells in a specific region (e.g., cancer region, cancer stroma region, tertiary lymphoid structure, normal region, necrosis, fat, blood vessel, high endothelial venule, lymphatic vessel, nerve, etc.).

[0058] As another example, a feature associated with one or more immune phenotypes may include a scalar value or vector value including a relative relationship (e.g., a histogram vector or a graph representation vector taking into account direction and distance) between positive / negative cells and specific cells (e.g., cancer cells, immune cells, fibroblasts, lymphocytes, plasma cells, macrophages, endothelial cells, etc.) based on the expression level of a biomarker, or a relative statistical value (e.g., a ratio of the number of specific cells to the number of immune cells, etc.). As another example, a feature associated with one or more immune phenotypes may include a scalar value or vector value including a statistical value (e.g., a ratio of the number of cancer stroma regions to the number of immune cells, etc.) or a distribution (e.g., a histogram vector or a graph representation vector, etc.) of positive / negative cells based on the expression level of a biomarker in a specific region (e.g., cancer region, cancer stroma region, tertiary lymphoid structure, normal region, necrosis, fat, blood vessel, high endothelial venule, lymphatic vessel, nerve, etc.).

[0059] 2, the information processing system 100 includes a target item detection unit 210, a region of interest determination unit 220, and an immune phenotype determination unit 230. However, the present invention is not limited to this, and some components may be omitted or other components may be added. In one embodiment, the information processing system 100 may further include an immune anticancer drug response prediction unit (not shown), which may generate a prediction result as to whether a patient will respond to an immune anticancer drug based on information associated with the immune phenotype. In another embodiment, the information processing system 100 may further include an output unit (not shown), which may output at least one of the detection results for one or more target items, the immune phenotypes of one or more regions of interest, the prediction result as to whether a patient will respond to an immune anticancer drug, or the density of immune cells in one or more regions of interest.

[0060] 3 is a block diagram illustrating an internal configuration of a user terminal 110 according to an embodiment of the present disclosure. According to an embodiment, as shown in the figure, the user terminal 110 may include an image generating unit 310 and an image output unit 320. In FIG. 3, each component of the user terminal 110 represents a functionally separated functional element, and multiple components may be embodied in a form where they are integrated with each other in an actual physical environment.

[0061] The image generating unit 310 can acquire information associated with an immunophenotype for one or more regions of interest in a pathology slide image. For example, the image generating unit 310 can receive information associated with an immunophenotype for one or more regions of interest generated by an information processing system. Additionally or alternatively, the user terminal 110 can generate information associated with an immunophenotype for one or more regions of interest, thereby allowing the image generating unit 310 to acquire the information associated with the immunophenotype for one or more regions of interest. Additionally or alternatively, the image generating unit 310 can receive information associated with an immunophenotype for one or more regions of interest stored in an internal and / or external device of the user terminal 110.

[0062] The image generator 310 can generate an image showing information associated with the immunophenotype based on information associated with the immunophenotype for one or more regions of interest. In one embodiment, if the immunophenotype of one or more regions of interest is acquired, the image generator 310 can generate an image including visual indicators corresponding to the immunophenotype of the one or more regions of interest. In another embodiment, if one or more immunophenotype scores for one or more regions of interest are received, the image generator 310 can generate an image including visual indicators corresponding to the one or more immunophenotype scores.

[0063] In still another embodiment, the image generating unit 310 may generate an image including visual indicators corresponding to features (e.g., feature values) associated with one or more immunophenotypes. For example, an image including a heatmap based on the expression rate of biomarkers, a classification map classified based on a specific threshold (e.g., a cut-off), etc. may be generated. Here, the classification map may include a map visualizing the results of classifying information associated with PD-L1 expression, such as Tumor Proportion Score (TPS) and / or Combined Proportion Score (CPS), based on a specific threshold.

[0064] The image output unit 320 can output an image showing information associated with the immunophenotype via a display device. In one embodiment, the image output unit 320 can output an image including one or more regions of interest and a visual indicator on a pathology slide image via a display device. Alternatively, the image output unit 320 can overlay an image including a visual indicator on one or more regions of interest on the pathology slide image.

[0065] 3, the image generating unit 310 is illustrated as being included in the user terminal 110, but is not limited thereto, and the image generating unit 310 may be included in any external device (e.g., the information processing system 100, etc.) capable of wired and / or wireless communication with the user terminal 110. According to this configuration, the image output unit 320 of the user terminal 110 can receive an image generated from an external device and display the received image on a display device to which the user terminal 110 is connected wired and / or wirelessly.

[0066] 2 and 3, the target item detection unit 210, the region of interest determination unit 220, the immunophenotyping unit 230, the image generation unit 310, and the image output unit 320 are shown as being executed separately in the information processing system 100 and the user terminal 110, but this is not limiting and these components may be executed by a single device. In other embodiments, these components may be distributed and processed in any combination by any number of devices (e.g., the information processing system 100 and the user terminal 110).

[0067] FIG. 4 is a flowchart illustrating a method 400 for providing information associated with an immunophenotype for a pathology slide image according to one embodiment of the present disclosure. In one embodiment, the method 400 for providing information associated with an immunophenotype for a pathology slide image can be performed by a processor (e.g., at least one processor of a user terminal and / or at least one processor of an information processing system). The method 400 for providing information associated with an immunophenotype for a pathology slide image can begin by the processor acquiring information associated with an immunophenotype for one or more regions of interest in the pathology slide image (S410). Here, the information associated with the immunophenotype for the one or more regions of interest can include an immunophenotype for the one or more regions of interest (e.g., at least one of an immunoreactivity score, an immunoremoval score, or an immunodeficiency score) and / or an immunophenotype score for the one or more regions of interest (e.g., at least one of an immunoreactivity score, an immunoremoval score, or an immunodeficiency score). Additionally or alternatively, the information associated with the immunophenotype for the one or more regions of interest can include features associated with one or more immunophenotypes for the one or more regions of interest (e.g., statistical values ​​or vectors associated with the immunophenotypes).

[0068] In one embodiment, the one or more regions of interest may be determined based on detection results of one or more target items in the pathology slide image. For example, the one or more regions of interest may include at least a partial region of the pathology slide image that satisfies conditions associated with one or more target items. As another example, the one or more regions of interest may be regions output by inputting detection results of one or more target items in the pathology slide image and / or the pathology slide image into a region of interest extraction model. In this case, the region of interest extraction model may correspond to a model trained to output a reference region of interest by inputting detection results of one or more target items in a reference pathology slide image (e.g., a reference pathology slide image including the target item detection results) and / or the reference pathology slide image.

[0069] The processor can generate an image showing information associated with the immunophenotype based on the information associated with the immunophenotype for one or more regions of interest (S420). In one embodiment, the processor can generate an image including visual indicators corresponding to the immunophenotype of one or more regions of interest. In another embodiment, the processor can generate an image including visual indicators corresponding to one or more immunophenotype scores. For example, the processor can generate an image including visual indicators corresponding to a score for immune activation. As another example, the processor can generate an image including visual indicators corresponding to a score for immune exclusion. As yet another example, the processor can generate an image including visual indicators corresponding to a score for immune deficiency. In yet another embodiment, the processor can generate an image including visual indicators corresponding to features associated with one or more immune phenotypes. For example, the processor can generate an image including visual indicators corresponding to values ​​of features associated with one or more immune phenotypes.

[0070] The processor can then output an image showing information associated with the immunophenotype (S430). In one embodiment, the processor can output both an image including one or more regions of interest in the pathology slide image and the visual indicator. In another embodiment, the processor can overlay an image including the visual indicator over one or more regions of interest in the pathology slide image.

[0071] In one embodiment, the processor can obtain detection results for one or more target items from the pathology slide image and generate an image indicative of the detection results for the one or more target items. The processor can then output the image indicative of the detection results for the one or more target items.

[0072] FIG. 5 illustrates an example of determining one or more regions of interest 522_1, 522_2, 522_3, 522_4, 524, and 526 within a pathology slide image 510 according to one embodiment of the present disclosure. To predict whether a patient will respond to an immunotherapy, a user (e.g., a doctor or researcher) can acquire patient tissue (e.g., tissue immediately before treatment or tissue after immunotherapy) and generate one or more pathology slide images. For example, the user can generate a pathology slide image by staining the acquired patient tissue with H&E and digitizing the H&E-stained tissue slide using a scanner. As another example, the user can generate a pathology slide image by staining the acquired patient tissue with IHC and digitizing the IHC-stained tissue slide using a scanner.

[0073] The region of interest determination unit 220 of the information processing system can determine one or more regions of interest in a pathology slide image. Here, the region of interest can correspond to various shapes such as a circle, a square, a rectangle, a polygon, and a contour. In one embodiment, the region of interest determination unit 220 can determine one or more regions of interest based on the detection result of one or more target items in the pathology slide image. For example, the region of interest determination unit 220 can determine one or more patches (e.g., 1 mm ) generated by dividing the pathology slide image into N grids (where N is an arbitrary natural number). 2 sized patch), a patch in which one or more items of interest (e.g., an item associated with cancer and / or an immune cell) are detected can be determined as a region of interest.

[0074] In one embodiment, the region of interest determination unit 220 may determine one or more regions of interest in the pathology slide image so as to include at least a partial region that satisfies a condition associated with one or more target items. For example, the region of interest determination unit 220 may determine, as the region of interest, a region in the pathology slide image where the number and / or size of target items (e.g., tumor cells, immune cells, cancer areas, cancer stroma, etc.) is equal to or greater than a reference value. Additionally or alternatively, the region of interest determination unit 220 may determine, as the region of interest, a region in the pathology slide image where a numerical value, such as a ratio or density of the target items, is equal to or greater than a reference value. Here, the target items may correspond to cells and / or regions detected for determining an immunophenotype. Furthermore, the reference value may refer to a numerical value for the target items that is set to define a statistically significant immunophenotype and / or a clinically significant immunophenotype.

[0075] In this case, the region of interest determination unit 220 can determine a region of any size in the pathology slide image as the region of interest. For example, the size of the region of interest can be dynamically determined to satisfy conditions associated with one or more target items. That is, the size of the region of interest is not fixedly predetermined, but can be dynamically determined by the region of interest determination unit 220 determining, as the region of interest, a region where the number, size, ratio, and / or density of the target items are equal to or greater than a reference value. Alternatively, the region of interest determination unit 220 can determine one or more regions of interest such that the size of the region of interest has a statically predetermined value.

[0076] In another embodiment, the region of interest determination unit 220 may determine one or more regions of interest to be output by inputting the pathology slide image and / or the pathology slide image into a region of interest extraction model. Here, the region of interest extraction model may correspond to a machine learning model (e.g., a neural network, a CNN, an SVM, etc.) trained to output a reference region of interest by inputting the pathology slide image and / or the detection result of one or more target items in a reference pathology slide image. In this case, the size of the region of interest may be determined dynamically and / or statically.

[0077] In one embodiment, when multiple regions of interest are determined for a pathology slide image, the region of interest determination unit 220 may determine the multiple regions of interest such that at least some of the multiple regions of interest overlap each other. For example, when the region of interest determination unit 220 determines a first region of interest and a second region of interest for a pathology slide image, at least a portion of the first region of interest and at least a portion of the second region of interest may overlap each other. In another embodiment, when multiple regions of interest are determined for a single pathology slide image, the region of interest determination unit 220 may determine the multiple regions of interest such that at least some of the multiple regions of interest do not overlap each other.

[0078] As shown in the figure, the region of interest determiner 220 can receive a pathology slide image (e.g., a pathology slide image including a detection result for a target item) 510 and determine one or more regions of interest 522_1, 522_2, 522_3, 522_4, 524, and 526 in the pathology slide image. For example, the region of interest determiner 220 can determine a region of interest of a particular size (e.g., 1 mm as a default size) that satisfies a condition associated with the target item. 2 ) may determine regions 522_1, 522_2, 522_3, and 522_4 as the regions of interest. As another example, the region of interest determination unit 220 may determine region 524 that satisfies a condition associated with the target item as the region of interest, and the size of the region of interest may be dynamically determined. As yet another example, the region of interest determination unit 220 may determine elliptical region 526 that satisfies a condition associated with the target item as the region of interest.

[0079] 6 is a diagram illustrating an example of generating an immunophenotyping result 620 according to an embodiment of the present disclosure. In one embodiment, the immunophenotyping unit 230 can generate the immunophenotyping result 620 for one or more regions of interest based on the detection of one or more items of interest (e.g., items associated with cancer and / or immune cells) in the one or more regions of interest. For example, the immunophenotyping unit 230 can determine the immunophenotype of the region of interest as at least one of immune activation, immune exclusion, or immune deficiency based on the detection of one or more items of interest in the one or more regions of interest.

[0080] In another embodiment, the immunophenotyping unit 230 may calculate an immunophenotype score for one or more regions of interest based on the detection results of one or more target items in one or more regions of interest. For example, the immunophenotyping unit 230 may calculate at least one of an immune activation score, an immune exclusion score, or an immune deficiency score for the region of interest based on the detection results of one or more target items in the region of interest. In this case, the immunophenotyping unit 230 may determine the immune phenotype of the region of interest based on at least one of the immune activation score, the immune exclusion score, or the immune deficiency score for the one or more regions of interest. For example, if the immune activation score of a specific region of interest is equal to or greater than a threshold, the immunophenotyping unit 230 may determine the immune phenotype of the region of interest as immune activation.

[0081] In one embodiment, the immunophenotyping unit 230 calculates at least one of the number, distribution, and density of immune cells in one or more regions of interest, and determines the immunophenotype and / or immunophenotype score of the one or more regions of interest based on at least one of the calculated number, distribution, and density of immune cells. For example, the immunophenotyping unit 230 calculates the density of immune cells in the cancer area (lymphocytes in cancer area) and the density of immune cells in the cancer stroma (lymphocytes in cancer stroma area) in one or more regions of interest, and determines the immunophenotype of the one or more regions of interest based on at least one of the density of immune cells in the cancer area or the density of immune cells in the cancer stroma. Additionally or alternatively, the immunophenotyping unit 230 can determine the immunophenotype of the one or more regions of interest as one of immune activation, immune exclusion, or immune deficiency based on the number of immune cells contained in a specific region within the cancer area.

[0082] For example, the immunophenotyping unit 230 may determine the immunophenotype of the first region of interest 612 as immune active if the density of immune cells in the cancerous region is equal to or greater than a first threshold density. The immunophenotyping unit 230 may determine the immunophenotype of the second region of interest 614 as immune exclusion if the density of immune cells in the cancerous region is less than the first threshold density and the density of immune cells in the cancer stroma is equal to or greater than a second threshold density. The immunophenotyping unit 230 may determine the immunophenotype of the third region of interest 616 as immune deficiency if the density of immune cells in the cancerous region is less than the first threshold density and the density of immune cells in the cancer stroma is less than the second threshold density. Here, the first threshold density may be determined based on the distribution of immune cell densities in the cancerous region in each of the multiple regions of interest in the multiple pathology slide images. Similarly, the second threshold density may be determined based on the distribution of immune cell densities in the cancer stroma in each of the multiple regions of interest in the multiple pathology slide images.

[0083] In another embodiment, the immunophenotyping unit 230 may input features for each of the one or more regions of interest into an artificial neural network immunophenotyping model to determine the immunophenotype and / or immunophenotype score for each of the one or more regions of interest. Here, the artificial neural network immunophenotyping model may correspond to a classifier trained to determine the immunophenotype of the reference region of interest as one of immune activation, immune exclusion, or immune deficiency by inputting features for the reference region of interest. Here, the features for each of the one or more regions of interest may include statistical features for one or more target items in each of the one or more regions of interest (e.g., the density or number of specific target items in the region of interest), geometric features for one or more target items (e.g., features including relative position information between specific target items, etc.), and / or image features corresponding to each of the one or more regions of interest (e.g., features extracted from multiple pixels included in the region of interest, image vectors corresponding to the region of interest, etc.). Additionally or alternatively, the features for each of the one or more regions of interest may include statistical features for one or more items of interest in each of the one or more regions of interest, geometric features for one or more items of interest, or image features corresponding to each of the one or more regions of interest that are a combination of two or more features.

[0084] As shown in the figure, the immunophenotyping unit 230 can receive one or more regions of interest 612, 614, and 616 and generate an immunophenotyping result 620. Here, one or more regions of interest 612, 614, and 616 can include target item detection results for the regions of interest. Therefore, the immunophenotyping unit 230 can generate an immunophenotyping result 620 for the regions of interest including the target item detection results. For example, the immunophenotyping unit 230 can generate an immunophenotyping result 620 including an immunophenotype of the region of interest and / or an immunophenotype score for the region of interest. The thus generated immunophenotyping result 620 and / or one or more regions of interest 612, 614, and 616 can be provided to a user terminal.

[0085] 7 is a diagram illustrating an example of outputting an immunophenotype determination result according to one embodiment of the present disclosure. An information processing system (e.g., at least one processor of the information processing system) provides information associated with an immunophenotype for one or more regions of interest in a pathology slide image to a user terminal, and the user terminal outputs the received information via an output device to provide it to a user. Here, the information associated with the immunophenotype for one or more regions of interest may include an immunophenotype score for the one or more regions of interest. The user terminal (e.g., at least one processor of the user terminal) may output an image showing the information associated with the immunophenotype for one or more regions of interest.

[0086] To this end, the user terminal can generate an image including visual indicators corresponding to one or more immunophenotype scores for one or more regions of interest. For example, the user terminal can generate an image including visual indicators corresponding to one or more immunophenotype scores for one or more regions of interest in an area corresponding to the region or regions of interest. Here, the one or more immunophenotype scores can include at least one of a score for immune activation, a score for immune exclusion, or a score for immune deficiency. In addition, the visual indicators can include color (e.g., color, brightness, saturation, etc.), text, images, marks, graphics, etc.

[0087] For example, for a score for immune activity (i.e., an immune activity score), the user terminal may generate an image including more saturated colors in areas corresponding to regions of interest with higher immune activity scores and less saturated colors in areas corresponding to regions of interest with lower immune activity scores. As another example, for a score for immune activity, the user terminal may generate an image including a first visual marker in areas corresponding to regions of interest where the immune activity score falls within a first score interval, a second visual marker in areas corresponding to regions of interest where the immune activity score falls within a second score interval, and a third visual marker in areas corresponding to regions of interest where the immune activity score falls within a third score interval. Here, the first visual marker, the second visual marker, and the third visual marker may be different from each other.

[0088] In one embodiment, the user terminal can output both one or more regions of interest in the pathology slide image and an image including a visual marker. For example, the user terminal can simultaneously display an image generated as described above (e.g., an image including a visual marker) and at least a partial region of the pathology slide image (e.g., a region corresponding to the generated image) on a display device. In another embodiment, the user terminal can overlay an image including a visual marker on one or more regions of interest in the pathology slide image. For example, the user terminal can display an image generated as described above (e.g., an image including a visual marker) on a display device, overlapping at least a partial region of the pathology slide image (e.g., a region corresponding to the generated image).

[0089] For example, the user terminal may generate image 710, as shown in the figure, in which regions corresponding to regions of interest whose immune activation scores fall within a first score interval are displayed in white, regions corresponding to regions of interest whose immune activation scores fall within a second score interval are displayed in light gray, regions corresponding to regions of interest whose immune activation scores fall within a third score interval are displayed in dark gray, and regions other than the regions of interest are displayed in black. The user terminal may then arrange image 710 including the visual indicators and a region 720 of the pathology slide image corresponding to image 710 side by side and display them together on the user interface.

[0090] Additionally or alternatively, the user terminal may further display, on the user interface, text, numerical values, graphs, etc., for "Total Tissue Region" (e.g., tissue region in a pathology slide image), "Cancer Region" (e.g., cancer region in a pathology slide image), "Analyzable Region" (e.g., region of interest in a pathology slide image), and "Immune Phenotype Proportion" (e.g., immune phenotype ratio) as information associated with the immune phenotype. Additionally or alternatively, a region of the pathology slide image displayed on the display device may include one or more target item detection results for that region. That is, when outputting an image of at least a partial region of the pathology slide image, the user terminal may output, on the display device, an image of at least a partial region in which detection results for one or more target items are displayed.

[0091] 8 is a diagram illustrating an example of outputting an immunophenotyping result according to another embodiment of the present disclosure. In one embodiment, the information associated with the immunophenotype for one or more regions of interest acquired by a user terminal (e.g., at least one processor of the user terminal) can include the immunophenotype of the one or more regions of interest. For example, the user terminal can output an image showing the information associated with the immunophenotype for the one or more regions of interest.

[0092] To this end, the user terminal can generate an image including visual markers corresponding to the immunophenotypes of one or more regions of interest. For example, the user terminal can generate an image including visual markers corresponding to the immunophenotypes of one or more regions of interest in regions corresponding to the regions of interest. That is, the image can include a first visual marker in a region corresponding to a region of interest whose immunophenotype is immunoactive, a second visual marker in a region corresponding to a region of interest whose immunophenotype is immunoexclusion, and a third visual marker in a region corresponding to a region of interest whose immunophenotype is immunodeficiency. In this case, the visual markers can include colors (e.g., color, brightness, saturation, etc.), text, images, marks, graphics, etc. that enable each immunophenotype to be distinguished. For example, the first visual marker indicating immunoactivity can be red, the second visual marker indicating immunoexclusion can be green, and the third visual marker indicating immunodeficiency can be blue. Additionally or alternatively, the first visual marker indicative of immune activity can be a circle, the second visual marker indicative of immune exclusion can be a triangle, and the third visual marker indicative of immune deficiency can be an X.

[0093] In one embodiment, the user terminal can output both an image including one or more regions of interest in the pathology slide image and a visual marker. For example, the user terminal can display an image generated as described above (e.g., an image including a visual marker) on a display device together with at least a partial region of the pathology slide image (e.g., a region corresponding to the generated image). In another embodiment, the user terminal can overlay an image including a visual marker on one or more regions of interest in the pathology slide image. For example, the user terminal can display an image generated as described above (e.g., an image including a visual marker) on a display device, transparently, semi-transparently, or opaquely overlapping at least a partial region of the pathology slide image (e.g., a region corresponding to the generated image).

[0094] For example, the user terminal may generate an image including diagonal markers in regions of interest corresponding to regions of interest with an immunoreactive immunophenotype, vertical markers in regions of interest corresponding to regions of interest with an immunorejection immunophenotype, and horizontal markers in regions of interest corresponding to regions of interest with an immunodeficiency immunophenotype, as shown in the figure. The user terminal may then display, on the user interface, an image 810 in which the image including the visual markers is overlaid on the corresponding region of the pathology slide image. Additionally, the user terminal may separately display (e.g., in the form of a minimap) an image 820 of at least a portion of the pathology slide image (e.g., the region corresponding to the image including the visual markers and / or the entire pathology slide image) on the user interface.

[0095] Additionally or alternatively, the user terminal may display on the user interface text, numerical values, graphs, etc., for information associated with the immunophenotype, such as "ANALYSIS SUMMARY," "Biomaker Findings," "Score" (e.g., immune activity score), "Cutoff" (e.g., reference value used in immunophenotyping), "Total Tissue Region," "Cancer Region," "Analyzable Region," "Immune Phenotype Proportion," and "Tumor Infiltrating Lymphocyte Density" (e.g., density of immune cells in a cancer region, density of immune cells in a cancer stroma region, etc.). Additionally or alternatively, an area of ​​the pathology slide image displayed on the display device may include detection results for one or more target items for that area. That is, when outputting an image of at least a portion of the pathology slide image, the user terminal may output, on the display device, an image of at least a portion of the area in which detection results for one or more target items are displayed.

[0096] 9 is a diagram illustrating an example of an artificial neural network model 900 according to an embodiment of the present disclosure. The artificial neural network model 900 is an example of a machine learning model, and is a statistical learning algorithm implemented based on the structure of a biological neural network in machine learning technology and cognitive science, or a structure for executing the algorithm.

[0097] According to one embodiment, the artificial neural network model 900 may be a machine learning model having problem-solving capabilities, in which nodes, which are artificial neurons forming a network through synaptic connections like a biological neural network, repeatedly adjust synaptic weights to learn to reduce the error between a normal output and an inferred output corresponding to a specific input. For example, the artificial neural network model 900 may include any probability model, neural network model, etc. used in artificial intelligence learning methods such as machine learning and deep learning.

[0098] According to one embodiment, the artificial neural network model 900 may include an artificial neural network model configured to detect one or more items of interest from an input pathology slide image. Additionally or alternatively, the artificial neural network model 900 may include an artificial neural network model configured to determine one or more regions of interest from an input pathology slide image.

[0099] The artificial neural network model 900 is embodied as a multilayer perceptron (MLP) composed of multiple nodes and connections between them. The artificial neural network model 900 according to this embodiment may be embodied using one of various artificial neural network model structures, including an MLP. As shown in FIG. 9 , the artificial neural network model 900 includes an input layer 920 that receives an input signal or data 910 from an external device, an output layer 940 that outputs an output signal or data 950 corresponding to the input data, and n hidden layers 930_1 through 930_n (where n is a positive integer) positioned between the input layer 920 and the output layer 940. The hidden layers 930_1 through 930_n receive signals from the hidden layers 930_1 through 930_n and output the signals to the external device.

[0100] The learning methods of the artificial neural network model 900 include a supervised learning method, in which the model learns to optimize problem solving in response to the input of a teacher signal (correct answer), and an unsupervised learning method, in which a teacher signal is not required. In one embodiment, the information processing system can train the artificial neural network model 900 through supervised learning and / or unsupervised learning to detect one or more target items from a pathology slide image. For example, the information processing system can train the artificial neural network model 900 through supervised learning to detect one or more target items from a pathology slide image using label information related to a reference pathology slide image and one or more reference target items.

[0101] In another embodiment, the information processing system may train the artificial neural network model 900 through supervised learning and / or unsupervised learning to determine one or more regions of interest from a pathology slide image. For example, the information processing system may train the artificial neural network model 900 through supervised learning to determine one or more regions of interest from a pathology slide image using a reference pathology slide image and / or detection results of one or more target items from the reference pathology slide image (e.g., a reference pathology slide image including the target item detection results) and label information for the reference regions of interest. Here, the regions of interest and / or reference regions of interest may include at least a portion of the pathology slide image and / or the reference pathology slide image that satisfies a condition associated with one or more target items.

[0102] The artificial neural network model 900 thus trained can be stored in a memory (not shown) of the information processing system and can detect one or more items of interest within the pathology slide image in response to inputs for the pathology slide image received from the communication module and / or memory. Additionally or alternatively, the artificial neural network model 900 can determine one or more regions of interest from the pathology slide image in response to the detection results of the one or more items of interest within the pathology slide image and / or the inputs for the pathology slide image.

[0103] According to one embodiment, the input variables of the artificial neural network model for detecting target items may be one or more pathology slide images (e.g., H&E-stained pathology slide images, IHC-stained pathology slide images). For example, the input variables input to the input layer 920 of the artificial neural network model 900 may be an image vector 910, which is a vector data element that represents one or more pathology slide images. In response to the image input, the output variable output from the output layer 940 of the artificial neural network model 900 may be a vector 950 that represents or characterizes one or more target items detected from the pathology slide images. That is, the output layer 940 of the artificial neural network model 900 may be configured to output a vector that represents or characterizes one or more target items detected from the pathology slide images. In the present disclosure, the output variables of the artificial neural network model 900 are not limited to the types described above and may include any information / data indicating one or more target items detected from the pathology slide images. Furthermore, the output layer 940 of the artificial neural network model 900 may be configured to output a vector that indicates the confidence and / or accuracy of the output target item detection results.

[0104] In another embodiment, the input variables of the machine learning model (i.e., artificial neural network model 900) that determines the region of interest may be the detection results of one or more target items in the pathology slide image (e.g., detection data for target items in the pathology slide image) and / or the pathology slide image. For example, the input variables input to the input layer 920 of the artificial neural network model 900 may be an image vector 910, which is a vector data element that represents the detection results of one or more target items in the pathology slide image and / or the pathology slide image. In response to the input of the detection results of one or more target items in the pathology slide image and / or the pathology slide image, the output variables output from the output layer 940 of the artificial neural network model 900 may be a vector 950 that represents or characterizes one or more regions of interest. In the present disclosure, the output variables of the artificial neural network model 900 are not limited to the types described above and may include any information / data that indicates one or more regions of interest.

[0105] In this manner, the artificial neural network model 900 is trained to extract a normal output corresponding to a specific input by matching multiple input variables with the input layer 920 and output layer 940, respectively, and adjusting synaptic values ​​between nodes included in the input layer 920, hidden layers 930_1 through 930_n, and output layer 940. Through this training process, the artificial neural network model 900 can grasp hidden characteristics of the input variables and adjust synaptic values ​​(or weights) between nodes of the artificial neural network model 900 to reduce the error between the output variables calculated based on the input variables and the desired output. The trained artificial neural network model 900 can be used to output a target item detection result in response to an input pathology slide image. Additionally or alternatively, the artificial neural network model 900 can be used to output one or more regions of interest in response to the input pathology slide image and / or one or more target item detection results for the pathology slide image (e.g., a pathology slide image including the target item detection result).

[0106] 10 is a block diagram of an exemplary computer device (e.g., a user terminal) 1000 for providing information associated with immunophenotypes for pathology slide images according to one embodiment of the present disclosure. As shown in the figure, the computer device 1000 may include one or more processors 1010, a bus 1030, a communication interface 1040, a memory 1020 for loading a computer program 1060 executed by the processor 1010, and a storage module 1050 for storing the computer program 1060. However, FIG. 10 shows only components associated with the embodiment of the present disclosure. Therefore, a person skilled in the art to which the present disclosure pertains will understand that other general-purpose components may be included in addition to the components shown in FIG. 10.

[0107] The processor 1010 controls the overall operation of each component of the computer device 1000. The processor 1010 may include a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphic Processing Unit), or any other type of processor known in the art. The processor 1010 may also perform calculations for at least one application or program for executing methods according to embodiments of the present disclosure. The computer device 1000 may include one or more processors.

[0108] The memory 1020 may store various data, instructions, and / or information. The memory 1020 may load one or more computer programs 1060 from the storage module 1050 to perform methods / operations according to various embodiments of the present disclosure. The memory 1020 may be embodied as a volatile memory such as RAM, although the scope of the present disclosure is not limited in this respect.

[0109] The bus 1030 can provide a communication function between components of the computer device 1000. The bus 1030 can be implemented as various types of buses such as an address bus, a data bus, and a control bus.

[0110] The communication interface 1040 can support wired and wireless Internet communication for the computer device 1000. The communication interface 1040 can also support various communication methods other than Internet communication. To this end, the communication interface 1040 can be configured to include a communication module known in the art.

[0111] The storage module 1050 can non-temporarily store one or more computer programs 1060. The storage module 1050 can include a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable PROM (EEPROM), a non-volatile memory such as flash memory, a hard disk, a removable disk, or any other form of computer-readable storage medium known in the art.

[0112] The computer program 1060 may include one or more instructions that, when loaded into the memory 1020, cause the processor 1010 to perform operations / methods according to various embodiments of the present disclosure. That is, the processor 1010 can perform operations / methods according to various embodiments of the present disclosure by executing the one or more instructions.

[0113] For example, the computer program 1060 may include one or more instructions for performing an operation of acquiring information associated with an immunophenotype for one or more regions of interest in a pathology slide image, an operation of generating an image showing the information associated with the immunophenotype based on the information associated with the immunophenotype for the one or more regions of interest, an operation of outputting the image showing the information associated with the immunophenotype, etc. In this case, a system for predicting a response to an immunological anticancer drug according to some embodiments of the present disclosure may be embodied via the computer device 1000.

[0114] 11 is a diagram illustrating an example of outputting a target item detection result according to one embodiment of the present disclosure. A processor (e.g., at least one processor of a user terminal) can obtain detection results of one or more target items from a pathology slide image. For example, an information processing system can use a target item detection model to detect one or more target items from a pathology slide image and provide the target item detection results to a user terminal. Here, the target item detection model can include a model trained to detect one or more reference target items from a reference pathology slide image.

[0115] The processor can generate an image showing the detection results of one or more target items and output the generated image showing the detection results of one or more target items. In one embodiment, the processor can generate and output an image including visual indicators that distinguish each target item based on the detection results of one or more target items from the pathology slide image. For example, the processor can generate an image including visual indicators that indicate each target item that can be considered when determining an immunophenotype (e.g., cancerous areas, cancerous target areas, blood vessels, cancer cells, immune cells, positive / negative cells based on biomarker expression levels, etc.). Additionally or alternatively, the processor can generate an image including a segmentation map for target items by region, contours for target items with specific structures (e.g., blood vessels), center points of target items by cell, or contours showing the morphology of target items by cell, based on the detection results of one or more target items.

[0116] In one embodiment, the processor may output one or more regions of interest in the pathology slide image and an image showing the detection result of one or more target items for the one or more regions of interest. For example, the processor may output both the image generated as described above (e.g., an image including visual indicators) and at least a partial region of the pathology slide image (e.g., a region corresponding to the generated image). As another example, the processor may overlay an image showing the detection result of one or more target items (e.g., an image including visual indicators) on one or more regions of interest in the pathology slide image. That is, the user terminal may display the image generated as described above (e.g., an image including visual indicators) on a display device connected to the user terminal, overlapping at least a partial region of the pathology slide image (e.g., a region corresponding to the generated image).

[0117] For example, as shown in the figure, the processor may display image 1110 on the user interface, in which a first color indicating an area other than the target item (background), a second color indicating a cancerous epithelial area, a third color indicating a cancerous stromal area, a fourth color indicating an immune cell (e.g., a center point of the immune cell), and a fifth color indicating a cancer cell (e.g., a center point of the cancer cell) are displayed in corresponding areas of the pathology slide image. Here, image 1110 may correspond to an image in which an image indicating the detection results of one or more target items is overlaid (or merged) on the corresponding area of ​​the pathology slide image. Furthermore, the first color, second color, third color, fourth color, and / or fifth color may correspond to different colors that are distinguishable from one another.

[0118] Additionally or alternatively, the user terminal may display, on a user interface, information associated with the immunophenotype, such as visual indicators corresponding to each target item (e.g., color information corresponding to each target item), text, numerical values, indicators, graphs, etc. for the "ANALYSIS SUMMARY." Additionally or alternatively, the user terminal may output a user interface that allows the user to select target items in the pathology slide image for which a visual indicator is to be displayed in a corresponding area. In FIG. 11, color is used as an example of a visual indicator indicating information associated with the immunophenotype, but the present invention is not limited to this.

[0119] 12 is a diagram illustrating an example of outputting target item detection results and immunophenotyping results according to one embodiment of the present disclosure. A processor (e.g., at least one processor of a user terminal) can output an image showing information associated with immunophenotypes for one or more regions of interest (e.g., an immunophenotype map). Further, the processor can output an image showing the detection results of one or more target items. In one embodiment, the processor can output an image showing the information associated with immunophenotypes for one or more regions of interest and / or the detection results of one or more target items overlaid (or merged) on corresponding regions of a pathology slide image.

[0120] For example, the processor may display, on the user interface, image 1210, in which a first color indicating immune deficiency, a second color indicating immune exclusion, a third color indicating immune inflamed, a fourth color indicating cancerous epithelial regions, a fifth color indicating cancerous stromal regions, a sixth color indicating immune cells (e.g., the centers of immune cells), and a seventh color indicating cancer cells (e.g., the centers of cancer cells) are displayed in corresponding regions of the pathology slide image, as shown in the figure. Here, image 1210 may correspond to an image in which an image indicating information associated with immune phenotypes for one or more regions of interest (e.g., an immune phenotype map) and an image indicating detection results of one or more target items are overlaid (or merged) in the corresponding regions of the pathology slide image. Furthermore, the first color, second color, third color, fourth color, fifth color, sixth color, and / or seventh color may correspond to different colors that are distinguishable from one another.

[0121] Additionally or alternatively, the processor may separately (e.g., in the form of a minimap) display an image 1220 of at least a portion of the pathology slide image (e.g., a region corresponding to the image including the visual indicator and / or the entire pathology slide image) on the user interface. Additionally or alternatively, the processor may display, on the user interface, information associated with the immunophenotype, such as visual indicators corresponding to each target item (e.g., color information corresponding to each target item), text, numerical values, graphs, etc. for the "ANALYSIS SUMMARY." Additionally or alternatively, a user interface may be output that allows the user to select information associated with the target item and / or immunophenotype for which a visual indicator is to be displayed in the corresponding region of the pathology slide image. In FIG. 12, color is used as an example of a visual indicator indicating information associated with the immunophenotype and / or target item, but this is not limiting.

[0122] The previous description of the present disclosure is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications of the present disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to various modifications without departing from the spirit or scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the examples described herein, but is intended to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0123] Although example embodiments may be referenced as utilizing aspects of the presently disclosed subject matter in the context of one or more stand-alone computer systems, the present subject matter is not so limited and may be implemented in connection with any computing environment, such as a network or distributed computing environment. Furthermore, aspects of the presently disclosed subject matter may be implemented on or across multiple processing chips or devices, and storage may be similarly affected across multiple devices. Such devices may include PCs, network servers, and handheld devices.

[0124] Although the present disclosure has been described in connection with some embodiments herein, it will be understood that various modifications and changes that would be understood by those of ordinary skill in the art to which the present invention pertains can be made without departing from the scope of the present disclosure, and such modifications and changes should be understood to fall within the scope of the claims appended hereto.

Claims

1. 1. A method for providing immunophenotypic associated information for a pathology slide image, the method being performed by at least one computer device, comprising: acquiring information associated with an immunophenotype for each of a plurality of unit areas included in one or more regions of interest in the pathology slide image; generating an image indicative of information associated with the immunophenotype based on information associated with the immunophenotype for each of the plurality of unit areas; and outputting an image showing information associated with the immunophenotype; Information associated with the immunophenotype, At least one of a plurality of classifications representing the immune environment of each of the plurality of unit areas; at least one score for each of the plurality of classifications; or a feature representing the immune environment of each of the plurality of unit areas; the obtaining step includes obtaining an immune phenotype for each of the plurality of unit areas based on a density of immune cells within the cancerous area; the generating step includes generating an image including visual indicators corresponding to the immunophenotype in each of the plurality of unit areas; A method for providing information associated with an immunophenotype for a pathology slide image, wherein the immunophenotype includes at least one of immune activity, immune exclusion, or immune deficiency.

2. The method for providing information associated with an immunophenotype for a pathology slide image according to claim 1 , wherein the one or more regions of interest are determined based on detection results of one or more target items for the pathology slide image.

3. The method for providing information associated with an immune phenotype for a pathology slide image described in claim 2, wherein the one or more regions of interest include at least a partial area of ​​the pathology slide image that satisfies conditions associated with the one or more target items.

4. the one or more regions of interest are regions output as a result of detecting one or more target items in the pathology slide image or as a result of inputting the pathology slide image into a region of interest extraction model; The method for providing information associated with an immune phenotype for a pathology slide image described in claim 2, wherein the region of interest extraction model is trained to output a reference region of interest by inputting the detection results of one or more target items for a reference pathology slide image or the reference pathology slide image.

5. The outputting step includes:

2. A method for providing information associated with an immunophenotype for a pathology slide image as described in claim 1, comprising a step of outputting an image including the one or more regions of interest in the pathology slide image and the visual marker.

6. The outputting step includes:

2. A method for providing information associated with an immunophenotype for a pathology slide image as described in claim 1, comprising the step of overlaying an image including the visual marker on the one or more regions of interest in the pathology slide image.

7. The obtaining step includes: obtaining one or more immunophenotype scores for each of the plurality of unit areas; The generating step includes: generating an image comprising a visual indicator corresponding to the one or more immunophenotype scores; The method of claim 1, wherein the one or more immunophenotype scores include at least one of a score for immune activity, a score for immune exclusion, or a score for immune deficiency.

8. The outputting step includes: A method for providing information associated with an immunophenotype for a pathology slide image as described in claim 7, comprising a step of outputting an image including the one or more regions of interest in the pathology slide image and the visual marker.

9. The outputting step includes:

8. A method for providing information associated with an immunophenotype for a pathology slide image as described in claim 7, comprising the step of overlaying an image including the visual marker on the one or more regions of interest in the pathology slide image.

10. The obtaining step includes: acquiring features associated with one or more immunophenotypes for each of the plurality of unit areas; The generating step includes: generating an image comprising visual indicators corresponding to features associated with the one or more immunophenotypes; The method of providing information associated with an immunophenotype for a pathology slide image of claim 1, wherein the features associated with one or more immunophenotypes further include at least one statistical value or vector associated with the immunophenotype.

11. The outputting step includes: A method for providing information associated with an immunophenotype for a pathology slide image as described in claim 10, comprising a step of outputting an image including the one or more regions of interest in the pathology slide image and the visual marker.

12. The outputting step includes:

11. A method for providing information associated with an immunophenotype for a pathology slide image as described in claim 10, comprising the step of overlaying an image including the visual marker on the one or more regions of interest in the pathology slide image.

13. obtaining detection results of one or more target items from the pathology slide image; generating an image indicative of the detection of the one or more items of interest; 2. The method of claim 1, further comprising the step of: outputting an image showing the detection results of the one or more target items.

14. A computer program stored on a computer-readable recording medium for executing the method for providing information associated with an immunophenotype for a pathology slide image according to any one of claims 1 to 13.

15. 1. A computer device comprising: a memory for storing one or more instructions; Executing the one or more stored instructions, acquiring information associated with an immunophenotype for each of a plurality of unit areas included in one or more regions of interest in the pathology slide image; generating an image showing information associated with the immunophenotype based on information associated with the immunophenotype for each of a plurality of unit areas included in the one or more regions of interest; a processor configured to output an image indicative of information associated with the immunophenotype; Information associated with the immunophenotype, At least one of a plurality of classifications representing the immune environment of each of the plurality of unit areas; at least one score for each of the plurality of classifications; or a feature representing the immune environment of each of the plurality of unit areas; the obtaining step includes obtaining an immune phenotype for each of the plurality of unit areas based on a density of immune cells within the cancerous area; the generating step includes generating an image including visual indicators corresponding to the immunophenotype in each of the plurality of unit areas; The computer device, wherein the immune phenotype comprises at least one of immune activity, immune exclusion, or immune deficiency.

16. The computer device of claim 15 , wherein the one or more regions of interest are determined based on detection of one or more items of interest in the pathology slide image.

17. The computer device of claim 16 , wherein the one or more regions of interest include at least a portion of the pathology slide image that satisfies a condition associated with the one or more items of interest.

18. the one or more regions of interest are regions output as a result of detecting one or more target items in the pathology slide image or as a result of inputting the pathology slide image into a region of interest extraction model; The computer device of claim 16, wherein the region of interest extraction model is trained to output a reference region of interest by receiving the reference pathology slide image or a detection result of one or more target items in the reference pathology slide image.

19. The processor: The computer device of claim 15 , further configured to output an image including the one or more regions of interest in the pathology slide image and the visual indicator together.

20. The processor: The computer device of claim 15 , further configured to overlay an image including the visual indicator onto the one or more regions of interest in the pathology slide image.

21. The processor: obtaining one or more immunophenotype scores for the plurality of unit regions; further configured to generate an image comprising a visual indicator corresponding to said one or more immunophenotype scores; The computer device of claim 15 , wherein the one or more immunophenotype scores include at least one of a score for immune activity, a score for immune exclusion, or a score for immune deficiency.

22. The processor:

22. The computer device of claim 21, further configured to output an image including the one or more regions of interest in the pathology slide image and the visual indicator together.

23. The processor:

22. The computer device of claim 21, further configured to overlay an image including the visual indicator onto the one or more regions of interest in the pathology slide image. 。

24. The processor: obtaining features associated with one or more immunophenotypes for each of the plurality of unit areas; further configured to generate an image comprising visual indicators corresponding to features associated with the one or more immunophenotypes; The computer device of claim 15 , wherein the features associated with the one or more immunophenotypes include at least one of a statistical value or vector associated with the immunophenotype.

25. The processor:

25. The computer device of claim 24, further configured to output an image including the one or more regions of interest in the pathology slide image and the visual indicator together.

26. The processor:

25. The computer device of claim 24, further configured to overlay an image including the visual indicator onto the one or more regions of interest in the pathology slide image.

27. The processor: obtaining detection results for one or more target items from the pathology slide image; generating an image indicative of the detection of the one or more items of interest; The computer device of claim 15 , further configured to output an image indicative of the detection of the one or more items of interest.

Images