Information processing device, information processing method, and program

Abstract

Analyze the spatial interactions between multiple cell types in the tumor microenvironment. [Solution] An information processing device classifies cells in an image of a tumor microenvironment obtained from a patient into multiple cell groups, extracts the positions of cells contained in the multiple cell groups in the image, assigns nodes to the cells contained in the multiple cell groups based on the extracted positions, connects the nodes with edges, assigns features representing the characteristics of the cells to each node, and assigns features representing the relationship between two cells corresponding to two nodes to each edge, thereby creating a graph, and using a graph neural network (GNN) in the graph, models the spatial relationship between cells based on the features of the assigned nodes and edges.

Description

[Technical Field]

[0001] The present invention relates to a technique for analyzing the tumor microenvironment using graph neural networks. [Background technology]

[0002] Analyzing the spatial interactions between tumor cells and immune cells in the tumor microenvironment (TME) is effective because it provides predictive and prognostic value for therapeutic efficacy. For example, Non-Patent Document 1 discloses analyzing the spatial interactions between tumor cells and immune cells by quantifying the spatial interactions between adjacent cells using distance metrics. For example, the interaction between a tumor cell and the immune cell closest to the tumor cell is quantified. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] James Monkman et al. “Spatial insights into immunotherapy response in non-small cell lung cancer (NSCLC) by multiplexed tissue imaging”, Journal of Translational Medicine, https: / / doi.org / 10.1186 / s12967-024-05035-8 Summary of the Invention [Problem to be solved by the invention]

[0004] The proximity-based method disclosed in Non-Patent Document 1 quantifies the interaction between only tumor cells and the immune cells closest to the tumor cells, and therefore lacks information on the cellular positions and interactions of multiple immune cells relative to tumor cells. Therefore, it cannot be said that the spatial interactions between multiple types of cells, such as tumor cells and immune cells, in the tumor microenvironment are fully analyzed.

[0005] In view of the above problems, the present disclosure provides a new method for analyzing spatial interactions between multiple types of cells in the tumor microenvironment. [Means for solving the problem]

[0006] In order to solve the above problem, one aspect of the information processing device according to the present invention includes a classification unit that classifies cells in an image of a tumor microenvironment obtained from a patient into multiple cell groups; a spatial information extraction unit that extracts the positions of cells contained in the multiple cell groups in the image; a graph creation unit that creates a graph by assigning nodes to cells contained in the multiple cell groups based on the extracted positions, connecting the nodes with edges, assigning features that represent the characteristics of the cells to each node, and assigning features that represent the relationship between two cells corresponding to the two nodes to each edge; and a graph modeling unit that models the spatial relationship between cells in the graph based on the features of the assigned nodes and edges using a graph neural network (GNN).

[0007] In order to solve the above problem, one aspect of the information processing method according to the present invention includes classifying cells in an image of a tumor microenvironment obtained from a patient into multiple cell groups, extracting the positions of cells included in the multiple cell groups in the image, creating a graph by assigning nodes to the cells included in the multiple cell groups based on the extracted positions, connecting the nodes with edges, assigning features representing cell characteristics to each node, and assigning features representing the relationship between two cells corresponding to each node to each edge, and modeling the spatial relationship between cells in the graph using a graph neural network (GNN) based on the features of the assigned nodes and edges.

[0008] In order to solve the above-mentioned problems, one aspect of the program according to the present invention is a program for causing a computer to execute information processing, the program causing the computer to execute processes including: a classification process for classifying cells in an image of a tumor microenvironment obtained from a patient into multiple cell groups; a spatial information extraction process for extracting the positions of cells contained in the multiple cell groups in the image; a graph creation process for creating a graph by assigning nodes to cells contained in the multiple cell groups based on the extracted positions, connecting the nodes with edges, assigning features representing cell characteristics to each node, and assigning features representing the relationship between two cells corresponding to two nodes to each edge; and a modeling process for modeling the spatial relationship between cells in the graph based on the features of the assigned nodes and edges using a graph neural network (GNN). [Effects of the Invention]

[0009] According to the present invention, it is possible to analyze spatial interactions between multiple types of cells in the tumor microenvironment using information on cell positions and interactions. The above-mentioned objects, aspects, and advantages of the present invention, as well as other objects, aspects, and advantages of the present invention not described above, will be understood by those skilled in the art from the following detailed description of the invention by referring to the accompanying drawings and the claims. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 shows an example of the functional configuration of an information processing device according to an embodiment. [Figure 2] FIG. 2 shows a flowchart of the overall processing in the first embodiment. [Figure 3] FIG. 3 shows an example of an image corresponding to one or more processes in the first embodiment. [Figure 4] FIG. 4 shows a conceptual diagram of a learning procedure for a responder prediction model using a subgraph in the first embodiment. [Figure 5] FIG. 5 shows a conceptual diagram of a prediction procedure using the responder prediction model in the first embodiment. [Figure 6] FIG. 6 shows a flowchart of the overall processing in the second embodiment. [Figure 7] FIG. 7 shows an example of an image corresponding to one or more processes in the second embodiment. [Figure 8] FIG. 8 shows a conceptual diagram of a procedure for learning a responder prediction model using a subgraph in the second embodiment. [Figure 9] FIG. 9 shows a conceptual diagram of a prediction procedure using a responder prediction model in the second embodiment. [Figure 10] FIG. 10 shows an example of the hardware configuration of an information processing device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, with reference to the accompanying drawings, an embodiment for carrying out the present invention will be described in detail. Among the components disclosed below, those having the same function will be given the same reference numerals, and their description will be omitted. Note that the embodiment disclosed below is an example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. The present invention is not limited to the following embodiment. Furthermore, not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention.

[0012] In the present disclosure, in an image of a tumor microenvironment (TME) containing tumor cells obtained from a patient (subject), cells are classified into multiple cell groups, spatial information of the multiple types of cells is extracted (specifically, cell locations are extracted (identified)), and a graph is created based on the spatial information, with nodes corresponding to the cells and edges corresponding to the connections between the cells. Here, patients are classified as responders or non-responders. However, these classifications are merely examples of predictive use cases, and other classifications, such as short / long survival time (prognostic use cases) and cancerous / non-cancerous (detective use cases), are also included. The created graph is then modeled based on the node features (i.e., cell features) and edge features (i.e., intercellular features), to model the graph for a graph neural network (GNN). The node locations correspond to cell locations, and the edges correspond to interactions between the nodes. In this way, spatial interactions between multiple types of cells in the TME are analyzed using information on cell locations and interactions. Next, information based on the modeled graph is associated with labels indicating responders or non-responders, which are true labels (ground truth data), to train a responder prediction model. In the present disclosure, the graph-based information is, but is not limited to, multiple subgraphs extracted from the modeled graph. The responder prediction model is a GNN-based machine learning model for learning the relationship between the subgraphs extracted from the modeled graph and the true labels. In other words, the responder prediction model is trained on the relationship between the subgraphs and responders or non-responders. Then, multiple subgraphs obtained from images containing the tumor microenvironment of a target patient are input to the trained responder prediction model to predict whether the target patient will be a responder or non-responder.

[0013] The TME is a microtissue that includes tumor cells (tumor tissue), blood vessels and stromal cells that are mixed around them, immune cells that infiltrate tumor cells, and normal tissue. It can also be described as a network constructed between multiple types of cells, including tumor cells and immune cells. It is known to have many effects on tumor initiation, development, and progression. While traditional cancer treatments have targeted cancer cells, the TME has recently attracted attention as a new target.

[0014] In the present disclosure, the terms responder and non-responder correspond to a responder or non-responder to a given treatment. For example, the term responder may correspond to a patient who has a relatively high level of immune cell infiltration into tumor cells as a result of treatment, while the term non-responder may correspond to a patient who has a relatively low level of immune cell infiltration into tumor cells as a result of treatment. Furthermore, the term responder may correspond to a patient who shows any response to cancer immunotherapy, while the term non-responder may correspond to a patient who does not show any response to cancer immunotherapy.

[0015] A GNN is a type of deep learning model designed to process data with a graph structure. In this disclosure, a GNN is constructed by assigning a node to each tumor cell and immune cell, and connecting the nodes with edges (links). The constructed GNN is trained using the features of each node, the features of the edges between the nodes, and the true labels.

[0016] [Functional configuration of information processing device] 1 shows an example of the functional configuration of an information processing device 10 configured to be able to perform the processing described in the present disclosure. As shown in FIG. 1, the information processing device 10 includes a classification unit 101, a spatial information extraction unit 102, a graph creation unit 103, a graph modeling unit 104, a learning data generation unit 105, a learning unit 106, a prediction unit 107, and a learning model storage unit 110. The learning model storage unit 110 is configured to be able to store a respondent prediction model 111. For example, the learning model storage unit 110 is configured to be able to store the architecture and various parameters of the respondent prediction model 111.

[0017] An overview of the processing of each functional component will be described. The classification unit 101 classifies cells in an image of the tumor microenvironment obtained from a patient into cell groups. These cell groups may be of two or more types depending on the use case. The proposed method can study not only interactions between tumor cells and immune cells, but also interactions between immune cells, interactions between tumor cells, and interactions between tumor cells and other cell types. The spatial information extraction unit 102 extracts positional information of cells contained in the cell groups in the image. The graph creation unit 103 creates a graph by assigning cells contained in the cell groups as nodes based on the extracted positional information and connecting the nodes with edges. Feature values ​​representing cell characteristics are assigned to the nodes, and feature values ​​representing the relationship between two cells corresponding to the two connected nodes are assigned to the edges. The graph modeling unit 104 uses a graph neural network (GNN) to model or learn spatial relationships between cells in the graph created by the graph creation unit 103 based on the feature values ​​of the assigned nodes and edges. In this disclosure, the modeling or learning process may simply be referred to as modeling.

[0018] Hereinafter, two embodiments, a first embodiment and a second embodiment, will be specifically described with reference to the functional configuration of the information processing device 10 shown in FIG.

[0019] [First embodiment] FIG. 2 shows a flowchart of the overall processing according to this embodiment. FIG. 3 shows an example of an image corresponding to one or more of the processes shown in FIG. 2. First, the classification unit 101 detects cells in an image (hereinafter referred to as a medical image) including a tumor microenvironment containing tumor cells obtained from a patient (responder or non-responder) and classifies the detected cells into tumor cells and immune cells (S21). In this embodiment, the classification unit 101 classifies the detected cells into two types of cell groups, namely, tumor cells and immune cells. However, this is not limited thereto, and the classification may be performed into three or more types of cell groups, such as tumor cells, immune cells, and stromal cells. This is to enable analysis of not only interactions between tumor cells and immune cells, but also interactions between immune cells, interactions between tumor cells, and interactions between tumor cells and other cell types. In this embodiment, the medical image is an image in which multiple types of cell groups can be distinguished (including images that can be visualized), such as a multiplex immunofluorescence (mIF) image. An mIF image is an image in which different types of cells, etc., are stained using multiple fluorescent dye-labeled antibodies. Image 30 in Figure 3 is an example of an mIF image, showing a tumor microenvironment containing tumor cells, where the cells are identified by multiple types of antibodies (Image 30 is a black and white image, but the actual mIF image is shown in different colors).

[0020] Cell detection in medical images can be performed, for example, as follows. When multiple types of cells in a medical image are identified by different colors, cells may be detected by identifying each color. When multiple types of cells in a medical image are identified by one or more pixels corresponding to predetermined numerical values, cells may be detected using a predetermined threshold. Watershed segmentation may be used to detect cells. Deep learning may also be used to detect cells. When deep learning is used, cells can be detected by using, for example, an encoder-decoder-based architecture or bounding box detection.

[0021] After cells are detected, the classification of multiple cell types can be performed using, for example, predetermined rules based on cell structure or predetermined features (e.g., cell shape, size, texture, intensity, etc.). Alternatively, machine learning / deep learning can be used to classify multiple cell types. When machine learning / deep learning is used, cells can be classified using classifiers such as random forests, logistic regression, support vector machines, convolutional neural networks, and visual transformers. Image 31 in Figure 3 is an example of an image in which cells are detected in image 30 and then classified into tumor cells and immune cells.

[0022] When cells are detected in a medical image and classified into tumor cells and immune cells, the spatial information extraction unit 102 extracts spatial information of the tumor cells and immune cells (S22). In this embodiment, the spatial information extraction unit 102 defines a predetermined two-dimensional coordinate system (i.e., a two-dimensional coordinate system, XY coordinate system) in the medical image and extracts (identifies) the cell positions of the tumor cells and immune cells in the two-dimensional coordinate system. The spatial information extraction unit 102 extracts the spatial information of the tumor cells and immune cells by extracting the X coordinate and the Y coordinate of the tumor cells and immune cells in the image classified into tumor cells and immune cells.

[0023] Once the spatial information of the tumor cells and immune cells is extracted, the graph creation unit 103 creates a graph based on the spatial information (S23). In this embodiment, the graph creation unit 103 creates a graph by assigning nodes to the positions of the tumor cells and immune cells specified by X and Y coordinates and connecting the nodes with edges. Here, the nearest node may be connected to each node with an edge. Alternatively, nodes located within a predetermined range may be connected to each node with an edge. Alternatively, a predetermined number of neighboring nodes may be connected to each node with edges. Alternatively, the nodes may be connected with edges using Delaunay triangulation. Delaunay triangulation is a method for creating a continuous, non-overlapping triangular mesh from a set of points (i.e., nodes). Alternatively, the nodes may be connected with edges using a hierarchical method. For example, a predetermined direction is defined in a two-dimensional coordinate system (XY coordinate system), and edges are used to connect a higher node in that direction to one or more lower nodes. Alternatively, the K-nearest neighbors method may be used to connect each node with an edge. This creates a graph with nodes corresponding to tumor cells and immune cells and with edges connecting each node.

[0024] Furthermore, the graph creation unit 103 assigns features representing cell characteristics to each node in the graph (i.e., features of the cell corresponding to the node) and features representing the relationship between two cells corresponding to two connected nodes to each edge (i.e., features between two cells corresponding to two nodes connected by each edge, and features of the interaction between the edges). The features representing cell characteristics may include, for example, at least one of the cell type (e.g., features related to the cell's function), cell size, characteristics of biomarker expression levels in the cell, the relationship with nodes corresponding to neighboring cells (e.g., the presence or absence of cells of the same type in the vicinity), and the coordinates of the center of the node corresponding to the cell. The features representing the relationship between two cells may include at least one of the edge type (the type of connection between cells) (e.g., information indicating whether the connection is between cells of the same type or between cells of different types) and the edge distance (e.g., the distance between nodes corresponding to two cells). Image 32 in FIG. 3 is an example of a graph in which nodes corresponding to tumor cells and immune cells in image 31 are connected by edges.

[0025] Next, the graph modeling unit 104 uses a GNN to model the spatial relationships between cells in the graph created by the graph creation unit 103 based on the features of the assigned nodes and edges (S24). The processing by the graph modeling unit 104 may include message passing, i.e., exchanging node features between nodes, and aggregating neighborhood information around each node and exchanging information between nodes. Message passing allows pairs of nodes to better learn the interactions between them. Examples of GNNs (i.e., neural networks with a GNN structure) include GCN (Graph Convolutional Network), GIN (Graph Isomorphism Network), GRAPHSAGE (Graph Sample and Aggregation), and GAT (Graph Attention Network). These GNN examples represent various methods for aggregating neighborhood information and transmitting messages between nodes. Through this graph modeling procedure, spatial interactions between multiple types of cells are analyzed (learned) using information on cell positions and interactions.

[0026] The graph modeling unit 104 also extracts multiple subgraphs from the modeled graph as graph-based information (S24). The graph modeling unit 104 extracts subgraphs from the modeled graph under the condition that the subgraph has at least (N+1) nodes (N is an integer greater than or equal to 2) and N hops (N consecutive edges). Setting the N-hop condition in this manner means extracting subgraphs that better represent the inter-node relationships due to edge connections. Because the extracted subgraphs are used as training data for training the respondent prediction model 111, it is preferable for the graph modeling unit 104 to extract multiple subgraphs from the entire modeled graph, rather than extracting multiple subgraphs locally from the modeled graph. This allows subgraphs of various patterns to be extracted from the entire modeled graph, contributing to accurate training. The subgraph group 33 in FIG. 3 is an example of multiple extracted subgraphs. The subgraph group 33 includes three subgraphs, but may include more than three subgraphs.

[0027] Next, the training data generation unit 105 generates training data for the responder prediction model 111 (S25). The training data is composed of the multiple subgraphs extracted in S24 and labels indicating responders or non-responders as true labels. The labels indicating responders or non-responders are determined based on whether the patient from whom the original medical images from which the multiple subgraphs are extracted were acquired is a responder or non-responder. As described above, a subgraph has at least (N+1) nodes (N is an integer equal to or greater than 2) and N hops (N consecutive edges).

[0028] Once the training data is generated, the training unit 106 uses the training data to train the respondent prediction model 111 (S26). The respondent prediction model 111 is a GNN for learning the relationship between subgraphs and true labels (labels indicating responders or non-responders). FIG. 4 shows a conceptual diagram of the training procedure of the respondent prediction model 111 using subgraphs. In FIG. 4, each subgraph of the subgraph group 33 is associated with a true label 41, and the training unit 106 inputs each subgraph of the subgraph group 33 to the respondent prediction model 111 to obtain an output (i.e., a predicted label 40). The respondent prediction model 111 according to this embodiment includes, for example, a feature extractor 43 and a classifier 44. The feature extractor 43 extracts a feature vector representing the feature of the input subgraph from the subgraph, and the classifier 44 is configured to classify the feature vector to predict a label indicating a responder or non-responder and output the predicted label 40.

[0029] When the predicted label 40 is output, the learning unit 106 trains the responder prediction model 111 using a loss function 42, which is a function that derives the magnitude of the difference (loss) between the true label 41 (i.e., the correct label) associated with the input subgraph and the predicted label. Specifically, the learning unit 106 trains the responder prediction model 111 so as to reduce the value of the loss derived from the loss function 42. When a subgraph is input, the trained responder prediction model 111 is configured to predict and output a responder or non-responder label for the input subgraph. Specifically, the trained responder prediction model 111 is configured to predict and output whether the original medical image of the graph from which the input subgraph was extracted was acquired from a responder or non-responder patient.

[0030] As described above, in this embodiment, the responder prediction model 111 is trained using a subgraph in which each node has a node characteristic and each edge has an edge characteristic. This enables the responder prediction model 111 to learn the relationship between the subgraph and responders or non-responders based on the cellular characteristics and the intercellular characteristics.

[0031] After training the responder prediction model 111, the prediction unit 107 uses the trained responder prediction model 111 to predict whether the target patient will be a responder or a non-responder from a medical image including a tumor microenvironment obtained from the target patient (S27). Specifically, the information processing device 10 performs the processes from S21 to S24 on the medical image obtained from the target patient to extract multiple subgraphs. The prediction unit 107 then inputs each of the multiple subgraphs extracted for the target patient into the trained responder prediction model 111 and predicts a responder or non-responder label for the input subgraph. The prediction unit 107 obtains multiple predicted labels by predicting a responder or non-responder label for all of the multiple subgraphs extracted for the target patient. Thereafter, the prediction unit 107 aggregates the multiple predicted labels to ultimately predict (determine) whether the target patient will be a responder or a non-responder. For example, the prediction unit 107 aggregates the multiple predicted labels to ultimately determine a responder prediction score that indicates the likelihood that the target patient is a responder. The score is indicated by a value between 0 and 1, for example, and the closer the score is to 1, the higher the likelihood that the target patient is a responder.

[0032] FIG. 5 shows a conceptual diagram of a prediction procedure using the responder prediction model 111. In FIG. 5, the subgraph group 50 includes multiple subgraphs extracted by performing the processes S21 to S24 on medical images including the tumor microenvironment obtained from a target patient. The prediction unit 107 inputs each subgraph included in the subgraph group 50 into the trained responder prediction model 111, predicts the responder or non-responder label for the input subgraph, and obtains predicted labels 51. The prediction unit 107 predicts the responder or non-responder label for all subgraphs included in the subgraph group 50, thereby obtaining multiple predicted labels 51. The prediction unit 107 then aggregates the multiple predicted labels 51 (aggregation 52) to finally obtain a prediction result 53 indicating whether the target patient is a responder or non-responder.

[0033] The aggregation 52 of the multiple predicted labels 51 indicating responders or non-responders is a statistical aggregation, and includes, for example, taking a majority vote of the multiple predicted labels 51. That is, the prediction unit 107 may acquire the label that is more prevalent among the multiple predicted labels 51 as the final prediction result 53. Alternatively, the prediction unit 107 may assign 1 to the labels of responders and 0 to the labels of non-responders among the multiple predicted labels 51, and acquire the average of the total values ​​as the final prediction result 53. In this case, the prediction result 53 indicates the possibility of being a responder as a value between 0 and 1.

[0034] As described above, the information processing device 10 according to the first embodiment assigns nodes to cells based on spatial information (i.e., cell positions) of multiple cells detected and classified from medical images, connects the nodes with edges, and models a graph using the node features and edge features (i.e., interaction information). Thus, spatial interactions between multiple types of cells in a tumor microenvironment are analyzed using the cell position and interaction information. The information processing device 10 then trains the responder prediction model 111 using subgraphs extracted from the modeled graph and true labels indicating responders or non-responders. The responder prediction model 111 can thus learn the relationship between subgraphs and responders or non-responders based on cellular features and intercellular features. Furthermore, the information processing device 10 uses the trained responder prediction model 111 to predict whether a patient is a responder or non-responder based on subgraphs extracted from medical images obtained from the patient, thereby achieving high prediction accuracy.

[0035] [Second embodiment] Next, a second embodiment will be described. The same description as in the first embodiment will be omitted. FIG. 6 shows a flowchart of the overall processing according to this embodiment. FIG. 7 shows an example of an image corresponding to one or more of the processes shown in FIG. 6. First, the classification unit 101 detects cells in an image (medical image) including a tumor microenvironment containing tumor cells obtained from a patient (responder or non-responder) and classifies the detected cells into two types of cell groups (a first cell type group and a second cell type group) based on the cell phenotype (phenotype marker) (S61). The first cell type group includes cells of a first phenotype, and the second cell type group includes cells of a second phenotype. A pair of cells of the first phenotype and cells of the second phenotype is referred to as a cell type pair. In this embodiment, the classification unit 101 ultimately classifies multiple types of cell type pairs into two types of cell groups. Here, the classification unit 101 classifies the first cell type pair into two types of cell groups. In this embodiment, the medical image is an image in which multiple types of cells can be identified (including an image in which multiple types of cells can be visualized), such as an mIF image. Image 70 in Figure 7 is an example of an mIF image, showing a tumor microenvironment containing tumor cells, where the cells are identified using multiple types of antibodies (although image 70 is a black and white diagram, an actual mIF image is shown in different colors).

[0036] Cell detection in a medical image can be performed using a procedure similar to that of S21 in FIG. 2 described in the first embodiment. After cell detection, classification into two types of cell groups can be performed using staining colors if image 70 is an mIF image. For example, the cell type pair is PANCK+ cells and CD8+ and CD69+ cells. In this case, image 70 can be classified into a first cell group of PANCK+ cells and a second cell group of CD8+ and CD69+ cells. PANCK stands for Pan-Cytokeratin. Image 71 in FIG. 7 is an example of an image classified into two types of cell groups from image 70.

[0037] When cells are detected in a medical image and classified into two types of cell groups, the spatial information extraction unit 102 extracts spatial information of the cells contained in each of the two types of cell groups (S62). In this embodiment, the spatial information extraction unit 102 defines a predetermined two-dimensional coordinate system (i.e., a two-dimensional coordinate system, XY coordinate system) in the medical image, and extracts (specifies) the positions of the cells contained in each of the first cell group and the second cell group in the two-dimensional coordinate system. The spatial information extraction unit 102 extracts the spatial information of the cells contained in each of the first cell group and the second cell group by extracting the X coordinates and Y coordinates of the cells contained in each of the first cell group and the second cell group in the image in which the two types of cell groups are classified.

[0038] Once the spatial information of the cells contained in each of the first and second cell groups has been extracted, the graph creation unit 103 creates a graph based on the spatial information (S63). In this embodiment, the graph creation unit 103 creates a graph by assigning nodes to the positions of the cells contained in each of the first and second cell groups, which are specified by X and Y coordinates, and connecting each node with an edge. Furthermore, the graph creation unit 103 assigns a feature representing the characteristics of a cell (i.e., a feature of the cell corresponding to the node) to each node in the graph, and assigns a feature representing the relationship between the two cells corresponding to the two connected nodes (i.e., a feature between the two cells corresponding to the two nodes connected by each edge, and a feature of the interaction between the edges) to each edge. The graph creation procedure is the same as the procedure in S23 of FIG. 2 described in the first embodiment. Image 72 in FIG. 7 is an example of a graph in which nodes corresponding to the cells contained in each of the first and second cell groups in image 71 are connected with edges.

[0039] Next, the graph modeling unit 104 uses a GNN to model the spatial relationships between cells in the graph created by the graph creation unit 103, based on the feature quantities of the assigned nodes and edges (S64). The graph modeling unit 104 also extracts multiple subgraphs from the modeled graph (S64). The graph modeling and subgraph extraction procedures are similar to the procedure of S24 in FIG. 2 described in the first embodiment. Subgraph group 73-1 in FIG. 7 is an example of multiple extracted subgraphs, and subgraph group 73-1 includes three subgraphs, but may include more than three subgraphs.

[0040] When the processing up to this point is completed, the process returns to S61, where the classification unit 101 classifies the cells detected in the medical image into two types of cell groups (a first cell group and a second cell group) based on the cell phenotype (S61). Since the first cell type pair has already been classified into two types of cell groups, in the second processing of S61, the classification unit 101 classifies a second cell type pair, which is different from the first cell type pair, into two types of cell groups. Then, the spatial information extraction unit 102 extracts spatial information of the cells included in each of the two types of cell groups (S62), and the graph creation unit 103 creates a graph based on the spatial information (S63). Thereafter, the graph modeling unit 104 models the graph, constructs a graph for the GNN, and extracts multiple subgraphs from the constructed graph (S64). In this way, by repeating the processing from S61 to S64 multiple times (twice or more), subgraph groups are generated for the two types of cell groups for each of the multiple types of cell type pairs. The repeated processes may be performed in parallel and are not limited to a specific order for multiple cell type pairs. Subgraph groups 73-1 to 73-3 in FIG. 7 show multiple subgraphs generated by performing the processes from S61 to S64 three times. Subgraph group 73-1 includes multiple subgraphs generated for a first cell type pair. Similarly, subgraph group 73-2 includes multiple subgraphs generated for a second cell type pair, and subgraph group 73-3 includes multiple subgraphs generated for a third cell type pair.

[0041] Next, the training data generation unit 105 generates training data for the responder prediction model 111 (S65). The training data is composed of a plurality of subgraphs and labels indicating responders or non-responders as true labels. In this embodiment, the training data generation unit 105 generates training data including a plurality of subgraphs included in a subgraph group generated for each of a plurality of cell-type pairs and the true labels.

[0042] Once the training data is generated, the training unit 106 uses the training data to train the respondent prediction model 111 (S66). The respondent prediction model 111 is a GNN for learning the relationship between subgraphs and true labels (labels indicating responders or non-responders). FIG. 8 shows a conceptual diagram of the training procedure for the respondent prediction model 111 using subgraphs. In FIG. 8, each of the subgraphs included in the subgraph groups 73-1 to 73-3 is associated with a true label 81, and the training unit 106 inputs each of the subgraphs in the subgraph groups 73-1 to 73-3 into the respondent prediction model 111 and obtains an output (i.e., a predicted label 80).

[0043] The respondent prediction model 111 according to this embodiment includes, for example, a feature extractor 83, a pooling layer 84, and a classifier 85. The feature extractor 83 extracts feature vectors representing the features of an input subgraph from the subgraph. Because the subgraph groups 73-1 to 73-3 each have a different cell-type pair, the feature extractor 83 extracts feature vectors in different spaces (feature vector spaces) for each of the subgraph groups 73-1 to 73-3. The pooling layer 84 reduces the size of the feature vectors extracted in the different spaces. The classifier 85 is configured to classify the size-reduced feature vectors to output a predicted label 80.

[0044] When the predicted label 80 is output, the learning unit 106 trains the responder prediction model 111 using a loss function 82, which is a function that derives the magnitude of the difference (loss) between the true label 81 (i.e., the correct label) associated with the input subgraph and the predicted label 80. Specifically, the learning unit 106 trains the responder prediction model 111 so as to reduce the value of the loss derived from the loss function 82. When a subgraph is input, the trained responder prediction model 111 is configured to predict and output a responder or non-responder label for the input subgraph. Specifically, the trained responder prediction model 111 is configured to predict and output whether the medical image from which the graph from which the input subgraph was extracted was acquired from a responder or non-responder patient.

[0045] After training the responder prediction model 111, the prediction unit 107 uses the trained responder prediction model 111 to predict whether the target patient will be a responder or a non-responder based on medical images including the tumor microenvironment obtained from the target patient (S67). Specifically, the information processing device 10 performs the processes from S61 to S64 multiple times on the medical images obtained from the target patient to extract multiple subgraphs for multiple cell type pairs. The prediction unit 107 then inputs each of the multiple subgraphs extracted for the target patient into the trained responder prediction model 111 and predicts a responder or non-responder label for the input subgraph. The prediction unit 107 obtains multiple predicted labels by predicting a responder or non-responder label for all of the multiple subgraphs extracted for the target patient. The prediction unit 107 then aggregates the multiple predicted labels to ultimately predict (determine) whether the target patient will be a responder or a non-responder. For example, the prediction unit 107 aggregates the multiple predicted labels to ultimately determine a responder prediction score that indicates the likelihood that the target patient is a responder. The score is indicated by a value between 0 and 1, for example, and the closer the score is to 1, the higher the likelihood that the target patient is a responder.

[0046] FIG. 9 illustrates a conceptual diagram of a prediction procedure using the responder prediction model 111. In FIG. 9, subgraph groups 90-1 to 90-3 include subgraphs extracted by performing the processes S61 to S64 three times on medical images including tumor microenvironments obtained from a target patient. That is, subgraph group 90-1 includes multiple subgraphs generated for a first cell type pair, subgraph group 90-2 includes multiple subgraphs generated for a second cell type pair, and subgraph group 90-3 includes multiple subgraphs generated for a third cell type pair. The prediction unit 107 inputs each of the subgraphs included in the subgraph groups 90-1 to 90-3 into the trained responder prediction model 111, predicts the responder or non-responder label for the input subgraph, and obtains predicted labels 91. The prediction unit 107 obtains multiple predicted labels 91 by predicting the responder or non-responder label for all subgraphs included in the subgraph groups 90-1 to 90-3. Thereafter, the prediction unit 107 aggregates the plurality of predicted labels 91 (aggregation 92) to finally obtain a prediction result 93 as to whether the target patient is a responder or a non-responder.

[0047] The aggregation 92 of the multiple predicted labels 91 indicating responders or non-responders is a statistical aggregation, and includes, for example, taking a majority vote of the multiple predicted labels 91. That is, the prediction unit 107 may acquire the label that is most prevalent among the multiple predicted labels 91 as the final prediction result 93. Alternatively, the prediction unit 107 may assign 1 to the labels of responders and 0 to the labels of non-responders among the multiple predicted labels 91, and acquire the average of the total values ​​as the final prediction result 93. In this case, the prediction result 93 indicates the possibility of being a responder as a value between 0 and 1.

[0048] As described above, the information processing device 10 according to the second embodiment assigns nodes to cells based on spatial information of cells of multiple phenotypes detected and classified from medical images, connects the nodes with edges, and constructs a graph using the node and edge characteristics. This allows spatial interactions between multiple types of cells in the tumor microenvironment to be analyzed using information on cell positions and interactions. The information processing device 10 then trains a responder prediction model 111 using subgraphs extracted from the constructed graph and true labels indicating responders or non-responders. This allows the responder prediction model 111 to learn the relationship between subgraphs and responders or non-responders. The responder prediction model 111 can learn the relationship between subgraphs and responders or non-responders based on cellular characteristics and intercellular characteristics. Furthermore, the information processing device 10 uses the trained responder prediction model 111 to predict whether a patient is a responder or non-responder based on subgraphs extracted from medical images obtained from the patient, thereby expecting high prediction accuracy.

[0049] In the above two embodiments, the graph modeling unit 104 extracts multiple subgraphs from the constructed graph as graph-based information. However, information other than subgraphs may be extracted. For example, the graph modeling unit 104 may extract node features and edge features from the constructed graph. In this case, the training data generation unit 105 may train the respondent prediction model 111 using the node features, edge features, and true labels.

[0050] Furthermore, in the above two embodiments, the responder prediction model 111 is trained using true labels, which are labels indicating responders or non-responders. However, a learning model for machine learning may be trained using other true labels. That is, if patients can be distinguished into any first type patient and any second type patient based on medical images, a patient type prediction model for machine learning (corresponding to the responder prediction model 111 in this embodiment) may be trained using true labels indicating the first type patient and the second type patient. In this case, the trained learning model can be configured to predict and output whether the original medical image of the graph from which the input subgraph was extracted was acquired from a first type patient or a second type patient. Examples of first type patients and second type patients include patients with a low survival rate and a high survival rate for a given treatment (a use case related to prognosis) and cancerous patients and non-cancerous patients (a use case related to detection).

[0051] [Hardware configuration of information processing device] An example of the hardware configuration of an information processing device 10 capable of executing the above-described embodiment will be described. FIG. 10 is a block diagram showing an example of the hardware configuration of the information processing device 10 according to this embodiment. The information processing device 10 according to this embodiment can be implemented on a single or multiple computers, mobile devices, or any other processing platform. While FIG. 10 illustrates an example in which the information processing device 10 is implemented on a single computer, the information processing device 10 according to this embodiment may be implemented in a computer system including multiple computers. The multiple computers may be connected to each other so as to be able to communicate with each other via a wired or wireless network.

[0052] 10, the information processing device 10 may include a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, a RAM (Random Access Memory) 1003, an HDD (Hard Disk Drive) 1004, an input unit 1005, a display unit 1006, a communication I / F (communication unit) (interface) 1007, and a system bus 1008. The information processing device 10 may also include an external memory. The CPU 1001 comprehensively controls the operations of the information processing device 10, and controls each of the components (1002 to 1007) via the system bus 1008, which is a data transmission path.

[0053] The ROM 1002 is a non-volatile memory that stores control programs and the like necessary for the CPU 1001 to execute processing. The programs include instructions (codes) that cause the CPU 1001 to execute the processing according to the above-described embodiment. The programs may be stored in a non-volatile memory such as the HDD 1004 or an SSD (Solid State Drive), or in an external memory such as a removable storage medium (not shown). The RAM 1003 is a volatile memory that functions as the main memory, work area, and the like of the CPU 1001. That is, when executing processing, the CPU 1001 loads the necessary programs and the like from the ROM 1002 into the RAM 1003, and executes the programs and the like to realize various functional operations.

[0054] The HDD 1004 stores, for example, various data and information required when the CPU 1001 performs processing using a program. The HDD 1004 also stores, for example, various data and information obtained when the CPU 1001 performs processing using a program. The input unit 1005 is configured with a pointing device such as a keyboard and a mouse. The display unit 1006 is configured with a monitor such as a liquid crystal display (LCD). The display unit 1006 may function as a GUI (Graphical User Interface) when configured in combination with the input unit 1005.

[0055] The communication I / F 1007 is an interface that controls communication between the information processing device 10 and an external device. The communication I / F 1007 provides an interface with a network and executes communication with the external device via the network. Various data, parameters, and the like are transmitted and received between the information processing device 10 and the external device via the communication I / F 1007. In this embodiment, the communication I / F 1007 may execute communication via a wired LAN (Local Area Network) or a dedicated line that conforms to a communication standard such as Ethernet (registered trademark). However, the network that can be used in this embodiment is not limited to this and may be configured as a wireless network. This wireless network includes wireless PANs (Personal Area Networks) such as Bluetooth (registered trademark), ZigBee (registered trademark), and UWB (Ultra Wide Band). It also includes wireless LANs (Local Area Networks) such as Wi-Fi (Wireless Fidelity) (registered trademark) and wireless MANs (Metropolitan Area Networks) such as WiMAX (registered trademark). It also includes wireless WANs (Wide Area Networks) such as 4G and 5G. The network may be any network that connects devices to each other so that they can communicate with each other, and the communication standard, scale, and configuration are not limited to those described above.

[0056] Although specific embodiments have been described above, these embodiments are merely examples and are not intended to limit the scope of the present invention. The devices and methods described herein may be embodied in forms other than those described above. Furthermore, appropriate omissions, substitutions, and modifications may be made to the above-described embodiments without departing from the scope of the present invention. Such omissions, substitutions, and modifications are included within the scope of the claims and their equivalents, and belong to the technical scope of the present invention.

[0057] The disclosure of this embodiment includes the following configuration. [1] An information processing device having: a classification unit that classifies cells in an image of a tumor microenvironment obtained from a patient into multiple cell groups; a spatial information extraction unit that extracts the positions of cells included in the multiple cell groups in the image; a graph creation unit that creates a graph by assigning nodes to cells included in the multiple cell groups based on the extracted positions, connecting the nodes with edges, assigning features that represent cell characteristics to each node, and assigning features that represent the relationship between two cells corresponding to two nodes to each edge; and a graph modeling unit that models the spatial relationship between cells in the graph based on the features of the assigned nodes and edges using a graph neural network (GNN).

[0058] [2] The information processing device described in [1], further comprising: a learning unit that trains a patient type prediction model for machine learning based on the modeled graph, and the patient is classified as either a first type patient or a second type patient; and a prediction unit that uses the trained patient type prediction model to predict whether the target patient is the first type patient or the second type patient from an image of the tumor microenvironment obtained from the target patient.

[0059] [3] The information processing device described in [2], wherein the graph modeling unit extracts multiple subgraphs from the modeled graph, and the learning unit uses the multiple subgraphs and true labels indicating the first type of patient or the second type of patient to train the patient type prediction model on the relationship between the multiple subgraphs and the true labels.

[0060] [4] The information processing device described in [3], wherein the graph modeling unit extracts the multiple subgraphs from the modeled graph, provided that the subgraphs have at least (N+1) nodes (N is an integer greater than or equal to 2) and N hops (N consecutive edges).

[0061] [5] An information processing device according to any one of [2] to [4], wherein the first type of patient or the second type of patient is a responder or non-responder to a predetermined treatment, respectively.

[0062] [6] An information processing device according to any one of [1] to [5], wherein the features representing the characteristics of the cell include at least one of the cell type, the cell size, the characteristics of the biomarker expression level in the cell, the relationship with neighboring nodes, and the coordinates of the center of the node.

[0063] [7] The information processing device according to any one of [1] to [6], wherein the feature representing the relationship between the two cells includes at least one of an edge type and an edge distance.

[0064] [8] The information processing device according to any one of [1] to [7], wherein the plurality of cell groups includes a cell group of tumor cells and a cell group of immune cells.

[0065] [9] The information processing device according to any one of [1] to [7], wherein the plurality of cell groups includes cell groups classified based on cell phenotype. [Explanation of symbols]

[0066] 10: Information processing device, 101: Classification unit, 102: Spatial information extraction unit, 103: Graph creation unit, 104: Graph modeling unit, 105: Learning data generation unit, 106: Learning unit, 107: Prediction unit, 110: Learning model storage unit, 111: Respondent prediction model

Claims

1. a classification unit that classifies cells in an image of a tumor microenvironment obtained from a patient into multiple cell groups; a spatial information extraction unit that extracts positions of cells included in the plurality of cell groups in the image; a graph creation unit that creates a graph by assigning nodes to cells included in the plurality of cell groups based on the extracted positions, connecting the nodes with edges, assigning features that represent cell characteristics to each node, and assigning features that represent a relationship between two cells corresponding to two nodes to each edge; a graph modeling unit that models spatial relationships between cells in the graph based on the feature quantities of the assigned nodes and edges using a graph neural network (GNN); An information processing device having the above.

2. The patient is classified as either a first type patient or a second type patient; a learning unit that learns a patient type prediction model for machine learning based on the modeled graph; a prediction unit that predicts whether the target patient is the first type patient or the second type patient based on an image of the tumor microenvironment obtained from the target patient using the trained patient type prediction model; The information processing device according to claim 1 , further comprising:

3. the graph modeling unit extracts a plurality of subgraphs from the modeled graph; the learning unit uses the plurality of subgraphs and a true label indicating the first type of patient or the second type of patient to cause the patient type prediction model to learn a relationship between the plurality of subgraphs and the true label; The information processing device according to claim 2 .

4. the graph modeling unit extracts the plurality of subgraphs from the modeled graph, provided that the subgraphs have at least (N+1) nodes (N is an integer equal to or greater than 2) and N hops (N consecutive edges); The information processing device according to claim 3 .

5. the first type of patient or the second type of patient is a responder or non-responder to a given treatment, respectively; The information processing device according to claim 2 .

6. The information processing device according to claim 1 , wherein the feature representing the characteristics of the cell includes at least one of the type of cell, the size of the cell, characteristics of biomarker expression levels in the cell, a relationship with nodes corresponding to neighboring cells, and coordinates of the center of the node corresponding to the cell.

7. The information processing device according to claim 1 , wherein the feature representing the relationship between the two cells includes at least one of a type of connection between the cells and a distance between nodes corresponding to the two cells.

8. The information processing device according to claim 1 , wherein the plurality of cell groups includes a cell group of tumor cells and a cell group of immune cells.

9. The information processing device according to claim 1 , wherein the plurality of cell groups includes cell groups classified based on cell phenotypes.

10. An information processing method executed by an information processing device, classifying cells in an image of the tumor microenvironment obtained from the patient into multiple cell populations; extracting positions of cells included in the plurality of cell groups in the image; creating a graph by assigning nodes to cells included in the plurality of cell groups based on the extracted positions, connecting the nodes with edges, assigning features representing cell characteristics to each node, and assigning features representing a relationship between two cells corresponding to two nodes to each edge; modeling spatial relationships between cells in the graph based on the assigned node and edge features using a graph neural network (GNN); An information processing method, including:

11. An information processing program for causing a computer to execute information processing, the program including: a classification process for classifying cells in an image of the tumor microenvironment obtained from a patient into multiple cell groups; a spatial information extraction process for extracting positions of cells included in the plurality of cell groups in the image; a graph creation process for creating a graph by assigning nodes to cells included in the plurality of cell groups based on the extracted positions, connecting the nodes with edges, assigning features representing cell characteristics to each node, and assigning features representing the relationship between two cells corresponding to two nodes to each edge; and a modeling process for modeling spatial relationships between cells based on the feature quantities of the assigned nodes and edges using a graph neural network (GNN) in the graph. program.

Images