A primary tumor staging multi-instance learning method, system, device and medium in a pathological image based on a hierarchical graph

Abstract

A primary tumor staging multi-instance learning method, system, device and medium based on hierarchical graph in pathological images, the method comprising: constructing a structure perception hierarchical graph through data preprocessing, feature extraction and graph construction; by learning the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of the key mode of

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and computer vision technology, and relates to a multi-instance learning method, system, device and medium for primary tumor staging in pathological images based on hierarchical graphs. Background Technology

[0002] Primary tumor staging (pT stage) is a crucial indicator in cancer assessment, aiding in determining patient prognosis and guiding clinical treatment selection. In routine clinical examinations, pT staging primarily involves measuring tumor size and identifying tumor infiltration into normal tissues. While the former can be easily assessed using gross examination, the latter requires pathologists to conduct a thorough and time-consuming microscopic examination of pathological slides. Typically, pathologists first identify the layered structure of the tissue under low magnification, then examine the junctions between the tumor and other normal tissues under high magnification to locate the deepest infiltration areas. Furthermore, some small clusters of cancer cells often remain distant from the primary tumor, further complicating the localization and identification of key staging features.

[0003] With the rapid development of digital pathology, many pathological identification and judgment processes can be modeled as machine learning tasks processed by deep learning algorithms. For primary tumor staging, it can be viewed as a classification task of a full-view digital pathological slide, with the category determined by the infiltration relationship between the tumor and other normal tissues. One straightforward solution is to first segment different types of tissue and then model the spatial infiltration relationship between them. However, segmenting different types of tissue requires a large amount of pixel-level fine annotation. Such pixel-level annotations rely on the pathologist's specialized pathological knowledge and are often difficult to obtain.

[0004] Currently, multiple instance learning (MIM) is a popular weakly supervised learning paradigm that relies solely on slice-level labels. It has been widely applied to classification tasks involving full-view digital slides, such as grading and typing. A classic MIM system typically consists of three steps: first, the full-view digital pathology slide is cropped into a series of non-overlapping small image patches, also known as "patches"; each patch is considered an instance, and all instances from a slide form a bag; then, features are extracted from each instance in the bag and fed into a pooling layer to aggregate the instances into a feature representation of the entire bag; finally, a classifier classifies the bag features.

[0005] However, directly applying classic multi-instance systems to primary tumor staging has two shortcomings: 1) The most discriminative region for primary tumor staging is the boundary between the tumor and other normal tissues. However, these regions are sparse in large-scale, full-view digital pathology slides, making it difficult for existing multi-instance learning-based methods to effectively capture the most critical instability features for staging; 2) Primary tumor staging is mainly based on the infiltration relationship between tissues, requiring examination at progressively higher magnifications. However, existing multi-instance learning-based methods typically extract morphological features based on a single magnification, failing to capture cross-scale spatial relationships between tissues.

[0006] Patent application CN112488234A discloses an end-to-end histopathological image classification method based on attention pooling. The method includes: S1: dividing the histopathological image into slices of a specified size, removing slices with excessive background areas, and forming a package from the remaining slices.

[0007] S2: Using the packet obtained in S1 as input, train a deep neural network using the standard multiple instance learning method; S3: Use the trained deep neural network to score all slices, and take the m slices with the highest and lowest scores for each whole slide image, and combine them into a new packet; S4: Build a deep neural network with an attention module, and train the network using the new packet obtained in step S3; S5: After processing the histopathological images to be classified through S1 and S3, use the model obtained in S4 to classify them. This invention can achieve good classification results in the current situation where there are only a few samples, providing doctors with an auxiliary diagnostic mechanism and alleviating the problem of medical resource shortage; however, this invention does not consider that the patches that play a key role in the classification of primary tumor staging only account for a small part, and it is difficult to accurately locate these patches and capture the most critical example features for staging; at the same time, this method only extracts the features of pathological slices at a single fold, losing some necessary information, which is not conducive to capturing cross-scale spatial relationships between tissues. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, the present invention aims to provide a multi-example learning method, system, device, and medium for primary tumor staging in pathological images based on hierarchical graphs. This graph-based example tissue method constructs a structure-aware hierarchical graph to represent full-view pathological slices with a pyramid structure, effectively promoting the model's capture of key features related to primary tumor staging. Based on the structure-aware hierarchical graph, a graph representation network based on hierarchical attention is constructed to capture key patterns of primary tumor staging by learning cross-scale spatial features. This allows for accurate classification of pathological images and provides multi-granular interpretability by identifying the deepest penetration regions, assisting clinical pathology identification and judgment, and effectively improving the work efficiency of pathologists.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] A multi-instance learning method for primary tumor staging in pathological images based on hierarchical graphs, comprising the following steps:

[0011] Step 1: Construct a structure-aware hierarchical map. Represent the full-view digital pathology slide with a pyramid structure as a heterogeneous structure-aware hierarchical map, which includes two types of sub-maps: the tissue map at the top level and multiple patch maps at the lower level, which are built on low-magnification and relatively high-magnification slides, respectively. During the switching between different magnification views, the dependency relationship between patch map nodes and corresponding tissue map nodes at different magnifications is determined by the association matrix structure.

[0012] Step 2: Construct a graph representation network based on hierarchical attention: A patch graph convolutional neural network is used to encode the local contextual features of nodes in the high-magnification patch graph using the structure-aware hierarchical graph constructed in Step 1. A hierarchical attention layer is then used to fuse the high-magnification patch features into their corresponding low-magnification tissue graph nodes using a cross-attention mechanism. The fused tissue graph is input into the tissue graph convolutional neural network to encode global spatial tissue structure features and capture the invasive relationship between tumors and normal tissues. Finally, a global attention layer aggregates all nodes in the tissue graph to obtain slice-level feature representations for predicting staging categories.

[0013] The specific implementation steps for constructing the structure-aware hierarchy graph described in step one are as follows:

[0014] 1) Data preprocessing

[0015] 1.1) For a full-view digital pathology slide with a pyramid structure, the resolutions from low to high are represented as w t w 1 w 2 ,…,w n ,w n+1 ; will w t The resolution is downsampled to half, and the Otsu binarization algorithm is used to separate the tissue from the background to obtain a binary segmentation mask. Then, the segmentation mask is subjected to 2... 1 ,2 2 ,2 3 , ...2 n ,2 n+1 ,2 n+2 Multiple upsampling steps to match w respectively t w 1 w 2 ,…,w n ,w n+1 The resolution;

[0016] 1.2) Using the sliding window method, the slices of different multiples obtained in step 1.1) are trimmed into non-overlapping patches, generating patch sets P respectively. t P 1 P 2 ...P n ,P n+1 Guided by the segmentation mask, the background of the slice is ignored, and the patch only includes the foreground region containing the tissue.

[0017] 1.3) In w t The image processed in step 1.2) is then segmented into different tissue blocks using a simple linear iterative clustering algorithm.

[0018] 2) Feature extraction

[0019] Extract the patch set P obtained in step 1.2) of step one. t P 1 P 2 ...P n ,P n+1 Node characteristics; P t P 1 P 2 ...P n ,P n+1 All patches are concatenated together and input into the feature encoder to obtain the initial node features X of the n+1 patch maps. 1 X 2 , ...X n ,X n+1 For a tissue diagram, since each tissue block t i All covered patch images P t Several patches in the middle can make t i The patch overlay is input into the feature encoder, and then average pooling is performed to obtain the tissue block t. i The characteristics; ultimately, the node features X of the organizational diagram are obtained. t ;

[0020] 3) Construct the structure-aware hierarchical graph G

[0021] Based on the node features of the organization blocks and organization graph obtained in step 1, step 2), a structure-aware hierarchical graph G is constructed; graph G contains three types of edges, among which E t Used to model the adjacency relationships of adjacent tissue blocks in an organizational diagram. The adjacency relationships between patches and their eight nearest neighbors in each patch graph were modeled. The dependency relationships between nodes in the organization diagram and patch diagram were modeled; simultaneously, an adjacency matrix A was generated based on the edge connectivity. t Adjacency matrix and the correlation matrix In A t In the middle, if organization t i , t j If there are edges between them, then the elements in the matrix... Conversely, it is 0; the same applies to the other matrices. Finally, the structure-aware hierarchical graph G can be represented as...

[0022] Step two involves constructing a graph representation network based on hierarchical attention, and the specific steps are as follows:

[0023] 1) Update the features of patch graph nodes using a patch graph convolutional neural network:

[0024] After constructing the structure-aware hierarchy graph G in step one, for each patch graph... Each patch graph is input into its respective patch graph convolutional neural network. The structure of each patch graph is identical, and each patch graph convolutional neural network has L layers. Each layer updates the features of each node. For each node, the feature update process of the l-th layer (l∈[0,L]) is as follows:

[0025] 1.1) For node i, its current feature is Find all its neighbor nodes N(i). For each neighbor node j∈N(i), add its features to the features of node i, and then activate the added features through the ReLU function.

[0026] 1.2) All the added features are accumulated into a single fused feature through an attention mechanism;

[0027] 1.3) Combine the obtained fusion features with After performing residual connections, the data is input into a multilayer perceptron to obtain updated features. Complete the feature update of layer l;

[0028] After feature updates are completed in layer L, the outputs of all layers are concatenated to form the node features of the final patch graph. After passing through the patch graph convolutional neural network, the patch graph is updated to...

[0029] 2) Integrate patch graph node features into organization graph node features through a hierarchical attention layer:

[0030] Input the patch graph into the hierarchical attention layer, based on the association matrix. Find the patches contained in each organization in the organizational graph. Guided by organizational features, use a cross-attention mechanism to perform a weighted summation of the features of the patch nodes within the organization, generating detail-aware organizational features. The weights are the similarity between the patch and its corresponding tissue; the detail-aware features at two scales are concatenated to obtain the tissue graph feature that incorporates the fine-grained patch node features. The fused tissue graph is represented as G. f ;

[0031] 3) Update organizational graph node features using an organizational graph convolutional neural network:

[0032] G f The input to the organizational graph is a convolutional neural network, which encodes the global spatial relationships of the organizational graph nodes into the features of each organizational node, resulting in an updated organizational graph G'. f The design of the organization graph convolutional neural network is the same as that of the patch graph convolutional neural network, and the feature update process is the same as steps 1.1) to 1.3) in step two.

[0033] 4) The node features in the organization graph are fused into package-level features for staging through a global attention layer:

[0034] G' f The input is a global attention layer, which weights and fuses the features of each node based on its importance to the slice, resulting in a bag-level feature representation x. bag Then x bag The input is a fully connected layer, and after a softmax operation, the final primary tumor staging prediction result is obtained.

[0035] A multi-instance learning system for primary tumor staging in pathological images based on hierarchical graphs includes a structure-aware hierarchical graph construction module and a graph representation network module based on hierarchical attention.

[0036] The structure-aware hierarchical graph construction module is used to represent the full-view digital slice of the pyramid structure as a heterogeneous structure-aware hierarchical graph, including two types of subgraphs: a tissue graph at the top level and multiple patch graphs at the lower levels, built on slices at low magnification and relatively high magnification, respectively. Specifically, in the tissue graph, different tissues and their nearest neighbor relationships are represented as nodes and edges, respectively, clearly representing the spatial relationships between different tissues, which is beneficial for the network model to perceive the infiltration relationship between tumors and normal tissues. In the patch graph, patches are used as nodes, and the eight-neighbor relationships between patches are used as edges, realizing the modeling of fine-grained features at high magnification. In addition, during the switching process between different magnification views, the dependency relationship between patch graph nodes and corresponding tissue graph nodes at different magnifications is determined by the association matrix structure.

[0037] The hierarchical attention-based graph representation network module comprises four parts: a patch graph convolutional neural network, a hierarchical attention layer, a tissue graph convolutional neural network, and a global attention layer. The patch graph convolutional neural network encodes the local contextual features of nodes in the high-magnification patch graph. The hierarchical attention layer uses a cross-attention mechanism to fuse the high-magnification patch features into their corresponding low-magnification tissue graph nodes. The fused tissue graph is input into the tissue graph convolutional neural network to encode global spatial tissue structure features, capturing the invasive relationship between tumors and normal tissues. Finally, the global attention layer aggregates all nodes in the tissue graph to obtain slice-level feature representations for predicting staging categories.

[0038] A multi-instance learning device for primary tumor staging in pathological images based on hierarchical graphs, comprising:

[0039] Memory, used to store computer programs;

[0040] The processor is configured to implement the multi-instance learning method for primary tumor staging in histopathological images based on structure-aware hierarchical maps as described in steps one and two when executing the computer program.

[0041] A computer-readable storage medium storing a computer program that, when executed by a processor, enables a user to stage primary tumors in histopathological images based on a structure-aware hierarchy map using a multi-instance learning method.

[0042] Compared with the prior art, the present invention has the following beneficial technical effects:

[0043] 1. The multi-instance learning system proposed in this invention simulates the discrimination and judgment process of a pathologist. By constructing a structure-aware hierarchical graph and a graph representation network based on hierarchical attention, it can perform accurate, efficient and interpretable primary tumor staging discrimination tasks.

[0044] 2. This invention constructs a structure-aware hierarchical diagram to represent full-view pathological slices with a pyramid structure, effectively promoting the capture of key features related to the staging of primary tumors by a multi-instance learning model.

[0045] 3. Based on the structure-aware hierarchical graph, a graph representation network based on hierarchical attention is proposed. By learning cross-scale spatial features, it captures key patterns of primary tumor staging and can accurately classify pathological images.

[0046] 4. The proposed system can accurately classify pathological images and provide multi-granular interpretability by identifying the deepest infiltration areas, assisting clinical pathology identification and judgment, and effectively improving the work efficiency of pathologists. Attached Figure Description

[0047] Figure 1 This is a flowchart of the overall processing.

[0048] Figure 2 This is a flowchart illustrating the specific construction process of the structure-aware hierarchy map, where 5×, 10×, and 20× represent magnifications of 5x, 10x, and 20x for pathological slides, respectively, corresponding to w. t w 1 w 2 .

[0049] Figure 3 This is a structural diagram of a graph representation network based on hierarchical attention. Detailed Implementation

[0050] The present invention will now be described in further detail:

[0051] See Figure 1 A multi-instance learning method for primary tumor staging in pathological images based on hierarchical graphs, the specific steps of which include:

[0052] Step 1: Construct a structure-aware hierarchy diagram. Specific implementation steps are as follows: (See...) Figure 2

[0053] 1) Data preprocessing

[0054] 1.1) For a full-view digital pathology slide with a pyramid structure, consider three resolutions as an example. The resolutions from low to high are represented as w... t w 1 w 2 , will w t The resolution is downsampled to half, and the Otsu binarization algorithm is used to separate the tissue from the background to obtain a binary segmentation mask. Then, the segmentation mask is upsampled by 2, 4, and 8 times respectively to match w. t w 1 w 2 The resolution;

[0055] 1.2) Using the sliding window method, the three multiple slices obtained in step 1.1) are trimmed into non-overlapping patches, generating three patch sets P respectively. t P 1 P 2 Guided by the segmentation mask, the background of the slice is ignored, and the patch only includes the foreground region containing the tissue.

[0056] 1.3) In w t The image processed in step 1.2) is then segmented into different tissue blocks using a simple linear iterative clustering algorithm.

[0057] 2) Feature extraction

[0058] Extract the three patch sets P generated in step 1.2) of step one. t P 1 P 2 Node characteristics; P 1 P 2 All patches are concatenated together and input into the feature encoder to obtain the initial node features X of the two patch maps. 1 X 2 For a tissue diagram, since each tissue block t i All covered patch images P t Several patches in the middle can make t i The patch overlay is input into the feature encoder, and then average pooling is performed to obtain the tissue block t. i The characteristics; ultimately, the node features X of the organizational diagram are obtained. t ;

[0059] 3) Construct the structure-aware hierarchical graph G

[0060] Based on the node features of the organization blocks and organization graph obtained in step 1, step 2), a structure-aware hierarchical graph G is constructed; graph G contains three types of edges, among which E t Used to model the adjacency relationships of adjacent tissue blocks in an organizational diagram. The adjacency relationships between patches and their eight nearest neighbors in each patch graph were modeled. The dependency relationships between nodes in the organization diagram and patch diagram were modeled. Simultaneously, an adjacency matrix and an affinity matrix A were generated based on the edge connectivity. t , In A t In the middle, if organization t i , t j If there are edges between them, then the elements in the matrix... Conversely, it is 0; the same applies to the other matrices. Finally, the structure-aware hierarchical graph G can be represented as...

[0061] Step two involves constructing a graph representation network based on hierarchical attention. The specific steps are as follows: See [link / reference]. Figure 3

[0062] 1) Update the features of patch graph nodes using a patch graph convolutional neural network:

[0063] After constructing the structure-aware hierarchy graph G in step one, for the two patch graphs... The data are then fed into two convolutional neural networks (CNNs) for each patch graph. The two patch graphs have identical structures, each containing L layers. Each layer updates the features of each node. For each node, the feature update process of the l-th layer (l∈[0,L]) is as follows:

[0064] 1.1) For node i, its current feature is Find all its neighbor nodes N(i). For each neighbor node j∈N(i), add its features to the features of node i, and then activate the added features through the ReLU function.

[0065] 1.2) All the added features are accumulated into a single fused feature through an attention mechanism;

[0066] 1.3) Combine the obtained fusion features with After performing residual connections, the data is input into a multilayer perceptron to obtain updated features. Complete the feature update of layer l;

[0067] After feature updates are completed in layer L, the outputs of all layers are concatenated to form the node features of the final patch graph. After passing through the patch graph convolutional neural network, the patch graph is updated to...

[0068] 2) By using a hierarchical attention layer, the features of patches in the patch graph are fused into the features of nodes in the organization graph:

[0069] Input the patch graph into the hierarchical attention layer, based on the association matrix. Find the patches contained in each organization in the organizational graph. Guided by organizational features, use a cross-attention mechanism to perform a weighted summation of the features of the patch nodes within the organization, generating detail-aware organizational features. The weights are the similarity between the patch and its corresponding tissue. Finally, the detail-aware features at the two scales are concatenated to obtain the tissue graph feature that integrates the fine-grained patch node features. The fused tissue graph is represented as G. f ;

[0070] 3) Update organizational graph node features using an organizational graph convolutional neural network:

[0071] G f The input to the organizational graph is a convolutional neural network, which encodes the global spatial relationships of the organizational graph nodes into the features of each organizational node, resulting in an updated organizational graph G'. f The design of the organization graph convolutional neural network is the same as that of the patch graph convolutional neural network, and the feature update process is the same as steps 1.1) to 1.3) in step two.

[0072] 4) The node features in the organization graph are fused into package-level features for staging through a global attention layer:

[0073] G' f The input is a global attention layer, which weights and fuses the features of each node based on its importance to the slice, resulting in a bag-level feature representation x. bag Then x bag The input is a fully connected layer, and after a softmax operation, the final primary tumor staging prediction result is obtained.

[0074] The entire multi-instance learning system is trained iteratively by minimizing the cross-entropy loss function between the predicted result and the true label.

[0075] The present invention discloses a multi-instance learning system for primary tumor staging in pathological images based on hierarchical graphs, comprising two modules: a structure-aware hierarchical graph construction module and a graph representation network module based on hierarchical attention.

[0076] The structure-aware hierarchical graph construction module represents the full-view digital slices of the pyramid structure as a heterogeneous structure-aware hierarchical graph, including two types of subgraphs: a tissue graph at the top level and multiple patch graphs at lower levels, built on slices at low and relatively high magnification, respectively. Specifically, in the tissue graph, different tissues and their nearest neighbor relationships are represented as nodes and edges, respectively, which can clearly represent the spatial relationships between different tissues, which is beneficial for the network model to perceive the infiltration relationship between tumors and normal tissues. In the patch graph, patches are used as nodes, and the eight-neighbor relationships between patches are used as edges, realizing the modeling of fine-grained features at high magnification. In addition, in order to simulate the switching process between different magnification fields when pathologists distinguish, we propose an association matrix structure to determine the dependency relationship between patch graph nodes and corresponding tissue graph nodes at different magnifications.

[0077] The graph representation network module based on hierarchical attention consists of four parts: a patch graph convolutional neural network, a hierarchical attention layer, a tissue graph convolutional neural network, and a global attention layer. The patch graph convolutional neural network encodes the local contextual features of nodes in the high-magnification patch graph. The hierarchical attention layer uses a cross-attention mechanism to fuse the high-magnification patch features into their corresponding low-magnification tissue graph nodes. The fused tissue graph is input into the tissue graph convolutional neural network to encode global spatial tissue structure features, capturing the invasive relationship between tumors and normal tissues. Finally, the global attention layer aggregates all nodes in the tissue graph to obtain slice-level feature representations for stage category prediction.

Claims

1. A method for primary tumor staging multi-instance learning in pathological images based on hierarchical graph, characterized in that: The specific steps include: Step one, constructing a structure-aware hierarchical graph, representing the pyramid structure of the full field of view digital pathology slide as a heterogeneous structure-aware hierarchical graph, including two types of subgraphs: the tissue graph at the top layer and multiple patch graphs at the lower layer, which are respectively established on the low magnification and relatively high magnification slides; and in the switching process between different magnification fields of view, the dependent relationship between the patch graph nodes and the corresponding tissue graph nodes at different magnifications is determined through the association matrix structure; Step two, constructing a graph representation network based on hierarchical attention: using the patch graph convolutional neural network to encode the local context features of the nodes in the high magnification patch graph constructed in step one; using the hierarchical attention layer to fuse the high magnification patch features into the corresponding low magnification tissue graph nodes through the cross-attention mechanism; the fused tissue graph is input into the tissue graph convolutional neural network to encode the global spatial tissue structure features and capture the infiltration relationship between the tumor and the normal tissue; finally, the global attention layer aggregates all the nodes of the tissue graph to obtain the feature representation at the slide level, which is used for the prediction of the staging category; The structure-aware hierarchical graph constructed in step one has the following specific implementation steps: 1) Data preprocessing 1.1) For a pyramid-structured whole slide digital pathology slide, w t , w 1 , w 2 , …, w n , w n+1 are respectively represented from low resolution to high resolution; t Downsample the resolution of w 1 by half, segment the tissue from the background using Otsu's thresholding algorithm to obtain a binary segmentation mask, and then upsample the segmentation mask by 2 2 , 2 3 , …, 2 n , 2 n+1 , 2 n+2 times respectively to match the resolutions of w t , w 1 , w 2 , …, w n , w n+1 respectively. 1.2) The different magnification patches obtained in step one 1.1) are cropped into non-overlapping patches using a sliding window approach, and patch sets P t , P 1 , P 2 , … P n , P n+1 are generated respectively; under the guidance of the segmentation mask, the background of the patches is ignored, and the patches only contain the foreground region containing the tissue; 1.3) In w t segment the image processed in step one, 1.2), into different tissue blocks using a simple linear iterative clustering algorithm; 2) Feature extraction Extract the patch set P obtained in step 1.2) of step one. t P 1 P 2 ...P n ,P n+1 Node characteristics; P t P 1 P 2 ...P n ,P n +1 All patches are concatenated together and input into the feature encoder to obtain the initial node features X of the n+1 patch maps. 1 X 2 , ...X n ,X n+1 For a tissue diagram, since each tissue block t i All covered patch images P t Several patches in the middle can make t i The patch overlay is input into the feature encoder, and then average pooling is performed to obtain the tissue block t. i The characteristics; ultimately, the node features X of the organizational diagram are obtained. t ; 3) Constructing a structure-aware hierarchical graph G According to the node features of the tissue blocks and the tissue graph obtained in step one 2), a structure-aware hierarchical graph G is constructed; the graph G contains three types of edges, wherein E t for modeling the adjacency relationship of adjacent tissue blocks in the tissue graph, for modeling the adjacency relationship of the patches and their eight nearest neighbors in each patch graph, for modeling the subordination relationship of the nodes in the tissue graph and the patch graph; meanwhile, an adjacency matrix A is generated according to the connection relationship of the edges t , the adjacency matrix and the association matrix In A t , if there is an edge between the tissues t i , t j , the element in the matrix is otherwise 0, and the rest of the matrix is the same; finally, the structure-aware hierarchical graph G can be represented as 2. The primary tumor staging multi-instance learning method based on hierarchical graph in pathological images according to claim 1, characterized in that: The specific steps of constructing a graph representation network based on hierarchical attention in step two are as follows: 1) Update the patch graph node features through the patch graph convolutional neural network: After the construction of the structure-aware hierarchical graph G in step one, for each patch graph The two are input into the respective patch graph convolutional neural network; each patch graph has the same structure, and each patch graph convolutional neural network has L layers, and each layer updates the features of each node. For each node, the feature update process of the lth layer (l ∈ [0, L]) is as follows: 1.1) for node i, its current features are find all its neighbor nodes N(i), for each neighbor node j ∈ N(i), add its features to the features of node i, and then activate the added features by a ReLU function; 1.2) Add all the features through the attention mechanism to form a fused feature; 1.3) The resulting fusion feature is input into a multi-layer perceptron after residual connection to obtain the updated feature The first layer feature update is completed.​ After the feature update of all L layers is completed, the outputs of all layers are spliced together as the node features of the final patch graph; after the patch graph convolutional neural network, the patch graph is updated to 2) Fuse the patch graph node features into the tissue graph node features through the hierarchical attention layer: The patch graph is input into a hierarchical attention layer, and a correlation matrix is generated according to the hierarchical structure of the patch graph The patch contained in each organization in the organization graph is found, and under the guidance of the organization feature, the patch node features contained in the organization are weighted and summed by using a cross-attention mechanism to generate a detail-aware organization feature wherein the weight is the similarity between the patch and the corresponding organization; the detail-aware features of the two scales are spliced to obtain an organization graph feature fused with fine-grained patch node features, and the fused organization graph is represented as G f ; 3) Update the tissue graph node features through the tissue graph convolutional neural network: G f The input organization graph convolutional neural network encodes the global spatial relationship of the organization graph nodes into each organization node feature to obtain an updated organization graph G' f The organization graph convolutional neural network and the patch graph convolutional neural network are designed identically, and the feature updating process is the same as steps 1.1) to 1.3) in step two. 4) Fuse the node features in the tissue graph into the package-level features for staging through the global attention layer: G' f The global attention layer is input, and the features of the organization nodes are weighted and fused according to the importance of the slices to obtain a package-level feature representation x bag ; x bag The fully connected layer is input, and the final primary tumor staging prediction result is obtained after the softmax operation.

3. A hierarchical graph-based primary tumor staging multi-instance learning system in pathological images, characterized in that: The method for realizing any one of claims 1 to 2; comprising a structure-aware hierarchical graph construction module and a graph representation network module based on hierarchical attention: The structure-aware hierarchical graph construction module is used to represent the pyramid structure of the full field of view digital slide as a heterogeneous structure-aware hierarchical graph, including two types of subgraphs: the tissue graph at the top layer and multiple patch graphs at the lower layer, which are respectively established on the low magnification and relatively high magnification slides; specifically, in the tissue graph, different tissues and their nearest neighbor relationships are represented as nodes and edges, respectively, clearly representing the spatial relationship between different tissues, which is conducive to the network model to perceive the infiltration relationship between the tumor and the normal tissue; in the patch graph, the patch is taken as the node, and the eight-neighborhood relationship between the patches is taken as the edge, realizing the modeling of fine-grained features at high magnification; In addition, in the switching process between different magnification fields of view, the dependent relationship between the patch graph nodes and the corresponding tissue graph nodes at different magnifications is determined through the association matrix structure; The graph representation network module based on hierarchical attention includes four parts: patch graph convolutional neural network, hierarchical attention layer, tissue graph convolutional neural network, and global attention layer; among them, the patch graph convolutional neural network is used to encode the local context features of the nodes in the high magnification patch graph; The hierarchical attention layer fuses the high-magnification patch features into the corresponding low-magnification tissue graph nodes by using a cross-attention mechanism; the fused tissue graph is input into a tissue graph convolutional neural network to encode global spatial tissue structure features and capture the infiltration relationship between the tumor and normal tissue; finally, the global attention layer aggregates all the nodes of the tissue graph to obtain a slice-level feature representation for predicting the staging category.

4. A hierarchical graph-based primary tumor staging multi-instance learning apparatus in pathological images, characterized by: The computer program is executed by the processor to implement the hierarchical graph-based primary tumor staging multi-instance learning method in the pathological image according to any one of claims 1 to 2. When the computer program is executed by the processor, a user can perform primary tumor staging on a histopathological image based on a structure-aware hierarchical graph by using the hierarchical graph-based primary tumor staging multi-instance learning method according to any one of claims 1 to 2. ​ 5. A computer readable storage medium storing a computer program, characterized in that: ​

Images