Cancer survival prediction method and system based on pathological image

Abstract

The invention provides a cancer survival prediction method based on a pathological image, and relates to the technical field of artificial intelligence and medical image analysis. The method comprises the following steps: firstly, dividing an acquired pathological image to obtain a plurality of image blocks; and obtaining the feature vector of each image block and identifying the tissue type to which each image block belongs. Calculating spatial proximity, feature similarity and tissue type compatibility between the image blocks according to the center coordinates of the image blocks, the feature vectors and the tissue type to which each image block belongs; and based on a calculation result, constructing a dynamic heterogeneous graph, and performing feature extraction to obtain a multi-scale feature graph of the dynamic heterogeneous graph. And performing multi-prototype learning on the basis of the belonging organization type of the image block and a cross-category attention mechanism to generate multiple prototypes. And finally, the multi-scale feature map and the multiple prototypes are aggregated to obtain final image representation, and cancer survival prediction is performed based on the final image representation, so that the prediction accuracy of the cancer survival state is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and medical image analysis, and more specifically, relates to a cancer survival prediction method and system based on pathological images. Background Art

[0002] With the rapid development of digital pathology and artificial intelligence technologies, whole-slide image (WSI)-based tissue regional modeling and survival prediction have become important areas for assisted tumor diagnosis. Existing research largely relies on expert pathologists to meticulously label image blocks to identify tissue types or functional regions, then construct graph neural networks (GNNs) or deep learning models for survival prediction. However, in real-world clinical practice, pixel-level or region-level labeling is not only time-consuming and labor-intensive, but also subject to significant subjectivity and consistency issues, severely limiting the scalability and generalization capabilities of the models.

[0003] Existing graph neural network-based survival prediction methods typically employ fixed strategies based on spatial proximity or feature similarity during the mapping phase. This strategy ignores the adaptability of graph structure to the task objective and makes it difficult to depict the complex underlying tissue structural relationships within the image. Furthermore, traditional graph models often lack the ability to model tissue semantic categories, cannot effectively distinguish structural differences between different tissue types, and struggle to incorporate category-level prior information to enhance representational capabilities. Some methods attempt to introduce prototype representations to assist modeling, but these methods often rely on manual label supervision and struggle to handle tissue regions with fuzzy boundaries or unknown subtypes.

[0004] In summary, existing survival prediction methods are disconnected from task objectives due to their fixed graph construction strategies and lack the ability to adaptively model tissue semantic categories and fuzzy boundaries. They are unable to effectively characterize the potential complex tissue structure relationships in pathological images and integrate prior information, resulting in insufficient accuracy in cancer survival prediction. Summary of the Invention

[0005] In order to solve the problem of insufficient prediction accuracy in the above-mentioned existing technologies when predicting cancer survival, the present invention provides a cancer survival prediction method and system based on pathological images. Under weakly supervised or unsupervised conditions, it dynamically constructs a graph structure to characterize the complex tissue interactions in the tumor microenvironment, realizes the generation and aggregation of pathological image structural views and prototype semantic views, improves the robustness and interpretability of the global image representation, and improves the prediction accuracy of cancer survival status.

[0006] In order to achieve the above technical effects, the technical solution of the present invention is as follows S1: Divide the acquired pathological image to obtain a plurality of image blocks; obtain a feature vector of each image block and identify the tissue type to which each image block belongs; S2: calculating spatial proximity, feature similarity, and tissue type compatibility between image blocks based on the center coordinates of the image blocks, the feature vectors, and the tissue type to which each image block belongs; and constructing a dynamic heterogeneous graph based on the calculation results; S3: extracting features from the dynamic heterogeneous graph to obtain a multi-scale feature graph of the dynamic heterogeneous graph; S4: Based on the tissue type of the image patch and the cross-category attention mechanism, multi-prototype learning is performed to generate multiple prototypes; S5: Aggregate multi-scale feature maps and multiple prototypes to obtain the final image representation, and perform cancer survival prediction based on the final image representation.

[0007] Furthermore, the method for identifying the tissue type to which each image block belongs includes: using a pre-trained tissue classifier to predict the tissue type of each image block to obtain the tissue type to which each image block belongs; The method also includes: performing unsupervised clustering on each image block, automatically dividing the image block into several tissue types, and obtaining the tissue type to which each image block belongs.

[0008] Furthermore, the process of calculating the spatial proximity, feature similarity, and tissue type compatibility between image blocks based on the center coordinates of the image blocks, the feature vectors, and the tissue type to which each image block belongs is as follows: Calculate the spatial proximity between image blocks, the expression is:

[0009] Where, represents the spatial proximity between the i-th image block and the j-th image block, represents the coordinates of the center point of the i-th image block, represents the coordinates of the center point of the j-th image block, . represents the Euclidean distance; According to the feature vectors of the image blocks, the feature similarity between the image blocks is calculated, and the expression is:

[0010] Where, represents the feature similarity between the i-th image block and the j-th image block, represents the feature vector of the i-th image block, represents the feature vector of the jth image block, Represents the transpose of the feature vector of the i-th image block; According to the tissue type to which each image block belongs, the tissue compatibility between the image blocks is calculated, and the expression is: g ( , ), where Indicates the tissue type to which the i-th image block belongs, Indicates the tissue type to which the jth image block belongs, g () represents the tissue compatibility function.

[0011] Furthermore, a dynamic edge determination mechanism is established based on the spatial proximity, feature similarity, and tissue type compatibility between image blocks, satisfying the expression:

[0012] Where, The formula for determining whether a node is connected is: represents the weight coefficient of the corresponding item, represents the threshold value,

[0013] Based on the dynamic edge judgment mechanism, a dynamic heterogeneous graph is constructed to represent the connection relationship between all image blocks in the pathological image; the dynamic heterogeneous graph is expressed as: G=(V,E), where V represents the node set, which is the set of the image blocks; E represents the edge set, which is the connection relationship between the image blocks.

[0014] Based on the above technical means, by jointly considering the spatial proximity, feature similarity and tissue type compatibility between image blocks, the edge connection relationship of the heterogeneous graph is dynamically determined, and adaptive heterogeneous graph construction is achieved, laying the foundation for subsequent cancer survival prediction.

[0015] Furthermore, a multi-layer graph convolutional neural network is used to extract features of the dynamic heterogeneous graph, and the expression of the multi-scale feature map of the dynamic heterogeneous graph is obtained as follows:

[0016] = Concat [ , ,…, ] H = MLP ( ) Where, represents the output of the lth layer of the multi-layer graph convolutional neural network, represents a multi-layer graph convolutional neural network, represents the output of the l-1th layer of the multi-layer graph convolutional neural network, A represents the adjacency matrix of the dynamic heterogeneous graph, represents the output of the 0th layer of the multi-layer graph convolutional neural network, X represents the set of feature vectors of all image blocks, Represents the output concatenation of each layer of the multi-layer graph convolutional neural network, Concat Represents a splicing operation, represents the output of the Lth layer of the multi-layer graph convolutional neural network, H represents a multi-scale feature map, MLP Represents a multilayer perceptron.

[0017] Furthermore, based on the tissue type of the image block and the cross-category attention mechanism, multi-prototype learning is performed. The process of generating multiple prototypes is as follows: Calculate the central prototype vector of each tissue type, the expression is:

[0018] Where, represents the central prototype vector, represents the set of image blocks belonging to the cth tissue type, Represents the feature vector of the i-th image block; Gaussian perturbation is introduced for multi-prototype learning to generate K prototypes for each type of tissue, expressed as:

[0019] Where, represents the K-th prototype of the c-th tissue type, represents Gaussian perturbation, represents the multivariate normal distribution, represents the covariance matrix, represents the identity matrix; The K prototypes of each tissue type are spliced ​​separately to generate the prototype of each tissue type. The expression is: =concat{ , ,..., } Where, Represents the prototype of the c-th tissue type, concat Represents a splicing operation; Archetypes based on each type of organizational archetype , using the cross-category attention mechanism to update each type of organizational prototype, the expression of the cross-category attention mechanism is defined as:

[0020] Where, represents the attention mechanism, Q represents the query vector, K represents the key vector, V represents a value vector, represents the transpose of the key vector, softmax represents the normalized exponential function, d Represents the key vector K Dimensions; Take the tissue type prototype as the query vector Q , the image block is used as the key vector K Sum value vector V , update each type of organization prototype, the expression is:

[0021] Where, Represents the prototype of the updated c-th tissue type, represents the learnable linear transformation matrix, represents the learnable linear transformation matrix, represents the prototype of the cth tissue type, represents a learnable linear transformation matrix; The prototype of each type of organizational prototype will be updated Splicing to form the initial multiple prototypes P.

[0022] Furthermore, the process of aggregating multi-scale feature maps and multiple prototypes to obtain the final image representation is as follows: The attention mechanism is used to aggregate multi-scale feature maps and multiple prototypes to obtain the aggregated image representation, which is expressed as:

[0023] Where, represents the image representation after aggregation, represents the attention mechanism, Represents multiple prototypes, represents a multi-scale feature map, represents a learnable linear transformation matrix, represents the learnable linear transformation matrix, represents a learnable linear transformation matrix; Based on the average pooling operation, the aggregated image representation is averagely pooled into the final image representation, which is expressed as: ） Where, represents the final image representation, Represents an average pooling operation.

[0024] According to the above technical means, the dynamic heterogeneous graph and the prototype are fused to obtain an overall image representation with dual structural and semantic perception capabilities.

[0025] Furthermore, the method further includes: using a joint loss function to train the process of cancer survival prediction; the joint loss function includes: a survival prediction loss function, a prototype compatibility loss function, and a prototype orthogonality loss function; The expression of the survival prediction loss function is:

[0026] Where, represents the survival prediction loss function, D represents the sample set of events, represents the risk set, represents the risk score of the a-th sample, represents the indexed risk score of the bth sample; The expression of the prototype compatibility loss function is:

[0027] Where, represents the prototype compatibility loss function, represents the similarity score between the nth image block and the cth tissue type, represents the similarity score between the nth image block and tissue types other than the cth class, represents the number of tissue types, N Indicates the number of image blocks, c Indicates the c-th tissue type, Indicates tissue types other than type c; The expression of the prototype orthogonality loss function is:

[0028] Where, represents the prototype orthogonality loss function, represents the prototype of the c-th tissue type, |⋅|F represents the Frobenius norm, represents the identity matrix, represents the transposition of the prototype of the cth tissue type; The expression of the joint loss function is:

[0029] Where, represents the joint loss function, represents an adjustable hyperparameter, represents an adjustable hyperparameter, represents a tunable hyperparameter.

[0030] Furthermore, the calculation expression of the similarity score is:

[0031] Where, represents the average similarity of the nth sample to the cth prototype, K Represents a set of multiple prototype indices of category c, represents the K-th prototype of the c-th tissue type, f( ) represents a multilayer perceptron, MEAN represents the average pooling operation, γ( , ) Represents the similarity function.

[0032] The present invention also provides a cancer survival prediction system based on pathological images, comprising: A feature extraction module is used to divide the acquired pathological image into a plurality of image blocks; obtain a feature vector of each image block and identify the tissue type to which each image block belongs; a heterogeneous graph construction module, configured to calculate spatial proximity, feature similarity, and tissue type compatibility between image blocks based on the center coordinates of the image blocks, the feature vectors, and the tissue type to which each image block belongs; and construct a dynamic heterogeneous graph based on the calculation results; A multi-scale feature map extraction module is used to extract features from the dynamic heterogeneous map to obtain a multi-scale feature map of the dynamic heterogeneous map; The multi-prototype generation module is used to learn and generate multiple prototypes based on the tissue type of the image patch and the cross-category attention mechanism; The prediction module is used to aggregate multi-scale feature maps and multiple prototypes to obtain the final image representation, and cancer survival prediction is performed based on the final image representation.

[0033] Compared with the prior art, the beneficial effects of this method are: The present invention provides a method and system for predicting cancer survival based on pathological images. First, the acquired pathological image is divided to obtain several image blocks, and the feature vector of each image block is obtained and the tissue type to which it belongs is identified. Then, based on the center coordinates, feature vectors and tissue types of the image blocks, the spatial proximity, feature similarity and tissue type compatibility are calculated to construct a dynamic heterogeneous graph. Feature extraction is performed on the dynamic heterogeneous graph to obtain a multi-scale feature map. Then, based on the tissue type to which the image blocks belong and the cross-category attention mechanism, multi-prototype learning is performed to generate multiple prototypes. Finally, the multi-scale feature map and multiple prototypes are aggregated to obtain the final image representation and realize cancer survival prediction. This method integrates the spatial position, feature information and tissue type relationship of the image blocks, uses the constructed dynamic heterogeneous graph to perform multi-scale feature extraction, and combines prototype learning with the cross-category attention mechanism to obtain a final image representation with dual perception capabilities of dynamic heterogeneous graph structure and prototype semantics, effectively improving the comprehensive representation capability of pathological image features, thereby improving the accuracy of cancer survival prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart showing a method for predicting cancer survival based on pathological images according to an embodiment of the present invention; Figure 2 A schematic diagram showing a dynamic heterogeneous graph proposed in an embodiment of the present invention; Figure 3 A schematic diagram showing the generation of multiple prototypes proposed in an embodiment of the present invention; Figure 4 A schematic diagram showing the aggregated multi-scale feature map and multiple prototypes proposed in an embodiment of the present invention; Figure 5 The figure shows the structure of the cancer survival prediction system based on pathological images proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent; In order to better illustrate this embodiment, some parts of the drawings may be omitted, enlarged, or reduced, and do not represent the actual size; It is understandable to those skilled in the art that descriptions of certain well-known contents may be omitted in the drawings.

[0036] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0037] The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limiting this patent; Example 1 This embodiment proposes a cancer survival prediction method based on pathological images, such as Figure 1The method proposed in this embodiment generally includes the following steps: S1: Divide the acquired pathological image to obtain a plurality of image blocks; obtain a feature vector of each image block and identify the tissue type to which each image block belongs; S2: calculating spatial proximity, feature similarity, and tissue type compatibility between image blocks based on the center coordinates of the image blocks, the feature vectors, and the tissue type to which each image block belongs; and constructing a dynamic heterogeneous graph based on the calculation results; S3: extracting features from the dynamic heterogeneous graph to obtain a multi-scale feature graph of the dynamic heterogeneous graph; S4: Based on the tissue type of the image patch and the cross-category attention mechanism, multi-prototype learning is performed to generate multiple prototypes; S5: Aggregate multi-scale feature maps and multiple prototypes to obtain the final image representation, and perform cancer survival prediction based on the final image representation.

[0038] In this embodiment, a whole slide image (WSI) of a cancer patient is obtained, and the image is divided into a number of fixed-size image blocks using a sliding window method. Each image block is feature-encoded to obtain a feature vector of each image block. ∈ .

[0039] Use a CNN network (such as ResNet-50) to perform feature encoding on each image block and obtain the feature vector of each image block. The expression is:

[0040] Where, represents the feature vector of the i-th image block, express CNN network, represents the i-th image block, Indicates belong d dimensional real space, d Represents the dimension of the feature vector.

[0041] In this embodiment, the method for identifying the tissue type to which each image block belongs includes: using a pre-trained tissue classifier to predict the tissue type of each image block to obtain the tissue type to which each image block belongs.

[0042] For example, a convolutional neural network (CNN) from deep learning, such as a ResNet, DenseNet, or U-Net network, is used as a tissue classifier. This classifier is trained using a dataset containing annotated pathological image samples of various tissue types to learn the morphological characteristics of each tissue type. Finally, the tissue type is predicted for each image block based on the pre-trained tissue classifier.

[0043] The method for identifying the tissue type to which each image block belongs further includes: performing unsupervised clustering on each image block, automatically dividing the image block into several tissue types, and obtaining the tissue type to which each image block belongs.

[0044] Exemplarily, K-means or a spectral clustering algorithm based on feature embedding is used to perform unsupervised clustering on each image block and automatically classify it into C tissue types.

[0045] In this embodiment, the expression for performing unsupervised clustering on each image block is:

[0046] Where, represents the feature vector of all image blocks, C represents the number of preset cluster centers (i.e., the number of tissue categories), Indicates the tissue type label to which image block i belongs.

[0047] The process of calculating the spatial proximity, feature similarity and tissue type compatibility between image blocks according to the center coordinates of the image blocks, the feature vectors and the tissue type to which each image block belongs is as follows: Calculate the spatial proximity between image blocks, the expression is:

[0048] Where, represents the spatial proximity between the i-th image block and the j-th image block, represents the coordinates of the center point of the i-th image block, represents the coordinates of the center point of the j-th image block, . represents the Euclidean distance; According to the feature vectors of the image blocks, the feature similarity between the image blocks is calculated, and the expression is:

[0049] Where, represents the feature similarity between the i-th image block and the j-th image block, represents the feature vector of the i-th image block, represents the feature vector of the jth image block, Represents the transpose of the feature vector of the i-th image block; According to the tissue type to which each image block belongs, the tissue compatibility between the image blocks is calculated, and the expression is: , where Indicates the tissue type to which the i-th image block belongs, Indicates the tissue type to which the j-th image block belongs, g () represents the tissue compatibility function.

[0050] The tissue compatibility function is used to measure the biological plausibility between tissue pairs. and For example, if the cancer tissue is next to the stromal tissue, .like and If there is no tissue incompatibility, such as no direct interaction between the gland and the necrotic area, .

[0051] For example, when there are C tissue types, a compatibility matrix G is constructed for calculation, and the expression is:

[0052] According to the compatibility between tissues shown in Table 1, the compatibility matrix G is constructed, and the tissue compatibility between image blocks is calculated using the tissue compatibility function.

[0053] Table 1

[0054] According to the spatial proximity, feature similarity and tissue type compatibility between image blocks, a dynamic edge determination mechanism is established to satisfy the expression:

[0055] Where, The formula for determining whether a node is connected is: represents the weight coefficient of the corresponding item, represents the threshold value,

[0056] Based on the dynamic edge judgment mechanism, a dynamic heterogeneous graph is constructed to represent the connection relationship between all image blocks in the pathological image, such as Figure 2 Schematic diagram shown; the dynamic heterogeneous graph is expressed as: G=(V,E), where V represents a node set, which is a set of image blocks; E represents an edge set, which is a connection relationship between the image blocks.

[0057] For example, a pathological image is divided into 256×256 pixel patches. Each patch is a node, and the connection between patches represents an edge (for example, the lymph nodes next to a tumor will suppress the tumor, so there will be an edge between them, and this edge has a corresponding weight). A dynamic heterogeneous graph is constructed to represent the pathological image.

[0058] The multi-layer graph convolutional neural network is used to extract features of the dynamic heterogeneous graph, and the expression of the multi-scale feature map of the dynamic heterogeneous graph is obtained as follows:

[0059] = Concat [ , ,…, ] H = MLP ( ) Where, represents the output of the lth layer of the multi-layer graph convolutional neural network, represents a multi-layer graph convolutional neural network, represents the output of the l-1th layer of the multi-layer graph convolutional neural network, A represents the adjacency matrix of the dynamic heterogeneous graph, represents the output of the 0th layer of the multi-layer graph convolutional neural network, X represents the set of feature vectors of all image blocks, Represents the output concatenation of each layer of the multi-layer graph convolutional neural network, Concat Represents a splicing operation, represents the output of the Lth layer of the multi-layer graph convolutional neural network, H represents a multi-scale feature map, MLP Represents a multilayer perceptron.

[0060] The first layer of a multi-layer graph convolutional neural network (GCN) takes as input the adjacency matrix of the dynamic heterogeneous graph and the set of feature vectors of all image patches, and outputs updated node features. The remaining layers take as input the adjacency matrix of the dynamic heterogeneous graph and the updated node features from the previous output. The outputs of each layer are concatenated as the final node representation, which is then converted into a multi-scale feature map of the required dimensions using a multi-layer perceptron (MLP).

[0061] In this embodiment, if Figure 3 As shown in the schematic diagram, based on the tissue type of the image block and the cross-category attention mechanism, multi-prototype learning is performed to generate multiple prototypes: Calculate the central prototype vector of each tissue type, the expression is:

[0062] Where, represents the central prototype vector, represents the set of image blocks belonging to the cth tissue type, Represents the feature vector of the i-th image block; Gaussian perturbation is introduced for multi-prototype learning to generate K prototypes for each type of tissue, expressed as:

[0063] Where, represents the K-th prototype of the c-th tissue type, represents Gaussian perturbation, represents the multivariate normal distribution, represents the covariance matrix, indicating that the perturbations are independent and identically distributed in each dimension. Represents the identity matrix, indicating that the perturbations have no correlation between the dimensions and are independent of each other. It indicates that it obeys the normal distribution, 0 indicates the mean vector of the normal distribution, which means that the mean is 0 in all dimensions and the disturbance is not biased in any direction as a whole. Represents the variance control parameter of Gaussian noise, which determines the strength of the disturbance. The larger it is, the greater the disturbance amplitude is.

[0064] The K prototypes of each tissue type are spliced ​​separately to generate the prototype of each tissue type. The expression is: =concat{ , ,..., } Where, represents the prototype of the cth tissue type, concat Represents a splicing operation; Archetypes based on each type of organizational archetype , using the cross-category attention mechanism to update each type of organizational prototype, the expression of the cross-category attention mechanism is defined as:

[0065] Where, represents the attention mechanism, Q represents the query vector, K represents the key vector, V represents a value vector, represents the transpose of the key vector, softmax represents the normalized exponential function, d Represents the key vector K Dimensions; In this embodiment, the feature dimension size is set to prevent the inner product values ​​in the query vector, key vector and value vector from being too large, causing the softmax function output to be close to one-hot, thereby suppressing gradient propagation.

[0066] Take the tissue type prototype as the query vector Q , the image block is used as the key vector K Sum vector V , update each type of organization prototype, the expression is:

[0067] Where, Represents the prototype of the updated c-th tissue type, represents a learnable linear transformation matrix, represents a learnable linear transformation matrix, Represents the prototype of the c-th tissue type, Represents a learnable linear transformation matrix. The learnable linear transformation matrix transforms the prototype and image patch nodes so that they are mapped to a unified attention space. Acting on the prototype, and The feature vector X applied to all image patches.

[0068] The prototype of each type of organizational prototype will be updated Splicing to form the initial multiple prototypes P.

[0069] In this embodiment, if Figure 4 As shown in the schematic diagram, the process of aggregating multi-scale feature maps and multiple prototypes to obtain the final image representation is: The attention mechanism is used to aggregate multi-scale feature maps and multiple prototypes to obtain the aggregated image representation, which is expressed as:

[0070] Where, represents the image representation after aggregation, represents the attention mechanism, Represents multiple prototypes, represents a multi-scale feature map, represents a learnable linear transformation matrix, represents a learnable linear transformation matrix, represents a learnable linear transformation matrix; Based on the average pooling operation, the aggregated image representation is averagely pooled into the final image representation, which is expressed as: ） Where, represents the final image representation, Represents an average pooling operation.

[0071] This method can achieve deep modeling of the structure and semantics of pathological images under weak or unsupervised conditions, and has good practical application potential and clinical promotion value.

[0072] Example 2 This embodiment proposes a cancer survival prediction method based on pathological images, which, based on Example 1, also includes: using a joint loss function to train the cancer survival prediction process; the joint loss function includes: a survival prediction loss function, a prototype compatibility loss function, and a prototype orthogonality loss function.

[0073] A survival prediction loss function is constructed to evaluate the difference between the actual cancer survival results and the cancer survival prediction results of this method. The expression of the survival prediction loss function is:

[0074] Where, represents the survival prediction loss function, D represents the sample set of events, represents the risk set, represents the risk score of the a-th sample, represents the exponential risk score of the b-th sample.

[0075] In this embodiment, based on the final image representation, a linear transformation is used to predict the patient risk score, which is expressed as:

[0076] Where, represents the risk score, represents the transpose of the training parameters.

[0077] Aggregate multi-scale feature maps and multi-prototype stages, introduce prototype compatibility loss, compare the loss of learning form, and optimize nodes to be close to similar prototypes and away from different ones. The expression of the prototype compatibility loss function is:

[0078] Where, represents the prototype compatibility loss function, represents the similarity score between the nth image block and the cth tissue type, represents the similarity score between the nth image block and tissue types other than the cth class, represents the number of tissue types, N Indicates the number of image blocks, c Indicates the c-th tissue type, Indicates tissue types other than type C.

[0079] The calculation expression of the similarity score is:

[0080] Where, represents the average similarity of the nth sample to the cth prototype, K represents the set of multiple prototype indexes of category c, represents the K-th prototype of the c-th tissue type, f( ) represents a multilayer perceptron, MEAN represents the average pooling operation, and γ( , ) represents the similarity function.

[0081] The prototype orthogonality loss function is introduced to improve the distinction between prototypes, avoid multiple prototypes concentrating in the same direction, and help to mine heterogeneous sub-groups within the organization. The expression of the prototype orthogonality loss function is:

[0082] Where, represents the prototype orthogonality loss function, represents the prototype of the c-th tissue type, |⋅|F represents the Frobenius norm, represents the identity matrix, represents the transposition of the prototype of the cth tissue type; The expression of the joint loss function is:

[0083] Where, represents the joint loss function, represents an adjustable hyperparameter, represents an adjustable hyperparameter, represents a tunable hyperparameter.

[0084] Example 3 This embodiment also provides a cancer survival prediction system based on pathological images, such as Figure 5 The structural diagram shown includes: A feature extraction module is used to divide the acquired pathological image into a plurality of image blocks; obtain a feature vector of each image block and identify the tissue type to which each image block belongs; a heterogeneous graph construction module, configured to calculate spatial proximity, feature similarity, and tissue type compatibility between image blocks based on the center coordinates of the image blocks, the feature vectors, and the tissue type to which each image block belongs; and construct a dynamic heterogeneous graph based on the calculation results; A multi-scale feature map extraction module is used to extract features from the dynamic heterogeneous map to obtain a multi-scale feature map of the dynamic heterogeneous map; The multi-prototype generation module is used to learn and generate multiple prototypes based on the tissue type of the image patch and the cross-category attention mechanism; The prediction module is used to aggregate multi-scale feature maps and multiple prototypes to obtain the final image representation, and cancer survival prediction is performed based on the final image representation.

[0085] The embodiments are provided merely to illustrate the present invention and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications may be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the claims.

Claims

1. A cancer survival prediction method based on pathological images, characterized in that: The following steps are involved: S1: Divide the acquired pathological image to obtain a plurality of image blocks; obtain a feature vector of each image block and identify the tissue type to which each image block belongs; S2: calculating spatial proximity, feature similarity, and tissue type compatibility between image blocks based on the center coordinates of the image blocks, the feature vectors, and the tissue type to which each image block belongs; and constructing a dynamic heterogeneous graph based on the calculation results; S3: extracting features from the dynamic heterogeneous graph to obtain a multi-scale feature graph of the dynamic heterogeneous graph; S4: Based on the tissue type of the image patch and the cross-category attention mechanism, multi-prototype learning is performed to generate multiple prototypes; S5: Aggregate multi-scale feature maps and multiple prototypes to obtain the final image representation, and perform cancer survival prediction based on the final image representation.

2. The method for predicting cancer survival based on pathological images according to claim 1, characterized in that: The method for identifying the tissue type to which each image block belongs includes: using a pre-trained tissue classifier to predict the tissue type of each image block to obtain the tissue type to which each image block belongs; The method also includes: performing unsupervised clustering on each image block, automatically dividing the image block into several tissue types, and obtaining the tissue type to which each image block belongs.

3. The method for predicting cancer survival based on pathological images according to claim 1, characterized in that: The process of calculating the spatial proximity, feature similarity and tissue type compatibility between image blocks according to the center coordinates of the image blocks, the feature vectors and the tissue type to which each image block belongs is as follows: Calculate the spatial proximity between image blocks, the expression is: Where, represents the spatial proximity between the i-th image block and the j-th image block, represents the coordinates of the center point of the i-th image block, represents the coordinates of the center point of the j-th image block, . represents the Euclidean distance; According to the feature vectors of the image blocks, the feature similarity between the image blocks is calculated, and the expression is: Where, represents the feature similarity between the i-th image block and the j-th image block, represents the feature vector of the i-th image block, represents the feature vector of the j-th image block, Represents the transpose of the feature vector of the i-th image block; According to the tissue type to which each image block belongs, the tissue compatibility between the image blocks is calculated, and the expression is: g ( , ), where Indicates the tissue type to which the i-th image block belongs, Indicates the tissue type to which the j-th image block belongs, g () represents the tissue compatibility function.

4. The method for predicting cancer survival based on pathological images according to claim 3, characterized in that: According to the spatial proximity, feature similarity and tissue type compatibility between image blocks, a dynamic edge determination mechanism is established to satisfy the expression: Where, The formula for determining whether a node is connected is: represents the weight coefficient of the corresponding item, represents the threshold value, Based on the dynamic edge judgment mechanism, a dynamic heterogeneous graph is constructed to represent the connection relationship between all image blocks in the pathological image; the dynamic heterogeneous graph is expressed as: G=(V,E), where V represents the node set, which is the set of the image blocks; E represents the edge set, which is the connection relationship between the image blocks.

5. The method for predicting cancer survival based on pathological images according to claim 4, characterized in that: The multi-layer graph convolutional neural network is used to extract features of the dynamic heterogeneous graph, and the expression of the multi-scale feature map of the dynamic heterogeneous graph is obtained as follows: = Concat [ , ,…, ] H = MLP ( ) Where, represents the output of the lth layer of the multi-layer graph convolutional neural network, represents a multi-layer graph convolutional neural network, represents the output of the l-1th layer of the multi-layer graph convolutional neural network, A represents the adjacency matrix of the dynamic heterogeneous graph, represents the output of the 0th layer of the multi-layer graph convolutional neural network, X represents the set of feature vectors of all image blocks, Represents the output concatenation of each layer of the multi-layer graph convolutional neural network, Concat Represents a splicing operation, represents the output of the Lth layer of the multi-layer graph convolutional neural network, H represents a multi-scale feature map, MLP Represents a multilayer perceptron.

6. The method for predicting cancer survival based on pathological images according to claim 1, characterized in that: Based on the tissue type of the image block and the cross-category attention mechanism, multi-prototype learning is performed. The process of generating multiple prototypes is as follows: Calculate the central prototype vector of each tissue type, the expression is: Where, represents the central prototype vector, represents the set of image blocks belonging to the cth tissue type, Represents the feature vector of the i-th image block; Gaussian perturbation is introduced for multi-prototype learning to generate K prototypes for each type of tissue, expressed as: Where, represents the K-th prototype of the c-th tissue type, represents Gaussian perturbation, represents the multivariate normal distribution, represents the covariance matrix, represents the identity matrix; The K prototypes of each tissue type are spliced ​​separately to generate the prototype of each tissue type. The expression is: =concat{ , ,..., } Where, Represents the prototype of the c-th tissue type, concat Represents a splicing operation; Archetypes based on each type of organizational archetype , using the cross-category attention mechanism to update each type of organizational prototype, the expression of the cross-category attention mechanism is defined as: Where, represents the attention mechanism, Q represents the query vector, K represents the key vector, V represents a value vector, represents the transpose of the key vector, softmax represents the normalized exponential function, d Represents the key vector K Dimensions; Take the tissue type prototype as the query vector Q , the image block is used as the key vector K Sum value vector V , update each type of organization prototype, the expression is: Where, Represents the prototype of the updated c-th tissue type, represents the learnable linear transformation matrix, represents the learnable linear transformation matrix, represents the prototype of the cth tissue type, represents a learnable linear transformation matrix; The prototype of each type of organizational prototype will be updated Splicing to form the initial multiple prototypes P.

7. The method for predicting cancer survival based on pathological images according to claim 1, characterized in that: The process of aggregating multi-scale feature maps and multiple prototypes to obtain the final image representation is: The attention mechanism is used to aggregate multi-scale feature maps and multiple prototypes to obtain the aggregated image representation, which is expressed as: Where, represents the image representation after aggregation, represents the attention mechanism, Represents multiple prototypes, represents a multi-scale feature map, represents the learnable linear transformation matrix, represents the learnable linear transformation matrix, represents a learnable linear transformation matrix; Based on the average pooling operation, the aggregated image representation is averagely pooled into the final image representation, which is expressed as: ） Where, represents the final image representation, Represents an average pooling operation.

8. The method for predicting cancer survival based on pathological images according to claim 1, characterized in that: Also includes: A joint loss function is used to train the cancer survival prediction process; the joint loss function includes: a survival prediction loss function, a prototype compatibility loss function, and a prototype orthogonality loss function; The expression of the survival prediction loss function is: Where, represents the survival prediction loss function, D represents the sample set of events, represents the risk set, represents the risk score of the a-th sample, represents the indexed risk score of the bth sample; The expression of the prototype compatibility loss function is: Where, represents the prototype compatibility loss function, represents the similarity score between the nth image block and the cth tissue type, represents the similarity score between the nth image block and tissue types other than the cth class, represents the number of tissue types, N Indicates the number of image blocks, c Indicates the c-th tissue type, Indicates tissue types other than type c; The expression of the prototype orthogonality loss function is: Where, represents the prototype orthogonality loss function, represents the prototype of the c-th tissue type, |⋅|F represents the Frobenius norm, represents the identity matrix, represents the transposition of the prototype of the cth tissue type; The expression of the joint loss function is: Where, represents the joint loss function, represents an adjustable hyperparameter, represents an adjustable hyperparameter, represents a tunable hyperparameter.

9. The method for predicting cancer survival based on pathological images according to claim 8, characterized in that: The calculation expression of the similarity score is: Where, represents the average similarity of the nth sample to the cth prototype, K Represents a set of multiple prototype indices of category c, represents the K-th prototype of the c-th tissue type, f( ) represents a multilayer perceptron, MEAN represents the average pooling operation, γ ( , ) Represents the similarity function.

10. A cancer survival prediction system based on pathological images, used to implement the cancer survival prediction method based on pathological images according to any one of claims 1 to 9, characterized in that: include: A feature extraction module is used to divide the acquired pathological image into a plurality of image blocks; obtain a feature vector of each image block and identify the tissue type to which each image block belongs; a heterogeneous graph construction module, configured to calculate spatial proximity, feature similarity, and tissue type compatibility between image blocks based on the center coordinates of the image blocks, the feature vectors, and the tissue type to which each image block belongs; and construct a dynamic heterogeneous graph based on the calculation results; A multi-scale feature map extraction module is used to extract features from the dynamic heterogeneous map to obtain a multi-scale feature map of the dynamic heterogeneous map; The multi-prototype generation module is used to learn and generate multiple prototypes based on the tissue type of the image patch and the cross-category attention mechanism; The prediction module is used to aggregate multi-scale feature maps and multiple prototypes to obtain the final image representation, and cancer survival prediction is performed based on the final image representation.

Images