Disease survival prediction method based on multi-scale pathological image and multi-modal data

Abstract

The invention belongs to the technical field of information science, and particularly relates to a disease survival prediction method based on a multi-scale pathological image and multi-modal data, which comprises the following specific steps: digital pathological image data processing, gene representation data processing, multi-modal data interaction and survival analysis. According to the method, image blocks corresponding to two magnification factors of x40 and x20 are used as input of a model, and integration of multi-scale information is realized by adopting an early fusion strategy; compared with a method of repeatedly calculating the same position by adopting a multi-view WSI, the method provided by the invention adopts a non-overlapping image segmentation and blocking mode, so that the over-fitting of the model can be prevented to a certain extent, and the efficiency of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of information science and technology, and in particular to a disease survival prediction method based on multi-scale pathological images and multi-modal data. Background Art

[0002] With the popularization of digital pathology, the application of deep learning technology in pathology has created conditions. Due to its powerful modeling and representation capabilities, deep learning technology has been widely used in computational pathology and has shown great potential in improving disease diagnosis. The application of deep learning technology helps to achieve accurate, objective and repeatable diagnosis. The quantitative analysis of pathological slice images based on deep learning has increased year by year. From the classification, segmentation, and detection of basic elements of pathological images (such as cells, glands, etc.) to the segmentation and detection of different areas of the entire pathological image, the rating of the degree of tumor deterioration, and the provision of survival analysis and personalized treatment reference, deep learning-based methods have been widely used and achieved remarkable results. Pathological images contain rich phenotypic information, which can be used to diagnose and monitor the condition of cancer patients, predict survival rates, and provide a reference for personalized cancer treatment. However, the traditional pathology diagnosis process is cumbersome and time-consuming, and is affected by the experience and subjective factors of pathologists, and the analysis results are prone to large differences. Moreover, compared with medical images, the time and energy required for labeling pathological images is enormous. And for pathological images themselves, due to their high resolution, if they need to be processed in large quantities, the demand for hardware equipment becomes very high.

[0003] So far, there is no good method to integrate multi-scale pathological image information and gene expression data at the same time. However, under the demand for fast and efficient pathological images, some methods for prognostic analysis of multimodal data have emerged. For example: "A cancer cell prognosis method based on multimodal interactive learning" 2024 actual review, using single-scale pathological images for segmentation and block, and the image blocks corresponding to each image constitute an image package, as a whole to represent WSI, and the patient's gene expression data is mapped to the genome to form a corresponding genomic representation, and then survival analysis is performed by capturing the optimal unit and performing multimodal fusion. "A method for predicting patient survival prognosis for multiple cancers" 2024 actual review, by obtaining multiple views from the same pathological image, feature extraction is performed through their respective graph convolutions to achieve feature representation of the image part. Then, the acquired transcriptome and genomic data are integrated first through a mixed attention encoder, and then the features of these three parts are fused through a triple encoder for survival analysis. The 2024 review of "Multimodal Fusion Survival Prediction Method Based on Slinkhorn Algorithm" divides the whole slice image WSI into 256×256 patches, and generates corresponding feature representations through the encoder for the original gene data. Then, through the cross-attention mechanism, the mutual influence between the two modal data is calculated respectively, and then the fusion features of the two parts are fused again for risk prediction. The 2024 review of "Cancer Survival Prediction Method, Model Pre-training Method and Device" randomly masks the pathological image, uses only part of the image blocks in the image, and splices them with the feature representations of RNA, miRNA, and DNA data. The fused feature representation is pre-trained to obtain a trained encoder, and this trained encoder is used as the encoder in the training process using all image blocks. Then, the image part and the text part are average pooled respectively to smooth the features for survival analysis. The 2024 review of "Cancer Patient Survival Prediction Method and Device Based on Artificial Intelligence" constructs corresponding heterogeneous graphs for the three modalities of pathology, genome, and clinical records, and then extracts key information from the existing data through feature edge reconstruction to enhance the feature representation of the three modal data. The high-dimensional feature representation obtained is then cross-learned through sparse encoders and ConvNeXt encoders to introduce the connection information between each node. Subsequently, the high-dimensional feature representations of the three parts are fused for survival prediction analysis.

[0004] Although the implementation scheme described above has achieved some results to a certain extent, there are still some problems. For "A Cancer Cell Prognosis Method Based on Multimodal Interactive Learning", after the tissue pathology image is segmented and divided into blocks, all image blocks are subjected to feature extraction, and all interact with the feature representation of the gene part. Each pathology image corresponds to an average of tens of thousands of image blocks, resulting in very high hardware conditions for this part of multimodal interaction. In addition, mapping the patient's genes to the genome may lack semantic correspondence with fine-grained pathological features, because in this way, genes can be divided into rough functional sets, representing different gene families, which can be used for labeling to simplify the correspondence, but such a set provides a basic and incomplete description of intracellular interactions, because a gene family may design different cellular functions. For "A method for predicting the survival prognosis of patients with various cancers", extracting blocks from a WSI through three views will greatly increase the amount of calculation compared to using one view, and there will be repeated parts between different views, causing the model to repeatedly learn the same area of ​​the WSI, which is very easy to overfit, resulting in a decrease in the generalization ability of the model. In addition, compared with the convolutional neural network to extract the feature information of the image block, the number of parameters required by the graph neural network used here will far exceed that of the convolutional neural network, and for large-scale data sets such as pathological images, the increase in these parameters will be even greater. As a result, the computing resources required for the overall model are very high, and there is a problem of repeated calculations, resulting in multiple repeated calculations. For "Multimodal fusion survival prediction method based on Slinkhorn algorithm", the image block features obtained after the whole slice image WSI is segmented and divided into blocks and the feature representation of the gene data are used to calculate the attention mechanism, but only the gene data guides the image data, but not the image data. In the calculation process of a large number of image blocks and gene data, the introduction of the Slinkhorn algorithm may face high computational complexity, especially when calculating pathological image data sets, the efficiency and scalability of the algorithm may be affected. For example, using the Slinkhorn algorithm to calculate the optimal transmission between two 256x256 pixel images may take several hours and consume nearly 100GB of memory, while a pathological image often generates tens of thousands of 256×256 image blocks. For the "Cancer Survival Prediction Method, Model Pre-training Method and Device", selecting image blocks representing the entire pathological image based on density may ignore the lesion information in the unselected areas because only part of the image blocks are selected, resulting in incomplete pathological diagnosis. The lesion areas in the pathological image may be unevenly distributed, and randomly or regularly selected image blocks may not cover all key areas.If the number of random selections is increased, the model can solve this problem, but the amount of calculation of the model will increase. It is better to use all the image blocks for training directly. This method is suitable for data with few samples, but for high-resolution pathological images, this method may not be very effective. In addition, in the preprocessing stage of omics data, only 300 genes with relatively large information are selected. This screening method can only be used in diseases with significant differences, which may lead to weak robustness of the model. For the "Method and Device for Predicting Survival of Cancer Patients Based on Artificial Intelligence", all input data are passed through the corresponding graph neural network, and then feature reconstruction is achieved to obtain a high-dimensional representation. First, the process of feature edge reconstruction may involve a lot of calculations, especially when processing high-resolution images or large-scale graphs, which may lead to increased consumption of computing resources and affect the speed of training and reasoning. It is definitely not suitable for high-resolution WSI. Secondly, tens of thousands of image blocks are constructed into heterogeneous graphs and the connections between them are calculated, which requires very high resources and takes a very long time to calculate. Furthermore, the high-dimensional feature vectors obtained by the pathology, genome, clinical records and other data are respectively obtained through their respective encoders and fused as the input for survival prediction. The late fusion strategy adopted in this way lacks the low-level correlation information between different modalities captured in the early stage, which helps to improve the robustness of the model. Even if the data of a certain modality is missing or noisy, the information of other modalities can still provide assistance to compensate.

[0005] In view of these existing problems, a disease survival prediction method based on multi-scale pathological images and multi-modal data is invented. Through the idea of ​​early fusion, features of different scales are combined, and combined with gene expression data and clinical record data, multiple modal data are fused in the late stage, so that the model can not only capture low-level correlation information between different modalities in the early stage, but also fully combine high-dimensional feature representations in the late stage. The overall idea is: first, high-resolution WSI, gene expression data corresponding to patients, 50 human biological pathways and clinical data are input, and WSI is divided and divided into blocks at x20 magnification and x40 magnification to obtain image blocks corresponding to two scales, and the two parts of feature representation are linked according to the position information in WSI, and then the feature representation corresponding to tens of thousands of image blocks is mapped to the expected prototype representation through the aggregation algorithm to achieve dimensionality reduction. Secondly, the gene expression data is mapped to the human biological pathway, and the expression products are integrated by corresponding to the positions on different pathways to integrate which life activities. Furthermore, the prototype representation corresponding to the image part and the pathway representation corresponding to the text part are mutually fused to obtain the fused high-dimensional feature representation. Finally, the risk score is calculated using the obtained high-dimensional features and combined with clinical data to predict the patient's survival period. Summary of the invention

[0006] In view of the above-mentioned and / or existing problems in a disease survival prediction method based on multi-scale pathological images and multi-modal data, the present invention is proposed.

[0007] Therefore, the purpose of the present invention is to provide a disease survival prediction method based on multi-scale pathological images and multimodal data, which can solve the above-mentioned existing problems.

[0008] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0009] A disease survival prediction method based on multi-scale pathological images and multi-modal data includes the following specific steps:

[0010] S1: digital pathology image data processing, the digital pathology image data processing includes WSI feature extraction and WSI feature prototype representation;

[0011] S2: Gene representation data processing, including human biological pathway mapping and feature extraction;

[0012] S3: Multimodal data interaction;

[0013] S4: Survival analysis.

[0014] As a preferred solution of the disease survival prediction method based on multi-scale pathological images and multi-modal data described in the present invention, the specific steps of the WSI feature extraction are as follows:

[0015] S11, data preprocessing: Hematoxylin-eosin-stained whole-slice images obtained from the GDC data portal were preprocessed using the CLAM toolbox, including tissue segmentation, smoothing, artifact removal, and extraction of non-overlapping fixed-size tissue region image patches from each WSI. Afterwards, CLAM was used to automatically identify subregions with high diagnostic value using attention-based learning, first segmenting the tissue and excluding any holes, and adjusting the size of the selected holes according to different diseases;

[0016] S12, self-supervised training: The UNI model is pre-trained using the DINOv2 self-supervised learning method, which is based on student-teacher knowledge distillation Where f is the student model, x is the input image, and θ is the parameter of the model. In this process, two loss targets, self-distillation loss and mask image modeling loss, are used to learn the high-level features and contextual information of the input image to optimize the target. In the pre-training stage, tissue slices extracted from WSIs are used as input and forward propagated through the UNI model to extract feature self-distillation loss and mask image modeling loss.

[0017] As a preferred solution of the disease survival prediction method based on multi-scale pathological images and multi-modal data described in the present invention, the formula of the self-distillation loss is:

[0018]

[0019] Where q(x) is the feature representation of the teacher model for the input x, q(x′) is the feature representation of the student model for the input x′, and |·|2 represents the Euclidean norm;

[0020] The formula for the mask image modeling loss is:

[0021]

[0022] where x masked is the masked region in the input image, is the reconstruction of the masked region predicted by the model;

[0023] The total loss is: [L total =L align +L recon ], through the online quoter strategy [x masked =M(x)], where M is a random mask operation. The model learns to predict the contextual information of the masked region in the input image, thereby capturing high-level visual features and contextual information.

[0024] As a preferred solution of the disease survival prediction method based on multi-scale pathological images and multi-modal data described in the present invention, the specific process of segmentation in S11 is as follows:

[0025] S111: Read the digitized slices into memory at a reduced resolution, I down = D(I, r), where I is the original image, r is the downsampling rate, D is the downsampling operation, and I down is a downsampled image that uses pathology sections at both x40 and x20 magnifications;

[0026] S112: Convert image information from RGB color space to HSV color space. HSV =C(I down ), where C is the conversion operation from RGB to color space, I HSV is the transformed image;

[0027] S113: After smoothing the edge through median filtering, threshold processing is performed on the saturation channel to calculate the binary mask of the region, I mask =T(I HSV), where T is the threshold operation, and the threshold is determined according to the median filtering result of the saturation channel, I mask is a binary mask image;

[0028] S114: Perform additional morphological closing operations to fill small gaps and holes, I close =MC(I mask ,k), where MC is the morphological closing operation, k is the structural element, and I close is the image after closing operation;

[0029] S115: The detected near-view contour of the foreground object is filtered based on the area threshold, C filtered =F(C,A threshold ), where F is based on the area threshold, A threshold is the filtering operation, C is the contour of the detected foreground object, and C filtered The filtered contours are stored for later processing steps, while the segmentation mask of each slice can be optionally visually inspected.

[0030] As a preferred solution of the disease survival prediction method based on multi-scale pathological images and multi-modal data described in the present invention, the specific steps of the WSI feature prototype representation are as follows:

[0031] S13, prototype construction: The feature representation of the image blocks obtained under the magnification of x20 and x40 is obtained through the idea of ​​early fusion to obtain multi-scale fusion information, and the multi-scale feature representation is used as the input of the aggregation algorithm to find the prototype representation; given: X = x1, x2, ..., x n is a set of n data points, each data point x i is a vector in d-dimensional space; K is the number of clusters into which you want to group the data points; C = c1, c2, …, c K is the aggregation of K cluster centers, each cluster center c j is also a d-dimensional vector; the goal is to minimize the total intra-cluster distance sum of squares, or WCSS:

[0032]

[0033] where ||x i -c j || is the data point x i and cluster center c j The Euclidean distance between them is firstly randomly selected as the initial cluster center; secondly, for each data point x i , according to its distance to each cluster center, it is assigned to the nearest cluster center c j ; Furthermore, for each cluster j, recalculate the cluster center cj is the mean of all data points in this cluster: Where S j is the set of all data points assigned to cluster j, N j YesS j Finally, S2 and S3 are iterated continuously until the cluster center no longer changes or the preset number of iterations is reached;

[0034] S14, prototype representation: Prototype representation: Assume that the data set is X = x1, x2, ..., x n , where each data point x i is a d-dimensional vector. Here, the center of the obtained cluster is used as the initialization of the Gaussian mixture model. If a mixture of K Gaussian distributions is used to fit this data set, the parameters of each Gaussian distribution include the mean vector μ k and the covariance matrix ∑ k , and the mixing weight π k (where k = 1, 2, ..., K), for each data point x i , the probability density function of which belongs to the kth Gaussian distribution is:

[0035]

[0036] Among them, |∑ k | is the covariance matrix∑ k The determinant of is the covariance matrix∑ k The inverse matrix of , d is the latitude of the data point, and the likelihood function of the entire data set is the weighted sum of each Gaussian distribution, which can be expressed as:

[0037]

[0038] Where, π={π1,π2,...,π k} is a set of mixing weights that satisfies μ={μ1,μ2,...,μ k} is the set of mean vectors, ∑={∑1,∑2,...,∑ k} is the covariance matrix of the multivariate Gaussian distribution, and the parameters of the Gaussian mixture model are determined by maximum likelihood estimation, which is implemented by the expectation maximization algorithm. The EM algorithm alternately performs two steps:

[0039] Step E: Calculate the posterior probability, also called responsibility, that each data point belongs to each Gaussian distribution;

[0040] M step: Update the parameters of the Gaussian distribution according to the posterior probability to maximize the likelihood function.

[0041] As a preferred solution of the disease survival prediction method based on multi-scale pathological images and multi-modal data described in the present invention, in S13, since the k-means algorithm is highly dependent on the initialized cluster center and the number of iterations is also random, before the feature representation of the data enters the aggregated k-means, a preliminary random cluster center will be generated according to the characteristics of the data, and the steps are as follows:

[0042] S131, Density Estimation: First, the algorithm performs density estimation on each point in the data set by calculating the k-distance of each point. For each point P in the data set i , its density density(P i ) can be estimated by the k-distance, that is, the distance to the kth nearest neighbor: Where d(P i ,k) is P i The distance to its kth nearest neighbor;

[0043] S132, construct conditional density reachability graph: Based on density estimation, construct a graph in which nodes represent data points and edges represent density reachability between two points. If the distance from one point to another point is less than the minimum value of its k-distance and ε value, the two points are considered to be density connected;

[0044] Its point P i Point P j The density reachability of is based on the distance between them and their respective density estimates:

[0045]

[0046] Among them, d(P j ,P i ) is P j and P i If the density accessibility of a point is greater than ε, it is considered a core point:

[0047]

[0048] S133, construct a mutual density reachable graph: remove low-density points from CDRG to obtain MDRG. In MDRG, if two points are mutually density reachable, there is an edge between them. MDRG is constructed based on density reachability, where if P i P j The density reachability is greater than ε and P j P i The density accessibility of is also greater than ε, then P i and P j There are edges between:

[0049]

[0050] S134, hierarchical clustering: Based on MDRG, a minimum spanning tree is constructed, and then clusters are identified through pruning operations. The basis for pruning is the density of the edges in the tree. If the density of the edge is lower than a certain threshold, it is considered that the edge connects points of different clusters and is pruned;

[0051] S135, cluster identification: Finally, according to the pruning results, the connected nodes are grouped to form the final clusters.

[0052] As a preferred solution of the disease survival prediction method based on multi-scale pathological images and multi-modal data described in the present invention, the specific steps of human biological pathway mapping are as follows:

[0053] Use biological pathway markers to encode specific cellular functions from transcriptomics data, first by selecting appropriate inference units for transcriptomics analysis and second by encoding pathways given a set of gene expression data. Where N G is the number of genes. In addition, the markers at the pathway level are constructed as Its N P is the number of channels, d is the dimension of the label, and for each channel P i , learn a specific weight vector φ i , and then map the gene expression data g to the pathway label x(P) through the weight vector i :x(P) i =φ i (gP i ), where gP i It is the path P i We now reshape X(P) into Then each path is labeled x(P) i It is expressed as: It involves a sparse multi-layer perceptron that maps gene expression data g to pathway labels X(P). The mapping is expressed as: X(P) = MLP(g), where MLP is a local neural network with multiple hidden layers. It learns how to map gene expression data to pathway label space. Finally, pathway labels are used as input for multimodal fusion to capture the multimodal interactions between pathways and histology.

[0054] As a preferred solution of the disease survival prediction method based on multi-scale pathological images and multi-modal data described in the present invention, the specific steps of feature extraction are as follows:

[0055] Its feature extraction consists of a linear layer, a ReLU layer, a maximum pooling layer, an LSTM layer, and a dropout layer. The gene expression data processed by MLP is input into the biological pathway and mapped into a low-dimensional feature representation. First, the convolutional layer is followed by a ReLU layer with a kernel of 16 to capture spatial information from the given human biological pathway and gene expression features. Then, a maximum pooling layer is used to select the maximum value based on the output of the convolutional layer, thereby reducing the dimensionality of the input and improving the computational efficiency of the model. The pooling operation definition is: pooling(X) ik =max({X iM,k ,X (iM+1),k ,…,X (iM+M-1,k}), the maximum pooling layer is followed by an LSTM layer to capture long-term dependencies between sequences as well as directional and spatial distances. RNN is an alternative to CNN for processing sequential data, and LSTM networks were first proposed to use special hidden units to remember inputs for a long time. The key to using LSTM is the unit state, which is regulated by structures called gates, including input gates, forget gates, and output gates. First, the forget gate decides which information to discard or save, and secondly, it decides how much new information should be added to the unit state, and finally, it decides what value to output. The specific implementation steps of LSTM are as follows: (Ⅰ)f t =sigmoid(W f *[h t-1 ,x t ]+b f )(Ⅱ)i t =sigmoid(W i *[h t-1 ,x t ]+b i )(Ⅲ)C t =tanh(W G *[h t-1 ,x t ]+b G )(Ⅳ)S t =f t ⊙S t-1 +i t ⊙C t (V)O t =sigmoid(W0*[h t-1 ,x t ]+b0)(VI)h t =O t ⊙tanh(S t ), where W represents the weight matrix, b represents the bias, and f t 、i t , O t Respectively represent the weight values ​​of the forget gate, input gate, and output gate, x t , Ct 、h t They represent the input vector, memory representation and hidden layer state at time t respectively. ⊙ is element-wise multiplication. A dropout layer with a probability of 0.2 is added to avoid overfitting caused by ignoring half of the feature detectors, making the model more generalizable. Then, all dropout results of gene expression information are combined into the feature vector of the corresponding pathway, and then the corresponding features are output.

[0056] As a preferred solution of the disease survival prediction method based on multi-scale pathological images and multi-modal data described in the present invention, the specific steps of the multi-modal data interaction are as follows:

[0057] S31: For the self-attention mechanism, the multimodal prototype representation first calculates the query (Q), key (K) and value (V): A = W Q X, K = W K X、V=W V X, where X is the input sequence, W Q , W K , W V is a learnable weight matrix; secondly, the attention score is calculated: where d k is the dimension of the key vector; then calculate the normalized weights: Finally, weighted sum: outpot = attention weights V;

[0058] S32: For the cross-attention mechanism, the model calculates the information of one modality in the context of one modality separately. The multimodal prototype first calculates the query (Q) and key (K): Q = W Q X1, K = W K X2, where X1 and X2 are input sequences from different modalities, respectively, Q , W K is a learnable weight matrix; secondly, the attention score is calculated: Then calculate the normalized weight: Finally, the weighted sum is: outpot = attention weights V, where V is the value vector calculated from the second mode, which can be obtained through V = W V X2 obtained;

[0059] S33: The obtained cross-attention and self-attention feature representations are used to determine the subtype of the patient's disease through a classifier constructed using the Kolmogorov-Arnold Networks network, in preparation for the subsequent calculation of the sub-line score.

[0060] As a preferred solution of the disease survival prediction method based on multi-scale pathological images and multi-modal data described in the present invention, the specific steps of the survival analysis are as follows:

[0061] S41: Obtain the previously obtained multimodal fusion feature representation and clinical survival time, and use the model to learn the relationship between each feature and survival time, and calculate the coefficient of each feature;

[0062] S42: Extract the coefficient of each feature from the trained model, which represents the intensity and direction of the impact of each feature on survival time;

[0063] S43: Obtain the values ​​of all relevant characteristics of each patient to calculate the risk score, and use the following formula to calculate the risk score for each individual: where β0 is the intercept term, β1,β2,...,β n are the coefficients of the features, x1,x2,…,x n is the characteristic value of the individual. According to the calculated risk score, the median of the risk score is used as the cutoff point to divide the patients into high-risk group and low-risk group;

[0064] S44: Generate all possible patient pairs from the dataset. If there are n patients, there will be n(n-1) / 2 possible pairs. For each pair of patients, if the survival time of one of the patients does not reach the endpoint, or the survival time of both patients does not reach the endpoint, then this pair of patients is considered an invalid pair. For the valid pairs below, if one patient has a longer survival time and the model-predicted survival probability is also higher, then this pair of patients is considered consistent. If the patient with a longer survival time has a lower predicted survival probability, then this pair of patients is considered inconsistent.

[0065] Compared with existing technologies:

[0066] 1. Compared with the existing method of using single-scale pathological images for problem analysis, the present invention uses image blocks corresponding to two magnifications of x40 and x20 as the input of the model, and adopts an early fusion strategy to achieve the integration of multi-scale information; compared with the method of using multi-view WSI that will generate repeated calculations of the same position, the present invention adopts a non-overlapping image segmentation and block method, which can prevent the overfitting of the model to a certain extent and improve the efficiency of the model;

[0067] 2. Compared with directly processing the feature representation of pathological images, facing the situation that an average WSI corresponds to tens of thousands of image blocks, the present invention constructs a prototype representation of WSI. Through a density-based aggregation algorithm, the model can spontaneously aggregate and find a suitable prototype based on the morphological characteristics of the image blocks, and then add the information between the image blocks to the obtained prototype representation through the GMM Gaussian mixture model, so that the obtained prototype information is more comprehensive;

[0068] 3. Compared with using gene family trees to represent gene expression data, the present invention uses a method of mapping to human biological pathways to construct a prototype. Compared with gene family trees, the number of biological pathways is relatively small. Mapping can effectively compress data to simplify subsequent data analysis. In addition, the use of biological pathways can reveal the biological processes and molecular mechanisms behind changes in gene expression, allowing the model to learn the connections between gene expressions.

[0069] 4. Compared with using a single cross attention or splicing before entering the encoder, the present invention adopts the cross attention mechanism, the self-attention mechanism and the Kolmogorov-Arnold idea to integrate multimodal data, and uses a tree structure to learn the connection between the models; since the prototype construction in the first two steps has achieved efficient dimensionality reduction, the computing resources required for this part are relatively very small, and the full interaction used here can make the multimodal data understand its own characteristics and mutual commonalities, making the final representation more complete. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic diagram of the structural design of the present invention;

[0071] Figure 2 It is a thermal schematic diagram of the present invention;

[0072] Figure 3 It is a schematic diagram of the experimental results of the present invention;

[0073] Figure 4 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0074] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0075] The present invention provides a disease survival prediction method based on multi-scale pathological images and multi-modal data. Figure 1-Figure 4 , including the following specific steps:

[0076] S1, digital pathology image data processing:

[0077] S11, WSI feature extraction:

[0078] Data preprocessing: Hematoxylin-eosin (H&E)-stained whole slide images (WSIs) obtained from the GDC data portal were preprocessed using the CLAM toolbox (Clustering Constrained Attention Multi-Instance Learning), including tissue segmentation, smoothing, artifact removal, and extraction of non-overlapping fixed-size tissue area image patches from each WSI;

[0079] CLAM uses attention-based learning to automatically identify sub-regions with high diagnostic value. First, it segments the tissue and excludes any holes. The size of the holes is adjusted according to different diseases. For example, in the pathological sections of breast cancer, lung cancer, colorectal cancer and other cancers, the necrotic area in the center of the tumor may be instructive for treatment decisions. These necrotic areas usually appear as holes or cavities, which may indicate the aggressiveness, grade and prognosis of the tumor. The segmentation process is as follows: ① Read the digitized slice (WSI) into memory at a reduced resolution, I down = D(I, r), where I is the original image, r is the downsampling rate, D is the downsampling operation, and I down is a downsampled image; the present invention uses pathological sections at both x40 and x20 magnifications; ② converting image information from RGB color space to HSV color space, I HSV =C(I down ), where C is the conversion operation from RGB to color space, I HSV is the transformed image; ③ After smoothing the edges through median filtering, the saturation channel is thresholded and the binary mask of the area (foreground) is calculated, I mask =T(I HSV ), where T is the threshold operation, and the threshold is determined according to the median filtering result of the saturation channel, I mask is a binary mask image; ④ then perform additional morphological closing operations to fill small gaps and holes; I close =MC(I mask ,k), where MC is the morphological closing operation, k is the structural element, and I close is the image after the closing operation; ⑤ The near-view contour of the detected foreground object is filtered based on the area threshold, C filtered =F(C,A threshold ), where F is based on the area threshold, A threshold is the filtering operation, C is the contour of the detected foreground object, and C filtered is the filtered contour; and is stored for later processing, while the segmentation mask of each slice can be selectively visually inspected;

[0080] Self-supervised training: The UNI model is pre-trained using the DINOv2 self-supervised learning method, which is based on student-teacher knowledge distillation. Where f is the student model, x is the input image, θ is the parameter of the model, and two main loss targets are used: self-distillation loss (alignment loss), and mask image modeling loss (reconstruction loss);

[0081] Self-distillation loss:

[0082]

[0083] Where q(x) is the feature representation of the teacher model for the input x, q(x′) is the feature representation of the student model for the input x′, and |·|2 represents the Euclidean norm;

[0084] Mask image modeling loss:

[0085]

[0086] where x masked is the masked region in the input image, is the reconstruction of the masked region predicted by the model; the total loss: [L total =L align +L recon ]; through the online tokenizer strategy [x masked =M(x)], where M is a random mask operation. The model learns to predict the contextual information of the masked region in the input image, thereby capturing high-level visual features and contextual information.

[0087] The high-level features and contextual information of the input image are learned by combining self-distillation and reconstruction loss to optimize the target; in the pre-training stage, tissue slices extracted from WSIs are used as input and forward propagated through the UNI model to extract features;

[0088] The overall process can be summarized into the following steps:

[0089] ① Data preprocessing: I preprocessed =T(I original ), where I original is the original pathological image, T is the preprocessing operation (such as cropping, scaling, normalization, etc.), I preprocessed is the preprocessed image;

[0090] ②Mask image modeling: Where M is a masking operation that randomly masks the silent parts of the image, is the masked image;

[0091] ③Feature extraction: Where E is the feature extraction network of ViT, and F is the feature representation extracted from the masked image;

[0092] ④Self-distillation loss: L distill =D(F teacher ,F student ), where F teacher and F student S represents the feature representation extracted by the teacher network and the student network, and D is the loss function, which measures the difference between the two features;

[0093] ⑤Masked image reconstruction loss: Where R is the reconstruction loss, which measures the feature representation F to reconstruct the masked image ability;

[0094] ⑥Total loss: L total =L distill +L recon , the total loss is the combination of the self-distillation loss and the masked image reconstruction loss, which is used to train the ViT model;

[0095] ⑦ Feature representation optimization: θ * =argmin θ L total (θ), where θ is the model parameter, θ * is the optimized parameter that minimizes the total loss;

[0096] S12: WSI feature prototype representation

[0097] Prototype construction: The feature representation of the image blocks obtained under the x20 and x40 magnifications obtained above is used to obtain multi-scale fusion information through the idea of ​​early fusion. The multi-scale feature representation is used as the input of the aggregation algorithm to find the prototype representation; given: X = x1, x2, ..., x n is a set of n data points, each data point x i is a vector in d-dimensional space; K is the number of clusters into which you want to group the data points; C = c1, c2, …, c K is the aggregation of K cluster centers, each cluster center c j is also a d-dimensional vector; the goal is to minimize the total intra-cluster distance sum of squares, or WCSS:

[0098]

[0099] where ||x i -c j || is the data point x i and cluster center c jThe Euclidean distance between them; first, randomly select K data points as the initial cluster centers; second, for each data point x i , according to its distance to each cluster center, it is assigned to the nearest cluster center c j ; Furthermore, for each cluster j, recalculate the cluster center c j is the mean of all data points in this cluster: Where S j is the set of all data points assigned to cluster j, N j YesS j Finally, the second and third steps are iterated continuously until the cluster center no longer changes or the preset number of iterations is reached;

[0100] However, since the k-means algorithm is highly dependent on the initialized cluster center and the number of iterations is also random, before the feature representation of the data enters the aggregate k-means, a preliminary random cluster center will be generated based on the characteristics of the data; the steps are as follows:

[0101] ① Density estimation: First, the algorithm will perform density estimation for each point in the data set; this is usually achieved by calculating the k-distance of each point (i.e., the distance to the kth nearest neighbor); for each point P in the data set i , its density density(P i ) can be estimated by the k-distance, that is, the distance to the kth nearest neighbor: Where d(P i ,k) is P i The distance to its kth nearest neighbor;

[0102] ②Construct a Conditional Density Reachability Graph (CDRG): Based on density estimation, a graph is constructed in which nodes represent data points and edges represent the density reachability between two points. If the distance from one point to another is less than the minimum value of its k-distance and ε (epsilon) value, the two points are considered to be density connected.

[0103] Click P i Point P j The density reachability of is based on the distance between them and their respective density estimates:

[0104]

[0105] Among them, d(P j ,P i ) is P j and P i If the density accessibility of a point is greater than ε, it is considered a core point:

[0106]

[0107] ③ Construct a mutual density reachability graph (MDRG): Remove low-density points (i.e., noise points) from CDRG to obtain MDRG; in MDRG, if two points are mutually density-reachable, there is an edge between them; based on density reachability, construct MDRG, where if P i P j The density reachability is greater than ε and P j P i The density accessibility of is also greater than ε, then P i and P j There are edges between:

[0108]

[0109] ④ Hierarchical clustering: Based on MDRG, a minimum spanning tree (MST) is constructed, and then clusters are identified through pruning operations; the basis for pruning is the density of the edges in the tree. If the density of the edge is lower than a certain threshold, it is considered that the edge connects points of different clusters and is pruned;

[0110] ⑤ Cluster identification: Finally, according to the results of pruning, the connected nodes are grouped to form the final clusters;

[0111] Prototype representation: Since the previous step only finds the cluster center of the data set and uses it as the prototype of the data set, the feature representation contained in each cluster is still independent. Therefore, the Gaussian mixture model is introduced here to introduce the relationship between the independent feature representations in the cluster.

[0112] Gaussian Mixture Model (GMM) is a probability model used to represent a data set with multiple subgroups, where the data distribution of each subgroup can be described by a Gaussian distribution (also called a normal distribution). GMM is a special case of a finite mixture model, which assumes that the data is generated by a mixture of a finite number of Gaussian distributions, each corresponding to a subgroup. The basic components of a Gaussian mixture model are: Mean: Each Gaussian distribution has a mean, which indicates the center position of the distribution; Covariance: Each Gaussian distribution has a covariance matrix, which describes the distribution range and shape of the data points in different dimensions; Weights: Each Gaussian distribution has a weight, which indicates the proportion of the distribution in the entire data set, and the sum of the weights of all Gaussian distributions is 1. Assume that the data set is X = x1, x2, …, xn , where each data point x i is a d-dimensional vector. Here, the center of the cluster obtained in the previous step is used as the initialization of the Gaussian mixture model. We want to fit this data set with a mixture of K Gaussian distributions (or components); the parameters of each Gaussian distribution include the mean vector μ k and the covariance matrix ∑ k , and the mixing weight π k (where k = 1, 2, ..., K); for each data point x i , the probability density function of which belongs to the kth Gaussian distribution is:

[0113]

[0114] Among them, |∑ k | is the covariance matrix∑ k The determinant of is the covariance matrix∑ k The inverse matrix of , d is the latitude of the data point; the likelihood function of the entire data set is the weighted sum of each Gaussian distribution, which can be expressed as:

[0115]

[0116] Where, π={π1,π2,...,π k} is a set of mixing weights that satisfies μ={μ1,μ2,...,μ k} is the set of mean vectors, ∑={∑1,∑2,...,∑ k} is a multivariate Gaussian distribution with a covariance matrix; and, the parameters of the Gaussian mixture model are usually determined by maximum likelihood estimation (MLE), which is usually implemented by the expectation maximization (EM) algorithm, which alternately performs two steps: ①E step: calculate the posterior probability that each data point belongs to each Gaussian distribution, also known as responsibility; ②M step: update the parameters of the Gaussian distribution based on these posterior probabilities to maximize the likelihood function;

[0117] S2, gene representation data processing:

[0118] Since this part of the data is not suitable for feature extraction using the idea of ​​block division like WSI, the idea of ​​grouping can be used. However, mapping the patient's genes to the genome may lack semantic correspondence with fine-grained pathological features, because in this way, the genes will be divided into rough functional sets, representing different gene families, for labeling to simplify the correspondence, but such a set provides a basic and incomplete description of intracellular interactions, because a gene family may design different cellular functions. Therefore, here we use the method of mapping gene expression data to human biological pathways, and then construct a prototype representation of gene expression data through feature extraction;

[0119] S21: Human Biological Pathway Mapping:

[0120] Use biological pathway tokens to encode specific cellular functions from transcriptomics data; these pathway tokens are generated from transcriptomics data, and they represent sets of genes known to interact with each other that are associated with specific cellular functions, such as the TGF-β signaling pathway, which plays a role in the epithelial-mesenchymal transition in breast cancer; first, choosing an appropriate unit of inference for transcriptomics analysis is challenging because of the complexity and hierarchical nature of cellular processes; pathways consist of a set of genes or subpathways involved in a specific biological process, providing a natural unit of inference for such analysis; second, encoding pathways, given a set of gene expression data Where N G is the number of genes, where we want to construct pathway-level markers Here N P is the number of pathways, d is the dimension of the label; for each pathway P i , learn a specific weight vector φ i , and then use this weight vector to map the gene expression data g to the pathway label x(P) i :x(P) i =φ i (gP i ), where gP i It is the path P i A subset of the gene expression data in ; now reshape X(P) into Then each path is labeled x(P) i It can be expressed as: This involves a sparse multi-layer perceptron (S-MLP) that maps gene expression data g to pathway labels X(P); this mapping can be expressed as: X(P) = MLP(g), where MLP is a local neural network with multiple hidden layers that learns how to map gene expression data to pathway label space; ultimately, these pathway labels are used as input to multimodal fusion to capture the multimodal interactions between pathways and histology;

[0121] S22: Feature extraction:

[0122] This part consists of a linear layer, a ReLU layer, a maximum pooling layer, an LSTM layer, and a dropout layer; the gene expression data processed by the MLP is input into the biological pathway and mapped into a low-dimensional feature representation; the advantage of mapping to the biological pathway is that it not only greatly reduces the parameters of the network but also can be trained in parallel; first, the convolutional layer is followed by a ReLU layer with a kernel of 16 to capture spatial information from given human biological pathways and gene expression features; one of the advantages of using CNN to analyze omics data is that the input data type is diverse, features can be easily integrated, and effective features can be automatically discovered through representation learning; then, a maximum pooling layer is used to select the maximum value based on the output of the convolutional layer, thereby reducing the dimensionality of the input and improving the computational efficiency of the model; pooling operation definition: pooling(X) ik =max({X iM,k ,X (iM+1),k ,…,X (iM+M-1,k}); The maximum pooling layer is followed by an LSTM layer to capture long-term dependencies between sequences as well as directional and spatial distances; RNN is an alternative to CNN for processing sequential data, and LSTM networks were first proposed to use special hidden units to remember inputs for a long time; The key to using LSTM is the cell state, which is regulated by structures called gates, including input gates, forget gates, and output gates; First, the forget gate decides which information to discard or keep, secondly, it decides how much new information should be added to the cell state, and finally, it decides what value to output; The specific implementation steps of LSTM are as follows: (Ⅰ)f t =sigmoid(W f *[h t-1 ,x t ]+b f )(Ⅱ)i t =sigmoid(W i *[h t-1 ,x t ]+b i )(Ⅲ)C t =tanh(W G *[h t-1 ,x t ]+b G )(Ⅳ)St =f t ⊙S t-1 +i t ⊙C t (V)O t =sigmoid(W0*[h t-1 ,x t ]+b0)(VI)h t =O t ⊙tanh(S t ), where W represents the weight matrix, b represents the bias, and f t 、i t , O t Respectively represent the weight values ​​of the forget gate, input gate, and output gate, x t , C t 、h t denote the input vector, memory representation, and hidden layer state at time t, respectively, and ⊙ is element-wise multiplication. A dropout layer with a probability of 0.2 is added to avoid overfitting caused by ignoring half of the feature detectors, making the model more generalizable. Next, all dropout results of gene expression information are combined into feature vectors of the corresponding pathways, and then the corresponding features are output.

[0123] S3, multimodal data interaction:

[0124] Since data of different modalities are represented by high-dimensional features by finding their prototype representations, the next step is how to fuse multimodal information. The strategy adopted here is the self-attention mechanism + cross-attention mechanism. Pathology and transcriptomics capture the long-distance dependencies within the sequence and focus on the information of another modality in the context of one modality. For the self-attention mechanism, the prototype representation of multimodality first calculates the query (Q), key (K) and value (V): Q = W Q X, K = W K X、V=W V X, where X is the input sequence, W Q , W K , W V is a learnable weight matrix; secondly, the attention score is calculated: where d k is the dimension of the key vector; then calculate the normalized weights: Finally, the weighted sum is taken: outpot = attention weights V; for the cross attention mechanism, the model calculates the information of focusing on one modality in the context of one modality respectively; the multimodal prototype first calculates the query (Q) and key (K): Q = W Q X1, K = W K X2, where X1 and W2 are input sequences from different modalities, respectively.Q , W K is a learnable weight matrix; secondly, the attention score is calculated: Then calculate the normalized weight: Finally, the weighted sum is: outpot = attention weights V, where V is the value vector calculated from the second modality, which can be obtained by V = W V X2 is obtained; finally, the obtained cross-attention and self-attention feature representations are used to determine the subtype of the patient's disease through the classifier constructed by the Kolmogorov-Arnold Networks network, in preparation for the subsequent calculation of the line score;

[0125] S4, Survival analysis:

[0126] The risk score is calculated based on the survival model designed above to estimate the risk of an individual experiencing an event (such as death or disease recurrence) within a specific period of time. The specific process is as follows:

[0127] ① First, obtain the previously obtained multimodal fusion feature representation and clinical survival time, and use the model to learn the relationship between each feature and survival time, and calculate the coefficient of each feature;

[0128] ② Extract the coefficients of each feature from the trained model. These coefficients represent the intensity and direction of the impact of each feature on survival time.

[0129] ③ Obtain the values ​​of all relevant features (feature representation and feature coefficient) of each patient to calculate the risk score, and use the following formula to calculate the risk score for each individual: where β0 is the intercept term, β1,β2,...,β n are the coefficients of the features, x1,x2,…,x n is the characteristic value of the individual; based on the calculated risk score, the median of the risk score can be used as the dividing point to divide the patients into high-risk group and low-risk group;

[0130] ④ Generate all possible patient pairs from the data set. If there are n patients, there will be n(n-1) / 2 possible pairs. For each pair of patients, if the survival time of one of the patients does not reach the endpoint (i.e., is censored), or the survival time of both patients does not reach the endpoint, then this pair of patients is considered an invalid pair. For the valid pairs below, if one patient has a longer survival time and the model-predicted survival probability is also higher, then this pair of patients is considered "consistent". If the predicted survival probability of the patient with a longer survival time is lower, then this pair of patients is considered "inconsistent". The C-index refers to the ratio of all consistent pairs to the total number of valid pairs, calculated as: c-index = consistent pairs / total number of valid pairs. If the model's prediction is completely accurate, the C-index will be 1. If the model's prediction is completely inaccurate, the C-index will be 0.5 (because in this case, the probability of the predicted result being consistent and inconsistent with the actual result is equal). The performance of the model in predicting patient survival time is indicated by measuring the value of the C-index.

[0131] Based on the above, the following implementation cases are included but not limited to:

[0132] S1: Digital pathology image data processing:

[0133] WSI feature extraction:

[0134] Data preprocessing

[0135] Hematoxylin and eosin (H&E)-stained whole-slide images (WSIs) were obtained from the GDC data portal;

[0136] The CLAM toolbox was used to preprocess WSIs, including tissue segmentation, smoothing, artifact removal, and extraction of non-overlapping fixed-size tissue region image patches;

[0137] Automatically identify sub-regions with high diagnostic value, exclude holes, and adjust hole sizes to suit different diseases;

[0138] Convert the image from RGB color space to HSV color space, perform median filtering to smooth edges, threshold processing, morphological closing operation, and foreground object filtering based on area threshold;

[0139] Self-supervised training

[0140] The UNI model is pre-trained using the DINOv2 self-supervised learning method based on student-teacher knowledge distillation;

[0141] Self-distillation loss (alignment loss) and mask image modeling loss (reconstruction loss) are used as the main loss targets;

[0142] Through the online quoter strategy, the model learns to predict the contextual information of the masked region in the input image, capturing high-level visual features and contextual information;

[0143] The WSI prototype represents:

[0144] Prototype building

[0145] Through the idea of ​​early fusion, we get multi-scale fusion information, use the multi-scale feature representation as the input of the aggregation algorithm, and find the prototype representation;

[0146] Use k-means algorithm for clustering to minimize the total intra-cluster distance sum of squares (WCSS);

[0147] To solve the initialization problem of k-means, density estimation is first performed, and the conditional density reachable graph (CDRG) and mutual density reachable graph (MDRG) are constructed to perform hierarchical clustering and cluster identification;

[0148] Prototype Representation

[0149] Introducing Gaussian mixture model (GMM) to introduce the relationship between independent feature representations in the cluster;

[0150] The parameters of GMM are determined by maximum likelihood estimation (MLE) and expectation maximization (EM) algorithms;

[0151] S2: Gene expression data processing:

[0152] Human Biological Pathway Mapping

[0153] Encode specific cellular functions from transcriptomics data using pathway tokens;

[0154] The gene expression data were mapped to pathway labels via a sparse multi-layer perceptron (S-MLP);

[0155] Feature extraction

[0156] The gene expression data processed by MLP are input into biological pathways and mapped into low-dimensional feature representations;

[0157] Gene expression data were processed using CNN, ReLU, max pooling, LSTM, and dropout layers to capture spatial information and long-term dependencies.

[0158] S3: Multimodal data interaction:

[0159] Self-attention mechanism and cross-attention mechanism

[0160] Compute the query (Q), key (K), and value (V) of the multimodal prototype representation;

[0161] Calculate the attention score and normalized weight, and perform weighted summation to get the output;

[0162] Compute queries (Q) and keys (K) across modalities, compute attention scores and normalized weights;

[0163] The weighted summation yields the output, where the value vector (V) comes from the second mode;

[0164] S4, Survival analysis:

[0165] Risk score calculation

[0166] Obtain multimodal fusion feature representation and clinical survival time;

[0167] The model learns the relationship between each feature and survival time and calculates the coefficient of each feature;

[0168] A formula was used to calculate each individual’s risk score;

[0169] C-Indicator Evaluation

[0170] Obtain multimodal fusion feature representation and clinical survival time;

[0171] The model learns the relationship between each feature and survival time and calculates the coefficient of each feature;

[0172] A risk score is calculated for each individual using a formula.

[0173] Furthermore, in the present invention:

[0174] according to Figure 1 It can be seen that Figure 1 The general structure of the proposed method is described, and the input is WSI, gene expression data and clinical data; the first step is to divide the WSI into two scales, obtain the feature representations corresponding to the two scales, and then integrate the two scale features through the idea of ​​early fusion; then the multi-scale feature representation is constructed into a prototype representation through an aggregation algorithm and feature extraction is performed; the second step is to construct a prototype representation by mapping transcriptomics data to human biological pathways to achieve dimensionality reduction; the third step is to fuse the multi-modal feature representations, and through the idea of ​​mid-term fusion, the features of each modality are interacted to obtain high-dimensional fusion features; then, through the idea of ​​late fusion, the high-dimensional feature representations of each part are integrated with the clinical data to calculate the risk value for patient survival prognosis.

[0175] according to Figure 2It can be seen that the risk value predicted by the model is fed back to the corresponding image block, and the depth of color is used to indicate the high and low risk value. Dark color indicates high risk and light color indicates low risk. It is reassembled into a complete pathological image for visualization to assist diagnosis and treatment.

[0176] according to Figure 3 It can be seen that after the experiment, train_loss (training loss) tends to 0.01, train_c_index (c-index indicator calculated during training) tends to 0.93, and the risk (risk score) corresponding to each patient is also significantly different. Based on this, it can be seen that as the input data increases, the model's loss is gradually decreasing, and most of the features in the data have been learned, proving that the designed model is feasible to a certain extent.

[0177] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed in the present invention may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A disease survival prediction method based on multi-scale pathological images and multi-modal data, characterized in that: The specific steps are as follows: S1: digital pathology image data processing, the digital pathology image data processing includes WSI feature extraction and WSI feature prototype representation; S2: Gene representation data processing, including human biological pathway mapping and feature extraction; S3: Multimodal data interaction; S4: Survival analysis.

2. The disease survival prediction method based on multi-scale pathological images and multi-modal data according to claim 1, characterized in that: The specific steps of WSI feature extraction are as follows: S11, data preprocessing: Hematoxylin-eosin-stained whole-slice images obtained from the GDC data portal were preprocessed using the CLAM toolbox, including tissue segmentation, smoothing, artifact removal, and extraction of non-overlapping fixed-size tissue region image patches from each WSI. Afterwards, CLAM was used to automatically identify subregions with high diagnostic value using attention-based learning, first segmenting the tissue and excluding any holes, and adjusting the size of the selected holes according to different diseases; S12, self-supervised training: The UNI model is pre-trained using the DINOv2 self-supervised learning method, which is based on student-teacher knowledge distillation Where f is the student model, x is the input image, and θ is the parameter of the model. In this process, two loss targets, self-distillation loss and mask image modeling loss, are used to learn the high-level features and contextual information of the input image to optimize the target. In the pre-training stage, tissue slices extracted from WSIs are used as input and forward propagated through the UNI model to extract feature self-distillation loss and mask image modeling loss.

3. The disease survival prediction method based on multi-scale pathological images and multi-modal data according to claim 2 is characterized in that: The formula for the self-distillation loss is: Where q(x) is the feature representation of the teacher model for the input x, q(x′) is the feature representation of the student model for the input x′, and |·|2 represents the Euclidean norm; The formula for the mask image modeling loss is: where x masked is the masked region in the input image, is the reconstruction of the masked region predicted by the model; The total loss is: [L total =L align +L recon ], through the online quoter strategy [x masked =M(x)], where M is a random mask operation. The model learns to predict the contextual information of the masked region in the input image, thereby capturing high-level visual features and contextual information.

4. The disease survival prediction method based on multi-scale pathological images and multi-modal data according to claim 1, characterized in that: The specific process of segmentation in S11 is as follows: S111: Read the digitized slices into memory at a reduced resolution, I down = D(I, r), where I is the original image, r is the downsampling rate, D is the downsampling operation, and I down is a downsampled image that uses pathology sections at both x40 and x20 magnifications; S112: Convert image information from RGB color space to HSV color space. HSV =C(I down ), where C is the conversion operation from RGB to color space, I HSV is the transformed image; S113: After smoothing the edge through median filtering, threshold processing is performed on the saturation channel to calculate the binary mask of the region, I mask =T(I HSV ), where T is the threshold operation, and the threshold is determined according to the median filtering result of the saturation channel, I mask is a binary mask image; S114: Perform additional morphological closing operations to fill small gaps and holes, I close =MC(I mask ,k), where MC is the morphological closing operation, k is the structural element, and I close is the image after closing operation; S115: The detected near-view contour of the foreground object is filtered based on the area threshold, C filtered =F(C,A threshold ), where F is based on the area threshold, A threshold is the filtering operation, C is the contour of the detected foreground object, and C filtered The filtered contours are stored for later processing steps, while the segmentation mask of each slice can be optionally visually inspected.

5. The disease survival prediction method based on multi-scale pathological images and multi-modal data according to claim 1, characterized in that: The specific steps of the WSI feature prototype are as follows: S13, prototype construction: The feature representations of the image blocks obtained at the magnifications of x20 and x40 are obtained, and the multi-scale fusion information is obtained through the idea of ​​early fusion. The multi-scale feature representations are used as the input of the aggregation algorithm to find the prototype representation; Given: X = x1, x2, …, x n is a set of n data points, each data point x i is a vector in d-dimensional space; K is the number of clusters into which you want to group the data points; C = c1, c2, …, c K is the aggregation of K cluster centers, each cluster center c j is also a d-dimensional vector; the goal is to minimize the total intra-cluster distance sum of squares, or WCSS: where ||x i -c j || is the data point x i and cluster center c j The Euclidean distance between them is firstly randomly selected as the initial cluster center; secondly, for each data point x i , according to its distance to each cluster center, it is assigned to the nearest cluster center c j ; Furthermore, for each cluster j, recalculate the cluster center c j is the mean of all data points in its cluster: Where S j is the set of all data points assigned to cluster j, N j YesS j Finally, S2 and S3 are iterated continuously until the cluster center no longer changes or the preset number of iterations is reached; S14, prototype representation: Assume that the data set is X = x1, x2, ..., x n , where each data point x i is a d-dimensional vector. Here, the center of the obtained cluster is used as the initialization of the Gaussian mixture model. If a mixture of K Gaussian distributions is used to fit this data set, the parameters of each Gaussian distribution include the mean vector μ k and the covariance matrix ∑ k , and the mixing weight π k (where k = 1, 2, ..., K), for each data point x i , the probability density function of which belongs to the kth Gaussian distribution is: Among them, |∑ k | is the covariance matrix∑ k The determinant of is the covariance matrix∑ k The inverse matrix of , d is the latitude of the data point, and the likelihood function of the entire data set is the weighted sum of each Gaussian distribution, which can be expressed as: Where, π={π1,π2,...,π k } is a set of mixing weights that satisfies μ={μ1,μ2,...,μ k } is the set of mean vectors, ∑={∑1,∑2,...,∑ k } is the covariance matrix of the multivariate Gaussian distribution, and the parameters of the Gaussian mixture model are determined by maximum likelihood estimation, which is implemented by the expectation maximization algorithm. The EM algorithm alternately performs two steps: Step E: Calculate the posterior probability, also called responsibility, that each data point belongs to each Gaussian distribution; M step: Update the parameters of the Gaussian distribution according to the posterior probability to maximize the likelihood function.

6. The disease survival prediction method based on multi-scale pathological images and multi-modal data according to claim 5, characterized in that: In S13, since the k-means algorithm is highly dependent on the initialized cluster center and the number of iterations is also random, before the feature representation of the data enters the aggregated k-means, a preliminary random cluster center is generated according to the characteristics of the data. The steps are as follows: S131, Density Estimation: First, the algorithm performs density estimation on each point in the data set by calculating the k-distance of each point. For each point P in the data set i , its density density(P i ) can be estimated by the k-distance, i.e. the distance to the kth nearest neighbor: Where d(P i ,k) is P i The distance to its kth nearest neighbor; S132, construct conditional density reachability graph: Based on density estimation, construct a graph in which nodes represent data points and edges represent density reachability between two points. If the distance from one point to another point is less than the minimum value of its k-distance and ε value, the two points are considered to be density connected; Its point P i Point P j The density reachability of is based on the distance between them and their respective density estimates: Among them, d(P j ,P i ) is P j and P i If the density accessibility of a point is greater than ε, it is considered a core point: S133, construct a mutual density reachable graph: remove low-density points from CDRG to obtain MDRG. In MDRG, if two points are mutually density reachable, there is an edge between them. MDRG is constructed based on density reachability, where if P i P j The density reachability is greater than ε and P j P i The density accessibility of is also greater than ε, then P i and P j There are edges between: S134, hierarchical clustering: Based on MDRG, a minimum spanning tree is constructed, and then clusters are identified through pruning operations. The basis for pruning is the density of the edges in the tree. If the density of the edge is lower than a certain threshold, it is considered that the edge connects points of different clusters and is pruned; S135, cluster identification: Finally, according to the pruning results, the connected nodes are grouped to form the final clusters.

7. The disease survival prediction method based on multi-scale pathological images and multi-modal data according to claim 1, characterized in that: The specific steps of human biological pathway mapping are as follows: Use biological pathway markers to encode specific cellular functions from transcriptomics data, first by selecting appropriate inference units for transcriptomics analysis and second by encoding pathways given a set of gene expression data. Where N G is the number of genes. In addition, the markers at the pathway level are constructed as Its N P is the number of channels, d is the dimension of the label, and for each channel P i , learn a specific weight vector φ i , and then map the gene expression data g to the pathway label x(P) through the weight vector i :x(P) i =φ i (gP i ), where gP i It is the path P i We now reshape X(P) into Then each path is labeled x(P) i It is expressed as: It involves a sparse multi-layer perceptron that maps gene expression data g to pathway labels X(P). The mapping is expressed as: X(P) = MLP(g), where MLP is a local neural network with multiple hidden layers. It learns how to map gene expression data to pathway label space. Finally, pathway labels are used as input for multimodal fusion to capture the multimodal interactions between pathways and histology.

8. The disease survival prediction method based on multi-scale pathological images and multi-modal data according to claim 1, characterized in that: The specific steps of feature extraction are as follows: Its feature extraction consists of a linear layer, a ReLU layer, a maximum pooling layer, an LSTM layer, and a dropout layer. The gene expression data processed by MLP is input into the biological pathway and mapped into a low-dimensional feature representation. First, the convolutional layer is followed by a ReLU layer with a kernel of 16 to capture spatial information from the given human biological pathway and gene expression features. Then, a maximum pooling layer is used to select the maximum value based on the output of the convolutional layer, thereby reducing the dimensionality of the input and improving the computational efficiency of the model. The pooling operation definition is: pooling(X) ik =max({X iM,k ,X (iM+1),k ,…,X (iM+M-1,k }), the maximum pooling layer is followed by an LSTM layer to capture long-term dependencies between sequences as well as directional and spatial distances. RNN is an alternative to CNN for processing sequential data, and LSTM networks were first proposed to use special hidden units to remember inputs for a long time. The key to using LSTM is the unit state, which is regulated by structures called gates, including input gates, forget gates, and output gates. First, the forget gate decides which information to discard or save, and secondly, it decides how much new information should be added to the unit state, and finally, it decides what value to output. The specific implementation steps of LSTM are as follows: (Ⅰ)f t =sigmoid(W f *[h t-1 ,x t ]+b f )(Ⅱ)i t =sigmoid(W i *[h t-1 ,x t ]+b i )(Ⅲ)C t =tanh(W G *[h t-1 ,x t ]+b G )(Ⅳ)S t =f t ⊙S t-1 +i t ⊙C t (V)O t =sigmoid(W0*[h t-1 ,x t ]+b0)(VI)h t =O t ⊙tanh(S t ), where W represents the weight matrix, b represents the bias, and f t 、i t , O t Respectively represent the weight values ​​of the forget gate, input gate, and output gate, x t , C t 、h t They represent the input vector, memory representation and hidden layer state at time t respectively. ⊙ is element-wise multiplication. A dropout layer with a probability of 0.2 is added to avoid overfitting caused by ignoring half of the feature detectors, making the model more generalizable. Then, all dropout results of gene expression information are combined into the feature vector of the corresponding pathway, and then the corresponding features are output.

9. The disease survival prediction method based on multi-scale pathological images and multi-modal data according to claim 1, characterized in that: The specific steps of the multimodal data interaction are as follows: S31: For the self-attention mechanism, the multimodal prototype representation first calculates the query (Q), key (K) and value (V): Q = W Q X, K = W K X、V=W V X, where X is the input sequence, W Q , W K , W V is a learnable weight matrix; secondly, calculate the attention score: attention where d k is the dimension of the key vector; then calculate the normalized weight: attention Finally, weighted sum: outpot = attention weights V; S32: For the cross-attention mechanism, the model calculates the information of one modality in the context of one modality separately. The multimodal prototype first calculates the query (Q) and key (K): Q = W Q X1, K = W K X2, where X1 and X2 are input sequences from different modalities, respectively, Q , W K is a learnable weight matrix; secondly, the attention score is calculated: Then calculate the normalized weight: Finally, the weighted sum is: outpot = attention weights V, where V is the value vector calculated from the second mode, which can be obtained through V = W V X2 obtained; S33: The obtained cross-attention and self-attention feature representations are used to construct a classifier through the Kolmogorov-Arnold Networks network to determine the patient's disease subtype, in preparation for the subsequent calculation of the sub-line score.

10. The disease survival prediction method based on multi-scale pathological images and multi-modal data according to claim 1, characterized in that: The specific steps of the survival analysis are as follows: S41: Obtain the previously obtained multimodal fusion feature representation and clinical survival time, and use the model to learn the relationship between each feature and survival time, and calculate the coefficient of each feature; S42: Extract the coefficient of each feature from the trained model, which represents the intensity and direction of the impact of each feature on survival time; S43: Obtain the values ​​of all relevant characteristics of each patient to calculate the risk score, and use the following formula to calculate the risk score for each individual: where β0 is the intercept term, β1,β2,...,β n are the coefficients of the features, x1,x2,…,x n is the characteristic value of the individual. According to the calculated risk score, the median of the risk score is used as the cutoff point to divide the patients into high-risk group and low-risk group; S44: Generate all possible patient pairs from the dataset. If there are n patients, there will be n(n-1) / 2 possible pairs. For each pair of patients, if the survival time of one of the patients does not reach the endpoint, or the survival time of both patients does not reach the endpoint, then this pair of patients is considered an invalid pair. For the valid pairs below, if one patient has a longer survival time and the model-predicted survival probability is also higher, then this pair of patients is considered consistent. If the patient with a longer survival time has a lower predicted survival probability, then this pair of patients is considered inconsistent.

Images