Multi-modal survival prediction method fusing digital pathological image and omics data

Abstract

The invention relates to a multi-modal survival prediction method fusing a digital pathological image and omics data. The method comprises the following steps: acquiring a full-view digital pathological image and omics data of an esophageal squamous cell carcinoma patient; carrying out preprocessing and feature extraction on the full-view digital pathological image and the omics data to respectively obtain a pathological embedding representation and an omics embedding representation; fusing the pathological embedding representation and the omics embedding representation by using a multi-head attention mechanism of Transform to obtain a preliminary fusion feature; performing deep interaction on the pathological embedding representation and the preliminary fusion feature to obtain a final fusion feature; and predicting a survival result based on the final fusion feature. According to the method, bimodal sharing of pathological images and omics data and full fusion of features are realized, and prognosis survival prediction of esophageal squamous cell carcinoma cancer patients can be better realized.

Description

Technical Field

[0001] The present application relates to the technical field of bioinformatics, and in particular to a multimodal survival prediction method integrating digital pathological images and omics data. Background Art

[0002] At present, there are relatively few studies or methods for multimodal survival prediction of esophageal squamous cell carcinoma. At the tissue scale, limited computational pathology studies have demonstrated the effectiveness of predicting overall survival and treatment response to chemoradiotherapy by pathological features; at the molecular scale, some studies have shown that prognosis prediction of esophageal squamous cell carcinoma based on gene expression features has good predictability and reliability. However, there is a lack of research on survival prediction of esophageal squamous cell carcinoma using deep learning technology in multimodal data of histological and genomic patterns. Pathological genomics combines synthetic complementary data patterns with cutting-edge multimodal artificial intelligence methods to enhance the diagnosis, prognosis and prediction decision-making process of oncology. In terms of medical multimodal fusion methods, several feature fusion-based technologies including genomic graph convolutional neural network (GSCNN), attention-based multiple instance learning (AMIL), deep attention multiple instance survival learning (DeepAttnMISL), and multimodal co-attention transformer (MCAT) have been introduced to merge feature representations of different modalities. These methods mainly focus on improving individual modality features and adopt direct fusion strategies (i.e., concatenation, bilinear pooling, global attention pooling, and collaborative attention fusion), but do not adequately explore the interactions within and between models. The above factors limit the effectiveness of deep learning for clinical prognosis prediction. Summary of the invention

[0003] Based on this, it is necessary to provide a multimodal survival prediction method that integrates digital pathology images and omics data, which includes:

[0004] S1: Acquire full-field digital pathology images and omics data of patients with esophageal squamous cell carcinoma;

[0005] S2: Preprocessing and feature extraction are performed on the full-field digital pathology image and the omics data to obtain a pathology embedding representation and an omics embedding representation, respectively;

[0006] S3: Utilize the Transformer's multi-head attention mechanism to fuse the pathology embedding representation and the omics embedding representation to obtain preliminary fusion features;

[0007] S4: deeply interacting the pathology embedding representation with the preliminary fusion feature to obtain a final fusion feature;

[0008] S5: Predicting survival outcomes based on the final fusion signature.

[0009] Beneficial effects: This method first preprocessed and extracted features of the full-field digital pathology images and genomic data of patients with esophageal squamous cell carcinoma, and then performed preliminary fusion and deep interaction on the obtained pathology embedding representation and omics embedding representation, respectively, to achieve dual-modal sharing of pathology images and omics data and full fusion of features, which can better realize the prognosis and survival prediction of patients with esophageal squamous cell carcinoma. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 This is a flowchart of a multimodal survival prediction method that integrates digital pathology images and omics data in an embodiment of the present application.

[0012] Figure 2 This is a schematic diagram of the survival results of an embodiment of the present application. DETAILED DESCRIPTION

[0013] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0014] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0015] like Figure 1 As shown, this embodiment provides a multimodal survival prediction method integrating digital pathological images and omics data, the method comprising:

[0016] S1: Acquire full-field digital pathology images and omics data of patients with esophageal squamous cell carcinoma.

[0017] In this embodiment, the training and validation data sets are derived from the TCGA esophageal cancer data set, which contains 159 cases, 249 full-view digital pathology images, and the omics data are copy number variation (94), gene mutation data (133), and transcriptome sequencing (2000).

[0018] S2: Preprocessing and feature extraction are performed on the full-field digital pathology image and the omics data to obtain a pathology embedding representation and an omics embedding representation, respectively.

[0019] Specifically, preprocessing and feature extraction of the full-field digital pathology image includes:

[0020] The tissue area of ​​the full-field digital pathology image is automatically segmented using the maximum inter-class variance threshold segmentation algorithm, and the full-field digital pathology image is read into the memory at a downsampled resolution (the default magnification is 20×), and the tissue area of ​​the full-field digital pathology image is evenly divided into multiple image blocks with an edge length of 360 μm and no overlap, and the full-field digital pathology image is adjusted to 256×256 pixels;

[0021] Use the fine-tuned pre-trained RseNet50 (clustering-guided contrastive learning, CCL) to extract the preprocessed full-field digital pathology image to obtain pathological features; reduce the dimension of the pathological features through a multi-layer perceptron combined with a feature dimensionality reduction network of proxy attention to obtain reduced dimension pathological features; and aggregate the reduced dimension pathological features and the proxy attention into the pathological embedding representation of the full-field digital pathology image through an attention pooling operation.

[0022] In this embodiment, the fine-tuning method is as follows: after loading the RseNet50 of CCL, a fully connected layer is added to the end of the model to ensure that the dimension of its output is 2048; the feature dimensionality reduction network first receives the pathological features with a dimension of 2048, and passes the pathological features through the first fully connected layer (including 1024 neurons) and the second fully connected layer (512 neurons) in the multi-layer perceptron in turn, and then passes through the proxy attention layer to obtain the reduced dimension pathological features with a dimension of 256;

[0023] In this embodiment, in the single-modality processing of pathological images, downsampling experiments are only performed at a magnification of 20×. If the hierarchical feature points of the full-field digital pathological image can be used to sample and extract features at 20×, 10× and 5×, and combined with the hierarchical pathological information, richer pathological features may be obtained.

[0024] Furthermore, the attention pooling operation is expressed as:

[0025]

[0026] Among them, H bag represents the pathology embedding representation; M represents the number of image blocks in the full-view digital pathology image; O m represents the proxy attention of the m-th image patch; h m represents the dimension-reduced pathological features of the mth image block; σ1(·) represents the first fully connected layer of the multilayer perceptron; σ2(·) represents the second fully connected layer of the multilayer perceptron; each fully connected layer of the multilayer perceptron has 256 neurons; Q m represents the query matrix of the mth image block; K m represents the key matrix of the mth image block; V m Represents the value matrix of the mth image block; C represents the dimension of the image block; A is the pooling of the query matrix of M image blocks, n represents the number of proxy tags, which is set to 81 in this embodiment; T represents transposition; B1 represents the first bias deviation, B2 represents the second bias deviation, DWC(·) denotes depthwise convolution.

[0027] Preprocessing and feature extraction of the omics data include:

[0028] Eliminating the molecular expressions that are invalid for survival prediction in the omics data to obtain valid genes and their expression values;

[0029] In this embodiment, the molecular expressions that are invalid for survival prediction include non-protein coding genes and genes that are not related to the prognosis of esophageal squamous cell carcinoma; the valid genes include the esophageal squamous cell carcinoma gene set found in the gene enrichment analysis database.

[0030] The preprocessed omics data is constructed into a sample feature matrix, wherein the sample feature matrix includes multiple valid genes and their expression values ​​in multiple samples; a gene interaction graph and a sample similarity network graph are constructed based on the sample feature matrix, wherein the gene interaction graph G (g) Includes gene set V (g) and the set of gene interaction edges E (g) , the sample similarity network graph G(s) Including sample set V (s) And the similarity set E between samples (s) Based on the gene interaction graph and the sample similarity network graph, a sparse graph convolutional neural network and a feature-weighted graph convolutional neural network are used to aggregate information on the sample feature matrix, and the aggregation result is reduced in dimension through a feedforward neural network containing two 256 neurons to obtain the omics embedding representation.

[0031] Further, the workflow of the sparse graph convolutional neural network includes:

[0032] A gene adjacency matrix is ​​calculated based on the gene interaction graph, wherein the gene adjacency matrix is ​​used to represent the connection relationship between genes, and a Hadamard product is performed on the gene adjacency matrix and the first weight matrix to obtain a new weight matrix;

[0033] After multiplying the new weight matrix by the sample feature matrix, a nonlinear transformation is performed to obtain a feature space aggregation result; the feature space aggregation result is used to calculate the omics embedding representation.

[0034] The workflow of the feature-weighted graph convolutional neural network includes:

[0035] Performing a Hadamard product on the feature space aggregation result and the first weight vector to obtain a feature weighting matrix;

[0036] Aggregating samples with consistent similarity in the sample similarity network graph to obtain a sample adjacency matrix;

[0037] After multiplying the feature weighted matrix with the sample adjacency matrix, a nonlinear transformation is performed to obtain a sample space aggregation result; the sample space aggregation result is used to pass through the feedforward neural network to obtain the omics embedding representation.

[0038] S3: The pathology embedding representation and the omics embedding representation are fused using the Transformer's multi-head attention mechanism to obtain preliminary fusion features.

[0039] Specifically, the step includes:

[0040] The pathology is embedded in the representation As a query, the omics embedding is represented as Make a copy, and use the two copies of the omics embedding representation as the key and the value respectively;

[0041] Based on the query, the key, and the value, the multi-head attention mechanism of Transformer is used for fusion to obtain the common attention, and the calculation formula is:

[0042]

[0043] Among them, CoA(·) represents shared attention; G bag Represents the omics embedding representation; H bag represents the pathological embedding representation; softmax(·) represents the softmax activation function; Q represents the query; K represents the key; d k Represents the vector dimension of the query; W q Represents the weight matrix of the query; W k represents the weight matrix of the key; W v represents the weight matrix of the value; T represents the transpose;

[0044] Based on the co-attention, the weight matrix of the value and the omics embedding representation, the preliminary fusion feature is obtained, and the calculation formula is:

[0045]

[0046] in, represents the initial fusion feature, A coa represents the co-attention matrix, which is used to describe the attention relationship between the image embedding and gene embedding of full-view digital pathology images in multimodal learning.

[0047] S4: Deeply interact the pathology embedding representation with the preliminary fusion feature to obtain the final fusion feature.

[0048] Specifically, the step includes:

[0049] The pathology is embedded in the representation and the preliminary fusion features After passing through the second fully connected layer, 32-dimensional pathological feature vectors are obtained respectively. 32-dimensional fused feature vector

[0050] Multiplying the pathological feature vector and the fusion feature vector by dot product to obtain pathological internal features and fusion internal features respectively;

[0051] Bilinearly fusing the pathological feature vector with the fused feature vector to obtain a shared mutual feature;

[0052] The pathological internal features, the fused internal features, and the mutual features are subjected to a reshape operation and a third fully connected layer (including 20 neurons) to obtain reshaped pathological feature vectors. Reshaped fused feature vector Reinventing mutual features

[0053] The pathological feature vector, the reshaped pathological feature vector, the reshaped mutual feature, the reshaped fusion feature vector, and the fusion feature vector are sequentially concatenated to obtain the final fusion feature. The calculation formula is:

[0054]

[0055] Among them, h final represents the final fusion feature, Represents vector concatenation.

[0056] S5: Predicting survival outcomes based on the final fusion signature.

[0057] In this embodiment, when performing risk prediction, the structure of the fusion architecture is optimized using a class-based survival loss function; for example: first, based on the quartiles of the event time of the uncensored patients, the continuous time scale T of the patient's total survival time (in days) is converted into os The survival prediction can be understood as a four-classification problem. The discrete survival time T of the mth patient is m It can be defined as:

[0058] T m =nIf T os ∈[t n ,t n+1 );

[0059] Where n represents the discretized event time index, which is used to refer to different intervals (or time periods) of patient survival time, T os represents the continuous time of the patient's overall survival time; [t n ,t n+1 ) represents the nth time interval; in the model, the survival time is divided into multiple non-overlapping intervals, such as four intervals, each of which corresponds to a specific time range.

[0060] For patients with the final fusion features, the corresponding weights are The prediction layer f haza Can be used to calculate the survival loss function f surv , the expression is:

[0061]

[0062] f haza (n|j final )=P(T m =n|T m ≥n,hfinal );

[0063] Then, the log-likelihood function is calculated, which is expressed as:

[0064] L m =-c i log(f surv (Y i |h final ))-(1-c i )log(f surv (Y i -1|h final ))-(1-

[0065] c i )log(f haza (Y i |h final ));

[0066] Among them, Y i represents the discretized true label of the mth patient; c i represents the patient's survival status, c i =1 means the event actually happened, c i = 0 means truncation; and the log-likelihood function L m is the loss function L for censored patients censored .

[0067] The loss function for uncensored patients is L uncensored It is expressed as:

[0068] L uncensored =-(1-c i )log(f surv (Y i -1|h final ))-(1-c i )log(f surv (Y i -1|h final ));

[0069] The loss function L for censored patients censored and the loss function L for uncensored patients uncensored Perform weighted summation to obtain the final training loss L surv , the expression is:

[0070] L surv =(1-α)L censored +L uncensored ;

[0071] Among them, α is the weight coefficient, which is used to increase the weight of uncensored patients.

[0072] This method first preprocessed and extracted features of the full-field digital pathology images and genomic data of patients with esophageal squamous cell carcinoma, and then performed preliminary fusion and deep interaction on the obtained pathology embedding representation and omics embedding representation, respectively, achieving dual-modal sharing of pathology images and omics data and full fusion of features, which can better realize the prognosis and survival prediction of patients with esophageal squamous cell carcinoma.

[0073] This embodiment also provides a multimodal cross-attention model based on a cross-domain bilinear network (MCCBN, Multimodal Cross-attention with Cross-domain Bilinear Network), which is used to implement a multimodal survival prediction method integrating digital pathological images and omics data. The model includes:

[0074] The acquisition module is used to obtain full-field digital pathological images and omics data of patients with esophageal squamous cell carcinoma;

[0075] A first preprocessing and feature extraction module, used for preprocessing and feature extraction of the full-field digital pathology image to obtain a pathology embedding representation;

[0076] A second preprocessing and feature extraction module, used for preprocessing and feature extraction of the omics data to obtain an omics embedded representation;

[0077] A pathology image-guided co-attention module is used to fuse the pathology embedding representation with the omics embedding representation using the Transformer's multi-head attention mechanism to obtain a preliminary fusion feature;

[0078] A cross-modal deep feature interaction module, used for deeply interacting the pathology embedding representation with the preliminary fusion feature to obtain a final fusion feature;

[0079] The risk prediction module predicts the survival outcome based on the final fusion features.

[0080] In this embodiment, the MCCB uses a learning rate of 1×10 -4 , L1 decay weight is 1×10 -5 The Adam optimizer is trained for 25 epochs.

[0081] In this embodiment, the survival results include risk scores, survival curves, and statistical significance. haza The output is the risk score of each patient, with the 50% quantile of the risk score as the dividing line. Patients below 50% are defined as low-risk groups, and those above 50% are defined as high-risk groups. The survival probability is calculated for each time point using the following formula:

[0082]

[0083] Among them, S(t) represents the survival probability at time t; n t represents the number of patients still alive at time t; d t represents the number of patients who died at time t; k represents the number of time points.

[0084] This example uses an esophageal squamous cell carcinoma dataset from a provincial cancer hospital as a test set, and uses the method and model provided in this example for prediction. The esophageal squamous cell carcinoma dataset from a provincial cancer hospital contains 41 cases and 41 full-view digital pathology images, and the omics data are copy number variation (94), gene mutation data (133), and transcriptome sequencing (2000). The predicted survival results are as follows: Figure 2 As shown, in Figure 2 In the example, the relationship between the probability of survival and time is plotted on the coordinate axis. Whenever an event (death) occurs, the survival curve will jump downward, and the magnitude of the jump corresponds to the number of patients at that time. The key survival time points and deaths are marked on the survival curve to facilitate observation of changes in survival rate. The KM curve also shows the changes in survival rate of different risk groups over time. For the significance test of patient stratification in the KM curve analysis, the log-rank test is used in this embodiment to measure whether the difference in survival distribution between the two groups (high-risk group and low-risk group) is statistically significant (ie, P-Value < 0.05). Finally, based on the shape of the KM curve and the results of inter-group comparison, the survival of different risk groups is explained to provide clinical decision support, and the average five-fold cross-validation consistency index is used to measure the predictive performance of the model.

[0085] The method and model provided in this embodiment have the following beneficial effects:

[0086] 1. This is the first study on multimodal survival research of esophageal squamous cell carcinoma pathological genomics. A multimodal cross-attention and cross-domain bilinear network method was designed to integrate genomic data (transcriptome sequencing, gene mutation data and copy number variation patterns) and digital pathology images for prognosis and survival prediction of esophageal squamous cell carcinoma patients.

[0087] 2. In terms of single-modal processing of pathological images, the pre-trained RseNet50 ResNet50 was introduced to extract in-domain features, eliminating the time-consuming and tedious manual labeling. At the same time, a multi-layer perceptron combined with a proxy attention module was designed to obtain a more informative pathological embedding representation.

[0088] 3. In the single-modal processing of omics data, a hierarchical graph convolutional neural network is used, which fully considers the information aggregation of gene expression data in feature space and sample space, which is more reasonable than using a feedforward neural network.

[0089] 4. In terms of information fusion and interaction, a pathology-guided co-attention module and a cross-modal deep feature interaction module are designed to fully interact and capture the intrinsic correlation between omics data and pathological images. Finally, the deep features of the two modalities are combined to complete risk prediction.

[0090] 5. The method and model were trained and validated on the TCGA esophageal cancer dataset and externally tested on the esophageal squamous cell carcinoma dataset of a provincial cancer hospital, verifying the effectiveness and robustness of the method and model.

[0091] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A multimodal survival prediction method integrating digital pathological images and omics data, characterized in that: include: S1: Acquire full-field digital pathology images and omics data of patients with esophageal squamous cell carcinoma; S2: Preprocessing and feature extraction are performed on the full-field digital pathology image and the omics data to obtain a pathology embedding representation and an omics embedding representation, respectively; S3: Utilize the Transformer's multi-head attention mechanism to fuse the pathology embedding representation and the omics embedding representation to obtain preliminary fusion features; S4: deeply interacting the pathology embedding representation with the preliminary fusion feature to obtain a final fusion feature; S5: Predicting survival outcomes based on the final fusion signature.

2. The multimodal survival prediction method integrating digital pathological images and omics data according to claim 1, characterized in that: In S2, preprocessing and feature extraction of the full-field digital pathology image includes: Automatically segmenting the tissue area of ​​the full-field digital pathology image using the maximum inter-class variance threshold segmentation algorithm, dividing the tissue area of ​​the full-field digital pathology image into a plurality of image blocks with an edge length of 360 μm and no overlap, and adjusting the full-field digital pathology image to 256×256 pixels; Use the pre-trained RseNet50 to extract the preprocessed full-field digital pathology image to obtain pathological features; reduce the dimension of the pathological features through a multi-layer perceptron combined with a feature dimensionality reduction network of proxy attention to obtain reduced dimension pathological features; and aggregate the reduced dimension pathological features and the proxy attention into the pathological embedding representation of the full-field digital pathology image through an attention pooling operation.

3. The multimodal survival prediction method integrating digital pathological images and omics data according to claim 2, characterized in that: The attention pooling operation is expressed as: Among them, H bag represents the pathology embedding representation; M represents the number of image blocks in the full-view digital pathology image; O m represents the proxy attention of the m-th image patch; h m represents the dimension-reduced pathological features of the mth image block; σ1(·) represents the first fully connected layer of the multilayer perceptron; σ2(·) represents the second fully connected layer of the multilayer perceptron; Q m represents the query matrix of the mth image block; K m represents the key matrix of the mth image block; V m Represents the value matrix of the mth image block; C represents the dimension of the image block; A is the pooling of the query matrix of M image blocks, n represents the number of proxy labels; T represents transposition; B1 represents the first bias deviation, B2 represents the second bias deviation, DWC(·) denotes depthwise convolution.

4. The multimodal survival prediction method integrating digital pathological images and omics data according to claim 1, characterized in that: In S2, preprocessing and feature extraction of the omics data include: Eliminating the molecular expressions that are invalid for survival prediction in the omics data to obtain valid genes and their expression values; The preprocessed omics data are constructed into a sample feature matrix, wherein the sample feature matrix includes multiple valid genes and their expression values ​​in multiple samples; a gene interaction graph and a sample similarity network graph are constructed based on the sample feature matrix, wherein the gene interaction graph includes a gene set and a set of interaction edges between genes, and the sample similarity network graph includes a sample set and a set of similarities between samples; based on the gene interaction graph and the sample similarity network graph, a sparse graph convolutional neural network and a feature-weighted graph convolutional neural network are used to aggregate information of the sample feature matrix, and the aggregation results are passed through a feedforward neural network to obtain the omics embedding representation.

5. The multimodal survival prediction method integrating digital pathological images and omics data according to claim 4, characterized in that: The workflow of a sparse graph convolutional neural network includes: A gene adjacency matrix is ​​calculated based on the gene interaction graph, wherein the gene adjacency matrix is ​​used to represent the connection relationship between genes, and a Hadamard product is performed on the gene adjacency matrix and the first weight matrix to obtain a new weight matrix; After multiplying the new weight matrix by the sample feature matrix, a nonlinear transformation is performed to obtain a feature space aggregation result; the feature space aggregation result is used to calculate the omics embedding representation.

6. The multimodal survival prediction method integrating digital pathological images and omics data according to claim 5, characterized in that: The workflow of the feature-weighted graph convolutional neural network includes: Performing a Hadamard product on the feature space aggregation result and the first weight vector to obtain a feature weighting matrix; Aggregating samples with consistent similarity in the sample similarity network graph to obtain a sample adjacency matrix; After multiplying the feature weighted matrix with the sample adjacency matrix, a nonlinear transformation is performed to obtain a sample space aggregation result; the sample space aggregation result is used to pass through the feedforward neural network to obtain the omics embedding representation.

7. The multimodal survival prediction method integrating digital pathological images and omics data according to claim 1, characterized in that: S3 includes: The pathology embedding representation is used as a query, and the omics embedding representation is copied, and the two omics embedding representations are used as a key and a value respectively; Based on the query, the key, and the value, the multi-head attention mechanism of Transformer is used for fusion to obtain the common attention, and the calculation formula is: Among them, CoA(·) represents shared attention; G bag Represents the omics embedding representation; H bag represents the pathological embedding representation; softmax(·) represents the softmax activation function; Q represents the query; K represents the key; d k Represents the vector dimension of the query; W q Represents the weight matrix of the query; W k represents the weight matrix of the key; W v represents the weight matrix of the value; T represents the transpose; Based on the co-attention, the weight matrix of the value and the omics embedding representation, the preliminary fusion feature is obtained, and the calculation formula is: in, Represents the initial fusion feature; A coa represents the co-attention matrix.

8. The multimodal survival prediction method integrating digital pathological images and omics data according to claim 1, characterized in that S4 include: Passing the pathology embedding representation and the preliminary fusion feature through a second fully connected layer respectively to obtain a pathology feature vector and a fusion feature vector respectively; Multiplying the pathological feature vector and the fusion feature vector by dot product to obtain pathological internal features and fusion internal features respectively; Bilinearly fusing the pathological feature vector with the fused feature vector to obtain a shared mutual feature; The pathological internal features, the fused internal features, and the mutual features are all subjected to a reshape operation and a third fully connected layer to obtain a reshaped pathological feature vector, a reshaped fused feature vector, and a reshaped mutual feature, respectively; The pathological feature vector, the reshaped pathological feature vector, the reshaped mutual feature, the reshaped fusion feature vector, and the fusion feature vector are sequentially concatenated to obtain the final fusion feature.

Images