Cancer cell identification method based on multi-modal characteristics

Abstract

The invention belongs to the field of pathological image classification, and particularly relates to a cancer cell recognition method based on multi-modal features, which comprises the following steps: acquiring pathological image data to be recognized; acquiring pathological image slice data according to the pathological image data, and performing feature extraction on the pathological image slice data to obtain image features; acquiring transcriptomic gene data of the pathological image according to the pathological image data, and performing feature extraction on the transcriptomic gene data to obtain transcriptomic data features; inputting the image features and the transcriptomics data features into a trained cancer cell recognition model to obtain a cancer cell recognition result; according to the method, the semantic difference between modals is effectively overcome, and the problem of semantic difference between a pathological image and transcriptomics data can be effectively solved through a multi-modal learning framework, especially in late fusion.

Description

Technical Field

[0001] The present invention belongs to the field of pathological image classification, and specifically relates to a cancer cell recognition method based on multimodal features. Background Art

[0002] With the rapid development of computer vision and deep learning technologies, medical image analysis has become an important application in the field of artificial intelligence, especially in disease diagnosis and survival prediction tasks. Traditional medical image analysis methods typically rely on models such as convolutional neural networks (CNNs) to analyze a single modality (e.g., image data only or clinical data only). However, single-modality methods are often limited by insufficient data or information. Therefore, in recent years, more and more research has begun to focus on multimodal learning, which combines data from different sources (such as pathological images, transcriptomics data, and clinical data) to improve the predictive accuracy and robustness of the model. To this end, more and more research has begun to focus on multimodal data fusion strategies, which combine data from different sources (such as pathological images, transcriptomics data, and clinical data) to improve the accuracy and reliability of predictions.

[0003] Multimodal learning techniques aim to leverage the complementarity of data from different modalities to improve model performance in biomedical tasks. To integrate information from different modalities, current multimodal learning methods primarily include two strategies: early fusion and late fusion. Late fusion combines features from each modality at the final stage of the network. Specifically, each modality first passes through a separate feature extraction network to obtain a unique feature representation. Finally, these features are concatenated, weighted, or otherwise combined. By performing fusion after features from each modality have been learned and extracted, this approach effectively avoids interference between modalities during feature extraction. Although late fusion methods sometimes fail to capture the fine-grained interaction information between modalities as well as early fusion, they demonstrate significant advantages in handling high-dimensional data, simplifying model training, and maintaining the independence of each modality. By performing fusion after feature extraction, late fusion effectively avoids interference between modalities during information extraction, enabling each modality to independently perform deep learning on its data.

[0004] In multimodal medical data analysis, late fusion provides flexibility in processing data from different modalities, especially when the data differ significantly. For example, pathological images and transcriptomics data have different feature spaces and representations. Late fusion can overcome the semantic differences between the modalities by first learning the features of each modality independently and then combining them. This is very beneficial for tasks such as disease diagnosis and survival prediction.

[0005] However, existing late-stage fusion methods still face several challenges, especially when there are significant semantic differences between modalities. Differences in data representation and feature spaces between modalities complicate their fusion, potentially making it difficult for the model to capture the deep connections between modalities. Furthermore, as data size increases, the computational complexity and memory consumption of the model also increase, placing higher demands on training efficient and accurate multimodal models. Summary of the Invention

[0006] To address the problems existing in the above-mentioned prior art, the present invention proposes a cancer cell recognition method based on multimodal features, which includes: obtaining pathological image data to be recognized; obtaining pathological image slice data based on the pathological image data, and performing feature extraction on the pathological image slice data to obtain image features; obtaining pathological image transcriptomic gene data based on the pathological image data, and performing feature extraction on the transcriptomic gene data to obtain transcriptomic data features; and inputting the image features and transcriptomic data features into a trained cancer cell recognition model to obtain cancer cell recognition results.

[0007] Beneficial effects of the present invention:

[0008] 1. The present invention effectively overcomes the semantic differences between modalities. The multimodal learning framework of the present invention, especially in late fusion, can effectively solve the problem of semantic differences between pathological images and transcriptomics data. By introducing contrast loss, hard negative sample mining loss and progressive feature fusion loss, the present invention can retain the independence of each modality during the fusion process and effectively capture fine-grained interactions between modalities, thereby improving the accuracy and robustness of the prediction. Compared with traditional late fusion methods, the present invention can perform more detailed feature fusion between different modalities, effectively improving the prediction performance, especially when faced with data with large semantic differences between modalities.

[0009] 2. The present invention uses the Swin-Transformer and S-MLP structures, where Swin-Transformer provides efficient image feature extraction capabilities and can capture spatial information in pathological images through a local windowed attention mechanism, while S-MLP effectively reduces computational complexity through sparse connections, making the processing of large-scale genetic data more efficient. By combining Swin-Transformer and S-MLP, the present invention can significantly improve computational efficiency when processing large-scale, high-dimensional medical data, avoiding the computational bottlenecks in traditional methods, thereby accelerating the model training and prediction process.

[0010] 3. This invention introduces a progressive feature fusion loss, which enables the model to gradually fuse data features from different modalities rather than fusing them all at once. This strategy avoids information loss, effectively improves the complementarity of multimodal features, and enhances the model's ability to understand fine-grained interactions between modalities. Compared with traditional fusion methods such as concatenation or weighted summation, the present invention's progressive feature fusion method can more meticulously handle the relationships between features, thereby improving the accuracy of survival prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is the overall flow chart of the present invention;

[0012] Figure 2 This is a model structure diagram of the present invention. DETAILED DESCRIPTION

[0013] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0014] A cancer cell recognition method based on multimodal features, such as Figure 1 and Figure 2 As shown, the method includes: obtaining pathological image data to be identified; obtaining pathological image slice data based on the pathological image data, and performing feature extraction on the pathological image slice data to obtain image features; obtaining pathological image transcriptomic gene data based on the pathological image data, and performing feature extraction on the transcriptomic gene data to obtain transcriptomic data features; inputting the image features and transcriptomic data features into a trained cancer cell recognition model to obtain cancer cell recognition results.

[0015] In multimodal learning, data representations of different modalities often differ significantly. For example, pathological images and transcriptomics data each have unique feature spaces: pathological images are typically high-dimensional pixel data containing rich spatial information, while transcriptomics data typically represents gene expression levels, reflecting molecular-level biological information. This semantic difference between modalities makes it difficult to directly and efficiently fuse them.

[0016] Transcriptomics data involves expression information for thousands of genes, while pathological images also contain numerous patches or blocks of pixels. Efficiently extracting useful features from this high-dimensional data and processing it is a challenge. Traditional neural networks (such as fully connected layers) are prone to high computational complexity and long training times when processing this type of data. This is especially true when dealing with large amounts of genetic data, where computational complexity increases significantly.

[0017] This paper uses the Swin-Transformer and S-MLP (sparse multi-layer perceptron) structures to efficiently extract features and reduce redundant calculations while maintaining computational efficiency when processing pathological images and transcriptomics data. The Swin-Transformer effectively captures local features through a local windowed attention mechanism, while the S-MLP improves parameter efficiency through sparse connections, enabling the model to reduce computational effort and improve training efficiency when processing large-scale data.

[0018] This deep learning framework is used for multimodal fusion of pathology images and transcriptomics data, aiming to combine these two types of information for survival prediction. The framework utilizes a pathology image feature extractor pre-trained on a large-scale Swim-Transformer model. It splits Whole Slide Images (WSIs) into multiple instances and extracts embedded features from these instances. The framework also performs feature extraction on transcriptomics data and processes gene expression through multiple channels, ultimately enabling multi-instance learning and survival prediction.

[0019] In this embodiment, feature extraction of the image includes: inputting the pathological image data (WSI) into Swim-Transformer, splitting the WSI into K instances (patches) through the feature extractor pre-trained by Swim-Transformer, each instance has N-dimensional features, and the embedded features of each instance are represented as X wsi ∈R K×N , where K is the number of instances and N is the feature dimension of each instance.

[0020] X wsi =Swim-Transformer (WSI)

[0021] The Swim-Transformer divides the image into fixed-size patches. Each patch is then mapped to a D-dimensional vector through a linear embedding layer, and a positional encoding is added. The sliding window partitioning reduces computational complexity. The attention within each window is calculated as follows:

[0022]

[0023] Where Q, K, and V are the query, key, and value, respectively, derived from the input embedding via linear transformations, and d is the dimension of the embedding. In the cross-window attention mechanism, the representation of the local window is shifted to allow information to propagate across windows. Within each layer, the Swin-Transformer aggregates information through cross-window operations. By performing parallel computations across multiple heads, the Swin-Transformer's self-attention mechanism can learn different attention patterns.

[0024] In this embodiment, feature extraction of transcriptomics gene data includes grouping the transcriptomics data according to pathways, performing feature extraction on each gene channel through S-MLP, and obtaining transcriptomics data features.

[0025] Specifically, the transcriptomics data (including NG gene expression information) is input into S-MLP, the transcriptomics data is grouped by pathway, and the gene set of each pathway is extracted through S-MLP (sparse attention model). The transcriptomics feature matrix X is output. rna ∈R NP×d . Extract features from each gene channel and generate X rna ∈R NP×d where NP is the number of paths and d is the token dimension.

[0026] X rna =S-MLP(RNA)

[0027] S-MLP is an improved multi-layer perceptron (MLP) that combines the idea of ​​sparse links, enabling the network to learn more efficiently when processing large amounts of genetic data.

[0028] In this embodiment, the multi-instance learning framework and the contrast feature fusion framework include:

[0029] Input: X wsi ∈R K×N (from pathology images) and X rna ∈R NP×d (from transcriptomics data).

[0030] The multi-instance learning framework is used to extract the package-level representations of pathological image instances and transcriptomics instances respectively, and obtain the global representation of pathological images. and global representation of transcriptomics Bag-level representations from different modalities are compared while controlling for differences in representation between them. Specifically, the optimization goal is to increase the similarity between representations from different modalities within the same category while reducing differences between representations from different categories. Through this comparative learning, the model is able to effectively overcome the semantic gap between features from different modalities. To achieve this goal, two memory banks are designed, storing bag-level representations from different categories. Representations belonging to the same category but from different modalities are stored in the same memory bank.

[0031] In the multi-stream structure, survival prediction loss (for classification tasks), contrast loss (for learning representation similarity), hard negative sample mining loss (for optimizing model robustness) and progressive feature fusion loss (for gradually fusing multimodal features) are used for optimization. The survival prediction probability is obtained through the feature fusion module IFAM and the multi-layer perceptron structure (MLP):

[0032]

[0033] In this embodiment, the survival prediction includes: inputting the survival prediction probability into a prediction model to perform survival prediction, and outputting the predicted survival time.

[0034] Survival status and event time definition:

[0035] Survival status c: c = 0 indicates the patient has died, and c = 1 indicates the patient's last follow-up.

[0036] Event time t i : represents the time from diagnosis to death of the patient (if c=0), or the time of the last follow-up (if c=1).

[0037] Prediction framework: This invention does not directly predict the specific time t of an event, but converts it into a time interval (t j-1 ,t j ), where j∈[1,…,n] represents non-overlapping time intervals of survival time. The model outputs the probability of death in each interval for each patient. The probability of death in all time intervals is calculated using the following formula:

[0038]

[0039] in, is the patient's re-time period (t j-1 ,t j ) probability of death, is the corresponding predicted value output by the model.

[0040] In this embodiment, a negative log-likelihood loss function (NLL) is defined. This loss function is used to deal with problems containing survival data. That is, the present invention not only considers patients who have died, but also considers those patients who have not died (for example, patients who are still alive and have not reached the last follow-up time).

[0041] Specifically, the loss function is defined as follows:

[0042]

[0043] Where ND is the number of samples in the dataset. (i) is the survival marker of the i-th sample, where 0 indicates that the patient has died and 1 indicates that the patient is still alive. Represents the multimodal embedding vector of the i-th sample. Indicates the time interval corresponding to the i-th sample. surv represents the survival function, which indicates the probability that the patient will survive in a certain time interval. hazard Represents the hazard function, which indicates the probability that the patient will die within a certain period of time.

[0044]

[0045] Where S is the sigmoid activation function.

[0046]

[0047] For patients who died (c = 0), the model is required to predict the probability of the patient surviving before the time of death (through the survival function f surv ) and the correct interval of death time (through the hazard function f hazard ).

[0048] For patients who survive (c=1), the model does not need to predict their death time, but rather their survival probability before the last follow-up time. To achieve this, the survival probability of the previous time interval (f surv (y j-1 )) to force the model to ensure a high probability of survival before the patient's last follow-up time point.

[0049] If c = 0, it means the patient has died, and we hope that the model will predict a higher probability of survival for the patient before this time interval.

[0050] If c = 1, it means the patient is still alive, and we hope that the model predicts that the patient has a higher probability of survival in the previous time interval.

[0051] If the patient is alive, we also want the model to correctly predict the probability of their death. If the patient does not die in the current time interval, we need to ensure that the probability of death in this time interval is low.

[0052] The goal of this loss function is to minimize the model's prediction error across all patients. By minimizing this loss function, the model can learn how to predict a patient's risk of death at different time intervals based on multimodal data, thereby improving the accuracy of survival predictions.

[0053] The goal of the contrastive loss is to optimize the similarity between features at different scales, making the feature vector distances between similar samples smaller and the distances between unrelated samples larger. This loss function is particularly important in multi-instance learning, as it helps the model learn how to distinguish between samples of different categories.

[0054]

[0055] Among them, z h and z l Corresponding to features from different modalities and

[0056] The goal of the hard negative loss is to improve the model's ability to learn from samples that are difficult to distinguish. Typically, in contrastive learning, some negative samples are easily recognized by the model due to their large difference from positive samples. However, difficult-to-distinguish negative samples (hard negative samples) are crucial for model training. Therefore, we designed a hard negative loss function specifically to handle these difficult-to-distinguish negative samples:

[0057]

[0058] In the task of multimodal feature fusion, features from different modalities provide rich image information. Multi-level features, ranging from detailed to global, can significantly enhance the model's representational capabilities. However, effectively fusing these features, particularly the weighting and aggregation of features from different modalities, remains a key challenge. The iterative feature fusion module addresses this challenge by gradually optimizing the fusion performance through multiple iterations. Combined with a progressive loss function, it can gradually optimize the fusion of features at each layer during model training, ensuring optimal alignment between the feature outputs of each layer and the final target features. The iterative feature fusion module employs a multi-round weighted approach to fuse features from different modalities. In each iteration, features from different modalities are initially fused through a weighted summation. Then, a structure comprising multiple fully connected networks is used to calculate weighting coefficients for each layer, achieving more accurate feature fusion. This process, through multiple iterations, allows for the gradual optimization of the weighting of each layer, ensuring that the final fused feature better reflects information correlations across different scales. Specifically, the design of the loss function is particularly important in multimodal information fusion, especially when using the iterative feature fusion module. The loss function not only needs to consider the difference between each layer’s features and the target features, but also needs to design a progressive training process to gradually optimize the weighted process of the target features and each layer’s features. The iterative feature fusion loss function can be expressed as:

[0059]

[0060] Where Fi(t) represents the fusion result of the i-th layer features after iteration t, Ftarget(t) is the target feature at iteration t, representing the ideal multi-scale fusion feature. λ(t) is the progressive weighting factor, representing the weighting coefficient at iteration t, which gradually increases as training progresses.

[0061] In this example, the model was evaluated using two methods: (1) the concordance index (c-index, higher is better), which measures the proportion of pairs of observations for which the model's predictions correctly predict the actual survival ranking (ranging from 0.5 (random prediction) to 1.0 (perfect prediction)); and (2) the Kaplan-Meier (KM) curve, which visualizes the survival probability of patients in different risk groups over a certain period of time. The log-rank statistical significance test was applied to determine whether the separation between the low-risk group and the high-risk group was statistically significant (p value < 0.05).

[0062] The consistency index (C-index) is used to evaluate the accuracy of the model's survival ranking. It measures whether the model's predicted survival ranking is consistent with the actual survival ranking for all possible patient pairs. Its formula is as follows:

[0063]

[0064] Concordant pair: If the survival time predicted by the model is consistent with the actual survival time, that is, for two patients, if the predicted survival time of the first patient is earlier than that of the second patient, and the actual survival time of the first patient is also earlier than that of the second patient, then the pair is considered "concordant".

[0065] Comparable pair: refers to a pair whose survival times can be compared. For example, the survival times of both patients are not censored (i.e., there is an observed death time).

[0066] The C-index value ranges from 0.5 (random prediction) to 1.0 (perfect prediction).

[0067] The Kaplan-Meier curve is used to estimate the probability of survival at different time points and can be used to compare survival differences between different risk groups. The calculation formula of the survival function is as follows:

[0068]

[0069] Where S(t) represents the probability of survival at time t, Di represents the number of deaths at time ti, and ni represents the number of survivors at time ti.

[0070] The Kaplan-Meier curve can show the survival status of different risk groups by calculating the survival probability at each time point and drawing the survival curve.

[0071] For comparison of risk groups, the Log-rank test is usually used to test whether the survival curves of the two groups are statistically significant. The formula of the Log-rank test is:

[0072]

[0073] Among them, O1 and O2 are the number of deaths in the two groups during the observation period; E1 and E2 are the number of deaths calculated according to the expected survival model; V1 and V2 are the variances of the number of deaths in the two groups.

[0074] The results of the Log-rank test (p-value) can be used to determine whether the survival difference between the two groups is statistically significant.

[0075] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for identifying cancer cells based on multimodal features, characterized in that: include: Acquire pathological image data to be identified; acquire pathological image slice data based on the pathological image data, and perform feature extraction on the pathological image slice data to obtain image features; acquire pathological image transcriptomic gene data based on the pathological image data, and perform feature extraction on the transcriptomic gene data to obtain transcriptomic data features; input the image features and transcriptomic data features into the trained cancer cell recognition model to obtain cancer cell recognition results.

2. The method for identifying cancer cells based on multimodal features according to claim 1, characterized in that: Feature extraction of pathological image slice data includes: inputting the pathological learning image slice data into Swim-Transformer for segmentation to obtain fixed-size blocks; mapping each block into a D-dimensional vector through a linear embedding layer, and adding position encoding to the D-dimensional vector; using sliding window block segmentation to perform attention calculation on the D-dimensional vector with position encoding to obtain image features.

3. The method for identifying cancer cells based on multimodal features according to claim 2, characterized in that: The attention calculation in the sliding window block is: Where Q, K, V are the query, key, and value, d is the dimension of the embedding, and T is the transpose.

4. The method for identifying cancer cells based on multimodal features according to claim 1, characterized in that: Feature extraction of transcriptomics gene data includes: grouping transcriptomics data according to pathways, extracting features of each gene channel through S-MLP, and obtaining transcriptomics data features.

5. The method for identifying cancer cells based on multimodal features according to claim 1, wherein: Training a cancer cell recognition model involves: S101, obtaining an original dataset, which includes pathological images and autotranscriptomics data; S102: Input the pathological image and autotranscriptomics data into the multi-instance framework backbone to extract the package-level representation and obtain the global representation of the pathological image. and global representation of transcriptomics S103. Global representation of pathological images and global representation of transcriptomics Perform multimodal feature fusion, use cross-modal contrast learning and progressive fusion attention mechanism to perform multiple rounds of cross fusion and differential supervision training, and obtain a unified fusion representation F containing two modal information target ; S104, the fused feature is represented as F target The data is input into the survival prediction network, and the risk function based on time segmentation is used to output the patient's risk score in each time period; S105. Construct a joint loss function based on the risk score, where the loss function includes a negative log-likelihood loss function, a contrastive learning loss, and a feature consistency loss, and jointly optimize the model parameters to improve the discriminability and predictive performance of the fused features; S106. Perform model training on the training set and evaluate model performance on the validation set using indicators such as C-index, consistency ranking, Kaplan-Meier curve, and log-rank test; S107. After training is completed, the final model is saved and applied to new pathological images and transcriptomics samples to achieve cancer cell identification and patient survival prediction.

6. The method for identifying cancer cells based on multimodal features according to claim 5, characterized in that: The backbone of the multi-instance framework extracts features from input data, including: Step 1: Slice the pathological image and pre-process it. The whole pathological image is divided into multiple patches. The local feature vector of each patch is extracted through the image coding network. The attention mechanism or graph attention mechanism is used to weightedly aggregate all local features to obtain the global representation of the pathological image. Step 2: Normalize the autotranscriptomics data and input it into the transcriptome feature extraction network. Use the fully connected neural network and graph neural network to encode the high-dimensional gene expression vector to obtain the global feature representation of the transcriptome data.

7. The method for identifying cancer cells based on multimodal features according to claim 5, characterized in that: Global representation of pathological images and global representation of transcriptomics The multimodal feature fusion includes: using feature alignment mechanism and cross-modal attention mechanism to align global features from different modalities; introducing modality weight adaptive control mechanism to dynamically adjust the fusion ratio of pathological images and transcriptomics modalities, and according to the adjusted fusion ratio, achieving the complementarity and reinforcement of image modality and gene modality in feature space through multi-layer fusion modules to construct a unified fusion representation F target .

8. The method for identifying cancer cells based on multimodal features according to claim 5, characterized in that: The loss function of the model is: Where ND is the number of samples in the dataset; c (i) is the survival marker of the i-th sample; Represents the multimodal embedding vector of the i-th sample; represents the time interval corresponding to the i-th sample; f surv represents the survival function; f hazard represents the hazard function.

Images