Sparse gating model using generic cancer genome information and pre-training method

Abstract

The invention belongs to the technical field of cancer classification pre-training models, and particularly relates to a sparse gating model and a pre-training method using panthenic cancer genome information. The Gene-MOE model further deepens the ability of extracting the deep correlation features of the high-dimensional genes by using a mixed expert layer, and explores the deep semantic association between the high-dimensional genetic features by using a mixed attention expert layer. And pre-training strategies including a data enhancement strategy, a joint training strategy combining a discriminant network and the Gene-MOE, a Gene-MOE feature distribution training strategy, a gene similarity training strategy, an expert utilization rate balance training strategy and total training loss calculation are utilized to further improve the performance of the Gene-MOE. The result shows that the Gene-MOE can obtain better performance in the aspects of cancer classification and survival analysis, and shows that the Gene-MOE can be used for classifying cancer types with larger sample sizes.

Description

Technical Field

[0001] The present invention belongs to the technical field of cancer classification pre-training models, and in particular relates to a sparse gating model and a pre-training method using pan-cancer genome information. Background Art

[0002] Currently, there is an urgent need to identify specific cancer types and perform accurate survival analysis to help personalize cancer treatment and accurately select drug therapy. The advent of the Human Genome Project has made high-throughput gene expression data available. Many genomic analysis techniques for large amounts of gene expression data have been developed, such as survival analysis, cancer classification, and co-expression analysis, which have been shown to be very useful for cancer diagnosis and prognosis. Deep learning-based algorithms have the ability to learn nonlinear functions from high-dimensional genes and are widely used in genomic diagnosis and prognosis tasks, such as the Cox-nnet model, DeepSurv model, VAECox model, DeepCC model, DeepCues model, and prognosis and classification methods using graph convolution.

[0003] Deep learning-based genomic diagnosis and prognosis methods are arguably one of the most successful areas applied to computational biology and bioinformatics, and are pioneers in end-to-end features and powerful fitting capabilities. In the past few years, pre-trained Transformer models have received widespread attention for their powerful feature learning capabilities. Currently, Transformer-based models are also becoming more and more diverse, and the number of parameters has increased dramatically due to the increase in computing power. Model features have become diverse due to sparse models and different self-attention methods. Some notable models are dense Transformer models and sparse models based on mixture of experts (MOE) models. The dense Transformer model introduces a self-attention mechanism that can learn the global semantic association of features during training. Compared with fully connected layers, the introduction of a self-attention mechanism enables the model to prioritize learning relevant features, thereby improving the accuracy of predictions. The sparse Transformer replaces the fully connected layers with a MOE module with multiple expert models. During training, the top K experts are selected to learn the same input features. Compared with the fully connected layer module, the introduction of MOE expands the parameter space of the entire model, enabling the model to learn richer feature representations. To date, many pre-trained models have been introduced to solve biological problems and have achieved remarkable results. For example, the BioBERT model, DNABERT model, and scBERT model. These methods transfer the self-attention mechanism from the Transformer model and MOE model to gene expression data analysis, thereby learning the significant correlation features of genes and achieving remarkable results in downstream tasks such as cancer classification and disease prediction.

[0004] However, there are still problems in applying pre-trained Transformer models to high-dimensional genetic data. First, the number of genes far exceeds the number of patient samples, which exacerbates the risk of overfitting during training. Second, most genes do not contain useful information. Appropriate data preprocessing and enhancement methods must be designed to avoid meaningless information as much as possible. Third, there are a large number of genes with similar expression patterns in the human genome, which are functionally relevant and reproducible. It can be seen that extracting such co-expression features from high-dimensional genetics requires special consideration to design appropriate methods suitable for genomic data. Summary of the invention

[0005] The present invention proposes a sparse gating model and pre-training method using pan-cancer genomic information. The model combines the principles of the MOE structure to create a pre-trained diagnosis and prognosis model named Gene-MOE for high-dimensional RNA-seq gene expression data. The model combines the features of MOE and the proposed mixture of attention experts (MOAE) to learn deep correlation features of high-dimensional genes. The Gene-MOE model achieves good performance in both diagnosis and prognosis, solving the above problems.

[0006] The technical solution of the present invention is achieved in this way:

[0007] A sparse gating model using pan-cancer genome information is named Gene-MOE. Gene-MOE includes a feature encoder and a decoder, wherein the encoder includes a mixed expert layer and a mixed attention expert layer, and the decoder is used to convert high-dimensional input gene features into low-dimensional feature encodings; the decoder includes a mixed expert layer, and the decoder is used to learn the reconstruction of low-dimensional feature encodings into high-dimensional gene features.

[0008] Optionally, the hybrid expert layer includes N experts, each of which independently learns and extracts features based on the characteristics of the input data. Given an input feature x∈R 1×N , the mixed expert layer with E experts is expressed as:

[0009]

[0010] Among them, y represents the output of the mixed expert layer, D i (x) is the output after inputting x into the i-th expert network, which is an independent fully connected layer, G(x) i is a sparse gating network used to predict the selection probability of the i-th expert. When G(x) i =1, the i-th expert is calculated. When G(x)i=0, the i-th expert is not selected for calculation. G(x) is expressed as:

[0011] G(x)=Softmax(TopK(H(x),k));

[0012] Among them, TopK(H(x), k) represents a discrete function that maps the input feature H(x) to the mask m∈R 1×N , which is expressed as:

[0013]

[0014] Among them, the values ​​of elements in H(x) corresponding to those outside the first k range are set to -∞. When the mask is input to the softmax function, the -∞ elements are set to 0. The input H(x) represents the dense layer in the noise gating model. This layer learns how to select experts during training. The corresponding learned i-th expert feature H(x)i is expressed as:

[0015] H(x) i =(x·W) i +z Softplus((x·W noise ) i );

[0016] Among them, z~N(0,1) is random Gaussian noise, W is the trainable weight matrix for learning sparse gated features, and W noise It is the weight that controls the noise increment. Different from the ordinary fully connected layer, H(x) i Contains trainable Gaussian noise to facilitate model load balancing during distributed training.

[0017] Optionally, the hybrid attention expert layer combines the attention mechanism with the hybrid expert layer, so that the Gene-MOE model learns rich relevant features from massive data sets.

[0018] Optionally, the Gene-MOE model is pre-trained, and the pre-training content includes data augmentation strategy, combined discriminant network and Gene-MOE joint training strategy, Gene-MOE feature distribution training strategy, gene similarity training strategy, expert utilization balance training strategy and calculation of overall training loss.

[0019] Optionally, the data enhancement strategy refers to constructing a data enhancement strategy by introducing Gaussian noise. For the input gene x, the enhanced input x is expressed as:

[0020] x=x+z;

[0021] Among them, z~N(0,σ 2 ), variance σ 2 Search via hyperparameter optimization.

[0022] Optionally, the strategy of combining the discriminant network with Gene-MOE joint training refers to constructing a discriminant network named D, where D is a fully connected layer network, the Gene-MOE model is the generator, and the Wasserstein loss is introduced to jointly train G and D, where G refers to the Gene-MOE model. For G and D, the loss is expressed as:

[0023]

[0024] Among them, λ gp is the hyperparameter of the gradient penalty, X is x and The formula enables the Gene-MOE model to learn how to perform dimensionality reduction and reconstruct the generated genes.

[0025] Optionally, the Gene-MOE feature distribution training strategy refers to: introducing KL divergence to measure The similarity of the generated latent code z to a standard Gaussian distribution;

[0026]

[0027] Among them, n represents the dimension of z, L KL The loss function is used to optimize the feature encoding z distribution of the input gene to maintain a standard normal distribution, optimize the training effect of the model, and simplify the training difficulty of Wasserstein loss.

[0028] Optionally, the gene similarity training strategy refers to: introducing L1 loss to measure the similarity of each gene between the sample reconstructed by Gene-MOE and the real sample;

[0029]

[0030] Optionally, the expert utilization balance training strategy refers to: in order to make each hybrid expert layer select each expert in a balanced manner, an importance loss is introduced for each sparse gating layer of the hybrid expert layer, and the importance loss can be expressed as:

[0031] L importance =||Importance(x k )|| 2 ;

[0032] Among them, x k represents the input feature of the k-th mixed expert layer, Importance(x k ) is expressed as:

[0033] Importance(x k )=∑ {x∈X} G(x k );

[0034] Where X∈R B×N , represents the total samples in the batch size; the expert will receive unbalanced samples, resulting in memory and performance issues. In order to further improve the balanced load, a load balancing loss L is introduced load , which is expressed as:

[0035]

[0036] Among them, Φ is the CDF of the standard normal distribution, TopK′ means selecting H(x k ) , excluding itself.

[0037] Optionally, calculate the overall training loss as:

[0038] L total =L gan +λ KL ·L KL +λ l1 ·L L1 +λ balance ·(L importance +L load );

[0039] Among them, λ KL , KL , 11 and λ balance is a hyperparameter; by calculating the overall training loss, we can fit the optimal parameters of the mixed expert layer

[0040]

[0041] After adopting the above technical solution, the beneficial effects of the present invention are:

[0042] The present invention designs a sparse gating model named Gene-MOE, which makes extensive use of a mixed expert layer (MOE) to further deepen the ability to extract deep correlation features of high-dimensional genes. In addition, a new mixed attention expert layer (MOAE) is proposed to explore the deep semantic associations between high-dimensional genetic features. Finally, a novel pre-training strategy is designed, including a data enhancement strategy, a joint training strategy combining a discriminant network and Gene-MOE, a Gene-MOE feature distribution training strategy, a gene similarity training strategy, an expert utilization balance training strategy, and calculating the overall training loss to further improve the performance of Gene-MOE. The results show that Gene-MOE can achieve good performance in cancer classification and survival analysis, indicating that Gene-MOE has strong application potential in these applications, especially focusing on the classification of cancer types with large sample sizes. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 is a statistical analysis of the Pan-Cancer dataset of 33 cancer types in the embodiment, wherein: (a) unified manifold approximation projection dimensionality reduction and visualization of each patient in the Pan-Cancer dataset, (b) the number of patients of each cancer type, (c) histogram statistics of the average expression of each gene in the dataset, and (d) histogram statistics of the standard deviation of gene expression in the dataset;

[0045] Figure 2 is a framework diagram of Gene-MOE in the embodiment, wherein (a) is a pre-training stage, and (b) is a fine-tuning stage;

[0046] Figure 3 It is a hybrid framework of the attention expert layer (MOAE) in the embodiment;

[0047] Figure 4 is an overview of downstream analyses in the embodiments, including (a) survival analysis, (b) cancer classification analysis, and (c) cancer subtypes;

[0048] Figure 5 is a comparison of consistency indexes of survival analysis of 14 cancer types in the embodiment, and the “+” in each box represents the average consistency index;

[0049] Figure 6 is the classification performance of Gene-MOE for 33 cancer types in the embodiment;

[0050] Figure 7 is the confusion matrix constructed for 33 cancers in the embodiment;

[0051] Figure 8 is a feature analysis of Gene-MOE in the embodiment, wherein (a) a Pearson correlation heat map between leading features and patient genes, features learned by Gene-MOE are closely related to patient genes, (b) a Pearson correlation heat map by selecting mean absolute coefficients greater than 0.5, (c) t-distributed random neighbor embedding and uniform manifold approximation and projection results of true genes and Gene-MOE reconstructed genes;

[0052] Fig. 9: is a comparison of the F1 scores of the pre-trained Gene-MOE and Gene-MOE for 33 cancer types in the Example. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The embodiments of the present application disclose a sparse gating model and a pre-training method using pan-cancer genome information.

[0055] Example

[0056] according to Figures 1 to 9 As shown, a sparse gating model and pre-training method using pan-cancer genomic information.

[0057] 1. Gene-MOE Model

[0058] 1. Data preparation

[0059] The present invention mainly utilizes the pan-cancer RNA-seq database of the TCGA data set from the Pan-Cancer Atlas project, which consists of 33 cancer types. In addition, specific RNA-seq data sets for each TCGA cancer type were downloaded. UCSC Xena already contains preliminary processing of these 34 data sets, so the present invention directly uses the FPKM-UQ normalized RNA-seq data of UCSC Xena. In the initial data set, each patient was associated with 60,484 genes, most of which showed zero expression and were not related to cancer.

[0060] according to Figure 1 (a) and Figure 1 (b), it can be found that the number of patient samples for each cancer is unbalanced. At the same time, some cancer types show significant overlap, which indicates that there is a clear correlation between certain types of cancer in the Pan-Cancer dataset. The dataset was then further analyzed. Figure 1 (c) and Figure 1 (d) shows a histogram of the mean and standard deviation of 60,484 genes. It can be seen that the mean expression and standard deviation of many genes are low, which indicates that these genes do not contribute to cancer. In addition, Figure 1There is a valley at 0-0.4 in (d), which indicates that the standard deviation of typical genes is not within this range. Based on these obstacles, the present invention introduces a filtering scheme to remove these non-contributing genes. First, the empty genes are deleted, and overlapping genes are selected in 34 data sets. Then, genes with variance less than 0.4 and mean less than 0.8 are filtered out. Through preparation, the present invention selects 25182 genes for each data set. Finally, before feeding the data to the network, min-max normalization is performed.

[0061] 2. Framework of Gene-MOE

[0062] Figure 2 The framework of Gene-MOE is illustrated, which consists of two stages: pre-training and fine-tuning. In the pre-training stage, Gene-MOE takes pre-processed pan-cancer genes as input to train the encoder-decoder network. In the fine-tuning stage, Gene-MOE transfers the pre-trained backbone and connects to the classification head to achieve accurate downstream prediction tasks such as prognosis prediction, cancer classification, and specific pathways of interest.

[0063] In the first stage, the present invention constructs a Gene-MOE model to learn the low-dimensional feature encoding of high-dimensional pan-cancer genes. At this stage, the present invention adopts a self-supervised learning method, in which the training label is the input gene, aiming to obtain the low-dimensional feature encoding of high-dimensional pan-cancer genes. The Gene-MOE model is mainly composed of an encoder backbone network and a decoder network. In the encoder network, the present invention introduces a mixed expert layer (MOE) based on sparse gating. The mixed expert layer involves multiple experts, enabling the encoder to learn rich feature representations of genomic information. In addition, the present invention designs a mixed attention expert layer (MOAE), which uses multiple attention mechanisms as different experts and uses a learnable sparse gating mechanism to adaptively merge attention features. The Gene-MOE model is different from the Transformer model (e.g., Switch Transformers) in the prior art. The Gene-MOE model only uses MOE and the proposed MOAE to apply the effects of self-attention and MOE to the entire genome, aiming to promote the discovery of potentially related genes.

[0064] In the second stage, the backbone of Gene-MOE was transferred and a new classification head was built to complete fast and accurate downstream tasks after fine-tuning. This approach enables the present invention to fully utilize the Gene-MOE model in two stages, helps to learn rich pan-cancer gene feature encoding, and achieve excellent performance in multiple downstream tasks.

[0065] Gene-MOE includes a mixed expert layer (MOE) and a mixed attention expert layer (MOAE), the specific contents are as follows:

[0066] (1) Mixed Experts (MOE): Each patient has more than 20,000 genes, among which there are intricate correlations. In the prior art, one or more fully connected layers are usually constructed to learn and extract key gene-related features. However, directly using fully connected layers cannot effectively help the model learn these intricate correlations of high-dimensional gene inputs. The learned features will result in significant losses. Compared with fully connected layers, MOE trains E experts, each of which independently learns and extracts features based on the characteristics of the input data. Compared with fully connected layers, MOE can integrate multiple features from multiple expert models, enhancing the feature extraction capability of high-dimensional genes and the overall performance of the model. In addition, the training of each expert is independent, allowing adjustments based on the characteristics of the genetic data, thus providing greater flexibility compared to fully connected layers. Given an input feature x∈R 1×N , the MOE layer with E experts is expressed as:

[0067]

[0068] Among them, D i (x) is the output after feeding x to the i-th expert network, which is an independent fully connected layer, (x) i is the choice of the i-th expert predicted by the sparse gating network, when (x) i =1, calculate the i-th expert. When (x) i = 0, no computational expert is required. G(x) is expressed as:

[0069] G(x) = Softmax(TopK(H(x), k)); Formula (2)

[0070] Among them, TopK(H(x), k) represents a discrete function that maps the input feature H(x) to the mask m∈R 1×N . It is expressed as:

[0071]

[0072] Among them, the values ​​of H(x) corresponding to elements outside the first k range are set to -∞. When m is fed to the softmax function, the -∞ elements are set to 0. The input H(x) represents the fully connected layer in the noise gating model. This layer learns to select experts during training. It is expressed as:

[0073] H(x) i =(x·W) i +zSoftplus((x·W noise ) i ); Formula (4)

[0074] Among them, z~N(0,1) is random Gaussian noise, W is the trainable weight matrix for learning sparse gated features, and W noise is the weight that controls the noise increment. Unlike ordinary fully connected layers, H(x) contains trainable Gaussian noise to facilitate model load balancing during distributed training.

[0075] (2) Mixed Attention Expert Layer (MOAE): The human genome exhibits significant correlations. Often, mutations in a gene are associated with multiple types of cancer, and the occurrence of a specific cancer may also be associated with modifications in multiple genes. Identifying such strong correlations among 60,484 genes poses many challenges to model design. In recent years, attention mechanisms have shown significant effects in many fields, and compared with traditional fully connected layers, this approach can effectively learn deep correlations between features. Incorporating attention mechanisms into models can effectively discover related genes and further improve model performance. In addition, combining attention mechanisms with MOE can help models learn rich relevant features from massive datasets. Therefore, we propose a mixed attention expert layer (MOAE) based on the MOE mechanism, replacing the fully connected layer expert with a residual self-attention network. Figure 3 It is explained that the purpose of using residuals in MOAE is to reduce the model degradation problem and improve the training effect. Similar to MOE, the present invention uses the same discrete function in formula (2) to select the first k self-attention models and merge features.

[0076] 3. Pre-training strategy

[0077] The present invention constructs a self-supervised pre-training strategy to learn the common features of pan-cancer genes and improve the feature extraction performance of the backbone network.

[0078] (1) Data augmentation strategy: The Pan-cancer dataset has 10,000 patients after preprocessing. If 500 million parameters are used, Gene-MOE may cause overfitting problems. Therefore, the present invention constructs a data augmentation strategy by introducing Gaussian noise. For the input gene x, the enhanced input It is expressed as:

[0079] x=x+z; Formula (5)

[0080] Among them, z~N(0,σ 2 ). Variance σ 2 The search is performed through hyperparameter optimization. In addition, a dropout strategy is introduced during the training process.

[0081] (2) Combined adversarial network and Gene-MOE joint training strategy: The present invention uses the training strategy of generative adversarial network (GAN) to pre-train Gene-MOE. The present invention constructs a discriminant network named D, which is a fully connected layer network, and the Gene-MOE model is the generator. Then, Wasserstein loss is introduced to jointly train G (Gene-MOE) and D (discriminator network composed of a fully connected network). For G and D, the loss is expressed as:

[0082]

[0083] where λgp is the hyperparameter of the gradient penalty, X is the value of x and The sample space of . Training with formula (6) can help Gene-MOE learn how to perform dimensionality reduction and reconstruct the generated genes.

[0084] (3) Gene-MOE feature distribution training strategy: The present invention further introduces KL divergence to measure The similarity of the generated latent code z to the standard Gaussian distribution. It can be expressed as:

[0085]

[0086] Where n represents the dimension of z. Using this loss can keep z in a standard normal distribution, thereby simplifying the training difficulty of Wasserstein loss.

[0087] (4) Gene similarity training strategy: The present invention introduces L1 loss to further measure the similarity of each gene between the samples reconstructed by Gene-MOE and the real samples. Its calculation method is:

[0088]

[0089] (5) Expert utilization balanced training strategy: In order to make each MOE layer select each expert in a balanced manner, the present invention introduces an importance loss for each sparse gating layer of MOE. The importance loss can be expressed as:

[0090] L importance =||Importance(x k )|| 2 ; Formula (9)

[0091] Among them, x k represents the input feature of the kth MOE layer, Importance(x k ) is expressed as:

[0092] Importance(x k )=∑{x∈X} G(x k ); Formula (10)

[0093] Where X∈R B×N , represents the total samples in the batch size. Formula (10) can ensure that each expert has the same importance, which makes the model balanced in assigning experts to training data. However, experts may receive unbalanced samples, which will cause memory and performance problems. In order to further improve the balanced load, the present invention also introduces the load balancing loss L load , which is expressed as:

[0094]

[0095] Among them, Φ is the cumulative distribution function of the standard normal distribution, and TopK′ represents the selection of H(x k ) , excluding itself.

[0096] (6) Calculate the overall training loss: Combining these losses, the overall loss of MOE can be expressed as:

[0097] L total =L gan +λ KL ·L KL +λ l1 ·L L1 +λ balance ·(L importance +L load ); formula (12)

[0098] Among them, λ KL , L KL , l1 and λ balance is a hyperparameter. Therefore, the pre-training phase aims to solve the optimal parameters for fitting MOE

[0099]

[0100] 4. COX survival analysis

[0101] The COX survival analysis method can help researchers apply further treatment and improve cancer prognosis. The present invention uses Gene-MOE to perform COX survival analysis on specific cancers. Figure 4 (a) shows the framework of the survival model. The present invention uses the pre-trained Gene-MOE skeleton as an encoder to extract useful features of high-dimensional genes. Subsequently, the gene features are input into a fully connected classifier to predict the patient's survival rate.

[0102] 5. Cancer classification

[0103] Accurate prediction of cancer based on gene expression helps to understand the disease mechanism and utilize customized treatment for specific cancer types.Then, the present invention selects cancer classification analysis as a downstream task to further evaluate the performance of Gene-MOE. Figure 4 (b) shows the framework of the classification model. Similar to the survival model, the present invention transfers the weights of the Gene-MOE skeleton to extract representative features of the gene. The features are then input into the fully connected layer classification head. The classification head of the classification model then predicts the probabilities of 33 different cancer types. The classification head is designed as a multi-class classification model that directly predicts the probabilities of 33 cancer types, and uses focal loss during training to alleviate the imbalance problem and improve model performance.

[0104] 6. Cancer subtypes

[0105] Cancer subtyping methods are crucial for cancer diagnosis, prognosis, and customized treatment. We selected this typical field for evaluation. We then transferred the encoder backbone and added a cancer subtype classification head to predict cancer subtypes through a feature decoupling contrast clustering learning method.

[0106] 7. Experimental Setup

[0107] Gene-MOE is implemented using the PyTorch framework. The present invention uses NVIDIA Tesla V100 (32GB) GPU to train and evaluate Gene-MOE. In the pre-training stage, the Gene-MOE model is initially trained on a standardized pan-cancer dataset including 33 cancer types. Then, the pan-cancer data is randomly divided into a training dataset and a test dataset in a ratio of 4:1, where the training dataset is used for training and the test dataset is used for hyperparameter search and feature analysis. The Adam optimizer is trained with Gene-MOE. For better convergence, a learning rate decay method is introduced, which sets the learning rate, which remains unchanged in the first half of the training rounds and linearly decays to 0 in the second half of the training rounds. In order to find a better hyperparameter setting, a random parameter search scheme is performed. The noise variance range is set to [0.2, 0.4, 0.6, 0.8], the epoch range is set to [100, 200, 300], the learning rate range is set to [0.002, 0.0002, 0.00002], the λkl range is set to [1, 10, 20], the λl1 range is set to [1, 10, 20], the λbalance range is set to [1, 10, 20], and the λgp range is set to [1, 10, 20]. Next, a hyperparameter search is performed and a noise variance of 0.2, 200 training epochs, a learning rate of 0.0002, a λkl of 10, a λl1 of 20, a λbalance of 10, and a λgp of 10 are obtained.

[0108] For downstream tasks, the present invention uses a Bayesian optimization strategy to select better hyperparameter settings. In the survival analysis stage, the present invention evaluates the survival model in the same way as Cox-nnet and VAE-Cox. The present invention trains specific Gene-MOE cancer types on 14 common TCGA datasets. For each dataset, the present invention uses the same method to randomly divide the training dataset and the test dataset in a ratio of 4:1. The better hyperparameters of each model are selected using a Bayesian optimization strategy. In addition, the present invention repeats the entire process 5 times and calculates the average result to avoid the bias of random splitting. In the cancer classification stage, due to the imbalance of samples for each cancer, directly dividing the pan-cancer data into training sets and test sets in a ratio of 4:1 will result in the loss of cancer types with a small number of samples in the test set. In response to the above problems, the present invention introduces a new partitioning method in the sample, divides each cancer into a training subset and a test subset in a ratio of 4:1, and then combines the training subsets and test subsets of 33 cancers to obtain the training set and test set for the classification task. At the cancer subtype stage, six cancer types were used for performance evaluation, namely BLCA, BRCA, KIRC, LUAD, STAD, and UCEC. Then, the corresponding model was trained 5 times and the average result was taken as the final result.

[0109] 8. Evaluation Metrics

[0110] In the survival analysis stage, the main evaluation method used is the consistency index, which is widely used in survival analysis models and ranges from 0 to 1. When the consistency index is ≤0.5, the model has completed an invalid survival analysis prediction. When the consistency index is ≥0.5 or higher, the model's prediction effect is better.

[0111] In the cancer classification stage, accuracy, precision, recall and F1 score are mainly used. Accuracy is expressed as:

[0112]

[0113] Among them, c represents the class number, TP i represents the true positive samples of class i, TN i Represents the true negative of class i, and Ni represents the total samples of class i. The accuracy index represents the ability of the classifier to correctly predict the accuracy of positive samples, which is expressed as:

[0114]

[0115] Among them, FP i Represents the false positive samples of class i. The recall rate reflects the ability of the classifier to correctly predict the fill degree of positive samples, which is expressed as:

[0116]

[0117] Among them, FN i represents the false negative samples of class i. Finally, the F1 score represents the harmonic mean of precision and recall, expressed as:

[0118]

[0119] In the cancer subtype stage, two evaluation criteria were used, namely, the -log10P value and the number of important clinical parameters, which were used in previous studies.

[0120] 2. Results Analysis

[0121] 1. Survival analysis performance

[0122] The present invention selects 14 representative cancer types and analyzes the performance of the model of the present invention and two advanced models Cox-nnet and VAE-Cox. Figure 5 The consistency indexes of 14 cancer types are shown. In order to better compare the consistency indexes of these models, the average consistency indexes of these models for 14 cancer types are listed in Table I. Compared with Cox-nnet and VAECox, Gene-MOE outperformed the advanced models on 13 cancer types with higher average consistency indexes, which indicates that Gene-MOE performed more accurate survival analysis compared with these two models. And it was found that the performance of Gene-MOE on the STAD dataset was not as good as some advanced models. By examining the STAD dataset, it was found that there was a significant imbalance between positive and negative samples, which indicates that Gene-MOE is not sufficient to learn from an imbalanced dataset.

[0123] 2. Performance of Cancer Type Classification

[0124] The present invention evaluates the precision, recall and F1 score of the classification model using Gene-MOE. The performance results are as follows Figure 6 As shown. Figure 6 As shown, Gene-MOE exhibited significant performance on 32 cancer types except rectal adenocarcinoma (READ), with an overall accuracy of 95.8%.

[0125] In order to demonstrate the novelty of the proposed classifier, the present invention selects several advanced classifiers for comparison, including classifier models based on machine learning methods, such as random forest and SVM, and classifier models based on deep learning, such as MLP and CNN.

[0126] Table I: Comparison of mean concordance index among 14 cancer types

[0127] BLCA BRCA HNSC KIRC LGG LIHC LUAD LUSC OV SARC CESC STAD COAD UCEC Ccx-nnet 0.643 0.692 0.632 0.728 0.847 0.691 0.649 0.558 0.590 0.722 0.720 0.607 0.674 0.699 VAE-cox 0.655 0.702 0.624 0.714 0.852 0.697 0.642 0.553 0.593 0.672 0.700 0.638 0.635 0.675 Gene-MOE 0.664 0.727 0.640 0.730 0.855 0.705 0.662 0.564 0.617 0.726 0.738 0.632 0.715 0.720

[0128] In order to ensure the fairness of the comparative experiment, the present invention performs five independent processes and calculates the average of the classification results to avoid bias. Table II shows the classification results of these five models. According to Table II, it is found that the classification results of the Gene-MOE model on the test data set are improved by 0.1-0.5 compared with the other four methods.

[0129] Table II: Comparison of classification indicators

[0130] Accuracy Accuracy Recall F1 score RandomForest 0.9010 0.8943 0.8338 0.8409 SVM 0.9498 0.9281 0.9114 0.9149 MLP 0.9250 0.8989 0.8756 0.8790 2D-Hybrid-CNN 0.9507 0.9300 0.9229 0.9232 Gene-MOE 0.9580 0.9554 0.9374 0.9333

[0131] The present invention further analyzes the classification performance by constructing a confusion matrix. Figure 7 The confusion matrix constructed for 33 cancer types is illustrated. The Gene-MOE model demonstrated accurate classification of all 33 cancer types, with very low misclassification rates according to the confusion matrix. In addition, it was observed that the Gene-MOE model misclassified 29 cases of READ as colon adenocarcinoma (COAD). This issue occurred because READ and COAD are genetically identical and the samples in the COAD sample are larger than READ, causing the model to misclassify some READ patients as COAD patients.

[0132] 3. Performance of cancer subtypes

[0133] The present invention compares the clustering performance of Gene-MOE with four advanced models: subtype DCC, subtype GAN, NEMO and SNF. The present invention selects 9 representative cancer types and analyzes the clustering performance of these models. According to relevant studies, these 9 representative cancer types are determined to be interpretable subtypes (i.e., BLCA has 5 subtypes, BRCA has 5 subtypes, KIRC has 4 subtypes, LUAD has 3 subtypes, STAD has 3 subtypes, UCEC has 4 subtypes, SKCM has 4 subtypes, UVM has 4 subtypes, and PAAD has 2 subtypes). Then, subtype identification and clustering are performed according to the known subtype numbers. The performance results are listed in Table III. The present invention uses -log10P values ​​and the number of important clinical parameters to evaluate the performance of cancer subtypes. Compared with other models, the Gene-MOE model achieved better results in at least one indicator in 7 data sets. According to Table III, the better results of the two indicators performed by Gene-MOE are achieved on the LUAD and STAD data sets. On the LUAD dataset, the survival-log10P value reached 2.07, and the number of significant clinical parameters was 5. On the STAD dataset, the survival-log10P value reached 1.58, and the number of significant clinical parameters was 3. These results indicate that the subtypes of lung adenocarcinoma and gastric adenocarcinoma are better compared with other advanced models.

[0134] Table III: Performance comparison of Gene-MOE and other advanced models on 9 datasets (-log10P value / number of important clinical parameters).

[0135] BLCA BRCA KIRC LUAD STAD UCEC UVM PAAD SKCM SNF 1.31 / 6 0.93 / 5 8.19 / 6 2.23 / 4 0.72 / 2 5 / 1 2.77 / 0 3.24 / 3 5.27 / 4 NEMO 2.8 / 5 1.21 / 6 5.72 / 5 2.63 / 4 1.8 / 2 5.96 / 1 2.38 / 0 3.04 / 1 5.01 / 4 Subtype-GAN 1.45 / 4 1.28 / 6 7.77 / 6 2.83 / 3 0.39 / 2 7.4 / 1 2.62 / 0 1.65 / 1 0.1 / 2 Subtype-DCC 2.35 / 6 0.11 / 5 8.97 / 6 1.69 / 4 1.48 / 2 5.46 / 1 2.77 / 0 3.75 / 1 5.94 / 4 Gene-MOE 2.21 / 5 1.33 / 5 11.99 / 5 2.07 / 5 1.85 / 3 4.08 / 1 2.6 / 2 2.3 / 3 1.81 / 3

[0136] 4. Characteristic analysis of gene MOE

[0137] (1) Correlation analysis: The present invention performs correlation analysis on the low-dimensional features extracted by the Gene-MOE model and the high-dimensional genes of the patient. Specifically, 1000 patient samples from the test data set are fed into the Gene-MOE model, which evaluates the low-dimensional features of each patient. Then, the top 20 features with the largest variance are selected as the main features, and their Pearson correlation with the original patient genetic information is calculated. It is not said that a normalization step is performed before selecting the main features to confirm that the features are in the same proportion. The results are as follows Figure 8 As shown in (a), it provides a heat map illustrating the correlation between these main features and the genes of 1000 patients, which clearly shows that there is a significant correlation between the low-dimensional features predicted by the Gene-MOE model and the original input features. Figure 8 (a), we further refined our analysis by identifying genes with an average absolute correlation of more than 0.4 with respect to the 20 main features. Then, we filtered the 29 strongly correlated genes, such as Figure 8 (b). Figure 8Results in (b) show that many genes closely associated with the main features are cancer-related genes. For example, the TMPRSS4 gene is an emerging potential therapeutic target for cancer. In addition, tensin4 expression shows prognostic relevance in gastric cancer. In addition, PRSS22 transcription initiated by E2F1 cleaves ANXA1 and activates the FPR2 / ERK signaling pathway. In addition, downregulation of FXYD3 expression was observed in lung cancer. In addition, ST14 gene expression affects breast cancer. In addition, RAB25 is also associated with various cancers, and it has been reported that it is both an oncogene and a tumor suppressor gene. Long intergenic noncoding RNA 00324 promotes gastric cancer cell proliferation by binding to HuR and stabilizing FAM83B expression. Studies have found that CBLC is expressed higher in breast cancer tissues and cells than in normal tissues and cells. In addition, Serpin B5 has been shown to be a CEA-interacting biomarker for colorectal cancer. The interaction between PKP3 and the MAPK-JNK-ERK1 / 2-mTOR pathway regulates autophagy and invasion in ovarian cancer. Furthermore, LAD1 expression is associated with the metastatic potential of colorectal cancer cells. Studies have found that ELF3 is a negative regulator of epithelial-mesenchymal transition in ovarian cancer cells. In addition, expression patterns of CEACAM5 and CEA-CAM6 were observed in primary and metastatic cancers. Grhl2 determines the epithelial phenotype of breast cancer and promotes tumor progression. KLF5 promotes proliferation, migration, and invasion of breast cancer, in part by upregulating the transcription of TNFAIP2. Finally, it was found that genetic susceptibility to colon and rectal adenocarcinoma is mediated by super enhancer polymorphisms that co-activate CD9 and PLEKHG6. According to Figure 8 (a) and Figure 8 From the analysis in (b), Gene-MOE can extract abundant cancer-related genes, and each feature is strongly correlated with a specific gene, which indicates that Gene-MOE can effectively learn the latent semantic correlation of genes.

[0138] (2) Visualization Analysis: The present invention further performs visualization analysis to measure the performance of Gene-MOE. 1000 patients were randomly selected from the test set for this evaluation. The true gene expression of the test patients was first fed into Gene-MOE to generate reconstructed expression. Then, TSNE and UMAP were used for visualization and to evaluate the similarity of the two distributions. Figure 8 (c) shows the visualization results using these two methods. Figure 8(c), the true gene distribution is completely consistent with the reconstructed gene distribution, which proves that the Gene-MOE model can reconstruct the input genes more accurately based on the input true genes. In addition, Gene-MOE completes the reconstruction based on the low-dimensional features obtained by the backbone model, which further strengthens that Gene-MOE can learn rich feature representations through the backbone network.

[0139] 5. Ablation study

[0140] The present invention extends the analysis by proposing ablation experiments to evaluate the performance of the model proposed in the present invention. To this end, four different models were constructed: 1) a baseline model containing two fully connected layers; 2) a model that integrates MOE by replacing the fully connected layer; 3) a Gene-MOE model without pre-training; 4) a Gene-MOE model with pre-training. Subsequently, these four models were used to perform survival analysis, cancer classification, and cancer subtyping tasks. The results of these tasks are shown in Tables IV, V, and VI.

[0141] Table IV: Comparison of the concordance index of 14 cancer types using 4 different models

[0142]

[0143] Table V: Performance comparison of four different models for nine cancer types (-log10P value / number of important clinical parameters)

[0144]

[0145]

[0146] Table VI: Comparison of classification metrics using four different models

[0147] Accuracy Accuracy Recall F1 score Baseline Model 0.9304 0.9235 0.8721 0.8799 MOE 0.9417 0.9214 0.9063 0.9098 MOAE 0.9584 0.9469 0.9349 0.9392 Gene-MOE after pre-training 0.9580 0.9554 0.9374 0.9333

[0148] (1) Performance of MOE: By comparing the survival analysis results in Table IV, it is found that the model with the MOE layer can achieve better consistency results than the baseline on 13 cancer types. By comparing the cancer subtype results in Table V, it is found that the model with the MOE layer outperforms the baseline on eight cancer types. In addition, in Table VI, the model with MOE shows better accuracy and F1 score. These results show that the use of the MOE layer can improve the performance of genomic analysis, which proves that the use of MOE can generate features rich in model learning during the feature extraction process.

[0149] (2) Performance of MOAE: By comparing the model consistency index of the models with MOE and Gene-MOE in Table IV, it is observed that Gene-MOE outperforms the model with MOE on nine cancer types. By comparing the cancer subtype analysis performance, it is observed that Gene-MOE outperforms the MOE model on six cancer types. In addition, in Table IV, it is found that Gene-MOE performs better in terms of accuracy, recall, precision, and F1 score. These findings demonstrate that the MOAE model can further improve the accuracy of the model by improving the ability to learn deep semantically relevant features.

[0150] (3) Performance of pre-training: By comparing the performance of Gene-MOE and pre-trained Gene-MOE models in Table IV, it was found that the pre-trained Gene-MOE model performed better in eight cancer types. At the same time, the Gene-MOE model showed the same performance on the KIRC and LGG datasets. By comparing the performance of Gene-MOE and pre-trained Gene-MOE in Table V, it was found that the pre-trained Gene-MOE model performed better in at least one indicator of log10P value and significant clinical quantity of six cancers. These results reveal the effectiveness of pre-training, especially in improving the performance of most survival analysis models. Further comparison of Gene-MOE and pre-trained Gene-MOE in Table VI shows that the pre-trained Gene-MOE model performs better in precision and recall, but the accuracy and F1 score are lower than Gene-MOE. By further comparison Fig. 9 By comparing the F1 scores of the two models, it is found that the pre-trained model performs worse for cancer types with fewer samples (such as the READ dataset). This difference can be attributed to the fact that the pre-training stage mainly captures common cancer features, and some features may be missing for cancer types with small sample sizes. Therefore, during the fine-tuning stage, the model tends to learn patterns from cancer types with larger sample sizes. The MOE module and MOAE module proposed in the present invention can further improve cancer classification, demonstrating the creativity and value of Gene-MOE.

[0151] The present invention proposes a sparse gated RNA-seq analysis framework named Gene-MOE. Gene-MOE uses MOE layers to extract features from high-dimensional RNA-seq genes. In addition, a self-attention mechanism is added to construct a MOAE model to further learn the deep semantic relationships within genetic features. The present invention uses a new self-supervised pre-training strategy to allow Gene-MOE to learn the common features of 33 types of cancer, and then transfers the pre-trained weights to specific analyses including survival analysis and cancer classification. According to the survival analysis results of 14 cancer types, Gene-MOE achieves a good consistency index on 12 cancer types. In addition, the classifier using pre-trained Gene-MOE achieved accurate classification of 33 cancer types with an overall accuracy of 95.8%.

[0152] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A sparse gating model using pan-cancer genomic information, characterized in that: The name of this model is Gene-MOE. Gene-MOE includes a feature encoder and a decoder, wherein the encoder includes a mixed expert layer and a mixed attention expert layer, and the decoder is used to convert high-dimensional input gene features into low-dimensional feature encodings; the decoder includes a mixed expert layer, and the decoder is used to learn the reconstruction of low-dimensional feature encodings to high-dimensional gene features.

2. The sparse gating model using pan-cancer genomic information according to claim 1, characterized in that: The hybrid expert layer includes N experts, each of whom independently learns and extracts features based on the characteristics of the input data. Given an input feature x∈R 1×N , the mixed expert layer with E experts is expressed as: Among them, y represents the output of the mixed expert layer, D i (x) is the output after inputting x into the i-th expert network, which is an independent fully connected layer, G(x) i is a sparse gating network used to predict the selection probability of the i-th expert. When G(x) i =1, then calculate the i-th expert, when G(x) i = 0, the i-th expert is not selected for calculation, and G(x) is expressed as: G(x)=Softmax(TopK(H(x),k)); Among them, TopK(H(x), k) represents a discrete function that maps the input feature H(x) to the mask m∈R 1×N , which is expressed as: Among them, the values ​​of H(x) corresponding to elements outside the first k range are set to -∞. When the mask is input to the softmax function, the -∞ elements are set to 0. The input H(x) represents the dense layer in the noise gating model. This layer learns how to select experts during training. The corresponding feature H(x) is the i-th expert feature learned. i It is expressed as: H(x) i =(x·W) i +z Softplus((x·W noise ) i ); Among them, z~N(0,1) is random Gaussian noise, W is the trainable weight matrix for learning sparse gated features, and W noise It is the weight that controls the noise increment. Different from the ordinary fully connected layer, H(x) i Contains trainable Gaussian noise to facilitate model load balancing during distributed training.

3. The sparse gating model using pan-cancer genomic information according to claim 1, characterized in that: The hybrid attention expert layer combines the attention mechanism with the hybrid expert layer, enabling the Gene-MOE model to learn rich relevant features from massive data sets.

4. A pre-training method for a sparse gating model using pan-cancer genomic information, characterized in that: The Gene-MOE model in claim 2 is pre-trained, and the pre-training content includes data enhancement strategy, combined discriminant network and Gene-MOE joint training strategy, Gene-MOE feature distribution training strategy, gene similarity training strategy, expert utilization balance training strategy and calculation of overall training loss.

5. The pre-training method of a sparse gating model using pan-cancer genomic information according to claim 4, characterized in that: The data enhancement strategy refers to constructing a data enhancement strategy by introducing Gaussian noise. For the input gene x, the enhanced input x is expressed as: x=x+z; Among them, z~N(0,σ 2 ), variance σ 2 Search via hyperparameter optimization.

6. The pre-training method of a sparse gating model using pan-cancer genomic information according to claim 5, characterized in that: The strategy of combining the discriminant network with Gene-MOE joint training refers to: constructing a discriminant network named D, D is a fully connected layer network, the Gene-MOE model is the generator, and the Wasserstein loss is introduced to jointly train G and D, G refers to the Gene-MOE model, and for G and D, the loss is expressed as: Among them, λ gp is the hyperparameter of the gradient penalty, X is x and The formula enables the Gene-MOE model to learn how to perform dimensionality reduction and reconstruct the generated genes.

7. The pre-training method of a sparse gating model using pan-cancer genomic information according to claim 6, characterized in that: The Gene-MOE feature distribution training strategy refers to the introduction of KL divergence to measure The similarity of the generated latent code z to the standard Gaussian distribution Among them, n represents the dimension of z, L KL The loss function is used to optimize the feature encoding z distribution of the input gene to maintain a standard normal distribution, optimize the training effect of the model, and simplify the training difficulty of Wasserstein loss.

8. The pre-training method of a sparse gating model using pan-cancer genome information according to claim 7, characterized in that: The gene similarity training strategy refers to: introducing L1 loss to measure the similarity of each gene between the samples reconstructed by Gene-MOE and the real samples; 9. The pre-training method of a sparse gating model using pan-cancer genomic information according to claim 8, characterized in that: The expert utilization balance training strategy refers to that in order to make each hybrid expert layer select each expert in a balanced manner, an importance loss is introduced for each sparse gating layer of the hybrid expert layer. The importance loss can be expressed as: L importance =||Importance(x k )||2; Among them, x k represents the input feature of the k-th mixed expert layer, Importance(x k ) is expressed as: Importance(x k )=∑ {x∈X} G(x k ); Where X∈R B×N , represents the total samples in the batch size; the expert will receive unbalanced samples, resulting in memory and performance issues. In order to further improve the balanced load, a load balancing loss L is introduced load , which is expressed as: Among them, Φ is the CDF of the standard normal distribution, TopK′ means selecting H(x k ) , excluding itself.

10. The pre-training method of a sparse gating model using pan-cancer genomic information according to claim 9, characterized in that: The overall training loss is calculated as: L total =L gan +λ KL ·L KL +λ l1 ·L L1 +λ balance ·(L importance +L load ); Among them, λ KL , KL , 11 and λ balance is a hyperparameter; by calculating the overall training loss, we can fit the optimal parameters of the mixed expert layer

Images