An expert fusion model for single-cell lineage tracing
By using an expert fusion model for single-cell lineage tracing, the problems of inaccurate CTC lineage tracing relationships and insufficient generalization ability in existing technologies are solved, achieving more stable CTC prediction and tracing results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG SHANGYIJIAN TECHNOLOGY CO LTD
- Filing Date
- 2026-04-22
- Publication Date
- 2026-06-19
Smart Images
Figure CN122245443A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to an expert fusion model for single-cell lineage tracing. Background Technology
[0002] Currently, there are machine learning techniques based on transcriptomics to identify and detect circulating tumor cells (CTCs). For example, there are models that train to identify CTCs using single-cell transcriptomics data of CTCs and peripheral blood. These models integrate published single-cell transcriptomics data of CTCs into single-cell transcriptomics data of blood immune cells. Based on some molecular marker genes of CTCs, they are trained using three classification techniques: Naive Bayes, gradient booster, and random forest, to obtain the iCTC model, which isolates CTCs from blood cells. Another example is the scTumorTrace model, which identifies tumor cells and blood leukocytes based on Bulk RNA sequencing data. When this model is applied to single-cell RNA sequencing data, it can identify known CTCs from mixed data.
[0003] Current machine learning or deep learning techniques for detecting circulating tumor cells (CTCs) only consider the differences between CTCs and peripheral blood immune cells in specific molecular markers, without taking into account the primary tumor origin of CTCs. While the cTumorTrace algorithm considers the relationship between CTCs and the primary tumor, it is not precise enough because the training set based on bulk RNA data cannot accurately reflect the nature of tumor cells at the cellular level. These existing techniques for tracing the origin of circulating tumor cells suffer from several drawbacks, including insufficient training data to cover the high heterogeneity of tumors, inadequate generalization ability of extracted cellular features and constructed models, and a lack of scalability for different lineage types. Therefore, these techniques cannot maintain good predictive performance on unseen datasets. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide an expert fusion model for single-cell lineage tracing.
[0005] The objective of this invention is achieved through the following technical solution: an expert fusion model for single-cell lineage tracing, comprising a dimension normalization layer, a feature extraction module, a feature fusion layer, and a multi-task head module connected in sequence;
[0006] The dimensionality normalization layer is used to receive input vector sets from different pre-trained single-cell transcriptome large models, map input vectors of different dimensions to a unified dimension and complete distribution normalization, and output a normalized vector set.
[0007] The feature extraction module includes a baseline branch, an expert network module, and a gating network module. The baseline branch is used to perform global feature extraction on the standardized vector set and output a baseline feature vector. The expert network module contains multiple parallel expert networks matching the number of input vectors. Each expert network independently extracts features from a single standardized vector in the standardized vector set and outputs the corresponding expert feature vector. The gating network module configures an independent gating unit for each prediction task. It dynamically outputs the weight coefficients of different expert networks for each task based on the input standardized vector set, and performs weighted fusion of each expert feature vector based on the weight coefficients to output the expert fused feature vector for the corresponding task.
[0008] The feature fusion layer is used to concatenate and fuse the baseline feature vectors and expert fusion feature vectors under the same task, and output the fused feature representation of the corresponding task.
[0009] The multi-task head module contains multiple parallel task heads that match the number of prediction tasks. Each task head independently receives the fusion feature representation of the corresponding task, performs task prediction for single-cell lineage tracing, and outputs the results.
[0010] Preferably, the dimension normalization layer performs linear transformation and layer normalization on each input vector. For the dimensionality normalization layer, ... input vectors Vectors after dimension standardization The calculation is as follows:
[0011] ;
[0012] in: It is a linear transformation matrix used to transform... The dimensions of the input vectors are from Mapping to a unified dimension , This is the bias vector, used for translation adjustments after dimension mapping. This is a layer normalization operation.
[0013] Preferably, the baseline branch processing steps are as follows:
[0014] S1: Concatenate all standardized vectors to obtain the concatenated features.
[0015] ;
[0016] in, The number of input vectors. A unified dimension for standardized vectors;
[0017] S2: Feature extraction is performed through a two-layer fully connected network, outputting a baseline feature vector.
[0018] ;
[0019] in, This is the first-layer weight matrix, used to map the concatenated features to the hidden layer dimensions. , This is the first-level bias vector, used for translation adjustments after dimension mapping. This is the second-layer weight matrix, used to map the hidden layer features to the baseline feature dimension. This is the second-layer bias vector, used for translation adjustments after dimension mapping. For activation function, This is a layer normalization operation.
[0020] Preferably, in the expert network module, for the first A network of experts to handle the first A standardized vector,
[0021] ;
[0022] in, For the first Each expert network weight matrix is used to map standardized vectors to expert feature dimensions. For the first A bias vector for the expert network is used for translation adjustment after dimension mapping.
[0023] Preferably, in the gating network module, for the first For each task, the output of the gating network is:
[0024] ;
[0025] in, This is the weight matrix of the first layer of the gated network, used to map the concatenated features to the hidden layer dimensions. , This is the bias vector for the first layer of the gated network. The second-layer weight matrix of the gated network is used to map the hidden layer features to the expert weight dimension. , This is the bias vector for the second layer of the gated network. This is the normalization function;
[0026] The expert feature vector is obtained by weighted summation using gating weights:
[0027] ;
[0028] in, For the first The first task, the first The weight of each expert.
[0029] Preferably, the feature fusion layer is for the first For each task, the output of the feature fusion layer is:
[0030] ;
[0031] in, For the baseline feature vector, For the first Expert feature vectors for each task.
[0032] Preferably, in the multi-task header module, the first The task header for each task is a two-layer fully connected neural network, and its processing flow is as follows:
[0033] The first layer performs a nonlinear transformation on the fused feature representation of the input.
[0034] ;
[0035] The second layer performs Dropout regularization on the hidden layer features to complete the prediction output, obtaining the... The prediction results for each task,
[0036] ;
[0037] in, This is the first-layer weight matrix of the task head, used to map the fused features to the hidden layer dimension. , This is the first layer bias vector of the task header. The task head outputs a layer weight matrix, used to map hidden layer features to the first layer. Number of task categories , Output the layer bias vector for the task header.
[0038] The present invention has the following advantages: The present invention uses a baseline branch to capture the global common information of all input features, and the expert network module mines the specific features of each pre-trained model. At the same time, it takes into account both global features and local specific features, reducing the dependence on a single feature source. Furthermore, the multi-task head module effectively improves the stable performance of the model in clinical samples through information sharing and complementarity between tasks, and reduces the impact of sequencing data quality fluctuations on prediction results. Attached Figure Description
[0039] Figure 1 A schematic diagram of the expert fusion model architecture for single-cell lineage tracing. Detailed Implementation
[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0041] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0042] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other.
[0043] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0044] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0045] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0046] In this embodiment, as Figure 1As shown, an expert fusion model for single-cell lineage tracing includes a dimension normalization layer, a feature extraction module, a feature fusion layer, and a multi-task head module connected in sequence.
[0047] The dimensionality normalization layer is used to receive input vector sets from different pre-trained single-cell transcriptome large models and normalize different dimensions ( , ... The input vector is mapped to a uniform dimension and its distribution is normalized, outputting a standardized vector set;
[0048] The feature extraction module includes a baseline branch, an expert network module, and a gating network module. The baseline branch is used to perform global feature extraction on the standardized vector set and output a baseline feature vector. The expert network module contains multiple parallel expert networks matching the number of input vectors. Each expert network independently extracts features from a single standardized vector in the standardized vector set and outputs the corresponding expert feature vector. The gating network module configures an independent gating unit for each prediction task. It dynamically outputs the weight coefficients of different expert networks for each task based on the input standardized vector set, and performs weighted fusion of each expert feature vector based on the weight coefficients to output the expert fused feature vector for the corresponding task.
[0049] The feature fusion layer is used to concatenate and fuse the baseline feature vectors and expert fusion feature vectors under the same task, and output the fused feature representation of the corresponding task.
[0050] The multi-task head module contains multiple parallel task heads matching the number of prediction tasks. Each task head independently receives the fused feature representation of its corresponding task, performs single-cell lineage tracing prediction, and outputs the results. A baseline branch is used to capture global common information from all input features, while the expert network module mines the unique features of each pre-trained model. This approach balances global and local specific features, reducing reliance on a single feature source. Furthermore, the multi-task head module effectively improves the model's stable performance on clinical samples and reduces the impact of sequencing data quality fluctuations on prediction results through information sharing and complementarity between tasks.
[0051] Furthermore, the dimension normalization layer applies a linear transformation and layer normalization to each input vector. For the dimensionality normalization layer... input vectors Vectors after dimension standardization The calculation is as follows:
[0052] ;
[0053] in: It is a linear transformation matrix used to transform... The dimensions of the input vectors are from Mapping to a unified dimension , This is the bias vector, used for translation adjustments after dimension mapping. This is a layer normalization operation used to stabilize feature distribution and accelerate convergence.
[0054] Furthermore, the steps for processing the baseline branch are as follows:
[0055] S1: Concatenate all standardized vectors to obtain the concatenated features.
[0056] ;
[0057] in, The number of input vectors. A unified dimension for standardized vectors;
[0058] S2: Feature extraction is performed through a two-layer fully connected network, outputting a baseline feature vector.
[0059] ;
[0060] in, This is the first-layer weight matrix, used to map the concatenated features to the hidden layer dimensions. , This is the first-level bias vector, used for translation adjustments after dimension mapping. This is the second-layer weight matrix, used to map the hidden layer features to the baseline feature dimension. This is the second-layer bias vector, used for translation adjustments after dimension mapping. For activation function, This is a layer normalization operation.
[0061] In this embodiment, the expert network module for the first... A network of experts to handle the first A standardized vector,
[0062] ;
[0063] in, For the first Each expert network weight matrix is used to map standardized vectors to expert feature dimensions. For the first Each expert network bias vector is used for translation adjustment after dimension mapping. This is a layer normalization operation used to stabilize feature distribution and accelerate convergence.
[0064] Furthermore, in the gating network module, for the first... For each task, the output of the gating network is:
[0065] ;
[0066] in, This is the weight matrix of the first layer of the gated network, used to map the concatenated features to the hidden layer dimensions. , This is the bias vector for the first layer of the gated network. The second-layer weight matrix of the gated network is used to map the hidden layer features to the expert weight dimension. , This is the bias vector for the second layer of the gated network. This is the normalization function;
[0067] The expert feature vector is obtained by weighted summation using gating weights:
[0068] ;
[0069] in, For the first The first task, the first The weight of each expert is determined by the output of the gating network.
[0070] Furthermore, the feature fusion layer for the first... For each task, the output of the feature fusion layer is:
[0071] ;
[0072] in, For the baseline feature vector, For the first Expert feature vectors for each task.
[0073] In this embodiment, in the multi-task header module, the first... The task header for each task is a two-layer fully connected neural network, and its processing flow is as follows:
[0074] The first layer performs a nonlinear transformation on the fused feature representation of the input.
[0075] ;
[0076] The second layer performs Dropout regularization on the hidden layer features to complete the prediction output, obtaining the... The prediction results for each task,
[0077] ;
[0078] in, This is the first-layer weight matrix of the task head, used to map the fused features to the hidden layer dimension. , This is the first layer bias vector of the task header. The task head outputs a layer weight matrix, used to map hidden layer features to the first layer. Number of task categories , Output the layer bias vector for the task header.
[0079] In this embodiment, a two-stage training strategy is employed during model training to ensure that the model can effectively learn the dynamic weight allocation of the expert network and the task-specific feature processing.
[0080] Phase 1: Training the gating network to learn the dynamic weight allocation of the expert network, ensuring that the gating mechanism can adaptively adjust the expert weights according to different tasks and input features. The core of judging the stability of the gating network parameters is to monitor the change rate of the gating weights. When the rate of change of the gating weights is less than a preset threshold for several consecutive epochs, it is considered that the gating network has learned a stable weight allocation strategy and training can be stopped.
[0081] In successive training steps, the rate of change of the gating weights is defined as follows:
[0082] ;
[0083] in, For the first In the step, the first The first task's gating network output The weight of each expert, The total number of experts, The total number of tasks. For the first The rate of change of weights compared to the previous step is used to determine whether the gating network has converged.
[0084] During the first phase of training, a learnable log-variance parameter is used to balance the loss across multiple tasks. Adjust task weights, number 1 The loss for each task is defined as:
[0085] ;
[0086] in, , for the first Cross-entropy loss for each task, For the first Predicted labels for each task, For the first The true label of each task It is a learnable log-variance parameter used to automatically adjust task weights and balance task importance in multi-task learning;
[0087] The total loss is the sum of the losses after all task adjustments:
[0088] .
[0089] The second stage involves training each task head sequentially, fine-tuning specific task heads for each task (while keeping the gating network and expert network parameters unchanged), to ensure that each task can fully utilize the features extracted by the expert network.
[0090] No. The loss function for each task head is:
[0091] ;
[0092] in, For the first Predicted labels for each task, For the first The true label of each task.
[0093] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. An expert fusion model for single cell lineage tracing, characterized in that: It includes a dimension normalization layer, a feature extraction module, a feature fusion layer, and a multi-task head module connected in sequence; The dimensional normalization layer is used to receive input vector sets from different pre-trained single-cell transcriptome large models, map input vectors of different dimensions to a unified dimension and complete distribution normalization, and output a normalized vector set. The feature extraction module includes a baseline branch, an expert network module, and a gating network module. The baseline branch is used to perform global feature extraction on the standardized vector set and output a baseline feature vector. The expert network module contains multiple parallel expert networks matching the number of input vectors. Each expert network independently extracts features from a single standardized vector in the standardized vector set and outputs a corresponding expert feature vector. The gating network module configures an independent gating unit for each prediction task. It dynamically outputs the weight coefficients of different expert networks for each task based on the input standardized vector set, and performs weighted fusion of each expert feature vector based on the weight coefficients to output the expert fused feature vector for the corresponding task. The feature fusion layer is used to concatenate and fuse the baseline feature vectors and expert fusion feature vectors under the same task, and output the fused feature representation of the corresponding task. The multi-task head module contains multiple parallel task heads that match the number of prediction tasks. Each task head independently receives the fusion feature representation of the corresponding task, performs task prediction for single-cell lineage tracing, and outputs the results.
2. The expert fusion model for single-cell lineage tracing according to claim 1, characterized in that: The dimension normalization layer applies a linear transformation and layer normalization to each input vector. input vectors Vectors after dimension standardization The calculation is as follows: ； wherein: is a linear transformation matrix for mapping the dimension of the input vector from to a unified dimension , is a bias vector for translation adjustment after dimension mapping, is a layer normalization operation.
3. The expert fusion model for single-cell lineage tracing of claim 2, wherein: The processing steps for the baseline branch are as follows: S1: Concatenate all standardized vectors to obtain the concatenated features. ； wherein, is the number of input vectors, is the uniform dimension of the normalized vectors; S2: Feature extraction is performed through a two-layer fully connected network, outputting a baseline feature vector. ； wherein, is a first layer weight matrix for mapping the concatenated features to the hidden layer dimension , is a first layer bias vector for translation adjustment after dimension mapping, is a second layer weight matrix for mapping the hidden layer features to the baseline feature dimension, is a second layer bias vector for translation adjustment after dimension mapping, is an activation function, is a layer normalization operation.
4. The expert fusion model for single-cell lineage tracing according to claim 3, characterized in that: The expert network module processes the first standardized vector for the first expert network of the first expert network ； in, For the first Each expert network weight matrix is used to map standardized vectors to expert feature dimensions. For the first A bias vector for the expert network is used for translation adjustment after dimension mapping.
5. The expert fusion model for single-cell lineage tracing according to claim 4, characterized in that: In the gated network module, for the first... For each task, the output of the gating network is: ； wherein, is a gating network first layer weight matrix for mapping the concatenated features to the hidden layer dimension , is a gating network first layer bias vector, is a gating network second layer weight matrix for mapping the hidden layer features to the expert weight dimension , is a gating network second layer bias vector, is a normalization function; The expert feature vector is obtained by weighted summation using gating weights: ； wherein, is the weight of the jth task, the ith specialist. is the weight of the jth task, the ith specialist. 6. The expert fusion model for single-cell lineage tracing according to claim 5, characterized in that: The feature fusion layer is for the first task, and the output of the feature fusion layer is: ； wherein, is a baseline feature vector, is an expert feature vector for the th task.
7. The expert fusion model for single-cell lineage tracing according to claim 6, characterized in that: In the multi-task header module, the first... The task header for each task is a two-layer fully connected neural network, and its processing flow is as follows: The first layer performs a nonlinear transformation on the fused feature representation of the input. ； The second layer performs Dropout regularization on the hidden layer features to complete the prediction output, obtaining the... The prediction results for each task, ； in, This is the first-layer weight matrix of the task head, used to map the fused features to the hidden layer dimension. , This is the first layer bias vector of the task header. The task head outputs a layer weight matrix, used to map hidden layer features to the first layer. Number of task categories , Output the layer bias vector for the task header.