A single-cell clustering system based on reinforcement learning driving

Abstract

A single cell clustering system based on reinforcement learning driving relates to a single cell clustering system. In order to solve the problems that the existing single cell self-supervised contrast learning method exists false negative sample interference causes the destruction of topological manifold and the static sampling strategy is difficult to adapt to the heterogeneity of single cell data. The present application firstly preprocesses the single cell sequencing data and constructs the cell topological adjacency network, extracts the single cell high-dimensional feature representation by using the graph attention network combined with the momentum contrast learning architecture, and introduces the topological mask filtering strategy to reduce the interference of local false negative samples; based on the PPO agent, the proportion of difficult negative sample sampling is dynamically adjusted through the multi-objective reward mechanism, so as to adaptively balance the discriminative information extraction and local topological structure protection; finally, the converged cell feature representation is used, the Leiden algorithm is combined, and the resolution optimization strategy of the target cluster number constraint is used to output the final single cell clustering result.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of bioinformatics processing and deep learning, and specifically relates to a single-cell clustering system. Background Technology

[0002] The development of single-cell RNA sequencing (scRNA-seq) technology has revealed cellular heterogeneity at an unprecedented resolution. In downstream analysis of single-cell data, the accurate clustering of cells and identification of different cell subtypes (including rare cell populations) is a key prerequisite for understanding tissue development, disease pathogenesis, and the discovery of precision medicine targets.

[0003] With the widespread application of deep learning in bioinformatics, researchers have proposed numerous single-cell clustering methods. In recent years, self-supervised contrastive learning techniques have shown great potential in single-cell sequencing data analysis due to their ability to learn highly discriminative feature representations in the absence of manually labeled data. This is achieved by bringing positive samples (such as different enhanced views of the same cell) closer together in the feature space while simultaneously widening the distance between negative samples (different cells). Building upon this, some existing methods further integrate graph neural networks with contrastive learning to process single-cell data. Specifically, these methods first construct a K-nearest neighbor topology graph using the gene expression features of single cells, and then aggregate local neighbor information using a graph neural network to generate primary cell embeddings that incorporate the underlying manifold structure. Subsequently, the spatial distribution of these primary embeddings is optimized using the objective function of contrastive learning. This paradigm of "graph network extracts topological features, contrastive learning optimizes feature discriminative power" results in higher intra-class aggregation and inter-class separation of the final output cell features, effectively supporting downstream cell clustering analysis.

[0004] While existing single-cell contrastive learning methods have made some progress in cluster analysis, significant room for improvement remains. Firstly, single-cell data exhibits extremely high dimensionality, sparsity, and complex local topological manifold structures. Most existing contrastive learning models employ an indiscriminate default negative sample allocation mechanism when constructing contrastive losses (blindly and indiscriminately treating all other cells in the current batch or memory queue, except for their own perturbed view, as negative samples). Although random negative sample sampling is easy to implement and widely applicable, it can lead to problems such as data distribution imbalance and misselection of negative samples, thus affecting model performance. Due to the random distribution characteristics of single-cell data, this passive construction method based on a globally uniform distribution is highly prone to misclassifying semantically similar and structurally highly similar cells (i.e., true topological neighbors) as negative samples and pushing them away. This severe "false negative sample" interference directly disrupts the inherent local manifold structure of the cell population. False negative samples are nodes that should be positive samples but are incorrectly labeled as negative samples. Since false negative samples and positive samples have the same high scores, optimizing them may lead to model overfitting.

[0005] To improve the model's ability to distinguish complex cell populations, some studies have attempted to introduce hard sample mining mechanisms. However, most existing mining strategies rely on static, fixed sampling ratios or hard-coded rules. Due to the heterogeneity of single-cell data and the significant differences in noise levels among different tissue types, static strategies cannot adaptively balance "discriminative information extraction" and "local topological manifold protection" during training, often leading to over-separation of rare cell populations or semantic drift.

[0006] Secondly, in the downstream clustering stage after extracting cell features, commonly used heuristic community detection algorithms (such as the Leiden algorithm) are extremely sensitive to resolution parameters. Existing methods lack a mechanism that can combine prior information such as the number of target clusters to effectively optimize clustering resolution parameters and lock in a better clustering state, which leads to low stability and accuracy of clustering results. Summary of the Invention

[0007] This invention aims to address the problems of topological manifold disruption caused by spurious negative sample interference in existing single-cell self-supervised contrastive learning methods, and the difficulty of adapting static sampling strategies to the heterogeneity of single-cell data.

[0008] A single-cell clustering system based on reinforcement learning, comprising:

[0009] Data preprocessing and graph construction module: Obtain the original single-cell gene expression matrix, perform quality control at the cell and gene levels, and generate a cell feature matrix; calculate the similarity between cells in the cell feature matrix, construct a K-nearest neighbor topology graph, generate a global adjacency matrix representing the spatial relationships of cells, and obtain a cell topology graph represented by the global adjacency matrix; apply random perturbations to the cell feature matrix and the global adjacency matrix respectively to generate their corresponding enhanced views; use the cell feature matrix and cell topology graph before applying random perturbations as the original view;

[0010] The dual-stream feature encoding module includes a query encoder and a key encoder. The query encoder and the key encoder receive the original view and the enhanced view as input, respectively. First, they generate hidden layer embeddings, and then they are mapped to projective space embeddings through a projection layer. The projective space embeddings are used for similarity calculation in contrastive learning, and the hidden layer embeddings are used for downstream clustering analysis after training convergence.

[0011] The topological neighbor filtering module maintains a first-in-first-out dynamic negative sample queue. This queue stores the projection space embeddings output by the key encoder in historical batches and serves as a candidate set of negative samples for other cells in subsequent batches. In each batch, with the current cell as the center, the module searches for its topological neighbor samples in the global topological graph based on the global adjacency matrix and retrieves the sample index corresponding to the topological neighbor cell from the negative sample queue. The retrieved topological neighbor samples are then masked in the contrastive loss to prevent them from participating in negative sample competition, resulting in a filtered candidate set of negative samples.

[0012] The reinforcement learning sampling auxiliary module employs a proximal policy optimization PPO agent as an auxiliary controller for the dynamic sampling module. The policy network outputs parameters of the continuous action distribution based on the current state vector, namely the action mean and variance. The sampling ratio of actual difficult samples in the current batch is determined based on the action or the action mean. Based on the current sampling ratio of difficult samples Difficult samples are selected from the filtered negative sample candidate set to participate in the InfoNCE contrastive loss calculation;

[0013] Adaptive clustering evaluation module: By introducing a dual constraint mechanism of target cluster number anchoring and graph module degree Q maximization, automatic optimization of clustering partitioning in the high-dimensional hidden layer embedding space is achieved.

[0014] Furthermore, both the query encoder and the key encoder in the dual-stream feature encoding module adopt a graph attention network structure, and each layer of the graph attention network uses a multi-head attention mechanism to aggregate neighbor node information.

[0015] Furthermore, the parameters of the query encoder are in a learnable state, while the gradient of the parameters of the key encoder is truncated. The query encoder is responsible for participating in backpropagation updates, while the key encoder maintains smooth parameter changes through a momentum update mechanism.

[0016] Furthermore, the sampling ratio of actual difficult samples in the current batch is determined based on the action or the average of the actions. In the process, the actual difficulty in sampling proportion Where A represents or , Actions are obtained by sampling from the action distribution. The mean of the action. This is the upper limit ratio. This is the lower limit ratio.

[0017] Furthermore, the upper limit ratio Set to 0.35, lower limit ratio Set to 0.15.

[0018] Furthermore, based on the current sampling ratio of difficult samples The process of selecting difficult samples from the filtered negative sample candidate set to participate in the InfoNCE contrastive loss calculation includes:

[0019] Extract negative sample candidates from the filtered negative sample candidate set, calculate the similarity between the current cell and the remaining negative sample candidates, and sort them from high to low similarity. The top-N most similar negative samples are selected to form a difficult negative sample set, which is then used to calculate the InfoNCE contrastive loss.

[0020] Furthermore, the InfoNCE contrast loss is as follows:

[0021]

[0022] in, This indicates the contrast loss corresponding to the current cell; This represents the projection space feature representation obtained after the original view corresponding to the current cell is processed by the query encoder and the projection head. This represents the positive sample projection space feature representation of the same cell corresponding to the current cell, obtained after processing by the key encoder and projection head in the enhanced view; This represents the set of Top-N difficult samples selected in the current batch; Represents a set The first in Projected spatial feature representations corresponding to each difficult sample; This is the temperature coefficient.

[0023] Furthermore, the elements of the state vector include the ratio of the average similarity of positive sample pairs to the average similarity of negative sample pairs, the normalized graph neighbor consistency score, the surrogate rate of false negative samples in the previous batch, the sampling ratio of hard samples, and the loss change characteristics after nonlinear compression; wherein,

[0024] The ratio of the average similarity of positive sample pairs to the average similarity of negative sample pairs. :

[0025]

[0026] in, This represents the average similarity of positive sample pairs in the current batch. This represents the average similarity of negative sample pairs in the current batch. To prevent constants with a denominator of zero;

[0027] Normalized graph neighbor consistency score :

[0028]

[0029]

[0030] in, This represents the set of edges in the cell topology graph of the current batch. Indicates the number of edges; sim( ) represents the similarity function of projected spatial features; and These represent the feature representations of the cell nodes at both ends of the connection in the projection space generated by the query encoder;

[0031] The proxy rate of the previous batch of spurious negative samples like:

[0032]

[0033] in, This represents the total number of difficult samples corresponding to the current cell. It is the projection space feature representation of the current cell output by the query encoder. It is the projected spatial feature representation of the i-th hard sample output by the key encoder. Represents the similarity function of projected space features; This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise; Preset safety threshold;

[0034] Loss variation characteristics after nonlinear compression :

[0035]

[0036]

[0037] in, This represents the smoothed contrast loss from the previous batch. This represents the smoothing contrast loss for the current batch. This indicates the amount of decrease in smoothing loss. This represents the loss scaling factor.

[0038] Furthermore, the proximal policy optimizes the multi-objective reward function during PPO agent updates. , As a reward for reduced losses, Rewards are given for feature consistency. As a reward for inter-class separation, This is a non-linear penalty term for false negative samples. A proportional regularization term is used to constrain the agent from falling into extreme sampling strategies;

[0039] The loss reduction reward , This represents the hyperbolic tangent activation function. As a reward weight, This is the scaling factor;

[0040] Feature Consistency Reward , As a reward weight;

[0041] Inter-class separation reward , As weight, and These represent the average similarity between positive and negative sample pairs, respectively.

[0042] Nonlinear spurious negative sample penalty term , For penalty weighting;

[0043] Proportional regularization term , For regularization weights, This refers to the proportion of difficult samples sampled, which is the actual proportion of difficult samples obtained from the previous training. ; This indicates the proportion of difficult samples in the preset benchmark. and These represent the upper limit and lower limit of the sampling ratio for difficult samples, respectively.

[0044] Furthermore, the reinforcement learning sampling assistance module also includes a security gate control step, which includes:

[0045] Real-time monitoring of spurious negative sample proxy rate in each batch When the observed false negative sample proxy rate exceeds the preset safety limit, the sampling ratio of difficult samples output by the PPO agent is restricted, forcibly reducing the current sampling ratio of difficult samples to a safe benchmark value.

[0046] The beneficial effects of this invention are:

[0047] This invention constructs a global topological adjacency relationship and introduces a topological neighbor filtering mechanism. This prioritizes filtering out topological neighbor-type negative sample candidates that can be explicitly identified during contrastive learning, reducing the phenomenon of similar cells being mistakenly pushed away as negative samples, thus helping to protect the integrity of the cell's local manifold structure. Furthermore, by introducing a reinforcement learning sampling assistance module, this invention models the optimization process of the difficult sample sampling ratio as a dynamic decision-making process. This allows the model to adaptively adjust the difficult sample sampling ratio according to the training state, overcoming the problem that traditional static sampling strategies are difficult to adapt to the heterogeneity of single-cell data and the inability to balance discriminability and robustness. In addition, this invention uses a dual-flow graph attention encoding structure to extract cell representations and employs a resolution optimization strategy combining target cluster number constraints and graph module degree maximization in the clustering evaluation stage, thereby improving the accuracy and stability of the clustering results. Experimental results show that this invention has good clustering performance and robustness on multiple standard single-cell datasets and has good application value. Attached Figure Description

[0048] Figure 1 This is a flowchart of a single-cell clustering system driven by reinforcement learning.

[0049] Figure 2 A visualization comparison of the clustering effects of UMAP cell embedding representation on representative single-cell datasets.

[0050] Figure 3 Performance graphs of ARI validation experiments on five datasets and compared with other methods.

[0051] Figure 4 Performance graphs of NMI validation experiments on twelve datasets compared with other methods.

[0052] Figure 5 A bar chart comparing the performance evaluation results of ARI and NMI in ablation experiments on datasets of varying complexity.

[0053] Figure 6The figure shows the results of the parameter sensitivity analysis of the model on the Quake_10x_Limb_Muscle dataset, including the parameter variation curves of the effect of the nearest neighbor number k and the temperature coefficient τ on the model's clustering performance (ARI).

[0054] Figure 7 This is a graph showing the dynamic co-change between the contrastive loss and the proportion of hard sample samples output by the agent during the training process of the model on the Quake_10x_Limb_Muscle dataset. Detailed Implementation

[0055] It should be noted that, where there is no conflict, the various embodiments disclosed in this invention can be combined with each other.

[0056] Specific implementation method one: Refer to Figure 1 This implementation method will be described in detail.

[0057] This embodiment describes a single-cell clustering system driven by reinforcement learning, comprising a data preprocessing and graph construction module, a two-stream feature encoding module, a topological neighbor filtering module, a reinforcement learning sampling assistance module, and an adaptive clustering evaluation module. The modules are sequentially connected according to the data flow order.

[0058] Data preprocessing and graph construction module: used to generate cell feature matrices and cell topological relationships;

[0059] Two-stream feature encoding module: used to generate query features and key features required for contrastive learning;

[0060] Topology neighbor filtering module: used to filter out topology neighbors in the dynamic negative sample queue;

[0061] Reinforcement learning sampling assistance module: used to dynamically adjust the sampling ratio of difficult samples according to the training status;

[0062] Adaptive clustering evaluation module: used to optimize clustering resolution and output final clustering results based on target cluster number constraints.

[0063] More specifically,

[0064] The data preprocessing and graph construction module's processing steps include:

[0065] Step a1: Obtain the original single-cell gene expression matrix and remove genes with extremely low expression and low-quality cells using quality control methods; perform quality control on the original single-cell gene expression matrix at the cell and gene levels to reduce the impact of noise on subsequent clustering analysis.

[0066] The process of quality control at the cellular and gene levels includes: for low-expression genes, removing genes expressed in fewer than 3 cells; for low-quality cells, removing cell samples with abnormally low sequencing counts, i.e., removing low-quality cell samples with abnormal sequencing counts or in apoptosis through multidimensional indicators. Specific filtering operations include: removing cells with fewer than 200 detected gene expression types, removing cells with a total sequencing count of less than 500, and removing cells with mitochondrial gene expression exceeding 5%.

[0067] Step a2: Perform normalization, logarithmic transformation, highly variable gene screening, and standardization on the retained high-quality single-cell data to generate a cell feature matrix;

[0068] First, sequencing and normalization are performed on each cell, followed by logarithmic transformation, which can weaken the influence of extreme expression values. Highly variable genes are screened as effective features to retain the most distinctive biological variations among single cells. Finally, the features are standardized to generate the cell feature matrix for model input.

[0069] In some embodiments, during the sequencing normalization process for each cell, a total count normalization method is used. The gene expression level of each cell is divided by the total cell count and multiplied by a scaling factor of 10,000 to eliminate sequencing depth differences. The top 4000 highly variable genes are selected as effective features to retain the most discriminative biological variations among single cells. Highly variable genes are defined as those for which the average expression level and variance of each gene are calculated across all cells. The trend of variance with the mean is observed using a fitted curve. Genes that deviate most from the fitted curve, i.e., whose variance is significantly higher than the variance corresponding to their expected mean, are selected as highly variable genes, and the top 4000 are chosen as highly variable genes.

[0070] Step a3: Calculate the similarity between cells in the cell feature matrix, construct a K-nearest neighbor topology graph, and generate a global adjacency matrix representing the spatial relationship between cells. Based on the preprocessed cell feature matrix, use Euclidean distance to measure the similarity between cells, construct a cell K-nearest neighbor topology graph, and generate a global adjacency matrix based on the topology graph to represent the local manifold structure and adjacency relationship between cells. That is, the K-nearest neighbor method is used to construct a cell topology graph represented by the global adjacency matrix.

[0071] The K-nearest neighbor number is set to 15. For each cell, 15 nearest neighbor cells are searched to construct the cell K-nearest neighbor topology graph.

[0072] Step a4: Apply random perturbations to the cell feature matrix and the global adjacency matrix respectively to generate their corresponding enhanced views. The cell feature matrix and cell topology map before applying the random perturbations are denoted as the original cell feature matrix and the original cell topology map.

[0073] During the application of random perturbations, a mask perturbation is applied to node features, and an edge discard perturbation is applied to topological edge connections. The perturbation probability is set to 0.2 for both.

[0074] The dual-stream feature encoding module is as follows:

[0075] The dual-stream feature encoding module includes a query encoder and a key encoder. The query encoder and key encoder receive the original view (original cell feature matrix and original cell topology map) and the enhanced view as input, respectively. They first generate hidden layer embeddings and then map them to projective space embeddings through a projection layer. The projective space embedding is used for similarity calculation in contrastive learning, and the hidden layer embedding is used for downstream clustering analysis after the model training converges.

[0076] The two-stream feature encoding module encodes the input high-dimensional single-cell features (the original cell feature matrix and its enhanced view) and their topological information (the original cell topology map and its enhanced view) into a low-dimensional representation suitable for contrastive learning, and further outputs cell embedding results that can be used for downstream clustering analysis. In this process, the enhanced view serves as the input to the key encoder, while the query encoder receives the undisturbed original view, thus forming an asymmetric two-view input structure to enhance the model's robustness to local perturbations.

[0077] The processing procedure of the dual-stream feature coding module includes:

[0078] Step b1: Construct a two-stream graph attention network containing a query encoder and a key encoder. Both the query encoder and the key encoder adopt a two-layer graph attention network structure. Each layer uses a multi-head attention mechanism to aggregate neighbor node information, with the number of attention heads set to 4. In the two-stream structure, the parameters of the query encoder are in a learnable state, while the gradients of the key encoder parameters are truncated. The query encoder is responsible for participating in backpropagation updates, while the key encoder maintains smooth parameter changes through a momentum update mechanism.

[0079] Step b2: The original view is input into the query encoder. The query encoder performs graph attention encoding based on the cell feature matrix and the cell topology graph represented by the global adjacency matrix. Under the constraints of the cell topology graph, the features of neighboring nodes are weighted and aggregated to obtain the hidden layer embedding of the cell. Then, it is mapped to the projection space embedding through the projection layer. The query encoder encodes the cell features and their topological relationships through a two-layer graph attention network to obtain the hidden layer embedding, with the hidden layer dimension set to 256. Subsequently, the hidden layer embedding is input into the multilayer perceptron projection head and mapped to the projection space embedding, with the projection space dimension set to 128. After normalization, it is used for similarity calculation in subsequent contrastive learning.

[0080] The enhanced view is input to the key encoder, which extracts the hidden layer embedding and maps it to the projection space embedding. The structure of the key encoder is consistent with that of the query encoder. It first generates a hidden layer embedding with a dimension of 256, and then maps it to a projection space embedding with a dimension of 128 through the projection layer.

[0081] In the graph attention network structure corresponding to the query encoder and key encoder, GAT captures node features and network topology information. The node feature update formula is as follows:

[0082]

[0083] in These are learnable weight parameters. For nodes The neighborhood group, Attention coefficient It is the hidden layer embedding after node update.

[0084] The final hidden layer embedding is mapped to a projection space embedding using MLP. .

[0085] This implementation uses a momentum update mechanism to update the key encoder in order to maintain the consistency of the negative sample representation. The parameters of the key encoder... Parameters from the query encoder Smooth Update:

[0086]

[0087] in The momentum coefficient is taken in this invention. .

[0088] After completing the contrastive learning for the current batch, the key encoder output is written to the dynamic negative sample queue. The representation of the current cell under the augmented view first participates in the training of the current batch as a positive sample for that cell, and then enters the queue; after entering the queue, the representation is no longer a positive sample for other cells, but is instead used as a negative sample candidate for other cells.

[0089] Positive samples are used to characterize the same cell across views in the original view and the enhanced view. Figure 1 Consistency, negative samples are used to characterize the separability between different cells. Difficult samples are high-similarity subsets of negative samples, that is, the top-K samples with the highest similarity to the current cell in the projected feature space among the negative sample candidates after initial screening. Although these samples still participate in contrastive learning as negative samples, they are more critical to improving the model's discrimination ability because they are closer to the cell.

[0090] False negative samples refer to samples that should have retained their proximity relationships but were mistakenly treated as negative samples during training. It's important to note that the subsequent topological neighbor filtering module does not directly filter all false negative samples, but rather those that can be explicitly identified based on the global adjacency matrix—samples that have an edge relationship with the current cell in the global topological graph. Once identified, these samples are directly removed from the negative sample candidate and no longer participate in subsequent difficult sample selection. The scope of false negative samples is broader than the "filtered samples": besides the topological neighbors that have already been explicitly identified and filtered out, there may be samples that, although not identified as explicit neighbors of the current cell in the global adjacency matrix, still have a high similarity to the current cell in the projected feature space. These samples may semantically still belong to the same type of cell or neighboring cells; continuing to push them away as negative samples could still disrupt the local manifold structure. Therefore, the false negative samples in this invention are not a complete and accurate set of explicit samples identified one by one during training, but rather their residual risk is quantitatively characterized through the false negative sample surrogate rate. In other words, the false negative sample proxy rate is not a virtual parameter, but a risk assessment metric based on the current set of difficult samples, used to measure the proportion of potential false negative samples in the set of difficult samples. Since difficult samples are a subset of negative samples with high similarity to the current cell, they are more likely to contain potential false negative samples that are semantically close to the current anchor cell. The false negative sample proxy rate is further fed as a feedback signal into the reinforcement learning state space to assist the PPO agent in dynamically adjusting the difficult sample sampling strategy in subsequent training batches.

[0091] Step b3: Update the query encoder and key encoder parameters:

[0092] The query encoder uses backpropagation to update in real time;

[0093] The key encoder does not directly participate in gradient updates, but adopts a momentum update strategy with a momentum coefficient set to 0.99. This allows the key encoder parameters to smoothly follow the evolution of the query encoder, thereby enhancing the consistency and stability of negative sample representations.

[0094] The processing procedure of the topology neighbor filtering module includes:

[0095] Step c1: Maintain a first-in, first-out (FIFO) dynamic negative sample queue. Construct and maintain a data queue of fixed capacity (e.g., 4096 cells). This dynamic negative sample queue stores the projected spatial embeddings output by the key encoder in historical training batches and serves as a candidate set of negative samples for other cells in subsequent training batches. It should be noted that for a given cell in the current training batch, its corresponding key encoder output is first used in conjunction with the query encoder output to calculate the contrastive loss; after the contrastive learning calculation for the current batch is completed, the key encoder output is then written into the dynamic negative sample queue.

[0096] During each batch iteration of model training, the dynamic negative sample queue is updated in a first-in, first-out (FIFO) manner: the projected spatial features of cells generated by the key encoder and projection layer in the current batch are written to the tail of the queue, while the earliest historical projected spatial features entering the queue at the head of the queue are removed and discarded. This method maintains a large historical negative sample candidate pool with low memory overhead, thereby improving the efficiency and stability of negative sample construction in contrastive learning. Furthermore, the queue also synchronously stores the corresponding cell node indexes for subsequent steps to perform topological neighbor filtering in conjunction with the cell topology graph.

[0097] Optionally, in some embodiments, the dynamic negative sample queue capacity is set to 4096 to store the projected spatial features of cells generated by the key encoder in historical batches, thereby expanding the size of the negative sample candidate set and improving the efficiency of contrastive learning. The queue also maintains the corresponding sample index for subsequent topological neighbor filtering.

[0098] Step c2: Retrieve topological neighbors based on the global adjacency matrix:

[0099] In each training batch, with the current cell as the center, its neighboring cells in the global topology graph are found according to the global adjacency matrix, i.e., topological neighbor samples. Furthermore, the sample index corresponding to the topological neighbor cells is retrieved in the negative sample queue to provide a basis for subsequent topological neighbor filtering.

[0100] Step c3: Perform topology neighbor filtering:

[0101] The similarity scores of the retrieved topological neighbor samples in the contrastive loss are explicitly replaced with negative infinity (equivalent to masking all sims), thus preventing them from participating in the negative sample competition and obtaining a filtered set of negative sample candidates. This step explicitly masks the situation where real neighbors of the same type are mistakenly identified as negative samples and provides a more reasonable candidate set for subsequent false negative sample risk assessment.

[0102] The processing procedure of the reinforcement learning sampling assistance module includes:

[0103] Step d1: Employ a Proximal Policy Optimization (PPO) agent as the auxiliary controller for the dynamic sampling module. In the current training batch t, ​​the system constructs a reinforcement learning state vector based on the contrastive learning statistics of the current batch. This is used to characterize the contrastive learning quality and sampling state in the current environment. Some components of the state vector are calculated by combining feedback information from the previous training batch (t-1). For example, the smoothing loss reduction is composed of the loss change between the current training batch and the previous training batch.

[0104] Optionally, a 5-dimensional reinforcement learning state vector is used. Specifically:

[0105]

[0106] in, This represents the ratio of the average similarity of positive sample pairs to the average similarity of negative sample pairs. The normalized graph neighbor consistency score is obtained by calculating the average cosine similarity of connected cell nodes in the projection space. This represents the spurious negative sample proxy rate of the previous batch. It is calculated by counting the number of Top-K difficult samples selected in the previous batch whose cosine similarity exceeds a preset safety threshold, and dividing this number by the total number of difficult samples to obtain the corresponding proportion. The preset safety threshold is preferably 0.9. This item is used to assess the risk of contamination of similar cells in the previous batch of sampling. This represents the proportion of hard samples sampled, which is the actual proportion of hard samples sampled in the previous training iteration. For the first batch of training samples, the value was set to 0.2; This indicates the characteristic of loss variation after nonlinear compression.

[0107] It should be noted that in some embodiments, one or more of the above-mentioned 5-dimensional reinforcement learning state vectors can be used to construct the state vectors, depending on actual needs. It is even possible to construct the state vector using other parameters. .

[0108] The positive and negative sample similarity ratio feature The ratio used to characterize the relative separability of positive and negative sample pairs in the current training batch is calculated as follows: The average similarity of positive and negative sample pairs in the current training batch is calculated separately, and their ratio is constructed, specifically expressed as:

[0109]

[0110] in, This represents the average similarity of positive sample pairs in the current training batch. This represents the average similarity of negative sample pairs in the current training batch. To prevent extremely small constants with a denominator of zero. When A larger value indicates that the positive and negative samples in the current training batch have better discriminative power.

[0111] The graph neighbor consistency score The consistency score is used to characterize the degree of aggregation of adjacent cells in the projection space of a topological graph. It is calculated as follows: First, the mean similarity of projection space features is calculated for each pair of cell nodes connected in the topological graph to obtain the original consistency score. Then normalize it to the [0,1] interval, specifically as follows:

[0112]

[0113]

[0114] in, This represents the set of edges in the cell topology graph of the current training batch. Indicates the number of edges; sim( ) represents the similarity function of the projection space features, preferably the cosine similarity function; and This represents the projection space feature representation generated by the cell nodes at both ends of the edge in the query encoder branch of the current training batch. The larger the value, the more consistent the topological neighbors are in the feature space, and the better the local manifold structure is preserved.

[0115] The spurious negative sample proxy rate The calculation method is as follows: the Top-K difficult samples selected in step d3 (obtained from the previous batch of training samples, set as a constant value in the first batch of training samples) have a cosine similarity exceeding a preset safety threshold. The number of samples is divided by the total number of difficult samples to obtain the corresponding proportion, wherein the preset safety threshold is... The value is 0.9. Its mathematical formula is:

[0116]

[0117] in, This represents the total number of difficult samples corresponding to the current cell. It is the projection space feature representation of the current cell output by the query encoder. It is the projected spatial feature representation of the i-th hard sample output by the key encoder. Represents the similarity function of projected space features; This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise; This is a preset safety threshold.

[0118] The loss change characteristics The method used to characterize the convergence trend of the current training phase is as follows: First, the difference in smoothed loss between the current training batch and the previous training batch is statistically analyzed, and then compressed using a nonlinear function, specifically expressed as follows:

[0119]

[0120]

[0121] in, This represents the smooth contrast loss from the previous training batch. This represents the smooth contrast loss of the current training batch. This indicates the amount of decrease in smoothing loss. This represents the loss scaling factor. Used to reflect the optimization trend of the current training batch relative to the previous training batch, when A larger value indicates that the model has a more obvious trend of loss reduction at the current stage.

[0122] Step d2: Generate the sampling ratio for difficult samples based on the state vector.

[0123] In this embodiment, the policy network of the PPO agent receives the current state vector. The output parameters are the distribution of continuous actions, i.e., the mean of a one-dimensional Gaussian distribution of the policy network output. and variance This allows us to construct the continuous action distribution corresponding to the current training batch:

[0124]

[0125] in, This represents the mean of the action distribution. The variance representing the distribution of actions. This represents the action sampled from the Gaussian distribution.

[0126] During the training phase, the system samples actions from the continuous action distribution. The actual hard sample sampling ratio of the current training batch is determined based on the actions obtained from the action distribution sampling. ; As the status of the next batch of training processes In the usage phase after training, to ensure output stability, the system no longer performs random sampling, but directly uses the mean of the action distribution. As a deterministic action, the sampling proportion of difficult samples is determined based on the action mean. .

[0127] In determining the sampling proportion of difficult samples based on actions or action mean, the actions... / Motion average The Sigmoid function is used for strict interval mapping to generate the actual hard sample sampling ratio for the current batch. Its mapping formula is:

[0128] During the training phase:

[0129]

[0130] The usage phase after training:

[0131]

[0132] Among them, the upper limit ratio Set to 0.35, lower limit ratio The value is set to 0.15. This mathematical mapping method theoretically ensures that the sampling ratio of difficult samples in the output is always limited to the range of [0.15, 0.35].

[0133] Step d3: Perform hard sample sampling and update the dual-flow graph attention network of the query encoder and key encoder:

[0134] Based on the current sampling ratio of difficult samples, Top-K difficult samples are selected from the filtered (topological neighbor sample masking) negative sample candidate set to participate in the InfoNCE contrastive loss calculation. The specific mathematical formula is as follows:

[0135]

[0136] in, This indicates the contrast loss corresponding to the current cell; This represents the projection space feature representation obtained after the original view corresponding to the current cell is processed by the query encoder and the projection head; the original view includes the original cell feature matrix and the original cell topology map represented by the global adjacency matrix. This represents the positive sample projection space feature representation of the same cell corresponding to the current cell, obtained after processing by the key encoder and projection head under the enhanced view; the enhanced view includes the enhanced cell feature matrix and the enhanced cell topology. This represents the set of Top-K difficult samples selected in the current batch; Represents a set The first in The projection space feature representation corresponding to each difficult sample; the difficult samples are derived from a dynamic negative sample queue, which stores the historical projection space feature representations obtained by processing the enhanced views of other cells through a key encoder and a projection head. The temperature coefficient represents the dot product operation between eigenvectors. Setting it to 0.2 adjusts the sharpness of the contrast similarity distribution.

[0137] In the current training batch, the system first extracts negative sample candidates from the dynamic negative sample queue, centered on the query representation of the current cell, and removes known topological neighbors using topological neighbor filtering. Then, it calculates the similarity between the query representation and the remaining negative sample candidates, and sorts them from high to low similarity. Select the top-K most similar negative samples to form a hard negative sample set ( (Limited number, Top-K limits which ones to select). Based on this, the system constructs the InfoNCE contrastive loss, which improves the discriminativeness of cell representation by bringing the current cell representation closer to the positive sample representation in the projection space, while simultaneously increasing the distance between the cell representation and the representations of each sample in the hard sample set.

[0138] The positive sample is represented by the projection space feature representation of the same cell corresponding to the current cell under the enhanced view, which is processed by the key encoder and the projection layer; the sample representations in the hard negative sample set are derived from the historical projection space feature representations of other cells that have been processed by the key encoder and the projection layer under the enhanced view and written into the dynamic negative sample queue.

[0139] After calculating the InfoNCE contrast loss, the system uses the backpropagation mechanism to update the query encoder and its projection layer parameters; the key encoder and its projection layer parameters do not directly participate in the gradient backpropagation update, but are smoothly and synchronously updated by the query encoder parameters through the momentum update mechanism.

[0140] Finally, the false negative sample surrogate rate (FNR) of the difficult samples obtained from this batch is statistically analyzed and sent as a feedback signal into the reinforcement learning state space to assist the PPO agent in dynamically adjusting the sampling strategy of subsequent training batches.

[0141] It is particularly noteworthy that the closed-loop control mechanism described above, which dynamically drives the sampling of difficult samples in contrastive learning through reinforcement learning, constitutes the core substantive innovation of this invention. Existing traditional contrastive learning methods, when processing single-cell transcriptome data, typically employ blind random negative sample sampling or a static difficult sample mining strategy with a fixed ratio. This static strategy, when faced with a highly heterogeneous single-cell feature space containing continuous differentiation trajectories, is prone to "over-mining," pushing biologically homologous cells away, severely disrupting the original graph topology and causing semantic drift. To address this technical problem, this invention creatively integrates the topological representation capabilities of graph neural networks, the self-supervised capabilities of contrastive learning, and the adaptive control capabilities of proximal strategy optimization. By constructing a dynamic state for reinforcement learning using five dimensions—positive-negative sample similarity ratio, graph topological consistency, false negative sample surrogate rate, difficult sample sampling ratio, and contrastive loss change state—the PPO agent adaptively outputs the optimal difficult sample sampling ratio for the current batch, and continuously iterates the agent's strategy using InfoNCE loss feedback during subsequent training. This mechanism breaks through the limitations of traditional contrastive learning's isolated training, constructing an intelligent collaborative closed-loop framework where "reinforcement learning dynamically guides difficult sampling, and contrastive learning provides real-time state feedback." This enables the model to accurately achieve the optimal balance between "feature recognition accuracy" and "graph topological integrity" in the highly challenging single-cell analysis scenario.

[0142] Step d4: Update the PPO agent according to the multi-objective reward function. The multi-objective reward function consists of a loss reduction reward term, a feature consistency reward term, an inter-class separation reward term, a non-linear false negative sample penalty term, and a proportional regularization term, used to comprehensively evaluate the impact of the current sampling action on the training of the contrastive learning model. The system updates the PPO agent according to the immediate reward, enabling it to gradually learn to output a more appropriate hard sample sampling ratio at different training stages. To improve the stability in the early stages of training, some embodiments set the number of warm-up rounds to 10, during which a fixed hard sample sampling ratio of 0.15 is used; after the warm-up, the PPO agent takes over dynamic sampling control.

[0143] Multi-objective reward function in step d4 It is shown as:

[0144]

[0145] in, As a reward for reduced losses, Rewards are given for feature consistency. As a reward for inter-class separation, This is a non-linear penalty term for false negative samples. A proportional regularization term is used to constrain the agent from falling into extreme sampling strategies.

[0146] Specifically, the construction logic for the five reward sub-items is as follows:

[0147] The loss reduction reward Constructed based on the current smoothed loss decrease, specifically expressed as:

[0148]

[0149] in, This represents the hyperbolic tangent activation function. As a reward weight, This represents the smoothed loss decrease at the current training moment relative to the previous moment. This is the scaling factor; this term is used to encourage the agent to take sampling actions that enable the model to converge quickly.

[0150] Feature Consistency Reward Constructed based on the normalized graph neighbor consistency score, specifically represented as follows:

[0151]

[0152] in, As a reward weight, The consistency score of the neighbors in the current batch;

[0153] pass It can be seen that when the angle between neighboring cells in the feature space is smaller (i.e., more clustered), their average cosine similarity is higher. The higher the feature consistency reward, the more the agent receives. The larger the size, the more it guides the model to remain anchored to the underlying cellular biological manifold structure during the comparative learning process.

[0154] Inter-class separation reward Constructed based on the average similarity difference between positive and negative sample pairs, specifically expressed as:

[0155]

[0156] in, As weight, and These are the average similarity of positive and negative sample pairs, respectively; this term guides the agent to select samples that can increase the difference between categories and improve the discriminative power of the embedding space.

[0157] Nonlinear spurious negative sample penalty term Nonlinear negative feedback is constructed using the hyperbolic tangent function, specifically expressed as follows:

[0158]

[0159] in, 0.5 is the penalty weight; 0.5 is the baseline threshold; when As the penalty increases, the intensity increases non-linearly, thereby suppressing false negative sample interference and semantic drift caused by over-mining.

[0160] Proportional regularization term : Used to constrain the deviation of the sampling ratio of difficult samples from a preset benchmark ratio, specifically expressed as:

[0161]

[0162] in, For regularization weights, The proportion of difficult samples. This indicates the preset baseline sampling ratio for difficult samples, set to 0.2; and These represent the upper limit and lower limit of the sampling ratio for difficult samples, respectively; this term is used to constrain the exploration boundary of the agent and avoid falling into extreme or unstable sampling strategies.

[0163] The reinforcement learning sampling assistance module also includes a safety gate mechanism, designed to prevent the agent from over-exploring and causing the model to crash. Specifically, it includes:

[0164] In each training batch, the system monitors the spurious negative sample surrogate rate in real time. When the observed false negative sample surrogate rate exceeds a preset safety limit, the system restricts the sampling ratio of difficult samples output by the PPO agent, forcibly reducing the current difficult sample sampling ratio to a safe baseline value to prioritize the integrity of the local topology and the stability of the semantic space. Through this hard boundary control, the integrity of the cell's local topology and the stability of the semantic space are prioritized. Once the situation returns to a safe range, dynamic control of the agent is restored.

[0165] The adaptive clustering evaluation module aims to overcome the shortcomings of traditional single-cell clustering methods, which heavily rely on manual experience to adjust resolution parameters. By introducing a dual constraint mechanism of anchoring the target cluster number (known cluster number) and maximizing the Q-degree of the graph module, it achieves automatic optimization of cluster partitioning in the high-dimensional hidden layer embedding space. Specific steps include:

[0166] Step e1: Obtain the final hidden layer embedding output by the query encoder after the training of the dual-flow graph attention network has converged; downstream clustering does not directly use the projection space embedding, but uses the hidden layer embedding output by the query encoder as the clustering input to preserve more complete cell biological semantic information.

[0167] Step e2: Based on the final hidden layer embedding, an automatic resolution optimization strategy with known cluster number constraints is adopted, and the Leiden algorithm is used to cluster the cells;

[0168] During the clustering process using an automatic resolution optimization strategy with known cluster number constraints, the cluster graph is reconstructed based on the final hidden layer embedding, and the Leiden algorithm is used to cluster the cells. By traversing and searching within a preset resolution range, a set of candidate partitions with the cluster number closest to the target cluster number is selected. Among the candidate cluster partitions that meet the target cluster number, the partition that maximizes the modularity of the cluster graph is selected as the final cell clustering result.

[0169] The adaptive clustering evaluation module is based on the Leiden algorithm (graph theory community detection algorithm), and evaluates the target cluster number. Automatic resolution optimization is achieved through a dual constraint of anchoring and maximizing modularity. The specific steps include:

[0170] First, set the resolution parameters. The search interval and search step size are determined, and the Leiden algorithm is used to perform multiple clustering iterations on the cell topology map to generate a series of clusters corresponding to different... The initial partitioning result is based on each resolution parameter. The partitioning scheme is obtained by multiple clustering iterations on the topological graph.

[0171] Secondly, the number of clusters selected is exactly equal to the number of target biological clusters. The partitioning scheme is used to construct a candidate set;

[0172] Finally, the graph module degree of each partitioning scheme in the candidate set is calculated. And select the module that makes the graph module degree The partition that yields the global maximum value is taken as the final cell clustering result; wherein, the graph module degree The formula used to measure the tightness of community boundaries is as follows:

[0173]

[0174] in, The total weight or number of edges in the topological graph. Representing cell nodes With cell nodes The adjacency weight matrix between them and Representing nodes respectively and nodes The degree, and These represent the cluster labels assigned to the nodes. Let Kronecker function be used when hour ,otherwise ; Graph module degree The larger the value, the tighter the intra-cluster connections and the higher the inter-cluster separation.

[0175] The final clustering result is selected based on the graph modularity; among the candidate partitions that meet the target cluster number constraint, the partition with the largest graph modularity is further selected as the final cell clustering result, thus balancing the consistency of the number of clusters and the compactness of the graph community structure.

[0176] Example

[0177] A single-cell clustering process based on reinforcement learning-driven single-cell clustering system includes:

[0178] S1: Collect single-cell sequencing data and perform preprocessing and graph construction to obtain model input features and cell topology.

[0179] S11: Obtain single-cell gene expression data. In this embodiment, the raw cell-gene expression count matrix is ​​obtained from a standard single-cell public database. In the expression matrix, rows represent cells, columns represent genes, and matrix elements represent the expression count value of the corresponding gene in the corresponding cell.

[0180] S12: Data Filtering and Preprocessing. First, low-quality cells and low-expression genes are removed; then, depth normalization and logarithmic transformation are performed on the remaining data; next, the top 4000 highly variable genes are selected to retain the most discriminative expression features; finally, standardization is performed to obtain the cell feature matrix for the input model. This step effectively reduces the impact of sequencing depth differences and noise.

[0181] S13: Constructing a cell topology map. Euclidean distances are calculated for the preprocessed cell features, and a K-nearest neighbor algorithm is used to construct the cell topology map. In this embodiment, the number of K-nearest neighbors is set to 15, thus obtaining a cell topology map characterizing the local structure of the cells. This cell topology map is used for subsequent topological neighbor filtering and graph neural network encoding.

[0182] S2: Cell features are extracted using a dual-flow graph attention network.

[0183] S21: Construct the query encoder and key encoder. Both the query encoder and key encoder are two-layer graph attention network structures with 4 attention heads. The first layer of the graph attention network performs multi-head aggregation on neighborhood information, and the second layer further outputs the hidden layer embedding. The hidden layer dimension is set to 256. Subsequently, the hidden layer embedding is mapped to a 128-dimensional projection space through a multilayer perceptron projector, and normalization is performed before the original view is input into the query encoder. GAT is used to capture node features and network topology information, where the node feature update formula is as follows:

[0184]

[0185] in These are learnable weight parameters. For nodes The neighborhood group, Attention coefficient This is the hidden layer embedding after node updates. Subsequently, it is mapped to a projection space embedding using MLP. .

[0186] S22: The key encoder is updated using a momentum update mechanism. To ensure the stationarity of the key encoder representation, the key encoder parameters are not updated directly through backpropagation, but rather smoothly updated by querying the encoder parameters using a momentum update mechanism. To maintain the consistency of negative sample representations, the key encoder parameters... Parameters from the query encoder Smooth Update:

[0187]

[0188] in The momentum coefficient is taken in this invention. .

[0189] S23: Construct the original view and the enhanced view. The query encoder receives the undisturbed original view, and the key encoder receives the enhanced, perturbed view. Enhancements include node feature masking and edge dropping, with both perturbed probabilities set to 0.2. This dual-view design enables the model to learn a stable and consistent representation under local perturbations.

[0190] S3: Perform topology neighbor filtering.

[0191] S31: Maintain a dynamic negative sample queue. The capacity of the dynamic negative sample queue is set to 4096, which is used to store the projection space features of the historical batch key encoder and projection layer output.

[0192] S32: For the current cell, utilize the constructed adjacency matrix Find its local neighbors and set their dot product similarity in the denominator of the InfoNCE loss to 0. ,make sure This physically shields some of the interference from false negative samples.

[0193] S4: Perform reinforcement learning dynamic sampling control.

[0194] S41: Construct the training state vector. In each training batch, extract a 5-dimensional training state vector containing the positive-negative sample similarity ratio, the normalized graph neighbor consistency score, the false negative sample surrogate rate, the current hard sample sampling ratio, and the loss change trend features to represent the current training state.

[0195] .

[0196] S42: Output the hard sample sampling ratio. The PPO agent outputs action values ​​based on the current training state, which are then mapped to generate the hard sample sampling ratio. The range of the hard sample sampling ratio is limited to 0.15 to 0.35, with a baseline hard sample sampling ratio set to 0.20. This limitation prevents the agent from outputting extreme actions.

[0197] S43: Calculate the multi-objective reward function. The system constructs an immediate reward based on the training feedback of the main model. The immediate reward consists of a loss reduction reward, a feature consistency reward, an inter-class separation reward, a nonlinear false negative sample penalty term, and a proportional regularization term, which are used to comprehensively evaluate the current sampling action.

[0198] .

[0199] Specifically, the loss reduction reward It is constructed based on the amount of loss reduction before and after smoothing (rather than the absolute value of the current loss) to accurately assess the convergence trend of the main model.

[0200] S44: Execute warm-up and safety control. To improve stability in the early stages of training, this embodiment sets the number of warm-up rounds to 10, and a fixed hard sample sampling ratio of 0.15 is used during the warm-up phase. After the warm-up, the PPO agent takes over dynamic sampling control. Simultaneously, the system monitors the false negative sample proxy rate in real time; when it exceeds a preset safety limit, the current hard sample sampling ratio is forcibly reduced to a safety benchmark value to prioritize the stability of the local topology and semantic space.

[0201] S5: Perform clustering and performance evaluation.

[0202] S51: The final hidden layer embedding is output using the query encoder. After model training converges, the final hidden layer embedding output by the query encoder is extracted as the input for downstream clustering to retain richer cell biology information. Community detection is performed on the feature topology graph using the Leiden algorithm, traversing the resolution parameters. The step size is 0.02.

[0203] S52: Select the optimal clustering result. From the candidate partition set that satisfies the target cluster number constraint, calculate its graph modularity and select those that result in a cluster number equal to the biologically accurate target cluster number. The candidate partitions are determined, and the graph module degree of each partition is calculated. :

[0204]

[0205] S53: Select The partitioning scheme with the largest value is taken as the final clustering result.

[0206] Subsequently, the clustering results were quantitatively evaluated using adjusted RAND index and normalized mutual information. The specific calculation formula is as follows:

[0207] S6: The Adjusted Rand Index (ARI) is used to evaluate clustering quality by measuring the consistency of sample pairs in their assignments to predicted clusters and true classes. Its calculation formula is as follows:

[0208]

[0209] in, This indicates that they were simultaneously assigned to the prediction cluster. Compared to the real category The number of cells; and They represent the predicted clusters respectively. and real category The total number of cells in the cell; The total number of cell samples to be clustered; ARI represents the number of combinations of drawing two samples from a given set. The range of values ​​for ARI is... The closer the value is to 1, the higher the match between the predicted cell cluster and the real biological label.

[0210] Normalized mutual information (NMI) is used to measure the statistical correlation between clustering results and true labels from an information theory perspective. Its calculation formula is as follows:

[0211]

[0212] in, Indicates the predicted partitioning result Distinguishing from real labels Mutual information between them; and Let NMI represent the entropy of the two partitioning schemes, respectively. The value range of NMI is... The larger the value, the more complete the true clustering information contained in the clustering results.

[0213] Experimental verification. For example... Figures 2 to 7 As shown.

[0214] A1: Training Dynamics and Sampling Co-analysis. To verify the dynamic control capability of the reinforcement learning module during training, Figure 7 This is a graph showing the dynamic co-change curves of the contrastive loss and the proportion of hard sample samples output by the agent during model training after introducing reinforcement learning-assisted control on the Quake_10x_Limb_Muscle dataset. Figure 7 As shown, the contrastive loss (Loss) and the proportion of hard sample samples in the agent's output are compared during the model's training on the Quake_10x_Limb_Muscle dataset. The graph shows the dynamic co-variance curves of the reinforcement learning controller. As can be seen, the controller is activated after the warm-up phase. With training progressing, the agent can dynamically and adaptively adjust the sampling ratio of difficult samples based on the current loss reduction state, effectively preventing the model from getting stuck in local optima or mode collapse, thus guiding the contrastive loss to continuously and steadily decrease until convergence.

[0215] A2: Qualitative analysis of cell embedding characterization. In terms of qualitative visualization, such as... Figure 2 The image shows a visualization comparison of the UMAP cell embedding representation clustering performance of the model of this invention (RL-MoCo) and the real biological labels (GroundTruth) on representative single-cell datasets (such as biase, Quake_10x_Limb_Muscle, Quake_Smart-seq2_Limb, and Klein). Figure 2 As shown, the cell clusters generated by RL-MoCo have clear boundaries, with similar cells tightly clustered together and different cell types showing obvious separation. The distribution pattern of each cluster is highly consistent with the real label, intuitively demonstrating the ability of this invention to extract high-quality characterization in low-dimensional manifold space.

[0216] A3: Parameter Sensitivity Verification. To evaluate the impact of key hyperparameters on prediction performance, this embodiment establishes different nearest neighbor numbers. With different temperature coefficients The comparison model was used, and the corresponding ARI value was calculated. Figure 6This is a graph showing the parameter sensitivity analysis verification results of this invention on the Quake_10x_Limb_Muscle dataset, including the nearest neighbor number. With temperature coefficient The parameter variation curves showing the impact on model clustering performance (ARI) are shown below. Figure 6 As shown, within the tested range of nearest neighbor numbers, the overall fluctuation of the model's ARI is small, indicating that the present invention has good robustness to the nearest neighbor number parameter k; regarding the temperature coefficient... Within the test range, the model performance remained at a high level overall, among which to The performance within the specified range is relatively good. Considering both model stability and parameter setting consistency, the temperature coefficient in this embodiment is taken as... .

[0217] A4: Comparative Experiment Verification. To verify the superiority of the overall scheme of this invention over existing methods, this embodiment selects representative datasets such as Klein, Quake_10x_Limb_Muscle, Camp, and Quake_Smart-seq2_Lung for comparative experiments with existing single-cell clustering methods. The results are shown in Tables 1 and 2. Figure 3 and Figure 4 As shown, Figure 3 This is a bar chart comparing the performance of the adjusted RAND index (ARI) clustering accuracy of the present invention and existing technologies on different heterogeneous single-cell datasets. Figure 4 The bar chart shows a comparison of the normalized mutual information (NMI) clustering consistency performance of this invention and existing technologies on different heterogeneous single-cell datasets. The results indicate that the method integrating reinforcement learning dynamic sampling and topological neighbor filtering significantly improves clustering performance. Furthermore, the invention performs well on datasets from different sequencing platforms (10x and Smart-seq2) and maintains relatively stable performance across different datasets, demonstrating good robustness and generalization ability.

[0218] A5: Ablation Experiment Verification. To further verify the effectiveness of the reinforcement learning dynamic sampling module and the topological neighbor filtering module in this invention, this embodiment selects the Quake_10x_Limb_Muscle, Quake_SS2_Heart, deng, and Plasschaert datasets, which are more sensitive to module changes, for ablation experiments, as shown in Tables 3 and 4. Figure 5 As shown, removing the reinforcement learning dynamic sampling module (w / oRL) or the topological neighbor filtering module (w / oFNF) both lead to a significant performance drop. This result fully demonstrates the synergistic benefit of "topological neighbor filtering" and "dynamic difficulty sampling" in dealing with the high heterogeneity of single cells. Figure 5This is a bar chart comparing the performance evaluation results of ablation experiments of the present invention on datasets of different complexities.

[0219] Table 1. Comparison of ARI performance in independent tests on different single-cell datasets.

[0220]

[0221] Table 2 Comparison of NMI performance in independent tests on different single-cell datasets

[0222]

[0223] Table 3. Comparison of ARI performance of the model in ablation experiments under different settings.

[0224]

[0225] Table 4 Comparison of NMI performance of the model in ablation experiments under different settings

[0226]

[0227] It should be noted that the specific embodiments are merely explanations and illustrations of the technical solution of the present invention and should not be used to limit the scope of protection. Any modifications made in accordance with the claims and specification of the present invention that are only partial should still fall within the protection scope of the present invention.

Claims

1. A single-cell clustering system based on reinforcement learning, characterized in that, include: Data preprocessing and graph construction module: Obtain the raw single-cell gene expression matrix, perform quality control at the cell and gene levels, and generate a cell feature matrix; Calculate the similarity between cells in the cell feature matrix, construct the K-nearest neighbor topology graph, generate a global adjacency matrix representing the spatial relationship between cells, and obtain the cell topology graph represented by the global adjacency matrix; Random perturbations are applied to the cell feature matrix and the global adjacency matrix respectively to generate their respective enhanced views; the cell feature matrix and cell topology map before the random perturbation are used as the original view; Dual-stream feature coding module: The dual-stream feature encoding module includes a query encoder and a key encoder. The query encoder and the key encoder receive the original view and the enhanced view as input, respectively. First, they generate hidden layer embeddings, and then they are mapped to projective space embeddings through a projection layer. The projective space embeddings are used for similarity calculation in contrastive learning, and the hidden layer embeddings are used for downstream clustering analysis after training convergence. The topological neighbor filtering module maintains a first-in-first-out dynamic negative sample queue. This queue stores the projection space embeddings output by the key encoder in historical batches and serves as a candidate set of negative samples for other cells in subsequent batches. In each batch, with the current cell as the center, the module searches for its topological neighbor samples in the global topological graph based on the global adjacency matrix and retrieves the sample index corresponding to the topological neighbor cell from the negative sample queue. The retrieved topological neighbor samples are then masked in the contrastive loss to prevent them from participating in negative sample competition, resulting in a filtered candidate set of negative samples. The reinforcement learning sampling auxiliary module employs a proximal policy optimization PPO agent as an auxiliary controller for the dynamic sampling module. The policy network outputs parameters of the continuous action distribution based on the current state vector, namely the action mean and variance. The sampling ratio of actual difficult samples in the current batch is determined based on the action or the action mean. Based on the current sampling ratio of difficult samples Difficult samples are selected from the filtered negative sample candidate set to participate in the InfoNCE contrastive loss calculation; Adaptive clustering evaluation module: By introducing a dual constraint mechanism of target cluster number anchoring and graph module degree Q maximization, automatic optimization of clustering partitioning in the high-dimensional hidden layer embedding space is achieved.

2. The single-cell clustering system based on reinforcement learning according to claim 1, characterized in that, The query encoder and key encoder in the dual-stream feature encoding module both adopt a graph attention network structure, and each layer of the graph attention network uses a multi-head attention mechanism to aggregate neighbor node information.

3. The single-cell clustering system based on reinforcement learning according to claim 1, characterized in that, The parameters of the query encoder are in a learnable state, while the gradients of the parameters of the key encoder are truncated. The query encoder is responsible for participating in backpropagation updates, while the key encoder maintains smooth parameter changes through a momentum update mechanism.

4. The single-cell clustering system based on reinforcement learning according to claim 1, characterized in that, The sampling proportion of actual difficulty samples in the current batch is determined based on the action or the average of the actions. In the process, the actual difficulty in sampling proportion Where A represents or , Actions are obtained by sampling from the action distribution. The mean of the action. This is the upper limit ratio. This is the lower limit ratio.

5. A single-cell clustering system based on reinforcement learning according to claim 4, characterized in that, upper limit ratio Set to 0.35, lower limit ratio Set to 0.

15.

6. A single-cell clustering system based on reinforcement learning according to any one of claims 1 to 5, characterized in that, Based on the current sampling ratio of difficult samples The process of selecting difficult samples from the filtered negative sample candidate set to participate in the InfoNCE contrastive loss calculation includes: Extract negative sample candidates from the filtered negative sample candidate set, calculate the similarity between the current cell and the remaining negative sample candidates, and sort them from high to low similarity. The top-N most similar negative samples are selected to form a difficult negative sample set, which is then used to calculate the InfoNCE contrastive loss.

7. A single-cell clustering system based on reinforcement learning according to claim 6, characterized in that, The InfoNCE contrast loss is as follows: in, This indicates the contrast loss corresponding to the current cell; This represents the projection space feature representation obtained after the original view corresponding to the current cell is processed by the query encoder and the projection head. This represents the positive sample projection space feature representation of the same cell corresponding to the current cell, obtained after processing by the key encoder and projection head in the enhanced view; This represents the set of Top-N difficult samples selected in the current batch; Represents a set The first in Projected spatial feature representations corresponding to each difficult sample; This is the temperature coefficient.

8. A single-cell clustering system based on reinforcement learning according to claim 7, characterized in that, The elements of the state vector include the ratio of the average similarity of positive sample pairs to the average similarity of negative sample pairs, the normalized graph neighbor consistency score, the surrogate rate of false negative samples in the previous batch, the sampling ratio of hard samples, and the loss variation characteristics after nonlinear compression; among which, The ratio of the average similarity of positive sample pairs to the average similarity of negative sample pairs. : in, This represents the average similarity of positive sample pairs in the current batch. This represents the average similarity of negative sample pairs in the current batch. To prevent constants with a denominator of zero; Normalized graph neighbor consistency score : in, This represents the set of edges in the cell topology graph of the current batch. Indicates the number of edges; sim( ) represents the similarity function of projected spatial features; and These represent the feature representations of the cell nodes at both ends of the connection in the projection space generated by the query encoder; The proxy rate of the previous batch of spurious negative samples like: in, This represents the total number of difficult samples corresponding to the current cell. It is the projection space feature representation of the current cell output by the query encoder. It is the projected spatial feature representation of the i-th hard sample output by the key encoder. Represents the similarity function of projected space features; This is an indicator function that takes the value 1 if the condition is met, and 0 otherwise; Preset safety threshold; Loss variation characteristics after nonlinear compression : in, This represents the smoothed contrast loss from the previous batch. This represents the smoothing contrast loss for the current batch. This indicates the amount of decrease in smoothing loss. This represents the loss scaling factor.

9. A single-cell clustering system based on reinforcement learning according to claim 8, characterized in that, Proximal policy optimization of the multi-objective reward function during PPO agent updates , As a reward for reduced losses, Rewards are given for feature consistency. As a reward for inter-class separation, This is a non-linear penalty term for false negative samples. A proportional regularization term is used to constrain the agent from falling into extreme sampling strategies; The loss reduction reward , This represents the hyperbolic tangent activation function. As a reward weight, This is the scaling factor; Feature Consistency Reward , As a reward weight; Inter-class separation reward , As weight, and These represent the average similarity between positive and negative sample pairs, respectively. Nonlinear spurious negative sample penalty term , For penalty weighting; Proportional regularization term , For regularization weights, This refers to the proportion of difficult samples sampled, which is the actual proportion of difficult samples obtained from the previous training. ; This indicates the proportion of difficult samples in the preset benchmark. and These represent the upper limit and lower limit of the sampling ratio for difficult samples, respectively.

10. A single-cell clustering system based on reinforcement learning according to claim 8, characterized in that, The reinforcement learning sampling assistance module also includes a safety gate control step, which includes: Real-time monitoring of spurious negative sample proxy rate in each batch When the observed false negative sample proxy rate exceeds the preset safety limit, the sampling ratio of difficult samples output by the PPO agent is restricted, forcibly reducing the current sampling ratio of difficult samples to a safe benchmark value.

Images