An automatic design method of a transcriptome processing scheme based on reinforcement learning

Abstract

The application discloses an automatic design method of a transcript processing scheme based on reinforcement learning, comprising the following steps: constructing a reinforcement learning intelligent agent module, which comprises an intelligent agent operation space and an intelligent agent recurrent neural network; obtaining a transcript processing scheme by sampling based on the two parts; applying the transcript processing scheme to transcript observation data to obtain processed transcript data; inputting the transcript data and independent modal observation data into a reinforcement learning environment module to obtain a quantitative evaluation result of the transcript processing scheme; calculating the parameter gradient of the intelligent agent recurrent neural network based on a reward function of the transcript processing scheme, and updating the intelligent agent recurrent neural network parameters through back propagation; determining whether the maximum number of iterations is reached, and if yes, stopping iteration, and regarding the processing scheme with the highest reward in the iteration process as the final design result. The method can design an optimal transcript processing scheme from scratch in the scene of a specified multi-modal analysis task.

Description

Technical Field

[0001] This invention belongs to the field of transcriptome processing and reinforcement learning technology, and in particular, it is an automatic design method for transcriptome processing schemes based on reinforcement learning. Background Technology

[0002] Advances in transcriptome sequencing technology have spurred the rapid development of various transcriptome analysis methods, such as differential expression analysis, cell category identification, and developmental trajectory inference. As a fundamental element of transcriptome analysis, transcription processing aims to remove technical noise during sequencing, thereby restoring accurate gene expression data with significant biological value. Popular transcriptome processing schemes can be broadly categorized into two types: 1) Expert-led schemes, defined by experts based on prior computational experience, are ordered sequences of operations including scaling, normalization, and feature selection; 2) Deep neural network-led schemes employ probabilistic statistical prior modeling of accurate gene expression or more advanced self-supervised paradigms as the loss function of the neural network, thus constraining the network to restore accurate expression. While both types of transcriptome processing schemes can restore accurate gene expression, the quantitative evaluation of transcriptome processing schemes remains a critical technical challenge due to the difficulty in effectively observing actual gene expression levels. This challenge makes it difficult for researchers to select appropriate transcriptome processing schemes for specific transcriptome analysis tasks.

[0003] Over the past decade, sequencing technology has evolved from initial single-modal transcriptome observation to simultaneous multimodal observation. With the aid of multimodal data, the quantitative evaluation of transcriptome processing protocols has achieved unprecedented breakthroughs; the quality of transcriptome processing protocols can be quantitatively assessed by comparing the consistency of the processed transcriptome with observational data from other independent modalities. Correspondingly, breakthroughs in sequencing technology have presented new challenges to transcriptome analysis methods: a complete transformation from "selecting an appropriate transcriptome processing protocol" to "designing the optimal transcriptome processing protocol de novo for a given multimodal analysis task." This design can best capture the tissue biological information contained in the transcriptome, such as the spatial three-dimensional distribution of cell types; it can also reveal the design patterns of processing protocols used to connect the transcriptome with other modalities, thereby inspiring researchers to design better transcriptome processing protocols. However, researchers in the field currently lack well-tailored tools for designing optimal transcriptome processing protocols de novo for a given multimodal analysis task.

[0004] Therefore, how to provide a method that can design the optimal transcriptional processing scheme from scratch in a given multimodal analysis task is a technical problem that urgently needs to be solved by those in the field. Summary of the Invention

[0005] In view of the above problems, the present invention provides an automatic design method for transcriptome processing schemes based on reinforcement learning that at least solves some of the above technical problems. This method can design the corresponding optimal transcription processing scheme from scratch in a scenario of a specified multimodal analysis task.

[0006] This invention provides an automated design method for transcriptome processing schemes based on reinforcement learning, comprising:

[0007] S1. Construct a reinforcement learning agent module; the reinforcement learning agent module includes an agent computation space and an agent recurrent neural network;

[0008] S2. Initialize the relevant parameters in the agent's recurrent neural network;

[0009] S3. Based on the agent's computational space and agent recurrent neural network, a transcriptome processing scheme is obtained through sampling;

[0010] S4. Apply the transcriptome processing protocol to the transcriptome observation data to obtain the processed transcriptome data.

[0011] S5. Input the processed transcriptome data and independent modality observation data into the reinforcement learning environment module, quantify the degree of consistency between the two and use it as a quantitative evaluation result of the transcriptome processing scheme.

[0012] S6. Calculate the reward function for the transcriptome treatment protocol;

[0013] S7. Calculate the parameter gradient of the agent recurrent neural network based on the reward function obtained in S6, and backpropagate to update the parameters of the agent recurrent neural network.

[0014] S8. Determine whether the maximum number of iterations has been reached. If yes, stop the iterative optimization process and take the processing solution with the highest reward during the iteration as the final design result. If no, return to S3 to continue updating the relevant parameters in the agent's recurrent neural network.

[0015] Furthermore, the transcriptome processing scheme is represented as an ordered operational sequence.

[0016]

[0017] in, S Indicates the transcriptome processing protocol; T The length of the operation sequence is determined by when the agent acquires the terminator; s t express t One-hot encoded operation vector at the location.

[0018] Furthermore, the agent's computation space includes: various scaling, normalization, and feature selection operations, as well as a terminator representing the end of the operation sequence.

[0019] Furthermore, the agent recurrent neural network includes basic time units with shared weights;

[0020] The implementation process of the basic time unit includes:

[0021] Step 1: Given the current sequence position t An encoder is used to perform one-hot encoding operations on the current position into a vector. Encoded as a low-dimensional dense representation:

[0022]

[0023] in, Represents a neural network; Represents a low-dimensional vector. ;

[0024] Step 2: Construct a Long Short-Term Memory (LSTM) network; the input to the LSM network is the current encoding operation vector. Memory of existing operation sequences And its current hidden state The network output is for t Prediction results for +1 positional operation type:

[0025]

[0026] in, Represents a long short-term memory network; h t+1 express t +1 position operation prediction results of latent variables ; This represents the updated sequence memory. ;

[0027] Step 3: Use a decoder to process the data obtained in Step 2. t +1 positional operation latent variable prediction results Interpreted as a one-hot encoded vector:

[0028]

[0029] in, This represents a neural network whose output layer uses the softmax activation function; the output of the decoder network. Provided The probability of setting each dimension to 1; using a type distribution. To model this probability for use based on existing computations Select new operation .

[0030] Furthermore, the sampling process in S3 includes:

[0031] Connect the initial operation s0, the hidden variable h0, and the memory vector c Initialize to the zero vector;

[0032] The agent-based recurrent neural network iteratively selects operations in the transcriptome processing scheme one by one until a terminator is selected, at which point the loop terminates, completing the sampling process. This sampling process can be characterized as follows:

[0033]

[0034] in, S Indicates the transcriptome processing protocol; express i One-hot encoded operation vector at the location; express t The one-hot encoded operation vector at position +1; T Represents the length of the operation sequence; The existing length is t The sequence of operations.

[0035] Furthermore, in S4, the process of applying the transcriptome processing scheme to the transcriptome observation data is characterized as follows:

[0036]

[0037] in, Indicates transcriptome-based processing scheme S The resulting mapping function can map the transcribed data from the original space before processing to the space after processing. X and These represent the transcriptional expression matrices before and after treatment, respectively. , ; M This represents the number of samples in the transcription data, and it remains unchanged before and after processing. N 1. N 2 represents the number of genes before and after treatment, respectively.

[0038] Furthermore, in S5, the process by which the reinforcement learning environment module quantitatively evaluates the transcriptome processing scheme can be characterized as follows:

[0039]

[0040] in, X and These represent the transcriptional expression matrices before and after treatment, respectively. , ; S Indicates the transcriptome processing protocol; Y A matrix or vector representing independent modal observation data; environment function Used to process the transcription expression matrix Mapped to Y The space in which it is located; the evaluation function Used for calculation and Y The degree of consistency between them and the output of quantitative assessment results of transcriptome processing protocols. .

[0041] Furthermore, in step S6, the reward function is expressed as:

[0042]

[0043] in, T Indicates transcriptome processing protocol S The length; if the transcriptome processing protocol S If the processing cannot produce a meaningful result, the reward function will output a negative number. Used for punishment.

[0044] Furthermore, in step S7, the process of calculating the parameter gradient of the agent's recurrent neural network specifically includes:

[0045] Step 1: Calculate the objective function for the agent's recurrent neural network; the objective function is expressed as:

[0046]

[0047] in, This indicates the transcriptome processing protocol obtained through sampling. S Expected value; The representation is defined as follows: The dominant function; b Historical advantage function The exponential moving average; Recurrent neural networks representing intelligent agents All parameters in; This represents the ratio of new to old probabilities in the recurrent neural network used by the agent. Used to Crop to and Between, the function and the minimum function The combination of these parameters is used to limit the update magnitude of the agent's recurrent neural network parameters, thereby accelerating the convergence of the network. This represents a hyperparameter used to balance the contribution of entropy constraints to the target expected value; This represents the probability distribution vector output by the decoder network, where the sum of the given values ​​is 1. Represents probability distribution The corresponding information entropy.

[0048] Step 2: Calculate the parameter gradients of the agent recurrent neural network and backpropagate to update the parameters of the agent recurrent neural network:

[0049]

[0050] in ， Indicates the parameter update step size; This represents the parameters of the recurrent neural network of the agent. Find the gradient.

[0051] Compared with existing technologies, the automatic design method for transcriptome processing schemes based on reinforcement learning described in this invention has the following beneficial effects:

[0052] 1. Achieved the goal of fully automating the de novo design method for transcriptome processing protocols:

[0053] Existing transcriptome processing methods are unable to automatically assemble candidate transcriptome processing operations into meaningful transcriptome processing schemes from scratch. This invention achieves this function for the first time using a reinforcement learning framework.

[0054] 2. The designed product best captures the tissue biological information contained in the transcriptome:

[0055] This invention employs a reinforcement learning framework with a learning mechanism that maximizes the expected value of the reward function. The reward function, defined as the degree of consistency between the processed transcriptome data and other independent modality observation data, encourages the designed product to best capture the tissue biological information contained in the transcriptome, thereby achieving a breakthrough compared to existing technologies.

[0056] 3. It can reveal design patterns for processing schemes used to connect the transcriptome with other modalities:

[0057] By comparing the design products of different datasets under the same embodiment and the design products under different embodiments, and analyzing their commonalities and differences, the design patterns of processing schemes used to connect the transcriptome with other modalities can be revealed, thereby inspiring relevant technicians to design better transcriptome processing schemes.

[0058] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0059] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0060] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0061] Figure 1 This is a schematic diagram of the automatic design method for transcriptome processing schemes based on reinforcement learning provided in an embodiment of the present invention.

[0062] Figure 2 This is a schematic diagram illustrating the design principle of the transcriptome processing scheme for the targeted electrophysiological feature prediction task provided in this embodiment of the invention.

[0063] Figure 3 This is a schematic diagram illustrating the design principle of the transcriptome processing scheme for the targeted protein expression prediction task provided in this embodiment of the invention.

[0064] Figure 4 This is a schematic diagram illustrating the design principle of the transcriptome processing scheme for the targeted histological phenotype recognition task provided in this embodiment of the invention. Detailed Implementation

[0065] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0066] See Figure 1 As shown, this embodiment of the invention provides an automatic design method for transcriptome processing schemes based on reinforcement learning, specifically including the following steps:

[0067] S1. Construct a reinforcement learning agent module; the reinforcement learning agent module includes an agent computation space and an agent recurrent neural network;

[0068] S2. Initialize the relevant parameters in the agent's recurrent neural network;

[0069] S3. Based on the agent's computational space and agent recurrent neural network, a transcriptome processing scheme is obtained through sampling;

[0070] S4. Apply the transcriptome processing protocol to the transcriptome observation data to obtain the processed transcriptome data.

[0071] S5. Input the processed transcriptome data and independent modality observation data into the reinforcement learning environment module, quantify the degree of consistency between the two and use it as a quantitative evaluation result of the transcriptome processing scheme.

[0072] S6. Calculate the reward function for the transcriptome treatment protocol;

[0073] S7. Calculate the parameter gradient of the agent recurrent neural network based on the reward function obtained in S6, and backpropagate to update the parameters of the agent recurrent neural network.

[0074] S8. Determine whether the maximum number of iterations has been reached. If yes, stop the iterative optimization process and take the processing solution with the highest reward during the iteration as the final design result. If no, return to S3 to continue updating the relevant parameters in the agent's recurrent neural network.

[0075] The following sections will provide a detailed explanation of each of the above steps.

[0076] In S1 above, the agent's computational space includes various types of scaling, normalization, and feature selection operations developed for transcriptome processing, as well as a terminator representing the end of the computation sequence. This invention employs one-hot encoded vectors. Represents an operation, where H This represents the total number of operations in the computation space. A transcriptome processing scheme (also called a computational sequence) is defined in this invention as an ordered sequence of operations, i.e., the transcriptome processing scheme is expressed as:

[0077]

[0078] in, T The length of the operation sequence is determined by when the agent acquires the terminator (this invention defines the maximum length of the operation sequence as...). ); s t express t One-hot encoded operation vector at the location;

[0079] The basic time unit for weight sharing in an agent recurrent neural network aims to learn conditional probability distributions. This enables the selection of new operations based on existing operations, where... The existing length is The sequence of operations; This represents the operation at the next position. Specifically, the implementation process of the basic time unit in an agent's recurrent neural network is as follows:

[0080] Step 1: Given the current sequence position t This invention employs an encoder to perform one-hot encoding operations on the current position into a vector. Encoded as a low-dimensional dense representation:

[0081]

[0082] in, It is a neural network; It is a low-dimensional vector;

[0083] Step 2: Use a Long Short-Term Memory (LSTM) network, whose input is the computations performed so far ( ), memory of existing operation sequences ( ) and the current hidden state ( The network output is for t Prediction results for +1 positional operation type:

[0084]

[0085] in, This represents a long short-term memory network; yes t +1 position operation prediction results of latent variables; It is an updated sequence memory (i.e., it includes the latest addition operation). (memories)

[0086] Step 3: This invention uses a decoder to process the data obtained in Step 2. t +1 positional operation latent variable prediction results Interpreted as a one-hot encoded vector:

[0087]

[0088] in, It is a neural network whose output layer uses the softmax activation function; the output of the decoder network... Provided The probability of setting each dimension to 1; this invention uses a type distribution. To model this probability for use based on existing computations Select new operation .

[0089] In S3 above, the sampling process of the transcriptome processing protocol includes:

[0090] Connect the initial operation (s0), the hidden variable (h0), and the memory ( c 0) Initialize the vector to zero;

[0091] The agent-based recurrent neural network iteratively selects operations in the transcriptome processing scheme one by one until a terminator is selected, at which point the loop terminates, completing the sampling process. This sampling process can be characterized as follows:

[0092]

[0093] in, S This represents a transcriptome processing scheme modeled as an ordered operational sequence; express i One-hot encoded operation vector at the location; express t The one-hot encoded operation vector at position +1; T Represents the length of the operation sequence; The existing length is The sequence of operations.

[0094] In S4 above, the process of applying the transcriptome processing protocol to transcriptome observation data can be characterized as follows:

[0095]

[0096] in, Transcriptome processing schemes modeled as ordered operational sequences S The corresponding mapping function maps the transcribed data from its original space before processing to its processed space. and These represent the transcriptional expression matrices before and after treatment, respectively. M This represents the number of samples in the transcription data, and it remains unchanged before and after processing. N 1. N 2 represents the number of genes (feature dimension) before and after treatment, respectively, and satisfies the following conditions: In S5 above, the process of quantitatively evaluating transcriptome processing protocols using the reinforcement learning environment module can be characterized as follows:

[0097]

[0098] in, Y It is a matrix or vector derived from independent modalities other than the transcriptome; environment function Used to process the transcription expression matrix Mapped to Y The space in which it is located; the evaluation function Used for calculation and YThe degree of consistency between them and the output of quantitative assessment results of transcriptome processing protocols. .if Y If the labels are different samples, then the environment function... It will be a classifier and a consistency index Use positive classification indicators; if Y If it is the feature matrix that guides the processing, then the environment function... It will be a regressor and a consistency index A positive regression indicator was used.

[0099] In S6 above, the reward function is defined in this invention as an extended version of the quantitative evaluation index in S5, used to avoid constructing extremely lengthy processing sequences and handling unreasonable outputs (such as NaN). and Penalize the operation sequence of )

[0100]

[0101] in, T It is a transcriptome processing scheme modeled as an ordered operational sequence. S The length of the reward function is reduced, and it can decrease the value of the reward function to encourage shorter operation sequences; if S If the processing cannot produce a meaningful result, the reward function will output a negative number. Used for punishment.

[0102] In S7 above, the parameter gradient calculation process of the agent recurrent neural network includes:

[0103] Step 1: Calculate the objective function value for the parameters of the agent's recurrent neural network. This invention defines the objective function as a combination of the standard PPO-Clip and additional entropy constraints:

[0104]

[0105] in, This refers to the processing scheme obtained from sampling from the agent during training. S Expected value; The representation is defined as follows: Advantage function ( b It is a historical advantage function The exponential moving average is used to reduce the variance of the gradient estimation process. Recurrent neural networks representing intelligent agents All parameters in; The ratio of new to old probabilities in a recurrent neural network used by an agent can be used to improve the sampling efficiency of reinforcement learning. Used to Crop to and Between, the function and the minimum function The combination of these parameters is used to limit the update magnitude of the agent's recurrent neural network parameters, thereby accelerating the convergence of the network. This represents the probability distribution vector whose sum is 1, output by the aforementioned decoder network; an additional information entropy constraint is introduced into the objective function of this invention. This is used to encourage intelligent agents to fully explore the computational space; It is a hyperparameter used to balance the contribution of entropy constraints to the target expected value;

[0106] Step 2: Calculate the parameter gradients of the agent recurrent neural network and backpropagate to update the parameters of the agent recurrent neural network:

[0107]

[0108] in ， Indicates the parameter update step size; This represents the parameters of the recurrent neural network of the agent. Find the gradient.

[0109] The present application will be further described in detail below through three specific embodiments.

[0110] Example 1: Design of a transcriptome processing scheme for a targeted electrophysiological feature prediction task

[0111] See appendix Figure 2 This invention discloses a schematic diagram of the design principle of a transcriptome processing scheme for a targeted electrophysiological feature prediction task. In practical biological applications, the Patch-seq sequencing platform exhibits strong heterogeneity between the transcriptome and electrophysiological feature modalities observed simultaneously in a single cell, requiring a high degree of consistency between the processed transcriptome and electrophysiological features. This invention proposes a reinforcement learning framework that can design, de novo, a processing scheme to best capture the electrophysiological biological information contained in the transcriptome. The method specifically includes:

[0112] S1. Construct a reinforcement learning agent module, which includes an agent computation space and a recurrent neural network.

[0113] S2. Randomly initialize the parameters of the agent's recurrent neural network;

[0114] S3. Based on the intelligent agent's computational space and recurrent neural network, a transcriptome processing scheme is obtained through sampling.

[0115] S4. Apply the transcriptome processing protocol to the transcriptome expression data observed on the Patch-seq sequencing platform to obtain processed transcriptome data.

[0116] S5. The processed transcriptome data and the electrophysiological feature data observed simultaneously by the Patch-seq sequencing platform are sent into the reinforcement learning environment module to quantify the degree of consistency between the two and use it as a quantitative evaluation result of the transcriptome processing scheme.

[0117] S6. Calculate the reward function for the transcriptome treatment protocol;

[0118] S7. Calculate the parameter gradient of the agent network based on the reward function obtained in S6, and backpropagate to update the agent network parameters.

[0119] S8. Determine whether the maximum number of iterations has been reached. If yes, stop the iterative optimization process and consider the processing scheme with the highest reward during the iteration process as the final design result of the electrophysiological feature prediction task. If no, return to S3 to continue updating the parameters of the agent network.

[0120] In this embodiment, the reinforcement learning environment module in step S5 Y It is the electrophysiological characteristic matrix and the environmental function. It is a K-nearest neighbor regression predictor (KNN, K=20), evaluation function It is a regression index of negative root mean square error after exponential transformation. This indicator is Positive numbers within the interval.

[0121] Example 2: Design of a transcriptome processing scheme for a targeted protein expression prediction task

[0122] See appendix Figure 3 This invention discloses a schematic diagram of the design principle of a transcriptome processing scheme for a targeted protein expression prediction task. In practical biological applications, the CITE-seq sequencing platform exhibits strong heterogeneity between the transcriptome and membrane protein expression modalities observed simultaneously in single cells, requiring a high degree of consistency between the processed transcriptome and membrane protein expression data. This embodiment proposes a reinforcement learning framework to design, de novo, a processing scheme that best captures the biological information of membrane protein expression contained in the transcriptome. The method specifically includes:

[0123] S1. Construct a reinforcement learning agent module, which includes an agent computation space and a recurrent neural network.

[0124] S2. Randomly initialize the parameters of the agent's recurrent neural network;

[0125] S3. Based on the intelligent agent's computational space and recurrent neural network, a transcriptome processing scheme is obtained through sampling.

[0126] S4. Apply the transcriptome processing protocol to the transcriptome expression data observed on the CIE-seq sequencing platform to obtain processed transcriptome data.

[0127] S5. The processed transcriptome data and the membrane protein expression data observed simultaneously by the CITE-seq sequencing platform are sent into the reinforcement learning environment module to quantify the degree of consistency between the two and use it as a quantitative evaluation result of the transcriptome processing scheme.

[0128] S6. Calculate the reward function for the transcriptome treatment protocol;

[0129] S7. Calculate the parameter gradient of the agent network based on the reward function obtained in S6, and backpropagate to update the agent network parameters.

[0130] S8. Determine whether the maximum number of iterations has been reached. If yes, stop the iterative optimization process and consider the processing scheme with the highest reward during the iteration process as the final design result of the protein expression prediction task. If no, return to S3 to continue updating the agent network parameters.

[0131] In this embodiment, the reinforcement learning environment module in step S5 Y This is the protein expression matrix; the rest of the settings are similar to the environment configuration in Example 1, including the environment function. It is a K-nearest neighbor regression predictor (KNN, K=20), evaluation function It is a regression index of negative root mean square error after exponential transformation. This indicator is Positive numbers within the interval.

[0132] Example 3: Design of a transcriptome processing scheme for a targeted histological phenotype recognition task

[0133] See appendix Figure 4 This invention discloses a schematic diagram of the design principle of a transcriptome processing scheme for a histological phenotype recognition task. For practical biological applications, the transcriptome of tiny tissue units (which may contain multiple cells) in tissue sections is observed using a 10X Visium sequencing platform. Then, experts label the histological phenotype (different layers of the cerebral cortex) to which the tiny tissue units belong based on the morphological characteristics of the tissue sections, thereby achieving simultaneous observation of both transcriptome and histological phenotype modalities.

[0134] Considering the strong heterogeneity between transcriptome and histological phenotype data, and the requirement for high consistency between the processed transcriptome and histological phenotype, this embodiment proposes a reinforcement learning framework to design, de novo, a processing scheme that best captures the histological phenotype biological information contained in the transcriptome. The method specifically includes:

[0135] S1. Construct a reinforcement learning agent module, which includes an agent computation space and a recurrent neural network.

[0136] S2. Randomly initialize the parameters of the agent's recurrent neural network;

[0137] S3. Based on the intelligent agent's computational space and recurrent neural network, a transcriptome processing scheme is obtained through sampling.

[0138] S4. The transcriptome processing protocol was applied to the transcriptome expression data obtained by the 10X Visium sequencing platform, which was based on micro-organisms, to obtain the processed transcriptome data.

[0139] S5. The processed transcriptome data and the histological phenotypic labels manually annotated by experts based on the morphological information of tissue sections are sent into the reinforcement learning environment module to quantify the degree of consistency between the two and use it as a quantitative evaluation result of the transcriptome processing scheme.

[0140] S6. Calculate the reward function for the transcriptome treatment protocol;

[0141] S7. Calculate the parameter gradient of the agent network based on the reward function obtained in S6, and backpropagate to update the agent network parameters.

[0142] S8. Determine whether the maximum number of iterations has been reached. If yes, stop the iterative optimization process and consider the processing scheme with the highest reward during the iteration process as the final design result of the histological phenotype recognition task. If no, return to S3 to continue updating the agent network parameters.

[0143] In this embodiment, the reinforcement learning environment module in step S5 Y It is a histological phenotypic label vector, an environment function. It is a nearest neighbor classifier (NCC), and its evaluation function is... It is the median of the F1 score for different categories of classification indicators. Positive numbers within the interval.

[0144] This invention describes an automated design method for transcriptome processing schemes based on reinforcement learning. This method employs a reinforcement learning framework to automatically assemble candidate transcriptome processing operations from scratch into meaningful transcriptome processing schemes. These schemes optimally reflect the tissue biological information contained within the transcriptome, such as the spatial and three-dimensional distribution of cell types. Simultaneously, they reveal design patterns for processing schemes that connect the transcriptome to other modalities, thereby bridging the gap between transcriptome processing and multimodal analysis, achieving the goal of fully automated de novo design of transcriptome processing schemes.

[0145] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An automated design method for transcriptome processing schemes based on reinforcement learning, characterized in that, include: S1. Construct a reinforcement learning agent module; the reinforcement learning agent module includes an agent computation space and an agent recurrent neural network; S2. Initialize the relevant parameters in the agent's recurrent neural network; S3. Based on the agent's computational space and agent recurrent neural network, a transcriptome processing scheme is obtained through sampling; S4. Apply the transcriptome processing protocol to the transcriptome observation data to obtain the processed transcriptome data. S5. Input the processed transcriptome data and independent modality observation data into the reinforcement learning environment module, quantify the degree of consistency between the two and use it as a quantitative evaluation result of the transcriptome processing scheme. S6. Calculate the reward function for the transcriptome treatment protocol; S7. Calculate the parameter gradient of the agent recurrent neural network based on the reward function obtained in S6, and backpropagate to update the parameters of the agent recurrent neural network. S8. Determine whether the maximum number of iterations has been reached. If yes, stop the iterative optimization process and take the processing solution with the highest reward during the iteration as the final design result. If no, return to S3 to continue updating the relevant parameters in the agent's recurrent neural network. The transcriptome processing scheme is represented as an ordered sequence of operations: Where S represents the transcriptome processing scheme; T represents the length of the operational sequence and is determined by when the agent collects the terminator; s t This represents the one-hot encoded operation vector at position t; The sampling process in S3 includes: Initialize the initial operation s0, the hidden variable h0, and the memory vector c0 to zero vectors; The agent-based recurrent neural network iteratively selects operations in the transcriptome processing scheme one by one until a terminator is selected, at which point the loop terminates, completing the sampling process. This sampling process can be characterized as follows: Where S represents the transcriptome processing protocol; S i The vector representing the one-hot encoding operation at position i; s t+1 This represents the one-hot encoded operation vector at position t+1; T represents the length of the operation sequence; (S i ) i≤t This represents an existing sequence of operations of length t; In S5, the process of quantitatively evaluating transcriptome processing schemes by the reinforcement learning environment module can be characterized as follows: Among them, X and These represent the transcriptional expression matrices before and after treatment, respectively. S represents the transcriptome processing protocol; Y represents the matrix or vector of independent modality observation data; the environment function f Env (·) is used to process the transcription expression matrix. Mapping to the space where Y resides; evaluating the function f Eval (·) is used for calculation The degree of consistency between X and Y is determined, and the quantitative assessment results of the transcriptome processing protocol r(X,X,Y) are output. In step S6, the reward function is expressed as: Where T represents the length of the transcriptome treatment protocol S; if the transcriptome treatment protocol S fails to produce a meaningful result, the reward function will output a negative number -r. penalty (r penalty >0) is used for punishment.

2. The automatic design method for transcriptome processing schemes based on reinforcement learning as described in claim 1, characterized in that, The agent's computation space includes various scaling, normalization, and feature selection operations, as well as a terminator representing the end of the operation sequence.

3. The automatic design method for transcriptome processing schemes based on reinforcement learning as described in claim 1, characterized in that, The agent recurrent neural network includes a basic time unit with shared weights; The implementation process of the basic time unit includes: Step 1: Given the current sequence position t, use an encoder to perform a one-hot encoding operation on the vector s of the current position. t Encoded as a low-dimensional dense representation: Among them, f Encoder (·) represents a neural network; Represents a low-dimensional vector. Step 2: Construct a Long Short-Term Memory (LSTM) network; the input to the LSM network is the current encoding operation vector. Memory of existing operation sequences c t And the current hidden state h t The network output is the prediction result for the operation type at position t+1: Among them, f LSTM (·) indicates Long Short-Term Memory (LSTM) network; h t+1 This represents the prediction result of the latent variable operation at position t+1. c t+1 This represents the updated sequence memory. Step 3: Use a decoder to calculate the prediction result h of the latent variable at position t+1 obtained in Step 2. t+1 Interpreted as a one-hot encoded vector: d t+1 =f Decoder (h t+1 ) s t+1 ~P(s t+1 |(s i ) i≤t )=Cat(d t+1 ) Among them, f Decoder (·) indicates a neural network whose output layer uses the softmax activation function; the output d of the decoder network t+1 ∈R H Provided s t+1 The probability of each dimension being set to 1 is used; a type distribution Cat(·) is employed to model this probability for use in computations based on existing operations (S). i ) i≤t Select new operation S t+1 .

4. The automatic design method for transcriptome processing schemes based on reinforcement learning as described in claim 1, characterized in that, In S4, the process of applying the transcriptome processing scheme to the transcriptome observation data is characterized as follows: Where S(·) represents the mapping function obtained based on the transcriptome processing scheme S, which maps the transcriptome data from the original space before processing to the space after processing; X and These represent the transcriptional expression matrices before and after treatment, respectively. M represents the number of transcribed data samples, which remains unchanged before and after treatment; N1 and N2 represent the number of genes before and after treatment, respectively.

5. The automatic design method for a transcriptome processing scheme based on reinforcement learning as described in claim 3, characterized in that, In step S7, the process of calculating the parameter gradient of the agent's recurrent neural network specifically includes: Step 1: Calculate the objective function for the agent's recurrent neural network; the objective function is expressed as: in, A(X, S, Y) represents the expected value of the transcriptome processing scheme S obtained through sampling; A(X, S, Y) is defined as r * The dominance function is (X, S, Y)-b; b represents the exponential moving average of the historical dominance function A(X, S, Y); θ represents the agent recurrent neural network {f Encoder (·), f LSTM (·), f Decoder All parameters in (·)}; ρ(θ) represents the ratio of new to old probabilities in the agent recurrent neural network; clip(·) is used to clip ρ(θ) to between 1-ε and 1+ε. The combination of this function and the minimization function min(·) is used to limit the update magnitude of the agent recurrent neural network parameters, thereby accelerating the convergence of the network; γ>0 represents a hyperparameter used to balance the contribution of entropy constraints to the target expected value; d t Entropy(·) represents the probability distribution vector output by the decoder network, where the sum of the digits is 1; · represents the probability distribution d. t The corresponding information entropy; Step 2: Calculate the parameter gradients of the agent recurrent neural network and backpropagate to update the parameters of the agent recurrent neural network: Where ξ>0 represents the parameter update step size; This indicates that the gradient of the parameters θ of the agent's recurrent neural network is calculated.

Images