Cancer patient gene methylation characteristic screening method based on DDPG reinforcement learning

Abstract

The invention discloses a cancer patient gene methylation feature screening method based on DDPG reinforcement learning. According to the method, a depth deterministic strategy gradient algorithm (DDPG) is combined with gene methylation data, and important features closely related to cancer classification are accurately screened out from massive gene methylation data. The system dynamically selects gene methylation features through effective exploration of a continuous state space and an action space, and optimizes a feature selection strategy through a reward mechanism of reinforcement learning. According to the method, the accuracy of a cancer classification model can be remarkably improved, and powerful support can be provided for early detection and personalized treatment of cancers by analyzing the screened features. According to the method, the challenge of high-dimensional gene data feature selection is effectively solved, the data processing efficiency in biomedical research is improved, and the method has wide application prospects and practical value.

Description

Technical Field

[0001] The present invention relates to the fields of bioinformatics, machine learning and medical technology, and in particular to a method for screening gene methylation features of cancer patients based on DDPG reinforcement learning. Background Art

[0002] Gene methylation has been widely studied as a key biomarker in the early detection and personalized treatment of cancer. Abnormal methylation patterns are often closely related to the development of cancer, so accurately screening out features related to cancer classification is crucial to improving the diagnosis and treatment effects. However, gene methylation data usually contains a large number of features, making it difficult for traditional feature selection methods to effectively screen in high-dimensional data. Traditional methods such as principal component analysis (PCA) or linear discriminant analysis (LDA) may not be able to fully explore the complex relationships in the data or handle the computational complexity of high-dimensional data. In recent years, the rapid development of deep learning and reinforcement learning techniques has provided new ideas for feature selection. The deep deterministic policy gradient (DDPG) algorithm, as an advanced reinforcement learning technique, can perform efficient policy optimization in continuous action space. This algorithm dynamically adjusts the feature selection strategy through a reward mechanism, so that the most predictive features can be gradually mined in high-dimensional data, thereby improving the classification accuracy and generalization ability of the model. The feature selection method based on DDPG is expected to solve the shortcomings of existing methods in high-dimensional data processing and achieve more accurate and efficient feature screening. Summary of the invention

[0003] This paper proposes a method for screening gene methylation features of cancer patients based on DDPG reinforcement learning. By effectively exploring and optimizing feature selection strategies, it aims to improve the accuracy of cancer classification models and provide support for early cancer detection and personalized treatment. This method is expected to solve the challenges in high-dimensional data processing and play an important role in biomedical research.

[0004] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a method for screening gene methylation features of cancer patients based on DDPG reinforcement learning, which specifically includes the following modules: a data preprocessing module, a DDPG reinforcement learning module, a feature extraction module, and a high-risk group and survival time classification module;

[0005] The data preprocessing module is used for data preprocessing, specifically dividing all data into two parts: methylation degree data and survival period and survival status data, and sending the preprocessed methylation degree data to the DDPG reinforcement learning module;

[0006] The DDPG reinforcement learning module is used to obtain a value matrix using a reinforcement learning method, specifically to select the value matrix using a DDPG reinforcement learning algorithm, and send the grouping results to a feature extraction module;

[0007] The feature extraction module is used to extract features from the pre-processed methylation degree data, specifically, multiplying the methylation degree matrix of each cancer patient by a value matrix to obtain a risk value matrix for each patient, arranging them in descending order of risk value, and then transmitting them to the high-risk group and low-risk group and survival time classification module;

[0008] The high- and low-risk groups and survival time classification module is used to divide the data after descending order into a high-risk group and a low-risk group from the middle, and use the log-rank test to compare the survival curves of the high- and low-risk groups respectively, and calculate the p-value. If the p-value is less than 0.05, the model is considered to be meaningful.

[0009] Furthermore, the DDPG reinforcement learning module in the cancer patient gene methylation feature screening method based on DDPG reinforcement learning specifically includes the following steps:

[0010] 1) Initialize the reinforcement learning network parameters and clear the playback buffer;

[0011] 2) Initialize the current state of the risk value of the cancer patient;

[0012] 3) Get actions based on the current state in the Actor network;

[0013] 4) Execute the action and apply it to the feature extraction module to obtain the next state and corresponding reward and determine whether to terminate the state;

[0014] 5) Store the current state, action, next state, reward, and whether the state is terminated in the playback buffer;

[0015] 6) Randomly sample a batch of data from the playback buffer to update the parameters of the reinforcement learning network;

[0016] 7) Repeat steps 3) to 6) until the training is terminated.

[0017] Furthermore, the gene methylation feature screening method for cancer patients based on DDPG reinforcement learning uses the DDPG (Deep Deterministic Policy Gradient) algorithm for automatic optimization and obtains the value matrix, which specifically includes:

[0018] The DDPG algorithm combines deterministic strategies with deep Q networks and uses the Actor-Critic architecture to learn deterministic strategies, which enables it to effectively handle problems in continuous action spaces and improve the robustness and stability of the system.

[0019] The core network structure of DDPG includes the Actor network and the Critic network. The Actor network is responsible for generating continuous actions to respond to the system state. The Critic network evaluates the impact of the actions output by the Actor network on the system performance and optimizes it through the learning value function. The Actor network generates actions through the environment state, noise and strategy function. The specific formula is as follows:

[0020] a t =μ θ (s t )+N;

[0021] Among them, the environmental state S t , noise N and strategy function μ θ (·) Generate a t For action, s t is the environmental state, the noise is N, μ θ (·) is the policy function;

[0022] Action a t Interact with the environment to obtain a new state s t+1 and reward r t ; Use the policy gradient function to update the Actor online network parameters. The formula is as follows:

[0023]

[0024] in, is the gradient operator, J is the expected value of the cumulative return based on the current strategy parameters, N' is the number of samples, Q ω (·) is the online Q-value function;

[0025] The Critic network includes the Critic online network and the Critic target network; the Critic online network generates an online Q value Q according to the online Q value function. ω (s t ,a t ), where ω represents the Critic online network parameter; the Critic target network calculates the target Q value according to the target Q value function, and the formula is as follows:

[0026]

[0027] Among them, y i is the target Q value, γ is the reward discount factor, Q ω′(·) is the target Q-value function ω′ is the Critic target network parameter;

[0028] The DDPG algorithm uses the loss function to update the online network parameters. The formula is as follows:

[0029]

[0030] Where N” is the number of samples;

[0031] The critic target network parameters are periodically copied and updated through a soft update method, and the formula is as follows:

[0032] ω′=τω+(1-τ)ω′;

[0033] Where τ is the update rate.

[0034] Furthermore, the feature extraction module in the cancer patient gene methylation feature screening method based on DDPG reinforcement learning specifically includes the following steps:

[0035] Get the risk value of each patient, and then arrange the risk values ​​of all patients in descending order into the risk value matrix. The specific process is as follows:

[0036] The methylation data matrix of each patient is multiplied by a value matrix to obtain the risk value of each patient. The calculation formula is as follows:

[0037] M i =X i ·V;

[0038] H=(h′ 1 ,h' 2 ,…,h′ i ,…,h' n-1 ,h' n ),h′ 1 ≥h' 2 ≥…≥h′ i ≥…≥h' n-1 ≥h' n ;

[0039] Where X i The methylation data matrix of the i-th patient, V is the value matrix, M i is the risk matrix of the ith patient, and H is the risk value matrix that arranges all risk values ​​in descending order.

[0040] The beneficial effect of the present invention is that by introducing the DDPG reinforcement learning algorithm, features closely related to cancer can be accurately screened out in high-dimensional gene methylation data, significantly improving the accuracy of the cancer classification model. In addition, the method improves data processing efficiency through dynamic feature selection and optimization strategies, overcomes the limitations of traditional methods in high-dimensional data, provides strong technical support for early cancer detection and personalized treatment, and has broad application prospects and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the system flow chart.

[0042] Figure 2 Calculate a flow chart for the risk matrix.

[0043] Figure 3 The flowchart of risk matrix training based on DDPG. DETAILED DESCRIPTION

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0045] Example:

[0046] like Figure 1 As shown: A method for screening gene methylation features of cancer patients based on DDPG reinforcement learning, including the following modules: data preprocessing module, DDPG reinforcement learning module, feature extraction module and high-risk and low-risk group and survival time classification module;

[0047] The data preprocessing module is used for data preprocessing, specifically dividing all data into two parts: methylation degree data and survival period and survival status data, and sending the preprocessed methylation degree data to the DDPG reinforcement learning module;

[0048] The DDPG reinforcement learning module is used to obtain a value matrix using a reinforcement learning method, specifically to select the value matrix using a DDPG reinforcement learning algorithm, and send the grouping results to a feature extraction module;

[0049] The feature extraction module is used to extract features from the pre-processed methylation degree data, specifically, multiplying the methylation degree matrix of each cancer patient by a value matrix to obtain a risk value matrix for each patient, arranging them in descending order of risk value, and then transmitting them to the high-risk group and low-risk group and survival time classification module;

[0050] The high- and low-risk groups and survival time classification module is used to divide the data after descending order into a high-risk group and a low-risk group from the middle, and use the log-rank test to compare the survival curves of the high- and low-risk groups respectively, and calculate the p-value. If the p-value is less than 0.05, the model is considered to be meaningful.

[0051] like Figure 2 As shown: The feature extraction module in the cancer patient gene methylation feature screening method based on DDPG reinforcement learning is used to extract features from the preprocessed methylation degree data. The methylation degree matrix is ​​the methylation degree of each cancer patient with a rows and b columns, and the value matrix is ​​b rows and one column. The two matrices are multiplied to obtain a risk value matrix with a rows and one column. The risk value matrix is ​​the risk value of each cancer patient.

[0052] like Figure 3 As shown: Risk matrix training flow chart based on DDPG, the steps are as follows:

[0053] 1) Medical environment:

[0054] The patient's genetic data and clinical information are modeled as state s, and the reward r is defined as an evaluation indicator based on survival differences, such as the extension of the patient's survival or improved prognosis. By adjusting the feature combination, the system generates the next state s', which is the adjusted patient data. This state transition process is the exploration link in reinforcement learning, which aims to identify key features related to cancer classification.

[0055] 2) Actor Network:

[0056] The Actor network is the core neural network used for decision-making. The input of the network is the medical state s, that is, the patient's current genetic data and medical information, and the output is the feature combination a, which represents the gene methylation features selected in the current state. This step is to learn the feature combination that should be selected in different states through network training, so as to gradually optimize the decision-making process.

[0057] 3) Gene feature screening:

[0058] In this step, the system screens the gene methylation features based on the feature combination generated by the Actor network. In this process, in addition to screening out features closely related to cancer, further etiology analysis can be performed to identify key genetic factors that may cause cancer, providing a basis for personalized treatment.

[0059] 4) Critic Network:

[0060] The Critic network evaluates the effectiveness of the feature combination selected by the Actor network. The input is the current medical state s and the feature combination a, and the output is the corresponding Q value Q(s, a), which represents the long-term reward that can be obtained by selecting this feature combination under the current state. Through the feedback of the Q value, the Critic network guides the feature selection of the Actor network, so that it continuously adjusts towards the optimal feature combination.

[0061] 5) Medical Risk Matrix:

[0062] In this step, the system calculates the corresponding risk value based on the feature combination (action) generated by the Actor network. The medical risk matrix contains multiple actions and their corresponding risk assessment values, for example:

[0063] - Action 1, the corresponding risk value is 0.8;

[0064] - Action 2, the corresponding risk value is 0.3;

[0065] - Action 3, the corresponding risk value is 0.5.

[0066] This matrix is ​​used to evaluate the impact of different feature combinations on patient prognostic risk and provide a reference for the next step of optimization.

[0067] 6) Experience Replay:

[0068] The experience replay mechanism is used to improve the training efficiency and stability of the model. The system stores the state s, feature combination a, reward r and next state s′ of each step, and randomly extracts these stored samples for model training in subsequent training. This experience replay mechanism can break the correlation between training samples and enhance the generalization ability of the model.

[0069] In addition, experience replay can be further extended to the target critic network. The target critic network receives the current medical state s and feature combination a, and outputs the corresponding Q value Q(s, a). This Q value is used to guide the next step of the optimization process.

[0070] 7) Target Actor Network:

[0071] The input of the target Actor network is the medical state s, and the output is the feature combination a in the current state. This network is used to evaluate the long-term performance of the Actor network and plays a stabilizing role in the model training process. By updating the target network, the system can avoid drastic policy fluctuations and ensure steady optimization of feature selection.

[0072] 8) Training process:

[0073] During the training process, the Critic network is first trained using the experience replay mechanism, and the Actor network is updated through the gradient back propagation of the Critic network. In addition, the system regularly updates the target Actor and target Critic network to ensure the stability of the model. The entire training process continuously adjusts the feature combination and optimization strategy to ultimately achieve effective improvement of the cancer classification model.

Claims

1. A method for screening gene methylation features of cancer patients based on DDPG reinforcement learning, characterized in that: include: Data preprocessing module, DDPG reinforcement learning module, feature extraction module and high-risk and low-risk group and survival time classification module; among them: The data preprocessing module is used for data preprocessing, dividing all data into two parts: methylation degree data and survival period and survival status data, and then sending the preprocessed methylation degree data to the DDPG reinforcement learning module; The DDPG reinforcement learning module is used to obtain a value matrix using a reinforcement learning method, select the value matrix using a DDPG reinforcement learning algorithm, and send the grouping results to a feature extraction module; The feature extraction module is used to extract features from the pre-processed methylation degree data, multiply the methylation degree matrix of each cancer patient by a value matrix, obtain a risk value matrix for each patient, arrange them in descending order of risk value, and then transmit them to the high-risk group and low-risk group and survival time classification module; The high- and low-risk groups and survival time classification module is used to divide the data after descending order into a high-risk group and a low-risk group from the middle, and use the log-rank test to compare the survival curves of the high- and low-risk groups respectively, and calculate the p-value. If the p-value is less than 0.05, the model is considered to be meaningful.

2. The method for screening gene methylation features of cancer patients based on DDPG reinforcement learning according to claim 1, characterized in that: The DDPG reinforcement learning module includes the following steps: 1) Initialize the reinforcement learning network parameters and clear the playback buffer; 2) Initialize the current state of the risk value of the cancer patient; 3) Get actions based on the current state in the Actor network; 4) Execute the action and apply it to the feature extraction module to obtain the next state and corresponding reward and determine whether to terminate the state; 5) Store the current state, action, next state, reward, and whether the state is terminated in the playback buffer; 6) Randomly sample a batch of data from the playback buffer to update the parameters of the reinforcement learning network; 7) Repeat steps 3) to 6) until the training is terminated.

3. The method for screening gene methylation features of cancer patients based on DDPG reinforcement learning according to claim 1, characterized in that: The DDPG reinforcement learning algorithm is used to select the value matrix, including: 1) Combining deterministic strategies with deep Q-networks and adopting the Actor-Critic architecture to learn deterministic strategies, it can effectively handle the problem of continuous action space and improve the robustness and stability of the system; 2) The core network structure of DDPG includes the Actor network and the Critic network. The Actor network is responsible for generating continuous actions to respond to the system state; the Critic network evaluates the impact of the actions output by the Actor network on the system performance and optimizes it through the learning value function; the Actor network generates actions through the environment state, noise and strategy function. The formula is as follows: a t =μ θ (s t )+N; Among them, the environmental state S t , noise N and strategy function μ θ (·) Generate a t For action, s t is the environmental state, the noise is N, μ θ (·) is the policy function; Action a t Interact with the environment to obtain a new state s t+1 and reward r t ; Use the policy gradient function to update the Actor online network parameters. The formula is as follows: Among them, ▽ is the gradient operator, J is the expected value of the cumulative return based on the current strategy parameters, N' is the number of samples, Q ω (·) is the online Q-value function; The Critic network includes the Critic online network and the Critic target network; the Critic online network generates an online Q value Q according to the online Q value function. ω (s t ,a t ), where ω represents the Critic online network parameter; the Critic target network calculates the target Q value according to the target Q value function, and the formula is as follows: Among them, y i is the target Q value, γ is the reward discount factor, Q ω′ (·) is the target Q-value function ω′ is the Critic target network parameter; The DDPG algorithm uses the loss function to update the online network parameters. The formula is as follows: Where N” is the number of samples; The critic target network parameters are periodically copied and updated through a soft update method, and the formula is as follows: ω′=τω+(1-τ)ω′; Where τ is the update rate.

4. The method for screening gene methylation features of cancer patients based on DDPG reinforcement learning according to claim 1, characterized in that: The feature extraction module obtains the risk value of each patient and then arranges the risk values ​​of all patients in descending order into a risk value matrix. The process includes: The methylation data matrix of each patient is multiplied by a value matrix to obtain the risk value of each patient. The calculation formula is as follows: M i =X i ·V; H=(h1',h'2,…,h i ',…,h' n-1 'h' n ),h1'≥h'2≥…≥h i '≥…≥h' n-1 ≥h' n ; Where X i The methylation data matrix of the i-th patient, V is the value matrix, M i is the risk matrix of the ith patient, and H is the risk value matrix that arranges all risk values ​​in descending order.

Images