Cancer patient gene methylation characteristic screening method based on DQN reinforcement learning

Abstract

The invention provides a cancer patient gene methylation feature screening method based on DQN reinforcement learning, which converts a gene methylation feature screening problem into a reinforcement learning problem by constructing a deep Q network model, and dynamically selects the most representative methylation feature by taking maximization of prediction accuracy as a target. The method comprises the following steps: firstly, preprocessing gene methylation data of a cancer patient, then continuously optimizing a feature selection strategy in a multi-round training process through a DQN model, and finally screening out key methylation features closely related to cancer diagnosis and prognosis. The method can effectively reduce the feature dimension, improves the generalization ability of the model, and has wide application prospects in early diagnosis, personalized treatment and prognosis evaluation of cancers.

Description

Technical Field

[0001] The present invention relates to the fields of bioinformatics, machine learning and medical technology, and in particular to a method for screening gene methylation features of cancer patients based on DQN reinforcement learning, which belongs to the application of artificial intelligence in medical data analysis. Background Art

[0002] With the rapid development of genomics and bioinformatics, gene methylation, as an important form of epigenetic modification, has been widely studied and confirmed to play a key role in the occurrence and development of cancer. Changes in gene methylation characteristics are not only closely related to the early diagnosis of cancer, but can also be used to predict the prognosis of patients and the effect of personalized treatment. However, gene methylation data are usually high-dimensional and highly redundant. How to screen out key features related to cancer has become an important challenge in cancer research and precision medicine. Traditional feature screening methods, such as feature selection algorithms based on statistical analysis or machine learning, can reduce the dimension of data and improve the predictive ability of the model to a certain extent, but these methods usually rely on pre-set assumptions or model structures and are difficult to dynamically adapt to changes in data distribution. In addition, many traditional methods show problems such as low computational efficiency and easy overfitting when processing large-scale high-dimensional data, which limits their effectiveness in practical applications. In recent years, the successful application of reinforcement learning, especially deep Q network (DQN) in complex decision-making problems, has provided a new solution for feature screening of high-dimensional data. By combining the advantages of deep learning and reinforcement learning, DQN can continuously optimize strategies through interaction with the environment in uncertain and dynamically changing environments, thereby automatically selecting the optimal sequence of actions. Applying DQN to gene methylation feature screening can effectively cope with the complexity of high-dimensional data and dynamically adjust feature selection strategies to maximize the prediction accuracy of the model. Summary of the invention

[0003] Based on the above background, the present invention proposes a method for screening gene methylation features of cancer patients based on DQN reinforcement learning. This method transforms the feature screening problem into a reinforcement learning problem by constructing a DQN model, and dynamically selects the most representative methylation features with the goal of maximizing prediction accuracy, thereby effectively reducing feature dimensions and improving the generalization ability of cancer diagnosis and prognosis models. It has broad application prospects in early diagnosis, personalized treatment and prognosis evaluation of cancer.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] A method for screening gene methylation features of cancer patients based on DQN reinforcement learning, including: data preprocessing module, DQN reinforcement learning module, feature extraction module, and high-risk and low-risk group and survival time classification module;

[0006] The data preprocessing module is used to preprocess the gene methylation data and divide the data into two parts: methylation degree data and survival period and survival status data. The preprocessed methylation degree data will be transmitted to the DQN reinforcement learning module;

[0007] The DQN reinforcement learning module is used to perform reinforcement learning using a deep Q network (DQN) algorithm, establish and optimize a feature selection strategy to generate a value matrix for screening key methylation features, and transmit the value matrix to a feature extraction module;

[0008] The feature extraction module is used to extract features from the preprocessed methylation degree data using the value matrix generated by the DQN reinforcement learning module. Specifically, the methylation degree matrix of each cancer patient is multiplied by the value matrix to obtain a risk value matrix for each patient, which is arranged in descending order of risk value and then transmitted to the high-risk group and low-risk group and survival time classification module;

[0009] The high-risk and low-risk group and survival time classification module is used to divide the sorted data into a high-risk group and a low-risk group, and compare the survival curves of the high-risk group and the low-risk group through a log-rank test, and calculate the p-value; if the p-value is less than 0.05, the model is considered to be statistically significant.

[0010] Further:

[0011] The DQN reinforcement learning module includes the following steps:

[0012] 1) Environment initialization: define a custom reinforcement learning environment, including the definition of state space and action space;

[0013] 2) State update and reward calculation: In each step, the current state is updated by selecting a specific action and the corresponding reward is calculated. The reward function is based on the log-rank test results of the patient survival data and is defined as 1 minus the p-value.

[0014] 3) Neural network construction and training: A DQN network is constructed, which contains two fully connected layers, which are used to estimate the Q value of the input state and the output action respectively. The network is trained using the mean square error (MSE) loss function and the Adam optimizer. During the training process, the state, action, reward and next state samples are stored in the experience replay pool, and learning is performed through batch random sampling.

[0015] 4) Action selection strategy: During the training process, the epsilon-greedy strategy is used to select actions. When epsilon is 0.9, the action with the largest current Q value is preferred; otherwise, an action is randomly selected;

[0016] 5) Q value update: Update the Q value through the Bellman equation. The goal of the Q value update is to make the current Q value close to the target Q value. The target Q value is calculated by the immediate reward and the maximum Q value of the next state;

[0017] 6) Target network synchronization: After each fixed number of learning operations, the parameters of the evaluation network are synchronized to the target network to stabilize the training process;

[0018] 7) Model training and iteration: In multiple training rounds, the cycle of action selection, state update, reward calculation and network parameter update is continuously executed until the reinforcement learning converges. After each round, the optimal feature subset is updated according to the training results;

[0019] 8) Output the optimal feature subset: After the training is completed, the optimal feature subset is output for subsequent classification of high / low risk groups and survival time of cancer patients;

[0020] 9) Form a survival feature set to guide clinical practice based on feature subsets to improve the generalization ability of cancer diagnosis and prognosis models.

[0021] Furthermore, the DQN reinforcement learning module updates the Q value and optimizes the model parameters through the following formula:

[0022] 1) Q value calculation formula:

[0023] Q(s,a)=DQN eval (s,a)

[0024] Among them, Q(s,a) is the Q value of taking action a in state s, DQN eval It is an evaluation network model used to calculate the Q value under the current state and action;

[0025] 2) Target Q value calculation formula:

[0026]

[0027] Among them, r is the immediate reward after taking action a in state s, γ is the discount factor (discount rate), is the maximum Q value at the next state s′, through the target network (DQN target ) is calculated;

[0028] 3) Loss function calculation formula:

[0029]

[0030] Where N is the size of the batch data, and the loss function is used to calculate the mean square error between the Q value of the current evaluation network and the target Q value to optimize the model parameters;

[0031] 4) Q network parameter update formula:

[0032]

[0033] Among them, θ eval are the parameters for evaluating the network, α is the learning rate, It is the gradient of the loss function to evaluate the network parameters, which is used to update the model parameters to reduce the loss;

[0034] 5) Target network parameter update formula:

[0035] θ target ←θ eval

[0036] Among them, θ target are the parameters of the target network. After every set learning step (such as every 100 learning times), the parameters of the evaluation network are directly assigned to the target network to stabilize the training process.

[0037] The DQN reinforcement learning module includes an experience replay mechanism, which is used to store and manage experience during the training process. Specifically, the experience replay mechanism maintains a memory array to save the state-action-reward-next state quadruple. During each training process, the mechanism adds new experience to the memory and stores it cyclically according to the size of the experience pool. In order to improve the stability and generalization ability of training, the experience replay mechanism randomly extracts a batch of data from the experience pool for training. This random sampling method breaks the correlation between data, helps to reduce the overfitting phenomenon of the model and accelerates the convergence of the model. In addition, DQN also regularly updates the target network to ensure stability and effectiveness during the learning process.

[0038] During the training process, the DQN reinforcement learning module performs the following steps to optimize model performance:

[0039] First, a batch of samples are randomly drawn from the experience pool, and the samples contain a four-tuple of state-action-reward-next state; then, the neural network model (evaluation network) is used to calculate the Q value of each possible action in the current state; then, the target network is used to calculate the maximum Q value of the next state as the basis for the target Q value; the parameters of the evaluation network are optimized by minimizing the mean square error between the current Q value and the target Q value; after several learnings, the parameters of the target network will be updated to the latest parameters of the evaluation network, thereby maintaining the stability and accuracy of the training process. This regular update mechanism ensures the consistency of the target network and the evaluation network, and improves the training effect and feature screening quality of the DQN algorithm.

[0040] The feature extraction module specifically includes the following steps:

[0041] Multiply the methylation data matrix of each patient by a value matrix to obtain the risk value of each patient. Then, arrange the risk values ​​of all patients in descending order to generate the risk value matrix. The calculation formula is as follows:

[0042] M i =X i ·V;

[0043] H=(h1',h'2,…,h i ',…,h' n-1 ,h' n ),h1'≥h'2≥…≥h i '≥…≥h' n-1 ≥h' n ;

[0044] Where X i The methylation data matrix of the i-th patient, V is the value matrix, M i is the risk matrix of the ith patient, and H is the risk value matrix that arranges all risk values ​​in descending order.

[0045] The beneficial effects of the present invention are: through the feature screening method based on DQN reinforcement learning, the accuracy of gene methylation feature selection is effectively improved, redundant features are reduced, model complexity and overfitting risks are reduced, thereby improving the generalization ability of the model. In addition, the method can dynamically optimize the feature screening strategy, improve the efficiency and accuracy of the screening process, and has broad application prospects in the early diagnosis, personalized treatment and prognosis evaluation of cancer. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is the system flow chart.

[0047] Figure 2 A schematic diagram of the classification based on the risk matrix.

[0048] Figure 3 This is a flowchart for risk matrix training based on DQN. DETAILED DESCRIPTION

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, not all embodiments.

[0050] Example:

[0051] like Figure 1As shown: A method for screening gene methylation features of cancer patients based on DQN reinforcement learning, comprising the following modules: a data preprocessing module, a DQN reinforcement learning module, a feature extraction module, and a high-risk group and a survival time classification module;

[0052] The data preprocessing module is used to preprocess the gene methylation data and divide the data into two parts: methylation degree data and survival period and survival status data. The preprocessed methylation degree data will be transmitted to the DQN reinforcement learning module;

[0053] The DQN reinforcement learning module is used to perform reinforcement learning using a deep Q network (DQN) algorithm, establish and optimize a feature selection strategy to generate a value matrix for screening key methylation features, and transmit the value matrix to a feature extraction module;

[0054] The feature extraction module is used to extract features from the preprocessed methylation degree data using the value matrix generated by the DQN reinforcement learning module. Specifically, the methylation degree matrix of each cancer patient is multiplied by the value matrix to obtain a risk value matrix for each patient, which is arranged in descending order of risk value and then transmitted to the high-risk group and low-risk group and survival time classification module;

[0055] The high-risk and low-risk group and survival time classification module is used to divide the sorted data into a high-risk group and a low-risk group, and compare the survival curves of the high-risk group and the low-risk group by log-rank test, and calculate the p value. If the p value is less than 0.05, the model is considered to be statistically significant.

[0056] like Figure 2 As shown: The feature extraction module in the cancer patient gene methylation feature screening method based on DQN reinforcement learning arranges the risk value matrix of each patient in descending order, and then transmits it to the high-risk and low-risk group and survival time classification module. This module performs threshold segmentation on the risk value matrix after descending order and divides it into a high-risk group and a low-risk group.

[0057] like Figure 3 As shown: The risk matrix training flow chart based on DQN, the steps are as follows:

[0058] 1) Medical data input:

[0059] A multidimensional dataset is extracted from the patient's medical data, mainly including physical sign data, examination results, and medical information such as historical cases. These input data are used to describe the patient's overall health status and condition. Gene methylation data and these physical sign data are used together as inputs to the DQN model to represent the current state space.

[0060] 2) Step 2: Feature extraction layer:

[0061] In the DQN model, the input data is first processed by the feature extraction layer. The feature extraction layer consists of one or more fully connected layers (FC Layer), which are used to reduce the dimension and represent the features of the input high-dimensional data. In this process, the fully connected layer extracts potential key features related to cancer, providing a basis for subsequent Q value calculation.

[0062] 3) Q value output layer:

[0063] The data after feature extraction is input to the Q value output layer. This layer calculates and outputs the Q value related to different actions through the reinforcement learning algorithm, indicating the potential rewards of different strategies in the current state. In a specific implementation, the present invention outputs the Q value of multiple actions, each Q value represents the expected reward value after selecting a certain action in the current state, which is used to evaluate the pros and cons of different actions.

[0064] 4) Generation of Medical Risk Matrix:

[0065] Based on the Q-value output, a medical risk matrix is ​​constructed. This matrix corresponds different actions or strategies to corresponding risk values. Specifically, the risk value of each action in the risk matrix is ​​calculated based on the Q-value in the reinforcement learning model and reflects the potential impact of the action on the patient's health. An example is as follows:

[0066] - Action 1: The risk value is 0.8, indicating a higher risk;

[0067] - Action 2: The risk value is 0.3, indicating a lower risk;

[0068] - Action 3: The risk value is 0.5, indicating medium risk.

[0069] The medical risk matrix helps doctors quantify the risks of different treatment options or interventions during clinical decision-making.

[0070] 5) Target network and experience replay mechanism:

[0071] In order to ensure the stability and convergence of the model, the present invention adopts the target network and experience replay mechanism. The weight of the target network is updated regularly to ensure that it remains unchanged during a certain training phase, thereby reducing the instability during the training process. Specifically, the weight parameters of the target network are periodically updated synchronously according to the weights of the evaluation network.

[0072] At the same time, the present invention adopts an experience replay mechanism to store the state-action pairs generated by the model during training in the experience pool. In subsequent training, these state-action pairs are randomly selected for model optimization. This mechanism enables the model to learn from different historical experiences and avoid overfitting problems caused by sample correlation.

[0073] The experience replay process acts on the medical risk matrix and target network, helping the reinforcement learning model to better adjust the feature selection strategy and improve the accuracy of cancer diagnosis and prognosis.

Claims

1. A method for screening gene methylation features of cancer patients based on DQN reinforcement learning, characterized by: include: Data preprocessing module, DQN reinforcement learning module, feature extraction module, and high-risk and low-risk group and survival time classification module; among them: The data preprocessing module is used to preprocess the gene methylation data and divide the data into two parts: methylation degree data and survival period and survival status data. The preprocessed methylation degree data will be transmitted to the DQN reinforcement learning module; DQN reinforcement learning module, used to perform reinforcement learning using the deep Q network algorithm, establish and optimize the feature selection strategy to generate a value matrix for screening key methylation features, and transmit this value matrix to the feature extraction module; The feature extraction module is used to extract features from the preprocessed methylation degree data using the value matrix generated by the DQN reinforcement learning module, multiply the methylation degree matrix of each cancer patient by the value matrix to obtain the risk value matrix of each patient, sort them in descending order by risk value, and then transmit them to the high-risk group and low-risk group and survival time classification module; The high-risk and low-risk group and survival time classification module is used to divide the sorted data into high-risk and low-risk groups, compare the survival curves of the high-risk and low-risk groups through the log-rank test, and calculate the p-value; if the p-value is less than 0.05, the model is considered to be statistically significant.

2. The method for screening gene methylation features of cancer patients based on DQN reinforcement learning according to claim 1, characterized in that: The DQN reinforcement learning module includes the following steps: 1) Environment initialization: define a custom reinforcement learning environment, including the definition of state space and action space; 2) State update and reward calculation: In each step, the current state is updated by selecting a specific action and the corresponding reward is calculated. The reward function is based on the log-rank test results of the patient survival data and is defined as 1 minus the p-value. 3) Neural network construction and training: Construct a DQN network, which contains two fully connected layers, used for estimating the Q value of the input state and output action respectively. The network is trained using the mean square error (MSE) loss function and the Adam optimizer. During the training process, the state, action, reward, and next state samples are stored in the experience replay pool, and learning is performed through batch random sampling. 4) Action selection strategy: During the training process, the epsilon-greedy strategy is used to select actions. When epsilon is 0.9, the action with the largest current Q value is preferred; otherwise, an action is randomly selected; 5) Q value update: Update the Q value through the Bellman equation. The goal of the Q value update is to make the current Q value close to the target Q value. The target Q value is calculated by the immediate reward and the maximum Q value of the next state; 6) Target network synchronization: After each fixed number of learning operations, the parameters of the evaluation network are synchronized to the target network to stabilize the training process; 7) Model training and iteration: In multiple training rounds, the cycle of action selection, state update, reward calculation and network parameter update is continuously executed until the reinforcement learning converges; after each round, the optimal feature subset is updated according to the training results; 8) Output the optimal feature subset: After the training is completed, the optimal feature subset is output for subsequent classification of high / low risk groups and survival time of cancer patients and genetic feature screening.

3. The method for screening gene methylation features of cancer patients based on DQN reinforcement learning according to claim 1, characterized in that: The DQN reinforcement learning module updates the Q value and optimizes the model parameters through the following formula: 1) Q value calculation formula: Q(s,a)=DQN eval (s,a) Among them, Q(s,a) is the Q value of taking action a in state s, DQN eval It is an evaluation network model used to calculate the Q value under the current state and action; 2) Target Q value calculation formula: Where r is the immediate reward after taking action a in state s, γ is the discount factor, is the maximum Q value at the next state s′, through the target network DQN target Calculated; 3) Loss function calculation formula: Where N is the size of the batch data, and the loss function is used to calculate the mean square error between the Q value of the current evaluation network and the target Q value to optimize the model parameters; 4) Q network parameter update formula: Among them, θ eval are the parameters for evaluating the network, α is the learning rate, It is the gradient of the loss function to evaluate the network parameters, which is used to update the model parameters to reduce the loss; 5) Target network parameter update formula: i target ←θ eval Among them, θ target are the parameters of the target network. After each set learning step, the parameters of the evaluation network are directly assigned to the target network to stabilize the training process.

4. The method for screening gene methylation features of cancer patients based on DQN reinforcement learning according to claim 1, characterized in that: The DQN reinforcement learning module includes an experience replay mechanism, which is used to store and manage experience during the training process, wherein the experience replay mechanism maintains a memory array for saving the four-tuple of state-action-reward-next state; during each training process, the mechanism adds new experience to the memory and stores it cyclically according to the size of the experience pool.

5. The method for screening gene methylation features of cancer patients based on DQN reinforcement learning according to claim 1, characterized in that: The DQN reinforcement learning module performs the following steps during training to optimize model performance: first, a batch of samples are randomly drawn from the experience pool, and the samples contain a four-tuple of state-action-reward-next state; then, the neural network model is used to calculate the Q value of each possible action in the current state; then, the target network is used to calculate the maximum Q value of the next state as the basis for the target Q value; the parameters of the evaluation network are optimized by minimizing the mean square error between the current Q value and the target Q value; after each several learnings, the parameters of the target network will be updated to the latest parameters of the evaluation network.

6. The method for screening gene methylation features of cancer patients based on DQN reinforcement learning according to claim 1, characterized in that: The feature extraction module comprises: Multiply the methylation data matrix of each patient by a value matrix to obtain the risk value of each patient; then, arrange the risk values ​​of all patients in descending order to generate a risk value matrix. The calculation formula is as follows: M i =X i ·V; H=(h′1,h'2,…,h′ i ,…,h' n-1 ,h' n ),h′1≥h'2≥…≥h′ i ≥…≥h' n-1 ≥h' n ; Where X i The methylation data matrix of the i-th patient, V is the value matrix, M i is the risk matrix of the ith patient, and H is the risk value matrix that arranges all risk values ​​in descending order.

Images