Offline reinforcement learning method and device for recommending safe treatment regimen

Abstract

The offline reinforcement learning method and device for recommending a safe treatment plan can improve the performance and safety of the reinforcement learning model when recommending a treatment plan, expand the distribution range of the evaluation data of the reinforcement learning model, and slow down the adverse effects of out-of-distribution samples and overestimation. The method comprises the following steps: (1) reading in a required data set and dividing it; (2) performing data preprocessing to obtain reduced dimension hidden space variables; (3) completing the reinforcement learning basic data composition of states, actions and reward values and storing them; (4) building a supervised learning model basis for anomaly detection; (5) building a nearest neighbor classification layer for anomaly detection and a nearest neighbor loss function; (6) dynamically weighting the original reward value according to the recurrence risk predicted by the above anomaly detection supervised model and the out-of-distribution sample detection situation; (7) building and training an action-evaluation offline reinforcement learning model for recommending a treatment plan; and (8) obtaining an optimal treatment plan.

Description

Technical Field

[0001] This invention relates to the technical field of medical electronic medical record data mining, and more particularly to an offline reinforcement learning method for recommending safe treatment plans, as well as an offline reinforcement learning device for recommending safe treatment plans. Background Technology

[0002] To improve the applicability of offline reinforcement learning models to out-of-distribution samples and reduce the overestimation problem inherent in offline learning methods, adjusting the reward information in the reinforcement learning components is one solution. Due to the ontological characteristics of medical data and clinical tasks, patients exhibit significant individual differences, and doctors with varying experience develop different treatment strategies, leading to frequent out-of-distribution samples. Offline reinforcement learning learns strategies using historical treatment trajectories, which can be divided into action prediction models and state-action evaluation models. The prediction model takes the patient's state as input and predicts the corresponding treatment plan, while the evaluation model takes the patient's state and the treatment plan as input and outputs the evaluation value of the treatment plan for the patient, i.e., the effectiveness of the treatment plan. For training samples, i.e., in-distribution samples, offline reinforcement learning typically performs well in evaluation. However, in actual treatment plan recommendations, out-of-distribution samples may appear—samples whose distribution differs from the training set samples—causing significant bias in the offline reinforcement learning's evaluation of these samples.

[0003] Therefore, the following conditions must be met for offline reinforcement learning to be applied to treatment plan recommendation: 1. Learning is based on offline data, but the test data may contain out-of-distribution samples; 2. Reasonable evaluation must be maintained for out-of-distribution samples; 3. The treatment plan recommended by the reinforcement learning model is similar to the treatment plan formulated by expert physicians.

[0004] Numerous offline reinforcement learning models exist to address the problems of overestimation and out-of-distribution samples. One approach utilizes regularization methods that minimize the distance between the recommendation and recording schemes, constructing multiple reinforcement learning components with different initialization methods for ensemble learning. However, these methods struggle to handle the frequent out-of-distribution sample problem encountered in clinical tasks. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the technical problem to be solved by the present invention is to provide an offline reinforcement learning method for recommending safe treatment plans, which can improve the performance and safety of reinforcement learning models when recommending treatment plans, expand the distribution range of evaluation data for reinforcement learning models, and mitigate the adverse effects of out-of-distribution samples and overestimation.

[0006] The technical solution of this invention is: an offline reinforcement learning method for recommending safe treatment plans, which includes the following steps:

[0007] (1) Read in the required dataset and divide it according to patient demographic information, lesion information, pathological information, preoperative information, treatment plan information, and postoperative information;

[0008] (2) Construct a self-supervised learning model based on autoencoder to perform data preprocessing on demographic information, lesion information, pathological information and preoperative information to obtain dimensionality-reduced latent space variables;

[0009] (3) Summarize the previous treatment plan, obtain the cumulative treatment dose and concatenate it with the above latent space variables to obtain the patient state for training reinforcement learning; then use the current treatment dose as the action for training reinforcement learning, and the postoperative recurrence status as the original reward value for training reinforcement learning; complete the reinforcement learning basic data of state, action, and reward value and store it.

[0010] (4) Based on the feedforward neural network framework, with the above patient status as input and recurrence risk as the prediction target, a supervised learning model for abnormality detection is constructed.

[0011] (5) Construct a nearest neighbor classification layer for anomaly detection, using Euclidean distance as the metric, and at the same time construct a nearest neighbor loss function to replace the original loss function;

[0012] (6) The original reward value is dynamically weighted based on the recurrence risk predicted by the above anomaly detection supervision model and the detection of out-of-distribution samples, and is used as the training basis for subsequent reinforcement learning.

[0013] (7) Using TD3-BC offline reinforcement learning as a framework, and using the above patient status, actions, and weighted reward values ​​as data, construct and train an action-evaluation offline reinforcement learning model for recommending treatment plans.

[0014] (8) Complete the training of the treatment plan recommendation reinforcement learning model, and obtain the optimal treatment plan from the action prediction model in the above model based on the current patient information.

[0015] This invention utilizes an anomaly detection mechanism to guide the training of a supervised learning model based on label information, predicting reward values. The dynamically weighted sum of the predicted and original reward values ​​serves as an evaluation signal to guide the reinforcement learning model in recommending the optimal treatment plan. The anomaly detection mechanism, composed of a nearest neighbor classification layer and a nearest neighbor loss function, is compatible with neural network-based classification models, offering high flexibility. The offline reinforcement learning model, based on the TD3-BC model framework, boasts high performance and robust prediction results. The anomaly detection mechanism identifies out-of-distribution samples, thereby altering the reward value to prevent overestimation by the reinforcement learning model. This mechanism allows for the construction of out-of-distribution sample data pairs, expanding the trainable data range for offline reinforcement learning. Encoding data with slow temporal changes, such as demographic information, lesion information, pathological information, and preoperative information, reduces dimensionality and improves the learning efficiency of the reinforcement learning model.

[0016] An offline reinforcement learning device for recommending safe treatment plans is also provided, which includes:

[0017] The reading module is configured to read in the required dataset and divide it according to patient demographic information, lesion information, pathological information, preoperative information, treatment plan information, and postoperative information;

[0018] The self-supervised learning module is configured to build a self-supervised learning model based on an autoencoder to perform data preprocessing on demographic information, lesion information, pathological information, and preoperative information to obtain dimensionality-reduced latent space variables.

[0019] The basic data construction module is configured to summarize previous treatment plans, obtain the cumulative treatment dose, and concatenate it with the aforementioned latent space variables to obtain the patient state used for training reinforcement learning; then, the current treatment dose is used as the action used for training reinforcement learning, and the postoperative recurrence status is used as the original reward value used for training reinforcement learning; the basic data construction of reinforcement learning, including state, action, and reward value, is completed and stored.

[0020] The anomaly detection module is configured based on a feedforward neural network framework, taking the aforementioned patient status as input and recurrence risk as the prediction target, to build a supervised learning model foundation for anomaly detection;

[0021] The nearest neighbor module is configured to build a nearest neighbor classification layer for anomaly detection, using Euclidean distance as the metric, and also to build a nearest neighbor loss function to replace the original loss function.

[0022] The dynamic weighting module is configured to dynamically weight the original reward value based on the recurrence risk predicted by the above-mentioned anomaly detection supervision model and the detection of out-of-distribution samples, so as to serve as the training basis for subsequent reinforcement learning.

[0023] The reinforcement learning modeling module is configured to use TD3-BC offline reinforcement learning as a framework, and uses the aforementioned patient state, action, and weighted reward value as data to construct and train an action-evaluation offline reinforcement learning model for recommending treatment plans.

[0024] The output module is configured to train the reinforcement learning model for recommending treatment plans, and obtains the optimal treatment plan from the action prediction model in the above model based on the current patient information. Attached Figure Description

[0025] Figure 1 This is a flowchart of an offline reinforcement learning method for recommending safe treatment plans according to the present invention.

[0026] Figure 2 This is a flowchart of an offline reinforcement learning method for recommending safe treatment plans according to the present invention. Detailed Implementation

[0027] like Figure 1 As shown, this offline reinforcement learning method for recommending safe treatment options includes the following steps:

[0028] (1) Read in the required dataset and divide it according to patient demographic information, lesion information, pathological information, preoperative information, treatment plan information, and postoperative information;

[0029] (2) Construct a self-supervised learning model based on autoencoder to perform data preprocessing on demographic information, lesion information, pathological information and preoperative information to obtain dimensionality-reduced latent space variables;

[0030] (3) Summarize the previous treatment plan, obtain the cumulative treatment dose and concatenate it with the above latent space variables to obtain the patient state for training reinforcement learning; then use the current treatment dose as the action for training reinforcement learning, and the postoperative recurrence status as the original reward value for training reinforcement learning; complete the reinforcement learning basic data of state, action, and reward value and store it.

[0031] (4) Based on the feedforward neural network framework, with the above patient status as input and recurrence risk as the prediction target, a supervised learning model for abnormality detection is constructed.

[0032] (5) Construct a nearest neighbor classification layer for anomaly detection, using Euclidean distance as the metric, and at the same time construct a nearest neighbor loss function to replace the original loss function;

[0033] (6) The original reward value is dynamically weighted based on the recurrence risk predicted by the above anomaly detection supervision model and the detection of out-of-distribution samples, and is used as the training basis for subsequent reinforcement learning.

[0034] (7) Using TD3-BC offline reinforcement learning as a framework, and using the above patient status, actions, and weighted reward values ​​as data, construct and train an action-evaluation offline reinforcement learning model for recommending treatment plans.

[0035] (8) Complete the training of the treatment plan recommendation reinforcement learning model, and obtain the optimal treatment plan from the action prediction model in the above model based on the current patient information.

[0036] This invention utilizes an anomaly detection mechanism to guide the training of a supervised learning model based on label information, predicting reward values. The dynamically weighted sum of the predicted and original reward values ​​serves as an evaluation signal to guide the reinforcement learning model in recommending the optimal treatment plan. The anomaly detection mechanism, composed of a nearest neighbor classification layer and a nearest neighbor loss function, is compatible with neural network-based classification models, offering high flexibility. The offline reinforcement learning model, based on the TD3-BC model framework, boasts high performance and robust prediction results. The anomaly detection mechanism identifies out-of-distribution samples, thereby altering the reward value to prevent overestimation by the reinforcement learning model. This mechanism allows for the construction of out-of-distribution sample data pairs, expanding the trainable data range for offline reinforcement learning. Encoding data with slow temporal changes, such as demographic information, lesion information, pathological information, and preoperative information, reduces dimensionality and improves the learning efficiency of the reinforcement learning model.

[0037] Preferably, in step (1), the data is read in and denoted as X, where X∈R. N×D N is the total number of samples, D is the feature dimension, and X d X l X pth X per X t X post The samples are defined as follows: demographic information, lesion information, pathological information, preoperative information, treatment plan information, and postoperative information features, where X = {X...} d ∪X l ∪X pth ∪X per ∪X t ∪X post Let X AE ={X d ∪X l ∪X pth ∪X per}

[0038] Preferably, in step (2), a self-supervised learning method based on an autoencoder is constructed to obtain the dimensionality-reduced latent space variables:

[0039] (2.1) Construct encoder E and decoder D based on a feedforward neural network framework. Let the latent variable be Z, then Z = E(X). X patient data AE As input, we obtain the latent variable Z. AE =E(X) AE Decode and obtain the reconstructed data.

[0040] (2.2) Calculate the objective function of mean square error Its O MSE The minimum mean squared error loss function is used, and then the parameters in the self-supervised learning model are updated based on the backpropagation algorithm.

[0041] Preferably, in step (3), a dataset for training the reinforcement learning model is constructed:

[0042] (3.1) Let the treatment dose be A, A∈X t The patient's condition is S. AE S AE =Z AE The original reward value is R, R∈X post ;

[0043] (3.2) Based on the level of medical care, accumulate the treatment doses completed before the current visit. m represents the current number of treatments during this visit;

[0044] (3.3) The cumulative treatment dose S A With patient status S AE Perform splicing to obtain the updated patient state S = [S AE S A ];

[0045] (3.4) (S, A, R, S') is a data pair used to train reinforcement learning, where S' is the state of S at the next time step.

[0046] Preferably, in step (4), a classifier is constructed based on the fully connected layer FNN(·) of the feedforward neural network component and the activation function ReLU(·). The classification supervision model, excluding the last layer, is a nested FNN with a fully connected layer and an activation function. i+1 (RELU i (FNN i (RELU i-1 (...)))), where i is the current layer number of the model, and let f be the output of the last layer.

[0047] Preferably, in step (5), a nearest neighbor classification layer and a nearest neighbor loss function are constructed for anomaly detection:

[0048] (5.1) Construct a classification layer based on an anomaly detection mechanism with f as input, then f i For the i-th sample, Let be the j-th sample belonging to class c, d(·) be the distance metric, and δ(·) be the activation function;

[0049] (5.2) Construct the objective function for calculating the nearest neighbor classification error loss Then, the parameters in the classification supervised learning model are updated based on the backpropagation algorithm.

[0050] Preferably, in step (6), the recurrence risk predicted by the anomaly detection supervision model for different sample distributions is . The dynamically weighted reward value is R. W =R+R AD .

[0051] Preferably, in step (7), an offline reinforcement learning model is constructed using TD3-BC offline reinforcement learning as the framework:

[0052] (7.1) Construct an action predictor μ, parameterize the reinforcement learning policy μ with a neural network model, and set the state s at time step t. t Select action a for input t Then a t =μ(s) t |θ μ );

[0053] (7.2) Construct a state-action evaluator, parameterize the reinforcement learning evaluation module Q with a neural network model, and set the state s at time step t. t With action a t Evaluate the value of state-action pairs.

[0054] (7.3) Constructing action predictors μ with the same network structure but different parameters tar and target evaluator Q tar Then construct the objective optimization function.

[0055]

[0056] Where γ is the reward discount coefficient, the parameters in the action predictor and estimator models are then updated based on the backpropagation algorithm. After a fixed number of iterations, the parameters of the target estimator are updated through soft target update.

[0057]

[0058] Where ∈, τ are the coefficients of the soft update of the target parameters.

[0059] Preferably, in step (8), the training of the treatment plan recommendation reinforcement learning model is completed based on the above steps, and the action prediction model μ is finally obtained. * Then the optimal treatment plan A = μ * (X).

[0060] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When executed, the program includes the steps of the methods of the above embodiments. The storage medium can be ROM / RAM, magnetic disk, optical disk, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes an offline reinforcement learning device for recommending safe treatment plans. This device is typically represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0061] The reading module is configured to read in the required dataset and divide it according to patient demographic information, lesion information, pathological information, preoperative information, treatment plan information, and postoperative information;

[0062] The self-supervised learning module is configured to build a self-supervised learning model based on an autoencoder to perform data preprocessing on demographic information, lesion information, pathological information, and preoperative information to obtain dimensionality-reduced latent space variables.

[0063] The basic data construction module is configured to summarize previous treatment plans, obtain the cumulative treatment dose, and concatenate it with the aforementioned latent space variables to obtain the patient state used for training reinforcement learning; then, the current treatment dose is used as the action used for training reinforcement learning, and the postoperative recurrence status is used as the original reward value used for training reinforcement learning; the basic data construction of reinforcement learning, including state, action, and reward value, is completed and stored.

[0064] The anomaly detection module is configured based on a feedforward neural network framework, taking the aforementioned patient status as input and recurrence risk as the prediction target, to build a supervised learning model foundation for anomaly detection;

[0065] The nearest neighbor module is configured to build a nearest neighbor classification layer for anomaly detection, using Euclidean distance as the metric, and to build a nearest neighbor loss function to replace the original loss function; the dynamic weighting module is configured to dynamically weight the original reward value based on the recurrence risk predicted by the above anomaly detection supervision model and the detection of out-of-distribution samples, so as to serve as the training basis for subsequent reinforcement learning.

[0066] The reinforcement learning modeling module is configured to use TD3-BC offline reinforcement learning as a framework, and uses the aforementioned patient state, action, and weighted reward value as data to construct and train an action-evaluation offline reinforcement learning model for recommending treatment plans.

[0067] The output module is configured to train the reinforcement learning model for recommending treatment plans, and obtains the optimal treatment plan from the action prediction model in the above model based on the current patient information.

[0068] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An offline reinforcement learning method for recommending safe treatment plans, characterized in that, It includes the following steps: (1) Read in the required dataset and divide it according to patient demographic information, lesion information, pathological information, preoperative information, treatment plan information, and postoperative information; (2) Construct a self-supervised learning model based on autoencoder to perform data preprocessing on demographic information, lesion information, pathological information and preoperative information to obtain dimensionality-reduced latent space variables; (3) Summarize the previous treatment plan, obtain the cumulative treatment dose and concatenate it with the above latent space variables to obtain the patient state for training reinforcement learning; then use the current treatment dose as the action for training reinforcement learning, and use the postoperative recurrence status as the original reward value for training reinforcement learning; complete the reinforcement learning basic data of state, action, and reward value and store it. (4) Based on the feedforward neural network framework, with the above patient status as input and recurrence risk as the prediction target, a supervised learning model for anomaly detection is constructed. (5) Construct a nearest neighbor classification layer for anomaly detection, using Euclidean distance as the metric, and at the same time construct a nearest neighbor loss function to replace the original loss function; (6) The original reward value is dynamically weighted based on the recurrence risk predicted by the above anomaly detection supervision model and the detection of out-of-distribution samples, and is used as the training basis for subsequent reinforcement learning. (7) Using TD3-BC offline reinforcement learning as a framework, and using the above patient status, actions, and weighted reward values ​​as data, construct and train an action-evaluation offline reinforcement learning model for recommending treatment plans; (8) Complete the training of the treatment plan recommendation reinforcement learning model, and obtain the optimal treatment plan from the action prediction model in the above model based on the current patient information; In step (2), a self-supervised learning model based on an autoencoder is constructed to obtain the dimensionality-reduced latent space variables: (2.1) Constructing an encoder based on a feedforward neural network framework With decoder Let the hidden variable be ,but , , patient data As input, we obtain latent variables. Decode and obtain reconstructed data ; (2.2) Calculate the objective function of mean square error ,That The minimum mean squared error loss function is used, and then the parameters in the self-supervised learning model are updated based on the backpropagation algorithm; In step (3), a dataset for training the reinforcement learning model is constructed: (3.1) Let the treatment dose be , The patient's condition is , The original reward value is , ; (3.2) Based on the level of medical care, accumulate the treatment doses completed before the current medical visit. , This refers to the current number of treatments received during this visit; (3.3) The cumulative treatment dose With the patient's condition The data is stitched together to obtain an updated patient status. ; (3.4) For the data pairs used to train reinforcement learning, where for The state of the next time step; In step (4), the fully connected layer based on the feedforward neural network component... With activation function Construct a classifier; the supervised classification model, excluding the last layer, consists of a nested fully connected layer and activation function. ,in Let be the current layer number of the model, and let the output of the last layer be . ; In step (5), a nearest neighbor classification layer and a nearest neighbor loss function are constructed for anomaly detection: (5.1) Construct a mechanism based on anomaly detection to For the input classification layer, then , For the first One sample, For belonging to the first The class of One sample, For distance measurement, For activation functions; (5.2) Construct the objective function for calculating the nearest neighbor classification error loss Then, the parameters in the classification supervised learning model are updated based on the backpropagation algorithm.

2. The offline reinforcement learning method for recommending safe treatment plans according to claim 1, characterized in that: In step (1), the data is read in and set to be , , The total number of samples, For feature dimension, where , , , , , The samples consist of demographic information, lesion information, pathological information, preoperative information, treatment plan information, and postoperative information features, respectively. ,make .

3. The offline reinforcement learning method for recommending safe treatment plans according to claim 2, characterized in that: In step (6), let the recurrence risk predicted by the anomaly detection supervision model for different sample distributions be . The dynamically weighted reward value is .

4. The offline reinforcement learning method for recommending safe treatment plans according to claim 3, characterized in that: In step (7), an offline reinforcement learning model is constructed using TD3-BC offline reinforcement learning as the framework: (7.1) Constructing an action predictor Reinforcement learning strategies parameterized by neural network models Based on time steps status Select action for input ,but ; (7.2) Construct a state-action evaluator, parameterize the reinforcement learning evaluation module Q using a neural network model, and use time steps as the time step. status With action Evaluate the value of state-action pairs. ; (7.3) Constructing action predictors with the same network structure but different parameters and target evaluator Then construct the objective optimization function. ， in To determine the reward discount factor, the parameters in the action predictor and estimator models are updated based on the backpropagation algorithm. After a fixed number of iterations, the parameters of the target estimator are updated using soft target updates. ， in, , The coefficients are used for soft updates of the target parameters.

5. The offline reinforcement learning method for recommending safe treatment plans according to claim 4, characterized in that: In step (8), the reinforcement learning model for recommending treatment plans is trained based on the above steps, and the action prediction model is finally obtained. Then the optimal treatment plan can be obtained. .

6. An offline reinforcement learning device for recommending safe treatment plans, which performs the method according to claim 1, characterized in that, It includes: The reading module is configured to read in the required dataset and divide it according to patient demographic information, lesion information, pathological information, preoperative information, treatment plan information, and postoperative information; The self-supervised learning module is configured to build a self-supervised learning model based on an autoencoder to perform data preprocessing on demographic information, lesion information, pathological information, and preoperative information to obtain dimensionality-reduced latent space variables. The basic data construction module is configured to summarize previous treatment plans, obtain the cumulative treatment dose, and concatenate it with the aforementioned latent space variables to obtain the patient state used for training reinforcement learning; then, the current treatment dose is used as the action used for training reinforcement learning, and the postoperative recurrence status is used as the original reward value used for training reinforcement learning; the basic data construction of reinforcement learning, including state, action, and reward value, is completed and stored. The anomaly detection module is configured based on a feedforward neural network framework, taking the aforementioned patient status as input and recurrence risk as the prediction target, to build a supervised learning model foundation for anomaly detection; The nearest neighbor module is configured to build a nearest neighbor classification layer for anomaly detection, using Euclidean distance as the metric, and also to build a nearest neighbor loss function to replace the original loss function. The dynamic weighting module is configured to dynamically weight the original reward value based on the recurrence risk predicted by the above-mentioned anomaly detection supervision model and the detection of out-of-distribution samples, so as to serve as the training basis for subsequent reinforcement learning. The reinforcement learning modeling module is configured to use TD3-BC offline reinforcement learning as a framework, and uses the aforementioned patient state, action, and weighted reward value as data to construct and train an action-evaluation offline reinforcement learning model for recommending treatment plans. The output module is configured to train the reinforcement learning model for recommending treatment plans, and obtains the optimal treatment plan from the action prediction model in the above model based on the current patient information.

Images