A pre-training and reinforcement learning-based hypertension early warning method

Abstract

The application discloses a hypertension early warning method based on pre-training and reinforcement learning, designs self-constituted prediction tasks and target prediction pre-training tasks, and applies a large amount of abandoned single physical examination data to pre-training and obtains suitable model initialization parameters; a hypertension early warning reinforcement learning framework is designed, and multiple physical examination data are applied to hypertension early warning, and through the strategies such as rewards and punishments, memory management and data sample selection, the problems of sample imbalance and overfitting in a small sample environment are solved. The application uses the physical examination data of a certain medical institution for testing, and experiments show that the model framework has achieved good results in hypertension prediction.

Description

Technical Field

[0001] This invention relates to the fields of smart healthcare, pre-training technology, and reinforcement learning technology, specifically to a hypertension early warning method based on pre-training and reinforcement learning. Background Technology

[0002] In recent years, deep learning has developed rapidly, achieving many disruptive results in image recognition, natural language processing, and speech recognition. Deep learning simulates how the brain processes information, using neural networks to gradually acquire features from input information. Deep learning has seen rapid development in assisting disease diagnosis and has achieved great success in disease prevention, clinical decision-making, and disease risk warning. With sufficient training samples, deep learning models can achieve performance comparable to or even better than domain experts in diagnosing certain diseases. However, in real-world clinical scenarios, due to differences in medical record recording methods, the quality of medical record data varies, making it difficult to obtain large amounts of standardized electronic health record data. To address this issue of insufficient training samples and data dependency, transfer learning has emerged.

[0003] In recent years, transfer learning has developed rapidly. The main idea of ​​transfer learning is to transfer knowledge from the source domain to the target domain to improve the performance of the learning model. Transfer learning can, to some extent, solve the problem of excessive human and time resource consumption caused by sample labeling tasks, and can also effectively address the problem of scarce labeled data. Pre-training is a specific implementation of the transfer learning idea. Pre-training obtains initial parameters by training on a large amount of unlabeled datasets, and then fine-tunes them on the training set to achieve knowledge transfer to the target task, thereby improving the effectiveness of predicting the target task. Currently, research has demonstrated the effectiveness of pre-training and fine-tuning in natural language processing. Currently, most studies focus on longitudinal patient consultation records, neglecting single consultation data. Using a large amount of single consultation data for pre-training can facilitate knowledge transfer and improve model transfer efficiency.

[0004] Reinforcement learning aims to optimize decision-making by using samples of an agent's interactions with its environment and potential delayed feedback. Unlike traditional supervised learning, which typically relies on one-time, exhaustive, and supervised reward signals, reinforcement learning simultaneously addresses sequential decision-making problems involving sampling, evaluation, and delayed feedback. In reinforcement learning, an individual takes action at each time step, deriving the value of their actions through an incentive function and maximizing the reward signal through continuous trial and error. This distinguishes reinforcement learning from traditional supervised learning, which primarily discovers hidden features in the training set. Furthermore, reinforcement learning is particularly well-suited for systems with inherent time delays, where decisions are not made based on immediate effectiveness but rather on long-term future reward evaluation. Therefore, reinforcement learning is suitable for solving continuous and delayed feedback processes in the medical field.

[0005] To address this, this patent proposes a hypertension early warning method based on pre-training and reinforcement learning. The method designs two pre-training tasks: self-composition prediction and target prediction. These tasks fully learn the relationships between various attributes and the relationship between predicted target values, providing reasonable initialization parameters. Furthermore, a reinforcement learning-based hypertension early warning framework is designed, transforming the hypertension prediction problem into an agent selection problem. A memory management mechanism alleviates the imbalance between positive and negative samples, a data sample selection strategy prioritizes data samples that contribute significantly to hypertension early warning, and a reward / penalty mechanism is designed to prevent overfitting caused by local optima under small sample conditions. Summary of the Invention

[0006] The purpose of this invention is to propose a hypertension early warning method based on pre-training and reinforcement learning for hypertension early warning prediction. First, it proposes pre-training tasks suitable for patient visit data, namely a self-composition prediction pre-training task and a target prediction pre-training task. Second, it proposes a reinforcement learning framework for hypertension early warning, designs a memory management mechanism to alleviate the positive-negative sample imbalance problem, designs a data sample selection strategy to select data that contributes significantly to hypertension early warning, and designs a reward and punishment mechanism to solve the overfitting problem caused by the model getting stuck in local optima.

[0007] The specific steps of this invention are as follows:

[0008] (1) Obtain the personal health record dataset, perform preprocessing such as deduplication on the data, establish dictionaries for each attribute of the medical data, and divide the medical data into single medical visit data and multiple medical visit data, and store them as PKL files.

[0009] (2) Based on the established medical visit attribute dictionary, the data of a single medical visit is encoded, and the Transformer model is used to pre-train the data of a single medical visit.

[0010] (3) For the characteristics of multiple sequence visit data, perform reinforcement learning modeling, and design parts such as the individual, behavior, state, memory management mechanism, reward and punishment mechanism, and sample selection strategy of the reinforcement learning framework.

[0011] (4) Use the Transformer pre-trained model to encode and embed the multiple visit data, and input the encoded multiple visit data into the reinforcement learning framework for hypertension warning.

[0012] In step (1), use personal physical examination data to construct a data set, and select hypertension as the disease to be warned. Select 16 attributes related to hypertension from the personal physical examination data table, namely height, weight, body temperature, heart rate, pulse, systolic blood pressure, diastolic blood pressure, waist circumference, BMI, total cholesterol, triglyceride, serum low-density lipoprotein cholesterol, serum high-density lipoprotein cholesterol, exercise frequency, drinking frequency, and exercise frequency. According to the two fields of "whether suffering from hypertension" and "date of hypertension diagnosis" in the basic information of the personal health record, and the "physical examination time" field in the personal health examination data, judge whether it is the physical examination data within three years before the disease. Set the physical examination within three years of the disease as positive samples; set the physical examination data without diagnosed hypertension and with the disease for more than three years as negative samples. Establish a dictionary for each of the selected 16 attribute fields, divide the visit data into single visit and multiple visit data, and store them as PKL files for preservation.

[0013] In step (2), based on the established attribute dictionary, encode each attribute field in the form of one-hot encoding, and splice the 16 attribute fields to obtain the vector representation of each single visit data:

[0014]

[0015] attr[i] represents the i-th attribute in the electronic health record, |M| represents the number of attributes in the electronic health record, and || represents the concatenation representation of attr[0], attr[1]…attr[|M|].

[0016] We randomly mask the attribute attr (n) [i] of the individual's single visit data to obtain the representation of each physical examination attribute of individual n:

[0017]

[0018] Among them, 0 < i ≤ |M|, i represents the i-th attribute in the physical examination record, and |M| represents the number of physical examination attributes in the electronic health record. random_mask is a random attribute masking method, which can be expressed by the following formula:

[0019]

[0020] Where 0≤i≤|P|, and i is a positive integer, and |P| represents the number of attributes in a physical examination record.

[0021] Therefore, a complete medical record of admission can be represented as:

[0022]

[0023] || represents the concatenation of adm[0], adm[1], ..., adm[|P|].

[0024] We set up two pre-training tasks to enhance the model's prediction performance on downstream tasks.

[0025] Self-Composition Prediction Task: We set up a self-composition prediction task to detect occlusion attributes. We designed a self-synthesis task to enhance the model's self-predictive capabilities. The loss function for the self-composition prediction task is as follows:

[0026]

[0027] Among them, attr (n) [i] represents the i-th physical examination record of the n-th patient, pre_attr (n) [i] represents the vector embedding obtained after the i-th physical examination record of the n-th patient is randomly processed and entered into the Transformer model. p i ∈{VOC i \attr (n) [i]} represents p i Besides attr (n) VOCs other than [i] i Any one of the values ​​in.

[0028] Target prediction task: We train a hypertension prediction task on a single patient physical examination record to facilitate prediction of downstream tasks. The loss function for this task is designed as follows for the final prediction target:

[0029]

[0030] in, This represents the is_hyper attribute of the nth patient, obtained through the Transformer and random masking tasks, attr (n) [i] represents the i-th physical examination record of the nth patient. We maximize the probability of each attribute predicting whether the patient has hypertension, thereby obtaining the minimum target task prediction loss value.

[0031] In step (3), we model the hypertension prediction process within a reinforcement learning framework. We treat each examinee as an individual under reinforcement learning; and we use each examinee's medical records from each visit as the basis for state changes in reinforcement learning. We set up two types of memory to store data samples with different states, solving the problem of uneven distribution of positive and negative samples; we designed a data selection strategy to select data that contributes significantly to hypertension early warning; and we designed a reward and punishment mechanism to solve the overfitting problem caused by the model getting stuck in local optima.

[0032] Individual (agent): We treat each examinee as an individual. Individuals provide personalized hypertension predictions based on their current environmental conditions.

[0033] Action: At time step t, take action a. t This determines whether an individual will develop hypertension. We defined two behavioral choices: behaviors that induce hypertension and behaviors that do not induce hypertension. The specific manifestations are as follows:

[0034]

[0035] State: We initialize the individual's own state as a zero vector. At time step t, we use the sequence obtained from the pre-trained model of the nth examinee's physical examination data as... To express the connection with previous electronic health record records, we averaged the sequence obtained from the pre-trained model before time step t. we will and By concatenating the vectors, we obtain the state vector of the individual at time step t. The calculation formula is as follows:

[0036]

[0037]

[0038]

[0039] in, This represents the sequence value of the i-th examination attribute of the n-th examinee at time step t, obtained through a pre-trained model. It is a splicing of physical examination attributes obtained from a pre-trained model. This represents the sequence vector obtained from the pre-trained electronic health record of the nth examinee at time step i.

[0040] Reward mechanism: In time step t, when an individual takes action a... t Subsequently, the environment provides feedback to evaluate action a. tThe resulting rewards and penalties, the reward function consists of two parts, reward1 t Reward2 is used to judge the accuracy of behavior. t Used to determine the reciprocal of the distance from the correct behavior. Rewards and punishments are expressed in the following ways:

[0041]

[0042]

[0043] reward t =reward1 t +reward2 t

[0044] Here, label represents the actual illness status of the examinee at time step t.

[0045] Priority allocation strategy based on time difference: We define the priority strategy score based on time difference as follows:

[0046]

[0047] y is the discount factor.

[0048] Sample selection strategy: After calculating the priority score x for each sample in memory... e Then, we need to sort them according to priority score x e For each data sample, we calculate the probability of it being selected to be taken from memory. In this paper, we choose to use the softmax function to map the priority score of each data sample, defined by the following formula:

[0049]

[0050] Where P(k) represents the k-th data sample in x (k) The probability of being selected under a priority score.

[0051] Memory management mechanism: In this experiment, due to the large difference between positive and negative samples, a sample imbalance problem exists. This imbalance causes the reinforcement learning model to favor larger samples over smaller ones, severely impacting the model's performance. To address this issue, we propose establishing an additional memory to store the data samples showing changes between positive and negative samples. Specifically, we establish two memories to store the data samples from the positive and negative samples respectively. t and s t+1 The corresponding labels are the same and different (s) t,a t ,r t ,s t+1 Data samples. Storage of s is done using memory m1. t and s t+1 The corresponding labels are the same (s) t ,a t ,r t ,s t+1 Data samples are stored in memory m2. t and s t+1 The corresponding labels are different (s) t ,a t ,r t ,s t+1 Data samples. By using different memories to store data samples of different analogies, it is possible to achieve (s) t ,a t ,r t ,s t+1 Learning from data samples whose states have changed can solve the problem of data imbalance and improve the training effect of the model.

[0052] In step (4), we input the individual's multiple medical visits data into the pre-trained model to obtain the multiple medical visits data vector representation, and input the multiple medical visits data vector representation into the reinforcement learning framework to obtain the probability that the individual will develop hypertension within 3 years. Attached Figure Description

[0053] Figure 1 This is a diagram of the overall architecture of the present invention. The overall architecture is divided into three modules: the input module, the model details, and the output module. The input is divided into single-visit data input and multiple-visit data input. Single-visit data is input into the pre-trained model for self-composition prediction and target prediction tasks to obtain appropriate model initialization parameters. The pre-trained model is then fed into the reinforcement learning prediction task. Multiple physical examination data are used in the reinforcement learning framework. Through reward and punishment mechanisms and sample selection strategies, the final hypertension prediction result is obtained through the neural network. Detailed Implementation

[0054] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0055] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0056] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0057] Example 1

[0058] Figure 1 The overall architecture diagram of the present invention is shown.

[0059] The overall architecture consists of three modules: an input module, a model framework, and an output module. Inputs are divided into single-visit data inputs and multiple-visit data inputs. Single-visit data is input into a pre-trained model for self-composition prediction and target prediction tasks, obtaining appropriate model initialization parameters. The pre-trained model is then fed into a reinforcement learning prediction task. Multiple-visit data is used in the reinforcement learning framework, employing reward and punishment mechanisms, memory management, and sample selection strategies, before being processed by a neural network to obtain the final hypertension prediction result.

[0060] The embodiments of the present invention have been described in detail above. Specific implementation methods have been used to illustrate the present invention. The description of the above embodiments is only for the purpose of helping to understand the method of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A hypertension early warning method based on pre-training and reinforcement learning, characterized in that, The implementation steps of this method are as follows: Step (1) Obtain the personal physical examination dataset, perform deduplication preprocessing on the physical examination data in the personal physical examination dataset, establish dictionaries for each attribute of the physical examination data, and divide the physical examination data into single physical examination data and multiple physical examination data, and store them as PKL files; Step (2) Based on the established medical visit attribute dictionary, encode the single medical visit data and use the Transformer model to pre-train the single physical examination data; Step (3) Based on the characteristics of multiple sequence physical examination data, reinforcement learning modeling is carried out, and the individual, behavior, state, memory management mechanism, reward and punishment mechanism, and data selection strategy of the reinforcement learning framework are designed. Step (4) Use the Transformer pre-trained model to encode and embed multiple physical examination data, and input the encoded multiple physical examination data into the reinforcement learning framework for chronic disease early warning; In step (3), the hypertension reinforcement learning modeling process is as follows: (1) Set each examinee in multiple physical examination data as an individual for reinforcement learning; (2) Set the behavior of "whether the examinee has hypertension" three years later as a reinforcement learning behavior, and divide it into having hypertension after three years and not having hypertension after three years; (3) Set the state to the vector obtained by the individual from the pre-trained model; (4) Set up reward and punishment functions to provide feedback on individual behavior; The calculation of the reward and punishment function is divided into two parts; One part is whether the individual's behavior is consistent with the actual disease record; if consistent, it returns 1, and if inconsistent, it returns -1. The other part is the reciprocal of the square difference between the predicted disease probability and the actual disease probability. (5) Set up a memory management mechanism to store data samples, and set up two storage devices to store time. With time Data samples showing changes and no changes in an individual's disease status; by using different storage devices to store different categories of data samples, the model can learn from data samples showing changes in status, solve the data imbalance problem, and improve the model training effect; (6) Set the priority score for each data sample according to the reward and penalty function; (7) Set a data sample selection strategy to select data samples, using The function maps the priority score of each data sample and randomly selects data samples based on the mapped values.

2. The hypertension early warning method based on pre-training and reinforcement learning according to claim 1, characterized in that, In step (1), the data preprocessing steps for individual physical examination data include: (1) Collect personal physical examination data and select 16 attributes related to hypertension in the physical examination form, including height, weight, body temperature, heart rate, pulse, systolic blood pressure, diastolic blood pressure, waist circumference, BMI, total cholesterol, triglycerides, serum low-density lipoprotein cholesterol, serum high-density lipoprotein cholesterol, exercise frequency, drinking frequency, and exercise frequency. (2) Remove duplicate data from the collected personal physical examination data and fill in the missing data values ​​with nearest neighbor values; (3) Determine whether the data is from a physical examination within the past three years based on the fields of "whether or not you have hypertension", "date of diagnosis of hypertension" and "time of physical examination", and divide the data into positive and negative samples; (4) Divide the individual physical examination data into single physical examination data and multiple physical examination data according to the number of physical examinations of the examinee; establish attribute dictionary tables for the 16 attributes of single physical examination data and multiple physical examination data according to the attribute values, and store them as txt files; (5) Divide the single physical examination data and multiple physical examination data into training and validation sets and test sets in a ratio of 1:

4. Then divide the training and validation sets into training and validation sets in a ratio of 4:1 and store the divided datasets as PKL files.

3. The hypertension early warning method based on pre-training and reinforcement learning according to claim 1, characterized in that, In step (2), a large amount of single physical examination data is used for pre-training to obtain model initialization parameters; the single physical examination data is randomly masked to obtain pre-training vectors; the masking attribute is detected by designing a self-composed prediction task to make the model have stronger self-detection ability; the target prediction task is designed to facilitate downstream hypertension prediction. The pre-training steps for single physical examination data include: (1) Load the single physical examination data PKL file to obtain the single physical examination dataset; (2) For each single physical examination data, one-hot encoding is performed on each physical examination attribute value according to the established attribute dictionary to obtain the embedded representation of each physical examination attribute value of the single physical examination data; (3) Input each single physical examination data into the Transformer model and perform random occlusion operation; in the random attribute occlusion operation, each physical examination attribute has an 80% probability of being used. If a field is masked, there is a 10% probability that the health check attribute remains unchanged, and a 10% probability that it will be replaced with another field in the same health check attribute. (4) The single physical examination data is used to perform a self-composition prediction task based on random masking operation; in the self-composition prediction task, the correctness of the attribute values ​​of each single physical examination data is predicted so that the model can better learn the characteristics of each physical examination attribute. (5) The physical examination attributes in each single physical examination data are concatenated to obtain the embedded representation of each single physical examination data, and the obtained embedded representation is used for the target prediction task; in the target prediction task, the embedded representation of the single physical examination data is used to predict whether the patient has hypertension, thereby improving the model's ability to predict hypertension in the end.

4. The hypertension early warning method based on pre-training and reinforcement learning according to claim 1, characterized in that, In step (4), the Transformer model is used to obtain the encoding embedding, and then reinforcement learning is used for hypertension early warning. The process is as follows: (1) Load the PKL file of multiple physical examination data to obtain the multiple physical examination dataset; (2) Input the multiple physical examination data of each examinee into the Transformer pre-trained model. For each physical examination data, concatenate the encoding vector of the current physical examination data with the average value of the encoding vectors of the previous physical examination data to obtain the final encoding vector for each time. (3) Input the final encoding vector of each examinee into the reinforcement learning model in sequence to obtain the prediction result for each time, and compare it with the true value to obtain the final f1, pr-auc, and jaccard model evaluation indexes.

Images