Medical data analysis early warning method and system based on inverse reinforcement learning and sequence data dynamic modeling

Abstract

The invention discloses a medical data analysis early warning method and system based on inverse reinforcement learning and sequence data dynamic modeling, and the method comprises the following steps: obtaining electronic health record data, and constructing a related data set; according to the constructed data set, a Markov process model is established according to the patient state and the treatment measures, a reward function capable of evaluating the effectiveness of the treatment measures is constructed based on the patient trajectory probability of a trajectory space, and each trajectory in the trajectory space is a state-action pair sequence formed by the patient state and the treatment measures; training the inverse reinforcement learning model by using the data set to obtain an optimal reward function; and obtaining a state prediction value of the current state of the patient after corresponding treatment measures are taken by using the optimal reward function, and if the state prediction value exceeds a set threshold value, carrying out early warning. According to the method, the adaptability to the dynamic change of the patient condition in the medical data analysis and prediction process is improved, and the accuracy of patient state prediction is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data analysis, and more specifically to a medical data analysis early warning system and system based on inverse reinforcement learning and sequence data dynamic modeling. Background Art

[0002] With the development of medical technology and information technology, medical data has become more and more abundant and diverse. For example, electronic health records (EHR) have greatly enhanced the depth and breadth of medical data through the integration and analysis of these data, playing a key role in the treatment of patients.

[0003] In healthcare, medical data early warning systems can detect potential health problems as early as possible, issue early warnings in time, and remind medical staff to intervene by real-time monitoring and analysis of these diverse data. This can not only improve the early detection rate of diseases, but also optimize resource allocation, so that medical resources can be used more reasonably and effectively. Such systems are increasingly widely used in the medical field, especially in chronic disease management, acute disease early warning, and public health monitoring. Traditional methods usually rely on defining static rules and thresholds, lack of early warning of potential risks, and affect the timeliness and effectiveness of preventive measures. In acute conditions, such as sepsis, real-time early warning is very important. The lag of traditional methods may lead to missing the best treatment time. In addition, it is impossible to accurately predict the future changes in the patient's condition, resulting in low early warning accuracy. Therefore, traditional methods have multiple shortcomings in medical data analysis and early warning, such as static rules and thresholds, difficulty in handling complex relationships, lack of real-time performance, and lack of predictive ability. These problems limit the development of medical data analysis and early warning. With the widespread application of machine learning in medicine, medical data analysis and early warning have improved performance in real-time monitoring and early warning, complex data processing, and resource utilization compared with traditional methods. Although machine learning models perform well on specific datasets, their performance may degrade when faced with different environments and populations.

[0004] In summary, the analysis and early warning of medical data are very important in the field of medical safety. At present, there are few dynamic early warning methods for patient status in medical data. How to improve the adaptability of medical data analysis and prediction models to dynamic changes in patient conditions and improve the generalization ability of medical data analysis and prediction models to different environments and populations are issues that need to be urgently addressed. Summary of the invention

[0005] In view of this, the present invention provides a medical data analysis and early warning method based on inverse reinforcement learning and dynamic modeling of sequence data, which is used to at least solve some of the technical problems in the background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] On the one hand, the present invention discloses a medical data analysis and early warning method based on inverse reinforcement learning and sequence data dynamic modeling, comprising the following steps:

[0008] Constructing a data set: acquiring electronic health record data, preprocessing the electronic health record data, and constructing a relevant data set;

[0009] Establishing an inverse reinforcement learning model: Based on the constructed data set, a Markov process model is established according to the patient status and treatment measures, and a reward function that can evaluate the effectiveness of the treatment measures is constructed based on the patient trajectory probability in the trajectory space, where each trajectory in the trajectory space is a state-action pair sequence consisting of the patient status and the treatment measures;

[0010] Model training: Use the data set to train the inverse reinforcement learning model to obtain the optimal reward function;

[0011] Data warning: Use the optimal reward function to obtain the patient's current state prediction value after taking corresponding treatment measures. If the state prediction value exceeds the set threshold, a warning is issued.

[0012] Furthermore, in the step of establishing the inverse reinforcement learning model, a reward function capable of evaluating the effectiveness of treatment measures is constructed based on the patient trajectory probability in the trajectory space, which specifically includes the following steps:

[0013] Obtain the trajectory probability p(τθ) under the maximum entropy distribution, where p represents the state transition probability function, τ represents the state-action pair sequence of each trajectory in the trajectory space, and θ represents the reward weight parameter of the reward function;

[0014] The objective function of maximum likelihood estimation is constructed based on the trajectory probability under the maximum entropy distribution:

[0015]

[0016] Among them, E represents the state trajectory space composed of patient states and treatment measures in the dataset.

[0017] Furthermore, the model training steps specifically include:

[0018] Construct the objective function gradient formula of maximum likelihood estimation:

[0019]

[0020] Among them, θ L represents the gradient of the reward weight parameter θ in the reward function, τ is the patient state trajectory, E represents the state trajectory space composed of the patient state and treatment measures in the dataset, M is the number of trajectories, D represents the trajectory space composed of all possible predicted state-action pairs, and Rθ To accumulate rewards;

[0021] Construct the normalized loss function according to the objective function gradient formula:

[0022]

[0023] Among them, s i represents the data value of the predicted i-th state s, and S represents the set of all predicted state values;

[0024] The normalized loss function is optimized using the conjugate gradient method to obtain the optimal reward function.

[0025] Furthermore, the normalized loss function is optimized using a conjugate gradient method, specifically comprising the following steps:

[0026] Randomly initialize the parameters θ and calculate the initial gradient g 0 =▽f(θ 0 ), set the initial search direction:

[0027] d 0 =-g 0 ;

[0028] Start iteration, set the current number of iterations to k, and calculate the step length α k :

[0029]

[0030] Where A is the Hessian matrix of the loss function, or an approximate matrix;

[0031] Update position θ k+1 :

[0032] θk +1 =θk+αkdk;

[0033] Calculate the new gradient g k+1 :

[0034] gk +1 =▽f(θk +1 );

[0035] Calculate the conjugation coefficient β k :

[0036]

[0037] Update search direction d k+1 :

[0038] d k+1 =-g k+1 +β k dk ;

[0039] If the gradient reaches the convergence condition: ||g k+1 ||<∈, ∈ is the set threshold, then the iteration stops.

[0040] Furthermore, the optimal reward function is used to obtain the predicted value of the patient's current state after taking corresponding treatment measures, specifically including constructing the following model:

[0041] S t+T =R(s)+S t

[0042] Among them, R(s) represents the optimized reward function, S t represents the patient status at the current time t, S t+T Indicates the patient's status at time T+t.

[0043] Furthermore, if the state prediction value exceeds a set threshold, an early warning is issued, which specifically includes the following steps:

[0044] The expert consensus patient status safety threshold is designed into the dictionary form of {'s':range(a,b)}, where a is the lower limit of the safety value and b is the upper limit of the safety value;

[0045] The safety change value of the patient status in the data set within the time interval T is designed as a dictionary form of {'s_change':c}, where c is the expert consensus safety value of the patient's status s change within a certain time T;

[0046] When the patient's status at the next moment is lower than the safety value lower limit a, or the patient's status at the next moment is higher than the safety value upper limit b, or the intermediate change value F(s) exceeds the safety change value c; it is judged that the state prediction value exceeds the set threshold and relevant warnings are issued.

[0047] Furthermore, the electronic health record data is preprocessed to construct a relevant data set, which specifically includes the following steps:

[0048] Missing values ​​in raw electronic health record data were filled and outliers were removed.

[0049] On the other hand, the present invention also discloses a medical data analysis and early warning system based on inverse reinforcement learning and sequence data dynamic modeling, comprising:

[0050] Dataset building module: used to obtain electronic health record data and pre-process the electronic health record data to obtain relevant data sets;

[0051] Inverse reinforcement learning model construction module: Based on the constructed data set, a Markov process model is established according to the patient status and treatment measures, and a reward function that can evaluate the effectiveness of treatment measures is constructed based on the patient trajectory probability in the trajectory space, where each trajectory in the trajectory space is a state-action pair sequence consisting of the patient status and treatment measures;

[0052] Model training module: Use the data set to train the inverse reinforcement learning model to obtain the optimal reward function;

[0053] Data warning module: Use the optimal reward function to obtain the patient's current state prediction value after taking corresponding treatment measures. When the state prediction value exceeds the set threshold, a warning message is generated.

[0054] Preferably, in the inverse reinforcement learning model construction module, a reward function capable of evaluating the effectiveness of treatment measures is constructed based on the patient trajectory probability in the trajectory space, specifically comprising the following steps:

[0055] Obtain the trajectory probability p(τθ) under the maximum entropy distribution, where p represents the state transition probability function, τ represents the state-action pair sequence of each trajectory in the trajectory space, and θ represents the reward weight parameter of the reward function;

[0056] The objective function of maximum likelihood estimation is constructed based on the trajectory probability under the maximum entropy distribution:

[0057]

[0058] Among them, E represents the state trajectory space composed of patient states and treatment measures in the dataset.

[0059] Preferably, in the model training module, the inverse reinforcement learning model is trained using the data set to obtain the optimal reward function, which specifically includes the following steps:

[0060] Construct the objective function gradient formula of maximum likelihood estimation;

[0061] Construct the normalized loss function according to the objective function gradient formula:

[0062] The normalized loss function is optimized using the conjugate gradient method to obtain the optimal reward function.

[0063] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a medical data analysis and early warning method and system based on inverse reinforcement learning and sequence data dynamic modeling, which has the following beneficial effects:

[0064] Under the framework of deep maximum entropy inverse reinforcement learning, the present invention adds the maximum entropy principle to inverse reinforcement learning, ensuring the diversity and flexibility of the strategy, so that the model can adapt to different clinical scenarios and patient needs. By learning the patient state motion trajectory in medical data, it can improve the accuracy of early warning, reduce false alarms and missed alarms, and improve the timeliness and effectiveness of medical intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0066] Figure 1 The present invention provides a schematic diagram of the overall process of the method.

[0067] Figure 2 A schematic diagram of the steps for obtaining the optimized reward function provided by the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] The embodiment of the present invention first discloses a medical data analysis and early warning method based on inverse reinforcement learning and sequence data dynamic modeling, such as Figure 1 As shown, the following steps are included:

[0070] Constructing a data set: acquiring electronic health record data, preprocessing the electronic health record data, and constructing a relevant data set;

[0071] Establishing an inverse reinforcement learning model: Based on the constructed data set, a Markov process model is established according to the patient status and treatment measures, and a reward function that can evaluate the effectiveness of the treatment measures is constructed based on the patient trajectory probability in the trajectory space, where each trajectory in the trajectory space is a state-action pair sequence consisting of the patient status and the treatment measures;

[0072] Model training: Use the data set to train the inverse reinforcement learning model to obtain the optimal reward function;

[0073] Data warning: Use the optimal reward function to obtain the patient's current state prediction value after taking corresponding treatment measures. If the state prediction value exceeds the set threshold, a warning is issued.

[0074] In the dataset construction step, electronic health record data can be collected from international open source medical databases, including patient data information within 15 days after diagnosis, including patient demographic characteristics, vital signs, laboratory test results, disease diagnosis results, and treatment records.

[0075] The missing values ​​in the original electronic health record data are filled and outliers are removed to obtain the complete dataset H.

[0076] In order to further reduce the complexity of data processing and improve the accuracy of medical data analysis, medical data can be clustered during the data set construction process, such as k-means clustering, to identify patient groups with different characteristics, such as high-risk groups and low-risk groups, so as to carry out targeted data processing.

[0077] In the step of establishing the inverse reinforcement learning model, the state space S and action space A are defined. The state space includes the patient's state data, and the action space is the treatment measures during the treatment process, including medication, mechanical ventilation, fluid resuscitation, etc.

[0078] Specifically, the state-related data in the data set H can be defined as a Markov process (MDP) form: {s, a, r, p}, s is the patient state, a is the treatment measure, r is the reward function, and p is the state transition probability, which represents the probability of going from the current state s to the next state s': P ss’ =[S t+1 =s'|S t =s];

[0079] The patient trajectory space has multiple trajectories, each of which is a sequence of state-action pairs:

[0080] τ={(s 1 ,a 1 ),(s 2 ,a 2 )...(s k ,a k )},

[0081] Among them, (s k ,a k ) is a patient state-action pair. The probability of each trajectory in the trajectory space is different. The probability of the trajectory is:

[0082]

[0083] Among them, P(τ|R) represents the probability of generating trajectory τ under the reward function R, and Z(R) is called the partition function, which is used to ensure that the sum of the probabilities is 1:

[0084]

[0085] The objective function of maximum likelihood estimation is constructed based on the trajectory probability under the maximum entropy distribution:

[0086]

[0087] Where E represents the state trajectory space composed of the patient status and treatment measures in the data set. Substituting the trajectory probability under the maximum entropy distribution, the objective function is expressed as:

[0088]

[0089] Where E is the trajectory space in the patient's historical data, τ is the patient's trajectory, p(τ|θ) is the final distribution of the parameterized Markov process trajectory, M is the number of trajectories, D is the trajectory space of the sample, and R θ is the cumulative reward, θ is the reward weight parameter, Z = ∫exp(R θ (τ))dτ. Cumulative reward R θ Obtained by the following formula:

[0090] Where γ is the discount factor, which indicates the importance of future rewards. The larger the value, the more important the future rewards are. r(st, at) is the reward at time t.

[0091] After obtaining the reward objective function of the inverse reinforcement learning model, the inverse reinforcement learning model is trained using the data set to obtain the optimal reward function.

[0092] First, calculate the gradient formula of the reward function parameter θ:

[0093]

[0094] Among them, θ L represents the gradient of the reward weight parameter θ in the reward function, τ is the patient state trajectory, E represents the state trajectory space composed of the patient state and treatment measures in the dataset, M is the number of trajectories, D represents the trajectory space composed of all possible predicted state-action pairs, and R θ For cumulative rewards.

[0095] Then the normalized loss function of the objective function is obtained according to the gradient formula:

[0096]

[0097] Among them, s irepresents the data value of the predicted i-th state s, and S represents the set of all predicted state values.

[0098] The normalized loss function is optimized using the conjugate gradient method to obtain the optimal reward function. Figure 2 As shown, the conjugate gradient method is used to optimize the normalized loss function, which specifically includes the following steps:

[0099] Randomly initialize the parameters θ and calculate the initial gradient g 0 =▽f(θ 0 ), set the initial search direction:

[0100] d 0 =-g 0 ;

[0101] Start iteration, set the current number of iterations to k, and calculate the step length α k :

[0102]

[0103] Where A is the Hessian matrix of the loss function, or an approximate matrix;

[0104] Update position θ k+1 :

[0105] θk +1 =θk+αkdk;

[0106] Calculate the new gradient g k+1 :

[0107] gk +1 =▽f(θk +1 );

[0108] Calculate the conjugation coefficient β k :

[0109]

[0110] Update search direction d k+1 :

[0111] d k+1 =-g k+1 +β k d k ;

[0112] If the gradient reaches the convergence condition: ||g k+1 ||<∈, ∈ is the set threshold, then the iteration is stopped, and the optimized reward function parameter θ is obtained.

[0113] Updating the position is actually updating the model parameters, which are usually directly related to the patient's characteristics. In the patient status prediction model, the parameters correspond to the patient's status. Each time the position is updated, the model adjusts these parameters during the optimization process to improve the accuracy of the prediction of the patient's status. When using the conjugate gradient method, it is necessary to determine the search direction based on the gradient information. This can be interpreted as looking for a direction that can improve the prediction effect in the parameter space of specific patient data. For example, the search direction can represent the potential effectiveness of a treatment regimen in a specific patient population.

[0114] Define the intermediate state change value as F(s) = |St+T-St|, where St and St+T represent the values ​​of the patient state s at time t and time T+t, respectively, and obtain the patient state data prediction model. Under this model, the probability of trajectories with the same reward is the same. The higher the reward, the greater the probability of the trajectory. Since different action selections will lead to different states of the patient, in the patient state data prediction model, actions will be randomly selected, and the model trained with data can predict the patient's state based on the selected action.

[0115] After the model is trained, the K-fold crossover method is used to verify the stability of the model. Specifically, the data of each class can be divided into K parts, and one of them is used in turn as the test set H. 1 , and the rest are used as training set H 2 .

[0116] In the inverse reinforcement learning algorithm, the agent can learn the potential reward function (which can be understood as the relationship between the patient's state and the treatment measures in the present invention) from the sample data, so it can predict the next state (i.e., the patient's state after treatment) based on this reward function and the current patient's state and treatment measures (actions), thereby realizing the prediction of the patient's state. The input includes the patient's state and the treatment measures: for example, the patient's state includes laboratory results (white blood cell count, platelet count, etc.), and the treatment measures include the use of certain drugs, mechanical ventilation, etc. The output is the patient's next state (i.e., the predicted state).

[0117] Finally, the reward function and expert consensus patient status safety threshold are used to construct the patient status safety threshold and issue an early warning.

[0118] The specific implementation method of this step is: design the expert consensus patient status safety threshold into the dictionary form of {'s':range(a,b)}, where a is the minimum safety value and b is the maximum safety value. For example, if the patient's heart rate ranges from 60 to 100 beats per minute, a can be set to 60 and b to 100.

[0119] The safe change value of the patient's status within the time interval T in the dataset is designed in the form of a dictionary {'s_change':c}, where c is the expert consensus safety value of the patient's status s change within a certain time T. For example, the safe change value of the patient's body temperature within 24 hours is 1°C, and the patient's status safety threshold is obtained.

[0120] In this step, if the predicted value of the current state after taking corresponding treatment measures exceeds the safety threshold (a, b) or the intermediate change value F(s) exceeds the safe change value c, the system generates an early warning message to remind medical personnel to take corresponding measures for the patient.

[0121] The design idea of ​​the present invention is: in the field of medical safety, using deep maximum entropy inverse reinforcement learning as a framework, constructing a patient status data prediction model, and by learning the hidden reward function in the data sample, more accurately predicting the patient's status and achieving early warning. Although the medical data early warning system has significant advantages in improving the quality of medical services, improving patient safety and optimizing resource management, it still faces many challenges and problems in actual application. Therefore, the present invention can simplify data analysis and monitoring by dividing the monitoring area into multiple clusters. Data points near the cluster center often have similar characteristics, which helps to reduce the number of variables that need to be monitored. In order to judge whether the patient's status is safe, the patient's intermediate dynamic safety change value is added on the basis of the expert consensus status safety threshold, realizing real-time dynamic early warning for patients and improving the accuracy of patient status prediction.

[0122] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0123] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A medical data analysis and early warning method based on inverse reinforcement learning and sequence data dynamic modeling, characterized in that: The following steps are involved: Constructing a data set: acquiring electronic health record data, preprocessing the electronic health record data, and constructing a relevant data set; Establishing an inverse reinforcement learning model: Based on the constructed data set, a Markov process model is established according to the patient status and treatment measures, and a reward function that can evaluate the effectiveness of the treatment measures is constructed based on the patient trajectory probability in the trajectory space, where each trajectory in the trajectory space is a state-action pair sequence consisting of the patient status and the treatment measures; Model training: Use the data set to train the inverse reinforcement learning model to obtain the optimal reward function; Data warning: Use the optimal reward function to obtain the patient's current state prediction value after taking corresponding treatment measures. If the state prediction value exceeds the set threshold, an early warning is issued.

2. According to claim 1, a medical data analysis and early warning method based on inverse reinforcement learning and sequence data dynamic modeling is characterized in that: In the step of establishing the inverse reinforcement learning model, a reward function that can evaluate the effectiveness of treatment measures is constructed based on the patient trajectory probability in the trajectory space, which specifically includes the following steps: Obtain the trajectory probability p(τθ) under the maximum entropy distribution, where p represents the state transition probability function, τ represents the state-action pair sequence of each trajectory in the trajectory space, and θ represents the reward weight parameter of the reward function; The objective function of maximum likelihood estimation is constructed based on the trajectory probability under the maximum entropy distribution: Among them, E represents the state trajectory space composed of patient states and treatment measures in the dataset.

3. According to claim 2, a medical data analysis and early warning method based on inverse reinforcement learning and sequence data dynamic modeling is characterized in that: The model training steps include: Construct the objective function gradient formula of maximum likelihood estimation: Among them, θ L represents the gradient of the reward weight parameter θ in the reward function, τ is the patient state trajectory, E represents the state trajectory space composed of the patient state and treatment measures in the dataset, M is the number of trajectories, D represents the trajectory space composed of all possible predicted state-action pairs, and R θ To accumulate rewards; Construct the normalized loss function according to the objective function gradient formula: Among them, s i represents the data value of the predicted i-th state s, and S represents the set of all predicted state values; The normalized loss function is optimized using the conjugate gradient method to obtain the optimal reward function.

4. The medical data analysis and early warning method based on inverse reinforcement learning and sequence data dynamic modeling according to claim 3 is characterized in that: The normalized loss function is optimized using the conjugate gradient method, specifically comprising the following steps: Randomly initialize the parameter θ, calculate the initial gradient g0 = ▽f(θ0), and set the initial search direction: d0 = -g0; Start iteration, set the current number of iterations to k, and calculate the step length α k : Where A is the Hessian matrix of the loss function, or an approximate matrix; Update position θ k+1 : i k+1 =θ k +a k d k ; Calculate the new gradient g k+1 : g k+1 =▽f(θ k+1 ); Calculate the conjugation coefficient β k : Update search direction d k+1 : d k+1 =-g k+1 +b k d k ; If the gradient reaches the convergence condition: ||g k+1 ||<∈, ∈ is the set threshold, then the iteration stops.

5. According to claim 1, a medical data analysis and early warning method based on inverse reinforcement learning and sequence data dynamic modeling is characterized in that: The optimal reward function is used to obtain the predicted value of the patient's current state after taking corresponding treatment measures, which specifically includes building the following model: S t+T =R(s)+S t Among them, R(s) represents the optimized reward function, S t represents the patient status at the current time t, S t+T Indicates the patient's status at time T+t.

6. The medical data analysis and early warning method based on inverse reinforcement learning and sequence data dynamic modeling according to claim 2 is characterized in that: If the state prediction value exceeds the set threshold, an early warning is issued, which specifically includes the following steps: The expert consensus patient status safety threshold is designed into the dictionary form of {'s':range(a,b)}, where a is the lower limit of the safety value and b is the upper limit of the safety value; The safety change value of the patient status in the data set within the time interval T is designed as a dictionary form of {'s_change':c}, where c is the expert consensus safety value of the patient's status s change within a certain time T; When the patient's status at the next moment is lower than the safety value lower limit a, or the patient's status at the next moment is higher than the safety value upper limit b, or the intermediate change value F(s) exceeds the safety change value c; it is judged that the state prediction value exceeds the set threshold and relevant warnings are issued.

7. The medical data analysis and early warning method based on inverse reinforcement learning and sequence data dynamic modeling according to claim 1 is characterized in that: Preprocessing the electronic health record data to construct a relevant data set specifically includes the following steps: Missing values ​​in raw electronic health record data were filled and outliers were removed.

8. A medical data analysis and early warning system based on inverse reinforcement learning and sequence data dynamic modeling, characterized in that: include: Dataset building module: used to obtain electronic health record data and pre-process the electronic health record data to obtain relevant data sets; Inverse reinforcement learning model construction module: Based on the constructed data set, a Markov process model is established according to the patient status and treatment measures, and a reward function that can evaluate the effectiveness of treatment measures is constructed based on the patient trajectory probability in the trajectory space, where each trajectory in the trajectory space is a state-action pair sequence consisting of the patient status and treatment measures; Model training module: Use the data set to train the inverse reinforcement learning model to obtain the optimal reward function; Data warning module: Use the optimal reward function to obtain the patient's current state prediction value after taking corresponding treatment measures. When the state prediction value exceeds the set threshold, a warning message is generated.

9. A medical data analysis and early warning system based on inverse reinforcement learning and sequence data dynamic modeling according to claim 8, characterized in that: In the inverse reinforcement learning model construction module, a reward function capable of evaluating the effectiveness of treatment measures is constructed based on the patient trajectory probability in the trajectory space, specifically including the following steps: Obtain the trajectory probability p(τθ) under the maximum entropy distribution, where p represents the state transition probability function, τ represents the state-action pair sequence of each trajectory in the trajectory space, and θ represents the reward weight parameter of the reward function; The objective function of maximum likelihood estimation is constructed based on the trajectory probability under the maximum entropy distribution: Among them, E represents the state trajectory space composed of patient states and treatment measures in the dataset.

10. The medical data analysis and early warning system based on inverse reinforcement learning and sequence data dynamic modeling according to claim 8, characterized in that: In the model training module, the inverse reinforcement learning model is trained using the data set to obtain the optimal reward function, which specifically includes the following steps: Construct the objective function gradient formula of maximum likelihood estimation; Construct the normalized loss function according to the objective function gradient formula: The normalized loss function is optimized using the conjugate gradient method to obtain the optimal reward function.

Images