Neurosurgical patient rehabilitation assessment and training method based on reinforcement learning

Abstract

The application provides a neurosurgical patient rehabilitation evaluation and training method based on reinforcement learning, comprising: S1. constructing a neurosurgical patient rehabilitation data acquisition system; S2. data preprocessing and feature engineering; S3. constructing a rehabilitation evaluation and training model fused with Q-learning-DQN-PPO; S4. model training and parameter optimization; S5. rehabilitation evaluation and training implementation; and S6. model effect evaluation and updating. Through the whole-process design of 'data-model-application-feedback', the application realizes the intelligentization, individualization and precision of neurosurgical patient rehabilitation.

Description

Technical Field

[0001] This invention relates to the fields of data processing, neurosurgical rehabilitation medicine and artificial intelligence, and in particular to a method for rehabilitation assessment and training of neurosurgical patients based on reinforcement learning. Background Technology

[0002] The rehabilitation process for neurosurgical patients (such as those recovering from brain tumors, cerebral hemorrhage, or traumatic brain injury) involves the coordinated recovery of multiple dimensions, including motor function, cognitive function, and physiological indicators. Their rehabilitation assessment and training need to be precisely matched to individual patient differences (such as age, severity of illness, and postoperative recovery stage). Currently used traditional methods have the following limitations:

[0003] The assessment is highly subjective: it relies on medical staff to use scales (such as MMSE, muscle strength classification) based on their experience, and the results are easily affected by differences in human judgment, making it difficult to quantify subtle changes in the patient's rehabilitation.

[0004] Homogeneous training programs: They often use standardized training procedures (such as fixed-set joint movement training) without fully considering the patient's real-time condition (such as daily muscle strength fluctuations and cognitive fatigue), which may lead to undertraining or overtraining.

[0005] Insufficient integration of technologies: Existing rehabilitation solutions based on reinforcement learning mostly use a single algorithm (such as DQN or PPO), each with its own limitations. Q-learning is difficult to handle high-dimensional continuous states (such as EEG features), DQN has poor training stability, and PPO is highly dependent on the initial strategy. None of them can meet the needs of complex rehabilitation scenarios on their own.

[0006] Lack of feedback loop: There is no dynamic correlation between the evaluation results and the adjustment of the training program. Medical staff need to manually analyze the data and adjust the program, which is slow and inefficient. Summary of the Invention

[0007] This invention provides a reinforcement learning-based method for rehabilitation assessment and training of neurosurgical patients. First, a comprehensive rehabilitation data acquisition system is constructed to collect multi-dimensional data, including patient basic information, physiological indicators, motor function, cognitive function, and daily living abilities. This data is preprocessed and feature-engineered into a standardized feature dataset. Then, in model construction, a collaborative fusion model is built by integrating three algorithms: Q-learning, Deep Q-Network (DQN), and Proximal Policy Optimization (PPO). During model training, the parameters of the three algorithms are synchronously updated according to interaction rules using partitioned training and validation sets, and performance evaluation and optimization strategies are employed to ensure the model meets standards. In practical applications, personalized plans are generated for new patients, feedback data is collected in real time to update the model, and the training plan is adjusted periodically. Simultaneously, the model's effectiveness is evaluated every three months, and data is supplemented and the model iterated to address issues, ensuring long-term adaptation to clinical needs and ultimately improving the efficiency and quality of neurosurgical patient rehabilitation.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] Reinforcement learning-based methods for rehabilitation assessment and training of neurosurgical patients include:

[0010] S1. Construct a data acquisition system for the rehabilitation of neurosurgical patients, collect patients' basic information, physiological indicators, motor function data, cognitive function data and daily activity ability data, and form a raw database for patient rehabilitation;

[0011] S2. Clean, standardize, and extract features from the original patient rehabilitation database to obtain a patient rehabilitation feature dataset;

[0012] S3. Construct a rehabilitation assessment and training model integrating Q-learning, DQN, and PPO. This model includes a Q-learning module, a Deep Q-Network (DQN) module, and a proximal policy optimization (PPO) module. Define the agent, environment, state, action, and reward. The agent is a decision-making system integrating the three modules. The environment is the patient's rehabilitation state and training scenario. The state consists of the patient's rehabilitation feature dataset. Actions include motor function training actions, cognitive function training tasks, and assessment actions. The reward is quantified based on the changes in rehabilitation indicators after the patient performs the action. The Q-learning module stores the value of state-action pairs through a Q-table. The Deep Q-Network module approximates the Q-value function through a neural network. The proximal policy optimization module outputs the action probability distribution through a policy network.

[0013] S4. Use the patient rehabilitation feature dataset to train and optimize the rehabilitation assessment and training model to obtain the optimized fusion model;

[0014] S5. Input the current state obtained by processing the rehabilitation data of new patients through S2 into the optimized fusion model, output the optimal rehabilitation action, collect the new state and reward after the patient performs the action and feed it back to the fusion model to update the parameters, evaluate and adjust the training program regularly until the patient reaches the rehabilitation standard.

[0015] In this specification, in step S3, the interaction process between the Q-learning module and the DQN module is as follows: the initial value of the Q-table of the Q-learning module is fused with the initial prediction value of the DQN module, and the target Q value of the DQN module is calculated and fused with the Q-table value of the Q-learning module.

[0016] In this specification, in step S3, the interaction process between the DQN module and the PPO module is as follows: the experience pool of the DQN module and the experience pool of the PPO module share interactive data, and the advantage function calculation of the PPO module merges the Q value of the DQN module.

[0017] In this specification, in step S3, the interaction process between the PPO module and the Q-learning module is as follows: when the Q-learning module selects an action using an ε-greedy policy, the probability distribution of the random action is referenced to the action probability output by the policy function of the PPO module.

[0018] In this instruction manual, in step S3, the action decision of the rehabilitation assessment and training model is obtained by weighted fusion of the outputs of three modules. The weights include the Q-table weight of the Q-learning module, the Q-value weight of the DQN module, and the logarithmic weight of the action probability of the PPO module, and the sum of the three is 1.

[0019] In this instruction manual, in step S5, a real-time feedback closed loop is formed: after the patient performs the optimal rehabilitation action, the new status and reward collected are synchronously updated in the Q table of the Q-learning module, the experience pool of the DQN module, and the trajectory data of the PPO module, so as to realize the online learning of the fusion model.

[0020] In this manual, the termination conditions for the real-time feedback loop are: the patient's daily activity ability score reaches the preset threshold, the muscle strength level recovers to the preset level, or the rehabilitation indicators in two consecutive assessments show no significant improvement (consistent or lower than the preset error).

[0021] In this specification, the model optimization strategy in step S4 includes: if the Q-learning module overfits, increasing the number of state discretization intervals and reducing its decision weights; if the loss function of the DQN module fluctuates too much, reducing the learning rate and increasing the experience pool capacity; if the policy update of the PPO module is unstable, decreasing the clipping parameters and increasing the trajectory collection length.

[0022] The reinforcement learning-based method for assessing and training neurosurgical patients’ rehabilitation also includes step S6: regularly assessing the effectiveness of the fusion model. The assessment indicators include the average rehabilitation time of patients, the incidence of complications, the satisfaction of medical staff, and the consistency between the model’s decisions and the expert’s plan.

[0023] In this manual, in step S6, if the evaluation indicators do not meet the standards, new patient data and new features are added, the model is retrained and the version is updated; the new patient data is obtained through the data acquisition system in S1, and the new features include joint load index and language fluency. The supplemented data is processed in S2 and then added to the training set.

[0024] In summary, the present invention has at least the following beneficial effects:

[0025] Enhancing the objectivity and accuracy of assessments: By integrating the stability of Q-learning, the high-dimensional feature fitting ability of DQN, and the probabilistic modeling advantages of PPO, subjective scale assessments are transformed into data-driven quantitative decisions, which can capture subtle rehabilitation changes that are difficult to identify using traditional methods (such as fluctuations in word comprehension ability in cognitive function).

[0026] Achieve dynamic adjustment of personalized training programs: Through real-time interaction of three algorithms (such as Q-table providing stable target values ​​for DQN and PPO guiding the direction of exploration), the training program can be adaptively adjusted according to the patient's real-time status (such as changes in heart rate after exercise and cognitive task completion), avoiding the limitations of homogeneous training.

[0027] Enhancing model adaptability and robustness: Complementary mechanisms between algorithms (such as DQN for handling high-dimensional physiological features and Q-learning for ensuring stable low-dimensional state decisions) enable the model to adapt to different types of neurosurgical patients (such as children and elderly patients, and patients with motor dysfunction and cognitive impairment), reducing performance degradation caused by differences in data distribution.

[0028] Constructing an efficient feedback loop: Evaluation results directly drive the update of training programs, and the model is continuously optimized through online learning (such as real-time updates of algorithm parameters based on feedback from patients after performing actions), reducing manual intervention by medical staff and improving the response speed and efficiency of the rehabilitation process.

[0029] Improving rehabilitation safety and patient compliance: The integrated strategy optimizes rehabilitation outcomes while reducing the risk of complications during training through PPO strategy constraints (such as avoiding high-risk movements) and safety penalties in the reward function; personalized programs are more tailored to the patient's abilities, which can reduce training frustration and increase active participation. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating the reinforcement learning-based neurosurgical patient rehabilitation assessment and training method involved in this invention.

[0032] Figure 2 This is a schematic diagram of the data preprocessing and feature engineering process involved in this invention.

[0033] Figure 3 This is a schematic diagram of the process for constructing a rehabilitation assessment and training model that integrates Q-learning-DQN-PPO, as described in this invention.

[0034] Figure 4 This is a schematic diagram of the model training and optimization process involved in this invention. Detailed Implementation

[0035] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0036] The following disclosure provides many different implementations or examples for carrying out different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the embodiments of the present invention; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0037] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0038] like Figure 1 As shown, this embodiment provides a reinforcement learning-based method for the rehabilitation assessment and training of neurosurgical patients, including:

[0039] S1. Construct a data acquisition system for the rehabilitation of neurosurgical patients, collect patients' basic information, physiological indicators, motor function data, cognitive function data and daily activity ability data, and form a raw database for patient rehabilitation;

[0040] S2. Clean, standardize, and extract features from the original patient rehabilitation database to obtain a patient rehabilitation feature dataset;

[0041] S3. Construct a rehabilitation assessment and training model integrating Q-learning, DQN, and PPO. This model includes a Q-learning module, a Deep Q-Network (DQN) module, and a Proximal Policy Optimization (PPO) module. Define the agent, environment, state, action, and reward. The agent is a decision-making system integrating the three modules. The environment is the patient's rehabilitation state and training scenario. The state consists of the patient's rehabilitation feature dataset. Actions include motor function training actions, cognitive function training tasks, and assessment actions. The reward is quantified based on the changes in rehabilitation indicators after the patient performs the action. The Q-learning module stores the value of state-action pairs through a Q-table. The DQN module approximates the Q-value function through a neural network. The PPO module outputs the action probability distribution through a policy network.

[0042] S4. Use the patient rehabilitation feature dataset to train and optimize the rehabilitation assessment and training model to obtain the optimized fusion model;

[0043] S5. Input the current state obtained by processing the rehabilitation data of new patients through S2 into the optimized fusion model, output the optimal rehabilitation action, collect the new state and reward after the patient performs the action and feed it back to the fusion model to update the parameters, evaluate and adjust the training program regularly until the patient reaches the rehabilitation standard.

[0044] In some embodiments, in step S3, the interaction process between the Q-learning module and the DQN module is as follows: the initial value of the Q-table of the Q-learning module is fused with the initial prediction value of the DQN module, and the target Q value of the DQN module is calculated and fused with the Q-table value of the Q-learning module.

[0045] In some embodiments, in step S3, the interaction process between the DQN module and the PPO module is as follows: the experience pool of the DQN module and the experience pool of the PPO module share interactive data, and the advantage function calculation of the PPO module merges the Q value of the DQN module.

[0046] In some embodiments, in step S3, the interaction process between the PPO module and the Q-learning module is as follows: when the Q-learning module selects an action using an ε-greedy policy, the probability distribution of the random action is referenced to the action probability output by the policy function of the PPO module.

[0047] In some embodiments, in step S3, the action decision of the rehabilitation assessment and training model is obtained by weighted fusion of the outputs of three modules. The weights include the Q-table weight of the Q-learning module, the Q-value weight of the DQN module, and the logarithmic weight of the action probability of the PPO module, and the sum of the three is 1.

[0048] In some embodiments, in step S5, a real-time feedback closed loop is formed: after the patient performs the optimal rehabilitation action, the new status and reward collected are synchronously updated in the Q table of the Q-learning module, the experience pool of the DQN module, and the trajectory data of the PPO module, so as to realize the online learning of the fusion model.

[0049] In some embodiments, the termination condition for the real-time feedback loop is: the patient's daily activity ability score reaches a preset threshold, the muscle strength level recovers to a preset level, or the rehabilitation indicators in two consecutive assessments show no significant improvement.

[0050] In some embodiments, in step S4, the model optimization strategy includes: if the Q-learning module overfits, increasing the number of state discretization intervals and reducing its decision weights; if the loss function of the DQN module fluctuates too much, reducing the learning rate and increasing the experience pool capacity; if the policy update of the PPO module is unstable, decreasing the clipping parameters and increasing the trajectory collection length.

[0051] The reinforcement learning-based method for assessing and training neurosurgical patients’ rehabilitation also includes step S6: regularly assessing the effectiveness of the fusion model. The assessment indicators include the average rehabilitation time of patients, the incidence of complications, the satisfaction of medical staff, and the consistency between the model’s decisions and the expert’s plan.

[0052] In some embodiments, in step S6, if the evaluation indicators do not meet the standards, new patient data and new features are added, the model is retrained and the version is updated; the new patient data is obtained through the data acquisition system in S1, and the new features include joint load index and language fluency. The supplemented data is processed in S2 and then added to the training set.

[0053] The technical concept of this invention is as follows:

[0054] Reinforcement learning-based methods for rehabilitation assessment and training of neurosurgical patients include the following:

[0055] S1. Constructing a data acquisition system for neurosurgical patient rehabilitation.

[0056] To provide comprehensive and accurate raw data for subsequent fusion models, a rehabilitation data acquisition system covering multiple dimensions needs to be built. The specific steps are as follows:

[0057] First, clarify the scope and indicators of data collection.

[0058] Patient basic information includes name, age, gender, height, weight, type of neurosurgical disease (such as brain tumor, cerebral hemorrhage, traumatic brain injury, etc.), name of surgery (such as craniotomy for tumor resection, hematoma evacuation, etc.), surgery time and postoperative duration, etc.

[0059] Physiological data include heart rate (beats / minute), systolic blood pressure (mmHg), diastolic blood pressure (mmHg), blood oxygen saturation (%), and electroencephalogram (EEG) (including the frequency and intensity of alpha, beta, theta, and delta waves); motor function data include hip range of motion (°), knee range of motion (°), ankle range of motion (°), upper limb muscle strength grade (0-5), lower limb muscle strength grade (0-5), balance score (0-100 points, measured by a balance tester), stride length (cm), and cadence (steps / minute).

[0060] Cognitive function data included Mini-Mental State Examination (MMSE) scores (0–30), Montreal Cognitive Assessment (MoCA) scores (0–30), digit memory scores (0–10), word comprehension scores (0–10), orientation to time scores (0–5), orientation to place scores (0–5), and calculation ability scores (0–10).

[0061] Daily activity ability data were obtained using the Barthel Index (BI) rating scale (0-100 points), which specifically included the completion of 10 behaviors such as eating, dressing, washing, toileting, walking, and climbing stairs.

[0062] Secondly, appropriate data collection equipment and tools should be selected. Physiological indicators are acquired using electrocardiogram (ECG) monitors (to record heart rate and blood pressure), pulse oximeters (to measure blood oxygen saturation), and electroencephalograms (EEGs) (to collect brain waves). Motor function is acquired using motion capture systems (such as Kinect for gait parameter acquisition and Vicon for precise measurement of joint range of motion), electromyography (EMG) sensors (to assist in muscle strength assessment), and balance testing devices (to generate balance ability scores). Cognitive function is recorded and scored by medical staff through face-to-face questioning based on scales such as MMSE and MoCA. Daily living activities are also assessed and recorded by medical staff using the Barthel Index rating scale.

[0063] Finally, a structured database is established to store the data. The database uses a MySQL architecture and stores data categorized by a three-dimensional index: "Patient ID - Data Type - Acquisition Time." For example, data for the same patient, such as "Motor Function - Hip Joint Range of Motion - August 1, 20xx," is archived separately to ensure that subsequent steps can quickly extract data from specific dimensions. The raw patient rehabilitation database output from this step will serve as the input source for S2 data preprocessing.

[0064] S2. Data Preprocessing and Feature Engineering

[0065] To transform raw data into features that can be directly used for training a fusion model, three steps are required: cleaning, transformation, and extraction. The specific process is as follows: Figure 2 As shown.

[0066] The first step is data cleaning. All data is extracted from the raw database output by S1. For missing values: continuous physiological indicators (such as heart rate and step length) are filled using the mean imputation method (calculating the mean of the indicator for patients with the same disease and postoperative stage); categorical data (such as muscle strength grades) are filled using the mode imputation method (selecting the grade with the highest frequency); for cognitive scale scores (such as MMSE), if ≤2 items are missing, they are filled using linear interpolation of two adjacent scores; otherwise, they are marked as invalid data and the record is removed. For outliers: box plots are used to identify outliers (data exceeding 1.5 times the interquartile range), combined with clinical knowledge (e.g., heart rate >180 bpm or <40 bpm is considered abnormal). After confirming the abnormality, the median of the patient's previous three measurements is used to replace the outlier.

[0067] The second step is data transformation. The cleaned data undergoes standardization: numerical data in physiological indicators and motor function (such as joint range of motion and stride length) are normalized using Min-Max normalization, with the formula: ( This is the original data; and , representing the maximum and minimum values ​​in the data set, respectively; the calculated result x' is the normalized value, which is transformed to the [0,1] interval; the cognitive scale score (e.g., MMSE 0-30) uses a standardized formula. ( The mean, Standard deviation, The values ​​are standardized to eliminate the magnitude differences between different scales; the muscle strength grades (0-5) are used as ordered categorical data and are converted into 6-dimensional binary vectors using one-hot encoding.

[0068] The third step is feature extraction. High-value features are extracted from the transformed data: For motor function, step length variation coefficient (reflecting gait stability) and left-right step length difference (reflecting symmetry) are extracted from gait parameters; the ratio of upper and lower limb range of motion (assessing limb coordination) is extracted from joint range of motion. For cognitive function, a comprehensive cognitive score (a weighted sum with weights of 0.6 and 0.4, respectively) is extracted from MMSE and MoCA scores; the memory-language comprehensive index (mean) is extracted from digit memory and word comprehension scores. For physiological indicators, the power ratio of alpha waves to beta waves (reflecting brain alertness) is extracted from EEG. The final result is a 20-dimensional patient rehabilitation feature dataset, including 8 motor function features, 7 cognitive function features, and 5 physiological indicator features. This dataset will serve as input for the construction and training of the S3 fusion model.

[0069] S3. Construct a rehabilitation assessment and training fusion model based on reinforcement learning.

[0070] This approach integrates three algorithms: Q-learning, Deep Q-Network (DQN), and Proximal Policy Optimization (PPO). Through pairwise interactions, these algorithms collaborate to clarify data transfer and parameter impact at each stage. The specific process is as follows: Figure 3 As shown.

[0071] 3.1 Detailed Definition of Core Elements (Shared by the Three Algorithms)

[0072] 3.1.1 Agent

[0073] The fusion agent is an integrated system comprising three sub-modules: Q-learning, DQN, and PPO. Its core function is to receive the patient's current state 's', calculate the comprehensive value of each action through interaction between the sub-modules, and ultimately output the optimal rehabilitation action. The goal is to maximize the patient’s cumulative rehabilitation reward (i.e., long-term rehabilitation effect).

[0074] 3.1.2 Environment

[0075] The environment is a combination of the patient's rehabilitation setting and physical condition, comprising two core components:

[0076] Patient state space: Consists of the rehabilitation feature dataset output by S2 (such as muscle strength, joint range of motion, cognitive score, etc.);

[0077] State transition rule: When the agent outputs action a, the environment will update the state according to the patient's actual response (e.g., after performing "straight leg raise" training, the patient's hip joint mobility may improve), and provide an immediate reward r.

[0078] The mathematical description is as follows: ;

[0079] in The environmental transfer function (determined by the patient's physiological response and training effect); It refers to the new state of the environment after performing action a (such as changes in rehabilitation characteristics after a patient performs training).

[0080] 3.1.3 State

[0081] State s is a quantitative representation of the patient's current recovery status, composed of features after S2 preprocessing, specifically: ;

[0082] Where d is the feature dimension (determined based on the feature engineering results of S2, for example, 20 dimensions). For the i-th feature, it includes:

[0083] Motor function characteristics (8 items): hip joint range of motion, knee joint range of motion, ankle joint range of motion, upper limb muscle strength grade, lower limb muscle strength grade, balance ability score, stride length, and stride frequency;

[0084] Cognitive function characteristics (7): MMSE score, MoCA score, digit memory score, word comprehension score, time orientation score, place orientation score, and calculation ability score;

[0085] Physiological indicators (5): heart rate, blood pressure, blood oxygen saturation, alpha wave intensity, and beta wave intensity.

[0086] 3.1.4 Action

[0087] Actions are rehabilitation training or assessment behaviors that the agent can choose from; there are 25 types in total, listed below:

[0088] 1. Motor function training exercises (10 types):

[0089] Straight leg raise (hip joint training), knee flexion (knee joint training), ankle pump exercise (ankle joint training), hip abduction (hip joint flexibility training), elbow flexion and extension (elbow joint training), wrist rotation (wrist joint training), finger grasp (hand muscle strength training), shoulder flexion (shoulder joint training), knee extension (knee joint muscle strength training), ankle dorsiflexion (ankle joint muscle strength training).

[0090] 2. Cognitive function training tasks (10 types):

[0091] Number recitation (short-term memory training), number recitation (working memory training), word matching (semantic memory training), image recognition (visual cognition training), simple calculation (logical thinking training), story retelling (language memory training), time orientation (orientation training), location orientation (spatial cognition training), object naming (language expression training), semantic understanding (language comprehension training).

[0092] 3. Assessment actions (5 types):

[0093] Muscle strength assessment (using a six-grade muscle strength grading system), joint range of motion measurement (using a goniometer), MMSE scale assessment (cognitive function screening), MoCA scale assessment (mild cognitive impairment assessment), and Barthel Index (BI) score (daily activities assessment).

[0094] 3.1.5 Rewards

[0095] The reward r is a quantification of the rehabilitation effect of action a, calculated based on the patient's state changes before and after performing the action, using the following formula:

[0096] ;

[0097] in:

[0098] 15 represents the number of key rehabilitation indicators (selected from condition characteristics, such as muscle strength, MMSE score, etc.).

[0099] The weights for the indicators are set by the rehabilitation physician based on their importance; for example, muscle strength has a weight of 0.15, MMSE score has a weight of 0.1, and all weights sum to 1.

[0100] ( The value of the k-th indicator after the action is performed. (The value before execution).

[0101] For example:

[0102] If, after performing "finger grasping" training, the patient's upper limb muscle strength grade improves from grade 2 to grade 3 ( , =0.15), then the contribution is 0.15 × 1 = 0.15;

[0103] If at the same time the MMSE score drops from 26 to 24 ( , =0.1), then the contribution is 0.1×(-2)=-0.2;

[0104] The total reward r = 0.15 - 0.2 + ... (summing up the contributions of all indicators; the final result can be positive or negative).

[0105] 3.2 Model Construction for Three Algorithms (Including Interaction Design)

[0106] 3.2.1 Q-learning model construction

[0107] Q-learning is a table-based model-free algorithm suitable for processing discrete state spaces. Its core is to store the value of state-action pairs through Q-tables and improve the initialization accuracy through interaction with DQN.

[0108] Model structure:

[0109] Table Q is a two-dimensional table with dimension 1. ,in The number of states after discretization (obtained by binning continuous features, for example, dividing each feature into 5 intervals). (This needs to be optimized using state compression technology). Each row corresponds to a state s, each column corresponds to an action a, and the cell value is... (The value of the state-action pair).

[0110] Interaction design with DQN:

[0111] The initial values ​​of the Q-table need to incorporate the prediction results of DQN (leveraging the advantages of DQN in handling high-dimensional states), and the formula is as follows:

[0112] (3.1)

[0113] in:

[0114] The initial values ​​for table Q;

[0115] This is an initial estimate based on historical rehabilitation data (e.g., the average reward of performing action a in state s from the raw database of S1).

[0116] Initial network for DQN (parameters) Predicted values ​​for state-action pairs;

[0117] (Historical data accounts for a larger proportion, ensuring initial stability).

[0118] 3.2.2 DQN Model Construction

[0119] DQN is an algorithm that combines deep learning and Q-learning. It approximates the Q-value function through a neural network and is suitable for high-dimensional continuous state spaces. Its core is to improve training stability and data utilization through interaction with Q-learning and PPO.

[0120] Model structure:

[0121] A multilayer perceptron (MLP) is used as the Q-value network, with the following structure:

[0122] Input layer: 20 neurons (corresponding to 20 features of state s);

[0123] Hidden layer 1: 128 neurons, activation function is ReLU (to handle non-linear relationships);

[0124] Hidden layer 2: 64 neurons, activation function is ReLU;

[0125] Output layer: 25 neurons (Q-values ​​corresponding to 25 actions, no activation function).

[0126] Simultaneously set the target network (its structure is consistent with the current network, and its parameters are...). ), used for stable training.

[0127] Interaction design with Q-learning:

[0128] The target value calculation of DQN requires incorporating the Q-table values ​​from Q-learning (utilizing the stability of the Q-table to suppress fluctuations in the neural network), and the formula is as follows:

[0129] (3.2)

[0130] in:

[0131] The target Q-value for DQN (used to calculate the loss);

[0132] This is a discount factor (a weighted measure of future rewards).

[0133] The Q-table value for Q-learning in the new state s';

[0134] is the target network's prediction of the new state s';

[0135] (Q table accounts for 30%, balancing stability and flexibility).

[0136] Interaction design with PPO:

[0137] DQN's experience pool (a collection storing interaction data) is shared with the PPO to expand the amount of training data. DQN's experience pool Defined as:

[0138] (3.3)

[0139] in It serves as an experience pool for PPOs (storing interaction data between PPOs and the environment), enhancing DQN's ability to generalize action effects through data sharing;

[0140] DQN's experience pool not only contains its own interaction data but also integrates PPO's interaction data. By sharing data, the training sample size is expanded, improving DQN's generalization ability to different action effects and achieving synergy between the two algorithms at the data level.

[0141] 3.2.3 PPO Model Construction

[0142] PPO is an algorithm based on policy gradient that directly optimizes the probability distribution of actions. It is suitable for continuous action spaces. Its core is to improve policy accuracy and exploration efficiency through interaction with DQN and Q-learning.

[0143] Model structure:

[0144] It includes two sub-networks: a policy network and a value network.

[0145] Policy network (outputs action probabilities):

[0146] Input layer (20 neurons) → Hidden layer 1 (128 neurons, ReLU) → Hidden layer 2 (64 neurons, ReLU) → Output layer (25 neurons, Softmax activation), output That is, the probability of action a ( These are parameters of the policy network in PPO.

[0147] Value network (estimating state value):

[0148] Input layer (20 neurons) → Hidden layer 1 (128 neurons, ReLU) → Hidden layer 2 (64 neurons, ReLU) → Output layer (1 neuron, no activation), output That is, the value of state s ( (Parameters of the value network in PPO).

[0149] Interaction design with DQN:

[0150] The advantage function of PPO (which measures the relative value of an action) needs to incorporate the Q-score of DQN (which uses DQN to accurately estimate the value of an action), and the formula is as follows:

[0151] (3.4)

[0152] in The advantage function represents the advantage function of the PPO algorithm when performing action a in state s. It is used to measure the superiority of the action relative to the current policy. The larger the value, the better action a is relative to the current policy. The Q-value of the current DQN network provides a reference for the absolute value of actions. These are the network parameters for DQN.

[0153] Interaction design with Q-learning:

[0154] Q-learning exploration strategies ( In a greedy approach, the probability of choosing a random action is referenced to the policy distribution of the Probabilistic Policy Object (PPO) (using the PPO to model the probability of action effects), i.e.:

[0155] When Q-learning When randomly selecting actions, the probability of action a being selected is... (3.5)

[0156] This avoids meaningless random exploration (e.g., actions that the PPO considers to be highly ineffective will be assigned a low probability even if randomly selected).

[0157] 3.3 Model training for the three algorithms (including the interactive process)

[0158] 3.3.1 Q-learning training process

[0159] The training objective of Q-learning is to update the Q-table. This allows it to more accurately reflect the actual value of state-action pairs, and the update magnitude is adjusted through policy probability adjustment via PPO during training.

[0160] Input data:

[0161] S2 output rehabilitation feature dataset (discretized), DQN initial network parameters Used for initializing the Q table in Equation 3.1), PPO real-time strategy parameters (For Equations 3.5 and 3.6).

[0162] Detailed steps:

[0163] 1. Initialize the Q-table: Calculate according to Equation 3.1 Complete the initialization of the Q table;

[0164] 2. Iterative training (1000 episodes in total, each episode corresponding to a patient's stage of rehabilitation):

[0165] a. Sample the initial state from the dataset. (A patient's condition before training);

[0166] b. For each time step t (each time step corresponds to one training / evaluation action):

[0167] i. Select Action: Use - Greedy strategy, Probability of choice (Optimal action), with The probability is selected according to the distribution in Equation 3.5. (Refer to PPO's exploration);

[0168] ii. Perform the action: Input environment, get new state and rewards ;

[0169] iii. Update the Q table: Adjust the update magnitude based on the probability assessment of the current action by the PPO, using the following formula:

[0170] (3.6)

[0171] in:

[0172] It is a state-action pair in the Q-table of Q-learning. The updated value is shown on the right; the update rule is shown on the right, which follows the temporal difference update idea of ​​Q-learning, that is, adjusting the current Q value based on the current reward and the value of the future state.

[0173] For PPO action Probability assessment (the higher the probability, the greater the update magnitude, and the more valuable the effective action).

[0174] It is the learning rate of Q-learning, which controls the overall update step size;

[0175] It is the time difference error (TD error), which reflects the gap between the current Q value and "actual reward + future optimal value";

[0176] iv. Status Update: Let If the patient reaches the preset rehabilitation goal (such as muscle strength recovering to level 5) or the number of training sessions reaches the upper limit (such as 30 times / episode), the current episode will end.

[0177] 3.3.2 DQN Training Process

[0178] The training objective of DQN is to optimize network parameters. ,make To more accurately predict action value, performance is improved during training by using the target value of the Q-table and the experience pool of PPO.

[0179] Input data:

[0180] S2 output rehabilitation feature dataset (original continuous features), Q-table of Q-learning (For Equation 3.2), PPO's experience pool (for expansion) ).

[0181] Detailed steps:

[0182] 1. Initialization parameters: Current network parameters (Random initialization), target network parameters Experience Pool (empty set);

[0183] 2. Iterative training (1000 episodes in total):

[0184] a. Initial sampling state (Continuous feature representation);

[0185] b. For each time step t:

[0186] i. Select Action: Use - Greedy strategy ( (from 0.9 linearly decaying to 0.1), with 1- Probability of choice ,by The probability of randomly selecting an action;

[0187] ii. Perform the action: obtain ,Will deposit ;

[0188] iii. Expand the experience pool: Every 10 time steps, expand the PPO experience pool. Data from [source] to [source] (According to formula 3.3);

[0189] iv. Batch update network: when When the data volume is ≥1000, 32 data points are randomly sampled. Calculate the target value according to formula 3.2 Define the loss function:

[0190] (3.7)

[0191] in:

[0192] The loss is averaged over 32 samples (32 is the batch sampling size).

[0193] Target Q-value (calculated by Equation 3.2, which combines the Q-table values ​​from Q-learning and the predictions from the target network)

[0194] The Q-value predicted by the current Q-network for the state-action pair (s,a).

[0195] Use the Adam optimizer (learning rate) ) minimize ,renew ;

[0196] v. Soft update target network: Execute every 100 time steps. ;

[0197] This is the soft update coefficient (usually a small value, such as 0.01), which controls the magnitude of the current network's impact on the target network;

[0198] vi. Status Update: Let The episode ends when the termination condition is met.

[0199] 3.3.3 PPO Training Process

[0200] The training objective of PPO is to optimize the parameters of the policy network. and value network parameters This makes the strategy more inclined to select high-reward actions, and improves the accuracy of value estimation during training by using the Q-value of DQN and the Q-table of Q-learning.

[0201] Input data:

[0202] S2 output rehabilitation feature dataset, DQN current network parameters (For Equation 3.4), Q-table for Q-learning (Used to initialize the value network).

[0203] Detailed steps:

[0204] 1. Initialization parameters: Policy network parameters (Random initialization), value network parameters (Initial values ​​are referenced from table Q:) (Empty set); Since the total number of actions is 25, divide by 25;

[0205] 2. Iterative training (1000 episodes in total):

[0206] a. Initial sampling state Collect trajectory data: through strategies Select an action, execute it, and obtain the trajectory. (T is the trajectory length);

[0207] b. Calculate the advantage function and cumulative reward:

[0208] Calculate the advantage at each time step using Equation 3.4. ;

[0209] Calculate cumulative discount rewards (As the goal of the value network);

[0210] in:

[0211] The cumulative reward (also known as "reward") at time step t is the learning objective of the value network (the value network needs to predict this value).

[0212] The immediate reward obtained at time step e (such as the quantitative value of the rehabilitation effect after the patient performs the action).

[0213] t represents the time difference, and the further into the future the reward (the larger e is), the greater the decay coefficient. The smaller;

[0214] c. Save trajectory data: deposit (For sharing with DQN);

[0215] d. Optimize the policy network: Fix the current policy parameters as follows The difference between the new and old strategies is limited by the objective function of editing, as shown in the formula:

[0216] (3.8)

[0217] in:

[0218] This reflects the improvement direction of the new strategy compared to the old strategy. Its advantage is that it encourages an increase in the probability of an action when the time is positive, and a decreases it when the time is negative.

[0219] The probability ratio of the new and old strategies is limited to [ ] using the clip function. Within the range (usually) =0.2), to avoid excessive policy update magnitude leading to training instability;

[0220] The overall calculation takes the average of the minimum values ​​of the above values ​​for all time steps t in the trajectory (1 / T∑) as the final optimization objective;

[0221] Use the Adam optimizer (learning rate) To maximize the above objectives, update ;

[0222] e. Optimize the value network: Minimize the mean squared error between value estimation and cumulative reward, as shown in the formula:

[0223] (3.9)

[0224] By minimizing this loss function, the state value output by the value network is maximized. Accumulated rewards for discounts that are as close as possible to reality ;

[0225] Update with Adam optimizer ;

[0226] f. Repeat step ae until all episodes have been trained.

[0227] 3.4 Model Application of Three Algorithms (Integrated Decision Process)

[0228] The core of the application phase is to select the optimal rehabilitation action by integrating the outputs of three algorithms based on the patient's current state s. This ensures that the decision-making process combines the stability of Q-learning, the high-dimensional fitting ability of DQN, and the probabilistic optimization advantages of PPO.

[0229] Input: The current state s of the new patient (20-dimensional features after S2 preprocessing).

[0230] Fusion Decision Formula:

[0231] (3.10)

[0232] in:

[0233] (Q-learning's Q-table provides a basic value reference);

[0234] (DQN's neural network dominates value estimation under high-dimensional features).

[0235] (The logarithmic probability of PPO reflects the relative advantage of the action.)

[0236] The action with the largest weighted sum of the three factors is taken as the final recommendation. ).

[0237] Application process:

[0238] 1. For new patients, data is collected according to S1 and preprocessed by S2 to obtain state s;

[0239] 2. Input the Q-learning module respectively (to obtain) ), DQN module (to obtain) ), PPO module (obtained) );

[0240] 3. Calculate the comprehensive value of each action according to Equation 3.10, and select the action corresponding to the maximum value. ;

[0241] 4. Medical staff guide patients to perform [the procedures / procedures]. The new state s' and reward r are collected according to the method of S1 and fed back to the fusion model to update the parameters of each module (to achieve online learning).

[0242] 3.5 Output

[0243] After the above construction, training, and interactive integration, the output is a rehabilitation assessment and training model integrating Q-learning-DQN-PPO, which includes:

[0244] Q-table after training ;

[0245] The current parameters of the trained DQN network With target network parameters ;

[0246] Parameters of the trained PPO policy network With value network parameters ;

[0247] Integration of decision weights .

[0248] This model will be used for further optimization of S4 (incorporating more patient data) and for the implementation of clinical rehabilitation assessment and training in S5.

[0249] S4. Model Training and Optimization

[0250] A Q-learning-DQN-PPO fusion model built on S3 is used for system training and parameter optimization, combined with the feature dataset output from S2, to ensure the accuracy of the model in rehabilitation assessment and training. The specific process is as follows: Figure 4 As shown.

[0251] First, the dataset is divided. The 20-dimensional feature dataset output by S2 is divided into a training set (used for model parameter updates) and a validation set (used for evaluating generalization ability) in a 7:3 ratio. The training set contains historical rehabilitation data of 700 patients (30 training records per person), and the validation set contains contemporaneous data of 300 patients. The division ensures that the disease type and postoperative stage of the two groups of patients are consistent (e.g., the proportion of brain tumor patients is 30% in both groups).

[0252] Secondly, multi-algorithm collaborative training. During training, the three algorithms are synchronously updated according to the interaction rules defined in S3: every 100 episodes (patient training cycles) completed by the Q-learning module, the updated Q-table is updated. Synchronize to the DQN module for target value calculation (Equation 3.2); the DQN module will update the experience pool every 500 iterations. The latest 1000 data points are shared with the PPO module to expand its trajectory data; the PPO module optimizes the policy network every 20 rounds and updates the current policy. Feedback is sent to the Q-learning module to guide its exploration strategy (Equation 3.5). The total training cycle is set to 10,000 iterations, and the parameters of the three algorithms are updated simultaneously in each iteration: the learning rate of Q-learning. DQN decreases linearly from 0.3 to 0.1. The policy network and value network of PPO are optimized synchronously as the value decreases from 0.9 to 0.1.

[0253] Next, model performance is evaluated. Every 1000 iterations, the model performance is evaluated using a validation set. Key metrics include: rehabilitation assessment accuracy (the consistency between the model-recommended assessment actions and the judgments of medical staff, target ≥85%), training action effectiveness (the proportion of patients whose rehabilitation indicators improve after the model-recommended actions, target ≥80%), and cumulative mean reward (the average reward of patients in the validation set, target ≥1.5 points). If any metric fails to meet the target in three consecutive evaluations, an optimization mechanism is triggered.

[0254] Finally, parameter optimization strategies are implemented. Targeted adjustments are made to address the weaknesses of different algorithms: if the Q-table of Q-learning exhibits overfitting (high accuracy on the training set but low accuracy on the validation set), its weight in the fusion decision is reduced. (From 0.2 to 0.15), and increase the number of state discretization intervals (from 5 to 8); if the loss function of DQN If the fluctuation is too large (variance > 0.5), reduce the learning rate. (Decrease from 0.001 to 0.0005), and increase the experience pool capacity (from 10,000 to 20,000); if the PPO strategy update is unstable (difference between the old and new strategies > 0.3), then reduce the clipping parameters. (Reduced from 0.2 to 0.15), and the trajectory collection length was increased (from 20 steps to 30 steps). After optimization, the model was retrained until all metrics met the requirements, and the final fusion model (including the trained Q-table, network parameters, and weights) was output.

[0255] S5. Rehabilitation Assessment and Training Implementation

[0256] Using the optimized S4 fusion model, personalized rehabilitation assessment and training programs are provided for new patients, and dynamically adjusted through real-time feedback. The specific steps are as follows:

[0257] The first step is the collection and preprocessing of new patient data. For newly admitted neurosurgical patients, basic information, physiological indicators, motor function, cognitive function, and daily living ability data (such as hip joint range of motion and MMSE score one week post-surgery) are collected according to method S1. Then, the data undergoes the preprocessing process S2 (cleaning, standardization, and feature extraction) to generate a 20-dimensional current status. .

[0258] The second step is to integrate the model's recommended actions. This involves combining the states... The fusion model output by inputting S4 is used to call the Q-learning module to compute the results. DQN module calculation PPO module calculation The optimal action is obtained according to Equation 3.10 (fusion decision formula). (e.g., "knee extension training" or "MoCA scale assessment").

[0259] The third step is the execution of the movements and data feedback. Medical staff guide patients to perform the movements. During the process, data is collected in real time: for training exercises, the patient's completion rate (0-100%) and changes in physiological indicators (such as heart rate fluctuations) are recorded; for assessment exercises, new muscle strength levels and scale scores are recorded. A new status is generated upon completion. (Updated 20-dimensional features) and rewards (Calculated according to the S3 reward formula, such as 0.15 points for a 1-level increase in muscle strength).

[0260] Step four: Online model updates. As new data, three modules are updated synchronously: the Q-learning module updates the Q table according to formula 3.6. The DQN module stores the data into the experience pool. A batch update is performed every 50 data points (Equation 3.7); the PPO module incorporates the data into the trajectory. The policy network is optimized after every 10 actions (Equation 3.8).

[0261] Step 5: Periodic Assessment and Treatment Adjustment. A comprehensive assessment of the patient is conducted weekly: the model adjusts the assessment based on the current status. Recommended assessment exercises (such as MMSE + muscle strength assessment) should be used to update the patient's status based on the assessment results. The overall value of each movement is recalculated, and the training plan for the following week is adjusted (e.g., switching from "ankle training" to "balance training"). This process continues until the patient reaches clinical rehabilitation standards (e.g., BI score ≥ 90 points, muscle strength ≥ grade 4).

[0262] S6. Model Performance Evaluation and Update

[0263] To ensure that the fusion model adapts to clinical needs in the long term, its effectiveness needs to be evaluated regularly and iteratively updated. The specific steps are as follows:

[0264] First, the evaluation cycle and indicators. A comprehensive evaluation of the model is conducted every 3 months, with data sourced from the rehabilitation records of 100 patients in S5. Evaluation indicators include: average rehabilitation time (compared with historical data, target reduction ≥10%), complication rate (joint injuries due to inappropriate training programs, etc., target ≤5%), satisfaction of medical staff (questionnaire survey, target ≥4.2 / 5 points), and consistency between model decisions and expert plans (Kappa coefficient, target ≥0.7).

[0265] Secondly, problem diagnosis and root cause analysis. If the assessment indicators are not up to standard, the problem can be located through source analysis: if the rehabilitation effect of new patients (such as pediatric neurosurgical patients) is poor, it may be because the training dataset lacks the characteristics of this group (insufficient data on children's muscle strength development); if the effectiveness of cognitive training decreases, it may be because the DQN module does not fit language features well enough (low weight of word comprehension features); if motor training leads to complications, it may be because the PPO strategy overemphasizes rewards and ignores movement safety (joint load features are not included).

[0266] Next, data supplementation and model iteration were performed. Supplemental data was added to address the underlying causes: data from 100 additional patients in special groups were collected (processed according to the S1-S2 workflow) and added to the training set; "Joint Load Index" (the 9th feature of motor function) and "Language Fluency" (the 8th feature of cognitive function) were added to the feature set, expanding it to 22 features. When retraining the model, the interaction parameters were adjusted: higher weights were assigned to the pediatric patient data (during Q-learning initialization). Increase the DQN's focus on language features (increase the number of hidden layer neurons from 128 to 160); add a safety penalty term to the PPO reward function (deduct 0.2 points for joint overload).

[0267] Finally, model version management and deployment. The updated model is marked as the new version (e.g., V2.0) and run in parallel with the old version for one month. The effect is verified through a double-blind test (the new and old models provide treatment plans for 50 patients respectively, and the recovery indicators are compared). After confirming that the new version is better, the old model in clinical application is replaced. At the same time, historical versions and evaluation reports are archived to provide a reference for subsequent iterations.

[0268] In some embodiments, a surface fitting algorithm is added to handle the nonlinear relationship of state features, forming a four-algorithm collaborative system with Q-learning, DQN, and PPO.

[0269] Surface fitting algorithms are used to handle nonlinear relationships between state features (such as the quadratic relationship between "alpha wave intensity" and "balance ability score"), outputting smooth feature correction values ​​and providing more robust state representations for reinforcement learning algorithms. Its mathematical form is:

[0270] (1)

[0271] in (Order) The fitting coefficients to be learned. Let i be the i-th, j-th component in the state feature.

[0272] Interaction with state preprocessing: Correcting the original state features, which are then used as input to the reinforcement learning algorithm, as shown in the formula:

[0273] (2)

[0274] in This is the corrected state; This is a weighting coefficient, typically ranging from [0,1], used to balance the influence of the original state and the result of the function processing.

[0275] Interaction with DQN: Nonlinear fitting of the auxiliary Q-value function, with the DQN input layer being a modified state. Furthermore, the prediction trend of the fusion surface fitting during target network updates is as follows:

[0276] (3)

[0277] in The function F represents the current network parameters. The gradient.

[0278] Interaction with PPO: Optimize the state input of the policy network, the policy function is Furthermore, the advantage function incorporates gradient information from surface fitting:

[0279] (4)

[0280] in is the advantage function of PPO when performing action a in state ŝ. The larger the value, the better the action is relative to the current policy. The Q-value (absolute value estimate) of the state-action pair (ŝ,a) output by the DQN network. For the PPO value network, the value of state ŝ is estimated. Let F be the gradient of the state characteristic function F with respect to the state ŝ. Let be the gradient of the PPO value function with respect to state ŝ.

[0281] Model training for surface fitting

[0282] The feature dataset output by inputting S2 is used for training by minimizing the error between the fitted value and the actual state transition. The loss function is:

[0283] (5)

[0284] in For state The actual corrected state after the action (the ideal state as indicated by clinical experts), where N is the sample size.

[0285] Collaborative training steps

[0286] 1. Initialization (Randomly generate polynomial coefficients);

[0287] 2. After every 100 reinforcement learning iterations (Q-learning / DQN / PPO training in S3.4), the surface fitting parameters are updated synchronously:

[0288] a. Sample 1000 state-action-new state samples from the S2 dataset. ;

[0289] b. Calculate the correction state (Equation 2) and the ideal correction state ;

[0290] c. Minimize using gradient descent ,renew (Learning rate 0.002);

[0291] 3. Update the Synchronize to all reinforcement learning modules and update correction status. The calculation.

[0292] Application of the four-algorithm fusion model

[0293] The feature correction effect of surface fitting is incorporated into the decision formula, as shown in the formula:

[0294] (6)

[0295] in (The gradient term of the surface fitting reflects the effect of the action on the smoothness of the state), and .

[0296] Expanded to a Q-learning-DQN-PPO-surface fitting fusion model, outputting: trained surface fitting parameters. and order ; Corrected state calculation function (Equation 2).

[0297] Data preprocessing and feature engineering

[0298] Add a "surface fitting preprocessing" step after "feature extraction":

[0299] For nonlinear correlation features in the 20-dimensional feature set (such as EEG alpha wave intensity and balance ability score, step length and lower limb muscle strength grade), surface fitting is used for smoothing to generate initial fitting parameters. ;

[0300] The smoothed features are compared with the original features according to Equation 2 ( =0.3) fusion, as the feature dataset of the final fusion model (still 20-dimensional, but with a smoother feature distribution).

[0301] Model training and optimization

[0302] Evaluation metrics expansion: Add "state fit error" ( The mean square error between the actual state and the target value is ≤0.05.

[0303] Optimization strategy: If the state fitting error is too large, improve... (Upgrade from 2nd to 3rd order) and increase (From 0.3 to 0.4); if the reinforcement learning algorithm converges slowly, reduce... (to 0.2) to reduce the interference of the fitting on the original features.

[0304] Rehabilitation assessment and training implementation

[0305] When pre-treating new patients, first use the training... Calculate the corrected state ;

[0306] After each action is executed, use the current Update new status This ensures that the state input to the reinforcement learning module is always smoothed out.

[0307] Real-time monitoring of surface fitting error ( (Difference from the actual measured value), if the error is >0.1, temporarily reduce... Up to 0.1 (original measurement data should be retained first).

[0308] Model performance evaluation and updating (iterative optimization of surface fitting)

[0309] The evaluation metric includes "feature smoothness" (corrected state). The variance of the target state is the ratio of the variance of the target state to the variance of the original state s (target value: 0.6~0.8).

[0310] If the smoothness is too low (<0.6), increase or If the smoothness is too high (>0.8), then reduce it;

[0311] When new data is added, the training set for surface fitting is updated synchronously, and retraining is performed every 6 months. This ensures that the fitted relationship adapts to the new patient characteristic distribution.

[0312] The embodiments described above are for illustrative purposes only and are not intended to limit the invention. Therefore, any changes in numerical values ​​or substitutions of equivalent elements should still fall within the scope of this invention.

[0313] The above detailed description will enable those skilled in the art to understand that the present invention can indeed achieve the aforementioned objectives and has complied with the provisions of the Patent Law.

[0314] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention. The above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

[0315] It should be noted that the above description of the process is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to the process under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0316] The basic concepts have been described above. Obviously, for those skilled in the art who have read this application, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore, such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this application.

[0317] Furthermore, this application uses specific terms to describe its embodiments. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different positions in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.

[0318] Furthermore, those skilled in the art will understand that aspects of this application can be described and illustrated through several patentable types or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Therefore, aspects of this application can be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. All of the above hardware or software can be referred to as a “unit,” “module,” or “system.” Furthermore, aspects of this application can take the form of a computer program product embodied in one or more computer-readable media, wherein computer-readable program code is contained therein.

[0319] The computer program code required for the operation of each part of this application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C, VB.NET, and Python; general programming languages ​​such as C; Visual Basic, Fortran2103, Perl, COBOL2102, PHP, and ABAP; dynamic programming languages ​​such as Python, Ruby, and Groovy; or other programming languages. This program code can run entirely on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as Software as a Service (SaaS).

[0320] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this application are not intended to limit the order of the processes and methods of this application. Although some currently considered useful embodiments of the invention have been discussed in the foregoing disclosure by way of various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the substance and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a purely software solution, such as an installation on an existing server or mobile device.

[0321] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this approach of the present application should not be construed as reflecting an intention that the claimed subject matter requires more features than expressly recited in each claim. Rather, the subject of the invention should possess fewer features than in any single embodiment described above.

Claims

1. A method for rehabilitation assessment and training of neurosurgical patients based on reinforcement learning, characterized in that, include: S1. Collect patients' basic information, physiological indicators, motor function data, cognitive function data, and daily activity ability data to form a raw database for patient rehabilitation; S2. Clean, standardize, and extract features from the original patient rehabilitation database to obtain a patient rehabilitation feature dataset; S3. Construct a rehabilitation assessment and training model integrating Q-learning, DQN, and PPO. This model includes a Q-learning module, a deep Q-network module, and a proximal policy optimization module. Define the agent, environment, state, action, and reward. The agent is a decision-making system integrating the three modules. The environment is the patient's rehabilitation state and training scenario. The state consists of the patient's rehabilitation feature dataset. Actions include motor function training actions, cognitive function training tasks, and assessment actions. The reward is quantified based on the changes in rehabilitation indicators after the patient performs the action. The Q-learning module stores the value of state-action pairs through a Q-table. The deep Q-network module approximates the Q-value function through a neural network. The proximal policy optimization module outputs the action probability distribution through a policy network. S4. Use the patient rehabilitation feature dataset to train and optimize the rehabilitation assessment and training model to obtain the optimized fusion model; S5. Input the current state obtained by processing the rehabilitation data of new patients through S2 into the optimized fusion model, output the optimal rehabilitation action, collect the new state and reward after the patient performs the optimal rehabilitation action and feed it back to the fusion model to update the parameters, evaluate and adjust the training program regularly until the patient reaches the rehabilitation standard. In step S3, the interaction process between the Q-learning module and the Deep Q-Network module (DQN) is as follows: the initial value of the Q-table of the Q-learning module is fused with the initial prediction value of the Deep Q-Network module, and the target Q-value of the Deep Q-Network module is calculated and fused with the Q-table value of the Q-learning module; specifically, the initial value of the Q-table is fused with the prediction result of DQN, using the following formula: ; The initial values ​​for table Q; This is an initial estimate based on historical rehabilitation data; Initial network parameters for DQN Predicted values ​​for state-action pairs; for The weights; the objective value of DQN is calculated by incorporating the Q-table values ​​of Q-learning, using the following formula: ; r is the target Q value of DQN; r is the reward. Discount factor; The Q-table value for Q-learning in the new state s'; is the target network's prediction of the new state s'; for The weights; In step S3, the interaction process between the Deep Q-Network module and the Proximal Policy Optimization (PPO) module is as follows: the experience pools of the Deep Q-Network module and the PPO module share interactive data, and the advantage function calculation of the PPO module fuses the Q-value of the Deep Q-Network module; specifically, the experience pool of DQN is shared with PPO to expand the amount of training data; the experience pool of DQN... Defined as: ; The PPO's experience pool stores interaction data between the PPO and the environment, improving DQN's generalization ability to action effects through data sharing. DQN's experience pool not only contains its own interaction data but also incorporates PPO interaction data, expanding the training sample size through data sharing and enhancing DQN's generalization ability to different action effects, thus achieving synergy between the two algorithms at the data level. The PPO's advantage function incorporates DQN's Q-value, with the formula: ; The advantage function represents the advantage function of the PPO algorithm when performing action a in state s. It is used to measure the superiority of the action relative to the current policy. The larger the value, the better action a is relative to the current policy. The Q-value of the current DQN network provides a reference for the absolute value of actions. These are the network parameters for DQN; In step S3, the interaction between the proximal policy optimization module and the Q-learning module is as follows: When the Q-learning module selects actions using an ε-greedy policy, the probability distribution of random actions is referenced to the action probabilities output by the policy function of the proximal policy optimization module; specifically, the exploration strategy of Q-learning... - In a greedy approach, the probability of choosing a random action is referenced to the policy distribution of the Probabilistic Policy Object (PPO). The PPO is used to model the probability of the action's effect, i.e., when Q-learning... When randomly selecting actions, the probability of action a being selected is... The update magnitude is adjusted based on the probability assessment of the current action using the PPO, and the formula is as follows: ; It is a state-action pair in the Q-table of Q-learning. The updated value is shown on the right; the update rule is shown on the right, which follows the temporal difference update idea of ​​Q-learning, that is, adjusting the current Q value based on the current reward and the value of the future state. For PPO action Probability assessment; It is the learning rate of Q-learning, which controls the overall update step size; It is the temporal difference error, which reflects the gap between the current Q value and the actual reward plus the future optimal value.

2. The method for rehabilitation assessment and training of neurosurgical patients based on reinforcement learning according to claim 1, characterized in that, It also includes a surface fitting algorithm for handling nonlinear relationships of state characteristics, forming a four-algorithm synergy system with Q-learning, DQN, and PPO; the mathematical form of the surface fitting algorithm is: ;in For order, The fitting coefficients to be learned. Let i be the i-th, j-th component in the state feature; Interaction with state preprocessing: Correcting the original state features, which are then used as input to the reinforcement learning algorithm, as shown in the formula: ;in This is the corrected state; For weighting coefficients; adjust the state The original state input is replaced by the Q-learning module, the deep Q-network module, and the proximal policy optimization module; Interaction with DQN: Nonlinear fitting of the auxiliary Q-value function, with the DQN input layer being a modified state. Furthermore, the prediction trend of the fusion surface fitting during target network updates is as follows: ;in The function F represents the current network parameters. The gradient; Interaction with PPO: Optimize the state input of the policy network, the policy function is Furthermore, the advantage function incorporates gradient information from surface fitting: ;in For PPO in state The advantage function for executing action 'a' is defined by the value of the action; a larger value indicates that the action is better than the current strategy. State-action pairs output by the DQN network ( The Q value of a); For PPO value network state Value estimation; Let F be the state characteristic function for the state. The gradient; For the PPO value function, the state The gradient.

3. The method for rehabilitation assessment and training of neurosurgical patients based on reinforcement learning according to claim 1, characterized in that, In step S3, the action decision of the rehabilitation assessment and training model is achieved by weighted fusion of the outputs of the three modules and the gradient term of the surface fitting. The weights obtained include the Q-table weights of the Q-learning module, the Q-value weights of the deep Q-network module, the logarithmic weights of the action probabilities of the proximal policy optimization module, and the decision weights of the surface fitting algorithm. And the sum of the four is 1.

4. The method for rehabilitation assessment and training of neurosurgical patients based on reinforcement learning according to claim 1, characterized in that, In step S5, a real-time feedback closed loop is formed: after the patient performs the optimal rehabilitation action, the new status and reward collected are synchronously updated in the Q table of the Q-learning module, the experience pool of the deep Q network module, and the trajectory data of the proximal strategy optimization module, so as to realize the online learning of the fusion model.

5. The method for rehabilitation assessment and training of neurosurgical patients based on reinforcement learning according to claim 4, characterized in that, The closing conditions for real-time feedback loop are: the patient's daily activity ability score reaches the preset threshold, the muscle strength level recovers to the preset level, or the rehabilitation indicators of two consecutive assessments are consistent.

6. The method for rehabilitation assessment and training of neurosurgical patients based on reinforcement learning according to claim 1, characterized in that, In step S4, the model optimization strategies include: if the Q-learning module overfits, increasing the number of state discretization intervals and reducing its decision weights; if the variance of the loss function of the deep Q-network module is greater than 0.5, reducing the learning rate and increasing the experience pool capacity; if the difference between the new and old strategies in the proximal policy optimization module is greater than 0.3, reducing the clipping parameters and increasing the trajectory collection length; the difference between the new and old strategies refers to the new strategy in the proximal policy optimization module. Compared to the old strategy The difference in probability distribution between them.

7. The method for rehabilitation assessment and training of neurosurgical patients based on reinforcement learning according to claim 1, characterized in that, It also includes step S6: regularly evaluating the effectiveness of the fusion model, with evaluation indicators including average patient recovery time, complication rate, healthcare staff satisfaction, and consistency between model decisions and expert plans.

8. The method for rehabilitation assessment and training of neurosurgical patients based on reinforcement learning according to claim 7, characterized in that, In step S6, if the evaluation indicators do not meet the standards, new patient data and new features are added, the model is retrained and updated. The new patient data is obtained through the data acquisition system in S1, and the new features include joint load index and language fluency. The supplemented data is processed in S2 and then added to the training set.

Images