Cognitive-motor rehabilitation warning and intervention method based on multi-modal time series modeling

Abstract

The application discloses a cognitive-motor rehabilitation early warning and intervention method based on multi-modal timing modeling, comprising the following steps: fusing multi-modal risk features to obtain multi-modal risk features; predicting health state prediction and risk trend estimation by simultaneously capturing dynamic multi-modal risk features of short-term fluctuations and long-term evolution; adaptively optimizing risk determination and intervention sensitivity; observing the comprehensive state of the patient through a Markov decision process, and selecting the corresponding intervention action accordingly, forming a reward signal based on the patient's response to the intervention, which is used to guide the next step of strategy optimization. Through multi-modal risk feature perception, cross-time scale modeling, individual baseline alignment and adaptive threshold adjustment, the application realizes the leap from static evaluation to dynamic prediction, from artificial intervention to intelligent closed loop, can capture the subtle changes of cognitive and motor function in the time dimension, and outputs individualized risk trend curve and intervention priority.

Description

Technical Field

[0001] This invention relates to the field of rehabilitation training technology, specifically to a cognitive-motor rehabilitation early warning and intervention method based on multimodal temporal modeling. Background Technology

[0002] Cognitive and motor dysfunction are among the most common functional declines in the elderly. Studies show a bidirectional relationship between cognition and motor function: cognitive decline leads to reduced motor coordination and executive function, while the persistence of motor impairment, in turn, exacerbates cognitive decline. This interaction typically exhibits temporal and phased fluctuations. Traditional rehabilitation systems are mostly based on static scales (such as the MoCA, MMSE, and Berg Balance Scale) for one-time assessments, which cannot capture changes in functional status over time, nor can they provide objective evidence for "when to retest" or "when to intervene."

[0003] In actual assessments, patients' reaction time, tone of voice, number of hesitations, drawing trajectories, and question-and-answer semantics, among other multimodal risk characteristics, often contain rich physiological and psychological information. These dynamic indicators reflect the potential state of the cognitive-motor system. However, traditional systems lack a unified multimodal modeling framework, and voice, text, action, and scale data cannot be aligned in the time dimension, resulting in models that can only analyze at a single-modal level and neglecting cross-modal complementary information.

[0004] With the rapid development of artificial intelligence technology, especially the hybrid learning approach combining data-driven and knowledge-guided methods, new technological pathways have emerged in the field of cognitive-motor rehabilitation. Deep neural networks can uncover hidden feature patterns in high-dimensional nonlinear spaces, modeling temporal signals of speech, images, and behavior through convolutional neural networks (CNNs), recurrent neural networks (RNNs), or Transformer structures to identify weak signal features related to functional decline. Simultaneously, the introduction of knowledge graphs enables the system to leverage medical knowledge and rehabilitation experience for reasoning constraints and result interpretation, in addition to data-driven learning, achieving a "knowledge-guided deep learning" model. By constructing questionnaire items, symptom descriptions, training plans, and medical concepts as knowledge graph nodes and running graph neural networks (GNNs) on the graph structure, the system can simultaneously capture semantic and physiological connections, achieving a structured understanding from symptoms to risk factors.

[0005] However, the application of existing technologies still has three limitations: First, there is a lack of a unified temporal alignment and correlation mechanism between multimodal data, and voice, scale and action signals are often processed separately, failing to form a complete time series; Second, there is a lack of an adaptive modeling framework oriented towards individual differences, and differences in functional baselines, recovery rates and living environments among different patients lead to a significant decrease in the performance of the model in cross-individual predictions; Third, intervention recommendations lack intelligent optimization and feedback correction mechanisms, and most systems still rely on fixed rules or manual judgment, failing to dynamically adjust prescription plans based on follow-up results.

[0006] Current rehabilitation systems generally suffer from problems such as static assessment, isolated information, and experience-based decision-making. Traditional scales or question-and-answer assessment methods can only reflect the patient's state at a certain moment, making it difficult to reveal the dynamic evolution of cognitive and motor functions over time; multimodal data (such as voice, movement, text, scale results, and follow-up information) lack a unified representation model, resulting in one-sided and unrelated assessment results; intervention program formulation relies on expert experience and manual judgment, lacking a dynamic optimization mechanism based on data and models, and failing to achieve individualized and real-time rehabilitation management (Second Medical Center of the General Hospital of the Chinese People's Liberation Army. Rehabilitation training system for MCI patients in different scenarios based on deep learning: 202410760633.6 [P]. 2025-01-10.).

[0007] Although the aforementioned existing technologies have introduced multimodal data acquisition and basic deep learning (such as basic CNN feature extraction or RNN sequence processing) or modular training methods into the field of cognitive and motor rehabilitation, they still have the following significant shortcomings when facing real-world, long-term clinical rehabilitation applications:

[0008] Lack of dynamic evolution modeling and early warning mechanisms across multiple time scales: While existing technologies utilize deep learning to process multimodal data, they often focus on assessing the state of cross-sectional data at a single time point, or simply use basic RNN networks to process short-term signals, failing to delve into the potential coupling relationships and evolutionary trends of multimodal data over long time. Faced with the complex characteristics of long-term evolution and short-term fluctuations in the rehabilitation process, existing systems cannot achieve high-precision temporal risk prediction and trend estimation, resulting in an inability to provide effective early warnings before patients enter a high-risk state.

[0009] The fusion of multimodal time series data has poor robustness: When processing multi-source heterogeneous data (speech, text, motion, etc.), existing technologies often use simple data splicing or static standardized calculations, ignoring the problem of missing time dimensions in real-world scenarios for each modality of data, and failing to consider the dynamic changes in the contribution of different modalities to the current health status, making it difficult to form unified and robust high-dimensional time series features. Summary of the Invention

[0010] To address the aforementioned issues, this invention proposes a cognitive-motor rehabilitation early warning and intervention method and system based on multimodal temporal modeling. This system comprehensively applies data-driven and knowledge-guided artificial intelligence methods, integrating core technologies such as deep learning, graph neural networks, and reinforcement learning. Through multimodal risk feature perception, cross-timescale modeling, individual baseline alignment, and adaptive threshold adjustment, it achieves a leap from static assessment to dynamic prediction, and from manual intervention to intelligent closed-loop management. The system can capture subtle changes in cognitive and motor function over time, outputting individualized risk trend curves and intervention priorities. It can be applied to early warning and precise intervention for the elderly, those recovering from stroke, and those at risk of cognitive impairment, providing rehabilitation physicians with scientific and interpretable decision-making support and patients with continuous, precise, and intelligent rehabilitation management support.

[0011] The present invention is achieved by at least one of the following technical solutions.

[0012] A cognitive-motor rehabilitation early warning and intervention method based on multimodal temporal modeling includes the following steps: 1) Fuse multimodal risk features to obtain multimodal risk features; 2) By using a hybrid model to capture the dynamic multimodal risk characteristics of short-term fluctuations and long-term evolution, the overall risk level can be predicted and the upward trend of risk can be predicted; 3) Adaptively adjust the sensitivity of individual baseline parameters according to the overall risk level to achieve adaptive optimization of sensitivity at the individual level; 4) Based on the state observed by the Markov decision model, select the appropriate intervention action, and generate a reward signal based on the patient's response to the intervention to guide the next step of strategy optimization; 5) Collect intervention feedback signals in real time, compare the real-time collected intervention feedback signals with the expected state output by the hybrid model at the current time to obtain the error. If the error is greater than the threshold, update the state transition matrix of the hybrid model.

[0013] Furthermore, the fusion in step 1) includes: reconstructing the input multimodal data using a VAE structure to obtain the completed multimodal risk features, and introducing an attention weighting mechanism to avoid equal weighting of all modalities. The weights are normalized by softmax to reflect the contribution of each modality to the overall state at the current time.

[0014] Further, step 2) includes the following steps: First, a multi-head self-attention structure is used to obtain short-term contextual features of adjacent time slices for multimodal risk features. ; the short-term context features Input into the Bayesian state transition layer for long-term dynamic modeling; Then, define the cognitive-motor risk function:

[0015] in, Indicates time The overall risk level For the Sigmoid function, and These are the linear weights and the bias term, respectively. Let be the short-term context feature vector after local context modeling; Finally, the patient's current risk is estimated by considering the overall risk level and the upward trend of risk.

[0016] Furthermore, the patient's current risk is estimated through risk level assessment and trend changes: Set a risk level threshold to determine whether a patient's current risk exceeds the safety limit. ,when This indicates that the patient's current overall risk is higher than the normal range; Set a risk change rate threshold to measure the rate of risk increase. ,when This indicates that the patient's risk is in a phase of rapid deterioration, even if the current risk level has not yet reached a critical point. ; When both conditions are met and When a patient is identified as being in a high-risk state, the early warning module is triggered, sending a notification signal to the doctor or caregiver.

[0017] Furthermore, step 3) includes the following steps: 31) Individual baseline calculation and risk standardization: Extract stable-phase risk characteristics from long-term patient monitoring data to form individual baseline parameters; determine patient status using the sliding window detection method; When the risk variance is less than the threshold over a continuous period of time, the patient is considered to be in a stable state. The risk value is then normalized to obtain a standardized risk value, which reflects the relationship between the risk at the corresponding moment and the patient's long-term average risk status. 32) Threshold Adaptive Smoothing Update: The multi-level thresholds are dynamically adjusted through a smoothing model. An individual sensitivity coefficient is introduced. By combining the smoothing factor and the individual sensitivity coefficient, automatic sensitivity adjustment is achieved to ensure that different patients can obtain a matching early warning response speed at their respective recovery stages.

[0018] Furthermore, the establishment of a Markov decision model includes the following steps: First, the defined state space Used to fully describe the recovery status:

[0019] in, Standardized risk values ​​for individuals reflect the degree of deviation of their current health status from their personal homeostasis; The risk change rate represents the rate at which cognitive-motor function increases or decreases. These are features extracted from a hybrid model, which characterize the potential coupling relationship between cognition and motor state. It is the average risk over a long-term stable period for an individual, used as a reference point for risk assessment; This is the standard deviation of individual risk volatility, used to measure the risk stability of that individual; Define selectable action space :

[0020] in Used to improve language, memory, and attention abilities. Regarding balance and physical coordination, To achieve joint stimulation of cognitive and motor pathways, During periods of stable condition, delayed or low-intensity interventions should be implemented to prevent excessive fatigue. Construct a reward function that integrates risk improvement and cost constraints:

[0021] in, For instant rewards; This represents the predicted future risk value after intervention. To predict the time step; This represents the central value of the ideal health interval; Let be the action cost function. This is used to quantify the time, energy, or resource consumption resulting from the intensity of intervention. This is the cost weighting coefficient, used to balance the risk improvement effect with the intervention cost.

[0022] Furthermore, the policy optimization in step 4) adopts an Actor-Critic dual network structure, where the Actor network outputs the action policy probability distribution and the Critic network is responsible for evaluating the state value.

[0023] The system for implementing the cognitive-motor rehabilitation early warning and intervention method based on multimodal temporal modeling includes: The multimodal risk feature fusion module is used to fuse multi-source data to obtain multimodal risk features; The State Prediction and Estimation module is used to predict health status and estimate risk trends using a hybrid model. The individual baseline alignment and adaptive threshold module is used to adaptively optimize the sensitivity of individual baseline parameters by introducing long-term individual characteristic statistics, standardized risk representation, and exponential smoothing dynamic threshold updates. The action intervention module is used to observe the patient's state through a Markov decision model, select intervention actions, and optimize the strategy based on the patient's feedback. The correction module is used to update the state transition matrix of the hybrid model through a backflow correction mechanism.

[0024] A computer device according to the present invention includes a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, which, when executed by the processor, causes the processor to implement the method described herein.

[0025] The present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor implements the method described herein.

[0026] Compared with existing technologies, the beneficial effects of the present invention are as follows: 1. At the assessment level, this invention achieves temporal dynamism and trend visualization of the rehabilitation process. Traditional rehabilitation assessments typically rely on static scale results, failing to reflect the continuity and phased fluctuations of functional changes. This invention, through multi-timescale Bayesian state transition modeling and Transformer temporal coding, captures the evolutionary trajectories of cognitive and motor indicators in real time, constructs individualized risk curves and rate-of-change functions, and uses a dual-threshold mechanism to achieve early risk identification and trend warning. This continuous assessment method based on time series enables the system to identify potential abnormalities before functional decline becomes significant, significantly improving the sensitivity and lead time of rehabilitation monitoring.

[0027] 2. At the modeling level, this invention achieves the fusion and robust representation of multimodal data. By fusing multi-source data such as speech, text, graphics, behavioral responses, and scale scores, the system employs cross-modal embedding mapping at the feature layer and a weighted attention mechanism at the decision layer to achieve semantic alignment and behavioral signal coupling, thereby forming a unified high-dimensional temporal feature vector. This method effectively solves the problems of fragmented information across different modalities and difficulty in quantifying weights in traditional models, enabling cognitive and motion signals to be correlated and analyzed in a unified space, thus enhancing the robustness and generalization of the model.

[0028] 3. At the intervention level, this invention introduces a reinforcement learning mechanism to achieve intelligent decision optimization. Through an Actor-Critic dual-network structure, the system can learn the optimal intervention strategy in a multimodal state space, maximizing the long-term expected return (i.e., rehabilitation benefits). This strategy is adaptive and evolutionary, automatically adjusting the intervention type and intensity according to changes in the patient's state, achieving closed-loop optimization through bidirectional learning between the agent and the environment. The system's strategy convergence is theoretically guaranteed, satisfying the Robbins-Monro stochastic approximation condition, thus ensuring the stability and reliability of intervention decisions in long-term operation. Compared to previous rehabilitation processes based on fixed templates, this system can form differentiated intervention paths based on individual characteristics, avoiding a one-size-fits-all training approach.

[0029] 4. This invention achieves precise personalization of rehabilitation strategies through individual baseline alignment and threshold adaptation mechanisms. The system calculates baseline risk and standard deviation of fluctuation based on individual stable-period data, and uses normalized risk and smoothed threshold update formulas to achieve dynamic adjustment and self-learning optimization of thresholds. Combined with an individual sensitivity coefficient, the system can adaptively adjust warning sensitivity and intervention response speed according to the patient's risk fluctuation characteristics and rehabilitation speed, thereby truly realizing a personalized rehabilitation plan "tailored to the individual."

[0030] 5. At the closed-loop control level, this invention proposes a parameter feedback correction and continuous learning mechanism to ensure the system maintains accuracy and stability during long-term operation. The system monitors the deviation between the patient's post-intervention feedback signal and the model's predicted values ​​in real time. When the error exceeds a threshold, it automatically triggers local retraining of the state transition matrix and policy parameters. This enables the system to continuously self-correct and update parameters in the face of different stages, individuals, or external disturbances, forming a dynamically evolving knowledge structure and significantly enhancing the model's long-term usability and anti-drift capability.

[0031] 6. This invention offers significant advantages in interpretability and safety in clinical applications. Every risk assessment, threshold change, and intervention decision made by the system can be traced back to specific input features and strategy output paths, transforming the "black box model" into an "interpretable decision-making model." When outputting intervention recommendations, the system not only provides training type and intensity but also generates textual explanations based on knowledge graphs, helping doctors and patients understand the decision-making basis and improving trust and compliance. Simultaneously, the dual-threshold risk assessment mechanism and delayed intervention strategy design ensure that the system does not over-intervene during periods of abnormal fluctuations, thereby improving the safety of the rehabilitation process.

[0032] In summary, this invention achieves intelligent, personalized, and closed-loop optimization of rehabilitation assessment and intervention through the organic combination of multimodal data fusion, temporal risk modeling, reinforcement learning decision-making, and adaptive threshold adjustment. Compared with traditional systems, this invention significantly improves detection sensitivity, response timeliness, intervention accuracy, and long-term stability, providing a comprehensive, full-cycle intelligent management solution for the elderly and neurorehabilitation patients. This technological innovation not only provides a new pathway for the early identification and rehabilitation of cognitive-motor dysfunction but also lays the technological foundation for the future intelligent development of clinical decision support systems. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a cognitive-motor rehabilitation early warning and intervention method based on multimodal temporal modeling, as an example.

[0034] Figure 2 This is a flowchart illustrating the structure of the hybrid model in the embodiment.

[0035] Figure 3 This is a flowchart of the Markov decision process in the example. Detailed Implementation

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] A cognitive-motor rehabilitation early warning and intervention system based on multimodal temporal modeling includes: The multimodal risk feature fusion module is used to fuse multi-source data to obtain multimodal risk features; The State Prediction and Estimation module is used to predict health status and estimate risk trends using a hybrid model. The individual baseline alignment and adaptive threshold module is used to adaptively optimize the sensitivity of individual baseline parameters by introducing long-term individual characteristic statistics, standardized risk representation, and exponential smoothing dynamic threshold updates. The action intervention module is used to observe the patient's state through a Markov decision model, select intervention actions, and optimize the strategy based on the patient's feedback. The correction module is used to update the state transition matrix of the hybrid model through a backflow correction mechanism.

[0038] like Figures 1-3As shown, this embodiment focuses on realizing the intelligence and dynamism of the rehabilitation assessment and intervention process. Using multimodal temporal modeling and adaptive optimization as the core carrier, it proposes a cognitive-motor rehabilitation early warning and intervention method based on multimodal temporal modeling, including the following steps: 1. Multimodal risk feature fusion and robust modeling, including: Set time step The corresponding multimodal input is:

[0039] in, For the multimodal input set corresponding to time t, It represents speech modalities (such as speech rate, pitch, and tone fluctuations). This indicates the text modality (such as the semantic and syntactic structure of the response). For behavioral modalities (such as reaction time, number of hesitations, trajectory deviation). This represents the scale modality (score vectors for each item). This indicates family follow-up modalities (such as sleep, mood, and exercise frequency).

[0040] Since some modes may be missing, reconstruction is first performed using a VAE structure. The VAE structure includes two mapping processes: an encoder and a decoder.

[0041]

[0042] in, For the actual observed modal subset, For latent variables, and These are the mean and covariance vectors of the encoder output, respectively. Indicated by For the mean, The covariance is a Gaussian distribution. Indicates that in the given observation data Under the condition of latent variables The probability distribution, This is a decoder function used to generate estimates of the missing modes. Through the above reconstruction, the completed multimodal risk characteristics can be obtained. .

[0043] To avoid equal weighting for all modalities, an attention-weighted mechanism is further introduced:

[0044]

[0045] in, For the first The weights of each modality For learnable parameter vectors, For the first Risk feature vectors of each modality This represents a loop variable from 1 to the total number of modes. Indicates the first Risk feature vectors of each modality Indicates the first The learnable parameter vector corresponding to each mode The normalization factor represents the sum of values ​​obtained by performing a dot product operation on the risk feature vectors of all modes and their corresponding learnable parameter vectors, followed by an exponential transformation. The weights, after softmax normalization, reflect the contribution of each mode to the overall state at the current time step. Based on the calculated weights... Information of each modality Kneading yields fused feature vectors First, missing data is imputed using VAE (Visual Augmentation Environment), and then an attention mechanism is used to adaptively weight the imputed data to form a fusion model. This fusion model can maintain stable feature representation capabilities under conditions of multimodal heterogeneity and partial missing data.

[0046] 2. Obtain the fused feature vector Subsequently, a time series model was used to predict the risk level and upward trend through a Transformer-Bayesian hybrid model.

[0047] This invention designs a Transformer-Bayesian hybrid model, which consists of a multi-head self-attention mechanism for capturing short-term fluctuations (connected to a Bayesian state transition layer for capturing long-term evolution). This hybrid model achieves high-precision health status prediction and risk trend estimation. Specifically, it includes the following steps: First, for the input sequence Short-term dependency modeling is performed. A multi-head self-attention structure is used to obtain contextual features from adjacent time slices:

[0048] in, This represents a multi-head self-attention function, used to capture feature dependencies in different subspaces in parallel; Indicated by Centered on time, with a length of The context of the feature sequence within the time window; The time window length, To represent the short-term context feature vector generated after local context modeling, it aggregates key information from the current time and its neighborhood.

[0049] Attention weights are calculated as follows:

[0050]

[0051] in Indicates the first Information at the time step for the _ _ Attention weights (i.e. importance) at each time step; and These are the query vector and the key vector, respectively. It has a vector dimension. This mechanism can automatically learn the importance of key information in the temporal neighborhood. Indicates at time step The local context feature representation vector output by a single attention head after weighted summation; The local time window representing the effect of the attention mechanism is the set of adjacent time slices centered on the current time, representing the actual features at that moment. This represents the learnable linear mapping weight matrix used to project input features onto the "value" space; Indicates time step The input feature vector (in this case, the multimodal fusion feature vector) ).

[0052] Then, the short-term context features generated above are... The data is fed into a Bayesian State Transition Layer for long-term dynamic modeling. This layer uses a probability distribution to describe the evolution of the health state over time.

[0053] in, This indicates that the state at the previous time step is known. and current observation features Under the given conditions, the current state is... The conditional transition probability; Indicates the first Distribution of potential health statuses at any given time (e.g., "stable", "mild deterioration", "high risk"). and B These are the state transition matrix and the feature mapping matrix, respectively.

[0054] The posterior probability of the complete sequence can be obtained through recursion:

[0055] Indicates that, given the entire observation sequence Under these conditions, potential health state sequence The probability of occurrence; This represents the complete state sequence from time 1 to time T. This represents the complete sequence of observation features from time 1 to time T. The expression "proportional to" omits the normalization constant. This represents the prior probability distribution of the health status at the initial moment; This represents a chain multiplication operation, which reflects the Markov chain property, namely that the current state depends only on the previous state and the current observation; This Transformer-Bayesian hybrid model combines the feature extraction capabilities of neural networks with the probabilistic interpretability of Bayesian methods.

[0056] Further define the cognitive-motor risk function:

[0057] in, Indicates time The overall risk level For the Sigmoid function, and These are linear weights and bias terms, respectively. Risk is trending upwards. Expressed using the time derivative:

[0058] in, To predict the time step, we represent the time window of the Transformer-Bayesian mixture model. The rate of risk change is calculated internally. To achieve early risk warning, this invention introduces a two-level dynamic threshold mechanism, used for risk level determination and trend change determination, respectively: Set a risk level threshold to determine whether a patient's current risk exceeds the safety limit. (Risk-LevelThreshold). when This threshold indicates that the patient's current overall risk is higher than the normal range. This threshold is initially determined by statistical analysis of training samples, such as the mean risk of healthy samples. The value is dynamically adjusted based on the individual baseline during system operation.

[0059] Set a risk change rate threshold to measure the rate of risk increase. (Risk-Trend Threshold). when This indicates that the patient's risk is in a phase of rapid deterioration, even if the current risk level has not yet reached a critical point. This may also indicate early abnormal trends. In one embodiment, The 90th percentile of the rate of risk increase in the sample can be used as the initial value.

[0060] When the system detects that both conditions are met and When a patient enters a high-risk state, the system automatically determines that the patient has entered a high-risk state and triggers the early warning module, sending a notification signal to the doctor or nursing staff. This dual-threshold linkage judgment mechanism combines two dimensions: "risk level" and "risk acceleration." It can prevent false alarms caused by a single abnormal fluctuation and identify potential deterioration trends in the early stages of a rapid increase in risk.

[0061] To further adapt to individual differences, the threshold is dynamically updated using an adaptive smoothing formula:

[0062] in express The latest update Class threshold (i.e.) or ); Indicates the current The threshold of time; As a smoothing factor, This is the individual's standardized risk value (the risk level adjusted to baseline). When the value is large, the threshold changes smoothly, making it suitable for individuals with low risk fluctuations; when... When the risk level is low, the system can respond quickly to changes in the patient's condition, thereby enabling individualized adjustment of risk sensitivity.

[0063] Using this formula, the Transformer-Bayesian hybrid model can automatically drift the threshold baseline as the patient's condition changes over a long period, avoiding the "false positive" or "false negative" problems caused by a fixed threshold.

[0064] 3. In clinical rehabilitation practice, there are significant differences in cognitive function, motor ability, psychological state, and lifestyle among individuals. These differences are not only reflected in absolute levels but also in the rate of risk change, stability, and sensitivity to intervention. If a uniform standard or fixed threshold is used for risk assessment, it is highly likely to lead to two types of problems: first, false positives in highly sensitive individuals (the system is too sensitive and misjudges normal fluctuations as risks); second, false negatives in less sensitive individuals (slow risk changes but serious cumulative effects fail to provide timely warnings).

[0065] To address this issue, this invention proposes an Individual Baseline Alignment and Adaptive Threshold Mechanism based on risk time series modeling. By introducing long-term individual characteristic statistics, standardized risk representation, and exponentially smoothed dynamic threshold updates, the mechanism achieves adaptive optimization of risk assessment and intervention sensitivity at the individual level.

[0066] (1) Individual baseline calculation and risk standardization.

[0067] First, stable-period risk characteristics are extracted from long-term patient monitoring data to form individual baseline parameters. These stable-period risk characteristics specifically include the comprehensive risk level output by the Transformer-Bayesian hybrid model within the time window deemed as a stable condition. mean ( ) and variance ).

[0068] The stable period is determined using a sliding window test; when the risk variance is less than a threshold over a consecutive time period, the patient is considered to be in a stable state. At this point, the mean risk for that period is calculated. with standard deviation , representing the patient's long-term average risk and the range of risk fluctuation, respectively.

[0069] Subsequently, the overall risk level was determined accordingly. Normalization is performed to obtain standardized risk values. :

[0070] in, Indicates time The original composite risk level predicted by the Transformer-Bayesian hybrid model (i.e., obtained in step 2) ); This represents an individual's baseline risk, reflecting their long-term steady-state level. The standard deviation of an individual's risk fluctuations This represents the standardized relative risk value of an individual.

[0071] Normalization transforms changes in patient risk into a change in the standard deviation relative to their long-term mean. For example, when This indicates that the risk is one standard deviation higher than the individual's baseline, while This indicates that the deviation from the expected status has exceeded the abnormal threshold, and an alert should be triggered. Through this individualized standardization, the system can eliminate the "baseline shift effect" across populations, achieving comparability of risk curves for different patients on a uniform scale.

[0072] (2) Threshold adaptive smooth update.

[0073] In acquiring standardized risks Then, the multi-level thresholds are dynamically adjusted using an exponential smoothing model:

[0074] in, Indicates the first Thresholds for risk levels (e.g., mild, moderate, severe). As a smoothing factor, it is used to balance historical memory with the current state. This is the standardized risk value at the current moment. When the value is larger (e.g., 0.9–0.95), the system has a stronger memory of historical thresholds, making it suitable for individuals with long-term stability and small fluctuations; when... At lower values ​​(e.g., 0.5–0.7), the system responds more quickly to recent changes and is suitable for patients with volatile conditions or in the early stages of recovery.

[0075] To further describe individual-differentiation sensitivity, in one embodiment, an individual sensitivity coefficient is introduced. It is defined as the ratio of the risk variance to the rate of change of the threshold:

[0076] in, It represents the variance of standardized risk values ​​within a recent time window, used to quantify the severity of current risk fluctuations in a patient; The derivative represents the rate of change of the threshold over time, reflecting how quickly the system threshold adapts to changes in risk. To prevent extremely small constants with a denominator of zero; if A larger threshold indicates greater individual risk fluctuations but a more gradual change in the threshold; the system can automatically reduce its warning sensitivity to avoid false alarms. A smaller threshold indicates that the individual risk changes are stable, and the system can increase the threshold update frequency to improve early detection capabilities.

[0077] Through joint use and The system can automatically adjust its sensitivity during operation to ensure that different patients receive a matching early warning response speed at their respective recovery stages.

[0078] (3) Threshold convergence and stability analysis.

[0079] To ensure the convergence stability of the threshold update, this invention performs mathematical verification on the above smoothing model. When the standardized risk value... When the process follows a weakly stationary process (i.e., with a finite mean and constant variance), the recursive formula is:

[0080] It can be viewed as an exponentially weighted moving average (EWMA) model, and its convergence condition is: When this condition is met, Converges at:

[0081] The threshold is a weighted average of historical risk sequences, with the weights decaying exponentially over time. This characteristic ensures the system is highly sensitive to recent conditions while retaining a memory of long-term trends, making threshold adjustments both stable and responsive. Indicates the time lag order, representing the time span for tracing back to history. Indicates the current moment. This indicates the previous moment, and so on. Indicates the first The weight decay coefficient for each historical moment; : indicates in Historical standardized risk value at any given moment.

[0082] This formula proves that the system's threshold is not a random walk, but a mathematically provable convergent stable value, which provides a theoretical guarantee for the safety of the medical system.

[0083] Through the aforementioned mechanism, this invention achieves refined adaptation at the individual level based on time-series risk modeling. The system establishes an individual risk reference system through long-term data statistics, achieving baseline alignment and enabling accurate differentiation between "true deterioration" and "normal fluctuations." Simultaneously, a dynamic self-learning threshold mechanism is introduced, using a smooth update formula to automatically adjust the risk threshold according to changes in the patient's condition, achieving a balance between long-term stability and short-term sensitivity. The system further utilizes individual sensitivity coefficients... The response speed is parameterized to maintain the robustness and individual adaptability of the model across different patients and stages.

[0084] 4. Optimization of intervention based on reinforcement learning.

[0085] In traditional rehabilitation systems, intervention plans often rely on physician experience or fixed procedures, lacking dynamic feedback and self-adjustment mechanisms. This makes it difficult to automatically optimize training content and intensity based on real-time changes in the patient's condition. This "static intervention" approach not only suffers from time lag but may also lead to mismatched intervention intensity in the early or declining stages of rehabilitation, reducing efficiency and increasing potential risks. Therefore, this invention introduces reinforcement learning (RL) principles based on risk time-series modeling. This allows the system to continuously learn within a closed-loop process of "state-action-feedback-correction," thereby forming optimal intervention strategies and achieving intelligent and individualized rehabilitation decision-making.

[0086] In this invention, the entire rehabilitation process is modeled as a Markov Decision Process (MDP). The MDP model constructed in this invention consists of a state space, an action space, a reward function, and a state transition mechanism. The system acts as an agent, and the patient and their dynamic physiological state constitute the environment. At each moment, the system observes the patient's state, selects intervention actions, and optimizes the strategy based on the patient's feedback (reward). The specific implementation steps are as follows: System-defined state space Composed of multimodal risk characteristics, time-series derivatives, and individual baseline information ( It consists of ) components used to fully describe the recovery status:

[0087] in, Standardized risk values ​​for individuals reflect the degree of deviation of their current health status from their personal homeostasis; The risk change rate represents the rate at which cognitive-motor function increases or decreases. It is a high-dimensional temporal feature vector extracted by the Transformer-Bayesian model, which describes the potential coupling relationship between cognition and motion state; It is the average risk over a long-term stable period for an individual, used as a reference point for risk assessment; This refers to the standard deviation of individual risk fluctuations, used to measure the risk stability of that individual. This multidimensional state space retains both short-term dynamic characteristics and incorporates long-term individual characteristics, enabling the system to identify the characteristic differences at different recovery stages and providing a comprehensive information basis for intervention decisions.

[0088] Selectable action space of the system It includes a variety of rehabilitation intervention methods, defined as:

[0089] in Used to improve language, memory, and attention abilities. Regarding balance and physical coordination, To achieve joint stimulation of cognitive and motor pathways, During periods of stable state, delayed or low-intensity interventions are implemented to prevent excessive fatigue. The system dynamically allocates the execution probability of each action through a policy network, achieving adaptive control of the intervention type and intensity.

[0090] To guide the system in learning appropriate intervention strategies, this invention constructs a reward function that integrates risk improvement and cost constraints:

[0091] in, For instant rewards; This represents the predicted future risk value after intervention. To predict the time step; This represents the central value of the ideal health interval; Let be the action cost function. This is used to quantify the time, energy, or resource consumption resulting from the intensity of intervention. The cost-weighting coefficient is used to balance the risk improvement effect with the intervention cost. The reward function aims to maximize the rate of risk reduction while avoiding excessive intervention that leads to energy waste or patient fatigue. When the risk level is close to the target value, the system tends to choose low-cost maintenance training; while when the risk rises or recovery stagnates, the system will adaptively increase the intensity of intervention to suppress the deterioration trend.

[0092] The optimization strategy employs an Actor-Critic dual-network structure, where the Actor network outputs the action-policy probability distribution, and the Critic network is responsible for evaluating the state value. The output of the Actor network is defined as follows:

[0093] Indicates the first Distribution of potential health states at any given time; This represents the policy function. It represents the overall state observed by the system at the current moment. (Right now Under the condition of ), choose to implement rehabilitation intervention actions. The probability distribution; This represents a deep feature extraction network (typically composed of a multilayer perceptron (MLP) or a Transformer coding layer). These are the learnable parameters of the network. It is the original high-dimensional state The mapped hidden layer feature vectors are used to capture nonlinear patterns in the state; This represents the action mapping weight matrix. It linearly projects the hidden layer feature vectors onto the dimension of the action space (i.e., the Logits values ​​corresponding to the four intervention actions).

[0094] The Critic network outputs the state value function:

[0095] in, and These are the policy and value network parameters, respectively. (In the aforementioned Actor network formula) In this context, ) represents the mapping function of the Actor network, used to output action probabilities; The mapping function (usually a multilayer perceptron) of the Critic network is used to output state values. The two networks communicate via the dominance function. For joint optimization, the update rules for the strategy parameters and value parameters are as follows:

[0096]

[0097] in, These represent the Actor (policy) network at the current time. And after the update Weight parameters at each time step; These represent the Critic (value) network at the current moment. And after the update Weight parameters at time points; This represents the gradient operator. This represents the log-policy probability relative to the parameters. The gradient direction indicates how to adjust the parameters to increase the probability of selecting high-quality actions. and For learning rate, This serves as a discount factor. Through continuous interaction and gradient updates, the system can gradually converge to a stable optimal strategy, maximizing long-term cumulative returns, which in turn minimizes the patient's overall risk level over time. .

[0098] This invention further analyzes the convergence of reinforcement learning strategies from a theoretical perspective. If the discount factor... Furthermore, since the learning rate follows a gradual decay law, the parameter update process of the Actor-Critic dual network satisfies the Robbins-Monro stochastic approximation convergence criterion, which guarantees that the strategy gradually approaches the local optimum. At this point, the expected state transition relation satisfies:

[0099] in Indicates the optimal strategy The mathematical expectation under the influence of an action. It does not refer to the result of a single experiment, but rather to the result of the system under long-term, rigorous execution of the optimal intervention strategy. Under the given conditions, the average of all possible probabilistic state transition outcomes is calculated. The physical meaning of this inequality is that, under optimal policy control, the patient's next state transition outcome... The expected risk will not be higher than the current value. This means that, statistically speaking, the non-deterioration and stability of the rehabilitation process are guaranteed.

[0100] This state transition inequality indicates that the patient's expected risk does not increase over time, thus achieving stability in functional recovery. Physiologically, this process manifests as a dynamic stabilization mechanism of "stimulus-adaptation-rebalancing" during patient rehabilitation: when the frequency or intensity of intervention is too high, leading to an increase in physiological load, the cost term... When the weight of an action increases, the system will automatically reduce the frequency of intervention; conversely, when the risk increases or the improvement is slow, the system will reallocate the probability of actions and increase the intensity of intervention in order to maintain the continuity and direction of functional improvement.

[0101] 5. To prevent the performance of the Transformer-Bayesian hybrid model from drifting over time, this invention further designs a closed-loop backflow and policy recalibration mechanism.

[0102] Real-time monitoring of patient feedback signals after receiving the system-recommended intervention. Intervention feedback signals will be collected in real time. (Including improvements in motor performance rate, semantic response speed, and scale score changes) and obtaining the predicted state or expected response value output by the Transformer-Bayesian hybrid model at the current time step. The error obtained from the comparison is:

[0103] in Indicates time The prediction error at any given time; Indicates time At any time, based on the patient's real intervention feedback signals The actual observed risk value obtained from the assessment (such as the rate of improvement in athletic performance, changes in scale scores, etc.); This indicates that the Transformer-Bayesian hybrid model is in time... The expected risk value output at any given time (i.e., the corresponding expected response value) The overall risk prediction level that is mapped.

[0104] Obtaining error Subsequently, the system is configured with an allowable error tolerance threshold. When absolute error This indicates that the state transition patterns predicted by the current hybrid model have significantly deviated from the patient's actual recovery trajectory. At this point, the system will automatically trigger a backflow correction mechanism, constructing a loss function using observational feedback and predicted output. The state transition matrix of the Bayesian state transition layer is updated using gradient descent.

[0105] in, This is the Bayesian state transition matrix, used to characterize the evolutionary relationship between cognitive and motor states; The learning rate is used to control the correction step size. The loss function measures the deviation between the prediction and the actual feedback. The core idea of ​​this correction process is "model and patient co-evolution": when the system overestimates the risk (false positives), gradient correction reduces the weight of the relevant channels; when the risk is underestimated (false negatives), the system strengthens the influence of that channel, thereby achieving self-regulation and continuous optimization of the parameters.

[0106] The feedback correction closed-loop mechanism, combined with actual intervention feedback, dynamically corrects model parameters, ensuring that predicted results remain consistent with actual rehabilitation performance and preventing model performance drift over time. The entire mechanism theoretically guarantees the convergence stability of the threshold and, in engineering practice, balances continuous learning and personalized adjustment capabilities. In summary, this mechanism enables the early warning system of this invention not only to accurately predict overall risk trends in the population but also to dynamically adapt to the physiological fluctuations and rehabilitation rhythm of each patient, achieving truly individualized, intelligent, and precise rehabilitation management.

[0107] In practice, the combination of reinforcement learning algorithms and closed-loop feedback mechanisms enables the system to automatically adjust intervention plans based on the patient's condition. When a decline in motor ability or postural instability is detected, the system increases the proportion of motor training and combined training; when a slowdown in cognitive response or semantic comprehension impairment is detected, the system increases the weight of cognitive tasks; and in the later stages of recovery, when the risk level stabilizes and the volatility decreases, the system automatically enters a delayed intervention phase, reducing training frequency to avoid over-fatigue. The entire process forms a dynamic equilibrium adaptive adjustment mechanism, naturally coupling the rehabilitation pace, intervention intensity, and the patient's actual recovery process.

[0108] In summary, this invention, by introducing an MDP decision model and a closed-loop correction mechanism, achieves a shift in intervention strategies from "rule-driven" to "data-driven." The system can not only automatically learn the optimal rehabilitation path in a high-dimensional state space, but also self-correct and continuously optimize through feedback signals during long-term operation. This theoretically ensures convergence stability and, in engineering terms, achieves dynamic adaptation and physiological interpretability of rehabilitation strategies, significantly improving the efficiency, accuracy, and safety of the intelligent rehabilitation system.

[0109] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A cognitive-motor rehabilitation early warning and intervention method based on multimodal temporal modeling, characterized in that, Includes the following steps: 1) Fuse multimodal risk features to obtain multimodal risk features; 2) By using a hybrid model to capture the dynamic multimodal risk characteristics of short-term fluctuations and long-term evolution, the overall risk level can be predicted and the upward trend of risk can be predicted; 3) Adaptively adjust the sensitivity of individual baseline parameters according to the overall risk level to achieve adaptive optimization of sensitivity at the individual level; 4) Based on the state observed by the Markov decision model, select the appropriate intervention action, and generate a reward signal based on the patient's response to the intervention to guide the next step of strategy optimization; 5) Collect intervention feedback signals in real time, compare the real-time collected intervention feedback signals with the expected state output by the hybrid model at the current time to obtain the error. If the error is greater than the threshold, update the state transition matrix of the hybrid model.

2. The cognitive-motor rehabilitation early warning and intervention method based on multimodal temporal modeling according to claim 1, characterized in that, Step 1) of the fusion includes: reconstructing the input multimodal data using a VAE structure to obtain the completed multimodal risk features, and introducing an attention weighting mechanism to avoid equal weighting of all modalities. The weights are normalized by softmax to reflect the contribution of each modality to the overall state at the current time.

3. The cognitive-motor rehabilitation early warning and intervention method based on multimodal temporal modeling according to claim 1, characterized in that, Step 2) includes the following steps: First, a multi-head self-attention structure is used to obtain short-term contextual features of adjacent time slices for multimodal risk features. ; the short-term context features Input into the Bayesian state transition layer for long-term dynamic modeling; Then, define the cognitive-motor risk function: in, Indicates time The overall risk level For the Sigmoid function, and These are the linear weights and the bias term, respectively. Let be the short-term context feature vector after local context modeling; Finally, the patient's current risk is estimated by considering the overall risk level and the upward trend of risk.

4. The cognitive-motor rehabilitation early warning and intervention method based on multimodal temporal modeling according to claim 3, characterized in that, Estimate the patient's current risk by assessing risk level and trend changes: Set a risk level threshold to determine whether a patient's current risk exceeds the safety limit. ,when This indicates that the patient's current overall risk is higher than the normal range; Set a risk change rate threshold to measure the rate of risk increase. ,when This indicates that the patient's risk is in a phase of rapid deterioration, even if the current risk level has not yet reached a critical point. ; When both conditions are met and When a patient is identified as being in a high-risk state, the early warning module is triggered, sending a notification signal to the doctor or caregiver.

5. The cognitive-motor rehabilitation early warning and intervention method based on multimodal temporal modeling according to claim 1, characterized in that, Step 3) includes the following steps: 31) Individual baseline calculation and risk standardization: Extract stable-phase risk characteristics from long-term patient monitoring data to form individual baseline parameters; determine patient status using the sliding window detection method; When the risk variance is less than the threshold over a continuous period of time, the patient is considered to be in a stable state. The risk value is then normalized to obtain a standardized risk value, which reflects the relationship between the risk at the corresponding moment and the patient's long-term average risk status. 32) Threshold Adaptive Smoothing Update: The multi-level thresholds are dynamically adjusted through a smoothing model. An individual sensitivity coefficient is introduced. By combining the smoothing factor and the individual sensitivity coefficient, automatic sensitivity adjustment is achieved to ensure that different patients can obtain a matching early warning response speed at their respective recovery stages.

6. The cognitive-motor rehabilitation early warning and intervention method based on multimodal temporal modeling according to claim 1, characterized in that, The establishment of a Markov decision model includes the following steps: First, the defined state space Used to fully describe the recovery status: in, Standardized risk values ​​for individuals reflect the degree of deviation of their current health status from their personal homeostasis; The risk change rate represents the rate at which cognitive-motor function increases or decreases. These are features extracted from a hybrid model, which characterize the potential coupling relationship between cognition and motor state. It is the average risk over a long-term stable period for an individual, used as a reference point for risk assessment; This is the standard deviation of individual risk volatility, used to measure the risk stability of that individual; Define selectable action space : in Used to improve language, memory, and attention abilities. Regarding balance and physical coordination, To achieve joint stimulation of cognitive and motor pathways, During periods of stable condition, delayed or low-intensity interventions should be implemented to prevent excessive fatigue. Construct a reward function that integrates risk improvement and cost constraints: in, For instant rewards; This represents the predicted future risk value after intervention. To predict the time step; This represents the central value of the ideal health interval; Let be the action cost function. This is used to quantify the time, energy, or resource consumption resulting from the intensity of intervention. This is the cost weighting coefficient, used to balance the risk improvement effect with the intervention cost.

7. The cognitive-motor rehabilitation early warning and intervention method based on multimodal temporal modeling according to claim 1, characterized in that, Step 4) The policy optimization adopts an Actor-Critic dual network structure, in which the Actor network outputs the action policy probability distribution, and the Critic network is responsible for evaluating the state value.

8. A system for implementing the cognitive-motor rehabilitation early warning and intervention method based on multimodal temporal modeling as described in claim 1, characterized in that, include: The multimodal risk feature fusion module is used to fuse multi-source data to obtain multimodal risk features; The State Prediction and Estimation module is used to predict health status and estimate risk trends using a hybrid model. The individual baseline alignment and adaptive threshold module is used to adaptively optimize the sensitivity of individual baseline parameters by introducing long-term individual characteristic statistics, standardized risk representation, and exponential smoothing dynamic threshold updates. The action intervention module is used to observe the patient's state through a Markov decision model, select intervention actions, and optimize the strategy based on the patient's feedback. The correction module is used to update the state transition matrix of the hybrid model through a backflow correction mechanism.

9. A computer device comprising a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, characterized in that: When the computer program is executed by the processor, it causes the processor to implement the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor implements the method as described in any one of claims 1 to 8.

Images