Mirror neuron system driven adaptive vr rehabilitation training method and system
By acquiring and cleaning multimodal data, combined with feature enhancement of mirror neuron systems and graph neural network analysis, the problems of insufficient data quality and adaptability in traditional systems are solved, achieving high-quality data fusion and neural modulation feedback, thereby improving the effectiveness of rehabilitation training and its clinical application value.
Patent Information
- Application Number
- CN202511864455.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-11
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2045-12-11
AI Technical Summary
Traditional adaptive virtual reality rehabilitation training systems have shortcomings in multi-source data temporal alignment and refined processing, resulting in low data signal-to-noise ratio, lack of deep intervention capability in adaptive logic, and lack of interpretability in AI models, which affects the optimization of rehabilitation effects and clinical translation value.
A multimodal data acquisition and processing module is used for data preprocessing and cleaning. A multimodal time-series dataset is generated using timestamp alignment and adaptive filtering algorithms. Neurophysiological and motor behavior features are extracted. VR scene parameters are enhanced by features of relevant brain regions of the mirror neuron system. Interpretable diagnostic reports are generated by combining graph neural networks and SHAP methods.
It improves the quality of multimodal data and optimizes the adaptive mechanism, enabling adaptive analysis from the behavioral level to the neuromodulation level, promoting the reorganization of patients' neural functions, generating accurate diagnostic and early warning reports, and enhancing the value of the system for clinical translation.
Smart Images
Figure CN121641353B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of rehabilitation training technology, and more specifically, to an adaptive VR rehabilitation training method and system driven by a mirror neuron system. Background Technology
[0002] The adaptive virtual reality rehabilitation training system is a new type of rehabilitation method that uses VR technology to create an immersive rehabilitation environment and integrates artificial intelligence algorithms to dynamically adjust the training program based on the patient's real-time physiological and behavioral performance. Its main purpose is to improve patient participation and rehabilitation efficiency by providing personalized and highly interactive training content.
[0003] However, traditional adaptive virtual reality rehabilitation training systems often suffer from the following shortcomings during use: First, while traditional systems can collect multi-source data, they generally lack refined processing of this data, leading to difficulties in accurately aligning the temporal sequence of the data. Furthermore, the raw data contains significant environmental noise, and inputting this "dirty data" into the analysis model results in low signal-to-noise ratios for extracted behavioral and neural features, further impacting the accuracy of the analysis. Second, the adaptive logic of traditional systems is mostly based on simple behavioral classifications for adjustments, such as task completion level or error rate. This limits adjustments to the level of task difficulty and makes it difficult to address the patient's overall training and rehabilitation needs. The solution involves deep intervention, making it difficult for traditional systems to fundamentally optimize rehabilitation outcomes. Thirdly, the AI models used in traditional systems lack interpretability of feature data, making it difficult for doctors to understand the specific assessment results and warning information output by the model when faced with data feedback. This hinders the conversion of system data into clinical use and greatly reduces the clinical value of adaptive virtual reality rehabilitation training systems. In general, how to effectively solve the problems of poor quality of multimodal fusion data, low degree of optimization of adaptive mechanisms, and insufficient value of system conversion into clinical applications in traditional systems has become the problem that adaptive virtual reality rehabilitation training systems need to face and solve.
[0004] In view of this, the present invention proposes an adaptive VR rehabilitation training method and system driven by a mirror neuron system to solve the above problems. Summary of the Invention
[0005] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution, including: The multimodal data acquisition and processing module is used to acquire raw data from multidimensional data sources based on the system command interface, and to preprocess the raw data to obtain a multimodal time series dataset. Furthermore, the steps of collecting raw data from multi-dimensional data sources based on the system command interface and preprocessing the raw data include: S1.1: Collect EEG data, eye movement data, and motion parameters based on the system command interface to obtain raw data; S1.2: Using a timestamp alignment algorithm, the original data is timestamped to obtain aligned original data. The specific calculation formula for timestamp synchronization is as follows: ; Get synchronization timestamp ,in, For the first The first item in the data source The timestamp of each data point For the first Time offset of the data source; S1.3: Using linear interpolation to generate aligned raw data with uniform sampling points, a multimodal time series dataset is obtained. The specific formula for linear interpolation is: ; in, To align the first in the dataset The data source is at a specific point in time. Time series data, , and The timestamp of the original data point; S1.4: Output the multimodal time series dataset to the data noise filtering and cleaning module; The data noise filtering and cleaning module is used to clean the multimodal time series dataset to obtain cleaned multimodal data. Furthermore, the steps for cleaning multimodal time-series datasets include: S2.1: Based on the multimodal time series dataset, the filter parameters are initialized. The least mean square adaptive filter with order 32 is used for the eye movement data and action parameters in the multimodal time series dataset, and the recursive least square adaptive filter with order 64 is used for the EEG data. S2.2: An adaptive filtering algorithm is used to process the multimodal time-series dataset to obtain pre-cleaned multimodal data. The output value of the least mean square adaptive filter is shown in section S2.2. The weight update rules are as follows: ; in, For time points Transpose of the filter weight coefficient vector For time points The input data vector, For time points The filter weight coefficient vector, For step size parameters, This is an error signal; The coefficient adjustment rule for the recursive least squares adaptive filter is as follows: ; in, For time points Estimation of the inverse correlation matrix, Forgetting factor, It is the gain vector; S2.3: Validate the pre-cleaned multimodal data based on the signal-to-noise ratio (SNR). If the SNR is less than or equal to the validation threshold, return to step S2.2 to adjust the filtering parameters in the adaptive filtering algorithm and perform filtering again. If the SNR is greater than the validation threshold, use the pre-cleaned multimodal data as the cleaned multimodal data and proceed to step S2.4. The specific calculation formula is as follows: ; in, For signal power, Noise power; S2.4: Output the cleaned multimodal data from step S2.3 to the neurophysiological and motor behavior feature enhancement module; The neurophysiological and motor behavior feature enhancement module is used to extract neurophysiological and motor behavior-related features based on cleaned multimodal data, and enhance key features to obtain enhanced neurophysiological and motor behavior features of brain regions related to the mirror neuron system. Furthermore, the steps of extracting neurophysiological and motor behavior-related features based on the cleaned multimodal data and enhancing key features include: S3.1: Use AI algorithms to extract feature vectors from the cleaned multimodal data. The feature vectors include behavioral feature data. Brain region feature data related to the mirror neuron system The specific extraction method is as follows: By substituting into the calculation formula: ; in, For time points Location data, To maximize the function, For time points short-time power, The average short-time power under resting conditions; S3.2: Use the feature vector as the key feature and use the attention mechanism to calculate the attention weight of the key feature; By multiplying the attention weights corresponding to the data items in the feature vector, the neurophysiological and motor behavior features of the relevant brain regions of the mirror neuron system after enhancement are obtained. The specific formula for calculating attention weights is as follows: ; Get the first The data source is at a specific point in time. attention weights ,in, It is an exponential function. For the first The learning coefficient of the data source. For feature vectors, The number of feature data; S3.3: Output the enhanced neurophysiological and motor behavior features of the relevant brain regions of the mirror neuron system to the VR scene adaptive control module and the neurophysiological and motor behavior coupling analysis module; The VR scene adaptive control module is used to dynamically adjust VR scene parameters based on the enhanced neurophysiological and motor behavior characteristics of the relevant brain regions of the mirror neuron system, and to obtain a dynamic parameter adjustment record report. Furthermore, the steps for dynamically adjusting VR scene parameters based on enhanced neurophysiological and motor behavioral characteristics of brain regions related to the mirror neuron system include: S4.1: Extract behavioral feature data and brain region feature data related to the mirror neuron system based on enhanced neurophysiological and motor behavioral features of the brain regions related to the mirror neuron system; S4.2: Based on the behavioral feature data and brain region feature data related to the mirror neuron system in step S4.1, adjust the task difficulty parameters and environmental complexity parameters in the VR scene to obtain task difficulty data. and environmental complexity data ; The specific formula group to be adjusted is as follows: ; in, Basic difficulty The behavioral feature data in step S4.1 For the purpose of coherence, Based on basic complexity, This is the gain coefficient. This refers to the brain region feature data related to the mirror neuron system in step S4.1; S4.3: Based on the VR scene control API, transmit task difficulty data and environmental complexity data, and adjust the task difficulty and environmental complexity in the scene in real time according to the transmitted task difficulty data and environmental complexity data. The adjustment includes: increasing the task difficulty when the patient's movements are smooth, decreasing the task difficulty when the patient's movements are disjointed, and increasing the environmental complexity when the mirror neural activity is enhanced. S4.4: Record the adjustment parameters and adjustment timestamps in step S4.3 to obtain a parameter dynamic adjustment record report; S4.5: Store the dynamic parameter adjustment record report to the database; The neurophysiological and motor behavior coupling analysis module is used to integrate and analyze the cleaned multimodal data and the enhanced neurophysiological and motor behavior features of the relevant brain regions of the mirror neuron system to obtain a neurophysiological and motor behavior coupling atlas. Furthermore, the steps for integrating and analyzing the cleaned multimodal data and the enhanced neurophysiological and motor behavioral features of the relevant brain regions of the mirror neuron system include: S5.1: The cleaned multimodal data from step S2.4 and the enhanced neurophysiological and motor behavioral features of the mirror neuron system related brain regions from step S3.3 are fused into a joint feature vector; S5.2: Analyze the coupling of joint feature vectors using graph neural networks, extract coupling features from the output of graph neural networks, and obtain a coupling map of neurophysiology and motor behavior; S5.3: Output the coupled neurophysiological and motor behavior map to the diagnostic interpretation and early warning report processing module; The diagnostic interpretation and early warning report processing module is used to analyze based on the coupling map of neurophysiology and motor behavior to obtain interpretable diagnostic reports and early warning reports; Furthermore, the steps for analysis based on the coupling map of neurophysiology and motor behavior include: S6.1: The contribution of each feature in the neurophysiological and motor behavior coupling atlas to the diagnostic results is calculated using the SHAP method, and the SHAP value is obtained. The formula for calculating the SHAP value is as follows: ; Get SHAP value ,in, For feature subset, For feature set, For prediction models; S6.2: Sort features based on SHAP values to obtain a feature importance sequence, and generate an interpretable diagnostic report based on the feature importance sequence; S6.3: Construct a logistic regression model based on the SHAP value and predict the risk probability to obtain the risk probability value. The specific calculation formula for the prediction is as follows: ; in, and These are model parameters; S6.4: When the risk probability value is greater than or equal to the risk threshold, generate an early warning report; S6.5: A visual panel that outputs interpretable diagnostic reports and early warning reports to the doctor's receiving end; Furthermore, S1: Collect raw data from multi-dimensional data sources based on the system instruction interface, and preprocess the raw data to obtain a multimodal time series dataset; S2: Clean the multimodal time series dataset to obtain cleaned multimodal data; S3: Extract neurophysiological and motor behavior-related features based on the cleaned multimodal data, and enhance key features to obtain enhanced neurophysiological and motor behavior features of brain regions related to the mirror neuron system; S4: Based on the enhanced neurophysiological and motor behavior characteristics of the relevant brain regions of the mirror neuron system, the VR scene parameters are dynamically adjusted to obtain a dynamic parameter adjustment record report; S5: Integrate and analyze the cleaned multimodal data and the enhanced neurophysiological and motor behavior features of the relevant brain regions of the mirror neuron system to obtain a coupled neurophysiological and motor behavior map; S6: Based on the analysis of the coupling map of neurophysiology and motor behavior, interpretable diagnostic reports and early warning reports are obtained.
[0006] The technical effects and advantages of the adaptive VR rehabilitation training method and system driven by the mirror neuron system of this invention are as follows: This invention collects raw data from multi-dimensional data sources via a system command interface, preprocesses the raw data to obtain a multimodal time-series dataset, cleans the multimodal time-series dataset to obtain cleaned multimodal data, extracts neurophysiological and motor behavior-related features based on the cleaned multimodal data, and enhances key features to obtain enhanced neurophysiological and motor behavior features of brain regions related to the mirror neuron system. Based on these enhanced neurophysiological and motor behavior features, VR scene parameters are dynamically adjusted, generating a parameter dynamic adjustment record report. The cleaned multimodal data and the enhanced neurophysiological and motor behavior features of brain regions related to the mirror neuron system are integrated and analyzed to obtain a neurophysiological and motor behavior coupling map. Analysis based on this coupling map yields interpretable diagnostic and early warning reports, enabling the system to collect multimodal data... The cascaded processing of the integrated processing module and the data noise filtering and cleaning module solves the quality problem of the raw data to the greatest extent, thereby greatly improving the reliability of the data. In addition, the present invention also extends the adaptive analysis of the system from the simple behavioral level to the neural regulation level through the neurophysiological and motor behavior feature enhancement module and the VR scene adaptive control module, and realizes a complete closed loop from neural activity to scene feedback, thereby effectively promoting the reorganization and rehabilitation of patients' neural functions. Finally, through the neurophysiological and motor behavior coupling analysis module and the diagnostic interpretation and early warning report processing module, the system can correlate behavioral performance with neural mechanisms, and on this basis, use interpretable AI technology to generate accurate and clear diagnostic and early warning reports, thereby greatly increasing the value of the system for clinical use. Overall, the present invention has significant advantages such as high quality of multimodal fusion data, good optimization effect of adaptive mechanism, and great potential for clinical use. Attached Figure Description
[0007] Figure 1 This is a schematic diagram of the adaptive VR rehabilitation training system driven by the mirror neuron system of the present invention. Figure 2 This is a schematic diagram of the adaptive VR rehabilitation training method driven by the mirror neuron system of the present invention. Detailed Implementation
[0008] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0009] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.
[0010] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”
[0011] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.
[0012] In practice, the server-side equipment deployed in the mirror neuron system-driven adaptive VR rehabilitation training system may consist of one or more devices. This mirror neuron system-driven adaptive VR rehabilitation training system can be implemented as a business instance, a virtual machine, or hardware devices. For example, it can be implemented as a business instance deployed on one or more devices in a cloud node. Simply put, it can be understood as software deployed on a cloud node, providing the mirror neuron system-driven adaptive VR rehabilitation training system to various user terminals. Alternatively, it can be implemented as a virtual machine deployed on one or more devices in a cloud node, with application software installed to manage each user terminal. Or, it can also be implemented as a server composed of numerous identical or different types of hardware devices, with one or more hardware devices configured to provide the mirror neuron system-driven adaptive VR rehabilitation training system to various user terminals.
[0013] In terms of implementation, the adaptive VR rehabilitation training system driven by the mirror neuron system and the user terminal are mutually adapted. That is, if the adaptive VR rehabilitation training system driven by the mirror neuron system is implemented as an application installed on a cloud service platform, then the user terminal is implemented as a client that establishes a communication connection with the application; or if the adaptive VR rehabilitation training system driven by the mirror neuron system is implemented as a website, then the user terminal is implemented as a webpage; or if the adaptive VR rehabilitation training system driven by the mirror neuron system is implemented as a cloud service platform, then the user terminal is implemented as a mini-program in an instant messaging application.
[0014] like Figure 1 The figure shown is a system architecture diagram of an adaptive VR rehabilitation training system driven by a mirror neuron system provided in an embodiment of the present invention.
[0015] The adaptive VR rehabilitation training system driven by the mirror neuron system described in this invention can be set up in a cloud server. In terms of implementation, it can be implemented as one or more service devices, or as an application installed in the cloud (e.g., a mobile service operator's server, server cluster, etc.), or it can be developed as a website. Depending on the implemented functions, the adaptive VR rehabilitation training system driven by the mirror neuron system may include a multimodal data acquisition and processing module, a data noise filtering and cleaning module, a neurophysiological and motor behavior feature enhancement module, a VR scene adaptive control module, a neurophysiological and motor behavior coupling analysis module, and a diagnostic interpretation, early warning, and report processing module. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.
[0016] In this embodiment of the invention, in the adaptive VR rehabilitation training system driven by the mirror neuron system, each of the above modules can be implemented independently and can call other modules. Here, "calling" can be understood as a module connecting to multiple modules of another type and providing corresponding services to those connected modules. For example, the sharing and assessment module can call the same information collection module to obtain the information collected by that module. Based on the above characteristics, in the adaptive VR rehabilitation training system driven by the mirror neuron system provided in this embodiment of the invention, without modifying the program code, the applicable scope of the adaptive VR rehabilitation training system architecture driven by the mirror neuron system can be adjusted by adding modules and directly calling them, achieving cluster-based horizontal expansion to quickly and flexibly expand the adaptive VR rehabilitation training system driven by the mirror neuron system. In practical applications, the above modules can be set in the same device or different devices, or they can be set in a virtual device, such as a service instance in a cloud server.
[0017] Example 1 Please see Figure 1 As shown in this embodiment, the adaptive VR rehabilitation training system driven by the mirror neuron system includes: The multimodal data acquisition and processing module is used to acquire raw data from multidimensional data sources based on the system command interface, and to preprocess the raw data to obtain a multimodal time series dataset. Furthermore, the steps of collecting raw data from multi-dimensional data sources based on the system command interface and preprocessing the raw data include: S1.1: Collect EEG data, eye movement data, and motion parameters based on the system command interface to obtain raw data; It should be explained that motion parameters refer to quantitative indicators related to the patient's operational functions collected by the tracking device or wearable sensor of the VR system, such as the three-dimensional coordinates of the hand, head or joints in virtual space, joint angles, movement trajectory, speed and acceleration, etc. S1.2: Using a timestamp alignment algorithm, the original data is timestamped to obtain aligned original data. The specific calculation formula for timestamp synchronization is as follows: ; Get synchronization timestamp ,in, For the first The first item in the data source The timestamp of each data point For the first Time offset of the data source; It should be explained that the time offset is obtained by using linear regression to fit and minimize the deviation of each data source from the standard clock; S1.3: Using linear interpolation to generate aligned raw data with uniform sampling points, a multimodal time series dataset is obtained. The specific formula for linear interpolation is: ; in, To align the first in the dataset The data source is at a specific point in time. Time series data, , and The timestamp of the original data point; S1.4: Output the multimodal time series dataset to the data noise filtering and cleaning module; The data noise filtering and cleaning module is used to clean the multimodal time series dataset to obtain cleaned multimodal data. Further steps for cleaning multimodal time-series datasets include: S2.1: Based on the multimodal time series dataset, the filter parameters are initialized. The least mean square adaptive filter with order 32 is used for the eye movement data and action parameters in the multimodal time series dataset, and the recursive least square adaptive filter with order 64 is used for the EEG data. S2.2: An adaptive filtering algorithm is used to process the multimodal time-series dataset to obtain pre-cleaned multimodal data. The output value of the least mean square adaptive filter is shown in section S2.2. The weight update rules are as follows: ; in, For time points Transpose of the filter weight coefficient vector For time points The input data vector, For time points The filter weight coefficient vector, For step size parameters, This is an error signal; It should be explained that the input data vector in the least mean square adaptive filter is eye-tracking data and action parameters from a multimodal time series dataset; The coefficient adjustment rule for the recursive least squares adaptive filter is as follows: ; in, For time points Estimation of the inverse correlation matrix, Forgetting factor, It is the gain vector; S2.3: Validate the pre-cleaned multimodal data based on the signal-to-noise ratio (SNR). If the SNR is less than or equal to the validation threshold, return to step S2.2 to adjust the filtering parameters in the adaptive filtering algorithm and perform filtering again. If the SNR is greater than the validation threshold, use the pre-cleaned multimodal data as the cleaned multimodal data and proceed to step S2.4. The specific calculation formula is as follows: ; in, For signal power, Noise power; It should be explained that the verification threshold is manually set and input into the system; the filtering parameters refer to the step size parameter and forgetting factor in step S2.2. S2.4: Output the cleaned multimodal data from step S2.3 to the neurophysiological and motor behavior feature enhancement module; The neurophysiological and motor behavior feature enhancement module is used to extract neurophysiological and motor behavior related features based on the cleaned multimodal data, and enhance key features to obtain neurophysiological and motor behavior features of brain regions related to the mirror neuron system after enhancement. Furthermore, the steps of extracting neurophysiological and motor behavior-related features based on the cleaned multimodal data and enhancing key features include: S3.1: Use AI algorithms to extract feature vectors from the cleaned multimodal data. The feature vectors include behavioral feature data. Brain region feature data related to the mirror neuron system The specific extraction method is as follows: By substituting into the calculation formula: ; in, For time points Location data, To maximize the function, For time points short-time power, The average short-time power under resting conditions; It should be explained that short-time power refers to the short-time power of the μ rhythm extracted from the EEG signal; S3.2: Use the feature vector as the key feature and use the attention mechanism to calculate the attention weight of the key feature; By multiplying the attention weights corresponding to the data items in the feature vector, the neurophysiological and motor behavior features of the relevant brain regions of the mirror neuron system after enhancement are obtained. The specific formula for calculating attention weights is as follows: ; Get the first The data source is at a specific point in time. attention weights ,in, It is an exponential function. For the first The learning coefficient of the data source. For feature vectors, The number of feature data; It should be explained that the learning coefficients are determined by optimizing them using gradient descent, retrieving historical feature vectors from the database to obtain the training set, and then iteratively updating the learning coefficients based on the training set. S3.3: Output the enhanced neurophysiological and motor behavior features of the relevant brain regions of the mirror neuron system to the VR scene adaptive control module and the neurophysiological and motor behavior coupling analysis module; The VR scene adaptive control module is used to dynamically adjust VR scene parameters based on the enhanced neurophysiological and motor behavior characteristics of the relevant brain regions of the mirror neuron system, and obtain a parameter dynamic adjustment record report. Furthermore, the steps for dynamically adjusting VR scene parameters based on enhanced neurophysiological and motor behavioral characteristics of brain regions related to the mirror neuron system include: S4.1: Extract behavioral feature data and brain region feature data related to the mirror neuron system based on enhanced neurophysiological and motor behavioral features of the brain regions related to the mirror neuron system; S4.2: Based on the behavioral feature data and brain region feature data related to the mirror neuron system in step S4.1, adjust the task difficulty parameters and environmental complexity parameters in the VR scene to obtain task difficulty data. and environmental complexity data ; The specific formula group to be adjusted is as follows: ; in, Basic difficulty The behavioral feature data in step S4.1 For the purpose of coherence, Based on basic complexity, This is the gain coefficient. This refers to the brain region feature data related to the mirror neuron system in step S4.1; S4.3: Based on the VR scene control API, transmit task difficulty data and environmental complexity data, and adjust the task difficulty and environmental complexity in the scene in real time according to the transmitted task difficulty data and environmental complexity data. The adjustment includes: increasing the task difficulty when the patient's movements are smooth, decreasing the task difficulty when the patient's movements are disjointed, and increasing the environmental complexity when the mirror neural activity is enhanced. S4.4: Record the adjustment parameters and adjustment timestamps in step S4.3 to obtain a parameter dynamic adjustment record report; S4.5: Store the dynamic parameter adjustment record report to the database; The neurophysiological and motor behavior coupling analysis module is used to integrate and analyze the cleaned multimodal data and the enhanced neurophysiological and motor behavior features of the relevant brain regions of the mirror neuron system to obtain a neurophysiological and motor behavior coupling map. Further steps in integrating and analyzing the cleaned multimodal data and the enhanced neurophysiological and motor behavioral features of the mirror neuron system's related brain regions include: S5.1: The cleaned multimodal data from step S2.4 and the enhanced neurophysiological and motor behavioral features of the mirror neuron system related brain regions from step S3.3 are fused into a joint feature vector; S5.2: Analyze the coupling of joint feature vectors using graph neural networks, extract coupling features from the output of graph neural networks, and obtain a coupling map of neurophysiology and motor behavior; It should be explained that coupling features refer to, for example, the correlation coefficient between feature terms; the neurophysiological and motor behavior coupling map includes a relationship matrix and feature importance. S5.3: Output the coupled neurophysiological and motor behavior map to the diagnostic interpretation and early warning report processing module; The diagnostic interpretation and early warning report processing module is used to analyze based on the coupling map of neurophysiology and motor behavior to obtain interpretable diagnostic reports and early warning reports; Further steps in the analysis based on the coupling map of neurophysiology and motor behavior include: S6.1: The contribution of each feature in the neurophysiological and motor behavior coupling atlas to the diagnostic results is calculated using the SHAP method, and the SHAP value is obtained. The formula for calculating the SHAP value is as follows: ; Get SHAP value ,in, For feature subset, For feature set, For prediction models; S6.2: Sort features based on SHAP values to obtain a feature importance sequence, and generate an interpretable diagnostic report based on the feature importance sequence; S6.3: Construct a logistic regression model based on the SHAP value and predict the risk probability to obtain the risk probability value. The specific calculation formula for the prediction is as follows: ; in, and These are model parameters; S6.4: When the risk probability value is greater than or equal to the risk threshold, generate an early warning report; S6.5: A visual panel that outputs interpretable diagnostic reports and early warning reports to the doctor's receiving end; In this embodiment, the beneficial effects are achieved by collecting raw data from multi-dimensional data sources through a system command interface, preprocessing the raw data to obtain a multimodal time-series dataset, cleaning the multimodal time-series dataset to obtain cleaned multimodal data, extracting neurophysiological and motor behavior-related features based on the cleaned multimodal data, and enhancing key features to obtain enhanced neurophysiological and motor behavior features of brain regions related to the mirror neuron system. Based on these enhanced neurophysiological and motor behavior features, VR scene parameters are dynamically adjusted, resulting in a parameter dynamic adjustment record report. The cleaned multimodal data and the enhanced neurophysiological and motor behavior features of brain regions related to the mirror neuron system are integrated and analyzed to obtain a neurophysiological and motor behavior coupling map. Analysis based on this coupling map yields interpretable diagnostic and early warning reports, enabling the system to utilize multimodal data... The cascaded processing of the data acquisition and processing module and the data noise filtering and cleaning module effectively addresses the quality issues of the raw data, thereby significantly improving data reliability. Furthermore, this invention utilizes a neurophysiological and motor behavior feature enhancement module and a VR scene adaptive control module to extend the system's adaptive analysis from a purely behavioral level to a neural modulation level, achieving a complete closed loop from neural activity to scene feedback. This effectively promotes the reorganization and rehabilitation of patients' neurological functions. Finally, through the neurophysiological and motor behavior coupling analysis module and the diagnostic interpretation and early warning report processing module, the system can correlate behavioral performance with neural mechanisms. Based on this, it uses interpretable AI technology to generate accurate and clear diagnostic and early warning reports, greatly increasing the system's value for clinical application. Overall, this invention has significant advantages in terms of high-quality multimodal fusion data, effective adaptive mechanism optimization, and significant potential for clinical application.
[0018] Example 2 Please see Figure 2 As shown, the parts not described in detail in this embodiment are described in Embodiment 1. An adaptive VR rehabilitation training method driven by a mirror neuron system is provided. The method includes: S1: collecting raw data from multi-dimensional data sources based on the system instruction interface, and preprocessing the raw data to obtain a multimodal time series dataset. S2: Clean the multimodal time series dataset to obtain cleaned multimodal data; S3: Extract neurophysiological and motor behavior-related features based on the cleaned multimodal data, and enhance key features to obtain enhanced neurophysiological and motor behavior features of brain regions related to the mirror neuron system; S4: Based on the enhanced neurophysiological and motor behavior characteristics of the relevant brain regions of the mirror neuron system, the VR scene parameters are dynamically adjusted to obtain a dynamic parameter adjustment record report; S5: Integrate and analyze the cleaned multimodal data and the enhanced neurophysiological and motor behavior features of the relevant brain regions of the mirror neuron system to obtain a coupled neurophysiological and motor behavior map; S6: Based on the analysis of the coupling map of neurophysiology and motor behavior, interpretable diagnostic reports and early warning reports are obtained.
[0019] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the present invention.
Claims
1. An adaptive VR rehabilitation training system driven by a mirror neuron system, characterized in that, The system includes: a neurophysiological and motor behavior feature enhancement module, a VR scene adaptive control module, a neurophysiological and motor behavior coupling analysis module, and a diagnostic interpretation, early warning, and report processing module, wherein: The neurophysiological and motor behavior feature enhancement module is used to extract the features and motor behavior features of the brain regions related to the mirror neuron system contained in the cleaned multimodal data, and to enhance the data weights of key features to obtain the neurophysiological and motor behavior features of the brain regions related to the mirror neuron system after feature enhancement. The VR scene adaptive control module is used to dynamically adjust VR scene parameters based on the enhanced neurophysiological and motor behavior characteristics of the relevant brain regions of the mirror neuron system, and obtain a parameter dynamic adjustment record report. The neurophysiological and motor behavior coupling analysis module is used to integrate and analyze the cleaned multimodal data and the enhanced neurophysiological and motor behavior features of the relevant brain regions of the mirror neuron system to obtain a neurophysiological and motor behavior coupling map. The diagnostic interpretation and early warning report processing module is used to analyze based on the coupling map of neurophysiology and motor behavior to obtain interpretable diagnostic reports and early warning reports; The steps involved in the analysis based on the coupling map of neurophysiology and motor behavior include: S6.1: The contribution of each feature in the neurophysiological and motor behavior coupling atlas to the diagnostic results is calculated using the SHAP method to obtain the SHAP value; S6.2: Sort features based on SHAP values to obtain a feature importance sequence, and generate an interpretable diagnostic report based on the feature importance sequence; S6.3: Construct a logistic regression model based on the SHAP value and predict the risk probability to obtain the risk probability value; S6.4: When the risk probability value is greater than or equal to the risk threshold, generate an early warning report; S6.5: Output interpretable diagnostic reports and early warning reports to a visual panel on the doctor's receiving end.
2. The adaptive VR rehabilitation training system driven by the mirror neuron system according to claim 1, characterized in that, The system further includes: a multimodal data acquisition and processing module and a data noise filtering and cleaning module, wherein: The multimodal data acquisition and processing module is used to acquire raw data from multidimensional data sources based on the system command interface, and to preprocess the raw data to obtain a multimodal time series dataset. The data noise filtering and cleaning module is used to clean the multimodal time series dataset to obtain cleaned multimodal data.
3. The adaptive VR rehabilitation training system driven by the mirror neuron system according to claim 1, characterized in that, The steps for collecting raw data from multi-dimensional data sources based on the system command interface and preprocessing the raw data include: S1.1: Collect EEG data, eye movement data, and motion parameters based on the system command interface to obtain raw data; S1.2: Use a timestamp alignment algorithm to synchronize the timestamps of the original data to obtain aligned original data; S1.3: Use linear interpolation to generate aligned raw data with uniform sampling points to obtain a multimodal time series dataset; S1.4: Output the multimodal time series dataset to the data noise filtering and cleaning module.
4. The adaptive VR rehabilitation training system driven by the mirror neuron system according to claim 1, characterized in that, The steps for cleaning multimodal time series datasets include: S2.1: Based on the multimodal time series dataset, the filter parameters are initialized. The least mean square adaptive filter with order 32 is used for the eye movement data and action parameters in the multimodal time series dataset, and the recursive least square adaptive filter with order 64 is used for the EEG data. S2.2: Use an adaptive filtering algorithm to calculate the multimodal time series dataset to obtain the preliminarily cleaned multimodal data; S2.3: Verify the pre-cleaned multimodal data based on the signal-to-noise ratio. When the signal-to-noise ratio is less than or equal to the verification threshold, return to step S2.2 to adjust the filtering parameters in the adaptive filtering algorithm and perform filtering again. When the signal-to-noise ratio is greater than the verification threshold, use the pre-cleaned multimodal data as the cleaned multimodal data and proceed to step S2.
4. S2.4: Output the cleaned multimodal data from step S2.3 to the neurophysiological and motor behavior feature enhancement module.
5. The adaptive VR rehabilitation training system driven by the mirror neuron system according to claim 1, characterized in that, The steps for extracting neurophysiological and motor behavior-related features from cleaned multimodal data and enhancing key features include: S3.1: Use AI algorithms to extract feature vectors from the cleaned multimodal data. The feature vectors include behavioral feature data and brain region feature data related to the mirror neuron system. S3.2: Use the feature vector as the key feature and use the attention mechanism to calculate the attention weight of the key feature; By multiplying the attention weights corresponding to the data items in the feature vector, the neurophysiological and motor behavior features of the relevant brain regions of the mirror neuron system after enhancement are obtained. S3.3: Output the enhanced neurophysiological and motor behavior features of the relevant brain regions of the mirror neuron system to the VR scene adaptive control module and the neurophysiological and motor behavior coupling analysis module.
6. The adaptive VR rehabilitation training system driven by the mirror neuron system according to claim 1, characterized in that, The steps for dynamically adjusting VR scene parameters based on enhanced neurophysiological and motor behavioral characteristics of brain regions related to the mirror neuron system include: S4.1: Extract behavioral feature data and brain region feature data related to the mirror neuron system based on enhanced neurophysiological and motor behavioral features of the brain regions related to the mirror neuron system; S4.2: Based on the behavioral feature data and brain region feature data related to the mirror neuron system in step S4.1, adjust the task difficulty parameters and environmental complexity parameters in the VR scene to obtain task difficulty data and environmental complexity data; S4.3: Based on the VR scene control API, transmit task difficulty data and environmental complexity data, and adjust the task difficulty and environmental complexity in the scene in real time according to the transmitted task difficulty data and environmental complexity data. The adjustment includes: increasing the task difficulty when the patient's movements are smooth, decreasing the task difficulty when the patient's movements are disjointed, and increasing the environmental complexity when the mirror neural activity is enhanced. S4.4: Record the adjustment parameters and adjustment timestamps from step S4.3 to obtain a parameter dynamic adjustment record report; S4.5: Store the dynamic parameter adjustment record report to the database.
7. The adaptive VR rehabilitation training system driven by the mirror neuron system according to claim 1, characterized in that, The steps for integrating and analyzing the cleaned multimodal data and the enhanced neurophysiological and motor behavioral features of the mirror neuron system's related brain regions include: S5.1: The cleaned multimodal data from step S2.4 and the enhanced neurophysiological and motor behavioral features of the mirror neuron system related brain regions from step S3.3 are fused into a joint feature vector; S5.2: Analyze the coupling of joint feature vectors using graph neural networks, extract coupling features from the output of graph neural networks, and obtain a coupling map of neurophysiology and motor behavior; S5.3: Output the coupled neurophysiological and motor behavior map to the diagnostic interpretation, early warning, and report processing module.
8. A mirror neuron system-driven adaptive VR rehabilitation training method, implemented according to any one of claims 1-7, characterized in that, The work includes the following steps: S1: Collect raw data from multi-dimensional data sources based on the system command interface, and preprocess the raw data to obtain a multimodal time series dataset; S2: Clean the multimodal time series dataset to obtain cleaned multimodal data; S3: Extract neurophysiological and motor behavior-related features based on the cleaned multimodal data, and enhance key features to obtain enhanced neurophysiological and motor behavior features of brain regions related to the mirror neuron system; S4: Based on the enhanced neurophysiological and motor behavior characteristics of the relevant brain regions of the mirror neuron system, the VR scene parameters are dynamically adjusted to obtain a dynamic parameter adjustment record report; S5: Integrate and analyze the cleaned multimodal data and the enhanced neurophysiological and motor behavior features of the relevant brain regions of the mirror neuron system to obtain a coupled neurophysiological and motor behavior map; S6: Based on the analysis of the coupling map of neurophysiology and motor behavior, interpretable diagnostic reports and early warning reports are obtained.