VR-based Parkinson's hand rehabilitation training system

The VR-based Parkinson's hand rehabilitation training system provides personalized training plans and real-time feedback, solving the problems of insufficient efficiency and effectiveness in hand rehabilitation training for Parkinson's patients. It achieves accurate recognition of hand posture data and movement guidance, improving training effectiveness and patient participation.

CN118370968BActive Publication Date: 2026-11-17BEIFANG UNIV OF NATITIES +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202410467971.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-18
Publication Date
2026-11-17
Estimated Expiration
2044-04-18

AI Technical Summary

Technical Problem

Existing technologies lack personalized VR training programs for hand rehabilitation in Parkinson's patients, and the accuracy of sensor detection data during virtual interaction is difficult to guarantee, resulting in insufficient efficiency and effectiveness of hand rehabilitation training.

Method used

A VR-based Parkinson's hand rehabilitation training system is adopted, including a planning subsystem, a training execution subsystem, and a result feedback subsystem. Through a VR hand posture recognition method based on multimodal data fusion, it provides personalized training plans and real-time feedback, and uses VR devices to accurately identify hand posture data and provide movement guidance.

Benefits of technology

This approach enables personalized and precise hand rehabilitation training for Parkinson's patients, improving training efficiency and effectiveness, and enhancing patients' motivation and interest in rehabilitation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118370968B_ABST
    Figure CN118370968B_ABST
Patent Text Reader

Abstract

This invention discloses a VR-based hand rehabilitation training system for Parkinson's disease, belonging to the field of medical rehabilitation technology. It includes a planning subsystem for generating a target training plan for the patient based on basic strength testing; a training execution subsystem for performing hand rehabilitation training according to the patient's target training plan, using a VR hand posture recognition method based on multimodal data fusion, and continuously obtaining hand posture data; and a result feedback subsystem for providing training movement guidance, evaluating the training process, generating training reports, and displaying the evaluation results based on the hand posture data. This invention solves the problem of insufficient effectiveness and efficiency in hand rehabilitation training for Parkinson's patients.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of medical rehabilitation technology, and in particular relates to a VR-based Parkinson's hand rehabilitation training system. Background Technology

[0002] Parkinson's disease is a neurodegenerative disease, one of the main symptoms of which is hand motor dysfunction, including hand tremors and muscle rigidity. These symptoms severely affect patients' quality of life and daily function. Traditional rehabilitation methods usually require direct guidance and supervision from medical staff, the training process is slow, patients need to spend a lot of time and energy participating in rehabilitation training, and they are prone to losing motivation during the rehabilitation process. With the development of technology, VR technology is gradually being used in the field of medical rehabilitation.

[0003] However, there is currently a lack of applications for VR technology in hand rehabilitation for Parkinson's patients. It is difficult to provide targeted and precise rehabilitation training programs based on the individual strength needs of different patients through interaction with a virtual environment. The accuracy of sensor data during virtual interaction is also a major challenge. These issues result in insufficient efficiency and effectiveness of hand rehabilitation training based on virtual interaction. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a VR-based Parkinson's hand rehabilitation training system. This system utilizes VR technology to provide personalized hand rehabilitation training for Parkinson's patients. Through interaction with a virtual environment, it trains hand grasping skills, thus solving the problem of insufficient effectiveness and efficiency in hand rehabilitation training for Parkinson's patients.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0006] This invention provides a VR-based Parkinson's hand rehabilitation training system, comprising:

[0007] The plan includes a subsystem for generating target training plans for patients based on basic strength tests.

[0008] The training execution subsystem is used to perform hand rehabilitation training based on a VR hand posture recognition method using multimodal data fusion, according to the patient's target training plan, by connecting to a VR device, and continuously obtain hand posture data.

[0009] The results feedback subsystem is used to provide training action guidance, evaluate the training process, and generate training reports based on hand posture data, and to display the evaluation results of the training process.

[0010] The beneficial effects of this invention are as follows: This invention provides a VR-based Parkinson's hand rehabilitation training system. Through a planning subsystem, it provides basic strength testing for hand rehabilitation training of Parkinson's patients and automatically generates a suitable target training plan based on the results of the basic strength test. Through a training execution subsystem, it can conduct personalized hand rehabilitation training for Parkinson's patients based on VR technology according to the patient's target training plan. Specifically, through a VR hand posture recognition method using multimodal data fusion, it can accurately and quickly identify the hand posture data of the patient during interactions with the virtual environment, such as grasping exercises. This provides a foundation for evaluating the standard of movement and providing relevant movement guidance during the patient's rehabilitation training. The VR-based Parkinson's hand rehabilitation training provided by this invention increases the patient's motivation for rehabilitation training and can provide more precise rehabilitation training programs based on the different strength needs of different patients. The result feedback subsystem can accurately guide and evaluate the patient's training, providing a foundation for understanding their own rehabilitation training progress and continuously improving training effectiveness.

[0011] Furthermore, the planning subsystem includes:

[0012] The basic strength testing module is used to conduct finger strength tests, palm strength tests, and grip strength tests on patients respectively, and obtain the strength test results.

[0013] The training plan generation module is used to generate initial training plans for several sets of training items with corresponding magnitudes of reaction force based on the strength test results.

[0014] The self-adjustment module is used to increase or decrease training items and the number of sets corresponding to the training items based on the initial training plan, so as to obtain the patient's target training plan.

[0015] The beneficial effects of adopting the above-mentioned further solution are as follows: The planning subsystem provided by the present invention can obtain the strength test results by conducting finger strength test, palm strength test and grip strength test on the patient. Based on the strength test results, it automatically generates an initial training plan for several sets of training items with corresponding magnitude reaction forces. On the basis of the initial training plan, the patient can add or subtract training items and the number of sets corresponding to the training items, thereby realizing the scientific generation and customized setting of the patient's target training plan.

[0016] Furthermore, the training execution subsystem includes:

[0017] The VR access module is used to provide a VR handheld device worn by the patient, wherein the VR device has a VR hand posture recognition method for multimodal data fusion, as well as a posture sensor and several hand pressure sensors for sensing hand movement.

[0018] The VR scene module is used to provide a VR head-mounted device worn by the patient, wherein the VR head-mounted device is used to provide virtual training scenes, training action guidance prompts, training time prompts, and training group and training project execution progress prompts;

[0019] The training reward module is used to generate training rewards when the patient completes each set of training items. The training rewards include reward visual stimuli and corresponding reward verbal stimuli.

[0020] The mode selection module is used to select active training mode, passive training mode, and specific movement training mode according to the patient's target training plan for hand rehabilitation training.

[0021] The training execution module is used to perform hand rehabilitation training based on training rewards after the patient wears a VR handheld device and a VR headset, according to the patient's target training plan and one of the selected active training mode, passive training mode or specific movement training mode, and continuously obtain hand posture data.

[0022] The beneficial effects of adopting the above-mentioned further solutions are as follows: The training execution subsystem provided by the present invention, based on the hand posture recognition method of multimodal data fusion, realizes accurate collection of sensor data in the process of hand rehabilitation training through the VR access module, and provides virtual reality training scenarios through the VR scene module. In the training scenario, training action guidance, training time, and relevant prompts on the execution progress of training groups and training projects are provided. On this basis, patients can choose training modes to carry out rehabilitation training with rewards, which effectively increases the efficiency of rehabilitation training for patients, and increases the patients' motivation and interest in rehabilitation while improving the rehabilitation effect.

[0023] Furthermore, the VR hand pose recognition method based on multimodal data fusion includes the following steps:

[0024] S1. Obtain acceleration and angle data generated by posture sensors and pressure data generated by hand pressure sensors during hand rehabilitation training, as well as data generated by sensor measurements at each time point, and construct a sensor measurement generation dataset.

[0025] S2. Construct a multimodal data fusion network;

[0026] S3. Use sensor measurements to generate a dataset to train a multimodal data fusion network, and obtain the trained multimodal data fusion network;

[0027] S4. Using the trained multimodal data fusion network, feature extraction and prediction are performed on the sensor measurement data generated at each input time point to obtain the hand posture recognition result, thus completing the VR hand posture recognition through multimodal data fusion.

[0028] The beneficial effects of adopting the above-mentioned further solutions are as follows: This invention provides a VR hand posture recognition method based on multimodal data fusion. By using data generated by sensors during the hand rehabilitation training of patients, a sensor measurement-generated dataset is constructed, and a multimodal data fusion network is constructed and trained. This enables accurate extraction of hand posture data during the hand rehabilitation training process through the trained multimodal data fusion network, providing a foundation for accurately evaluating the effectiveness of hand rehabilitation training and continuously improving the efficiency of patients' hand rehabilitation training.

[0029] Furthermore, the multimodal data fusion network includes an input layer, an SSA module, a CNN sub-network, an LSTM module, a GRU module, a fully connected layer, a fourth activation layer, and an output layer connected in sequence; the CNN sub-network includes a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer, a third convolutional layer, and a third activation function layer connected in sequence.

[0030] The input layer is used to input the data generated by the sensor measurements at each time point;

[0031] The SSA module is used to denoise the data generated by the sensor at each time point to obtain hand posture data;

[0032] The CNN subnetwork is used to extract basic spatial features from hand pose data and fuse these basic spatial features to obtain spatial fusion features.

[0033] The LSTM module is used to extract long short-term memory features based on spatial fusion features through input gate, forget gate and output gate;

[0034] The GRU module is used to optimize the long short-term memory features through reset gates and update gates to obtain spatiotemporal features;

[0035] The fully connected layer is used for feature classification based on spatiotemporal features. After the feature classification result is activated by the fourth activation function layer, the output layer outputs the hand gesture recognition result.

[0036] The beneficial effects of adopting the above-mentioned further solution are as follows: The structure of the multimodal data fusion network provided by the present invention can input the sensor data acquired during the hand rehabilitation training process into the SSA module through the input layer for noise reduction and temporal dimension processing to obtain hand posture data. Through the CNN sub-network, spatial features of the hand posture data can be extracted. Through the LSTM module and GRU module, the spatiotemporal features of the hand posture data can be extracted based on the spatial features. Then, through the fully connected layer and the fourth activation layer, the hand posture can be recognized to obtain the hand posture recognition result.

[0037] Furthermore, the method for denoising the data generated by the sensor measurements at each time point to obtain hand posture data includes the following steps:

[0038] A1. Construct the sensing embedding time series of the attitude sensor and pressure sensor respectively;

[0039] The calculation expression for the sensor embedding time series is as follows:

[0040]

[0041] Among them, Y n This represents the data generated by the nth sensor. This represents the data generated by the nth sensor at time 1. This represents the data generated by the nth sensor at time 2. This represents the data generated by the nth sensor at time T′.

[0042] A2. Transform the sensor embedding time series into a multi-dimensional sequence to obtain the trajectory matrix;

[0043] The formula for calculating the trajectory matrix is ​​as follows:

[0044]

[0045]

[0046] Among them, X n Represents the trajectory matrix. This represents the 1D embedded time series of the nth sensor. This represents the 2D embedded time series of the nth sensor. This represents the i-dimensional embedded time series of the nth sensor. This represents the K-dimensional embedding time series of the nth sensor. This represents the data generated by the nth sensor at time 1. This represents the data generated by the nth sensor at time k. This represents the data generated by the nth sensor at time L. Let L represent the data generated by the nth sensor at time T′, where L represents the embedding dimension and K represents the number of columns in the trajectory matrix. This represents the data generated by the nth sensor at time i. This represents the data generated by the nth sensor at time i+1. This represents the data generated by the nth sensor at time i+L-1;

[0047] A3. Perform singular value decomposition on the trajectory matrix to obtain the left and right singular value decomposition results;

[0048] The calculation expressions for the left and right singular value decomposition results are as follows:

[0049]

[0050] Δ=diag(σ1,σ2,...,σ d )

[0051] Among them, U L×L Let Δ denote the left singular matrix of the trajectory matrix, Δ denote the diagonal matrix, diag(·) denote the diagonal matrix function, σ1 denote the first positive singular value of the trajectory matrix, σ2 denote the second positive singular value of the trajectory matrix, and σ d This represents the d-th positive singular value of the trajectory matrix. The transpose of the right singular matrix of the trajectory matrix is ​​denoted by T, where T represents the transpose of the matrix.

[0052] A4. Based on the results of the left and right singular value decompositions, select several large singular value descriptions of the trajectory matrix to obtain the denoised approximate trajectory matrix;

[0053] The expression for calculating the approximate trajectory matrix is ​​as follows:

[0054]

[0055] in, U represents the approximate trajectory matrix. L×r Let Δ be a left singular matrix with r singular values. r×r Represents a diagonal matrix with r singular values. Represents the transpose of a right singular matrix with r singular values, (h i′,j′ ) L×K This represents the denoised signal element located in the i-th row and j-th column of the approximate trajectory matrix;

[0056] A5. Diagonally average the approximate trajectory matrix to obtain hand posture data;

[0057] The calculation expression for the hand posture data is as follows:

[0058]

[0059]

[0060] L p =min(L,K), K p =max(L,K),

[0061] Among them, Y n_res This represents hand posture data. This represents the data generated by the nth sensor during the denoising and reconstruction at time 1. This represents the data generated by the nth sensor during the second time step in the denoising and reconstruction process. This represents the data generated by the nth sensor at time k during the denoising and reconstruction process. h represents the data generated by the nth sensor at time T′ during denoising and reconstruction. m,k-m+1 L represents the denoised signal element at the (k-m+1)th position in the m-th row of the approximate trajectory matrix. p K represents the left boundary of the diagonal element. p This indicates the right boundary of the diagonal elements. min(·) represents the minimum value, and max(·) represents the maximum value.

[0062] The beneficial effects of adopting the above-mentioned further solution are as follows: The present invention provides a method for the SSA module to denoise the data generated by the sensor at each time point to obtain hand posture data. By constructing a time series, matrix processing and singular value decomposition and reconstruction, the time dimension and denoising processing of the sensor-generated data obtained during hand rehabilitation training are realized, thereby providing high-precision data support for the spatiotemporal feature extraction and hand posture recognition of hand posture data.

[0063] Furthermore, the result feedback subsystem includes:

[0064] The training movement guidance module is used to determine whether the patient's movements meet the standards based on hand posture data in a specific movement training mode. It also displays the changes in the specific movement, non-compliance prompts, and compliance prompts in the virtual training scene through the VR scene module, until the hand rehabilitation training in the specific movement training mode is completed.

[0065] The training process evaluation module is used to score and evaluate the training time and achievement level of each training group and each training item in the complete hand rehabilitation training, and obtain the training process evaluation results.

[0066] The training report generation module is used to print the evaluation results of the training process, and after the hand rehabilitation training is completed, the evaluation results are displayed in a virtual training scene through the VR scene module.

[0067] The beneficial effects of adopting the above-mentioned further solutions are as follows: The result feedback subsystem provided by the present invention can determine the compliance rate of the patient's rehabilitation movements based on the hand posture data through the training movement guidance module, and promote the compliance of the patient's movements in the hand rehabilitation training process by prompting the training movements. The training process evaluation module and training report generation module realize the feedback and display of the movement evaluation results in the rehabilitation training, thereby gaining a clearer understanding of the patient's hand rehabilitation status.

[0068] Furthermore, it also includes an information entry subsystem; the information entry subsystem is used to enter personal basic information for patients undergoing VR-based Parkinson's hand rehabilitation training for the first time; the personal basic information includes user ID, name, age, gender, medical history and current symptoms.

[0069] The beneficial effects of adopting the above-mentioned further solutions are as follows: The information entry subsystem provided by the present invention can enter basic information for patients who are undergoing VR-based Parkinson's hand rehabilitation training for the first time through the system of the present invention, thereby realizing unified management of patient information according to specific circumstances, and facilitating data management of patient hand rehabilitation information and rehabilitation status.

[0070] Furthermore, it also includes a data storage subsystem; the data storage subsystem is used to store the patient's basic personal information and the evaluation results of the training process corresponding to each hand rehabilitation training session.

[0071] The beneficial effects of adopting the above-mentioned further solutions are as follows: The present invention provides a data storage subsystem, which realizes the associated storage of patients' basic personal information and training process evaluation results, providing a basis for observing patients' rehabilitation status.

[0072] Other advantages of the present invention will be analyzed in more detail in the following embodiments. Attached Figure Description

[0073] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This is a structural block diagram of a VR-based Parkinson's hand rehabilitation training system according to Embodiment 1 of the present invention.

[0075] Figure 2 This is a structural diagram of the multimodal data fusion network in Embodiment 1 of the present invention.

[0076] Figure 3 This is a structural block diagram of a VR-based Parkinson's hand rehabilitation training system according to Embodiment 2 of the present invention. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0078] VR, or Virtual Reality, is a comprehensive technology that integrates computer technology, sensor technology, human psychology, and physiology. It uses computer simulation systems to simulate the external environment, mainly simulating the environment, skills, sensing devices, and perception, providing users with a multi-information, three-dimensional dynamic, and interactive simulation experience.

[0079] Example 1:

[0080] like Figure 1 As shown, in one embodiment of the present invention, the present invention provides a VR-based Parkinson's hand rehabilitation training system, comprising:

[0081] The plan includes a subsystem for generating target training plans for patients based on basic strength tests.

[0082] The planning subsystem includes:

[0083] The basic strength testing module is used to conduct finger strength tests, palm strength tests, and grip strength tests on patients respectively, and obtain the strength test results.

[0084] The training plan generation module is used to generate initial training plans for several sets of training items with corresponding magnitudes of reaction force based on the strength test results.

[0085] The self-adjustment module is used to increase or decrease training items and the number of sets corresponding to the training items based on the initial training plan, so as to obtain the patient's target training plan.

[0086] The training execution subsystem is used to perform hand rehabilitation training based on a VR hand posture recognition method using multimodal data fusion, according to the patient's target training plan, by connecting to a VR device, and continuously obtain hand posture data.

[0087] The training execution subsystem includes:

[0088] The VR access module is used to provide a VR handheld device worn by the patient. The VR device has a VR hand posture recognition method with multimodal data fusion, as well as a posture sensor and several hand pressure sensors to sense the hand movement state. In this embodiment, the posture sensor is a six-axis hand gyroscope.

[0089] The VR hand pose recognition method based on multimodal data fusion includes the following steps:

[0090] S1. Obtain acceleration and angle data generated by posture sensors and pressure data generated by hand pressure sensors during hand rehabilitation training, as well as data generated by sensor measurements at each time point, and construct a sensor measurement generation dataset.

[0091] S2. Construct a multimodal data fusion network;

[0092] like Figure 2 As shown, the multimodal data fusion network includes an input layer, an SSA module, a CNN sub-network, an LSTM module, a GRU module, a fully connected layer, a fourth activation layer, and an output layer connected in sequence; the CNN sub-network includes a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer, a third convolutional layer, and a third activation function layer connected in sequence.

[0093] The input layer is used to input the data generated by the sensor measurements at each time point;

[0094] The SSA module is used to denoise the data generated by the sensor at each time point to obtain hand posture data;

[0095] The method for denoising the data generated by the sensor at each time point to obtain hand posture data includes the following steps:

[0096] A1. Construct the sensing embedding time series of the attitude sensor and pressure sensor respectively;

[0097] The calculation expression for the sensor embedding time series is as follows:

[0098]

[0099] Among them, Y n This represents the data generated by the nth sensor. This represents the data generated by the nth sensor at time 1. This represents the data generated by the nth sensor at time 2. This represents the data generated by the nth sensor at time T′.

[0100] A2. Transform the sensor embedding time series into a multi-dimensional sequence to obtain the trajectory matrix;

[0101] The formula for calculating the trajectory matrix is ​​as follows:

[0102]

[0103]

[0104] Among them, X n Represents the trajectory matrix. This represents the 1D embedded time series of the nth sensor. This represents the 2D embedded time series of the nth sensor. This represents the i-dimensional embedded time series of the nth sensor. This represents the K-dimensional embedding time series of the nth sensor. This represents the data generated by the nth sensor at time 1. This represents the data generated by the nth sensor at time k. This represents the data generated by the nth sensor at time L. Let L represent the data generated by the nth sensor at time T′, where L represents the embedding dimension and K represents the number of columns in the trajectory matrix. This represents the data generated by the nth sensor at time i. This represents the data generated by the nth sensor at time i+1. This represents the data generated by the nth sensor at time i+L-1;

[0105] A3. Perform singular value decomposition on the trajectory matrix to obtain the left and right singular value decomposition results;

[0106] The calculation expressions for the left and right singular value decomposition results are as follows:

[0107]

[0108] Δ=diag(σ1,σ2,…,σ d )

[0109] Among them, U L×LLet Δ denote the left singular matrix of the trajectory matrix, Δ denote the diagonal matrix, diag(·) denote the diagonal matrix function, σ1 denote the first positive singular value of the trajectory matrix, σ2 denote the second positive singular value of the trajectory matrix, and σ d This represents the d-th positive singular value of the trajectory matrix. The transpose of the right singular matrix of the trajectory matrix is ​​denoted by T, where T represents the transpose of the matrix.

[0110] A4. Based on the results of the left and right singular value decompositions, select several large singular value descriptions of the trajectory matrix to obtain the denoised approximate trajectory matrix;

[0111] The expression for calculating the approximate trajectory matrix is ​​as follows:

[0112]

[0113] in, U represents the approximate trajectory matrix. L×r Let Δ be a left singular matrix with r singular values. r×r Represents a diagonal matrix with r singular values. Represents the transpose of a right singular matrix with r singular values, (h i′,j′ ) L×K This represents the denoised signal element located in the i-th row and j-th column of the approximate trajectory matrix;

[0114] A5. Diagonally average the approximate trajectory matrix to obtain hand posture data;

[0115] The calculation expression for the hand posture data is as follows:

[0116]

[0117]

[0118] L p =min(L,K), K p =max(L,K),

[0119] Among them, Y n_res This represents hand posture data. This represents the data generated by the nth sensor during the denoising and reconstruction at time 1. This represents the data generated by the nth sensor during the second time step in the denoising and reconstruction process. This represents the data generated by the nth sensor at time k during the denoising and reconstruction process. h represents the data generated by the nth sensor at time T′ during denoising and reconstruction. m,k-m+1 L represents the denoised signal element at the (k-m+1)th position in the m-th row of the approximate trajectory matrix. pK represents the left boundary of the diagonal element. p This indicates the right boundary of the diagonal elements. min(·) represents the minimum value, and max(·) represents the maximum value.

[0120] The CNN subnetwork is used to extract basic spatial features from hand pose data and fuse these basic spatial features to obtain spatial fusion features. In this embodiment, the CNN subnetwork is a one-dimensional CNN convolutional neural network. The number of filters in the first, second, and third convolutional layers is 60, the kernel size is 3×3, and the first, second, and third activation function layers all use the Sigmoid activation function.

[0121] The LSTM module is used to extract long short-term memory features based on spatial fusion features through input gate, forget gate, and output gate; the LSTM module has 128 neurons and uses the tanh activation function;

[0122] The GRU module is used to optimize long short-term memory features through reset gates and update gates to obtain spatiotemporal features; the GRU module has 128 neurons and uses the tanh activation function.

[0123] The fully connected layer is used for feature classification based on spatiotemporal features. After the feature classification result is activated by the fourth activation function layer, the output layer outputs the hand gesture recognition result. In this embodiment, the fully connected layer has one neuron, and the fourth activation function layer uses the sigmoid activation function.

[0124] S3. Use sensor measurements to generate a dataset to train a multimodal data fusion network, and obtain the trained multimodal data fusion network;

[0125] S4. Using the trained multimodal data fusion network, feature extraction and prediction are performed on the sensor measurement data generated at each input time point to obtain the hand posture recognition result, thus completing the VR hand posture recognition through multimodal data fusion.

[0126] The VR scene module is used to provide a VR head-mounted device worn by the patient, wherein the VR head-mounted device is used to provide virtual training scenes, training action guidance prompts, training time prompts, and training group and training project execution progress prompts;

[0127] The training reward module is used to generate training rewards when the patient completes each set of training items. The training rewards include reward visual stimuli and corresponding reward verbal stimuli.

[0128] The mode selection module is used to select active training mode, passive training mode, and specific movement training mode according to the patient's target training plan for hand rehabilitation training.

[0129] In this embodiment, the active training mode involves the patient actively exerting force with their hand to obtain active training data collected by the posture sensor and hand pressure sensor. The collected active training data is then processed by the central processing unit and displayed on the HMI screen. The passive training mode involves driving a motor at a preset speed to move the finger, palm, and other hand joint tissues for hand rehabilitation training. The specific movement training mode involves selecting several training items from the target training plan and performing hand rehabilitation training by actively exerting force with the hand according to the selected training items to obtain specific movement training data collected by the posture sensor and hand pressure sensor. The collected specific movement training data is then processed by the central processing unit and displayed on the HMI screen.

[0130] The training execution module is used to perform hand rehabilitation training based on training rewards after the patient wears a VR handheld device and a VR headset, according to the patient's target training plan and one of the selected active training mode, passive training mode or specific movement training mode, and continuously obtain hand posture data.

[0131] The results feedback subsystem is used to provide training action guidance, evaluate the training process, and generate training reports based on hand posture data, and to display the evaluation results of the training process.

[0132] The result feedback subsystem includes:

[0133] The training movement guidance module is used to determine whether the patient's movements meet the standards based on hand posture data in a specific movement training mode. It also displays the changes in the specific movement, non-compliance prompts, and compliance prompts in the virtual training scene through the VR scene module, until the hand rehabilitation training in the specific movement training mode is completed.

[0134] The training process evaluation module is used to score and evaluate the training time and achievement level of each training group and each training item in the complete hand rehabilitation training, and obtain the training process evaluation results.

[0135] The training report generation module is used to print the evaluation results of the training process, and after the hand rehabilitation training is completed, the evaluation results are displayed in a virtual training scene through the VR scene module.

[0136] Example 2:

[0137] like Figure 3As shown, based on the planning subsystem, training execution subsystem and result feedback subsystem provided in Example 1, the VR-based Parkinson's hand rehabilitation training system provided by the present invention also includes an information input subsystem and a data storage subsystem.

[0138] The information entry subsystem is used to enter basic personal information for patients undergoing VR-based Parkinson's hand rehabilitation training for the first time; the basic personal information includes user ID, name, age, gender, medical history, and current symptoms.

[0139] The data storage subsystem is used to store the patient's basic personal information and the evaluation results of the training process corresponding to each hand rehabilitation training session.

[0140] In this invention, the patient's basic personal information and the evaluation results of each hand rehabilitation training session are all associated with the user ID.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A VR-based Parkinson's hand rehabilitation training system, characterized in that, include: The plan includes a subsystem for generating target training plans for patients based on basic strength tests. The training execution subsystem is used to perform hand rehabilitation training based on a VR hand posture recognition method using multimodal data fusion, according to the patient's target training plan, by connecting to a VR device, and continuously obtain hand posture data. The VR hand pose recognition method based on multimodal data fusion includes the following steps: S1. Obtain acceleration and angle data generated by posture sensors and pressure data generated by hand pressure sensors during hand rehabilitation training, as well as data generated by sensor measurements at each time point, and construct a sensor measurement generation dataset. S2. Construct a multimodal data fusion network; The multimodal data fusion network includes an input layer, an SSA module, a CNN sub-network, an LSTM module, a GRU module, a fully connected layer, a fourth activation layer, and an output layer connected in sequence. The input layer is used to input the data generated by the sensor measurements at each time point; The SSA module is used to denoise the data generated by the sensor at each time point to obtain hand posture data; The method for denoising the data generated by the sensor at each time point to obtain hand posture data includes the following steps: A1. Construct the sensing embedding time series of the attitude sensor and pressure sensor respectively; The calculation expression for the sensor embedding time series is as follows: , in, This represents the data generated by the nth sensor. This represents the data generated by the nth sensor at time 1. This represents the data generated by the nth sensor at time 2. Indicates the nth sensor at the th... Data generated at each time point; A2. Transform the sensor embedding time series into a multi-dimensional sequence to obtain the trajectory matrix; The formula for calculating the trajectory matrix is ​​as follows: , in, Represents the trajectory matrix. This represents the 1D embedded time series of the nth sensor. This represents the 2D embedded time series of the nth sensor. This represents the i-dimensional embedded time series of the nth sensor. This represents the K-dimensional embedding time series of the nth sensor. This represents the data generated by the nth sensor at time 1. This represents the data generated by the nth sensor at time k. This represents the data generated by the nth sensor at time L. Indicates the nth sensor at the th... The data generated at each time point, where L represents the embedding dimension and K represents the number of columns in the trajectory matrix. This represents the data generated by the nth sensor at time i. This represents the data generated by the nth sensor at time i+1. This represents the data generated by the nth sensor at time i+L-1; A3. Perform singular value decomposition on the trajectory matrix to obtain the left and right singular value decomposition results; The calculation expressions for the left and right singular value decomposition results are as follows: in, The left singular matrix representing the trajectory matrix. Represents a diagonal matrix. Represents a diagonal matrix function. This represents the first positive singular value of the trajectory matrix. This represents the second positive singular value of the trajectory matrix. This represents the d-th positive singular value of the trajectory matrix. The transpose of the right singular matrix of the trajectory matrix is ​​denoted by T, where T represents the transpose of the matrix. A4. Based on the results of the left and right singular value decompositions, select several large singular value descriptions of the trajectory matrix to obtain the denoised approximate trajectory matrix; The expression for calculating the approximate trajectory matrix is ​​as follows: in, Represents the approximate trajectory matrix. Represents a left singular matrix with r singular values. Represents a diagonal matrix with r singular values. Let r represent the transpose of a right singular matrix with r singular values. This represents the denoised signal element located at the i-th row and j-th column of the approximate trajectory matrix; A5. Diagonally average the approximate trajectory matrix to obtain hand posture data; The calculation expression for the hand posture data is as follows: , , , in, This represents hand posture data. This represents the data generated by the nth sensor during the denoising and reconstruction at time 1. This represents the data generated by the nth sensor during the second time step in the denoising and reconstruction process. This represents the data generated by the nth sensor at time k during denoising and reconstruction. This indicates that the nth sensor in the denoising reconstruction is at the 1st... Data generated at each time point This represents the denoised signal element at the (k-m+1)th position in the m-th row of the approximate trajectory matrix. Indicates the left boundary of the diagonal element. Indicates the right boundary of the diagonal element. This indicates taking the minimum value. This indicates taking the maximum value; The results feedback subsystem is used to provide training action guidance, evaluate the training process, and generate training reports based on hand posture data, and to display the evaluation results of the training process.

2. The VR-based Parkinson's hand rehabilitation training system according to claim 1, characterized in that, The planning subsystem includes: The basic strength testing module is used to conduct finger strength tests, palm strength tests, and grip strength tests on patients respectively, and obtain the strength test results. The training plan generation module is used to generate initial training plans for several sets of training items with corresponding magnitudes of reaction force based on the strength test results. The self-adjustment module is used to increase or decrease training items and the number of sets corresponding to the training items based on the initial training plan, so as to obtain the patient's target training plan.

3. The VR-based Parkinson's hand rehabilitation training system according to claim 1, characterized in that, The training execution subsystem includes: The VR access module is used to provide a VR handheld device worn by the patient, wherein the VR device has a VR hand posture recognition method for multimodal data fusion, as well as a posture sensor and several hand pressure sensors for sensing hand movement. The VR scene module is used to provide a VR head-mounted device worn by the patient, wherein the VR head-mounted device is used to provide virtual training scenes, training action guidance prompts, training time prompts, and training group and training project execution progress prompts; The training reward module is used to generate training rewards when the patient completes each set of training items. The training rewards include reward visual stimuli and corresponding reward verbal stimuli. The mode selection module is used to select active training mode, passive training mode, and specific movement training mode according to the patient's target training plan for hand rehabilitation training. The training execution module is used to perform hand rehabilitation training based on training rewards after the patient wears a VR handheld device and a VR headset, according to the patient's target training plan and one of the selected active training mode, passive training mode or specific movement training mode, and continuously obtain hand posture data.

4. The VR-based Parkinson's hand rehabilitation training system according to claim 1 or 3, characterized in that, The VR hand pose recognition method based on multimodal data fusion further includes the following steps: S3. Use sensor measurements to generate a dataset to train a multimodal data fusion network, and obtain the trained multimodal data fusion network; S4. Using the trained multimodal data fusion network, feature extraction and prediction are performed on the sensor measurement data generated at each input time point to obtain the hand posture recognition result, thus completing the VR hand posture recognition through multimodal data fusion.

5. The VR-based Parkinson's hand rehabilitation training system according to claim 4, characterized in that, The CNN subnetwork includes a first convolutional layer, a first activation function layer, a second convolutional layer, a second activation function layer, a third convolutional layer, and a third activation function layer connected in sequence. The CNN subnetwork is used to extract basic spatial features from hand pose data and fuse these basic spatial features to obtain spatial fusion features. The LSTM module is used to extract long short-term memory features based on spatial fusion features through input gate, forget gate and output gate; The GRU module is used to optimize the long short-term memory features through reset gates and update gates to obtain spatiotemporal features; The fully connected layer is used for feature classification based on spatiotemporal features. After the feature classification result is activated by the fourth activation function layer, the output layer outputs the hand gesture recognition result.

6. The VR-based Parkinson's hand rehabilitation training system according to claim 4, characterized in that, The result feedback subsystem includes: The training movement guidance module is used to determine whether the patient's movements meet the standards based on hand posture data in a specific movement training mode. It also displays the changes in the specific movement, non-compliance prompts, and compliance prompts in the virtual training scene through the VR scene module, until the hand rehabilitation training in the specific movement training mode is completed. The training process evaluation module is used to score and evaluate the training time and achievement level of each training group and each training item in the complete hand rehabilitation training, and obtain the training process evaluation results. The training report generation module is used to print the evaluation results of the training process, and after the hand rehabilitation training is completed, the evaluation results are displayed in a virtual training scene through the VR scene module.

7. The VR-based Parkinson's hand rehabilitation training system according to claim 1, characterized in that, It also includes an information entry subsystem; the information entry subsystem is used to enter personal basic information for patients undergoing VR-based Parkinson's hand rehabilitation training for the first time; the personal basic information includes user ID, name, age, gender, medical history and current symptoms.

8. The VR-based Parkinson's hand rehabilitation training system according to claim 7, characterized in that, It also includes a data storage subsystem; the data storage subsystem is used to store the patient's basic personal information and the evaluation results of the training process corresponding to each hand rehabilitation training session.

Citation Information

Patent Citations

  • Upper limb rehabilitation training system based on VR and multi-position sensors

    CN109350923A