Virtual reality rehabilitation system oriented to aged cognitive training and based on gesture interaction

Abstract

The invention discloses a virtual reality rehabilitation system oriented to aged cognitive training and based on gesture interaction. The virtual reality rehabilitation system comprises the following steps: constructing a virtual reality rehabilitation scene by using a Unity development platform; a Unity development platform and VR equipment are used for designing and developing cognitive rehabilitation training tasks covering five cognitive domains; capturing gesture hand data by using a camera of VR equipment, and designing gesture interaction based on gesture tracking, recognition and a hand data processing algorithm; gesture interaction, sound interaction and animation interaction design are integrated into a virtual reality rehabilitation task, so that aging-adaptive cognitive training is achieved; and performing analysis based on the behavior data when the subject completes the task and the questionnaire data after the task is completed. The method can be effectively used for cognitive rehabilitation training, the training initiative of a subject is stimulated, and a reference is provided for clinical research of VR cognitive rehabilitation suitable for aging.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality, and specifically to a virtual reality rehabilitation system for aging-friendly cognitive training and gesture interaction-based rehabilitation system. Background Art

[0002] Mild Cognitive Impairment (MCI) has an insidious onset and low recognition of its onset characteristics. If effective rehabilitation can be carried out at the MCI stage, the decline of cognitive function can be effectively slowed down. The cognitive rehabilitation system based on Virtual Reality (VR) is an important way to achieve cognitive intervention for MCI patients. However, existing VR cognitive training has problems such as insufficient multi-sensory coordination, lack of feedback, and insufficient aging-friendly design. In addition, the gesture interaction method has advantages such as multi-sensory coordination and more convenient use for the elderly. Therefore, it is of great significance to propose a multi-sensory, highly interactive, and aging-friendly gesture interaction VR cognitive rehabilitation training system. Summary of the Invention

[0003] Based on the problems existing in the background art, the present application proposes a virtual reality rehabilitation system for aging-friendly cognitive training and gesture interaction-based rehabilitation system.

[0004] To achieve the above object, the present application adopts the following technical solutions to implement:

[0005] In a first aspect, the present application proposes a cognitive rehabilitation training system, including:

[0006] S1. After the scene is loaded, automatically enter the "Gesture Action Training" task and perform gesture action training according to the prompts;

[0007] S2. Click the menu button through gesture interaction, and the scene jumps to enter the "Color Matching" task;

[0008] S3. Click the menu button through gesture interaction, and the scene jumps to enter the "Shape and Size Matching" task;

[0009] S4. Click the menu button through gesture interaction, and the scene jumps to enter the "Puzzle Block Assembly (3×3)" task. After success, enter the next difficulty level "4×4".

[0010] Further, the step S1 includes the following steps:

[0011] S12. The subject presses the button to start the countdown, and under the indication of a series of virtual gesture actions (the outer contours of the prompt gestures are blackened) in front of him / her, the left and right hands respectively complete the corresponding postures;

[0012] S22. Each time a correct action is completed with one hand, a prompt animation and corresponding prompt sound for that action will appear. At the same time, the scores at the corresponding positions on the scoreboard (two scoreboards for the left and right hands respectively) will increase accordingly.

[0013] S23. The task time and score data of the subject will be exported in real time.

[0014] The step S2 includes the following steps:

[0015] S21. The subject presses the button to start the countdown and needs to place the small balls of the corresponding color on the circular track aperture of the corresponding color.

[0016] S22. Each time a small ball is correctly placed in the track aperture of the corresponding color, the scoreboard will count 1 point. If 4 points (i.e., the colors of all four small balls match correctly) are obtained within the specified time (before the countdown ends), it is considered that the training is successfully completed.

[0017] S23. The task time and score data of the subject will be exported in real time.

[0018] The step S3 includes the following steps:

[0019] S31. The subject presses the button to start the countdown and needs to place the items of the corresponding size and shape on the corresponding circular track aperture.

[0020] S32. Each time an object is correctly placed in the track aperture of the corresponding shape and size, the scoreboard will count 1 point. If 4 points (i.e., the shapes and sizes of all four objects match correctly) are obtained within the specified time (before the countdown ends), it is considered that the training is successfully completed.

[0021] S33. The task time and score data of the subject will be exported in real time.

[0022] The step S4 includes the following steps:

[0023] S41. The subject presses the button to start the countdown and needs to use their hand to grab three building blocks in the "to-be-completed area" and assemble them to match the shape of the white building blocks (successful assembled examples) in the "example area".

[0024] S42. If the assembly is the same as the example building blocks on the left within the specified time, it is considered that the level is successfully passed and it will jump to the next level: 4×4 specification.

[0025] S43. The task time and score data of the subject will be exported in real time.

[0026] In the second aspect, the present application proposes gesture interaction, sound interaction, and animation interaction designs for age-friendly cognitive training, including:

[0027] S5. The system abandons the handle interaction in traditional VR projects and changes to gesture interaction. The forms of gesture interaction between the subject and the objects in the scene include pinching, grasping, tapping, pressing, free grasping, etc.;

[0028] S6. Button prompt sounds, countdown prompt sounds, task success / failure prompt sounds, sounds of grasping objects, etc. are set in the training tasks;

[0029] S7. Animations are added to the training tasks to prompt the subject on how to correctly complete the tasks and improve their enthusiasm for completing the tasks.

[0030] Thirdly, the present application proposes a data collection and analysis module, including:

[0031] S8. The subject fills out two questionnaires, namely the "Subject Acceptance Questionnaire" and the "Subject Task Experience Questionnaire";

[0032] S9. Analyze the score and time data generated by the subject during training, as well as the questionnaire information.

[0033] Compared with the prior art, the advantages of the present invention are as follows:

[0034] (1) Training mode of the integration of human-computer interaction technology and multi-sensory collaborative stimulation:

[0035] By communicating with clinicians and combining the patient's own preferences, human-computer interaction technology is used in the system, covering multi-sensory collaborative stimulation such as gestures, sounds, and animations, to design and construct an immersive VR scene and interaction method suitable for the elderly. The training mode can be adjusted according to the specific situation and training feedback of the patient, more effectively meeting the needs of different patients.

[0036] (2) Design of cognitive training tasks for the elderly suitable for gesture interaction:

[0037] The system breaks away from the handle interaction in traditional VR projects and changes to gesture interaction. Patients can directly interact with the objects in the scene using gestures. Using gesture tracking and interaction technology, five cognitive training tasks, namely "Gesture Action Training", "Color Matching", "Shape and Size Matching", "Intelligent Building Blocks", and "Supermarket Shopping", are designed and implemented, covering five major cognitive domains, breaking through the limitations of the traditional VR cognitive training mode, which is single, has poor interactivity, and rarely considers the suitability for the elderly.

[0038] (3) The task design comprehensively covers five major cognitive domains, promoting the early screening and rehabilitation of MCI diseases:

[0039] The system can be used to evaluate the cognitive function of MCI patients, including aspects such as memory, attention, and executive function. Through Quest3 gesture interaction, the system can provide a series of cognitive tasks and games to help patients train their cognitive function and delay the progression of the disease.

[0040] (4) Innovation in the Clinical Research of VR Cognitive Rehabilitation:

[0041] Through clinical verification and data analysis, the project can be effectively used for cognitive rehabilitation training, stimulate the initiative of the subjects in training, and provide a reference for the clinical research of age-friendly VR cognitive rehabilitation. Description of the Drawings

[0042] Figure 1 It is the overall architecture diagram of the present invention;

[0043] Figure 2 It is the gesture action training task (long shot) diagram of the present invention;

[0044] Figure 3 It is the gesture action training task (close shot) diagram of the present invention;

[0045] Figure 4 It is the color matching task (long shot) diagram of the present invention;

[0046] Figure 5 It is the color matching task (close shot) diagram of the present invention;

[0047] Figure 6 It is the shape and size matching task (long shot) diagram of the present invention;

[0048] Figure 7 It is the shape and size matching task (close shot) diagram of the present invention;

[0049] Figure 8 It is the diagram of the intelligent building block assembly (loop-playing video window with large screen countdown + operation tips) of the present invention;

[0050] Figure 9 It is the diagram of the intelligent building block assembly (difficulty level one) of the present invention;

[0051] Figure 10 It is the diagram of the intelligent building block assembly (difficulty level two) of the present invention;

[0052] Figure 11 It is the data collection and analysis flow chart of the present invention;

[0053] Figure 12 It is the radar chart of the scores of the five major cognitive domains of some subjects of the present invention. Detailed Description of the Invention

[0054] The present invention will be further described in detail below in conjunction with the drawings and specific embodiments, but the protection scope of the present invention is not limited thereto;

[0055] Regarding the problems of weakened hand flexibility, slight hand tremors, and weakened grip strength in MCI patients, this training task is based on the gesture interaction Interaction SDK officially provided by Meta: the gesture pose example scenario in com.meta.xr.sdk.interaction.ovr.samples. It further completes the training of the gesture actions of MCI patients, aiming to help patients improve the flexibility and strength of their hand muscles.

[0056] As Figure 2 and 3 shown, under the guidance of a series of virtual gesture actions (the outer contours of the prompt gestures are blackened) in front of the subject, the left and right hands respectively complete the corresponding postures. Each time a correct action is completed with one hand, a prompt animation and corresponding prompt sound for that action will appear, and at the same time, the scores at the corresponding positions on the scoreboard (two scoreboards for the left and right hands respectively) will increase accordingly.

[0057] When participating in the training, the subject first pats the green button on the left side of the table in front of them. After the countdown numbers appear directly in front, they can start the gesture action training.

[0058] During the process of the gesture action training task, the following are designed:

[0059] (1) A small video window that continuously loops on the left side of the table, where the subject can learn how to perform hand posture exercises.

[0060] (2) Countdown system:

[0061] 1) Pat the button to start the countdown;

[0062] 2) When the countdown ends (reaches 0), the interaction stops and the scoring stops.

[0063] (3) Scoring system:

[0064] 1) Score by different gesture postures;

[0065] 2) One correct action scores 1 point;

[0066] 3) Separate scoreboards for the left and right hands are implemented.

[0067] (4) Sound prompt interaction:

[0068] 1) The prompt sound when the subject correctly makes the corresponding gesture;

[0069] 2) The sound prompt when the subject pats / releases the button;

[0070] 3) The sound of the countdown;

[0071] 4) The sound prompt for successfully completing the training task;

[0072] 5) Sound prompt for failure to complete the training task.

[0073] In addition, during the gesture training process, if the subject feels any discomfort and wants to terminate the task, they can press the red emergency button, which will directly exit the training task.

[0074] As shown in Figures 4 and Figure 5 When the subject participates in the "color matching task", they first press the green button on the left side of the table, and a large screen will appear in front to start the countdown. Initially, four small balls of blue, red, green, and yellow (from left to right) will be placed in the automatic adsorption slots in front of the subject. And there are four-color light rings of blue, red, green, and yellow (from inside to outside) on the corresponding four circular tracks. The subject needs to place the small balls of the corresponding color on the circular track light rings of the corresponding color.

[0075] During the process of placing the small balls onto the light rings, the following are designed:

[0076] (1) Countdown system:

[0077] 1) Press the button to start the countdown;

[0078] 2) When the countdown ends (reaches 0), the interaction stops and the scoring stops.

[0079] (2) Distance detection + automatic adsorption:

[0080] 1) On the track light ring: When the subject grabs the small ball with their hand and is within a certain distance range from the track light ring, this distance will be detected and the small ball will be automatically adsorbed onto the light ring of the track.

[0081] 2) On the long strip slot (in front): When the subject grabs the small ball with their hand and is at a certain distance from the long strip slot in front, the small ball will be automatically adsorbed onto the slot, and the length of the slot will change according to the number of small balls placed.

[0082] (3) Scoring board:

[0083] Each time a small ball is correctly placed into the corresponding color track light ring, the scoring board will count 1 point. If 4 points (i.e., the color matching of all four small balls is correct) are obtained within the specified time (before the countdown ends), it is considered that the training is successfully completed.

[0084] (4) Sound prompt interaction:

[0085] 1) Sound prompt when the subject grabs / puts down the small ball;

[0086] 2) Sound prompt of the second hand during the countdown;

[0087] 3) Sound prompt when the subject presses / releases the button;

[0088] 4) Sound prompt for automatic adsorption of small balls;

[0089] 5) Sound prompt for successful completion of training tasks;

[0090] 6) Sound prompt for failure to complete training tasks within the specified time.

[0091] In addition, during the training process of the color matching task, if the subject feels any discomfort and wants to terminate, they can press the red emergency button, and it will directly exit the training task.

[0092] As Figure 6 and Figure 7 shown, when the subject participates in the "Shape and Size Matching Task", they first press the green button on the left side of the table, and a large screen will appear in front to start the countdown.

[0093] At the initial state, the automatic adsorption slots in front of the subject will place four gesture-interactive items: a large sphere, a large cube, a small cube, and a small sphere (from left to right). And there are four corresponding light circles on the four circular tracks: a large circle, a large square, a small square, and a small circle (from the inside to the outside). The subject needs to place the items of corresponding size and shape on the corresponding circular track light circles.

[0094] During the process of placing the small ball onto the light circle, the following are designed:

[0095] (1) Countdown system:

[0096] 1) Press the button to start the countdown;

[0097] 2) When the countdown ends (reaches 0), the interaction stops and the scoring stops.

[0098] (2) Distance detection + automatic adsorption:

[0099] 1) On the track light circle: When the subject grabs the small ball (or cube) with their hand and is within a certain distance range from the track light circle, this distance will be detected and the small ball (or cube) will be automatically adsorbed onto the track light circle.

[0100] 2) On the long strip slot (in front): When the subject grabs the small ball (or cube) with their hand and is at a certain distance from the long strip slot in front, the small ball (or cube) will be automatically adsorbed onto the slot, and the length of the slot will change according to the number of placed objects.

[0101] (3) Scoring board:

[0102] Each time a correct object is placed into the corresponding shape and size track light circle, the scoring board will count 1 point. After obtaining 4 points (i.e., the shape and size matching of all four objects is correct) within the specified time (before the countdown ends), it is regarded as the successful completion of the training.

[0103] (4) Voice prompt interaction:

[0104] 1) Voice prompt when the subject grabs / drops the ball (or cube);

[0105] 2) Voice prompt of the second hand for countdown;

[0106] 3) Voice prompt when the subject presses / releases the button;

[0107] 4) Voice prompt for the automatic adsorption of the ball (or cube);

[0108] 5) Voice prompt for successfully completing the training task;

[0109] 6) Voice prompt for not completing the training task within the specified time.

[0110] In addition, during the training of the shape and size matching task, if the subject feels any discomfort and wants to terminate, they can press the red emergency button, and it will directly exit the training task.

[0111] Such as Figure 8 and 9 As shown, when the subject is assembling building blocks, they first press the green button on the left side of the table, and a large screen will appear in the front to start the countdown. The subject needs to compare the shape of the white building blocks in the "example area" (the successfully assembled examples) and use their hands to grab the three building blocks in the "to-be-completed area" for assembly. When the assembled shape is the same as that in the "example area", it is considered that the task is successfully completed.

[0112] During the assembly process, the following are designed:

[0113] (1) A small video window that continuously loops on the left side of the table, through which the subject can learn how to use free gestures to grab building blocks.

[0114] (2) Countdown system:

[0115] 1) Press the white button to start the countdown;

[0116] 2) When the countdown ends (reaches 0), the interaction stops and the scoring stops.

[0117] (3) Distance detection + automatic adsorption:

[0118] 1) Distance detection: When the subject interacts with gestures to grab two building blocks and makes them approach each other, within a certain correct distance (compared with the target building block), it will detect that they enter this correct distance and accumulate scores on the scoreboard.

[0119] 2) Automatic adsorption: After the two building blocks enter this correct distance, they will automatically adsorb into the same shape as the building blocks in the "example area", which greatly improves the operation convenience of the subject.

[0120] (4) Voice prompt interaction:

[0121] 1) Voice prompt when the subject grabs / drops the building block;

[0122] 2) Second hand voice prompt for the countdown;

[0123] 3) Voice prompt when the subject presses / releases the button;

[0124] 4) Voice prompt for automatic adsorption between two building blocks;

[0125] 5) Voice prompt for successfully completing the training task;

[0126] 6) Voice prompt for not completing the training task within the specified time.

[0127] If it is consistent with the example building block on the left within the specified time, it is considered that the level has been successfully passed and jumps to the next level: 4×4 specification (such as Figure 10 ). In addition, during the process of assembling the building block puzzle, if the subject feels unwell and wants to terminate, they can press the red emergency button, and it will directly exit the training task.

[0128] In order to evaluate the usefulness and acceptability of the cognitive rehabilitation training system, as well as the subject's experience of the training task, an acceptance questionnaire and a task experience questionnaire were designed, which respectively included 11 acceptance question items and 7 task experience question items, and each questionnaire also included the collection of the subject's basic information (as shown in Table 1 and Table 2). Table 1 is the subject acceptance questionnaire of the present invention; Table 2 is the subject task experience questionnaire of the present invention.

[0129] Table 1:

[0130]

[0131]

[0132] Table 2:

[0133]

[0134] In these two questionnaires, the score for each question ranges from 1 to 5, corresponding to from completely disagree to strongly agree respectively. 1 means completely not, and 5 means very much so. If the score ≥ 4, it is considered that the subject has a supportive attitude towards the question, while if the score ≤ 2, it is considered that the subject has a negative attitude towards the question.

[0135] Such as Figure 11, which is the process of subject data collection and analysis. Among them, the order in which the subjects participated in the cognitive training tasks was: "Task 1: Gesture movement training", "Task 2: Color matching", "Task 3: Shape and size matching", "Task 4 Difficulty 1: Puzzle block assembly (3×3)", and "Task 4 Difficulty 2: Puzzle block assembly (4×4)".

[0136] By analyzing the training time and task score data collected from Task 1 to Task 4, such as Figure 12 , which is the radar chart of the scores of five cognitive domains of some subjects.

[0137] It should be noted that the above embodiments are not used to limit the protection scope of the present invention, and equivalent transformations or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention.

Claims

1. A virtual reality rehabilitation system for aging-friendly cognitive training and gesture-based interaction, characterized in that: include: Use the Unity development platform to build virtual reality rehabilitation scenes; Design and develop cognitive rehabilitation training tasks using the Unity development platform and VR devices; Use the camera of the VR device to capture hand gesture data, and design gesture interaction based on gesture tracking, recognition and hand data processing algorithms; By integrating gesture interaction, sound interaction, and animation interaction design into virtual reality rehabilitation tasks, we can achieve cognitive training suitable for the elderly. The analysis is based on the behavioral data of the subjects when completing the task and the questionnaire data after completing the task.

2. The virtual reality rehabilitation system for aging-friendly cognitive training and gesture interaction according to claim 1, characterized in that: Virtual reality training scenarios, virtual reality rehabilitation training tasks include four major tasks for aging-friendly cognitive rehabilitation training: "gesture training", "color matching", "shape and size matching", and "building blocks", including the following steps: S1. After the scene is loaded, it will automatically enter the "gesture training" task and perform gesture training according to the prompts; S2. Click the menu button through gesture interaction, and the scene jumps to the "color matching" task; S3. Click the menu button through gesture interaction, and the scene jumps to the "Shape and Size Matching" task; S4. Click the menu button through gesture interaction, and the scene will jump to the "Puzzle Block Building (3×3)" task. After success, it will enter the next difficulty "4×4".

3. The virtual reality rehabilitation system for aging-friendly cognitive training and gesture interaction according to claim 2, characterized in that: For aging-friendly cognitive training and virtual reality rehabilitation tasks based on gesture interaction, step S1 includes the following steps: S11. The subject presses the button to start the countdown, and then performs corresponding gestures with his left and right hands respectively under the guidance of a series of virtual gestures in front of him (the outline of the gestures is marked in black); S12. Each time each hand correctly completes an action, a prompt animation and a corresponding prompt sound will appear, and the score of the corresponding position on the scoreboard (two scoreboards for the left and right hands) will increase accordingly; S13. The task time and score data of the subjects will be exported in real time.

4. The virtual reality rehabilitation system for aging-friendly cognitive training and gesture interaction according to claim 2, characterized in that: Step S2 includes the following steps: S21. The subject presses the button to start the countdown, and needs to place the ball of the corresponding color on the center track aperture of the corresponding color; S22. Each time a ball is correctly placed in the corresponding color track aperture on the scoreboard, 1 point is scored. If 4 points are scored within the specified time (before the countdown ends) (i.e., the colors of the four balls are correctly matched), the training is considered successfully completed; S23. The task time and score data of the subjects will be exported in real time.

5. The virtual reality rehabilitation system for aging-friendly cognitive training and gesture interaction according to claim 2, characterized in that: Step S3 includes the following steps: S31. The subject presses the button to start the countdown, and needs to place objects of corresponding size and shape on the corresponding circular track aperture; S32. Each time an object is correctly placed in the corresponding shape and size of the track aperture, 1 point is counted on the scoreboard. The training is considered successfully completed when 4 points are obtained within the specified time (before the countdown ends) (i.e., the shapes and sizes of the four objects are correctly matched); S33. The task time and score data of the subjects will be exported in real time.

6. The virtual reality rehabilitation system for aging-friendly cognitive training and gesture interaction according to claim 2, characterized in that: Step S4 includes the following steps: S41. The subject presses the button to start the countdown. The subject must compare the shape of the white building blocks in the "example area" (the completed successful example) and use his hands to grab the three building blocks in the "to be completed area" to assemble them until the shape of the "example area" is consistent; S42. If the blocks are placed in the same way as the example blocks on the left within the specified time, the level is considered to have been successfully passed and the player will jump to the next level: 4×4 specification. S43. The task time and score data of the subjects will be exported in real time.

7. The virtual reality rehabilitation system for aging-friendly cognitive training and gesture interaction according to claim 1, characterized in that: Gesture interaction, voice interaction, and animation interaction design, including: S5. The system is set to gesture interaction. The forms of gesture interaction between the subject and the objects in the scene include pinching, grasping, tapping, pressing, and free grasping. S6. In the training task, the button prompt sound, countdown prompt sound, task success / failure prompt sound, and object grabbing sound are set. S7. Add animations to training tasks to remind subjects how to complete the tasks correctly and increase their motivation to complete the tasks.

8. The virtual reality rehabilitation system for aging-friendly cognitive training and gesture interaction according to claim 1, characterized in that: The system also includes a data collection and analysis module, which is used to record the questionnaire data of the subjects and the data in the task training, and perform statistical analysis on the data to evaluate the training effect, and is characterized by including: S8. Subjects fill in two questionnaires: "Subject Acceptance Questionnaire" and "Subject Task Experience Questionnaire"; S9. Analyze the scores and time data generated by the subjects during training, as well as the questionnaire information.

Images