A method and system for somatosensory interactive virtual rehabilitation training

Abstract

The present invention discloses a method and system for somatosensory interactive virtual rehabilitation training, which belongs to the field of artificial intelligence technology and is applied to rehabilitation training systems; collects the action information and voice information of the user after following the NPC-guided behavior; simulates the user's motion posture according to the action information; calculates the score of the user's rehabilitation action according to the pre-set requirements and standards of the rehabilitation action and the user's action information and voice information, and provides corresponding voice and graphic feedback to the user according to the score of the user's rehabilitation action; analyzes the user's rehabilitation process and effect according to the data of the user's motion posture simulation and the score of the user's rehabilitation action, and outputs a rehabilitation report. Advantages: It can use virtual reality technology and human motion recognition technology to provide users with an immersive rehabilitation training environment, and can also monitor and evaluate the user's rehabilitation actions in real time, give feedback and suggestions, and increase the user's rehabilitation efficiency and experience.

Description

Technical Field

[0001] The invention relates to a somatosensory-based interactive virtual rehabilitation training method and system, belonging to the technical field of artificial intelligence. Background Art

[0002] Limb incoordination refers to the abnormality of the smoothness, speed, range, strength and duration of limb movement, resulting in clumsy, inflexible, inaccurate and uneven limb movements. The causes of limb incoordination may be related to damage or dysfunction of the central nervous system, vestibular organs, deep sensation, vision and other systems. People with limb incoordination include patients with stroke, cerebral palsy, ataxia, dyscoliasis, coordination dysfunction, etc. Their limb motor functions are affected to varying degrees, seriously affecting their daily life and work ability. The recovery of limb motor function requires long-term, high-intensity and diversified training to promote the adaptation and reorganization of the nervous system and improve the sensitivity and coordination of the limbs.

[0003] At present, there are mainly the following methods for limb rehabilitation training:

[0004] (1) In traditional physical therapy, professional physical therapists provide one-on-one guidance and assistance to patients, such as limb flexion and extension, fist clenching, thumb-to-finger movements, etc. The advantage of this method is that a personalized training plan can be formulated according to the patient's specific situation, but it also has some disadvantages, such as low training efficiency, difficulty in controlling training intensity, boring training process, and difficulty in evaluating training results.

[0005] (2) Mechanical rehabilitation equipment, which consists of a mechanical structure and a motor drive, acts on the patient's limbs through external force to make them perform passive or active movements, such as bending, straightening, and rotation. The advantage of this method is that it can provide stable and controllable training intensity, but it also has some disadvantages, such as high equipment cost, complex structure, difficult maintenance, single training method, and difficulty in ensuring training results.

[0006] (3) Virtual reality technology, through computer-generated virtual environments and interactive devices, simulates real physical movement scenes, such as picking up, grasping, throwing, etc., to stimulate patients' interest and motivation and enhance the fun and immersion of training. The advantage of this method is that it can provide rich and diverse training content, but it also has some disadvantages, such as high equipment cost, high technical requirements, difficulty in quantifying training effects, and difficulty in adjusting training intensity.

[0007] In summary, the existing limb rehabilitation training methods have certain limitations and cannot meet the limb rehabilitation training needs of stroke patients. Therefore, there is an urgent need for a new limb rehabilitation training system that can overcome the shortcomings of existing technologies and provide an effective, convenient, intelligent and personalized limb rehabilitation training method. Summary of the invention

[0008] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a somatosensory-based interactive virtual rehabilitation training method and system.

[0009] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0010] On the one hand, the present invention provides a somatosensory-based interactive virtual rehabilitation training method, which is applied to a rehabilitation training system;

[0011] Collect the user's action information and voice information after following the NPC's guidance actions;

[0012] Performing user motion posture simulation according to the motion information;

[0013] Calculate the rehabilitation action score of the user according to the pre-set rehabilitation action requirements and standards and the action information and voice information of the user, and provide the user with corresponding voice and graphic feedback according to the rehabilitation action score of the user;

[0014] The user's rehabilitation progress and effect are analyzed based on the user's motion posture simulation data and the scores of the user's rehabilitation movements, and a rehabilitation report is output.

[0015] Furthermore, the rehabilitation movements of the NPC-guided behavior adopt Ba Duan Jin movements.

[0016] Furthermore, the collecting of the user's motion information after following the NPC's guided action and simulating the user's motion posture includes:

[0017] The distance and orientation changes of any two joint points among a plurality of skeletal joint points of the user are collected by a Kinect device, a human joint motion trajectory is determined according to the changes, and the user's motion posture is simulated according to the human joint motion trajectory; when determining the human joint motion trajectory, the human joint motion trajectory is smoothed, including: taking the coordinate value of the active joint at the current moment and the average value of the joint coordinates of the previous N-1 sampling cycles as the joint position at the current moment, and progressively moving the acquired coordinate position forward in a time sequence with N as a step length;

[0018] The user's voice information is received through a Kinect device, and the source direction and distance of the voice information are converted into angles and delays through beamforming technology. Based on the angles and delays, the voice information is segmented into a series of phoneme sequences, and then the phoneme sequences are mapped into word sequences through a statistical model or a neural network model, and then the word sequences are converted into sentences or commands for interacting with the rehabilitation training system through a language model or a semantic analysis model;

[0019] Analyze the user's emotions and needs based on the statements or commands;

[0020] Furthermore, the user's rehabilitation progress and effect analysis is performed based on the user's motion posture simulation data and the user's rehabilitation action scores, and a rehabilitation report is output, including:

[0021] Use UDP protocol to obtain the user's motion posture simulation data and the user's rehabilitation action score;

[0022] Use deep learning and pattern recognition technology to identify key rehabilitation indicators from the user's motion posture simulation data and the user's rehabilitation action scores;

[0023] generating a rehabilitation report including personalized suggestions based on the key rehabilitation indicators;

[0024] A deep learning model is used to determine optimal rehabilitation recommendations based on the rehabilitation report.

[0025] Furthermore, the requirements and standards of the rehabilitation movements are self-adjusted according to the rehabilitation progress and individual differences of each user;

[0026] The self-adjustment process includes: adding a BOXcollider and a kinect camera to the rehabilitation training system to detect whether the character's movements are standard. When the character touches an object with a boxcollider, the score will be increased by one and the process will be carried out in depth. Once the score is less than the set value, the current rehabilitation movement will be reset.

[0027] In a second aspect, the present invention further provides a somatosensory-based interactive virtual rehabilitation training system, comprising:

[0028] A virtual rehabilitation scene construction module, used to build a virtual rehabilitation scene through Unity, wherein the virtual rehabilitation scene includes a simulated rehabilitation treatment room and multiple virtual objects and tasks related to rehabilitation actions;

[0029] A human motion recognition module is used to recognize the user's body movements in the virtual rehabilitation scene by connecting to a Kinect device, and transmit the user's real motion information to the main control module;

[0030] The rehabilitation movement design module designs rehabilitation movements based on the Eight-Section Brocade movements and transmits the requirements and standards of the rehabilitation movements to the main control module;

[0031] The voice interaction module is used for voice interaction between the user and the system, receives the user's voice commands and feedback, and transmits the voice information to the main control module;

[0032] The NPC guidance module includes a virtual NPC character used to guide the user to perform rehabilitation actions. According to the instructions of the main control module, the virtual NPC character is used to guide the user how to correctly complete the rehabilitation actions through voice and action demonstration;

[0033] The main control module is used to control the operation of the entire system, judge the user's rehabilitation effect according to the user's motion information, voice information and rehabilitation motion requirements and standards, control the changes of the virtual rehabilitation scene, control the behavior of the NPC guidance module, control the display of the motion posture simulation module, control the recording of the rehabilitation motion data recording module, and control the analysis of the data analysis module.

[0034] Furthermore, it also includes: a motion posture simulation module, which is controlled by the main control module and simulates and displays the motion posture corresponding to the user's limbs, so that the user can observe whether his or her movements are correct;

[0035] The motion posture simulation module includes a display screen and a graphics processor, wherein the display screen is used to display a three-dimensional model of the user's limbs, and the graphics processor is used to generate a three-dimensional model of the user's limbs according to the user's motion information;

[0036] It also includes: a rehabilitation evaluation module, which is used to receive the user's motion information, voice information and rehabilitation motion requirements and standards transmitted by the main control module, and judge the user's rehabilitation effect;

[0037] The rehabilitation assessment module includes a scoring algorithm and a feedback mechanism. The scoring algorithm is used to calculate the score of the user's rehabilitation action based on the user's action information, voice information and the requirements and standards of the rehabilitation action. The feedback mechanism is used to provide the user with corresponding voice and graphic feedback based on the score of the user's rehabilitation action.

[0038] Furthermore, it also includes: a rehabilitation action data recording module, which is used to record the rehabilitation action data completed by the user in the virtual rehabilitation scene, so as to conduct long-term tracking and analysis of the rehabilitation effect;

[0039] The rehabilitation action data recording module includes a data storage device and a data transmitter, wherein the data storage device is used to store the user's rehabilitation action data, and the data transmitter is used to transmit the user's rehabilitation action data to the main control module or other devices;

[0040] It also includes: a data analysis module for analyzing rehabilitation movement data, generating rehabilitation reports, and providing personalized rehabilitation suggestions;

[0041] The data analysis module includes a data processor and a data display. The data processor is used to analyze the user's rehabilitation progress and effects based on the user's rehabilitation movement data, and generate a rehabilitation report and rehabilitation suggestions. The data display is used to display the rehabilitation report and rehabilitation suggestions to the user or doctor in the form of charts or text.

[0042] Furthermore, the virtual rehabilitation scene construction module is also used to select or generate virtual objects and tasks suitable for the user according to the user's rehabilitation action requirements and standards, and adjust the appearance and personality of the virtual NPC character;

[0043] The human body motion recognition module is also used to calculate the user's limb flexibility and strength according to the user's limb movements, and transmit the calculation results to the main control module;

[0044] The voice interaction module is also used to analyze the user's emotions and needs based on the user's voice information, and transmit the analysis results to the main control module;

[0045] The NPC guidance module is also used to adjust the voice and actions of the virtual NPC character according to user feedback and ratings.

[0046] Furthermore, the rehabilitation movement design module is also used to automatically or manually adjust the difficulty and sequence of rehabilitation movements according to the user's rehabilitation needs and preferences, so as to improve the user's rehabilitation effect and experience.

[0047] The beneficial effects achieved by the present invention are:

[0048] The present invention is suitable for whole-body limb rehabilitation training for people with all kinds of limb incoordination. The system can use virtual reality technology and human motion recognition technology to provide users with an immersive rehabilitation training environment. At the same time, it can monitor and evaluate the user's rehabilitation movements in real time, give feedback and suggestions, and increase the user's rehabilitation efficiency and experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a schematic diagram of the physical structure of the virtual rehabilitation system of the present invention;

[0050] Figure 2 is a schematic diagram of modules of the virtual rehabilitation system of the present invention;

[0051] Figure 3 is a processing flow chart of the main control module of the virtual rehabilitation system of the present invention;

[0052] Figure 4 It is a processing flow chart of a human motion recognition module and a feedback module of a virtual rehabilitation system of the present invention;

[0053] Figure 5It is a processing flow chart of the virtual human collision prediction and collision detection module of the virtual rehabilitation system of the present invention;.

[0054] Figure 6 It is a processing flow chart of the Kinect module of the virtual rehabilitation system of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0056] Embodiment 1, this embodiment introduces a method for somatosensory interactive virtual rehabilitation training, such as Figure 1 and Figure 2 It is characterized by being applied to a rehabilitation training system;

[0057] Collect the user's action information and voice information after following the NPC's guidance actions;

[0058] Performing user motion posture simulation according to the motion information;

[0059] Calculate the rehabilitation action score of the user according to the pre-set rehabilitation action requirements and standards and the action information and voice information of the user, and provide the user with corresponding voice and graphic feedback according to the rehabilitation action score of the user;

[0060] The user's rehabilitation progress and effect are analyzed based on the user's motion posture simulation data and the scores of the user's rehabilitation movements, and a rehabilitation report is output.

[0061] The rehabilitation movements of the NPC-guided behavior adopt Ba Duan Jin movements.

[0062] The user's motion information after following the NPC's guidance action is collected to simulate the user's motion posture, such as Figure 6 As shown, including:

[0063] The distance and orientation changes of any two joint points among a plurality of skeletal joint points of the user are collected by a Kinect device, a human joint motion trajectory is determined according to the changes, and the user's motion posture is simulated according to the human joint motion trajectory; when determining the human joint motion trajectory, the human joint motion trajectory is smoothed, including: taking the coordinate value of the active joint at the current moment and the average value of the joint coordinates of the previous N-1 sampling cycles as the joint position at the current moment, and progressively moving the acquired coordinate position forward in a time sequence with N as a step length;

[0064] The user's voice information is received through a Kinect device, and the source direction and distance of the voice information are converted into angles and delays through beamforming technology. Based on the angles and delays, the voice information is segmented into a series of phoneme sequences, and then the phoneme sequences are mapped into word sequences through a statistical model or a neural network model, and then the word sequences are converted into sentences or commands for interacting with the rehabilitation training system through a language model or a semantic analysis model;

[0065] According to the statements or commands, the user's emotions and needs are analyzed (the user's emotions and needs can be determined based on preset options for the user to choose from).

[0066] The user's rehabilitation progress and effect analysis is performed based on the user's motion posture simulation data and the user's rehabilitation action scores, and a rehabilitation report is output, including:

[0067] Use UDP protocol to obtain the user's motion posture simulation data and the user's rehabilitation action score;

[0068] Use deep learning and pattern recognition technology to identify key rehabilitation indicators from the user's motion posture simulation data and the user's rehabilitation action scores;

[0069] generating a rehabilitation report including personalized suggestions based on the key rehabilitation indicators;

[0070] A deep learning model is used to determine optimal rehabilitation recommendations based on the rehabilitation report.

[0071] The requirements and standards of the rehabilitation movements are self-adjusted according to the rehabilitation progress and individual differences of each user;

[0072] The self-adjustment process, such as Figure 5 As shown, it includes: adding a boxcollider and a kinect camera to the rehabilitation training system to detect whether the character's movements are standard. When the character touches an object with a boxcollider, the score will increase by one and the process will go on. Once the score is less than the set value, the current rehabilitation action will be restarted.

[0073] Example 2 is based on the same inventive concept as Example 1. Figure 1 and Figure 2 As shown, this embodiment introduces a somatosensory-based interactive virtual rehabilitation system, including:

[0074] Virtual rehabilitation scene building blocks, such as Figure 3As shown, a virtual scene built by the inventor himself based on Unity development includes multiple virtual objects and tasks related to rehabilitation actions, as well as a virtual NPC character; the module is also used to select or generate virtual objects and tasks suitable for the user according to the user's rehabilitation action requirements and standards, and adjust the appearance and personality of the virtual NPC character; the module uses Unity's game engine and scene editor, and the specific principles are as follows:

[0075] Game engine: Unity is a cross-platform game development tool that provides a range of functions, such as graphics rendering, physics simulation, audio processing, user interface, scripting, etc., for creating and running various types of games and applications. Unity's game engine is component-based. It abstracts each object in the game (such as characters, objects, lights, etc.) into a game object (GameObject). Each game object can be attached with multiple components (Component) to define the properties and behaviors of the game object, such as position, rotation, scaling, collider, renderer, animation controller, etc. The relationship between game objects and components can be described by the following formula:

[0076] GameObject=Transform+Component1+Component2+…

[0077] Scene Editor: Unity provides a visual scene editor for creating and arranging game objects in the game, as well as setting the game environment and parameters. The scene editor consists of the following parts:

[0078] Hierarchy View: Used to display and manage all game objects in the scene, as well as the parent-child relationships between them.

[0079] Inspector View: Used to display and edit the properties and components of the selected game object.

[0080] Scene View: Used to view and manipulate game objects in the scene in three-dimensional space, and to set the camera's perspective and position.

[0081] Game View: Used to preview and run the game, as well as view the game's output and debugging information.

[0082] Project View: Used to display and manage all resources in the project, such as models, textures, sound effects, scripts, etc.

[0083] Console View: Used to display and filter game logs, warnings, and error messages.

[0084] The human motion recognition module is connected to the Kinect device and is used to recognize the user's whole body movements in the virtual rehabilitation scene and transmit the motion information to the main control module. The module is also used to calculate the user's limb flexibility and strength based on the user's limb movements and transmit the calculation results to the main control module. The module uses the depth detection, skeleton tracking and motion trajectory generation functions of the Kinect device, such as Figure 6 As shown, the specific principles are as follows:

[0085] Depth detection: The Kinect device includes an infrared projector and an infrared camera. The infrared projector projects a near-infrared spectrum into the scene, forming random reflective spots, called speckles. This principle is based on the interference and diffraction phenomena of light. When a beam of monochromatic light (such as near-infrared light) is irradiated onto a rough surface (such as a human body or an object), it will be reflected by the irregular structure of the surface into multiple sub-waves. These sub-waves will interfere with each other to form light and dark stripes or spots, which are speckles. The shape and size of the speckle depends on factors such as the wavelength of the light source, the angle of illumination, the roughness of the reflective surface, and the distance. The image of the speckle can be described by the following formula:

[0086]

[0087] Where I(x,y) is the brightness of the speckle image, I0 is the intensity of the light source, λ is the wavelength of the light source, z is the average distance of the reflecting surface, x and y are the coordinates of the speckle image, and φ is the initial phase.

[0088] Skeleton tracking: The infrared camera captures the speckle image and calculates the phase difference of each pixel through the phase shift method to obtain the distance from each pixel to the camera plane, that is, the depth value. This principle is based on the phase change of light. When a beam of light enters from one medium to another, its phase changes, and this change is related to the refractive index of the two media and the incident angle of the light. If we know the initial phase and final phase of the light, we can calculate the phase difference of the light and thus deduce the propagation path and distance of the light. In order to obtain the phase difference of light, the Kinect device uses the phase shift method. Its basic idea is to illuminate the scene in turn with four light sources of different phases, and then use the depth camera to capture four speckle images respectively. According to the pixel values ​​of these four images, the phase difference of each pixel can be calculated, and then the depth value of each pixel can be obtained. The formula of the phase difference method is as follows:

[0089]

[0090] Among them, φ is the phase difference, I1, I2, I3, I4 are the pixel values ​​of the speckle images of four different phases received by the depth camera. The calculation formula of the depth value is as follows:

[0091]

[0092] Among them, d is the depth value, f is the focal length of the depth camera, λ is the wavelength of the infrared projector, and Δφ is the change in phase difference. Through the depth value, the user can be separated from the background and the different parts of the user, such as the head, hands, feet, etc., can be identified. Then, through machine learning methods, the user's joint positions, such as shoulders, elbows, knees, etc., are determined based on the user's part information. Finally, by connecting the joint positions, the user's skeleton diagram is constructed. Kinect can track the skeleton nodes of 6 users at the same time, and each user has 25 skeleton nodes.

[0093] Based on these twenty-five skeletal nodes, the distance and orientation changes between any two joint points can be calculated to obtain the human joint motion trajectory.

[0094] Kinect can obtain scene depth information and further calculate the spatial distance between the person and the camera. Assuming the distance from the human body to the Kinect device is d, the following formula can be used to obtain:

[0095] d = K tan(H d K +L)-O(4)

[0096] Where d K is the object depth value, obtained by the Kinect device, k = 12.36 cm, H = 3.5-10-4 rad, L = 1.18 rad, O = 3.7 cm.

[0097] Motion trajectory generation: transform the depth image coordinates (x K ,y K , z k ) is converted into actual coordinates (x r ,y r , z r ), the conversion formula is:

[0098]

[0099] Where F = 0.0021, D′ = -10, and the resolution of Kinect is w×h = 680×480. Thus, using equations (4) and (5), we can obtain the spatial coordinates M(x r1 ,y r1 ,z r1 ) and N(x r2 ,y r2 ,zr2 ), the distance between the two joint points is:

[0100]

[0101] Due to a series of factors such as Kinect's hardware error or jitter, the relative position of the skeletal joints changes greatly in a short period of time, and there are outliers in the data. To address this problem, when the system generates the motion trajectory of the skeletal nodes, it is necessary to filter and delete such outliers and smooth the motion trajectory. In order to achieve the effect of noise reduction, the coordinate value of the active joint at the current moment and the average value of the joint coordinates of the previous N-1 sampling cycles are used as the joint position at the current moment, and the acquired coordinate position is progressively moved forward in a time sequence with N steps. This method can make the human joint activity trajectory acquired and displayed by the system smoother and more beautiful, giving rehabilitation patients a better visual perception to enhance their confidence in rehabilitation.

[0102] The rehabilitation action design module designs rehabilitation actions based on the traditional Ba Duan Jin actions and transmits the requirements and standards of the rehabilitation actions to the main control module. The module is also used to automatically or manually adjust the difficulty and sequence of rehabilitation actions according to the user's rehabilitation needs and preferences to improve the user's rehabilitation effect and experience. The module designs actions based on the movement principle of Ba Duan Jin and strives to achieve a similar rehabilitation effect in this rehabilitation system. The specific principles are as follows:

[0103] Ba Duan Jin is a traditional Qigong fitness method. It consists of eight simple movements, each of which has certain effects, such as regulating Qi and blood, relaxing muscles and tendons, and strengthening the body. The movements of Ba Duan Jin can enhance cardiopulmonary function, regulate the nervous system, dredge meridians, enhance muscle strength, regulate visceral function, correct bad posture, improve vision, etc., and have certain effects on the prevention and treatment of various diseases. The movements of Ba Duan Jin require the body to be upright, chest to sink the Qi, and deep breathing to make the breathing deep and long, increase lung capacity, relax the nerves, eliminate fatigue, and enhance mental vitality. Ba Duan Jin can promote the blood circulation of the internal organs and enhance the metabolic function of the internal organs through massage and traction of the internal organs. It has certain effects on regulating the functions of the spleen, stomach, liver, gallbladder, kidneys, bladder, etc., and has certain effects on the prevention and treatment of indigestion, gastrointestinal diseases, hepatitis, nephritis, etc. Ba Duan Jin can enhance the stability and flexibility of the waist and back through stretching the muscles and bones of the waist and back. It has a certain effect on correcting bad postures such as shoulder adduction and round back, and has a certain effect on preventing and treating scoliosis, cervical spondylosis, lumbar spondylosis, etc. Ba Duan Jin can increase the range of eye movement and enhance the strength of the eye muscles through the rotation and glaring of the eyeballs. It has a certain effect on preventing and treating myopia, hyperopia, presbyopia, etc. The movements of Ba Duan Jin are as follows:

[0104] First posture: Hold your hands up to the sky to regulate the triple burner. Raise your hands from both sides, palms facing up, fingers straightened, arms stretched upward as far as possible, inhale at the same time, then lower your hands from the top, palms facing down, arms hanging naturally, exhale at the same time. Repeat 8 times. This posture can regulate the flow of qi and blood in the triple burner meridian, and promote the functions of the digestive, respiratory and urinary systems.

[0105] Second posture: Open the bow with both hands, like shooting an eagle. Raise both hands from both sides, palms facing up, stretch both arms horizontally, and inhale at the same time. Then make a fist with the left hand, and stretch the palm of the right hand to the left and back, like shooting an arrow, and exhale at the same time. Then make a fist with the right hand, and stretch the palm of the left hand to the right and back, like shooting an arrow, and inhale at the same time. Repeat 4 times on each side. This posture can regulate the circulation of qi and blood in the hand Taiyin lung meridian and the hand Yangming large intestine meridian, and enhance the functions of the respiratory system and immune system.

[0106] The third posture: To regulate the spleen and stomach, you must raise your hands. Raise both hands from both sides, palms facing up, arms stretched out, inhale at the same time, then turn your left palm up and stretch it upwards, turn your right palm down and hang it down, exhale at the same time, then turn your right palm up and stretch it upwards, turn your left palm down and hang it down, inhale at the same time. Repeat 4 times on each side. This posture can regulate the Qi and blood circulation of the Taiyin spleen meridian and the Yangming stomach meridian, and promote the functions of the digestive system and blood system.

[0107] Fourth posture: Look back to prevent five kinds of fatigue and seven kinds of injuries. Raise both hands from both sides, palms facing up, arms stretched out, inhale at the same time, palms facing each other, turn backwards, exhale at the same time, return to the original position, and inhale at the same time. Repeat 4 times. This posture can regulate the blood circulation of the hand Shaoyin heart meridian and hand Taiyang small intestine meridian, and promote the function of the heart and nervous system.

[0108] Fifth posture: Shake your head and tail to relieve internal heat. Raise your hands from both sides, palms up, arms outstretched, inhale at the same time, then palms down, press down, bend your body to the left, turn your head to the right, exhale at the same time, return to the original position, inhale at the same time, then palms down, press down, bend your body to the right, turn your head to the left, exhale at the same time, return to the original position, inhale at the same time. Repeat 4 times on each side. This posture can regulate the flow of qi and blood in the foot Shaoyin kidney meridian and foot Taiyang bladder meridian, and promote the functions of the urinary system and endocrine system.

[0109] Sixth posture: Hold the feet with both hands to strengthen the kidneys and waist. Raise both hands from both sides, palms facing up, arms stretched out, inhale at the same time, then put the palms down, stretch forward and downward, bend the body forward, try to touch the toes with your hands, exhale at the same time, then return to the original position, and inhale at the same time. Repeat 8 times. This posture can regulate the circulation of qi and blood in the liver meridian of foot jueyin and gallbladder meridian of foot yangming, and promote the functions of liver and gallbladder system and nervous system.

[0110] Seventh posture: Clenching fists and glaring to increase energy. Raise both hands from both sides, palms facing up, arms stretched out, inhale at the same time, then clench fists, palms facing up, close to the chest, lean back slightly, stare upward, exhale at the same time, return to the original position, and inhale at the same time. Repeat 8 times. This posture can regulate the circulation of qi and blood in the hand Jueyin pericardium meridian and hand Shaoyang triple burner meridian, and promote the function of the heart and circulatory system.

[0111] Eighth posture: Shake your back seven times to get rid of all diseases. Raise your hands from both sides, palms up, arms outstretched, inhale at the same time, then palms down, stretch back, bend your body backward, try to touch your lower back with your hands, exhale at the same time, return to the original position, and inhale at the same time. Repeat 8 times. This posture can regulate the flow of qi and blood in the Ren and Du meridians, and promote the functions of the spine and nervous system.

[0112] The voice interaction module has a built-in voice function and is used for voice interaction between the user and the system. It receives the user's voice commands and feedback and transmits the voice information to the main control module, such as Figure 3 As shown; this module is also used to analyze the user's emotions and needs according to the user's voice information (specifically, the user can choose according to the pre-set options to determine the user's emotions and needs, such as: plot dialogue: NPC: Is this rehabilitation action good? We have four options: A. Good! Let's continue; B. It's okay, let's keep up the good work. C. I feel that the previous few actions were not done well. D. Oops, I did it all wrong, let's do it again!), and transmit the analysis results to the main control module; this module uses the voice recognition function of the Kinect device. The specific principle is as follows: Voice recognition: The Kinect device includes a four-element linear microphone array for receiving sound signals and locating the direction and distance of the sound source through beamforming technology. This principle is based on the propagation and interference of sound waves. When a sound source makes a sound, it generates a series of pressure waves, which propagate at a certain speed in the air. When they reach the microphone, the microphone converts the sound waves into electrical signals. Due to the different distances and angles between the sound source and the microphone, the time and intensity of the sound waves reaching the microphone will also be different, which produces time difference and amplitude difference. If we know the position of the microphone and the speed of the sound wave, we can calculate the direction and distance of the sound source based on the time difference and amplitude difference. In order to improve the accuracy of sound source positioning and noise resistance, Kinect uses a beamforming technology. Its basic idea is to weight and phase the signals received by the microphone array so that the signals from the target direction are enhanced and the signals from other directions are suppressed, thereby achieving the positioning of the sound source. The formula of beamforming technology is as follows:

[0113]

[0114] Among them, y(t) is the output signal, w i is the weight coefficient, x i is the signal received by the i-th microphone, τ i is the delay time of the signal reaching the i-th microphone, and N is the number of microphones. The direction and distance of the sound source can be converted into angle and delay through beamforming technology, and then the sound signal can be converted into text information through speech recognition algorithm for user interaction with the system. The basic idea of ​​speech recognition algorithm is to divide the sound signal into a series of phonemes (Phoneme), and then map the phoneme sequence into a word sequence through statistical model or neural network model, and then convert the word sequence into a sentence or command through language model or semantic analysis model.

[0115] The NPC guidance module includes a virtual NPC (non-player character) character, which is used to guide the user to perform rehabilitation actions. According to the instructions of the main control module, the module teaches the user how to correctly complete the rehabilitation actions through voice and action demonstration; the module is also used to adjust the voice and action of the virtual NPC character according to the user's feedback and ratings to adapt to the user's rehabilitation level and preferences; the module uses Unity's animation system and voice system. The specific principles are as follows:

[0116] Animation system: Unity provides a powerful animation system for creating and controlling the animation of game objects, such as the character's walking, jumping, attacking, etc. The animation system includes the following parts:

[0117] Animation Clip: Used to store animation data of game objects, such as position, rotation, scale, etc., which can be recorded in the scene editor or created in external tools.

[0118] Animator Controller: Used to manage the animation states and transitions of game objects, as well as animation parameters and levels, which can be edited in the Animator Controller window.

[0119] Animator State Machine: Used to define the relationship between the animation states and transition conditions of a game object. It can be edited in the Animator State Machine window2.

[0120] Animator Parameter: A variable used to control animation states and transitions, such as Boolean values, integer values, floating point values, etc., which can be set in the script.

[0121] Animator Layer: It is used to layer the animation state machine to achieve animation blending of different parts, such as the animation of the upper and lower body, which can be set in the animation controller window.

[0122] Voice system: Unity provides a simple voice system for playing and controlling the voices of game objects, such as character dialogues, narrations, instructions, etc. The voice system consists of the following parts:

[0123] Audio Clip: Used to store the voice data of game objects, such as audio files, volume, pitch, etc., which can be created in external tools.

[0124] Audio Source: Used to play and control the audio of game objects, such as play mode, loop mode, sound effects, etc., which can be edited in the Inspector.

[0125] Audio Listener: Used to receive and process the audio of game objects, such as volume, stereo, reverb, etc., which can be edited in the inspector.

[0126] The main control module is used to control the operation of the entire system. According to the user's action information, voice information and the requirements and standards of rehabilitation actions, it determines the user's rehabilitation effect, controls the changes of the virtual rehabilitation scene, controls the behavior of the NPC guidance module, controls the display of the motion posture simulation module, controls the recording of the rehabilitation action data recording module, and controls the analysis of the data analysis module. This module uses Unity's script system and event system. The specific principles are as follows:

[0127] Script system: Unity provides a flexible script system for writing and executing the logic of game objects, such as user input, game state, game logic, etc. The script system supports multiple programming languages, such as C#, JavaScript, Boo, etc. The script system includes the following parts:

[0128] Script: used to store the logic code of game objects, such as variables, functions, classes, etc., which can be written in external tools.

[0129] Script Component: Used to attach scripts to game objects to implement the logical functions of game objects. It can be edited in the Inspector.

[0130] Script Lifecycle: used to define the execution order and time points of the script, such as initialization, update, destruction, etc., which can be written in the script.

[0131] Event System: Unity provides a unified event system for processing and distributing various events in the game, such as mouse events, keyboard events, touch events, etc. 2. The event system includes the following parts:

[0132] Event: used to indicate a certain situation that occurs in the game, such as mouse click, keyboard press, touch movement, etc., which can be created in the script.

[0133] Event Source: A game object used to generate events, such as mouse, keyboard, touch screen, etc., which can be edited in the Inspector.

[0134] Event Listener: A game object used to receive and process events, such as buttons, text, images, etc., which can be edited in the Inspector.

[0135] Event Handler: It is used to define the processing logic of events, such as click events, drag events, collision events, etc., which can be written in the script.

[0136] The motion posture simulation module is controlled by the main control module and simulates and displays the motion posture corresponding to the user's finger so that the user can observe whether his or her movements are correct. The module includes a display screen and a graphics processor. The display screen is used to display the 3D model of the user's finger, and the graphics processor is used to generate the 3D model of the user's finger according to the user's movement information. The module uses Unity's graphics rendering system. The specific principles are as follows:

[0137] Graphics rendering system: Unity provides an efficient graphics rendering system for creating and displaying 3D graphics in games, such as models, textures, lighting, shadows, etc. The graphics rendering system includes the following parts:

[0138] Model: used to store the three-dimensional shape of game objects, such as vertices, faces, normals, etc., which can be created in external tools.

[0139] Texture: Used to store surface details of game objects, such as color, texture, gloss, etc., which can be created in external tools.

[0140] Material: Used to define the surface properties of a game object, such as color, texture, gloss, transparency, etc., which can be edited in the inspector.

[0141] Shader: used to define the rendering method of game objects, such as lighting, shadows, reflections, etc., which can be written in external tools.

[0142] Light: Used to simulate lighting effects in games, such as light color, intensity, direction, range, etc., which can be edited in the Inspector.

[0143] Camera: used to capture and display images in the game, such as perspective, position, direction, cropping, etc., which can be edited in the inspection view.

[0144] The rehabilitation evaluation module receives the user's motion information, voice information, and requirements and standards of rehabilitation actions transmitted by the main control module, and is used to judge the user's rehabilitation effect. The module includes a scoring algorithm and a feedback mechanism. The scoring algorithm is used to calculate the score of the user's rehabilitation action based on the user's motion information, voice information, and requirements and standards of rehabilitation actions. The feedback mechanism is used to provide the user with corresponding voice and graphic feedback based on the user's rehabilitation action score to encourage the user to improve the rehabilitation action; Figure 3 , Figure 4 As shown, this module utilizes the following principles:

[0145] Scoring algorithm: Scoring algorithm is a method to quantitatively evaluate the user's rehabilitation movements based on certain standards and rules, which can be described by the following formula:

[0146] S=f(A,V,R) (8)

[0148] Among them, S is the score of the user's rehabilitation action, A is the user's action information, V is the user's voice information, R is the requirements and standards of the rehabilitation action, and f is the scoring function, which can be designed and adjusted according to different rehabilitation actions and evaluation indicators.

[0149] Feedback mechanism: The feedback mechanism is a method of providing users with corresponding voice and graphic feedback based on the scores of their rehabilitation movements to encourage them to improve their rehabilitation movements. It can be described by the following formula:

[0150] F=g(S) (9)

[0152] Among them, F is the user's voice and graphic feedback, S is the score of the user's rehabilitation action, and g(S) is the feedback function, which can be designed and adjusted according to different feedback strategies and methods.

[0153] The rehabilitation action data recording module is used to record the rehabilitation action data completed by the user in the virtual rehabilitation scene, so as to track and analyze the rehabilitation effect for a long time. The module includes a data storage device and a data transmitter. The data storage device is used to store the user's rehabilitation action data, and the data transmitter is used to transmit the user's rehabilitation action data to the main control module or other devices. The module uses the following principles:

[0154] Data storage: Data storage is a device used to store the user's rehabilitation action data, such as hard disk, flash memory, cloud storage, etc. The appropriate type and capacity can be selected according to different storage requirements and performance. The basic principle of data storage is to convert the user's rehabilitation action data into a binary digital signal, and then store the digital signal on a storage medium such as a disk, chip, CD, etc. through electromagnetic or optical means. The storage method of data storage can be described by the following formula:

[0155] D=h(E) (10)

[0157] Among them, D is the digital signal stored in the data storage device, E is the user's rehabilitation motion data, and h is the data conversion function, which can be designed and adjusted according to different data formats and encoding methods.

[0158] Data transmitter: A data transmitter is a device used to transmit the user's rehabilitation motion data to the main control module or other devices, such as a network card, Bluetooth, Wi-Fi, etc. The appropriate type and speed can be selected according to different transmission requirements and performance. The basic principle of a data transmitter is to convert the user's rehabilitation motion data into an analog signal of electromagnetic waves or light waves, and then transmit the analog signal to the target device through wired or wireless means, such as cables, optical fibers, air, etc. The transmission method of a data transmitter can be described by the following formula:

[0159] S=g(E) (11)

[0161] Among them, S is the analog signal transmitted to the target device, E is the user's rehabilitation motion data, and g is the signal conversion function, which can be designed and adjusted according to different signal formats and modulation methods.

[0162] The data analysis module is used to analyze the rehabilitation action data, generate rehabilitation reports, and provide personalized rehabilitation suggestions. The module includes a data processor and a data display device. The data processor is used to analyze the user's rehabilitation process and effect based on the user's rehabilitation action data, generate rehabilitation reports, and provide personalized rehabilitation suggestions by using methods such as data mining and machine learning. The data display device is used to display the rehabilitation report and suggestions to the user or doctor in the form of charts or texts; Figure 4As shown in the figure, this module utilizes the following principles: Data processor: A data processor is a device used to analyze the user's rehabilitation motion data, such as CPU, GPU, FPGA, etc. The appropriate type and speed can be selected according to different analysis requirements and performance. The basic principle of the data processor is to convert the user's rehabilitation motion data into a computable data structure, such as an array, matrix, vector, etc., and then use algorithms and models to calculate and process the data structure, such as classification, clustering, regression, prediction, etc., so as to obtain the analysis results of the user's rehabilitation process and effect, such as rehabilitation curve, rehabilitation index, rehabilitation evaluation, etc. The analysis method of the data processor can be described by the following formula:

[0163] R=m(E) (11)

[0165] Among them, R is the analysis result of the user's rehabilitation process and effect, E is the user's rehabilitation movement data, and m is the analysis function, which can be designed and adjusted according to different analysis methods and models.

[0166] Data display device: A data display device is a device used to display the analysis results of the user's rehabilitation process and effect to the user or doctor, such as a display, printer, projector, etc. The appropriate type and resolution can be selected according to different display requirements and effects. The basic principle of the data display device is to convert the analysis results of the user's rehabilitation process and effect into visual graphics or text, such as line charts, bar charts, tables, reports, etc., and then display the graphics or text on a display medium such as a screen, paper, wall, etc. through optical or electronic means. The display method of the data display device can be described by the following formula:

[0167] V=n(R) (12)

[0169] Among them, V is the graph or text of the user's rehabilitation process and effect, R is the analysis result of the user's rehabilitation process and effect, and n is the display function, which can be designed and adjusted according to different display forms and styles.

[0170] The data analysis module can receive and analyze rehabilitation action data in real time, and can also generate personalized rehabilitation reports and suggestions based on the data, including:

[0171] Real-time data mining and analysis module: Receive rehabilitation data in real time through UDP protocol and analyze it immediately to provide rapid feedback and early warning, thereby optimizing the rehabilitation plan.

[0172] Personalized rehabilitation report generation algorithm module: uses machine learning algorithms to analyze key rehabilitation indicators and generate rehabilitation reports containing personalized suggestions to guide future rehabilitation plans.

[0173] Dynamic rehabilitation effect evaluation model: Dynamically adjust the evaluation criteria based on real-time data to adapt to individual differences and changes in rehabilitation stages, ensuring the accuracy and personalization of the evaluation results.

[0174] Rehabilitation process visualization module: Displays rehabilitation reports and suggestions in the form of intuitive charts and texts to improve the readability of rehabilitation data and enhance users' and doctors' understanding of the rehabilitation process.

[0175] Adaptive rehabilitation advice module: Dynamically adjust rehabilitation advice based on data in the rehabilitation report to ensure timeliness and personalization of advice and optimize rehabilitation outcomes.

[0176] These innovations combine data mining, machine learning, dynamic assessment and visualization technologies to provide users with a comprehensive, accurate and personalized rehabilitation experience. Through the application of these technologies, our system can help users recover more effectively, while providing doctors with valuable insights to support better rehabilitation decisions. The design and implementation of this data analysis module demonstrates our innovation capabilities and technological leadership in the field of rehabilitation technology.

[0177] Real-time data mining and analysis module:

[0178] The core of this module is the application of real-time data mining technology, which can analyze and process rehabilitation action data immediately when it is transmitted to the system through the UDP protocol. The application of this technology makes the data in the rehabilitation process not only passively recorded, but also actively used to generate real-time feedback, thus providing data support for the timely adjustment of the rehabilitation plan. In addition, real-time data analysis can also help identify potential rehabilitation problems, such as inaccurate action execution or abnormal rehabilitation progress, so as to give early warnings at the early stage of the problem and avoid the decline of rehabilitation effect.

[0179] Mathematical formula:

[0180]

[0181] Among them, P 实时 (d) represents the prediction effect of real-time data, d is the input data, a i is the model parameter.

[0182] Personalized rehabilitation report generation algorithm module:

[0183] The innovation of this algorithm lies in its personalized nature. Through deep learning and pattern recognition technology, the algorithm can learn from the user's rehabilitation data and identify key rehabilitation indicators. These indicators include muscle strength level, range of motion of joints, Brunnstrom stage, etc. The algorithm associates these indicators with rehabilitation effects and generates a rehabilitation report with personalized suggestions. Such a report is not only a summary of past rehabilitation activities, but also a guide for future rehabilitation plans.

[0184] Mathematical formula:

[0185]

[0186] Where R(u) is the rehabilitation report of user u, x i,u is the i-th rehabilitation indicator of user u, w i is the weight and f is the conversion function.

[0187] Dynamic rehabilitation effect evaluation model:

[0188] The innovation of the dynamic rehabilitation effect evaluation model is that it can dynamically adjust the evaluation criteria based on real-time data. This means that the evaluation model is not static, but can adjust itself according to the rehabilitation process and individual differences of each user. For example, for users in the early stages of rehabilitation, the evaluation model will focus more on the accuracy of basic movements; while for users in the later stages of rehabilitation, it will focus more on the fluency and coordination of movements. This dynamic adjustment ensures the accuracy and personalization of the evaluation results.

[0189] Mathematical formula:

[0190]

[0191] Among them, E(t) is the rehabilitation effect evaluation at time t, y j,t is the jth evaluation indicator at time t, v j is the indicator weight, and g is the evaluation function.

[0192] Rehabilitation process visualization module:

[0193] The innovation of rehabilitation process visualization technology lies in its intuitiveness and interactivity. Through charts and text, this technology can transform complex data into easy-to-understand visual information. Users and doctors can intuitively see the rehabilitation progress through these charts, such as the growth curve of muscle strength and the expansion trend of joint range of motion. In addition, these charts can also be operated interactively, such as zooming in to view details and filtering data. This visualization technology not only improves the readability of rehabilitation data, but also enhances users and doctors' understanding and control of the rehabilitation process.

[0194] Mathematical formula:

[0195]

[0196] Among them, V(p) is the visual representation of the rehabilitation process, p is the rehabilitation progress parameter, and h(z) is the mapping function of the rehabilitation status.

[0197] Adaptive Rehabilitation Recommendations Module:

[0198] The innovation of the adaptive rehabilitation advice system lies in its ability to dynamically adjust rehabilitation advice based on the data in the rehabilitation report. The system uses a predictive model to determine the best rehabilitation advice by analyzing key indicators in the rehabilitation report, such as muscle strength, range of motion, Brunnstrom stage, etc. These suggestions are not static, but change as the user's rehabilitation status changes, ensuring the timeliness and personalization of the advice. For example, if the user's muscle strength does not improve significantly at a certain stage, the system will recommend adding specific muscle strength training; if the range of motion improves well, it will recommend maintaining the current rehabilitation plan.

[0199] Mathematical formula:

[0200]

[0201] Where S(c) is the recommendation for rehabilitation situation c, l,s is the lth indicator in suggestion s, u l is the utility value of the indicator.

[0202] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0203] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0206] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A somatosensory interactive virtual rehabilitation training method, characterized in that: Applied to rehabilitation training system; Collect the user's action information and voice information after following the NPC's guidance actions; Performing user motion posture simulation according to the motion information; Calculate the rehabilitation action score of the user according to the pre-set rehabilitation action requirements and standards and the action information and voice information of the user, and provide the user with corresponding voice and graphic feedback according to the rehabilitation action score of the user; Analyze the user's rehabilitation progress and effect based on the user's motion posture simulation data and the user's rehabilitation action scores, and output a rehabilitation report; The collecting of the action information of the user following the NPC-guided action and simulating the user's motion posture includes: The distance and orientation changes of any two joint points among a plurality of skeletal joint points of the user are collected by a Kinect device, a human joint motion trajectory is determined according to the changes, and the user's motion posture is simulated according to the human joint motion trajectory; when determining the human joint motion trajectory, the human joint motion trajectory is smoothed, including: taking the coordinate value of the active joint at the current moment and the average value of the joint coordinates of the previous N-1 sampling cycles as the joint position at the current moment, and progressively moving the acquired coordinate position forward in a time sequence with N as a step length; The user's voice information is received through a Kinect device, and the source direction and distance of the voice information are converted into angles and delays through beamforming technology. Based on the angles and delays, the voice information is segmented into a series of phoneme sequences, and then the phoneme sequences are mapped into word sequences through a statistical model or a neural network model, and then the word sequences are converted into sentences or commands for interacting with the rehabilitation training system through a language model or a semantic analysis model; Analyze the user's emotions and needs based on the statements or commands; The user's rehabilitation progress and effect analysis is performed based on the user's motion posture simulation data and the user's rehabilitation action scores, and a rehabilitation report is output, including: Use UDP protocol to obtain the user's motion posture simulation data and the user's rehabilitation action score; Use deep learning and pattern recognition technology to identify key rehabilitation indicators from the user's motion posture simulation data and the user's rehabilitation action scores; generating a rehabilitation report including personalized suggestions based on the key rehabilitation indicators; Determining optimal rehabilitation recommendations based on the rehabilitation report using a deep learning model; The requirements and standards of the rehabilitation movements are self-adjusted according to the rehabilitation progress and individual differences of each user; The self-adjustment process includes: adding a BOXcollider and a kinect camera to the rehabilitation training system to detect whether the character's movements are standard. When the character touches an object with a boxcollider, the score will be increased by one and the process will be carried out in depth. Once the score is less than the set value, the current rehabilitation movement will be reset.

2. The somatosensory interactive virtual rehabilitation training method according to claim 1, characterized in that: The rehabilitation movements of the NPC-guided behavior adopt Ba Duan Jin movements.

3. A system based on the somatosensory interactive virtual rehabilitation training method according to any one of claims 1-2, characterized in that: include: A virtual rehabilitation scene construction module, used to build a virtual rehabilitation scene through Unity, wherein the virtual rehabilitation scene includes a simulated rehabilitation treatment room and multiple virtual objects and tasks related to rehabilitation actions; A human motion recognition module is used to recognize the user's body movements in the virtual rehabilitation scene by connecting to a Kinect device, and transmit the user's real motion information to the main control module; The rehabilitation movement design module designs rehabilitation movements based on the Eight-Section Brocade movements and transmits the requirements and standards of the rehabilitation movements to the main control module; The voice interaction module is used for voice interaction between the user and the system, receives the user's voice commands and feedback, and transmits the voice information to the main control module; The NPC guidance module includes a virtual NPC character used to guide the user to perform rehabilitation actions. According to the instructions of the main control module, the virtual NPC character is used to guide the user how to correctly complete the rehabilitation actions through voice and action demonstration; The main control module is used to control the operation of the entire system, judge the user's rehabilitation effect according to the user's action information, voice information and the requirements and standards of rehabilitation actions, control the changes of the virtual rehabilitation scene, control the behavior of the NPC guidance module, control the display of the motion posture simulation module, control the recording of the rehabilitation action data recording module, and control the analysis of the data analysis module; It also includes: a motion posture simulation module, which is controlled by the main control module and simulates and displays the motion posture corresponding to the user's limbs, so that the user can observe whether his or her movements are correct; The motion posture simulation module includes a display screen and a graphics processor, wherein the display screen is used to display a three-dimensional model of the user's limbs, and the graphics processor is used to generate a three-dimensional model of the user's limbs according to the user's motion information; It also includes: a rehabilitation evaluation module, which is used to receive the user's motion information, voice information and rehabilitation motion requirements and standards transmitted by the main control module, and judge the user's rehabilitation effect; The rehabilitation assessment module includes a scoring algorithm and a feedback mechanism, wherein the scoring algorithm is used to calculate the score of the user's rehabilitation action according to the user's action information, voice information and the requirements and standards of the rehabilitation action, and the feedback mechanism is used to provide the user with corresponding voice and graphic feedback according to the score of the user's rehabilitation action; It also includes: a rehabilitation action data recording module, which is used to record the rehabilitation action data completed by the user in the virtual rehabilitation scene, so as to conduct long-term tracking and analysis of the rehabilitation effect; The rehabilitation action data recording module includes a data storage device and a data transmitter, wherein the data storage device is used to store the user's rehabilitation action data, and the data transmitter is used to transmit the user's rehabilitation action data to the main control module or other devices; It also includes: a data analysis module for analyzing rehabilitation movement data, generating rehabilitation reports, and providing personalized rehabilitation suggestions; The data analysis module includes a data processor and a data display. The data processor is used to analyze the user's rehabilitation progress and effects based on the user's rehabilitation movement data, and generate a rehabilitation report and rehabilitation suggestions. The data display is used to display the rehabilitation report and rehabilitation suggestions to the user or doctor in the form of charts or text.

4. The system according to claim 3, characterized in that The virtual rehabilitation scene construction module is also used to select or generate virtual objects and tasks suitable for the user according to the user's rehabilitation action requirements and standards, and adjust the appearance and personality of the virtual NPC character; The human body motion recognition module is also used to calculate the user's limb flexibility and strength according to the user's limb movements, and transmit the calculation results to the main control module; The voice interaction module is also used to analyze the user's emotions and needs based on the user's voice information, and transmit the analysis results to the main control module; The NPC guidance module is also used to adjust the voice and actions of the virtual NPC character according to user feedback and ratings.

5. The system according to claim 4, characterized in that The rehabilitation action design module is also used to automatically or manually adjust the difficulty and sequence of rehabilitation actions according to the user's rehabilitation needs and preferences, so as to improve the user's rehabilitation effect and experience.

Images