SDR postoperative rehabilitation training system and method for cerebral palsy children

Through markless motion capture technology and virtual reality technology, combined with three-dimensional digital mannequin model and biomechanical computing model, personalized virtual scenes and interactive games are designed, which solves the targeted and cost problems of postoperative rehabilitation training for children with cerebral palsy, and realizes the Efficient and low-cost rehabilitation training effects.

CN118942625APending Publication Date: 2024-11-12SHANGHAI YIBO YUNFAN TECHNOLOGY CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410821032.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-24
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing virtual reality rehabilitation system lacks targeted rehabilitation training for children with cerebral palsy after SDR surgery, lacks personalized settings, fails to make full use of biomechanical indicators, has single feedback, lacks data records, and is costly, which limits its popularity.

Method used

Using markerless motion capture technology and virtual reality technology, capture three-dimensional human motion data through RGB cameras, create three-dimensional digital human body models and biomechanical computing models, design personalized virtual scenes and interactive games, realize data storage and management, and provide immersive Rehabilitation training environment.

Benefits of technology

It improves the accessibility and fun of rehabilitation training, provides real-time feedback and quantitative data, enhances the pertinence and effectiveness of training, reduces equipment costs, is suitable for a variety of scenarios, and reduces the work burden of the therapist.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942625A_ABST
    Figure CN118942625A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of virtual reality, and discloses a cerebral palsy child SDR postoperative rehabilitation training system and method based on a mark-point-free motion capture technology and a virtual reality technology. The cerebral palsy child SDR postoperative rehabilitation training system based on the mark-point-free motion capture technology and the virtual reality technology is a rehabilitation training system which is low in cost, suitable for various scenes and more personalized. The device is expected to be applied to different rehabilitation environments such as medical clinic, families, communities and the like in the future, and is used for SDR postoperative rehabilitation training of cerebral palsy children. Meanwhile, a therapist can evaluate the rehabilitation condition of the patient by referring to game data of the child patient during training at home, so that one rehabilitation therapist can serve multiple patients, the working efficiency is improved, the workload is reduced, and the financial burden of rehabilitation services of the patient and the society is correspondingly reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of virtual reality technology, and in particular relates to a SDR postoperative rehabilitation training system and method for children with cerebral palsy based on marker-free motion capture technology and virtual reality technology. Background Art

[0002] Cerebral palsy is a long-term movement disorder and abnormal posture caused by abnormal brain development or damage before birth, during birth, or infancy. Selective Dorsal Rhizotomy (SDR) is a neurosurgical procedure used to treat spastic cerebral palsy. SDR surgery usually results in a significant reduction in spasticity, which improves the patient's movement ability, giving them better muscle control and further promoting independence in bed, chairs and other daily activities, development of motor skills, and improved gait.

[0003] After SDR surgery, physical therapy is a vital component of rehabilitation. The goal of physical therapy is to maximize the child's muscle function, motor control, and overall motor ability through systematic exercise and rehabilitation programs. Although physical therapy plays a key role in the field of SDR postoperative rehabilitation, it usually requires patients to undergo face-to-face rehabilitation training under the guidance of a therapist, and physical therapy is a long-term process, resulting in high time and money costs; the relatively single training model makes the training process monotonous and boring, and patients have low compliance with long-term training. Studies have shown that virtual reality technology can create an immersive environment, enhance patient training participation, and promote rehabilitation effects. Therefore, virtual reality-based rehabilitation systems have gradually been introduced into rehabilitation training for people such as stroke.

[0004] Through the above analysis, the problems and defects of the prior art are as follows:

[0005] (1) Insufficient targeting for children with cerebral palsy: Current virtual reality rehabilitation systems often lack customization for the specific needs of children with cerebral palsy after SDR surgery. Current rehabilitation systems mainly involve stroke patients and fall prevention for the elderly. Such systems often involve gait and gross motor skills. However, due to the phased characteristics of SDR postoperative rehabilitation for children with cerebral palsy, these trainings cannot meet the special recovery needs of children.

[0006] (2) Lack of personalized settings: Existing systems often lack sufficient personalized setting options and are unable to adjust the training content, difficulty, and intensity according to the child’s stage of rehabilitation or the child’s own functional level recovery.

[0007] (3) Insufficient integration with clinically significant biomechanical indicators and insufficient provision of quantitative data: The current virtual reality system fails to make full use of scientific biomechanical indicators to monitor the rehabilitation process, resulting in the inability to provide sufficient clinically significant quantitative data, thus affecting therapists' monitoring of rehabilitation progress.

[0008] (4) Insufficient diversified feedback that complies with rehabilitation principles: Diversified feedback, including visual and auditory feedback, can enhance the realism of the virtual environment. Studies have shown that diversified and personalized feedback can help promote training effects. However, the feedback provided by existing systems is often single and lacks diversity and personalization, which cannot fully motivate patients to participate in rehabilitation training.

[0009] (5) Lack of a database to record the training process: The existing virtual reality rehabilitation system can only save some simple game data, such as the training duration of the game, etc., and lacks a comprehensive database to record more clinically relevant training parameters such as the number of movements, movement amplitude, etc., and cannot track and monitor the patient's rehabilitation process.

[0010] (6) High cost: Cost is one of the main barriers to existing virtual reality rehabilitation devices, which limits their popularity in clinical and home settings. Even the lower-cost portable motion capture device Kinect still costs RMB 2,500. Summary of the invention

[0011] In view of the problems existing in the prior art, the present invention provides a SDR postoperative rehabilitation training system and method for children with cerebral palsy based on markerless motion capture technology and virtual reality technology.

[0012] The present invention is implemented as follows: a SDR postoperative rehabilitation training system for children with cerebral palsy based on marker-free motion capture technology and virtual reality technology, comprising:

[0013] Motion data capture module, model creation module, virtual scene creation module, situational interactive game design module, user management and data storage module.

[0014] The motion data capture module is used to capture three-dimensional human motion data using an RGB camera.

[0015] The model creation module is used to create three-dimensional digital human body models and biomechanical calculation models.

[0016] The virtual scene creation module is used for virtual scenes and is composed of ambient light, three-dimensional models, model rendering and core engine components.

[0017] The scenario interactive game design module is used to obtain human motion information through RGB cameras, motion parameters calculated by biomechanical models, and data generated by scene objects during human-computer interaction.

[0018] The user management and data storage module is used to record the user's numbered personal information and automatically store the motion information, biomechanical parameters, game settings and game performance obtained by the RGB camera during the game.

[0019] Furthermore, the three-dimensional digital human body model and the biomechanical calculation model are created:

[0020] A static three-dimensional digital human body is built through a 3D mesh model. The human motion information collected by the RGB camera is used to bind the bone nodes of the character model in Unity3D with the patient's bone information, and the three-dimensional character model is dynamically synchronized with the patient's movements. At the same time, based on the human motion information collected by the RGB camera, important human motion parameters including joint angles are calculated to build a biomechanical calculation model.

[0021] Further, the motion data capture module:

[0022] Based on the ThreeDPoseTracker machine learning algorithm and a custom joint position coordinate extraction algorithm, the 3D heat map estimation technology is used to estimate the 3D relative coordinates of the human body using an RGB camera. The coordinate positions of 24 key joints of the human body are inferred from a single frame, including the navel, nose, left and right ears, left and right eyes, left and right shoulders, left and right elbows, left and right wrists, left and right thumbs, left and right middle fingers, left and right hip joints, left and right knee joints, left and right ankle joints, and left and right big toes.

[0023] When the player's body is within the camera's field of view, the system can quickly estimate the user's body joints in real time and map the joints to the human skeleton model. Through the joint coordinate extraction algorithm, biomechanical indicators including joint angles are calculated.

[0024] Furthermore, the virtual scene creation module:

[0025] Create and use multiple virtual scenes suitable for children, including amusement parks, magic worlds, and underwater worlds.

[0026] The virtual scene is composed of ambient light, 3D model, model rendering and core engine components. The virtual scene is developed through Unity3D, and virtual objects are spliced ​​and size parameters are adjusted according to the scene construction requirements.

[0027] Furthermore, the scenario interactive game design module:

[0028] Create a canvas in the Unity editor as the root object of the UI interface, and create a game selection panel under the canvas as the main interface for game selection. Create a UI element for each game - a "button", associate the button with each game through C# code, create a selection page for each game, and create UI elements including drop-down menus, input boxes to display different difficulty levels, and input different game settings. Each difficulty corresponds to different training parameters, including game duration, action amplitude, and number of actions.

[0029] Furthermore, the user management and data storage module:

[0030] The data obtained from this game and the patient's name, number and personal information are stored, and this information is stored together with the game settings and game performance.

[0031] Game performance mainly includes game scores, three-dimensional joint position data captured during movement, relevant parameters calculated by the biomechanical model, and game settings including game duration.

[0032] Another object of the present invention is to provide a method for SDR postoperative rehabilitation training for children with cerebral palsy based on markerless motion capture technology and virtual reality technology, comprising:

[0033] Step 1: markerless motion capture system;

[0034] Step 2: Create a three-dimensional digital human body model and a biomechanical model;

[0035] Step 3: Create a virtual scene;

[0036] Step 4: Design of situational interactive games;

[0037] Step 5: Data storage.

[0038] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the SDR postoperative rehabilitation training method for children with cerebral palsy based on markerless motion capture technology and virtual reality technology.

[0039] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the SDR postoperative rehabilitation training method for children with cerebral palsy based on markerless motion capture technology and virtual reality technology.

[0040] Another object of the present invention is to provide an information data processing terminal, which is used to implement the SDR postoperative rehabilitation training system for children with cerebral palsy based on markerless motion capture technology and virtual reality technology.

[0041] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are analyzed from the following aspects:

[0042] First, the present invention is based on the markerless motion capture technology, which solves the problem that traditional rehabilitation training is time-consuming and laborious, improves the accessibility of rehabilitation training, and solves the problem of lack of objective quantitative indicators in the traditional rehabilitation process through kinematic algorithms, and can provide real-time feedback and progress evaluation through motion capture and data analysis. By introducing virtual reality technology, the fun and interactivity of postoperative rehabilitation training are improved. At the same time, a database for data storage, display and management is designed to solve the problem that the existing rehabilitation process is difficult to quantify and continuously track.

[0043] Step 1: Using computer vision-based markerless motion capture technology, compared with traditional marker-based (such as Vicon), depth sensor-based (such as Kinect), and IMU-based motion capture technology, only ordinary RGB cameras can be used to capture the three-dimensional coordinates of the joints of children in real time, eliminating the need for physical contact of traditional rehabilitation equipment, and can avoid the physical interference and discomfort caused by the installation of sensors to children, thereby enhancing the training acceptance of children. At the same time, there is no need to install sensors or markers, which greatly saves the time of therapists and children, making the training process more convenient and faster. In addition, the use of this technology can make the training system more adaptable to different environments and space requirements, without being restricted by specific equipment, so that it can be used in various rehabilitation scenarios, including home, rehabilitation center, etc., greatly increasing the flexibility and applicability of the system.

[0044] Step 2: Through the developed biomechanical model, the system can monitor the movement quality of the child in real time, and provide real-time feedback and evaluation of the movement quality during the child's training process by comparing with the rule-based algorithm. This helps the child to adjust the training movement in time to improve the training quality. At the same time, it can help the therapist to adjust the rehabilitation plan in time to ensure the effectiveness and safety of the training.

[0045] Step 3 and Step 4: Through virtual reality technology, combined with the principles of postoperative rehabilitation for children with cerebral palsy, a highly personalized rehabilitation training program can be designed according to the specific needs and rehabilitation progress of the child. This personalized training can more accurately intervene in the child's defects and improve the pertinence and efficiency of rehabilitation. In addition, the immersive experience provided by virtual reality technology can greatly improve the training participation and motivation of children. By interacting and playing games in a virtual environment, children are more willing to participate in rehabilitation training, thereby increasing the frequency and effectiveness of training.

[0046] Step 5: The data storage and management system of the invention can store the training time of the child, the number of movements completed in this training, etc., which is convenient for the therapist to remotely monitor and evaluate the training quality of the child and the progress of the training plan, thereby improving work efficiency. At the same time, data storage and management can more effectively manage the data of the child and avoid data loss and other factors that affect the tracking of health progress.

[0047] The present invention adopts the markerless motion capture technology based on computer vision, which can capture the three-dimensional position of the joints of the child in real time through an ordinary RGB camera, thereby eliminating the need for physical contact or installation of any sensors, avoiding physical discomfort or interference to the child, and at the same time improving the convenience and speed of the training process as well as the flexibility and scope of application of the system. The present invention focuses on the rehabilitation needs of children with cerebral palsy after SDR surgery, and designs six sports games to adapt to different stages of the rehabilitation period, aiming to improve the key rehabilitation indicators of the child, such as muscle strength, joint flexibility, balance ability and explosive power, which is more targeted.

[0048] The present invention is based on virtual reality technology and provides a three-dimensional immersive virtual environment, which can improve the training participation and motivation of children, thereby improving the training effect. Secondly, the biomechanical model developed by this system can calculate important motion parameters related to SDR postoperative rehabilitation. These parameters can quantify the movement quality and balance ability of SDR postoperative patients, and provide therapists with more clinically significant indicators. Finally, special postures defined based on joint point coordinates and kinematic parameters (such as the trunk leaning forward) are used for abnormal posture detection in the game. The detected abnormal postures will be used for feedback design to give children timely movement corrections and improve training effects.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] By using markerless motion capture technology based on computer vision, the three-dimensional coordinates of the joints of children can be captured in real time using only an ordinary RGB camera, avoiding the physical interference and discomfort caused by installing sensors to the children, making the training process more convenient and faster, and expanding the applicability of the system.

[0051] The developed biomechanical model includes the acquisition and calculation of important motion parameters related to SDR postoperative rehabilitation and abnormal posture detection, etc. It can provide more clinically significant biomechanical indicators, monitor the movement quality of children in real time, and provide movement correction.

[0052] According to the relevant theories of neurological disease rehabilitation, the sports games developed by this system are more targeted at the postoperative rehabilitation of children with cerebral palsy. They combine the task requirements of different rehabilitation periods and improve the physical functions of children at each rehabilitation stage through game actions such as sitting, standing, walking, and grasping. At the same time, the system allows children to choose targeted games and game difficulty according to their own rehabilitation progress and ability level, making the game tasks more targeted.

[0053] The SDR postoperative rehabilitation training system for children with cerebral palsy based on the markerless motion capture technology and virtual reality technology of the present invention is a low-cost, multi-scenario, and more personalized rehabilitation training system. In the future, the system equipment is expected to be applied to different rehabilitation environments such as medical clinics, families, and communities, as well as for postoperative SDR rehabilitation training for children with cerebral palsy. At the same time, the therapist can refer to the game data of the children during training at home to evaluate the patient's rehabilitation status, so that one rehabilitation therapist can serve multiple patients, improve work efficiency and reduce workload, and correspondingly reduce the financial burden of rehabilitation services for patients and society.

[0054] Second, the present invention provides an SDR postoperative rehabilitation training system for children with cerebral palsy based on markerless motion capture technology and virtual reality technology. The following is a detailed analysis of its technical features, the technical problems it solves, and the technical progress it brings:

[0055] 1) Motion data capture module: Use RGB camera to capture 3D human motion data without additional markers, reducing setup difficulty and cost.

[0056] 2) Model creation module: Create a three-dimensional digital human model and a biomechanical computational model to simulate and analyze the patient's movements.

[0057] 3) Virtual scene creation module: Creates virtual scenes including ambient light, three-dimensional models, model rendering and other elements to provide an immersive rehabilitation training environment.

[0058] 4) Situational interactive game design module: Combine human motion information, biomechanical parameters and human-computer interaction data to design interactive games to improve the fun and effectiveness of rehabilitation training.

[0059] 5) User management and data storage module: manage user information and store various data during the training process to facilitate subsequent analysis and evaluation.

[0060] Technical issues solved:

[0061] 1) Setup difficulty and cost: Traditional motion capture systems usually require a large number of markers, which not only increases the setup difficulty but also increases the cost. This system uses markerless technology to reduce setup difficulty and cost.

[0062] 2) Patient movement simulation and analysis: By creating a three-dimensional digital human model and a biomechanical computational model, the patient's movement can be accurately simulated and analyzed, providing a scientific basis for rehabilitation training.

[0063] 3) Rehabilitation training environment: The traditional rehabilitation training environment is monotonous and difficult to arouse the interest of patients. This system provides an immersive rehabilitation training environment by creating virtual scenes and interactive games, which increases the fun and attractiveness of training.

[0064] 4) Data management and analysis: Through the user management and data storage module, patients' personal information and training data can be easily recorded and managed, providing convenience for subsequent analysis and evaluation.

[0065] Technological advancements brought about:

[0066] 1) Markerless motion capture technology: improves the convenience and accuracy of motion capture and reduces costs.

[0067] 2) Three-dimensional digital human body model and biomechanical computational model: provide a scientific basis for rehabilitation training and help develop personalized training plans.

[0068] 3) Virtual scenes and interactive games: increase the fun and appeal of rehabilitation training, and improve patients' training enthusiasm and effectiveness.

[0069] 4) Data management and analysis: It is easy to record, manage and analyze patients' training data, which facilitates subsequent evaluation and improvement.

[0070] In summary, the system provides a new solution for SDR postoperative rehabilitation training for children with cerebral palsy and brings significant technological progress by adopting markerless motion capture technology, creating three-dimensional digital human models and biomechanical computational models, designing virtual scenes and interactive games, and realizing user management and data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a structural block diagram of the SDR postoperative rehabilitation training system for children with cerebral palsy based on markerless motion capture technology and virtual reality technology provided by an embodiment of the present invention.

[0072] Figure 2 It is a device connection diagram provided by an embodiment of the present invention.

[0073] Figure 3 It is a development and workflow diagram of the system provided by the embodiment of the present invention.

[0074] Figure 4 It is a flow chart of the SDR postoperative rehabilitation training method for children with cerebral palsy based on markerless motion capture technology and virtual reality technology provided by an embodiment of the present invention.

[0075] Figure 1 In: 1. Motion data capture module; 2. Model creation module; 3. Virtual scene creation module; 4. Situational interactive game design module; 5. User management and data storage module. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0077] like Figure 1 As shown, an embodiment of the present invention provides an SDR postoperative rehabilitation training system for children with cerebral palsy based on marker-free motion capture technology and virtual reality technology, comprising:

[0078] Motion data capture module 1, model creation module 2, virtual scene creation module 3, situational interactive game design module 4, user management and data storage module 5.

[0079] The motion data capture module 1 is used to capture three-dimensional human motion data using an RGB camera;

[0080] Model creation module 2, used to create a three-dimensional digital human body model and a biomechanical calculation model;

[0081] Virtual scene creation module 3, which is used for virtual scene, is composed of ambient light, three-dimensional model, model rendering and core engine components;

[0082] Situational interactive game design module 4, used for human motion information obtained through RGB cameras, motion parameters calculated by biomechanical models, and data generated by scene objects during human-computer interaction;

[0083] The user management and data storage module 5 is used to record the user's numbered personal information and automatically store the motion information, biomechanical parameters, game settings and game performance obtained by the RGB camera during the game.

[0084] The invention provides a method for creating a three-dimensional digital human body model and a biomechanical calculation model:

[0085] A static three-dimensional digital human body is built through a 3D mesh model. The human motion information collected by the RGB camera is used to bind the bone nodes of the character model in Unity3D with the patient's bone information, and the three-dimensional character model is dynamically synchronized with the patient's movements. At the same time, based on the human motion information collected by the RGB camera, important human motion parameters including joint angles are calculated to build a biomechanical calculation model.

[0086] The motion data capture module provided by the embodiment of the present invention:

[0087] Based on the ThreeDPoseTracker machine learning algorithm and a custom joint position coordinate extraction algorithm, 3D heat map estimation technology is used. The RGB camera is used to estimate the three-dimensional relative coordinates of the human body, and the coordinate positions of 24 key joints of the human body are inferred from a single frame, including the navel, nose, left and right ears, left and right eyes, left and right shoulder joints, left and right elbow joints, left and right wrist joints, left and right thumbs, left and right middle fingers, left and right hip joints, left and right knee joints, left and right ankle joints, and left and right big toes;

[0088] When the player's body is within the camera's field of view, the system can quickly estimate the user's body joints in real time and map the joints to the human skeleton model; through the joint coordinate extraction algorithm, it calculates biomechanical indicators including joint angles.

[0089] The virtual scene creation module provided by the embodiment of the present invention:

[0090] Create and use multiple virtual scenes suitable for children, including amusement parks, magic worlds, and underwater worlds;

[0091] The virtual scene is composed of ambient light, three-dimensional model, model rendering and core engine components. The virtual scene is developed through Unity3D, and virtual objects are spliced ​​and size parameters are adjusted according to the scene construction requirements.

[0092] The scenario interactive game design module provided by the embodiment of the present invention:

[0093] Create a canvas in the Unity editor as the root object of the UI interface, create a game selection panel under the canvas as the main interface for game selection; create a UI element for each game - "button". Associate the button with each game through C# code, create a selection page for each game, create UI elements including drop-down menus, input boxes to display different difficulty levels and input different game settings, and each difficulty corresponds to different training parameters, including game duration, action amplitude and number of actions.

[0094] The user management and data storage module provided by the embodiment of the present invention:

[0095] Store the data obtained from this game and the patient's name, number and personal information, and store this information together with the game settings and game performance;

[0096] Game performance mainly includes game scores, three-dimensional joint position data captured during movement, and relevant parameters calculated by the biomechanical model; game settings include game duration.

[0097] like Figure 4 As shown, an SDR postoperative rehabilitation training method for children with cerebral palsy based on markerless motion capture technology and virtual reality technology provided by an embodiment of the present invention includes:

[0098] S101, markerless motion capture system;

[0099] S102, creating a three-dimensional digital human body model and a biomechanical model;

[0100] S103, creating a virtual scene;

[0101] S104, situational interactive game design;

[0102] S105, data storage.

[0103] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the SDR postoperative rehabilitation training method for children with cerebral palsy based on markerless motion capture technology and virtual reality technology.

[0104] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the SDR postoperative rehabilitation training method for children with cerebral palsy based on markerless motion capture technology and virtual reality technology.

[0105] Another object of the present invention is to provide an information data processing terminal, which is used to implement the SDR postoperative rehabilitation training system for children with cerebral palsy based on markerless motion capture technology and virtual reality technology.

[0106] The present invention is based on the markerless motion capture technology, which solves the problem that traditional rehabilitation training is time-consuming and labor-intensive, and improves the accessibility of rehabilitation training. At the same time, through the kinematic algorithm, it solves the problem of lack of objective quantitative indicators in the traditional rehabilitation process, and can provide real-time feedback and progress evaluation through motion capture and data analysis. By introducing virtual reality technology, the fun and interactivity of postoperative rehabilitation training are improved. At the same time, a database for data storage, display and management is designed to solve the problem that the existing rehabilitation process is difficult to quantify and continuously track.

[0107] Device connection diagram Figure 2 As shown:

[0108] The system integrates motion capture, digital human modeling, virtual reality and other technologies. It obtains human motion information through a depth camera, uses digital human modeling technology to create a virtual human model, creates a virtual scene, imports human motion information data and binds it to the character model, and realizes the control of the virtual human skeleton movements in the virtual environment.

[0109] System development and workflow Figure 3 As shown:

[0110] 1) Motion data capture: The RGB camera captures three-dimensional human motion data and provides human motion information (joint position information) for the Unity3D virtual environment.

[0111] 2) Create a 3D digital human model and biomechanical calculation model: Use the 3D mesh model to build a static 3D digital human body. Use the human motion information collected by the RGB camera to bind the bone nodes of the character model in Unity3D with the patient's bone information to achieve dynamic synchronization between the 3D character model and the patient's movements. At the same time, based on the human motion information collected by the RGB camera, calculate important human motion parameters (such as joint angles) and build a biomechanical calculation model.

[0112] 3) Create virtual scenes: Virtual scenes are composed of ambient light, 3D models, model rendering and core engine components.

[0113] 4) Situational interactive game design: The human body motion information (joint position) obtained by the RGB camera, the motion parameters calculated by the biomechanical model, and the data generated by the scene objects during human-computer interaction (collision, distance, etc.) are used to achieve animation effects, thereby providing training feedback to patients.

[0114] 5) User management and data storage: The system can record personal information such as the user's ID, and automatically store motion information, biomechanical parameters, game settings, and game performance (such as scores) obtained by the RGB camera during the game, making it convenient for patients and medical staff to view and monitor the progress of rehabilitation, so as to adjust the next training content and difficulty according to the personalized rehabilitation progress.

[0115] Step 1: Markerless Motion Capture System

[0116] The markerless motion capture system used in this system is mainly based on the ThreeDPoseTracker (TDPT) machine learning algorithm and a custom joint position coordinate extraction algorithm. TDPT is a 3D body posture estimation algorithm based on machine learning. It uses 3D heat map estimation technology to estimate the three-dimensional relative coordinates of the human body using an RGB camera without using any markers. It can infer the coordinate positions of 24 key joints of the human body from a single frame, including the navel, nose, left and right ears, left and right eyes, left and right shoulder joints, left and right elbow joints, left and right wrist joints, left and right thumbs, left and right middle fingers, left and right hip joints, left and right knee joints, left and right ankle joints, and left and right big toes. When the player's body is within the field of view of the camera, the system can quickly estimate the user's body joint points in real time and map the joint points to the human skeleton model. Through the joint coordinate extraction algorithm, biomechanical indicators such as joint angles can be calculated.

[0117] Step 2: Create a 3D digital human body model and biomechanical model

[0118] This system has developed multiple children's animation characters, allowing users to make personalized character selections when using this system for training, and to fit the image of the user group. A three-dimensional digital human body is established through a 3D mesh model, and the animation character is imported into Unity3D for real-time action demonstration. The specific production process is: modeling (model) - material (texture) - setting bone binding (rig) - animation (animation). First, the model is made and the real model effect is created through the modeling tool. After completing the three-dimensional modeling, the material map is made according to the material characteristics of different models, and the relevant settings are adjusted to form a real surface rendering effect. Then, the character is skeletally bound, and a three-dimensional skeleton similar to the human body structure is created for the character, so that the skeleton can control the movement of the model. Through motion binding, the real human body three-dimensional joint point information captured by the RGB camera is used to drive the movement of the virtual character, and the virtual character can reflect the real-time movement of the patient. This process will be achieved by using the lightweight multi-platform neural network library-Barracuda toolkit in the development environment of the Unity3D engine.

[0119] Based on the three-dimensional joint point coordinate data obtained by the RGB camera, a motion biomechanical calculation model is established in the Unity 3D environment. The biomechanical model includes obtaining and calculating important motion parameters related to SDR postoperative rehabilitation and abnormal posture detection. Visual Studio 2019 is used for C# coding, and the code file is mounted on the virtual character of Unity3D to achieve real-time parameter acquisition and calculation. First, the joint point coordinates related to the main movement are extracted (such as elbow, hip, knee, ankle joint coordinates, etc.). Then, the kinematic parameters (such as important joint point positions, joint angles, center of gravity offset, etc.) are calculated based on the joint point coordinates. These parameters can quantify the movement quality and balance ability of patients after SDR surgery, and are closely related to clinical analysis. The parameter results can be provided to therapists for reference. Finally, special postures (such as the forward leaning of the trunk) are defined based on the joint point coordinates and kinematic parameters for abnormal posture detection in the game. The detected abnormal posture will be used for feedback design. If this abnormal posture is detected in the game, the corresponding feedback will be provided to the patient to improve his own movements (see the game design section for feedback design).

[0120] Step 3: Create a virtual scene

[0121] This system creates and uses multiple virtual scenes suitable for children, such as amusement parks, magic worlds, underwater worlds, etc. Different rehabilitation games are configured with different scenes. The specific virtual scene is composed of ambient light, three-dimensional models, model rendering, and core engine components. The virtual scene is developed through Unity3D. According to the needs of scene construction, virtual objects are spliced ​​and the size and other parameters are adjusted. Other aspects of the scene (such as mirror reflection) are optimized. These parameter adjustments and optimizations make the scene more realistic.

[0122] Step 4: Scenario-based interactive game design

[0123] According to the relevant theories of neurological disease rehabilitation, this system has developed a total of 6 sports games for the three rehabilitation stages (recovery stage, ascending stage, and strengthening stage) after SDR surgery (Table 1), aiming to scientifically improve the muscle strength, joint mobility, balance ability, and explosive power of children with cerebral palsy after SDR surgery. These games combine the task requirements of different rehabilitation periods, including game tasks based on sitting, standing, walking, etc. suitable for different stages, to improve the physical functions of children in different rehabilitation periods. The scenario interactive game was developed using Unity3D. The navigation interface was made based on the Canvas component, and a menu was provided to facilitate users to personalize the game tasks.

[0124] Through the motion capture system without markers, the abnormal posture defined by the biomechanical model, and the data (collision, distance, etc.) generated by the scene objects during human-computer interaction, the present invention can provide personalized, multi-sensory motion feedback information for children. Specifically, the real human three-dimensional joint point information captured by the RGB camera combined with the TDPT algorithm can drive the movement of the virtual character in real time, and the patient can correct his own movements by observing the movements of the virtual character. The C# code control system monitors the game actions in real time, and when the system detects the abnormal posture defined by the biomechanical model, it gives a prompt to correct the wrong action. The collision, position distance, etc. generated by the virtual character and the virtual objects in the game scene will also become the basis for feedback generation, such as giving encouraging feedback when a certain object is successfully touched. The above feedback is integrated into the system in the form of multi-sensory, including information visual feedback (such as text, pictures, etc.) and auditory feedback (including auditory effect sound and verbal reminders). The relevant picture materials are imported into the Unity3D Assest folder, the feedback materials are presented using the "Canvas" component, and the feedback information is triggered by the custom C# code control.

[0125] At the same time, the system allows children to choose targeted games and game difficulty according to their own rehabilitation progress and ability level, to ensure that children can choose personalized games that suit their own situation and effectively support their rehabilitation process. By creating a canvas in the Unity editor as the root object of the UI interface, a game selection panel is created under the canvas as the main interface for game selection. By creating a UI element - "button" for each game, the button is associated with each game through C# code. Create a selection page for each game, create UI elements such as drop-down menus, input boxes, etc. to display different difficulty levels and enter different game settings. Each difficulty corresponds to different training parameters, such as game duration, movement amplitude, and number of movements, to meet the personalized needs of children with different abilities. The difficulty level of each game task is set by consulting the therapist's advice.

[0126] Step 5: Data Storage

[0127] After the game is over, the data obtained from the game and the patient's name, number and other personal information can be stored, and this information can be stored together with the game settings and game performance. Game performance mainly includes game scores, three-dimensional joint position data captured during exercise, and related parameters calculated by the biomechanical model. Game settings include game duration, etc. These data are displayed in the "canvas" interface in the form of tables or bar graphs. Rehabilitation therapists and patients can log in to the system to remotely obtain the patient's game data and related information, so as to evaluate the rehabilitation training of children after SDR surgery, which is helpful for timely adjustment of training plans and for patients to further strengthen their exercises in a targeted manner.

[0128] Table 1. Introduction to virtual reality games for SDR postoperative rehabilitation of children with cerebral palsy

[0129]

[0130]

[0131] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0132] The system provided by the embodiment of the present invention utilizes the markerless motion capture technology and virtual reality technology, and combines multiple modules to implement SDR postoperative rehabilitation training for children with cerebral palsy. The following is the working principle of each module in the system:

[0133] 1) Motion data capture module:

[0134] -Use RGB camera to capture the 3D human motion data of the child. The RGB camera can obtain the child's motion information in real time, including posture, movement, etc.

[0135] 2) Model creation module:

[0136] -This module is used to create three-dimensional digital human models and biomechanical computational models.

[0137] -The static three-dimensional digital human body model is established through a 3D mesh model, referring to the standard human anatomy model and customized in combination with the individual characteristics of the child.

[0138] -The biomechanical computational model uses the movement data of the child to calculate important movement parameters such as joint angles, thereby constructing a biomechanical model to provide a basis for subsequent training.

[0139] 3) Virtual scene creation module:

[0140] -This module is responsible for creating virtual scenes, including ambient light, 3D models, model rendering and core engine components.

[0141] -The design of virtual scenes needs to be customized according to the needs of rehabilitation training, including simulation of real-life rehabilitation environments, game scenes, etc.

[0142] 4) Situational interactive game design module:

[0143] -Use RGB cameras to obtain the patient's body movement information, and combine it with the movement parameters calculated by the biomechanical model and the object data in the virtual scene to design an interactive game for rehabilitation training.

[0144] -The design of the game needs to take into account the rehabilitation needs of the children and promote their rehabilitation training by interacting with objects in the virtual environment.

[0145] 5) User management and data storage module:

[0146] -Used to record the user's ID and personal information, and automatically store the motion information, biomechanical parameters, game settings and game performance obtained by the RGB camera during the game.

[0147] -The storage and management of data helps to evaluate the recovery of children and provide a reference for the adjustment and optimization of the recovery process.

[0148] In summary, the system integrates multiple modules, combines markerless motion capture technology and virtual reality technology, and realizes SDR postoperative rehabilitation training for children with cerebral palsy. The system uses real-time captured motion data and interactive design of virtual scenes to provide children with personalized and effective rehabilitation training programs.

[0149] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A SDR postoperative rehabilitation training system for children with cerebral palsy based on markerless motion capture technology and virtual reality technology. It is characterized in that include: Motion data capture module, model creation module, virtual scene creation module, scenario interactive game design module, user management and data storage module; A motion data capture module is used to capture three-dimensional human motion data using an RGB camera; Model creation module, used to create three-dimensional digital human body models and biomechanical calculation models; Virtual scene creation module, which is used for virtual scenes and consists of ambient light, 3D models, model rendering and core engine components; The scenario interactive game design module is used to obtain human motion information through the RGB camera, motion parameters calculated by the biomechanical model, and data generated by scene objects during human-computer interaction; The user management and data storage module is used to record the user's numbered personal information and automatically store the motion information, biomechanical parameters, game settings and game performance obtained by the RGB camera during the game.

2. The SDR postoperative rehabilitation training system for children with cerebral palsy based on markerless motion capture technology and virtual reality technology as claimed in claim 1, It is characterized in that The three-dimensional digital human body model and biomechanical calculation model are created: A static three-dimensional digital human body is established through a 3D mesh model. The human body motion information collected by the RGB camera is used to bind the bone nodes of the character model in Unity3D to the patient's bone information, and the three-dimensional character model is dynamically synchronized with the patient's movements. At the same time, based on the human body motion information collected by the RGB camera, important human body motion parameters including joint angles are calculated to construct a biomechanical calculation model.

3. The SDR postoperative rehabilitation training system for children with cerebral palsy based on markerless motion capture technology and virtual reality technology as claimed in claim 1, It is characterized in that The motion data capture module: Based on the ThreeDPoseTracker machine learning algorithm and a customized joint position coordinate extraction algorithm; using 3D heat map estimation technology, the RGB camera is used to estimate the three-dimensional relative coordinates of the human body; the coordinate positions of 24 key joints of the human body are inferred from a single frame, including the navel, nose, left and right ears, left and right eyes, left and right shoulders, left and right elbows, left and right wrists, left and right thumbs, left and right middle fingers, left and right hip joints, left and right knee joints, left and right ankle joints, and left and right big toes; When the player's body is within the camera's field of view, the system can quickly estimate the user's body joints in real time and map the joints to the human skeleton model; through the joint coordinate extraction algorithm, it calculates biomechanical indicators including joint angles.

4. The SDR postoperative rehabilitation training system for children with cerebral palsy based on markerless motion capture technology and virtual reality technology as claimed in claim 1, It is characterized in that The virtual scene creation module: Create and use multiple virtual scenes suitable for children, including amusement parks, magic worlds, and underwater worlds; The virtual scene is composed of ambient light, three-dimensional model, model rendering and core engine components. The virtual scene is developed through Unity3D; virtual objects are spliced ​​and size parameters are adjusted according to the scene construction requirements.

5. The SDR postoperative rehabilitation training system for children with cerebral palsy based on markerless motion capture technology and virtual reality technology as claimed in claim 1, It is characterized in that The scenario interactive game design module: Create a canvas in the Unity editor as the root object of the UI interface, and create a game selection panel under the canvas as the main interface for game selection; By creating a UI element - "button" for each game, and associating the button with each game through C# code; creating a selection page for each game, creating UI elements including drop-down menus, input boxes to display different difficulty levels, and input different game settings. Each difficulty corresponds to different training parameters, including game duration, action amplitude, and number of actions.

6. The SDR postoperative rehabilitation training system for children with cerebral palsy based on markerless motion capture technology and virtual reality technology as claimed in claim 1, It is characterized in that The user management and data storage module: Store the data obtained from this game and the patient's name, number and personal information, and store this information together with the game settings and game performance; Game performance mainly includes game scores, three-dimensional joint position data captured during movement, and relevant parameters calculated by the biomechanical model; Game settings include game duration.

7. A method for SDR postoperative rehabilitation training for children with cerebral palsy based on markerless motion capture technology and virtual reality technology, which implements the SDR postoperative rehabilitation training system for children with cerebral palsy based on markerless motion capture technology and virtual reality technology as claimed in any one of claims 1 to 6, It is characterized in that The SDR postoperative rehabilitation training method for children with cerebral palsy based on markerless motion capture technology and virtual reality technology includes: Step 1: markerless motion capture system; Step 2: Create a three-dimensional digital human body model and a biomechanical model; Step 3: Create a virtual scene; Step 4: Design of situational interactive games; Step 5: Data storage.

8. A computer device, It is characterized in that The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the SDR postoperative rehabilitation training method for children with cerebral palsy based on markerless motion capture technology and virtual reality technology as described in claim 7.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the SDR postoperative rehabilitation training method for children with cerebral palsy based on markerless motion capture technology and virtual reality technology as described in claim 7.

10. An information data processing terminal, It is characterized in that The information data processing terminal is used to implement the SDR postoperative rehabilitation training system for children with cerebral palsy based on the marker-free motion capture technology and virtual reality technology as described in any one of claims 1 to 6.

Citation Information

Cited By

  • Patient remote rehabilitation management method, device, equipment and medium

    CN120089281A