Rehabilitation training method and system based on action recognition and virtual reality
Through the virtual rehabilitation system of Unity and Leap Motion technology, identify and compare user hand movements, provide personalized feedback and suggestions, solving the problem of lack of fun and insufficient accessibility of existing rehabilitation methods, and improving patient engagement and Rehabilitation effect.
Patent Information
- Application Number
- CN202410806131.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-21
- Publication Date
- 2024-11-22
AI Technical Summary
The lack of fun in existing hand rehabilitation methods leads to low patient engagement, increased psychological stress, and limited accessibility to professional rehabilitation equipment and experts, especially in remote areas.
Using virtual rehabilitation scenarios based on the Unity development platform and the action recognition technology of Leap Motion equipment, the user's hand movements are captured through the action recognition algorithm, the similarity between the recognized action and the preset rehabilitation action is compared, feedback and suggestions are generated, and voice assistants and mobile phones are combined. Assistant provides interactive and personalized training.
Improve patient engagement and rehabilitation effects, reduce loneliness and stress, improve rehabilitation experience through gamification and social interaction, and solve the problem of insufficient accessibility of equipment and experts, especially providing effective rehabilitation treatments in remote areas.
Smart Images

Figure CN119007925A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a rehabilitation training method and system based on motion recognition and virtual reality, belonging to the technical field of artificial intelligence. Background Art
[0002] In the current hand rehabilitation industry, the recovery process after hand trauma is often long and boring, which makes patients feel lonely and stressed during the rehabilitation process. Traditional rehabilitation methods usually rely on repetitive physical therapy exercises, which lack fun and are difficult to stimulate patients' continued participation, which may not only affect patients' rehabilitation progress, but also increase their psychological pressure, making them feel lonely and stressed during the rehabilitation process; In addition, a major pain point faced by the rehabilitation industry is the limited accessibility of professional rehabilitation equipment and experts, especially in remote areas. Summary of the invention
[0003] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a rehabilitation training method and system based on motion recognition and virtual reality.
[0004] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.
[0005] On the one hand, the present invention provides a rehabilitation training method based on motion recognition and virtual reality, comprising: Build virtual rehabilitation scenes based on the Unity development platform; Using a Leap Motion device to capture the user's hand motion data, performing motion recognition on the hand motion data based on a motion recognition algorithm, and simulating the recognized motion in a virtual rehabilitation scene according to the result of the motion recognition; comparing the identified movement with the preset rehabilitation movement to determine the movement similarity; The rehabilitation effect is determined according to the action similarity, and feedback and suggestions are generated for the user based on the rehabilitation effect.
[0006] Furthermore, the construction of a virtual rehabilitation scene based on the Unity development platform includes: Based on the Unity development platform, build multiple virtual rehabilitation scenes with different sizes, shapes, colors, lighting conditions, and add different 3D models; A scene switching window is set in the Unity editor of the Unity development platform, and the constructed multiple virtual rehabilitation scenes are reduced and set in the scene switching window. The reduced virtual rehabilitation scenes in the scene switching window are switched by scrolling the mouse, and the reduced virtual rehabilitation scenes after switching are selected by clicking the mouse for main display in the Unity editor.
[0007] Furthermore, the method of capturing the user's hand motion data using a Leap Motion device and performing motion recognition on the hand motion data based on a motion recognition algorithm includes: Use the Leap Motion device to capture the user's hand movement data; Filter and normalize the captured user's hand motion data to obtain stable data; Using a deep learning model to extract features from the stable data, the user's hand posture, finger bending angle features, hand position in space, hand direction, hand speed, and hand self-occlusion information are obtained; Machine learning or pattern matching methods are used to classify the user's hand posture, finger bending angle characteristics, hand position in space, hand direction, hand speed, and hand self-occlusion information in advance to identify the user's rehabilitation movements.
[0008] Furthermore, comparing the identified action with the preset rehabilitation action to determine the action similarity includes: adding the recognized action and the preset rehabilitation action to the first action sequence and the second action sequence respectively; Obtain the lengths of the first action sequence and the second action sequence, and store the obtained lengths as variables n and m respectively; Create a two-dimensional array dtw for storing the calculation results of dynamic time warping DTW, wherein the size of the two-dimensional array dtw is n rows and m columns; Initialize all elements of the first column and the first row of the two-dimensional array dtw except the first element to positive infinity, indicating that the action sequence at these positions has not yet started; Set the first element of the two-dimensional array dtw to 0, indicating that the similarity between the two action sequences at the beginning is 0; Use two nested for loops to traverse each element of the first action sequence and the second action sequence respectively. In each loop, calculate the absolute difference between the current element of the first action sequence and the current element of the second action sequence as the cost of the current position; Add the cost cost to the minimum value of the left, top and top left positions in the two-dimensional array dtw, and use it as the dtw value of the current position; Returns the last element of the two-dimensional array dtw as the final similarity between the two action sequences.
[0009] Furthermore, the acquisition of the preset rehabilitation action includes: Use Leap Motion device to capture Chinese martial arts hand motion data, the motion data includes hand position, speed, acceleration information; Filter and normalize the captured motion data for preprocessing; According to the preprocessed action data, each action is decomposed into a series of key frames, each of which contains the position, speed, and acceleration information of the hand; Calculate the digital representation of each key frame based on the position, velocity, and acceleration information of the hand in each key frame; The digital representation of each key frame is stored in a database of the virtual rehabilitation scene, so that during rehabilitation training, the action corresponding to the corresponding key frame can be retrieved from the database and displayed to the user in the virtual rehabilitation scene.
[0010] Furthermore, between decomposing each action into a series of key frames and calculating the digital representation of each key frame according to the position, speed and acceleration information of the hand in each key frame, the method further includes: performing interpolation processing on the key frame of each action to generate a continuous action sequence; The interpolation process is expressed as: ; in, M ( t ) is at time t The action posture, P i ( t ) is the i The interpolation function of key frames, K i It is i Keyframe poses.
[0011] Furthermore, it also includes: collecting voice data of the customer, identifying the voice command and voice feedback of the customer through the voice data, wherein the voice command is used to control the operation in the virtual rehabilitation scene, and the voice feedback is used to provide the customer with an evaluation of the rehabilitation training; It also includes: projecting the user's mobile phone screen onto a display device of a virtual rehabilitation scene through wireless communication, and directly operating and using the user's mobile phone on the rehabilitation device through a Bluetooth connection.
[0012] Furthermore, the method further includes: storing the hand motion data, the steps are as follows: Create a temporary file m_temp and open the stream m_stream of the temporary file m_temp for reading and writing operations; Use the temporary file m_temp to receive the data stream of the hand motion data and the pre-stored path of the data stream, check whether the length of the data stream exceeds the preset length, if it exceeds, convert the path into bytes, and obtain the path length and data length; write the path length, path, data length and data to the file stream in sequence; read the data of the specified length from the file stream into the buffer for storage; Traverse all records in the file stream. For each record, first read the length and path of the path, then read the length and data of the data, and finally return a key-value pair containing the path and data stream. Determine whether all data has been stored based on the key-value pair. If so, release the resources of the file stream and temporary file. If not, continue reading the key-value pair.
[0013] In a second aspect, the present invention provides a rehabilitation training system based on motion recognition and virtual reality, comprising: Virtual scene development module, used to develop and design various rehabilitation training scenes in the Unity environment; The user motion recognition module is used to capture the user's hand and finger movements using Leap Motion technology, convert the user's hand and finger movements into corresponding movements in the virtual environment, and transmit these data to the main control module in real time; The rehabilitation action development module is used to design a series of rehabilitation actions according to martial arts hand movements, and transmit the requirements and standards of the rehabilitation actions to the main control module and the rehabilitation action display module; The virtual assistant module includes a voice assistant and a mobile assistant. The voice assistant is used for real-time voice interaction between the user and the system to provide instructions and feedback. The mobile assistant is used to project the mobile phone screen onto the display device of the virtual rehabilitation scene through Bluetooth technology. The rehabilitation action display module is used to present the residual image of the standard action according to the requirements and standards of the rehabilitation action development module. The user can perform rehabilitation training by imitating the standard action and overlapping the residual image in the virtual scene. The main control module is used to receive data and information from other modules, coordinate the work of all modules, analyze the user's motion data in real time, perform rehabilitation assessment based on these data, control the transformation of rehabilitation scenes, control the presentation of rehabilitation motion display modules, and control the operation of rehabilitation data recording modules and data analysis modules.
[0014] Furthermore, the main control module includes: a game manager; The game manager is used to: define private serialized fields, including task manager, game status, items and events, and the serialized fields are used to store game status and data; The task manager is used to manage tasks in the game; The game state is used to indicate the current state of the game; The item is used to represent an item in the game, and is added to the inventory when a corresponding event is triggered; The event is used to represent an event in the game; Call the start game program to start the game, update the game status to in-game, and start the task manager task; Add OnEnable function or OnDisable function to each event as required, so that the OnEnable function or OnDisable function is called when the event occurs.
[0015] The beneficial effects achieved by the present invention are: The invention not only improves the patient's participation through gamified rehabilitation exercises, but also helps alleviate their loneliness and stress by providing opportunities for social interaction. It has a positive impact on improving rehabilitation outcomes and the quality of life of patients. Access through remote or mobile technology can solve the problem of limited accessibility of professional rehabilitation equipment and experts, especially in remote areas. This will enable more patients to receive effective rehabilitation treatment, thereby further improving the service level of the rehabilitation industry. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 It is a schematic diagram of the main control module directing and coordinating the various sub-modules in the system through a series of commands and data streams; Figure 2 It is part of the software workflow, which mainly includes the process of user rehabilitation data processing, storage, analysis and feedback; Figure 3 It is a schematic diagram of the entire rehabilitation process and background data processing, storage, analysis and feedback from the user's perspective; Figure 4 is a schematic diagram of a virtual assistant module realizing projection of a user's personal mobile phone onto a user's viewing screen; Figure 5 It is a schematic diagram of the rehabilitation assessment module cooperating with other modules to realize the assessment function; Figure 6 It is a schematic diagram of the storage, retrieval and display of rehabilitation data; Figure 7 It is a schematic diagram of the working principle of the virtual scene development module; Figure 8 This is a schematic diagram of how Leap Motion obtains user rehabilitation data. DETAILED DESCRIPTION
[0017] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0018] Embodiment 1, as Figure 3 , Figure 5 and Figure 8As shown, this embodiment introduces a rehabilitation training method based on motion recognition and virtual reality, including: The virtual rehabilitation scene is built based on the Unity development platform. The specific process is as follows: Figure 7 As shown; The user's hand motion data is captured using a Leap Motion device, and motion recognition is performed on the hand motion data based on a motion recognition algorithm. The recognized motion is simulated in a virtual rehabilitation scene according to the result of the motion recognition. The specific process is as follows: Figure 8 As shown; comparing the identified movement with the preset rehabilitation movement to determine the movement similarity; The rehabilitation effect is determined according to the action similarity, and feedback and suggestions for the user are generated based on the rehabilitation effect. The specific process is as follows Figure 5 shown.
[0019] The construction of a virtual rehabilitation scene based on the Unity development platform includes: Based on the Unity development platform, build multiple virtual rehabilitation scenes with different sizes, shapes, colors, lighting conditions, and add different 3D models; A scene switching window is set in the Unity editor of the Unity development platform, and the constructed multiple virtual rehabilitation scenes are reduced and set in the scene switching window. The reduced virtual rehabilitation scenes in the scene switching window are switched by scrolling the mouse, and the reduced virtual rehabilitation scenes after switching are selected by clicking the mouse for main display in the Unity editor.
[0020] The method of capturing the hand motion data of the user using the Leap Motion device and performing motion recognition on the hand motion data based on a motion recognition algorithm includes: Use the Leap Motion device to capture the user's hand movement data; Filter and normalize the captured user's hand motion data to obtain stable data; Using a deep learning model to extract features from the stable data, the user's hand posture, finger bending angle features, hand position in space, hand direction, hand speed, and hand self-occlusion information are obtained; Machine learning or pattern matching methods are used to classify the user's hand posture, finger bending angle characteristics, hand position in space, hand direction, hand speed, and hand self-occlusion information in advance to identify the user's rehabilitation movements.
[0021] The comparing the identified action with the preset rehabilitation action to determine the action similarity includes: adding the recognized action and the preset rehabilitation action to the first action sequence and the second action sequence respectively; Obtain the lengths of the first action sequence and the second action sequence, and store the obtained lengths as variables n and m respectively; Create a two-dimensional array dtw for storing the calculation results of dynamic time warping DTW, wherein the size of the two-dimensional array dtw is n rows and m columns; Initialize all elements of the first column and the first row of the two-dimensional array dtw except the first element to positive infinity, indicating that the action sequence at these positions has not yet started; Set the first element of the two-dimensional array dtw to 0, indicating that the similarity between the two action sequences at the beginning is 0; Use two nested for loops to traverse each element of the first action sequence and the second action sequence respectively. In each loop, calculate the absolute difference between the current element of the first action sequence and the current element of the second action sequence as the cost of the current position; Add the cost cost to the minimum value of the left, top and top left positions in the two-dimensional array dtw, and use it as the dtw value of the current position; Returns the last element of the two-dimensional array dtw as the final similarity between the two action sequences.
[0022] The acquisition of the preset rehabilitation action includes: Use Leap Motion device to capture Chinese martial arts hand motion data, the motion data includes hand position, speed, acceleration information; Filter and normalize the captured motion data for preprocessing; According to the preprocessed action data, each action is decomposed into a series of key frames, each of which contains the position, speed, and acceleration information of the hand; Calculate the digital representation of each key frame based on the position, velocity, and acceleration information of the hand in each key frame; The digital representation of each key frame is stored in a database of the virtual rehabilitation scene, so that during rehabilitation training, the action corresponding to the corresponding key frame is retrieved from the database and displayed to the user in the virtual rehabilitation scene, such as Figure 2 .
[0023] Between decomposing each action into a series of key frames and calculating the digital representation of each key frame according to the position, speed and acceleration information of the hand in each key frame, the method further includes: interpolating the key frames of each action to generate a continuous action sequence; The interpolation process is expressed as: ; in, M ( t ) is at time t The action posture, P i ( t ) is the i The interpolation function of key frames, K i It is i Keyframe poses.
[0024] It also includes: collecting the customer's voice data, identifying the customer's voice commands and voice feedback through the voice data, the voice commands are used to control the operations in the virtual rehabilitation scene, and the voice feedback is used to provide the customer with an evaluation of the rehabilitation training.
[0025] It also includes: projecting the user's mobile phone screen onto a display device of a virtual rehabilitation scene through wireless communication, and directly operating and using the user's mobile phone on the rehabilitation device through a Bluetooth connection, such as Figure 4 .
[0026] The method also includes: storing the hand motion data, the steps are as follows: Create a temporary file m_temp and open the stream m_stream of the temporary file m_temp for reading and writing operations; Use the temporary file m_temp to receive the data stream of the hand motion data and the pre-stored path of the data stream, check whether the length of the data stream exceeds the preset length, if it exceeds, convert the path into bytes, and obtain the path length and data length; write the path length, path, data length and data to the file stream in sequence; read the data of the specified length from the file stream into the buffer, store it, and store it in Figure 6 In the database; Traverse all records in the file stream. For each record, first read the length and path of the path, then read the length and data of the data, and finally return a key-value pair containing the path and data stream. Determine whether all data has been stored based on the key-value pair. If so, release the resources of the file stream and temporary file. If not, continue reading the key-value pair.
[0027] Example 2 is based on the same inventive concept as Example 1. This example introduces a rehabilitation training system based on motion recognition technology and virtual reality technology, which is suitable for rehabilitation training after hand trauma. Figure 1 As shown, the system includes six modules: virtual scene development module, user action recognition module, rehabilitation action development module, virtual assistant module, rehabilitation action display module and main control module.
[0028] The system realizes an immersive rehabilitation training environment by optimizing the motion recognition algorithm and designing the rehabilitation assessment algorithm, combining the design and development of virtual rehabilitation scenes, and the digital integration of traditional Chinese martial arts hand movements. The system modules include user motion recognition, rehabilitation motion display, personalized user image simulation, etc., aiming to provide an interactive rehabilitation training experience. The innovation lies in the application of the motion overlap algorithm, which improves the accuracy of motion recognition and scientifically quantifies the rehabilitation effect through the rehabilitation assessment algorithm. The advantage of this invention is that it improves user participation and provides a new technological solution for hand trauma rehabilitation.
[0029] (1) Virtual scene development module. This module develops and designs various Chinese garden rehabilitation training scenes in the Unity environment to provide a rich and diverse training environment.
[0030] (2) User action recognition module: This module uses Leap Motion technology to capture the user's hand and finger movements, converts the user's actual movements into corresponding movements in the virtual environment, and transmits this data to the main control module in real time.
[0031] (3) Rehabilitation movement development module. This module designs a series of rehabilitation movements based on rehabilitation training methods such as Chinese martial arts hand movements, and transmits the requirements and standards of the rehabilitation movements to the main control module and the rehabilitation movement display module.
[0032] (4) Virtual assistant module. This module consists of two parts: voice assistant and mobile assistant. The voice assistant can realize real-time voice interaction between users and the system, provide instructions and feedback; the mobile assistant uses Bluetooth technology to project the mobile phone screen into the VR environment, allowing users to see the virtual interface of their personal mobile phone without using their personal mobile phone, such as Figure 4 .
[0033] (5) Rehabilitation action display module. This module presents the afterimages of standard actions according to the requirements and data of the rehabilitation action development module. Users can perform rehabilitation training by imitating these standard actions or trying to overlap with these afterimages in the virtual scene.
[0034] (6) Main control module. This module is responsible for receiving data and information from other modules, coordinating the work of all modules, analyzing the user's motion data in real time, and conducting rehabilitation assessments based on these data, controlling the transformation of rehabilitation scenes, controlling the presentation of rehabilitation motion display modules, and controlling the operation of rehabilitation data recording modules and data analysis modules.
[0035] It also includes: a user image simulation module. The user image simulation module is controlled by the main control module and can simulate and display the user image corresponding to the actual action in the virtual space, so that the user can observe whether his rehabilitation action meets the preset standard. The module includes a display device and a graphics processor. The main function of the display device is to display the three-dimensional model of the user's limbs. The graphics processor is responsible for generating a corresponding three-dimensional model of the user image based on the user's hand posture scatter plot collected by the Leap Motion camera.
[0036] It also includes: a rehabilitation assessment module. The rehabilitation assessment module receives the user's hand posture information from the user action recognition module, as well as the voice instructions and rehabilitation action requirements and standards transmitted by the main control module, and is used to comprehensively evaluate the user's rehabilitation actions. The module includes a scoring algorithm and a feedback mechanism. The scoring algorithm calculates the user's rehabilitation action score based on the user's hand posture information, voice instructions, and rehabilitation action requirements and standards. The feedback mechanism provides the user with corresponding voice and graphic feedback based on the user's rehabilitation action score to motivate the user to improve the rehabilitation action.
[0037] It also includes: a rehabilitation action data recording module. The rehabilitation action data recording module is responsible for recording the user's hand posture data and the similarity data between the user's action and the standard action during the entire rehabilitation process, so as to conduct long-term tracking and analysis of the rehabilitation effect. The module includes a data storage device and a data transmitter. The main function of the data storage device is to store various rehabilitation data of the user, including but not limited to hand posture data and action similarity data. The data transmitter is responsible for transmitting the user's rehabilitation data to the main control module or other devices for further processing and analysis.
[0038] It also includes: data analysis module. The function of this data analysis module is to conduct in-depth analysis of rehabilitation data, generate detailed rehabilitation reports, and provide personalized rehabilitation suggestions. The data analysis module includes a data processor and a data display. The data processor is responsible for conducting a comprehensive and in-depth analysis of the user's rehabilitation process and effect based on the user's rehabilitation data, using advanced data mining and machine learning technologies, thereby generating a detailed rehabilitation report and providing targeted rehabilitation suggestions. The data display is responsible for presenting the rehabilitation report and suggestions to the user or doctor in the form of intuitive charts or clear texts so that they can understand and adopt them.
[0039] The virtual rehabilitation scene development module is responsible for selecting or generating virtual objects and tasks suitable for the user, as well as the appearance of the teaching afterimage according to the user's rehabilitation action requirements and standards. The user action recognition module is responsible for calculating the user's hand flexibility and strength according to the user's hand movements, and transmitting the calculation results to the main control module. The virtual assistant module is responsible for analyzing the user's emotions and needs according to the user's voice instructions, and transmitting the analysis results to the main control module. The rehabilitation action display module is responsible for adjusting the actions of the teaching afterimage according to the user's feedback and ratings to adapt to the user's rehabilitation level and preferences.
[0040] The rehabilitation action development module can be responsible for automatically or manually adjusting the difficulty and sequence of rehabilitation actions according to the user's rehabilitation needs and personal preferences. This flexible adjustment mechanism is intended to optimize the user's rehabilitation effect and experience.
[0041] Example 3, this example introduces the design process in detail for the content of Example 2 and Example 3. In the design and development of the virtual rehabilitation scene, we first use the Unity development platform to design and create the virtual rehabilitation scene. Unity is a powerful cross-platform 3D game development tool, through which we can create a rich and realistic 3D environment. In this environment, users can perform various rehabilitation training.
[0042] When designing the virtual rehabilitation scene, we took into account the user's comfort and interactivity. We created an immersive environment in which users can move and explore freely. In addition, we designed various elements and objects that users can interact with to make the training more interesting. In order to enable users to perform rehabilitation training in the virtual environment, we used Leap Motion technology to capture and recognize the user's movements. Leap Motion is a contact-free hand motion capture technology that can accurately capture and recognize the user's hand movements. In the development of the virtual rehabilitation scene, we also took into account the personalized needs of rehabilitation training. We designed a set of digital solutions for Chinese martial arts hand movements, and users can choose appropriate movements for training according to their needs and abilities. In general, the design and development of our virtual rehabilitation scene aims to provide a comfortable, interactive, and personalized rehabilitation training environment to help users perform rehabilitation training more effectively. We believe that through our efforts, users will be able to achieve the best rehabilitation results while enjoying their training.
[0043] In the optimization and implementation of the motion recognition algorithm, we used an improved algorithm based on Leap Motion technology to improve the accuracy and efficiency of motion recognition. The following is a detailed description of the algorithm: First, we use the Leap Motion device to capture the user's hand movements. These motion data include information such as the position, direction, and speed of each finger and the rest of the hand. Then, we input these motion data into our motion recognition algorithm.
[0044] Our action recognition algorithm mainly consists of two parts: feature extraction and action classification.
[0045] In the feature extraction stage, we first preprocess the raw motion data, including denoising and normalization. Then, we use a deep learning-based method to extract motion features. Specifically, we use a recurrent neural network model called long short-term memory (LSTM). The formula of the LSTM model is as follows: ; in, f t , i t , o t are the activation values of the forget gate, input gate, and output gate, respectively. c t is the cell state, h t is the hidden state, x t is the input, W and b are weights and biases, σ is the sigmoid function, * represents element multiplication, [ h t-1 , x t ]express h t-1 and x t connection.
[0046] In the action classification stage, we use a machine learning model called support vector machine (SVM). The formula of the SVM model is as follows: ; in, x are input features, y i It is i The categories of training samples, ( x i , x ) is the kernel function, α i is the Lagrange multiplier, bis the bias and sign is the sign function. In this way, our motion recognition algorithm can accurately identify the user's rehabilitation motions and compare them with the predefined Chinese martial arts hand motions, thereby achieving precise motion recognition and evaluation. The advantage of this method is that it can process a large amount of motion data and can adapt to a variety of different users and motion types. In addition, it can provide motion recognition and evaluation results in real time, thereby helping users to adjust their rehabilitation motions in a timely manner and improve the rehabilitation effect.
[0047] In the design and application of rehabilitation evaluation algorithms, we focus on how to scientifically quantify the rehabilitation effect. This involves accurately tracking and evaluating the user's rehabilitation movements in order to provide targeted feedback and suggestions. Our rehabilitation evaluation algorithm mainly includes the following four steps: motion data collection, motion data processing, rehabilitation movement evaluation, and rehabilitation effect feedback: The first step is to collect motion data. First, we use Leap Motion technology to capture the user's motion data. This data includes information such as hand position, speed, acceleration, etc. Figure 8 .
[0048] The second step is motion data processing. Then, we preprocess the collected motion data, including filtering, normalization and other operations, for subsequent analysis.
[0049] The third step is rehabilitation action evaluation. Next, we use the action coincidence algorithm to compare the user's actions with the preset rehabilitation actions. This algorithm can quantify the similarity between the user's actions and the preset actions, thereby evaluating the user's rehabilitation progress.
[0050] Specifically, if we represent the user's action as a vector , the preset rehabilitation action is represented as a vector , then their similarity can be calculated by cosine similarity: ; in, represents the dot product of vectors, and They represent the magnitude of the vectors respectively.
[0051] Step 4: Feedback on rehabilitation effects. Finally, we provide feedback based on the evaluation results. If the user's movements are highly similar to the preset movements, it means that the user's rehabilitation progress is good; if the similarity is low, it means that the user needs to improve his movements.
[0052] In this way, our rehabilitation assessment algorithm can scientifically quantify the rehabilitation effect and provide users with targeted feedback and suggestions, thereby improving the effectiveness and efficiency of rehabilitation. At the same time, this method also increases user participation and makes rehabilitation training more interesting and challenging.
[0053] In the digitization and integration of Chinese martial arts hand movements, we mainly focus on how to convert Chinese martial arts hand gestures into digital movements that can be performed in a virtual environment and integrate these movements into our rehabilitation system.
[0054] Our digitization and integration of Chinese martial arts hand movements mainly includes the following four steps: The first step is to collect motion data. First, we use Leap Motion technology to capture the data of Chinese martial arts hand movements. This data includes information such as hand position, speed, acceleration, etc.
[0055] The second step is motion data processing. Then, we preprocess the collected motion data, including filtering, normalization and other operations, for subsequent analysis.
[0056] The third step is to digitize the motion data. Next, we convert the processed motion data into digital motion. Specifically, we decompose each motion into a series of key frames, each of which contains information such as the position, velocity, and acceleration of the hand and the hand. Then, we use the following formula to calculate the digital representation of each key frame: ; in, POS i , vel i and acc i Respectively represent i The position, velocity and acceleration of each keyframe.
[0057] Step 4: Motion data integration. Finally, we integrate the digitized motions into our rehabilitation system. Specifically, we store the digitized representation of each motion in a database, and then during rehabilitation training, we can retrieve the corresponding motion from the database and present it to the user in a virtual environment.
[0058] In this way, the digitization and integration of our Chinese martial arts hand movements can transform Chinese martial arts hand movements into digital movements that can be performed in a virtual environment, thereby improving the effectiveness and efficiency of rehabilitation training. At the same time, this method also increases user participation, making rehabilitation training more interesting and challenging. This is one of the innovations and advantages of our invention.
[0059] The following is a detailed explanation of eleven system modules, including the virtual scene development module, user motion recognition module, and rehabilitation motion development module.
[0060] 1. Virtual scene development module. It is mainly responsible for creating an immersive rehabilitation environment. The design and development of this module is based on the Unity development platform, which uses its powerful 3D graphics processing capabilities and physics engine to build a realistic virtual rehabilitation scene.
[0061] In the virtual scene development module, we first need to define the basic parameters of the scene, including the size, shape, color, and lighting conditions of the scene. Then, we can add various 3D models to the scene, such as plants and buildings in the forest, to enhance the realism and immersion of the scene. In addition, we can also add some special effects, such as particle systems and water surface reflections, to enhance the visual effects of the scene.
[0062] In the process of scene design, we need to take into account the characteristics and needs of rehabilitation training. For example, we can design some specific rehabilitation tasks, such as grasping objects, completing actions, etc., and then set corresponding goals and obstacles in the scene. In addition, we can also dynamically adjust the difficulty and complexity of the scene according to the user's rehabilitation progress and ability.
[0063] In the process of scene development, we also need to consider the interactive design. We can use Leap Motion technology to realize gesture recognition and tracking, so that users can directly operate in the virtual environment with their hands. In addition, we can also design some feedback mechanisms, such as visual prompts, sound effects, etc., to enhance the user's sense of participation and satisfaction.
[0064] Here is a Unity editor window that shows how to switch scenes: Step 1: Define the enumeration type `ScenesSource`, including two options `Assets` and `BuildSettings`, which are used to specify the source of the scene.
[0065] Step 2: Define `scrollPosition` of `Vector2` type to control the position of the scroll bar.
[0066] Step 3: Define `scenesSource` of `ScenesSource` type. The default value is `Assets`, which is used to store the currently selected scene source.
[0067] Step 4: Define `openSceneMode` of `OpenSceneMode` type, with the default value of `Single`, which is used to store the currently selected scene opening mode.
[0068] Step 5: Define the static method `Init`, which is called when the `Tools / Scene Switcher` menu item is clicked to open the scene switching window.
[0069] Step 6: Define the virtual method `OnGUI`, which is called when the GUI event of the Unity editor occurs, and is used to draw the interface of the scene switching window. In this method, first get all the built scenes and scenes in the resource library, then list all the scenes in the scroll view, and add a button for each scene. If the button is clicked, the `Open` method will be called to open the corresponding scene. Finally, add a `Create New Scene` button to create a new scene.
[0070] Step 7: Define the virtual method `Open`, which accepts a path as a parameter and is used to open the scene of the specified path. Before opening the scene, it will check whether the current unnamed scene has been saved. If it has not been saved, a prompt will pop up to let the user choose whether to save it.
[0071] The innovation of this method is that it provides a function to switch scenes directly in the Unity editor. In the traditional development process, developers need to manually select and open scenes in the Unity menu, which may be cumbersome and time-consuming. This method provides a dedicated scene switching window, allowing developers to see all scenes in one place and open scenes directly by clicking a button, greatly improving development efficiency.
[0072] In addition, this method also solves some pain points in the industry. For example, when developers need to frequently switch between different scenes, the traditional method may cause developers to repeatedly search for scenes in the menu, while this method provides a scene list so that developers can quickly find and open scenes. At the same time, this method also provides the function of creating new scenes, allowing developers to complete the creation and switching of scenes in the same window, further improving work efficiency.
[0073] The advantages of this method are mainly reflected in the following aspects: Improve efficiency: By providing a dedicated scene switching window, developers can quickly switch between different scenes without having to repeatedly search for scenes in the menu.
[0074] Easy to use: The interface design of this method is simple and clear. Developers can open or create scenes through intuitive button operations without having to remember complicated operation steps.
[0075] Flexibility: This method supports selecting scenes from the resource library or build settings, and choosing single scene or multi-scene mode, meeting different development needs.
[0076] 2. User motion recognition module. It is mainly responsible for capturing and recognizing the user's rehabilitation movements. The design and implementation of this module is based on Leap Motion technology. Through its precise motion capture capabilities, it can obtain the user's hand position and movement information in real time.
[0077] In the user motion recognition module, we first need to initialize and configure the Leap Motion device, including device connection, calibration, parameter settings, etc. Then, we can obtain the user's hand raw data, including position, speed, direction, etc., through the device's API interface.
[0078] Next, we need to process and analyze these raw data to identify the user's rehabilitation movements. This process usually includes the following four steps: The first step is preprocessing. The raw data is filtered and normalized to eliminate noise and equipment errors and improve the stability and reliability of the data.
[0079] The second step is feature extraction. According to the characteristics and requirements of the rehabilitation movements, meaningful feature parameters are extracted, such as hand posture, finger bending angle, etc.
[0080] The third step is action recognition. Using machine learning or pattern matching methods, the user's rehabilitation actions are identified based on feature parameters. This process may require the use of some algorithms, such as support vector machines, hidden Markov models, etc.
[0081] The fourth step is action feedback. Based on the recognition results, the user is provided with real-time feedback, such as whether the action is correct or not, the degree of action completion, etc.
[0082] In this process, we may need to use some formulas, such as the Euclidean distance formula, the cosine similarity formula, etc., to calculate feature parameters or evaluate the similarity of movements. The specific formulas and algorithms may vary depending on the specific type and needs of the rehabilitation movement.
[0083] The following is the principle used to calculate the similarity of two action sequences: Step 1: First, we define a class called `DTW`, which contains a static method called `Compute`. This method accepts two parameters, `seq1` and `seq2`, which are arrays of type `double`, representing two action sequences respectively.
[0084] Step 2: In the `Compute` method, we first get the length of `seq1` and `seq2` and store them in variables `n` and `m` respectively.
[0085] Step 3: Then, we create a two-dimensional array named `dtw` with a size of `n` rows and `m` columns to store the calculation results of Dynamic Time Warping (DTW).
[0086] Step 4: Next, we initialize all elements in the first column (except the first element) and the first row (except the first element) of the `dtw` array to positive infinity, indicating that the action sequence at these positions has not yet started.
[0087] Step 5: We set the first element of the `dtw` array (i.e., `dtw[0, 0]`) to 0, indicating that the similarity between the two action sequences at the beginning is 0.
[0088] Step 6: Then, we use two nested `for` loops to traverse each element of `seq1` and `seq2` respectively. In each loop, we calculate the absolute difference between the current elements of `seq1` and `seq2` as the cost (`cost`) of the current position.
[0089] Step 7: We add this cost to the minimum value of the three positions on the left, above, and above the left in the `dtw` array as the `dtw` value of the current position. This step implements the core idea of the DTW algorithm, which is to find the optimal alignment of two action sequences through dynamic programming.
[0090] Step 8: Finally, we return the last element of the `dtw` array (i.e., `dtw[n - 1, m - 1]`) as the final similarity between the two action sequences.
[0091] This method uses the dynamic time warping (DTW) algorithm to calculate the similarity of two action sequences, which has important application value in the design of human action recognition and feedback modules. The following are the innovations of this method compared to existing methods, as well as the pain points and advantages it solves in the industry: Innovation: 1. Dynamic Time Warping (DTW): This method uses the DTW algorithm, which is an algorithm that can measure the similarity between two time series. The DTW algorithm can handle situations where the lengths and time offsets of the two time series are different³, which has great advantages when dealing with non-linear action sequences.
[0092] Pain points solved: 1. Matching of nonlinear time series: In human action recognition, a common problem is that the action sequence may produce nonlinear changes due to different execution speeds. The DTW algorithm can effectively handle this problem, thereby improving the accuracy of action recognition.
[0093] 2. Handling speed changes: The DTW algorithm can handle changes in the speed at which actions are performed. Even if one person moves faster than another, or there are accelerations and decelerations during observation, DTW can detect similarities in walking.
[0094] Advantages: 1. Wide range of application areas: The DTW algorithm has been applied to time series of video, audio and graphic data. It has applications in many fields such as speech recognition, data mining and financial markets.
[0095] 2. Robustness: The DTW algorithm is very robust to outliers, which gives it an advantage when processing data containing noise.
[0096] 3. Handling time series of different lengths and time offsets: The DTW algorithm can handle situations where the lengths and time offsets of two time series are different, which makes it very flexible in dealing with practical problems.
[0097] In general, this method effectively solves some key problems in human action recognition by using the DTW algorithm, such as matching nonlinear time series and processing speed changes, thereby improving the accuracy and robustness of action recognition. At the same time, the wide application field of the DTW algorithm and its ability to process time series of different lengths and time offsets also make this method have great application potential and value.
[0098] 3. Rehabilitation movement development module. Mainly responsible for the programming implementation of Chinese martial arts hand movements. The design and implementation of this module is based on the Unity development platform and Leap Motion technology. In this module: First, we need to digitize the hand movements of Chinese martial arts. This includes capturing and recording the key frames of each movement, and then generating a continuous sequence of movements through interpolation technology. This process can be expressed by the following formula: ; in, M ( t ) is at time t The action posture, P i ( t ) is the i The interpolation function of key frames, K i It is i The pose of the keyframe. Interpolation function P i ( t ) can be linear interpolation, cubic spline interpolation, etc. The specific choice needs to be determined according to the characteristics and requirements of the action.
[0099] Then, we need to use Leap Motion technology to achieve real-time motion capture and recognition. This includes real-time tracking of the user's hand movements, and then using the motion recognition algorithm to determine which Chinese martial arts hand movement the user is currently performing. This process can be expressed by the following formula: ; in, A ( t ) is at time t The actions identified, F is the action recognition function, M ( t ) is at time t The action posture, D ( t ) is at time t action data.
[0100] Finally, we need to compare the recognized movements with the preset Chinese martial arts hand movements, and then evaluate the user's rehabilitation effect through the rehabilitation evaluation algorithm. This process can be expressed by the following formula: ; in, E ( t ) is at time t The results of rehabilitation effect evaluation were G is the rehabilitation assessment function, T ( t ) is at time t target action.
[0101] 4. Virtual assistant module. This module is an important part of this system. It mainly includes two sub-modules: voice assistant and mobile assistant. The specific functions and interaction methods are as follows: (I) Voice assistant. Its main function is to achieve natural language interaction with users through voice recognition and speech synthesis technology. Users can control the operation of the system through voice commands, such as starting or ending rehabilitation training, adjusting the difficulty of training, etc. At the same time, the voice assistant can also guide users to correctly complete rehabilitation movements through voice feedback, provide real-time training feedback, and summarize and evaluate the user's rehabilitation effect after the training.
[0102] (ii) Mobile Assistant. It is a supporting mobile phone application that can be connected to the main system via Bluetooth. The main function of the mobile phone assistant is to project the mobile phone screen onto the rehabilitation display device, so that users can directly operate and use their personal mobile phones on the rehabilitation device without picking up the phone. In this way, users can conveniently use their mobile phones to perform other operations such as answering calls, replying to messages, browsing the web, etc. while undergoing rehabilitation training. In addition, the mobile phone assistant can also provide some auxiliary functions, such as adjusting the brightness and contrast of the projection, selecting the content of the projection (for example, only projecting the demonstration of rehabilitation movements, or projecting the movement demonstration and training data at the same time), and controlling the start and end of the projection. These functions can all be operated through the projection screen on the rehabilitation device, and the operation is simple and convenient.
[0103] When necessary, the virtual assistant module can also use some algorithms to improve the effect of interaction. For example, we can use the confidence of speech recognition to determine whether the user's voice command is clear. If the confidence is lower than a certain threshold, the system can prompt the user to reissue the command. This threshold can be expressed as: ; in, is the average confidence of all voice commands received by the system over a period of time, α and β It is an adjustment parameter that can be adjusted according to the actual situation of the system. In this way, we can ensure accurate command recognition while avoiding erroneous operations caused by misrecognition.
[0104] 5. Rehabilitation action display module. It is mainly responsible for guiding users to perform rehabilitation actions and providing action demonstrations. The following is a detailed explanation of this module: The main task of this module is to provide users with accurate demonstrations of rehabilitation movements and guide users to perform these movements correctly. It includes the following four key parts: 1. Action Demonstration. This module displays pre-programmed Chinese martial arts hand movements so that users can clearly see the correct way to perform each movement. These action demonstrations are displayed through high-precision 3D models to simulate real hand movements as much as possible.
[0105] (ii) User guidance. In addition to action demonstrations, this module also provides real-time guidance and feedback when users perform actions. For example, if the user's actions do not match the demonstration actions, the system will guide the user to adjust their actions through visual and sound prompts.
[0106] (iii) Action synchronization. To ensure that users can keep up with the rhythm of the action demonstration, the module also includes an action synchronization function. This function works by comparing the time sequence of the user's action and the demonstration action. If the user's action lags behind the demonstration action, the system will remind the user to speed up their action.
[0107] (IV) Movement difficulty adjustment. This module can adjust the difficulty of movements according to the user's rehabilitation progress and ability. For example, for beginner users, the system may display simplified versions of movements, while for more advanced users, the system may display more complex or challenging movements.
[0108] The design and implementation of this module are based on the Unity development platform and Leap Motion technology, both of which provide strong support for high-precision gesture capture and real-time user feedback. In addition, this module also utilizes our optimized motion recognition algorithm to improve the accuracy and efficiency of motion recognition.
[0109] 6. Main control module. It is responsible for coordinating and managing the operations of all other modules. The following is a detailed explanation of this module from four aspects.
[0110] (I) System operation. The main control module is responsible for processing user inputs, such as gestures and voice commands, and converting these inputs into system operations. For example, if the user selects a specific rehabilitation action through gestures, the main control module will instruct the rehabilitation action display module to display the action.
[0111] (ii) User Interface. The master control module is also responsible for managing the user interface, including menus, buttons, and other interactive elements. It ensures that the layout and design of the user interface are intuitive and user-friendly so that users can easily navigate and use the system.
[0112] (III) Module coordination. The main control module is also responsible for coordinating the operations of other modules. For example, when a user starts a rehabilitation action, the main control module will instruct the user action recognition module to start capturing the user's action, and at the same time instruct the rehabilitation action display module to display the corresponding action demonstration.
[0113] (IV) Error handling. If an error occurs in the system, such as a failure in motion recognition or the inability of the rehabilitation assessment algorithm to calculate the assessment result, the main control module will be responsible for handling these errors and providing corresponding feedback to the user.
[0114] The design and implementation of the main control module are based on the Unity development platform and Leap Motion technology, both of which provide strong support for efficient system operation and user interface. In addition, the main control module also utilizes our optimized motion recognition algorithm and rehabilitation assessment algorithm to improve system performance and user experience.
[0115] The following is the principle of the game manager in the main control module: Step 1: Define variables In the GameManager class, first define some private serialized fields, including task manager, game status, items, and events. These fields are mainly used to store game status and data.
[0116] The quest manager is used to manage quests in the game. In this example, the quest manager is a QuestManagerSO object, which is serialized and stored in the _questManager field.
[0117] The game state is or is used to represent the current state of the game. In this example, the game state is a GameStateSO object, which is serialized and stored in the _gameState field.
[0118] The items are or are used to represent items in the game. For example, items are ItemSO objects, which are serialized and stored in the _rockCandyRecipe, _sweetDoughRecipe, _finalRecipes, and _inventory fields.
[0119] The event is or is used to represent an event in the game. For example, events are VoidEventChannelSO objects, which are serialized and stored in the _addMartialArtsSecerts field. These events are triggered in the game to add items to the inventory or update the game state. For example, when the _addMartialArtsSecerts (add martial arts secrets) event is triggered, the AddMartialArtsSecerts method is called to add the _MartialArts item to the inventory. Listeners for these events are added in the OnEnable method and removed in the OnDisable method to ensure that listening for events starts when the game object is enabled and stops when the game object is disabled. This is a common event listening and handling pattern that ensures that events are handled correctly during the life cycle of the game object.
[0120] Step 2: Game Start In the Start method, the StartGame method is called to start the game. The StartGame method updates the game state to Gameplay and starts the game start method of the task manager.
[0121] Step 3: Event monitoring In the OnEnable method, add monitoring methods for each event. For example, when the AaddMartialArtsSecertsEvent event is triggered, the AddMartialArtsSecerts method will be called to add the martial arts secret book to the inventory.
[0122] Step 4: Cancel event monitoring In the OnDisable method, all event monitoring methods are canceled to prevent null reference exceptions caused by events still referencing these methods after the object is disabled or destroyed.
[0123] Step 5: Add items In the AddMartialArtsSecerts method, add the corresponding items to the inventory. These methods will be called when the corresponding events are triggered.
[0124] This algorithm enables users to obtain new martial arts secrets during the rehabilitation process, that is, unlock new rehabilitation movements. Compared with existing algorithms, the advantages of this method and the pain points it solves are mainly reflected in the following aspects: Advantage 1: Modular design This method adopts a modular design, encapsulating the game manager, task manager, game status, items and events in different classes or objects, making the code structure clear and easy to maintain and expand. This design method is also conducive to team collaborative development, and each developer can focus on the module he is responsible for, improving development efficiency.
[0125] Advantage 2: Event-driven This method uses an event-driven approach to process game logic. For example, the operation of adding an item to the inventory is triggered by a corresponding event. This approach makes the code more flexible and can dynamically add or remove event listeners according to the actual needs of the game without modifying the original code.
[0126] Advantage 3: Serialized Objects This method uses serialized objects (SerializedObject), so that the values of these objects can be modified directly in the Unity editor without writing additional code. This greatly simplifies the game development process and improves development efficiency.
[0127] Pain points solved: Code complexity: Through modular and event-driven design, this approach effectively reduces the complexity of the code, making the code easier to understand and maintain.
[0128] Development efficiency: Through serialized objects and event-driven design, this method improves the efficiency of game development and reduces duplication of work in the development process.
[0129] Code coupling: This method reduces the coupling between codes through modular design, allowing each module to be developed and tested independently, improving the reusability of the code.
[0130] 7. User image simulation module. The main function of this module is to generate a virtual user image based on the user's characteristics and actions, providing users with a personalized rehabilitation experience. The image can perform various rehabilitation actions in the virtual rehabilitation scene, giving users a more intuitive sense of participation and immersion.
[0131] The implementation of this module mainly relies on the Unity development platform and Leap Motion technology. Unity provides a wealth of 3D models and animation functions that can be used to create and control virtual user images. Leap Motion provides precise hand motion capture functions that can be used to drive the movements of virtual user images. Specifically, the workflow of the user image simulation module is as follows: First, create a model. Use Unity's 3D model function to create a basic user image model. This model can be adjusted according to the user's height, weight and other characteristics to achieve a personalized effect.
[0132] Secondly, motion capture. Use Leap Motion's motion capture function to capture the user's hand movements. These motion data will be transmitted to Unity in real time to drive the movements of the virtual user image.
[0133] Then, the action is simulated. In Unity, the virtual user image is controlled to perform corresponding actions based on the captured action data. This process needs to be optimized through some algorithms to ensure the smoothness and accuracy of the virtual action.
[0134] Finally, the image display shows the virtual user image and its movements in the virtual rehabilitation scene, allowing the user to see his or her movements and rehabilitation effects.
[0135] The key to this module is the accuracy of motion capture and motion simulation. To improve the accuracy, we can use some optimization algorithms, such as Kalman filter, to smooth the motion data. The specific formula is as follows: Assumptions x t yes t The true state of the moment, z t yes t The observation value at the moment, then the workflow of the Kalman filter can be expressed by the following formula: Prediction steps: Prediction status: ; Prediction covariance: ; Update steps: Calculate the Kalman gain: ; Update status: ; Update the covariance: ; in, F is the state transition matrix, B is the control matrix, u t-1 is the control vector, H is the observation matrix, Q is the process noise covariance, R is the observation noise covariance, K t is the Kalman gain, P t is the state covariance.
[0136] In this way, the user image simulation module can provide users with a real and smooth rehabilitation experience, thereby improving the rehabilitation effect and user satisfaction. At the same time, this module also provides important data support for other modules (such as rehabilitation assessment module and data analysis module).
[0137] 8. Rehabilitation assessment module. It is mainly responsible for monitoring and providing feedback on rehabilitation effects. The design and application of this module is based on the rehabilitation assessment algorithm, which can scientifically quantify the rehabilitation effect. In terms of specific implementation, the rehabilitation assessment module may include the following five steps: The first step is to collect motion data. When the user performs rehabilitation movements, the system accurately captures the user's motion data through the user motion recognition module.
[0138] The second step is data processing. The collected motion data will be preprocessed, such as denoising and normalization, to facilitate subsequent evaluation.
[0139] The third step is rehabilitation effect evaluation. The system calculates the quantitative indicators of rehabilitation effect based on the pre-processed motion data through the rehabilitation evaluation algorithm. This indicator may include the accuracy, continuity, speed and other aspects of the movement.
[0140] The fourth step is feedback generation. Based on the evaluation results, the system will generate corresponding feedback information, including text, charts, etc., to intuitively demonstrate the rehabilitation effect.
[0141] Step 5: Feedback display. The system displays feedback information to users to help them understand their rehabilitation progress and effects.
[0142] The rehabilitation assessment module may use some formulas when necessary. For example, suppose we used Indicates the accuracy of the user's action, c Indicates continuity, s Indicates speed, then the quantitative index of rehabilitation effect E It can be calculated by the following formula: ; in, w d , w c and w s are weighting factors used to adjust the importance of each indicator in the overall assessment. These weighting factors can be adjusted according to specific rehabilitation goals and the user's personal situation.
[0143] 9. Rehabilitation action data recording module. It is responsible for recording and storing all the action data of the user during the rehabilitation process. The main functions of this module include three aspects.
[0144] 1. Progress tracking. By recording the user's action data for each rehabilitation training, the system can track the user's rehabilitation progress. This data may include the degree of completion, completion time, completion frequency, etc.
[0145] (ii) Data storage. All movement data will be securely stored in the database. This way, users can easily access this data whenever they need to review their rehabilitation history.
[0146] (III) Data update. Every time a user completes a rehabilitation training, this module will update the data in the database. This ensures the real-time and accuracy of the data.
[0147] If necessary, we can use the following formula to calculate the completion of the action: ; The overlap between the actual action and the target action can be calculated by the action overlap algorithm, and the total amount of the target action is preset. This formula can help us quantify the effect of each rehabilitation training, so as to better track the user's rehabilitation progress.
[0148] The principle of data storage and display module: The method is a class called `RestoreHandlerMetadataStorage`, which is under the `Duplicati.Library.Main.Operation` namespace. The main function of this class is to handle the storage of metadata. The following are the main steps of this class: Step 1: Initialization In the constructor of the `RestoreHandlerMetadataStorage` class, a temporary file `m_temp` is created and the stream `m_stream` of this file is opened for reading and writing operations.
[0149] Step 2: Add data In the `Add` method, a path and a data stream are received. First, check whether the length of the data stream exceeds `Int32.MaxValue`, and throw an exception if it does. Then, convert the path to bytes and get the path length and data length. Next, write the path length, path, data length, and data to the file stream in sequence. Finally, check whether the total size written is correct, and throw an exception if it is not correct.
[0150] Step 3: Read data In the `CheckedRead` method, read the specified length of data from the file stream into the buffer. If the read length does not match the specified length, an exception is thrown.
[0151] Step 4: Get the records In the `Records` property, iterate over all the records in the file stream. For each record, first read the length of the path and the path, then read the length of the data and the data. Finally, return a key-value pair containing the path and the data stream.
[0152] Step 5: Release resources In the `Dispose` method, release the resources of the file stream and temporary file.
[0153] The innovation and advantages of this method are mainly reflected in the following aspects: 1. Use of temporary files: This method uses temporary files to store metadata, which avoids storing a large amount of metadata in memory and reduces memory usage.
[0154] 2. Data streaming: When adding and reading data, it is done in a streaming manner, which can process large amounts of data without loading all the data into memory at once.
[0155] 3. Error checking: Error checking is performed when writing and reading data. If the length of the data does not meet the expectations, an exception will be thrown, so that errors can be discovered and handled in time.
[0156] 4. Release of resources: In the `Dispose` method of the class, the resources of file streams and temporary files are released to avoid waste of resources.
[0157] This method solves the problems in the industry when processing large amounts of metadata, such as excessive memory usage, low data processing efficiency, untimely error handling, and waste of resources.
[0158] 10. Data analysis module. Responsible for statistics and analysis of rehabilitation results. Specifically, the main functions and implementation methods of this module are as follows:
[0159] (I) Data collection. This module will collect various data generated by the user during the rehabilitation process, including but not limited to the accuracy of the action execution, the time to complete, the number of times each action is completed, etc.
[0160] (ii) Data processing. The collected raw data needs to be processed before it can be used for subsequent analysis. For example, we may need to calculate the average completion time of each action, or count the number of actions completed by users within a certain period of time.
[0161] (III) Data analysis. The processed data will be used for various analyses to help us understand the user's rehabilitation progress. For example, we can evaluate the user's rehabilitation effect by analyzing the accuracy of their action execution, or observe the user's rehabilitation progress by comparing data from different time periods.
[0162] (iv) Data visualization. In order to help users better understand their rehabilitation progress, we will also display the analysis results in the form of charts. For example, we can use a line chart to show the user's rehabilitation progress, or use a bar chart to compare the completion of different movements.
[0163] (V) Data feedback. Finally, we will provide the analysis results to users to help them understand their own recovery status and adjust their recovery plans as needed.
[0164] In the process of data analysis, we may use some statistical methods and formulas. For example, we may use mean and standard deviation to describe the central tendency and dispersion of data. The specific formula is as follows: Mean: ; Standard Deviation: ; in, N is the amount of data, x i is each data point, μ is the mean, σ is the standard deviation. These statistics can help us better understand the characteristics of the data and provide a basis for subsequent analysis. In actual implementation, we may also use other statistical methods and machine learning algorithms to improve the accuracy and efficiency of data analysis.
[0165] The action coincidence algorithm is an algorithm used to compare user actions with preset rehabilitation actions. Its main goal is to determine whether the user has performed the rehabilitation action correctly by calculating the similarity between the user action and the preset action. The working principle of the algorithm is as follows: First, motion representation. We need to represent the motion in a form that can be mathematically operated. In this system, we use Leap Motion technology to capture the user's hand motions and convert them into a series of three-dimensional coordinate points.
[0166] Second, action alignment. Since the speed and amplitude of each person performing the same action may be different, we need to normalize the action in time and space so that different actions can be compared under the same standard.
[0167] Third, action comparison. Compare the similarity between the user action and the preset action. This can be achieved by calculating the Euclidean distance between the two actions or other similarity measurement methods.
[0168] Fourth, action evaluation. Based on the similarity of actions, we can evaluate the accuracy of the user's action. If the similarity exceeds a certain threshold, then we believe that the user has performed the action correctly.
[0169] The key to this algorithm lies in how to accurately capture and represent actions, and how to define a suitable similarity measurement method. This requires a lot of experiments and data for optimization and verification.
[0170] Mathematically, suppose we have two action sequences A and B, both of which consist of n three-dimensional coordinate points, namely , We can define the similarity S(A,B) between action A and action B as: ; in, and Action A and action B are i Coordinate points x , y , z This formula actually calculates the average Euclidean distance between action A and action B, and subtracts this distance from 1 to get the similarity. When two actions are exactly the same, their similarity is 1; when two actions are completely different, their similarity is 0.
[0171] The rehabilitation assessment algorithm is mainly based on the following formula: ; in,N is the total number of rehabilitation training and the accuracy of the movements i , Movement speed i and action frequency i They are i The formula can quantitatively evaluate the user's rehabilitation effect and thus achieve scientific rehabilitation evaluation.
[0172] Regarding the user engagement improvement part, this is a key innovation because it can improve user engagement through interactive rehabilitation training. Here are some possible methods and corresponding formulas: First, real-time feedback. By providing real-time feedback on the user's movements and rehabilitation progress, user engagement can be improved. For example, after the user completes each movement, feedback can be provided to the user immediately through voice or graphics to let the user know their performance and progress. This feedback can be quantified through some indicators, such as the accuracy of the movement, the speed of completion, etc.
[0173] Second, personalized experience. By providing a personalized rehabilitation experience, user engagement can be improved. For example, rehabilitation movements and scenarios can be customized according to user preferences and needs. This personalized experience can be quantified through some indicators, such as user satisfaction, user engagement, etc.
[0174] Third, gamification elements. By introducing gamification elements, user participation can be improved. For example, some goals and challenges can be set so that users can feel the fun of the game while completing rehabilitation training. This gamification element can be quantified through some indicators, such as user points, user rankings, etc.
[0175] The above methods can be quantified by some formulas. For example, user engagement can be calculated by the following formula: ; The number of completed actions refers to the number of actions completed by the user within a certain period of time, and the total number of actions refers to the total number of actions designed by the system for the user within this period of time.
[0176] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0177] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0178] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0180] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A rehabilitation training method based on motion recognition and virtual reality, characterized in that: include: Build virtual rehabilitation scenes based on the Unity development platform; Using a Leap Motion device to capture the user's hand motion data, performing motion recognition on the hand motion data based on a motion recognition algorithm, and simulating the recognized motion in a virtual rehabilitation scene according to the result of the motion recognition; comparing the identified movement with the preset rehabilitation movement to determine the movement similarity; The rehabilitation effect is determined according to the action similarity, and feedback and suggestions are generated for the user based on the rehabilitation effect.
2. The rehabilitation training method based on motion recognition and virtual reality according to claim 1, characterized in that: The construction of a virtual rehabilitation scene based on the Unity development platform includes: Based on the Unity development platform, build multiple virtual rehabilitation scenes with different sizes, shapes, colors, lighting conditions, and add different 3D models; A scene switching window is set in the Unity editor of the Unity development platform, and the constructed multiple virtual rehabilitation scenes are reduced and set in the scene switching window. The reduced virtual rehabilitation scenes in the scene switching window are switched by scrolling the mouse, and the reduced virtual rehabilitation scenes after switching are selected by clicking the mouse for main display in the Unity editor.
3. The rehabilitation training method based on motion recognition and virtual reality according to claim 1, characterized in that: The method of capturing the hand motion data of the user using the Leap Motion device and performing motion recognition on the hand motion data based on a motion recognition algorithm includes: Use the Leap Motion device to capture the user's hand movement data; Filter and normalize the captured user's hand motion data to obtain stable data; Using a deep learning model to extract features from the stable data, the user's hand posture, finger bending angle features, hand position in space, hand direction, hand speed, and hand self-occlusion information are obtained; Machine learning or pattern matching methods are used to classify the user's hand posture, finger bending angle characteristics, hand position in space, hand direction, hand speed, and hand self-occlusion information in advance to identify the user's rehabilitation movements.
4. The rehabilitation training method based on motion recognition and virtual reality according to claim 1, characterized in that: The comparing the identified action with the preset rehabilitation action to determine the action similarity includes: adding the recognized action and the preset rehabilitation action to the first action sequence and the second action sequence respectively; Obtain the lengths of the first action sequence and the second action sequence, and store the obtained lengths as variables n and m respectively; Create a two-dimensional array dtw for storing the calculation results of dynamic time warping DTW, wherein the size of the two-dimensional array dtw is n rows and m columns; Initialize all elements of the first column and the first row of the two-dimensional array dtw except the first element to positive infinity, indicating that the action sequence at these positions has not yet started; Set the first element of the two-dimensional array dtw to 0, indicating that the similarity between the two action sequences at the beginning is 0; Use two nested for loops to traverse each element of the first action sequence and the second action sequence respectively. In each loop, calculate the absolute difference between the current element of the first action sequence and the current element of the second action sequence as the cost of the current position; Add the cost cost to the minimum value of the left, top and top left positions in the two-dimensional array dtw, and use it as the dtw value of the current position; Returns the last element of the two-dimensional array dtw as the final similarity between the two action sequences.
5. The rehabilitation training method based on motion recognition and virtual reality according to claim 1, characterized in that: The acquisition of the preset rehabilitation action includes: Use Leap Motion device to capture Chinese martial arts hand motion data, the motion data includes hand position, speed, acceleration information; Filter and normalize the captured motion data for preprocessing; According to the preprocessed action data, each action is decomposed into a series of key frames, each of which contains the position, speed, and acceleration information of the hand; Calculate the digital representation of each key frame based on the position, velocity, and acceleration information of the hand in each key frame; The digital representation of each key frame is stored in a database of the virtual rehabilitation scene, so that during rehabilitation training, the action corresponding to the corresponding key frame can be retrieved from the database and displayed to the user in the virtual rehabilitation scene.
6. The rehabilitation training method based on motion recognition and virtual reality according to claim 5, characterized in that: Between decomposing each action into a series of key frames and calculating the digital representation of each key frame according to the position, velocity and acceleration information of the hand in each key frame, it also includes: interpolating the key frames of each action to generate a continuous action sequence; The interpolation process is expressed as: ; in, M ( t ) is at time t The action posture, P i ( t ) is the i The interpolation function of key frames, K i It is i Keyframe poses.
7. The rehabilitation training method based on motion recognition and virtual reality according to claim 1, characterized in that: The method further includes: collecting voice data of the client, and identifying voice commands and voice feedback of the client through the voice data, wherein the voice commands are used to control operations in the virtual rehabilitation scene, and the voice feedback is used to provide the client with an evaluation of the rehabilitation training; It also includes: projecting the user's mobile phone screen onto a display device of a virtual rehabilitation scene through wireless communication, and directly operating and using the user's mobile phone on the rehabilitation device through a Bluetooth connection.
8. The rehabilitation training method based on motion recognition and virtual reality according to claim 1, characterized in that: Also includes: To store hand motion data, the steps are as follows: Create a temporary file m_temp and open the stream m_stream of the temporary file m_temp for reading and writing operations; Use the temporary file m_temp to receive the data stream of the hand motion data and the pre-stored path of the data stream, check whether the length of the data stream exceeds the preset length, if it exceeds, convert the path into bytes, and obtain the path length and data length; write the path length, path, data length and data to the file stream in sequence; read the data of the specified length from the file stream into the buffer for storage; Traverse all records in the file stream. For each record, first read the length and path of the path, then read the length and data of the data, and finally return a key-value pair containing the path and data stream. Determine whether all data has been stored based on the key-value pair. If so, release the resources of the file stream and temporary file. If not, continue reading the key-value pair.
9. A rehabilitation training system based on motion recognition and virtual reality, characterized in that: include: Virtual scene development module, used to develop and design various rehabilitation training scenes in the Unity environment; The user motion recognition module is used to capture the user's hand and finger movements using Leap Motion technology, convert the user's hand and finger movements into corresponding movements in the virtual environment, and transmit these data to the main control module in real time; The rehabilitation action development module is used to design a series of rehabilitation actions according to martial arts hand movements, and transmit the requirements and standards of the rehabilitation actions to the main control module and the rehabilitation action display module; The virtual assistant module includes a voice assistant and a mobile assistant. The voice assistant is used for real-time voice interaction between the user and the system to provide instructions and feedback. The mobile assistant is used to project the mobile phone screen onto the display device of the virtual rehabilitation scene through Bluetooth technology. The rehabilitation action display module is used to present the residual image of the standard action according to the requirements and standards of the rehabilitation action development module. The user can perform rehabilitation training by imitating the standard action and overlapping the residual image in the virtual scene. The main control module is used to receive data and information from other modules, coordinate the work of all modules, analyze the user's motion data in real time, perform rehabilitation assessment based on these data, control the transformation of rehabilitation scenes, control the presentation of rehabilitation motion display modules, and control the operation of rehabilitation data recording modules and data analysis modules.
10. The rehabilitation training system based on motion recognition and virtual reality according to claim 9, characterized in that: The main control module includes: a game manager; The game manager is used to: define private serialized fields, including task manager, game status, items and events, and the serialized fields are used to store game status and data; The task manager is used to manage tasks in the game; The game state is used to indicate the current state of the game; The item is used to represent an item in the game, and is added to the inventory when a corresponding event is triggered; The event is used to represent an event in the game; Call the start game program to start the game, update the game status to in-game, and start the task manager task; Add OnEnable function or OnDisable function to each event as required, so that the OnEnable function or OnDisable function is called when the event occurs.
Citation Information
Cited By
Gesture interaction method, device and equipment based on desktop three-dimensional interaction all-in-one machine
CN119271053A
VR management system and method for preventing venous thromboembolism, electronic equipment and storage medium
CN119943264A
GMP drug production non-contact virtual-real interactive simulation system and method
CN120510750A
Hand function rehabilitation training system and method integrating multi-sensory feedback and virtual-real interaction
CN121287165A
