Rehabilitation training method, system and computer readable storage medium

Abstract

The present invention discloses a rehabilitation training method, system and computer-readable storage medium, the rehabilitation training method comprising: using an ordinary monocular camera to obtain a virtual training scene and the user's motion data, the virtual training scene including the target position of the motion; estimating the user's current three-dimensional position and the user's motion intention in real time according to the acquired user's motion data, so as to obtain the current estimated value of the three-dimensional position and the intention recognition result; predicting the three-dimensional position at the next moment according to the current estimated value of the three-dimensional position and the intention recognition result, and constructing the user's motion model according to the predicted three-dimensional position at the next moment, so as to update and render the virtual training scene in real time, and display the virtual training scene. The present invention provides a low-cost rehabilitation training method that can be built and used in a home environment, has non-contact interaction capabilities, can guide rehabilitation training in real time, so as to meet the needs of later home rehabilitation training.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality interaction technology, and in particular to a rehabilitation training method, system and computer-readable storage medium. Background Art

[0002] Stroke rehabilitation is generally divided into three stages: early, middle and late stages. Corresponding rehabilitation training is carried out in hospitals, rehabilitation institutions and home environments. In the late stage of home rehabilitation training, there is a lack of on-site guidance from rehabilitation therapists, making it difficult to make timely evaluation and feedback on the patient's rehabilitation training. In addition, the rehabilitation process is not only boring, but also the enthusiasm for active participation is low, resulting in poor rehabilitation results. Summary of the invention

[0003] The main purpose of the present invention is to provide a rehabilitation training method, system and computer-readable storage medium, aiming to meet the needs of late-stage home rehabilitation training.

[0004] To achieve the above object, the present invention provides a rehabilitation training method, which comprises:

[0005] A common monocular camera is used to obtain a virtual training scene and the user's motion data, wherein the virtual training scene includes a motion target posture;

[0006] estimating the current three-dimensional posture of the user and the user's movement intention in real time according to the acquired movement data of the user, so as to obtain a current estimated value of the three-dimensional posture and an intention recognition result;

[0007] The three-dimensional posture at the next moment is predicted according to the current estimated value of the three-dimensional posture and the intention recognition result, and the user's motion model is constructed according to the predicted three-dimensional posture at the next moment to update and render the virtual training scene in real time, and display the virtual training scene.

[0008] Optionally, the rehabilitation training method further includes:

[0009] The three-dimensional posture of the limb end in the current estimated value of the three-dimensional posture is used as the starting point, and the target posture is used as the target point. A teaching symbol is generated according to the starting point and the target point, and is displayed in the virtual training scene.

[0010] Optionally, before the step of acquiring the virtual training scene and the user's motion data, the rehabilitation training method further includes:

[0011] The stored historical training data is obtained, and the user's training status is evaluated according to the historical training data and a status evaluation report is generated, so that the user can select a corresponding virtual training scene according to the status evaluation report.

[0012] Optionally, the step of estimating the current three-dimensional posture of the user and the user's movement intention in real time according to the acquired movement data of the user to obtain a current estimated value of the three-dimensional posture and an intention recognition result specifically includes:

[0013] Acquire, according to the motion data of the user, a two-dimensional key point distribution of the user during motion;

[0014] The current three-dimensional posture of the user is estimated according to the distribution of two-dimensional key points when the user moves and a generative adversarial network to obtain a current estimated value of the three-dimensional posture.

[0015] Optionally, the step of estimating the current three-dimensional posture of the user and the user's movement intention in real time according to the acquired movement data of the user to obtain a current estimated value of the three-dimensional posture and an intention recognition result specifically includes:

[0016] estimating the user's three-dimensional posture during movement according to the acquired movement data of the user to obtain a three-dimensional posture sequence;

[0017] The user's movement intention is estimated based on the user's three-dimensional posture sequence and a recurrent neural network to obtain an intention recognition result.

[0018] Optionally, after the step of estimating the current three-dimensional posture of the user and the user's movement intention in real time according to the acquired movement data of the user to obtain a current estimated value of the three-dimensional posture and an intention recognition result, the rehabilitation training method further includes:

[0019] storing the current three-dimensional pose of the user estimated in real time as historical training data;

[0020] The historical training data is sent to an external storage device.

[0021] Optionally, the step of obtaining the virtual training scene and the user's motion data includes:

[0022] Image information of a user exercising is acquired, and image processing is performed on the image information to obtain exercise data of the user.

[0023] Optionally, the number of target postures of the movement is multiple;

[0024] Before the step of obtaining the virtual training scene and the user's motion data, the rehabilitation training method further includes:

[0025] The user's rehabilitation status is obtained, and one that is adapted to the user's rehabilitation status is selected from a plurality of target postures of the movement, and is added to the virtual training scene.

[0026] The present invention also proposes a rehabilitation training system, which includes a processor, a memory, and a rehabilitation training program stored in the memory and executable on the processor, wherein the rehabilitation training program implements the steps of the rehabilitation training method described above when executed by the processor.

[0027] Optionally, the rehabilitation training system further includes:

[0028] An image acquisition device, electrically connected to the processor, for acquiring image information of the user during exercise and outputting the image information to the processor;

[0029] A display device is electrically connected to the processor and is used to display the virtual training scene output by the processor.

[0030] The present invention further provides a computer-readable storage medium, on which a rehabilitation training program is stored. When the rehabilitation training program is executed by a processor, the steps of the rehabilitation training method described above are implemented.

[0031] The present invention obtains a virtual training scene and the user's motion data, wherein the virtual training scene includes a target posture of the motion, and estimates the user's current three-dimensional posture and the user's motion intention in real time based on the acquired motion data of the user, so as to obtain a current estimated value of the three-dimensional posture and an intention recognition result, thereby predicting the three-dimensional posture at the next moment based on the current estimated value of the three-dimensional posture and the intention recognition result, and constructing a motion model of the user based on the predicted three-dimensional posture at the next moment, so as to update and render the virtual training scene in real time, and display the virtual training scene. The virtual scene constructed by the present invention includes the user's avatar and teaching character, which can predict the user's movements and display them synchronously. The present invention is a low-cost rehabilitation training method that can be built and used in a home environment, has non-contact interaction capabilities, can guide rehabilitation training in real time, and can meet the needs of later home rehabilitation training. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0033] Figure 1 It is a flow chart of an embodiment of a rehabilitation training method of the present invention;

[0034] Figure 2 It is a flowchart of another embodiment of the rehabilitation training method of the present invention;

[0035] Figure 3 A schematic diagram of a flow chart of another embodiment of the rehabilitation training method of the present invention;

[0036] Figure 4 for Figure 3 A detailed flow chart of step S200 in an embodiment;

[0037] Figure 5 for Figure 3 A detailed flow chart of another embodiment of step S200;

[0038] Figure 6 A schematic diagram of a flow chart of another embodiment of the rehabilitation training method of the present invention;

[0039] Figure 7 It is a structural schematic diagram of an embodiment of a rehabilitation training system of the present invention;

[0040] Figure 8 A schematic diagram of the distribution of key points of two-dimensional posture of a human body involved in the rehabilitation training method of the present invention;

[0041] Fig. 9 A schematic diagram of a generative adversarial network for three-dimensional posture estimation involved in the rehabilitation training method of the present invention;

[0042] Fig.10 A schematic diagram of a recurrent network for intention recognition in the rehabilitation training method of the present invention;

[0043] Fig.11 A schematic diagram of posture prediction and teaching symbols related to the rehabilitation training method of the present invention;

[0044] Fig.12 It is a schematic diagram of the terminal structure of the hardware operating environment of the rehabilitation training device involved in the embodiment of the present invention.

[0045] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0048] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0049] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0050] The invention provides a rehabilitation training method.

[0051] With the advent of an aging society, the number of stroke patients has increased year by year. Stroke can easily cause hemiplegia, and patients need long-term rehabilitation training. Early, mid-term and late-stage rehabilitation training are carried out in hospitals, rehabilitation institutions and home environments respectively. At present, there is a lack of upper and lower limb auxiliary training equipment suitable for late-stage home rehabilitation training. In addition, during the late-stage home rehabilitation training process, there is a lack of on-site guidance from rehabilitation therapists, making it difficult to make timely evaluations and feedback on the patient's rehabilitation training. Existing rehabilitation training equipment needs to collect the patient's movement information and is equipped with a large number of wearable sensor devices, which brings many inconveniences to patients' use in a home environment. In addition, existing rehabilitation training equipment is generally only applicable to hospitals or rehabilitation institutions, requires special personnel to operate, has high maintenance costs, and is difficult to meet the requirements of home rehabilitation training.

[0052] To solve the above problems, refer to Figures 1 to 12 In one embodiment of the present invention, the rehabilitation training method includes:

[0053] Step S100, obtaining a virtual training scene selected by a user and the user's motion data, wherein the virtual training scene includes a target position of the motion;

[0054] In this implementation, there are multiple virtual training scenes, and different application scenes can be provided according to the needs of users. For example, corresponding application scenes can be set for the user's upper limb rehabilitation training needs, or corresponding application scenes can be set for the user's lower limb rehabilitation training needs, or corresponding application scenes can be set for the user's whole body. Specifically, virtual scenes such as cutting fruit, fishing, billiards, and simulated driving can be provided for the user's upper limb rehabilitation training needs. For lower limb rehabilitation training, virtual scenes such as walking, going up and down stairs, and kicking a ball can be provided. For example, driving training games and fishing master training games can be used to complete upper limb horizontal rehabilitation training; shark eating small fish training games can be used to complete upper limb vertical rehabilitation training; parkour training games can be used to complete lower limb rehabilitation training. Virtual training scenes can use the Unity 3D platform to develop life scenes suitable for user rehabilitation, and at the same time design some basic life skills training in the scene as rehabilitation tasks. Rendering is used to enhance the immersion and realism of user training.

[0055] The virtual training scene can be configured based on the user's selection, for example, it can be displayed on the display device in the form of a menu, button, etc. The user can make a selection through a mouse, keyboard, touch screen, etc. Based on the user's selection, the corresponding virtual training scene is obtained and displayed. That is, the various virtual training scenes of rehabilitation training can be displayed to the user through a graphical interface display, and the user can select and set the virtual training scene by clicking or touching the mouse. Then, based on the virtual training scene selected by the user, the virtual training scene can be initialized, such as loading the virtual training scene image, and configuring other data for completing the virtual training.

[0056] In each virtual training scene, the displayed content is different. For example, when simulating driving, the virtual scene may include roads, scenery on both sides of the road, etc. When simulating kicking a ball, the virtual scene may include a stadium, an auditorium, etc., which can improve the user's experience and enable the user to integrate and immerse into the rehabilitation environment to improve the rehabilitation training effect. The user's motion data may include the user's motion state. When obtaining the user's motion state, it can be obtained at a certain frame rate. The motion state includes but is not limited to the position of the user's body joints, that is, the posture. The target posture is the target point of the standard rehabilitation training action corresponding to each virtual training scene, including the target position and posture. In the same virtual training scene, multiple target postures can be set. Each target posture can be set according to the difficulty of the user to complete the standard rehabilitation training action. Specifically, in the initial stage of the user's rehabilitation training, it is difficult to complete the standard rehabilitation training action. At this time, the target posture can be set lower. In the middle stage of the user's rehabilitation training, the completion of the standard rehabilitation training action is improved. At this time, the target posture can be set to increase accordingly. At the end of the user's rehabilitation training, it is easier to complete the standard rehabilitation training action. At this time, the target posture can be set higher. Taking arm-raising rehabilitation training as an example, when performing rehabilitation training of raising the right arm, the user's lower limbs and left arm do not need to exercise. According to the difficulty of the action, the user's right arm needs to extend the upper limb forward to a 45° angle with the body, raise the upper limb to a horizontal height, and raise the upper limb to the top of the head. Correspondingly, when the upper limb is extended forward to a 45° angle with the body, the target posture is at a 45° angle between the end of the limb and the body, when the upper limb is raised to a horizontal height, the target posture is at a 90° angle between the end of the limb and the body, and when the upper limb is raised to the top of the head, the target posture is at a 180° angle between the end of the limb and the body. The motion data can also include the user's height, gender, motion trajectory, the time it takes the user to do each action, the comparison between the user's actions and the standard actions of rehabilitation training, etc.

[0057] After the user selects the virtual training scene, the system initializes a rehabilitation training model based on virtual reality, such as a game model, and configures the rehabilitation training category, rehabilitation training mode, rehabilitation training time, rehabilitation training scene difficulty category, etc. corresponding to the virtual training scene selected by the user. The rehabilitation training mode can be an upper limb training mode, a lower limb training mode, a single-handed training mode, a double-handed training mode, and so on.

[0058] Step S200, estimating the current three-dimensional posture of the user and the user's movement intention in real time according to the acquired movement data of the user, so as to obtain a current estimated value of the three-dimensional posture and an intention recognition result;

[0059] In this embodiment, the current three-dimensional posture of the user can be based on the trunk of the body as a reference, and the current three-dimensional posture of the user can also be an absolute position and posture. Depending on the rehabilitation training that the user is performing, the current three-dimensional posture of the user is also different. For example, the three-dimensional posture when the user is performing the fruit cutting training action and the fishing fruit cutting training action are different, and according to the different progress of the training, it will also change accordingly. For example, the current three-dimensional posture and the three-dimensional posture of the previous moment, as well as the three-dimensional posture of the next moment will all change. The motion data is a collection of motion data recorded in real time at each moment, and the user's motion intention is identified based on the recorded motion data sequence, wherein the motion intention may include but is not limited to upper limb extension, upper limb extension and then lowering, upper limb lateral extension, upper limb lateral extension and then lowering, upper limb lifting, upper limb lifting and then lowering, stepping, standing, etc.

[0060] Step S300, predicting the three-dimensional posture at the next moment according to the current estimated value of the three-dimensional posture and the intention recognition result, and constructing the user's motion model according to the predicted three-dimensional posture at the next moment, so as to update and render the virtual training scene in real time, and display the virtual training scene.

[0061] In this embodiment, the three-dimensional posture at the next moment can be an action that is ahead of the current three-dimensional posture for a certain time, such as 0.5s, 1s, etc., and the action that the user will make. According to the predicted three-dimensional posture at the next moment, the inverse kinematics method of the virtual reality engine Unity can be used to construct the user's motion model, that is, the three-dimensional posture of the end of the limb is known, the three-dimensional posture of other joints of the limb is calculated and the whole limb model is established, and a virtual game character or limb is mapped in the virtual scene. The virtual limb is mapped with these three-dimensional postures at the corresponding position in the virtual scene, and its posture and relative position are consistent with the three-dimensional posture of the real user, so that the action of the limb interacts with the virtual training scene, and this is used as the user's avatar in the virtual scene to complete the designated action of rehabilitation training. Specifically, during the training process, the three-dimensional posture of each moment in the user's motion data can be converted into the update and rendering data of the virtual training scene. For example, in the rehabilitation training of simulating driving, when the user makes a left turn, the car model will turn to the left in proportion, so that the user is brought into the game, so that the user is immersed in virtual reality to complete specific rehabilitation training. The actions established in the virtual scene using the above method are ahead of the user's current actions, which can play a role in motion guidance and teaching.

[0062] The present invention obtains a virtual training scene and the user's motion data, wherein the virtual training scene includes a target posture of the motion, and estimates the user's current three-dimensional posture and the user's motion intention in real time based on the acquired motion data of the user, so as to obtain a current estimated value of the three-dimensional posture and an intention recognition result, thereby predicting the three-dimensional posture at the next moment based on the current estimated value of the three-dimensional posture and the intention recognition result, and constructing a motion model of the user based on the predicted three-dimensional posture at the next moment, so as to update and render the virtual training scene in real time, and display the virtual training scene. The virtual scene constructed by the present invention includes the user's avatar, which can predict the user's movements and display them synchronously. The present invention provides a low-cost technical solution that can be built and used in a home environment, has non-contact interaction capabilities, can guide rehabilitation training in real time, and can meet the needs of later home rehabilitation training.

[0063] Reference Figure 2 and Fig.11 In one embodiment, the rehabilitation training method further comprises:

[0064] Step S400: taking the three-dimensional posture of the limb end in the current estimated value of the three-dimensional posture as the starting point, taking the target posture as the target point, generating a teaching symbol according to the starting point and the target point, and displaying it in the virtual training scene.

[0065] It should be noted that, due to the difference between the user's motion state and the standard action, the current three-dimensional posture of the user's wrist or ankle joint is used as the starting point, and the three-dimensional posture corresponding to the standard action is used as the target point. The vector obtained by connecting the two is used as the teaching symbol. The teaching symbol can be extended to game characters such as moving objects. For example, the virtual scene teaching symbol can be an arrow, a gold coin, an apple, a small fish, or other symbols that match the ongoing rehabilitation virtual training scene. The teaching symbol will be updated and rendered in real time as the rehabilitation virtual training action proceeds, and displayed in the virtual scene, replacing the rehabilitation therapist to teach the action. In this way, the rehabilitation training method of the present invention includes action teaching and guiding roles, which can teach and correct the user's actions and play the role of on-site guidance.

[0066] Reference Figure 3 In one embodiment, before the step of obtaining the virtual training scene and the user's motion data, the rehabilitation training method further includes:

[0067] Step S500: Acquire stored historical training data, evaluate the user's training status based on the historical training data, and generate a status evaluation report, so that the user can select a corresponding virtual training scene based on the status evaluation report.

[0068] It is understandable that the actions performed by the user in each rehabilitation virtual training scene will be recorded and stored as historical training data, so as to evaluate the user's training situation of implementing rehabilitation training using the rehabilitation virtual training scene in real time and give feedback to the user. After the rehabilitation training is completed, the training data is uploaded and an evaluation report is generated. That is, the corresponding rehabilitation training is implemented according to the user's rehabilitation level. Specifically, the upper computer program can obtain the angle data during rehabilitation training in real time, interact with the virtual reality of rehabilitation training, complete real-time evaluation and instant motion feedback; and after the game is over, the upper computer program automatically analyzes the training data and generates a training result report. Specifically, when the user uses the rehabilitation game to implement limb rehabilitation training, the motion data of rotating in the left and right directions, swinging in the up and down directions, or ankle joint movement, is compared with the standard action based on the motion data, or the time used for the three-dimensional posture data at each moment is compared with the time required for the standard action. At the same time, the action curve trend chart can be drawn according to the three-dimensional posture data at each moment, etc., to generate a real-time evaluation report in the form of. The status assessment report may also include the training completion status of each action, the training completion time, whether the target action is completed, etc. In some embodiments, the user's training score can also be derived based on the acquired user's motion data, and intuitive tables, pie charts, bar charts, etc. can be generated to analyze the completion status of the target action of each rehabilitation training, as well as whether the specific upper limb and lower limb movements are coherent, whether there are any obstacles, etc.

[0069] Reference Figure 4 In one embodiment, the step of estimating the current three-dimensional posture of the user and the user's movement intention in real time based on the acquired movement data of the user to obtain the current estimated value of the three-dimensional posture and the intention recognition result specifically includes:

[0070] Step S211, obtaining the two-dimensional key point distribution of the user during the movement according to the movement data of the user;

[0071] Step S212: Estimate the current three-dimensional posture of the user according to the distribution of the two-dimensional key points when the user moves and generate an adversarial network to obtain a current estimated value of the three-dimensional posture.

[0072] In this embodiment, Figure 7 , Figure 7 It is the distribution and definition of the user's two-dimensional key points, including the head, shoulder, elbow, wrist, hip, knee and ankle, a total of 17 key points. Fig. 9As shown, the present invention specifically adopts the VGG16 network structure to estimate the two-dimensional key points of the user. The network has a total of 16 hidden layers, including 13 convolutional layers and 3 fully connected layers, the convolution kernel size is 3×3, the step size is 1, and the maximum pooling layer size is 2×2. RGB images of size 224×224 at 30 frames per second are extracted from the video captured by the monocular camera as the input of the VGG16 network, and the VGG16 network is trained and tested using the MPII data set to estimate the two-dimensional key points of the user at each moment, and obtain the result of the two-dimensional key point position estimation. This embodiment also uses a generative adversarial network to estimate the three-dimensional posture of the user at each moment, and the generative adversarial network includes a generative network G and a discriminative network D. The generative network G takes the result of the two-dimensional key point position estimation as input, generates a predicted value of the three-dimensional posture, and then projects the three-dimensional prediction result to a specific direction to simulate the observation result of the user's posture under a specific perspective, such as simulating the perspective of the user's posture control when the rehabilitation trainer is located during actual rehabilitation. The discriminant network D evaluates the accuracy of the prediction result, that is, compares the two-dimensional projection data obtained when the three-dimensional prediction result is projected to a specific direction with the real two-dimensional projection data to obtain an evaluation result, and uses the sigmoid function to feed the evaluation result back to the generative network, thereby training the generative network to generate a more realistic three-dimensional prediction result. The present invention uses Human3.6M as the user three-dimensional posture estimation training and test data set for the generative adversarial network.

[0073] Reference Figure 5 In one embodiment, the step of estimating the current three-dimensional posture of the user and the user's movement intention in real time based on the acquired movement data of the user to obtain the current estimated value of the three-dimensional posture and the intention recognition result specifically includes:

[0074] Step S221, estimating the three-dimensional posture of the user during movement according to the acquired movement data of the user to obtain a three-dimensional posture sequence;

[0075] Step S222, estimating the user's movement intention based on the user's three-dimensional posture sequence and the recurrent neural network to obtain an intention recognition result.

[0076] In this embodiment, during the user training process, the user's 3D posture during exercise at each moment is recorded and stored in real time. The user's 3D posture during exercise at multiple moments can form a 3D posture sequence, which can specifically include the user's 3D posture at the current moment and the 3D postures at different points in the past period of time. After obtaining the user's 3D posture sequence, Fig.10As shown, the present embodiment can use a recurrent neural network (RNN) to estimate the user's motion intention. In this embodiment, the recurrent neural network model can be a recurrent neural network with weights as a three-dimensional tensor, and the recurrent neural network of the three-dimensional tensor has a hidden layer, an input layer and an output layer. xh represents the weight matrix input to the hidden layer, W hh Represents the weight matrix from the previous hidden layer to the current hidden layer, W hz represents the weight matrix from hidden layer to output. t represents the input vector at time t. In this embodiment, X t is the 3D posture data at time t (specifically, it can be expressed as the 3D posture at the current time), which is used as the input of the recurrent neural network at time t. t+1 It is represented as the three-dimensional posture at the next moment. t is the movement intention at time t (specifically, it can be expressed as the movement intention at the current moment), Z t+1 is the movement intention at the next moment. There are 8 types of movement intentions, namely, upper limb extension, upper limb extension and then lowering, upper limb lateral extension, upper limb lateral extension and then lowering, upper limb raising, upper limb raising and then lowering, stepping, and standing. t is the hidden layer state vector. In this embodiment, h t It is the motion state memory information stored at time t (specifically, it can be expressed as the motion state memory information at the current moment).

[0077] Reference Figure 6 In one embodiment, after step S200, the step of estimating the current three-dimensional posture of the user and the user's movement intention in real time according to the acquired movement data of the user to obtain a current estimated value of the three-dimensional posture and an intention recognition result, the rehabilitation training method further includes:

[0078] Step S600, storing the current three-dimensional posture of the user estimated in real time as historical training data;

[0079] The historical training data is output to an external storage device.

[0080] In this embodiment, the motion data of the user at each moment during the exercise process can be synchronously stored, and the training data can be stored in the local database after the rehabilitation training is completed. The training data can also be synchronized to the cloud database, remote diagnosis and treatment system, rehabilitation training big data system, etc. through wired or wireless methods. In this way, the user himself or the rehabilitation trainer can view the user's rehabilitation training related data according to needs. For example, when stored in the local database or external storage device, the user can retrieve the historical training data stored in the local database by clicking, sliding, voice input, etc. in the display interface of the local terminal or external storage device. The historical training data can be fed back to the local terminal or external storage device in the form of a corresponding evaluation report, and the local terminal or external storage device displays the corresponding evaluation report in the display interface.

[0081] In one embodiment, the step of obtaining the virtual training scene and the user's motion data includes:

[0082] Image information of a user exercising is acquired, and image processing is performed on the image information to obtain exercise data of the user.

[0083] In this embodiment, an image acquisition device can be used to capture the user's own motion state at a certain frame rate, and send it to a processor for executing and processing the above-mentioned rehabilitation training method in real time. The processor can complete image processing and human posture estimation, update and render the virtual scene in real time, and display the virtual scene on the display device. The image acquisition device can use a monocular camera, a camera, a mobile phone, a computer and other monocular image acquisition devices for image acquisition. The present invention uses a monocular image acquisition device to perform image acquisition, which can achieve low-cost real-time recognition without GPU, binocular camera or depth camera. The required acquisition device uses a popular monocular image acquisition device. Compared with professional high-end image acquisition equipment, it can be more universally realized. Only using existing monocular image acquisition devices such as mobile phones can achieve the technical effects achieved by this solution, and the application range is wider. The present invention is based on the image information of the user's movement when the monocular image acquisition device is used, and at the same time, it is matched with the algorithm recognition of the above-mentioned rehabilitation training method, which can be realized on a low-cost hardware platform to complete the real-time recognition of the user's motion data.

[0084] In addition, in the embodiment of the invention, the image data collected by the monocular image acquisition device, the two-dimensional key point distribution when the user moves, that is, the two-dimensional key points of the human body are detected from the image information collected by the monocular image acquisition device, and the result of the two-dimensional key point position estimation is obtained through the key point description. The present invention uses monocular vision to estimate the patient's posture in real time, providing a non-contact interactive technical solution, which can avoid the complicated work of wearing equipment and is more suitable for use in a home environment.

[0085] In a specific embodiment, during the rehabilitation training process, there is usually only one person in the field of view of the monocular camera; if there are multiple people, the middle position of the field of view is taken as the area of ​​interest, and the person closest to the middle position is taken as the detection object.

[0086] In one embodiment, the number of target postures of the movement is multiple;

[0087] Before the step of obtaining the virtual training scene and the user's motion data, the rehabilitation training method further includes:

[0088] The user's rehabilitation status is obtained, and one that is adapted to the user's rehabilitation status is selected from a plurality of target postures of the movement, and is added to the virtual training scene.

[0089] In this embodiment, as the user's own body recovers and rehabilitation training proceeds, the user's limb movements will become more and more flexible. Therefore, the training difficulty will be different for the user's rehabilitation conditions at different stages, so as to achieve gradual rehabilitation training for the user. Therefore, when conducting rehabilitation training for the same limb movement in the same virtual training scene, the training difficulty will be different for the user's rehabilitation conditions, which can be specifically reflected in the differences in target posture, training duration, etc. Specifically, the difference in target posture can include the height of the target position, the degree of direct bending of the joint, etc. For example, in the initial stage, the target posture can be set to a position that is in a straight line with the end of the user's limb. In the middle stage of training, the target posture can be set to a position that is in a straight line with the end of the user's limb and is higher than the height in the initial stage. In the final stage of training, the target posture can be set to a position that is in a broken line with the end of the user's limb. For example, in the case of upper limb movement, it can be a combination of lifting + lateral movement, etc. The virtual training scene of this embodiment may include at least two rehabilitation training difficulty categories for users to choose from, and obtain the user's rehabilitation status, select one that is adapted to the user's rehabilitation status from multiple target postures of the movements, and add it to the virtual training scene, so that during the rehabilitation training process, the user can perform rehabilitation training with the actual three-dimensional posture corresponding to the target posture as the end point of the movement.

[0090] The present invention also proposes a rehabilitation training system, which includes a processor, a memory, and a rehabilitation training program stored in the memory and executable on the processor, wherein the rehabilitation training program implements the steps of the rehabilitation training method described above when executed by the processor.

[0091] Reference Figure 7 and Fig.12, the rehabilitation training system of this embodiment can be a monocular camera, a computer host, a background server, a cloud server, etc., wherein the rehabilitation training system includes: a processor 1001, such as a CPU (Central Processing Unit, central processing unit 1001), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The optional user interface 1003 can also include a standard wired interface and a wireless interface. The network interface 1004 can optionally include a standard wired interface and a wireless interface (such as Wireless Fidelity Wireless-Fidelity, Wi-Fi interface). The memory 1005 can be a high-speed RAM memory 1005, or it can be a stable memory 1005 (non-volatile memory), such as a disk memory 1005. The memory 1005 can also be a storage device independent of the aforementioned processor 1001. Those skilled in the art can understand that, Fig.12 The structure of the rehabilitation training system shown in the figure does not constitute a limitation on the rehabilitation training system, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently. The network interface 1004 is mainly used to connect to the background server and communicate data with the background server; and the processor 1001 can be used to call the rehabilitation training program stored in the memory 1005 and execute the above-mentioned rehabilitation training method. The memory 1005 can also store historical training data generated during the rehabilitation training process.

[0092] In this embodiment, the processor can perform image processing, real-time estimation of human posture, real-time generation and rendering of dynamic virtual scenes. Image processing mainly completes pre-processing work such as image filtering and image enhancement. This implementation can also be provided with an external software interface, which can be a data interface for a rehabilitation training big data platform, a remote diagnosis and treatment system, and a health cloud platform with which data is interacted. After the system is started, the user or the caregiver selects a suitable virtual training scene according to historical training data and the user's own rehabilitation situation, and the system completes the initialization of the virtual scene. Subsequently, the user's motion data is obtained based on monocular vision, and the user's human posture is estimated in real time. The user's own action sequence formed by the human posture estimation result is used as input to identify the user's own motion intention, and generate the user's own limb avatar and teaching signal in the virtual scene. The system updates the dynamic virtual scene, the user's own posture and the teaching signal in real time at a certain frequency. The three-dimensional posture information generated during the user's own training process can be stored in a local computer or cloud as historical data for training effect evaluation. The present invention adopts a real-time estimation technology of human posture based on monocular vision interaction, and by constructing a virtual reality rehabilitation training scene in real time, it has both interactive and teaching functions, and is suitable for upper and lower limb rehabilitation training in a home environment. The present invention can transfer the rehabilitation training that currently needs to be conducted in a rehabilitation institution for at least one month to a home environment, which can save the user's own hospitalization and training costs, and can significantly reduce the workload of rehabilitation therapists. It is of great significance to alleviate the problems of insufficient rehabilitation training resources and high rehabilitation costs in my country. The present invention can use household appliances such as mobile phones, televisions and home computers as system hardware, without the need to purchase additional equipment, and its cost is significantly lower than that of existing rehabilitation training equipment. In addition, the present invention can be connected with the remote diagnosis and treatment system and rehabilitation training big data system of the health care community, which helps to increase the service functions of the health care community and improve its intelligence level.

[0093] Reference Figure 7 In one embodiment, the rehabilitation training system further includes:

[0094] An image acquisition device 1007, electrically connected to the processor, for acquiring image information of the user during exercise and outputting it to the processor;

[0095] The display device 1008 is electrically connected to the processor and is used to display the virtual training scene output by the processor.

[0096] In this example, the display device 1008 can be a large-screen TV, a projection device, etc., and the image acquisition device 1007 can be a digital camera, a mobile phone camera, etc. The present invention uses monocular vision to estimate the patient's posture in real time, without setting sensors and contacting the user. It is a non-contact interactive technical solution that can avoid complicated equipment wearing work and is more suitable for use in a home environment. The virtual scene constructed by the present invention includes the patient's avatar, which can predict the patient's movements and display them synchronously through the display device 1008. In addition, the present invention can also perform action teaching and guidance on the display device 1008, and can teach and correct the patient's movements, playing the role of on-site guidance. The image acquisition device 1007 captures the patient's motion state at a certain frame rate and sends it to the processor in real time. The processor completes image processing and human posture estimation, updates and renders the virtual scene in real time, and displays the virtual scene on the display device 1008.

[0097] The present invention also provides a computer-readable storage medium, on which a rehabilitation training program is stored, and when the rehabilitation training program is executed by a processor, the steps of the rehabilitation training method described above are implemented. In the embodiment of the computer-readable storage medium provided by the present invention, all technical features of the various embodiments of the rehabilitation training method of the above-mentioned rehabilitation training device are included, and the expansion and explanation content of the specification are basically the same as those of the various embodiments of the above-mentioned method, and will not be repeated here.

[0098] The above descriptions are only optional embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings, or directly / indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A rehabilitation training method, characterized in that: The rehabilitation training method comprises: A common monocular camera is used to obtain a virtual training scene and the user's motion data, wherein the virtual training scene includes a motion target posture; estimating the current three-dimensional posture of the user and the user's movement intention in real time according to the acquired movement data of the user, so as to obtain a current estimated value of the three-dimensional posture and an intention recognition result; Predicting the three-dimensional posture at the next moment according to the current estimated value of the three-dimensional posture and the intention recognition result, and constructing a motion model of the user according to the predicted three-dimensional posture at the next moment, so as to update and render the virtual training scene in real time, and display the virtual training scene; The rehabilitation training method further comprises: Taking the three-dimensional posture of the limb end in the current estimated value of the three-dimensional posture as the starting point, taking the target posture as the target point, generating a teaching symbol according to the starting point and the target point, and displaying it in the virtual training scene; The target posture is set according to the difficulty of the user completing the standard movements of the rehabilitation training.

2. The rehabilitation training method according to claim 1, characterized in that: Before the step of obtaining the virtual training scene and the user's motion data, the rehabilitation training method further includes: The stored historical training data is obtained, and the user's training status is evaluated according to the historical training data and a status evaluation report is generated, so that the user can select a corresponding virtual training scene according to the status evaluation report.

3. The rehabilitation training method according to claim 1, characterized in that: The step of estimating the current three-dimensional posture of the user and the user's movement intention in real time according to the acquired movement data of the user to obtain a current estimated value of the three-dimensional posture and an intention recognition result specifically includes: Acquire, according to the motion data of the user, a two-dimensional key point distribution of the user during motion; The current three-dimensional posture of the user is estimated according to the distribution of two-dimensional key points when the user moves and a generative adversarial network to obtain a current estimated value of the three-dimensional posture.

4. The rehabilitation training method according to claim 1, characterized in that: The step of estimating the current three-dimensional posture of the user and the user's movement intention in real time according to the acquired movement data of the user to obtain a current estimated value of the three-dimensional posture and an intention recognition result specifically includes: estimating the user's three-dimensional posture during movement according to the acquired movement data of the user to obtain a three-dimensional posture sequence; The user's movement intention is estimated based on the user's three-dimensional posture sequence and a recurrent neural network to obtain an intention recognition result.

5. The rehabilitation training method according to claim 1, characterized in that: After the step of estimating the current three-dimensional posture of the user and the user's movement intention in real time according to the acquired movement data of the user to obtain a current estimated value of the three-dimensional posture and an intention recognition result, the rehabilitation training method further includes: storing the current three-dimensional pose of the user estimated in real time as historical training data; The historical training data is sent to an external storage device.

6. The rehabilitation training method according to claim 1, characterized in that: The step of obtaining the virtual training scene and the user's motion data comprises: Image information of a user exercising is acquired, and image processing is performed on the image information to obtain exercise data of the user.

7. The rehabilitation training method according to any one of claims 1 to 6, characterized in that: The number of target postures of the movement is multiple; Before the step of obtaining the virtual training scene and the user's motion data, the rehabilitation training method further includes: The user's rehabilitation status is obtained, and one that is adapted to the user's rehabilitation status is selected from a plurality of target postures of the movement, and is added to the virtual training scene.

8. A rehabilitation training system, characterized in that: The rehabilitation training system includes a processor, a memory, and a rehabilitation training program stored in the memory and executable on the processor, wherein the rehabilitation training program implements the steps of the rehabilitation training method according to any one of claims 1 to 7 when executed by the processor.

9. The rehabilitation training system according to claim 8, characterized in that: The rehabilitation training system also includes: A monocular camera is used as an image acquisition device, which is electrically connected to the processor to acquire image information of the user during movement and output it to the processor; A display device is electrically connected to the processor and is used to display the virtual training scene output by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a rehabilitation training program, and when the rehabilitation training program is executed by the processor, the steps of the rehabilitation training method according to any one of claims 1 to 7 are implemented.

Images