Auxiliary rehabilitation teaching system and method based on somatosensory interaction

Abstract

The invention provides an auxiliary rehabilitation teaching system and method based on somatosensory interaction, and the method comprises the steps: determining an initial motion track of a target user during somatosensory interaction, capturing a body image of the target user during somatosensory interaction through a depth camera, determining skeleton points with the positions changed in the body image, and obtaining the skeleton points with the positions changed in the body image; performing drift compensation on the initial movement track based on the dynamic drift features of all skeleton points to obtain an actual interaction movement track when the target user performs somatosensory interaction; based on a standard rehabilitation action matched with the rehabilitation training action when the target user performs somatosensory interaction, performing deviation recognition on the interaction motion track to obtain a posture deviation degree between the rehabilitation training action and the standard rehabilitation action; and generating a feedback adjustment instruction of the target user in the somatosensory interaction process based on the posture deviation degree. By adopting the scheme of the invention, the drift error of inertial sensing can be corrected through the depth camera in the somatosensory interaction process so as to improve the motion capture precision of the auxiliary rehabilitation teaching system.

Description

Technical Field

[0001] This application relates to the field of somatosensory interaction technology. More specifically, this application relates to an assisted rehabilitation teaching system and method based on somatosensory interaction. Background Art

[0002] Somatosensory interaction technology is a technology that uses natural human movements (such as gestures, body movements, expressions, and voices) to interact with a computer or intelligent device, achieving natural, contactless, and intelligent human-computer interaction through sensors, computer vision, and artificial intelligence; while the assisted rehabilitation teaching system is a new type of system that combines modern technology and rehabilitation medicine, aiming to help users carry out rehabilitation training teaching through technology to restore their physical functions; this assisted rehabilitation teaching system usually guides users to complete specific rehabilitation actions through real-time monitoring, data analysis, and personalized feedback, thereby accelerating the rehabilitation process and improving the training effect.

[0003] Existing assisted rehabilitation teaching systems use somatosensory interaction technology for motion capture, pose recognition, and real-time motion feedback of target users. However, when collecting motion data, inertial sensing has drift errors, and long-term training may lead to motion trajectory deviation, while optical sensing is affected by light, occlusion, and camera angle, which may lead to unstable motion recognition, thus affecting the rehabilitation teaching effect. Therefore, how to correct the drift error of inertial sensing through a depth camera during the somatosensory interaction process to improve the motion capture accuracy of the assisted rehabilitation teaching system has become a difficult problem faced by the industry. Summary of the Invention

[0004] This application provides an assisted rehabilitation teaching system and method based on somatosensory interaction, which can correct the drift error of inertial sensing through a depth camera during the somatosensory interaction process to improve the motion capture accuracy of the assisted rehabilitation teaching system.

[0005] In a first aspect, this application provides a feedback adjustment method based on somatosensory interaction, including: When a target user performs a rehabilitation training action of somatosensory interaction, collect the rehabilitation training action data of the target user, and determine the initial motion trajectory of the target user during somatosensory interaction according to the rehabilitation training action data; Capture the body image of the target user during somatosensory interaction through a depth camera, determine the bone points whose positions have changed in the body image, and perform drift compensation on the initial motion trajectory based on the dynamic drift characteristics of all bone points to obtain the actual interaction motion trajectory of the target user during somatosensory interaction; Perform deviation recognition on the rehabilitation training action of the target user based on the standard rehabilitation action of somatosensory interaction and the actual interaction motion trajectory to obtain the posture deviation degree between the actual rehabilitation training action of the target user during somatosensory interaction and the standard rehabilitation action; Generate a feedback adjustment instruction for the target user during the somatosensory interaction based on the posture deviation degree, and execute the feedback adjustment instruction to prompt the target user to adjust the rehabilitation training action.

[0006] In some embodiments, determining the initial motion trajectory of the target user during somatosensory interaction according to the rehabilitation training action data specifically includes: Denoise the rehabilitation training action data to obtain motion data with sensor noise removed; Based on the quaternion fusion algorithm, fuse the acceleration information and angular velocity information in the motion data with sensor noise removed to obtain the posture angles of each motion joint on the target user; Determine the spatial displacement information of each motion joint on the target user based on the acceleration information in the motion data with sensor noise removed; Construct the initial motion trajectory of the target user during somatosensory interaction according to the posture angles and spatial displacement information of all motion joints on the target user.

[0007] In some embodiments, capturing the body image of the target user during somatosensory interaction through a depth camera, and determining the bone points whose positions have changed in the body image specifically includes: Obtain the depth visual data of the target user during somatosensory interaction; Based on the human bone detection algorithm, identify the key bone points of the target user in the body image; According to the bone tracking algorithm, capture the position changes of each key bone point in the depth visual data, and further the bone points whose positions have changed in the body image.

[0008] In some embodiments, performing drift compensation on the initial motion trajectory based on the dynamic drift characteristics of all bone points to obtain the actual interaction motion trajectory of the target user during somatosensory interaction specifically includes: Determine the dynamic drift characteristics of each bone point; Determine the drift information in the initial motion trajectory of the target user during somatosensory interaction through all the dynamic drift characteristics; Compensate the initial motion trajectory based on the drift information to generate the actual interaction motion trajectory of the target user during somatosensory interaction.

[0009] In some embodiments, performing deviation identification on the rehabilitation training action of the target user based on the standard rehabilitation action of somatosensory interaction and the actual interaction motion trajectory to obtain the posture deviation degree between the actual rehabilitation training action of the target user during somatosensory interaction and the standard rehabilitation action specifically includes: Determine the standard rehabilitation action of the target user during somatosensory interaction; Determine the pose deviation of each bone point in the interactive motion trajectory based on the standard rehabilitation motion; Determine the posture deviation degree between the actual rehabilitation training motion of the target user during somatosensory interaction and the standard rehabilitation motion according to all the pose deviations.

[0010] In some embodiments, determining the standard rehabilitation motion of the target user during somatosensory interaction specifically includes: Extract the motion features of the target user's rehabilitation motion in the interactive motion trajectory; Based on the motion features, perform temporal segmentation on the interactive motion trajectory to obtain all independent action segments of the target user during somatosensory interaction; Perform action matching on all the independent action segments to obtain the standard rehabilitation motion of the target user during somatosensory interaction.

[0011] In some embodiments, generating a feedback adjustment instruction for the target user during somatosensory interaction based on the posture deviation degree, and executing the feedback adjustment instruction to prompt the target user to adjust the rehabilitation training motion specifically includes: Determine all the parts to be adjusted of the target user when performing the rehabilitation training motion based on the posture deviation degree; Determine the adjustment direction and adjustment amplitude of each part to be adjusted; Generate a feedback adjustment instruction for the target user during somatosensory interaction according to the adjustment direction and adjustment amplitude of each bone point.

[0012] In a second aspect, the present application provides an assisted rehabilitation teaching system based on somatosensory interaction, including a feedback adjustment unit, and the feedback adjustment unit includes: An acquisition module, configured to collect the rehabilitation training motion data of the target user when the target user performs the rehabilitation training motion of somatosensory interaction, and determine the initial motion trajectory of the target user during somatosensory interaction according to the rehabilitation training motion data; A processing module, configured to capture the body image of the target user during somatosensory interaction through a depth camera, determine the bone points whose positions have changed in the body image, and perform drift compensation on the initial motion trajectory based on the dynamic drift characteristics of all bone points to obtain the actual interactive motion trajectory of the target user during somatosensory interaction; The processing module is configured to perform deviation identification on the rehabilitation training motion of the target user based on the standard rehabilitation motion of somatosensory interaction and the actual interactive motion trajectory, and obtain the posture deviation degree between the actual rehabilitation training motion of the target user during somatosensory interaction and the standard rehabilitation motion; An execution module, configured to generate a feedback adjustment instruction for the target user during somatosensory interaction based on the posture deviation degree, and execute the feedback adjustment instruction to prompt the target user to adjust the rehabilitation training motion.

[0013] In a third aspect, the present application provides a computer device, which includes a memory and a processor. The memory stores code, and the processor is configured to obtain the code and execute the above-mentioned feedback adjustment method based on somatosensory interaction.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned feedback adjustment method based on somatosensory interaction is implemented.

[0015] The technical solutions provided by the disclosed embodiments of the present application have the following beneficial effects: In the assisted rehabilitation teaching system and method based on somatosensory interaction provided by the present application, first, when a target user performs a rehabilitation training action of somatosensory interaction, the rehabilitation training action data of the target user is collected, and the initial movement trajectory of the target user during somatosensory interaction is determined according to the rehabilitation training action data; the body image of the target user during somatosensory interaction is captured by a depth camera, the bone points with changed positions in the body image are determined, and the initial movement trajectory is drift-compensated based on the dynamic drift characteristics of all bone points to obtain the actual interaction movement trajectory of the target user during somatosensory interaction; the deviation of the rehabilitation training action of the target user is identified based on the standard rehabilitation action of somatosensory interaction and the actual interaction movement trajectory to obtain the posture deviation degree between the actual rehabilitation training action of the target user during somatosensory interaction and the standard rehabilitation action; a feedback adjustment instruction for the target user during somatosensory interaction is generated based on the posture deviation degree, and the feedback adjustment instruction is executed to prompt the target user to adjust the rehabilitation training action.

[0016] It can be seen that this application executes the feedback adjustment instruction to prompt the target user to adjust the rehabilitation training action; first, determining the dynamic drift feature of the bone points can obtain a time series describing the spatial displacement of the bone points of the target user over time during somatosensory interaction. The determination of the dynamic drift feature helps to accurately capture the motion pattern of the target user, providing data support for trajectory drift compensation and reducing sensor noise interference; second, determining the interactive motion trajectory can obtain a continuous trajectory that reflects the changes in the motion path and posture of the target user during somatosensory interaction after drift compensation of the initial trajectory. The determination of the interactive motion trajectory solves the problem of drift error in inertial sensing, which may lead to the deviation of the motion trajectory during long-term training, and helps to accurately record the actual motion path of the target user during rehabilitation exercise, reflecting the posture changes of the target user in real time, providing a scientific basis for subsequent deviation analysis and action adjustment; then, determining the posture deviation degree can obtain an index that measures the spatial position difference and rotation angle difference between the motion trajectory of the target user during rehabilitation exercise and the motion trajectory of the standard rehabilitation action during somatosensory interaction. The determination of the posture deviation degree helps to quantify the spatial position and rotation angle differences between the motion trajectory of the target user and the standard rehabilitation action, providing a scientific basis for evaluating the motion deviation of the target user, thereby supporting personalized posture adjustment to help the target user gradually approach the standard action, thus improving the rehabilitation training effect; finally, generating a feedback adjustment instruction for the target user during somatosensory interaction based on the posture deviation degree, and executing the feedback adjustment instruction to prompt the target user to adjust the rehabilitation training action; in summary, based on the above solution, the drift error of inertial sensing can be corrected by a depth camera during somatosensory interaction to improve the action capture accuracy of the assisted rehabilitation teaching system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is an exemplary flowchart of a feedback adjustment method based on somatosensory interaction according to some embodiments of the present application; Figure 2 is an exemplary flowchart of determining an initial motion trajectory according to some embodiments of the present application; Figure 3 is a schematic flowchart of implementing drift compensation according to some embodiments of the present application; Figure 4 is a schematic structural diagram of a feedback adjustment unit according to some embodiments of the present application; Figure 5 is an internal structural diagram of a computer device for implementing a feedback adjustment method based on somatosensory interaction according to some embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] In order to better understand the technical solution of the present application, the technical solution of the present application will be described in detail below in conjunction with the accompanying drawings of the specification and specific embodiments.

[0019] Reference Figure 1 , which is an exemplary flowchart of a feedback adjustment method based on somatosensory interaction according to some embodiments of the present application. The feedback adjustment method 100 based on somatosensory interaction mainly includes the following steps: In step 101, when the target user performs a rehabilitation training action of somatosensory interaction, collect the rehabilitation training action data of the target user, and determine the initial motion trajectory of the target user during somatosensory interaction according to the rehabilitation training action data.

[0020] It should be noted that in the present application, the rehabilitation training action data is information describing the changes in the body spatial position and motion state of the target user when performing a rehabilitation training action during somatosensory interaction; in specific implementation, the inertial detection unit can be fixed on the moving joints of the target user (such as the wrist, elbow or knee), and the linear acceleration of the moving joints of the target user when performing a rehabilitation training action during somatosensory interaction is collected by the accelerometer in the inertial detection unit at every preset acquisition time interval (such as 1 second), and all the collected linear accelerations are used as acceleration information. The angular velocity of the moving joints of the target user when performing a rehabilitation training action during somatosensory interaction is collected by the gyroscope in the inertial detection unit at every preset acquisition time interval (such as 1 second), and all the collected angular velocities are used as angular velocity information. Then, the set of all the collected acceleration information and angular velocity information is used as the rehabilitation training action data. Among them, the inertial detection unit is a sensor device integrating an accelerometer and a gyroscope, and is used to measure the acceleration, angular velocity and direction change of an object.

[0021] In some embodiments, reference Figure 2 , which is an exemplary flowchart of determining the initial motion trajectory according to some embodiments of the present application. In the present application, the initial motion trajectory of the target user during somatosensory interaction can be determined according to the rehabilitation training action data by the following steps: In step 1011, denoise the rehabilitation training action data to obtain motion data with sensor noise removed; In step 1012, fuse the acceleration information and angular velocity information in the motion data with sensor noise removed based on the quaternion fusion algorithm to obtain the attitude angles of each moving joint on the target user; In step 1013, determine the spatial displacement information of each moving joint on the target user based on the acceleration information in the motion data with sensor noise removed; In step 1014, construct the initial motion trajectory of the target user during somatosensory interaction according to the attitude angles and spatial displacement information of all the moving joints on the target user.

[0022] When specifically implemented, denoising the rehabilitation training motion data to obtain motion data with sensor noise removed can be achieved in the following way, that is: an existing denoising algorithm (such as the wavelet transform algorithm) can be used to denoise the rehabilitation training motion data, and the denoised rehabilitation training motion data is used as the motion data with sensor noise removed; wherein, the motion data is noise-free information describing the spatial position and motion state changes of the target user during the somatosensory interaction process.

[0023] It should be noted that in this application, the pose angle is a parameter describing the rotation angle and rotation amplitude of the motion joints of the target user in three-dimensional space during somatosensory interaction; when specifically implemented, fusing the acceleration information and angular velocity information in the motion data with sensor noise removed based on the quaternion fusion algorithm to obtain the pose angles of each motion joint on the target user can be achieved in the following way, that is: for each motion joint on the target user's body, the acceleration information and angular velocity information of the motion joint in the motion data can be first subjected to time synchronization processing, then a quaternion is initialized, the angular velocity information of the motion joint is integrated to obtain an angle increment to update the quaternion, and the quaternion is error-corrected by the gravity direction in the acceleration information of the motion joint, and finally the quaternion is normalized, and the normalized quaternion is used as the pose angle of the motion joint. Through the above method, the pose angles of each motion joint on the target user can be obtained; wherein, the quaternion is a four-dimensional vector used to represent the three-dimensional rotation of an object in space.

[0024] When specifically implemented, determining the spatial displacement information of each motion joint on the target user based on the acceleration information in the motion data with sensor noise removed can be achieved in the following way, that is: for each motion joint on the target user's body, the acceleration information of the motion joint can be first extracted from the motion data, and the linear acceleration in the acceleration information is integrated to calculate the velocity information of the motion joint, and then the velocity information is integrated twice to calculate the spatial displacement information of the motion joint. Through the above method, the spatial displacement information of each motion joint on the target user can be obtained; wherein, the spatial displacement information is a data set describing the position changes of the motion joints of the target user in three-dimensional space during somatosensory interaction.

[0025] It should be noted that in this application, the initial motion trajectory is the original motion path of the target user's body during somatosensory interaction; specifically, when implemented, the initial motion trajectory of the target user during somatosensory interaction can be constructed according to the posture angles and spatial displacement information of all motion joints on the target user, which can be achieved in the following way: for each motion joint on the target user's body, the posture angle and spatial displacement information of the motion joint point can be sorted according to the time sequence first to obtain the change time sequence of the motion joint, and then the existing interpolation method (such as cubic spline interpolation method) can be used to smooth the change time sequence to obtain the motion path of the motion joint. By the above method, the motion paths of each motion joint on the target user's body can be obtained, and then the set of all motion paths is used as the initial motion trajectory of the target user during somatosensory interaction.

[0026] In step 102, the body image of the target user during somatosensory interaction is captured by a depth camera, the bone points whose positions have changed in the body image are determined, and the initial motion trajectory is compensated for drift based on the dynamic drift characteristics of all bone points to obtain the actual interaction motion trajectory of the target user during somatosensory interaction.

[0027] It should be noted that the body image refers to the depth image that shows the body spatial position and motion state of the target user when performing rehabilitation training actions during somatosensory interaction; specifically, when implemented, the depth camera can be used to collect the depth image of the target user in the first second when performing rehabilitation training actions during somatosensory interaction, and this depth image is used as the body image of the target user during somatosensory interaction.

[0028] In some embodiments, determining the bone points whose positions have changed in the body image can be implemented by the following steps: Obtain the depth visual data of the target user during somatosensory interaction; Based on the human bone detection algorithm, identify the key bone points of the target user in the body image; According to the bone tracking algorithm, capture the position changes of each key bone point in the depth visual data, and then the bone points whose positions have changed in the body image.

[0029] Specifically, when implemented, obtaining the depth visual data of the target user during somatosensory interaction can be achieved in the following way: the depth camera can be used to collect the depth image of the target user during somatosensory interaction at every preset acquisition time interval (such as 1 second), and the existing smoothing processing method (such as bilateral filtering algorithm) can be used to smooth all the depth images, and then the set of all processed depth images arranged in time sequence is used as the depth visual data; wherein, the depth visual data is an image set that describes the position information of the target user in the three-dimensional space.

[0030] In specific implementation, to identify the key bone points of the target user in the body image based on the human bone detection algorithm, the following method can be adopted, that is: an existing human bone detection algorithm (such as a deep learning model based on a convolutional neural network) can be used to detect the human joint points of the body image and parse the coordinates of each joint point, thereby obtaining the key bone points of the target user; wherein, the key bone points refer to the joint nodes on the target user that represent the human body posture and movement trajectory, such as the head, shoulders, elbows, wrists, hips, knees, and ankles.

[0031] In specific implementation, according to the bone tracking algorithm, the position changes of each key bone point are captured in the depth visual data, and then the bone points whose positions change in the body image can be implemented in the following way, that is: for each key bone point, the Farneback optical flow method (such as the Farneback optical flow method) can be used to perform frame-by-frame matching and tracking of the key bone point in the depth visual data to obtain the spatial coordinates of the key bone point in different time frames, and all the spatial coordinates are combined into the position information of the key bone point in chronological order. By the above method, the position information of each key bone point can be obtained, and then based on all the position information, the key bone points with position drift are extracted from all the key bone points as the bone points whose positions change in the body image; wherein, the bone points whose positions change refer to the moving joints with position drift when the target user performs somatosensory interaction.

[0032] In some embodiments, referring to Figure 3 , this figure is a schematic flowchart of implementing drift compensation according to some embodiments of the present application. In the present application, based on the dynamic drift characteristics of all bone points, drift compensation is performed on the initial motion trajectory to obtain the actual interaction motion trajectory when the target user performs somatosensory interaction, which can be implemented by the following steps: Determine the dynamic drift characteristics of each bone point; Determine the drift information in the initial motion trajectory when the target user performs somatosensory interaction through all the dynamic drift characteristics; Compensate the initial motion trajectory based on the drift information to generate the actual interaction motion trajectory when the target user performs somatosensory interaction.

[0033] It should be noted that in this application, the dynamic drift feature is a sequence describing the spatial position of skeletal points changing with time during the somatosensory interaction of the target user. This dynamic drift feature helps to accurately capture the motion pattern of the target user and provides data support for trajectory drift compensation and reduction of sensor noise interference. Specifically, when implemented, the dynamic drift feature of each skeletal point can be determined in the following way: for each skeletal point, the position information of the skeletal point can be smoothed using a noise filtering technique (such as Kalman filtering or first-order low-pass filtering), and the smoothed position information is used as the dynamic drift feature of the skeletal point. Through the above method, the dynamic drift feature of each skeletal point can be obtained.

[0034] Specifically, when implemented, the drift information in the initial motion trajectory during the somatosensory interaction of the target user can be determined through all the dynamic drift features in the following way: for each skeletal point, the dynamic skeletal node can be first matched with the motion joint in the initial motion trajectory according to the human skeletal structure, and then an existing trajectory comparison method (such as the trajectory comparison method based on the Hausdorff distance) is used to compare the dynamic drift feature of the skeletal point and the motion path of the matched motion joint. The Hausdorff distance between the dynamic drift feature of the skeletal point and the motion path of the matched motion joint can be used as the trajectory deviation of the skeletal point. Through the above method, the trajectory deviation of each skeletal point can be obtained, and then the set of trajectory deviations of all skeletal points is used as the drift information in the initial motion trajectory during the somatosensory interaction of the target user. Among them, the drift information is data describing the deviation between the captured motion trajectory and the actual motion trajectory of the target user due to sensor errors during the rehabilitation movement process.

[0035] In addition, it should be noted that in this application, the interactive motion trajectory is a continuous trajectory reflecting the changes in the motion path and posture during the execution of rehabilitation training actions by the target user during the somatosensory interaction after drift compensation of the initial trajectory. Specifically, when implemented, compensating the initial motion trajectory based on the drift information to generate the actual interactive motion trajectory during the somatosensory interaction of the target user can be achieved in the following way: an existing trajectory compensation algorithm (such as the long short-term memory network algorithm) can be used to correct the trajectory deviation in the initial motion trajectory through the drift information, and the corrected initial motion trajectory is used as the actual interactive motion trajectory during the somatosensory interaction of the target user. Among them, using the trajectory compensation algorithm to correct the initial motion trajectory can eliminate the trajectory inaccuracy caused by sensor errors and user posture deviations by correcting the trajectory deviation in the target user's rehabilitation movement, thereby obtaining a more accurate interactive motion trajectory and providing reliable basic data for subsequent deviation analysis and feedback adjustment, improving the accuracy of rehabilitation training.

[0036] In step 103, deviation recognition is performed on the rehabilitation training actions of the target user based on the standard rehabilitation actions of somatosensory interaction and the actual interaction motion trajectory, and the posture deviation degree between the actual rehabilitation training actions of the target user during somatosensory interaction and the standard rehabilitation actions is obtained.

[0037] In some embodiments, deviation recognition is performed on the rehabilitation training actions of the target user based on the standard rehabilitation actions of somatosensory interaction and the actual interaction motion trajectory, and the posture deviation degree between the actual rehabilitation training actions of the target user during somatosensory interaction and the standard rehabilitation actions can be implemented by the following steps: Determine the standard rehabilitation actions of the target user during somatosensory interaction; Based on the standard rehabilitation actions, determine the pose deviation of each bone point in the interaction motion trajectory; According to all the pose deviations, determine the posture deviation degree between the actual rehabilitation training actions of the target user during somatosensory interaction and the standard rehabilitation actions.

[0038] In some embodiments, determining the standard rehabilitation actions of the target user during somatosensory interaction can be implemented by the following steps: Extract the motion features of the target user's rehabilitation movement from the interaction motion trajectory; Based on the motion features, perform temporal segmentation on the interaction motion trajectory to obtain all independent action segments of the target user during somatosensory interaction; Perform action matching on all the independent action segments to obtain the standard rehabilitation actions of the target user during somatosensory interaction.

[0039] Specifically, when implemented, extracting the motion features of the target user's rehabilitation movement from the interaction motion trajectory can be achieved in the following way, that is: for each bone point, the pose angle and spatial displacement information of the bone point can be obtained in the interaction motion trajectory, the angular change rate of the bone point (i.e., the first derivative of the pose angle with respect to time) is calculated through the pose angle, and the velocity change rate of the bone point (i.e., the first derivative of the spatial displacement information with respect to time) is calculated through the spatial displacement information, and the vector composed of the angular change rate and the velocity change rate is used as the motion vector of the bone point. By the above method, the motion vector of each bone point can be obtained, and then the set of motion vectors of all bone points is used as the motion features of the target user's rehabilitation movement; wherein, the motion features are feature vectors that describe the motion angle change and motion speed change of the target user during the rehabilitation movement process to analyze the difference between the user's motion pattern and the standard rehabilitation actions, and this motion feature can help accurately evaluate the motion performance of the target user, guide action correction and optimize the rehabilitation training effect.

[0040] When specifically implemented, the interactive motion trajectory is segmented in time sequence based on the motion features to obtain all independent action segments of the target user during the somatosensory interaction, which can be achieved in the following manner, i.e., an existing trajectory cutting method (such as a trajectory cutting method based on dynamic time warping) can be used to segment the interactive motion trajectory in time according to the significant change points of the angular change rate and the speed change rate of the skeleton points in the motion features (for example, both the angular change rate and the speed change rate are greater than 0.5), and each independent action segment is output, thereby obtaining all independent action segments of the target user during the somatosensory interaction; wherein, the independent action segment refers to a trajectory segment extracted during the rehabilitation movement of the target user, which has complete action features and can independently express the trajectory of a certain motion unit from the start to the end of the action. This independent action segment has a clear starting point and ending point, can independently reflect the posture and dynamic changes of a specific rehabilitation action, and does not depend on the continuity of other action segments.

[0041] It should be noted that in this application, the standard rehabilitation action refers to the motion trajectory of the ideal rehabilitation action used to guide the target user for rehabilitation training; when specifically implemented, action matching is performed on all independent action segments to obtain the standard rehabilitation action of the target user during somatosensory interaction, which can be achieved in the following manner, i.e., for each independent action segment, an existing action matching model (such as a hidden Markov model) can be loaded, the independent action segment is used as the input of this action matching model to match with the human action trajectory database, the action matching model is executed, and the human action with the highest matching probability in the output is used as the matching action of the independent action segment. Through the above method, the matching actions of each independent action segment can be obtained. The motion trajectories of the matching actions of each independent action segment can be extracted from the human action trajectory database, and all the motion trajectories are merged in the time sequence of all independent action segments in the interactive motion trajectory, and the merged motion trajectory is used as the standard rehabilitation action for rehabilitation training action matching.

[0042] It should be noted that the human action trajectory database is a database used to store and manage human motion trajectory data, which can be used to train models, match the motion trajectories of users, and provide action correction and feedback, and is applied to the fields of rehabilitation training, action recognition, sports analysis, and human-computer interaction. In this application, the Karlsruhe Institute of Technology Motion-Language Dataset human action trajectory database is used. In other embodiments, other human action trajectory databases can also be used, which are not specifically limited here.

[0043] In specific implementation, to determine the pose deviation of each bone point in the interactive motion trajectory based on the standard rehabilitation motion, the following method can be adopted, that is: for each bone point, first extract the motion path of the bone point in the interactive motion trajectory, extract the motion trajectory of the bone point in the standard rehabilitation motion, and calculate the distance deviation of the bone point in the spatial position (i.e., the Euclidean distance between the motion path and the spatial displacement information in the motion trajectory), then calculate the angle deviation of the bone point in joint rotation (i.e., the difference between the pose angles of the motion path and the motion trajectory), and use the vector composed of the distance deviation and the angle deviation as the pose deviation of the bone point. In this way, the pose deviation of each bone point can be obtained; wherein, the pose deviation is a vector that measures the difference between the motion path of the bone point and the motion trajectory of the bone point in the standard rehabilitation motion in terms of spatial position and rotation angle.

[0044] It should be noted that in this application, the pose deviation degree is an index that measures the spatial position difference and rotation angle difference between the motion trajectory of the target user during the somatosensory interaction for rehabilitation exercise and the motion trajectory of the standard rehabilitation motion; in specific implementation, to determine the pose deviation degree between the actual rehabilitation training motion of the target user during the somatosensory interaction and the standard rehabilitation motion according to all the pose deviations, the following method can be adopted, that is: first calculate the Euclidean norm of all the pose deviations, and then use the average value of all the Euclidean norms as the pose deviation degree between the rehabilitation training motion of the target user during the somatosensory interaction and the standard rehabilitation motion; wherein, the rehabilitation training motion refers to the action sequence extracted from the motion trajectory of the target user during the rehabilitation exercise to evaluate the difference between the action of the target user and the standard rehabilitation motion.

[0045] In step 104, generate a feedback adjustment instruction for the target user during the somatosensory interaction based on the pose deviation degree, and execute the feedback adjustment instruction to prompt the target user to adjust the rehabilitation training motion.

[0046] In some embodiments, to generate a feedback adjustment instruction for the target user during the somatosensory interaction based on the pose deviation degree and execute the feedback adjustment instruction to prompt the target user to adjust the rehabilitation training motion, the following steps can be adopted: Determine all the parts to be adjusted of the target user when performing the rehabilitation training motion based on the pose deviation degree; Determine the adjustment direction and adjustment amplitude of each part to be adjusted; Generate a feedback adjustment instruction for the target user during the somatosensory interaction according to the adjustment direction and adjustment amplitude of each bone point.

[0047] In specific implementation, all the parts to be adjusted for the target user during the execution of the rehabilitation training action can be determined based on the posture deviation degree in the following manner: for each bone point, calculate the absolute value of the difference between the Euclidean norm of the pose deviation of the bone point and the posture deviation degree. If the absolute value is less than or equal to a preset deviation threshold (for example: 1), the movement posture of the bone point is normal and does not need to be adjusted. If the absolute value is greater than the preset deviation threshold (for example: 1), the movement posture of the bone point is abnormal, and the bone point can be used as a part to be adjusted. All the parts to be adjusted for the target user during the execution of the rehabilitation training action can be obtained through the above method; wherein, the part to be adjusted refers to all the joints that the target user needs to correct the posture during the rehabilitation movement. By determining the part to be adjusted, the movement deviation can be accurately located, and the accuracy of the rehabilitation training can be improved.

[0048] In specific implementation, the adjustment direction and adjustment amplitude of each part to be adjusted can be determined in the following manner: for each part to be adjusted, the direction represented by the imaginary part of the angle deviation in the pose deviation of the bone point can be used as the adjustment direction of the bone point, and the amplitude represented by the real part can be used as the adjustment amplitude of the bone point. Through the above method, the adjustment direction and adjustment amplitude of each part to be adjusted can be obtained; wherein, the adjustment direction refers to the moving direction of the joint that needs to be changed; the adjustment amplitude refers to the size of the angle adjustment of the joint that needs to be changed.

[0049] It should be noted that in this application, the feedback adjustment instruction is an adjustment prompt for guiding the target user to correct the movement posture to reach the standard target; in specific implementation, the feedback adjustment instruction for the target user during the somatosensory interaction process can be generated according to the adjustment direction and adjustment amplitude of each bone point, for example: "Please raise your right arm by 3 degrees".

[0050] In specific implementation, the execution of the feedback adjustment instruction to prompt the target user to adjust the rehabilitation training action can be implemented in the following manner: the user can be informed of the feedback adjustment instruction through voice prompt, and vibration prompt can be provided at the bone points that need to be adjusted through tactile feedback (such as a vibrating bracelet) to enhance the user's perception.

[0051] In addition, on the other hand of this application, in some embodiments, this application provides an assisted rehabilitation teaching system based on somatosensory interaction. The system includes a feedback adjustment unit. Refer to Figure 4 This figure is a schematic structural diagram of the feedback adjustment unit according to some embodiments of this application. The feedback adjustment unit 400 includes: an acquisition module 401, a processing module 402, and an execution module 403, which are described as follows: The acquisition module 401. In this application, the acquisition module 401 is mainly used to acquire the rehabilitation training action data of the target user when the target user performs the rehabilitation training actions of somatosensory interaction, and determine the initial motion trajectory of the target user during somatosensory interaction according to the rehabilitation training action data; The processing module 402. In this application, the processing module 402 is mainly used to capture the body image of the target user during somatosensory interaction through a depth camera, determine the bone points whose positions have changed in the body image, and perform drift compensation on the initial motion trajectory based on the dynamic drift characteristics of all bone points to obtain the actual interaction motion trajectory of the target user during somatosensory interaction; It should be noted that in this application, the processing module 402 is also used to; identify the deviation of the rehabilitation training actions of the target user based on the standard rehabilitation actions of somatosensory interaction and the actual interaction motion trajectory, and obtain the posture deviation degree between the actual rehabilitation training actions of the target user during somatosensory interaction and the standard rehabilitation actions The execution module 403. In this application, the execution module 403 is mainly used to generate a feedback adjustment instruction for the target user during somatosensory interaction based on the posture deviation degree, and execute the feedback adjustment instruction to prompt the target user to adjust the rehabilitation training actions.

[0052] Each module in the above-mentioned somatosensory interaction-based assisted rehabilitation teaching system can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0053] In addition, in one embodiment, this application provides a computer device, which can be a server, and its internal structure diagram can be as Figure 5 shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data of the feedback adjustment method based on somatosensory interaction. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it realizes a feedback adjustment method based on somatosensory interaction.

[0054] Those skilled in the art can understand, Figure 5The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0055] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the steps in the above-mentioned embodiment of the feedback adjustment method based on somatosensory interaction are implemented.

[0056] In one embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned embodiment of the feedback adjustment method based on somatosensory interaction are implemented.

[0057] In one embodiment, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in the above-mentioned embodiment of the feedback adjustment method based on somatosensory interaction.

[0058] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the processes of the above-mentioned method embodiments. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in this application may include at least one of non-volatile and volatile memories. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0059] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0060] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A feedback adjustment method based on somatosensory interaction, used for performing somatosensory interaction feedback adjustment in an auxiliary rehabilitation teaching system, characterized in that: The method comprises the following steps: When the target user performs a somatosensory interactive rehabilitation training action, collecting rehabilitation training action data of the target user, and determining an initial motion trajectory of the target user when performing the somatosensory interaction according to the rehabilitation training action data; Capturing a body image of a target user when performing somatosensory interaction through a depth camera, determining the skeletal points whose positions have changed in the body image, and performing drift compensation on the initial motion trajectory based on the dynamic drift characteristics of all skeletal points to obtain an actual interactive motion trajectory of the target user when performing somatosensory interaction; Based on the standard rehabilitation action of somatosensory interaction and the actual interactive motion trajectory, deviation identification is performed on the rehabilitation training action of the target user to obtain the posture deviation between the actual rehabilitation training action of the target user and the standard rehabilitation action when performing somatosensory interaction; A feedback adjustment instruction of the target user in the somatosensory interaction process is generated based on the posture deviation, and the feedback adjustment instruction is executed to prompt the target user to adjust the rehabilitation training action.

2. The method according to claim 1, characterized in that Determining the initial motion trajectory of the target user when performing somatosensory interaction according to the rehabilitation training motion data specifically includes: De-noising the rehabilitation training motion data to obtain motion data without sensor noise; The acceleration information and angular velocity information in the motion data from which the sensor noise has been removed are fused based on a quaternion fusion algorithm to obtain the posture angles of each motion joint of the target user; Determine spatial displacement information of each motion joint of the target user based on the acceleration information in the motion data from which sensor noise has been removed; The initial motion trajectory of the target user during somatosensory interaction is constructed based on the posture angles and spatial displacement information of all motion joints of the target user.

3. The method according to claim 1, characterized in that Determining the bone points whose positions have changed in the body image specifically includes: Obtain the deep visual data of the target user when performing somatosensory interaction; identifying key skeleton points of the target user in the body image based on a human skeleton detection algorithm; The position changes of each key bone point are captured in the depth vision data according to the bone tracking algorithm, and then the bone points whose positions are changed in the body image are captured.

4. The method according to claim 1, characterized in that The initial motion trajectory is drift compensated based on the dynamic drift characteristics of all skeleton points to obtain the actual interactive motion trajectory when the target user performs somatosensory interaction, specifically including: Determine the dynamic drift characteristics of each skeleton point; Determine the drift information in the initial motion trajectory of the target user when performing somatosensory interaction through all dynamic drift features; The initial motion trajectory is compensated based on the drift information to generate an actual interactive motion trajectory when the target user performs somatosensory interaction.

5. The method according to claim 1, characterized in that Based on the standard rehabilitation action of somatosensory interaction and the actual interactive motion trajectory, the rehabilitation training action of the target user is identified for deviation, and the posture deviation between the actual rehabilitation training action of the target user and the standard rehabilitation action when performing somatosensory interaction is obtained, specifically including: Determine the standard rehabilitation actions for target users when performing somatosensory interaction; Determining the position deviation of each skeletal point in the interactive motion trajectory based on the standard rehabilitation action; The posture deviation between the actual rehabilitation training action and the standard rehabilitation action when the target user performs somatosensory interaction is determined according to all posture deviations.

6. The method according to claim 5, characterized in that The standard rehabilitation actions for target users to perform somatosensory interaction include: extracting the motion features of the target user performing rehabilitation exercise from the interactive motion trajectory; Performing time-series segmentation on the interactive motion trajectory based on the motion features to obtain all independent motion segments of the target user during the somatosensory interaction process; All independent action clips are matched to obtain the standard rehabilitation actions of the target user when performing somatosensory interaction.

7. The method according to claim 1, characterized in that Generating a feedback adjustment instruction for the target user during the somatosensory interaction process based on the posture deviation, and executing the feedback adjustment instruction to prompt the target user to adjust the rehabilitation training action specifically includes: Determining all parts to be adjusted of the target user when performing rehabilitation training actions based on the posture deviation; Determine the adjustment direction and adjustment range of each part to be adjusted; According to the adjustment direction and adjustment range of each skeleton point, feedback adjustment instructions are generated for the target user during the somatosensory interaction process.

8. An auxiliary rehabilitation teaching system based on somatosensory interaction, the system includes a feedback adjustment unit, characterized in that: The feedback adjustment unit comprises: A collection module, used for collecting rehabilitation training action data of the target user when the target user performs a somatosensory interactive rehabilitation training action, and determining an initial motion trajectory of the target user when performing the somatosensory interaction according to the rehabilitation training action data; A processing module, used to capture a body image of a target user when performing somatosensory interaction through a depth camera, determine the skeletal points whose positions have changed in the body image, and perform drift compensation on the initial motion trajectory based on the dynamic drift characteristics of all skeletal points to obtain an actual interactive motion trajectory of the target user when performing somatosensory interaction; The processing module is used to identify the deviation of the rehabilitation training action of the target user based on the standard rehabilitation action of the somatosensory interaction and the actual interactive motion trajectory, and obtain the posture deviation between the actual rehabilitation training action of the target user and the standard rehabilitation action when the target user performs somatosensory interaction; The execution module is used to generate a feedback adjustment instruction for the target user during the somatosensory interaction process based on the posture deviation, and execute the feedback adjustment instruction to prompt the target user to adjust the rehabilitation training action.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the feedback adjustment method based on somatosensory interaction described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the feedback adjustment method based on somatosensory interaction as described in any one of claims 1 to 7 are implemented.

Images