Motion recognition and evaluation method and system for rehabilitation training based on transformer

Abstract

The embodiment of the application provides a rehabilitation training motion recognition and evaluation method and system based on a Transformer, which comprises the following steps: acquiring an initial image containing a patient, performing human body detection on the initial image to obtain a target region image and target region feature information; performing posture regression processing on the target region image and the target region feature information through a neural network based on individualized calibration tokens to determine key points and confidence; determining the key points of each continuous frame containing the patient based on the key points; performing filtering processing and time sequence prediction on the key points of each continuous frame to obtain key point trajectories and the uncertainty of the key points; calculating a plurality of joint parameters based on the key point trajectories; comparing the plurality of joint parameters with preset joint parameters to determine an error type; and generating a correction instruction for the patient according to the error type, so that the patient performs rehabilitation training according to the correction instruction. The above scheme can improve the accuracy of rehabilitation training motion recognition and evaluation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and rehabilitation engineering, in particular to a motion recognition and evaluation method and system for rehabilitation training based on a Transformer. BACKGROUND

[0002] With the gradual popularity of home rehabilitation and remote rehabilitation, videos collected by monocular cameras or mobile terminals have become an important data source for rehabilitation training. However, rehabilitation motions have the characteristics of small amplitude, slow speed, clear stages, and often accompanied by auxiliary instruments (braces, protectors, crutches, wheelchairs). The body posture differences of patients and the individualized boundaries of joint range of motion (ROM) caused by surgical history, as well as the strict control requirements of safety thresholds and compensatory motions (such as shoulder raising, pelvic tilting, and trunk rotation) in clinical practice. Existing general human pose estimation (HPE) methods perform well in fitness or security scenes, but lack stability in rehabilitation scenes (low resolution, occlusion, abnormal proportions, and small motion amplitude). At the same time, traditional evaluation schemes only have simple threshold judgments for joint points (reaching the threshold is simply determined to be qualified), and lack evaluation schemes centered on clinical indicators. SUMMARY

[0003] The embodiments of the present application aim to provide a motion recognition and evaluation method and system for rehabilitation training based on a Transformer, which can improve the accuracy and usability of motion recognition and evaluation for rehabilitation training.

[0004] The technical solution of the present application is implemented as follows:

[0005] In a first aspect, the embodiments of the present application provide a motion recognition and evaluation method for rehabilitation training based on a Transformer, which comprises:

[0006] An initial image containing a patient is obtained, human body detection is performed on the initial image, and a target region image and target region feature information are obtained;

[0007] A pre-determined neural network based on individualized calibration tokens is used to perform pose regression processing on the target region image and the target region feature information, to determine key points and confidence levels;

[0008] Based on the key points, key points of each of the continuous frames containing the patient are determined, and filtering processing and time series prediction are performed on the key points of each of the continuous frames to obtain key point trajectories and uncertainties of the key points;

[0009] Based on the key point trajectories, a plurality of joint parameters are calculated, and the plurality of joint parameters are compared with preset joint parameters to determine error types; wherein the plurality of joint parameters include joint range of motion, angular velocity, stage events, and symmetry.

[0010] generate a correction instruction for the patient according to the error type, so that the patient performs rehabilitation training according to the correction instruction.

[0011] In the scheme, the human body detection on the initial image to obtain the target region image and the target region feature information comprises:

[0012] The initial image is divided into grids, and local features are extracted from the divided initial image;

[0013] The local features are extracted by a backbone network to obtain a feature map, and human body detection is performed on the feature map to obtain an initial human body bounding box and class information;

[0014] Based on the initial human body bounding box and the class information, redundant boxes are removed to determine a human body bounding box;

[0015] Based on the human body bounding box, the target region image and the target region feature information containing scale information, rotation angle and detection confidence are determined.

[0016] In the scheme, the pose regression processing of the target region image and the target region feature information by the neural network based on the individualized calibration token is performed to determine the key points and the confidence, comprising:

[0017] The target region image is subjected to region feature extraction to obtain a target region feature;

[0018] Based on the scale information in the target region feature information, the target region feature is subjected to position encoding and channel normalization processing to obtain a normalized feature;

[0019] The normalized feature is subjected to pose regression processing by a plurality of constraint conditions in the neural network based on the individualized calibration token to determine the key points and the confidence.

[0020] In the scheme, the plurality of constraint conditions comprises a skeleton adjacency matrix, a left-right symmetry mapping matrix and an upper and lower limit of an anatomical angle;

[0021] The pose regression processing of the normalized feature by the plurality of constraint conditions in the neural network based on the individualized calibration token to determine the key points and the confidence comprises:

[0022] The irrelevant points in the normalized feature are removed by the skeleton adjacency matrix in the neural network based on the individualized calibration token to obtain screened points;

[0023] Based on the left-right symmetric mapping matrix and the upper and lower limits of the anatomical angle, posture regression processing is performed on the screened points to determine bone length consistency loss value, joint angle penalty loss value, uncertainty loss value, and joint consistency loss value.

[0024] Based on the bone length consistency loss value, the joint angle penalty loss value, the uncertainty loss value, and the joint consistency loss value, weighted processing is performed to determine a total loss value.

[0025] Based on the total loss value, the key points and the confidence are determined.

[0026] In the above scheme, based on the key point trajectory, a plurality of joint parameters are calculated; and the plurality of joint parameters are compared with preset joint parameters to determine an error type, including:

[0027] Based on the key point trajectory, joint range of motion, angular velocity, and symmetry are calculated.

[0028] Based on the joint range of motion, the angular velocity, and the symmetry, a stage query is performed to determine a stage event; wherein the stage event includes: preparation, acceleration, maintenance, deceleration, and recovery; and the joint range of motion, the angular velocity, the symmetry, and the stage event are determined as the plurality of joint parameters.

[0029] The plurality of joint parameters are compared with the preset joint parameters to determine the error type.

[0030] In the above scheme, before the posture regression processing of the target region image and the target region feature information is performed by the neural network based on the individualized calibration token, the method further includes:

[0031] Before the action recognition and evaluation of the patient's first rehabilitation training, the baseline action of the patient is obtained.

[0032] Based on the baseline action, the limb segment ratio, the joint range of motion, and the affected side label of the patient are determined; based on the limb segment ratio, the joint range of motion, and the affected side label, an individualized calibration token of the patient is determined.

[0033] Based on the individualized calibration token, a pre-determined structure prior token is personalized adjusted to obtain the neural network based on the individualized calibration token; wherein the structure prior token is an adjacency / Laplacian matrix containing a skeleton graph or a structure prior token sequence of relative position bias based on a preset key point set and a skeleton edge set.

[0034] In the foregoing solution, after the patient performs rehabilitation training according to the correction instruction, the method further includes:

[0035] determining a current rehabilitation training degree of the patient based on the error type and the correction instruction;

[0036] updating the individualized calibration token-based neural network based on the current rehabilitation training degree, to obtain an updated individualized calibration token-based neural network, so that the patient is next time rehabilitation evaluated according to the updated individualized calibration token-based neural network.

[0037] In a second aspect, the embodiments of the present application provide a motion recognition and evaluation system for rehabilitation training based on a Transformer, which comprises an acquisition module, a determination module, a prediction module and a generation module, wherein

[0038] The acquisition module is configured to acquire an initial image containing a patient, perform human body detection on the initial image, and obtain a target region image and target region feature information.

[0039] The determination module is configured to perform pose regression processing on the target region feature information and the target region feature information by a pre-determined individualized calibration token-based neural network, to determine key points and confidence.

[0040] The prediction module is configured to determine respective key points of consecutive frames containing the patient based on the key points, and perform filtering processing and time series prediction on the respective key points of the consecutive frames, to obtain key point trajectories and uncertainty of the key points.

[0041] The determination module is further configured to calculate a plurality of joint parameters based on the key point trajectories, and compare the plurality of joint parameters with preset joint parameters to determine an error type, wherein the plurality of joint parameters include joint range of motion, angular velocity, phase events and symmetry.

[0042] The generation module is configured to generate a correction instruction for the patient according to the error type, so that the patient performs rehabilitation training according to the correction instruction.

[0043] In a third aspect, the embodiments of the present application provide a motion recognition and evaluation device for rehabilitation training based on a Transformer, which comprises a processor and a memory, wherein

[0044] The memory is configured to store a computer program.

[0045] The processor is configured to invoke and run the computer program from the memory to execute the method according to the first aspect.

[0046] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium storing executable instructions for causing a processor to execute the method according to the first aspect.

[0047] The embodiments of the present application provide a rehabilitation training motion recognition and evaluation method and system based on a Transformer. The method comprises the following steps: acquiring an initial image containing a patient, performing human body detection on the initial image to obtain a target region image and target region feature information; performing posture regression processing on the target region image and the target region feature information through a pre-determined neural network based on individualized calibration tokens to determine key points and confidence; determining key points of each of continuous frames containing the patient based on the key points; and performing filtering processing and time sequence prediction on the key points of each of the continuous frames to obtain key point trajectories and uncertainty of the key points; calculating a plurality of joint parameters based on the key point trajectories; and comparing the plurality of joint parameters with preset joint parameters to determine an error type; wherein the plurality of joint parameters comprise joint range of motion, angular velocity, phase events and symmetry; and generating a correction instruction for the patient according to the error type to enable the patient to perform rehabilitation training according to the correction instruction. In the above scheme, the key points are determined through the pre-determined neural network based on individualized calibration tokens to perform posture regression processing on the target region image and the target region feature information, so that the accuracy of the key points is higher, and the accuracy of rehabilitation training motion recognition and evaluation can be improved. The rehabilitation motion is converted into clinical indicators through the key point trajectories, compared with standard motions, the amplitude and angle differences between the actual motion and the standard motion are determined, and a correction instruction is generated, so that individualized, interpretable and safe and compliant rehabilitation guidance can be realized, and the usefulness and usability of patient rehabilitation training can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0048] The drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application. It is expressly understood that the drawings are only illustrative and that the present application can be embodied in other ways not necessarily depicted in the drawings.

[0049] The flowcharts shown in the drawings are only illustrative, and do not necessarily include all contents and operations / steps, nor do they necessarily have to be executed in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to the actual situation.

[0050] Figure 1 An optional flowchart of a rehabilitation training action recognition and evaluation method based on a Transformer provided by an embodiment of the present application Figure 1

[0051] Figure 2 An optional flowchart of a rehabilitation training action recognition and evaluation method based on a Transformer provided by an embodiment of the present application Figure 2

[0052] Figure 3 An optional flowchart of a rehabilitation training action recognition and evaluation method based on a Transformer provided by an embodiment of the present application Figure 3

[0053] Figure 4 An optional flowchart of a rehabilitation training action recognition and evaluation method based on a Transformer provided by an embodiment of the present application Figure 4

[0054] Figure 5 A structural diagram of a rehabilitation training action recognition and evaluation system based on a Transformer provided by an embodiment of the present application

[0055] Figure 6 A structural diagram of a rehabilitation training action recognition and evaluation device based on a Transformer provided by an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the specific technical solutions of the present application will be further described below with reference to the accompanying drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0058] In the following description, “some embodiments”, “the embodiment”, “the embodiments of the present application” and the like describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0059] ​​​​If the application file contains similar descriptions of "first / second", the following description is added: in the following description, the terms "first\second\third" involved only distinguish similar objects, and do not represent the specific order of the objects. Understandably, "first\second\third" can be interchanged in specific order or sequence as allowed, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0060] Based on this, the embodiments of the present application provide a motion recognition and evaluation method for rehabilitation training based on Transformer, Figure 1 An optional flowchart of a motion recognition and evaluation method for rehabilitation training based on Transformer provided by the embodiments of the present application Figure 1 The steps shown will be described. Figure 1 The steps shown will be described.

[0061] S101, obtain an initial image containing a patient, perform human body detection on the initial image, and obtain a target region image and target region feature information.

[0062] In some embodiments of the present application, the target region image is a region of interest image containing a human body bounding box. The target region feature information includes scale information, rotation angle and detection confidence.

[0063] In some embodiments of the present application, the scale information represents the size of the object in the three-dimensional space, determines the specific size of the person in the whole photo, facilitates subsequent normalization processing, and prevents errors caused by body proportions. The rotation angle (i.e., rotation pose estimation) represents the direction of the object in the three-dimensional space. The rotation pose estimation is used to adjust the angle in the subsequent process, so that the cropping result is aligned with the human body pose, which is convenient for subsequent recognition and is also beneficial to distinguish the effective human body and the interference of the background.

[0064] In some embodiments of the present application, the motion recognition and evaluation method for rehabilitation training based on Transformer is suitable for scenarios such as motion recognition and evaluation of patient rehabilitation training.

[0065] In some embodiments of the present application, the motion recognition and evaluation method for rehabilitation training based on Transformer is suitable for a motion recognition and evaluation system for rehabilitation training based on Transformer.

[0066] In some embodiments of the present application, the initial image is divided into a grid, and local features are extracted from the divided initial image; the local features are extracted by a backbone network to obtain a feature map; and human body detection is performed through the feature map to obtain an initial human body bounding box and class information; based on the initial human body bounding box and the class information, redundant boxes are removed to determine the human body bounding box; and based on the human body bounding box, a target region image and target region feature information containing scale information, a rotation angle, and a detection confidence are determined.

[0067] In S102, the target region image and the target region feature information are subjected to pose regression processing through a pre-determined neural network based on individualized calibration tokens to determine the key points and the confidence.

[0068] In some embodiments of the present application, the target region image is subjected to region feature extraction to obtain target region features; the target region features are subjected to position encoding and channel normalization processing based on the scale information in the target region feature information to obtain normalized features; and the normalized features are subjected to pose regression processing through multiple constraint conditions in the neural network based on individualized calibration tokens to determine the key points and the confidence.

[0069] In S103, based on the key points, the key points of each of the continuous frames containing the patient are determined; and the key points of each of the continuous frames are subjected to filtering processing and time series prediction to obtain a key point trajectory and an uncertainty of the key points.

[0070] In some embodiments of the present application, based on the key points, the key points in each of the continuous frames containing the patient are determined to obtain the key points of each of the continuous frames, and the key points in each of the frames are the same. The key points of each of the continuous frames are subjected to Kalman filtering processing to obtain processed key points of each of the continuous frames; and the processed key points of each of the continuous frames are input into a time series Transformer model for time series prediction to obtain a key point trajectory and an uncertainty of the key points.

[0071] For example, after the continuous frame key points are preprocessed by a Kalman filter as time series input features and input into a time series Transformer model, a smoothed trajectory and a fusion variance are output, the smoothed trajectory is a stable time series prediction result, and the fusion variance is used to represent a stable typical quantization. According to the fusion variance, a threshold of the uncertainty can be set, and threshold self-adaptation and feedback throttling are realized based on actual values. When the threshold is higher than the threshold, a feedback mechanism is automatically triggered, the decision is delayed, or the user is requested to adjust the pose / photographing condition to improve the photographing stability of subsequent frames, and the weight parameters of the Transformer model or the filter are dynamically adjusted.

[0072] S104, based on the key point trajectory, calculate a plurality of joint parameters; and compare the plurality of joint parameters with the preset joint parameters to determine the error type; wherein the plurality of joint parameters include: joint range of motion, angular velocity, phase event and symmetry.

[0073] In some embodiments of the application, based on the key point trajectory, the joint range of motion, angular velocity and symmetry are calculated; based on the joint range of motion, angular velocity and symmetry, the phase query is performed to determine the phase event; wherein the phase event includes: preparation, acceleration, maintenance, deceleration and recovery; and the joint range of motion, angular velocity, symmetry and phase event are determined as the plurality of joint parameters; the plurality of joint parameters are compared with the preset joint parameters to determine the error type.

[0074] It should be noted that the key point trajectory: the key point refers to the joint position, which is used to determine the motion trajectory of the patient; the uncertainty: used to detect the error and reliability of the predicted trajectory; the joint range of motion (ROM): the range of joint activity, such as the knee joint usually 0-180 degrees activity; angular velocity: the activity speed of the joint, which is used to detect the patient's action; symmetry: the symmetry of the two sides of the joint, which ensures the consistency of the two sides of the joint. For example, when the range of motion (ROM) of the left knee joint is greater than 160°, and the ROM of the right knee joint is 120°, it may indicate that the left knee is overactive, which has a risk of injury. At this time, the system can prompt the user to rest or check accordingly. The phase event token, i.e., the phase event, is automatically learned by the model during the training process and is used to represent the phase state in the motion state (such as preparation, acceleration, maintenance, deceleration, recovery, etc.).

[0075] S105, according to the error type, generate a correction instruction for the patient, so that the patient performs rehabilitation training according to the correction instruction.

[0076] In some embodiments of the application, according to the error type, a correction instruction for the patient is generated; the patient can perform rehabilitation training according to the correction instruction.

[0077] For example, according to the comparison of the indicators, the error type e ∈ {ROM insufficient / over limit, speed insufficient / fast, phase out of order, symmetry poor, compensation exists} on (joint j, phase s) is output, and the correction instruction such as "please reduce the forward inclination of the trunk, keep the pelvis stable, and then extend 5°" is generated.

[0078] It can be understood that, by performing pose regression processing on the target region image and the target region feature information through the pre-determined individualized calibration token-based neural network, the accuracy of the determined key points is higher, and the accuracy of motion recognition and evaluation of rehabilitation training can be improved. By converting the rehabilitation motion into clinical indicators (joint range of motion, angular velocity, symmetry, phase event) through the key point trajectory and comparing with the standard motion, the amplitude angle difference between the actual motion and the standard motion is determined, and correction instructions are generated, realizing individualized, interpretable and safe and compliant rehabilitation guidance, and the usefulness and usability of patient rehabilitation training can be improved.

[0079] In some embodiments of the present application, S101 can be implemented through S201-S204, as follows:

[0080] S201, grid division is performed on the initial image, and local features are extracted from the divided initial image.

[0081] S202, feature extraction is performed on the local features through a backbone network to obtain a feature map, and human body detection is performed through the feature map to obtain an initial human body bounding box and class information.

[0082] S203, based on the initial human body bounding box and the class information, redundant boxes are removed to determine the human body bounding box.

[0083] S204, based on the human body bounding box, a target region image is determined, and target region feature information containing scale information, rotation angle and detection confidence is determined.

[0084] As shown in the example, Figure 2 As shown, the input image is divided into a grid, and after feature extraction through the backbone network, a feature map output (i.e. initial prediction result of the human body bounding box) is obtained. After non-maximum suppression NMS according to the bounding box and class prediction, redundant boxes are removed, and finally the human body bounding box output is determined. The output RoI and its scale information s, rotation pose estimation r. When the detection confidence is lower than the threshold, retry or multi-scale strategy is entered. To be compatible with the home scene, single-person rapid screening and human body proportion adaptive cropping are supported.

[0085] It should be noted that the initial image is the input image; and the RoI is the target region image.

[0086] In some embodiments of the present application, S102 can be implemented through S301-S303, as follows:

[0087] S301, region feature extraction is performed on the target region image to obtain target region features.

[0088] S302, based on the scale information in the target region feature information, position encoding and channel normalization processing are performed on the target region features to obtain normalized features.

[0089] S303. By applying various constraints in the neural network based on individualized calibration tokens, pose regression is performed on the normalized features to determine key points and confidence levels.

[0090] In some embodiments of this application, various constraints include: skeleton adjacency matrix, left-right symmetry mapping matrix, and upper and lower limits of anatomical angles.

[0091] In some embodiments of this application, irrelevant points in the normalized features are removed by using the skeleton adjacency matrix in a neural network based on individualized calibration tokens to obtain filtered points; pose regression processing is performed on the filtered points based on the left-right symmetry mapping matrix and the upper and lower limits of anatomical angles to determine bone length consistency loss value, penalized joint angle loss value, uncertainty loss value, and joint consistency loss value; weighted processing is performed on the bone length consistency loss value, penalized joint angle loss value, uncertainty loss value, and joint consistency loss value to determine the total loss value; and key points and confidence levels are determined based on the total loss value.

[0092] For example, such as Figure 3 As shown, CNN or ViT backbone is used to extract RoI features. Perform positional encoding (PE) and channel normalization on F. Encode the human topology (joint set K, bone edge set E) as prior. It includes: skeleton adjacency matrix A, left-right symmetry mapping M, and upper and lower limits of anatomical angles. Constructing a mask in self-attention This suppresses non-adjacent or transsymmetric but unrelated attention weights; and introduces bone length consistency and angle out-of-bounds penalty terms into the loss function:

[0093]

[0094]

[0095] Where e represents a skeletal edge (a line segment connecting two joints, which is a set of joint pairs). This indicates the positions of the i-th and j-th joints in the skeleton. This represents the actual length of the edges in the skeleton, i.e. The actual distance between them The bone length consistency loss term, i.e., the bone length consistency loss value, is used to ensure that the distance between joints in the posture estimation matches the actual anatomical structure.

[0096] in, This represents the set of all joints in the skeleton. Let j be the actual bending angle of the j-th joint. maximal allowed angle of the jth joint, minimal allowed angle of the jth joint, to punish joint angle, ensure that the joint angle cannot exceed the physiological allowable range. penalty joint angle loss value.

[0097] The patient completes the baseline action (such as T-pose, flexion and extension) through guidance for the first time, estimates the limb segment ratio , joint range of motion , the affected side is marked h, etc. Parameters such as , h, height / weight / visual distance estimation, etc. are encoded into , and the query is spliced into the input decoder to realize the judgment of "relative to the baseline of itself".

[0098] In the decoding output, N key point queries (k=1..N) are used, and after multi-layer cross-attention and self-attention, the key point mean value and confidence are output; At the same time, the logarithmic variance is output for uncertainty representation (which can be realized by negative log-likelihood regression):

[0099]

[0100]

[0101] wherein, represents the joint coordinate point predicted by the model, is the real joint coordinate point, is the variance of the predicted coordinate, indicating the uncertainty of the model. Mainly evaluate the uncertainty of the predicted value to ensure the reliability of the model; is the uncertainty loss value and is the joint consistency loss value.

[0102] and are joint coordinates at different scales. This loss term ensures that the pose estimation at different scales remains consistent.

[0103] The total loss value .

[0104] It should be noted that the joint set is a set covering all joint coordinates in the picture. The skeleton edge set is a set of lines connecting joints by bones in the input of the neural network, which is represented as a line from one joint to another. The skeleton adjacency matrix is composed of the joint set and the skeleton edge set, and is used to determine whether there is a skeleton between two joints. The left-right symmetry mapping M is a symmetry matrix used to determine which two joints are stacked to maintain the consistency of left and right joint predictions. The upper and lower limits of the anatomical angle are used to ensure that the joint will only flex within a certain range and will not appear in an impossible motion posture, such as a knee joint rotation angle of 270 degrees.

[0105] In some embodiments of the present application, S104 can be implemented by S401-S403 as follows:

[0106] S401, based on the key point trajectory, calculate the joint activity, angular velocity and symmetry.

[0107] S402, based on the joint activity, angular velocity and symmetry, perform stage query to determine the stage event; wherein the stage event includes: preparation, acceleration, maintenance, deceleration and recovery; and the joint activity, angular velocity, symmetry and stage event are determined as a plurality of joint parameters.

[0108] S403, compare the preset joint parameters corresponding to the joint activity, angular velocity, symmetry and stage event to determine the error type.

[0109] For example, for the recognized joint information, the joint angle is calculated by the joint three-point method or the bone segment vector method , and the

[0110]

[0111] wherein ROM represents the joint activity, i.e., limits the joint activity range, represents the angle of the jth joint at time t. The knee joint angular velocity is obtained as follows:

[0112] ,

[0113] The symmetry is defined as:

[0114]

[0115] wherein, The angles representing the left and right joints, respectively, are used to detect compensation according to symmetry. Compensation is determined when the trunk inclination, pelvic inclination, or shoulder elevation amplitude exceeds a threshold or is not synchronized with the phase. For example, if the range of motion (ROM) of the left knee joint is greater than 160°, while the ROM of the right knee joint is 120°, it may indicate that the left knee is overactive and at risk of injury. In this case, the system can prompt the user to take appropriate rest or check.

[0116] Extracting peak values and durations, after acquisition, the phase query token, which is automatically learned by the model during training to represent the phasic state in the movement state (such as preparation, acceleration, maintenance, deceleration, recovery, etc.), is used. The previously acquired joint angles, angular velocities, and symmetries are input into the timing Transformer encoder along with the general token. The phase query token interacts with the movement data in the sequence as a special query vector. After multi-head self-attention mechanism, the phase query token aggregates key features related to movement in the input sequence and generates a phase sequence. This sequence contains phase labels corresponding to each time slice in the movement process, thus directly predicting the phase sequence S t ∈ {preparation, force, backstroke, end}

[0117] ; or auxiliary segmentation with speed threshold and zero crossing.

[0118] In some embodiments of the present application, before S102 is performed, S106-S108 are also performed, as follows:

[0119] S106, before the action recognition and evaluation of the patient's first rehabilitation training, the baseline action of the patient is acquired.

[0120] S107, based on the baseline action, the limb segment ratio, joint range of motion, and affected side label of the patient are determined; based on the limb segment ratio, joint range of motion, and affected side label, the individualized calibration token of the patient is determined.

[0121] S108, based on the individualized calibration token, the pre-determined structural prior token is personalized adjusted to obtain a neural network based on the individualized calibration token; wherein the structural prior token is an adjacency / Laplacian matrix containing a skeleton graph or a relative position bias structural prior token sequence based on a pre-set key point set and a skeleton edge set.

[0122] It should be noted that the structural prior token: pre-models the anatomical position and possible range of motion of the patient's joints to constrain and correct the key point detection results. Individualized calibration: adaptively adjusts the model parameters according to the patient's physical condition, rehabilitation stage, and baseline joint activity to ensure the accuracy and relevance of the evaluation.

[0123] In some embodiments of the present application, after S105 is performed, S109 and S1010 are also performed, as follows:

[0124] S109, based on the error type and the correction instruction of the patient, determining the current rehabilitation training degree of the patient.

[0125] S1010, based on the current rehabilitation training degree, updating the individualized calibration token-based neural network to obtain an updated individualized calibration token-based neural network, so that the next rehabilitation assessment of the patient is performed according to the updated individualized calibration token-based neural network.

[0126] For example, the construction and injection of the structure prior token are as follows:

[0127] Let the set of key points be K, and the set of skeleton edges be E. For each key point Construct a learnable embedding vector The initial features can be obtained by concatenating and linearly mapping the following information.

[0128] 1. The standardized coordinates of the joint under the standard posture (such as T-pose) (unit bone length);

[0129] 2. The bone vector / bone length ratio and left / right side label of the adjacent joint;

[0130] 3. The upper and lower limits of the anatomical range of motion (ROM) of the joint, and the type identification (shoulder / elbow / knee, etc.).

[0131] On the basis of the initial, the adjacency / Laplacian matrix or relative position bias of the skeleton graph is also constructed as a structure bias of attention, so that the attention is more focused on the adjacent joints. Finally, the structure prior token sequence

[0132] It should be noted that the joint set is a set of all joint coordinates in the picture. The skeleton edge set is the connection between joints by the skeleton, which is represented as a line from one joint to another in the input of the neural network. The set of these lines is the skeleton edge set.

[0133] In some embodiments of the present application, the motion recognition and evaluation method for rehabilitation training based on the Transformer further comprises:

[0134] Based on the inertial measurement device installed at the joint of the patient, the joint angular velocity and joint acceleration of the patient are obtained;

[0135] ​The joint angular velocity and joint acceleration based on the initial image acquisition of the patient image are fused by the joint angular velocity and joint acceleration of the patient to obtain updated joint angular velocity and joint acceleration, thereby fusing various joint parameters to determine updated various joint parameters; and the updated various joint parameters are compared with preset joint parameters to determine an error type.

[0136] The purpose of the present application is to provide a pose estimation algorithm for instrument-free rehabilitation training action guidance (i.e. the patient does not need to use an instrument for rehabilitation training, and the algorithm is to estimate the rehabilitation pose of the patient and provide evaluation guidance), which is used to solve the stable key point regression of small, slow and partially occluded actions in the rehabilitation scene, and introduces individualized baseline, outputs multi-dimensional indicators with clinical interpretability and error positioning, and provides executable correction commands.

[0137] The present application specifically provides the technical solutions as shown in the following Figure 4 As shown in the following

[0138] A, human body detection: human body detection is performed on the input image / video to obtain the RoI of the target individual.

[0139] B, pose regression: input the RoI feature into the Transformer containing the structure prior token and individualized calibration token, and output the key point coordinates and confidence.

[0140] C, time sequence stabilization: the time sequence Transformer takes the key points of consecutive frames as input, and outputs the smooth key point trajectory and the variance of each key point to represent the uncertainty.

[0141] D, index calculation: calculate the ROM, angular velocity, phase event (preparation / power return / end), symmetry, compensation index, etc. according to the key point trajectory.

[0142] E, compliance feedback: compare the indicators with the medical order template / contraindication threshold to generate error site-type positioning and correction instructions.

[0143] F, progress modeling: input the training history data into the progress model to update the individualized training prescription or difficulty.

[0144] G, privacy and deployment: complete inference and index generation on the edge, and use quantization, distillation and sparse attention to ensure real-time performance; multi-modal such as IMU / depth can be fused as needed.

[0145] The system of the present application can include but is not limited to: an acquisition module, a detection module, a pose estimation module, a time sequence stabilization module, an index calculation module, and a feedback module. The above-mentioned modules can be realized in the same device, or distributedly realized in the form of edge device+terminal.

[0146] It can be understood that the rehabilitation action is converted into clinical indicators (joint range of motion, angular velocity, symmetry, phase event) through the key point trajectory, compared with the standard action, the amplitude angle difference in the actual action and the standard action is determined, and correction instructions are generated, realizing individualized, interpretable and safe compliance rehabilitation guidance. The system supports edge deployment and privacy protection, and can optionally fuse multi-modal information such as IMU / depth to improve the robustness in occlusion and low-resolution scenes. The application improves the accuracy and usability of pose estimation and evaluation in rehabilitation scenarios.

[0147] Based on the motion recognition and evaluation method of the rehabilitation training based on the Transformer in the above embodiment, the embodiment of the application further provides a motion recognition and evaluation system of rehabilitation training based on the Transformer, as shown in Figure 5 Figure 5 The structure diagram of the motion recognition and evaluation system of rehabilitation training based on the Transformer provided by the embodiment of the application, the motion recognition and evaluation system of rehabilitation training based on the Transformer 5 comprises an acquisition module 501, a determination module 502, a prediction module 503 and a generation module 504, wherein

[0148] The acquisition module 501 is configured to acquire an initial image containing a patient, perform human body detection on the initial image, and obtain a target region image and target region feature information;

[0149] The determination module 502 is configured to perform pose regression processing on the target region feature information and the target region feature information through a pre-determined neural network based on individualized calibration tokens, determine key points and confidence;

[0150] The prediction module 503 is configured to determine respective key points of consecutive frames containing the patient based on the key points, and perform filtering processing and time series prediction on the respective key points of the consecutive frames to obtain a key point trajectory and an uncertainty of the key points;

[0151] The determination module 502 is further configured to calculate a plurality of joint parameters based on the key point trajectory, and compare the plurality of joint parameters with preset joint parameters to determine an error type; wherein the plurality of joint parameters comprise joint range of motion, angular velocity, phase event and symmetry;

[0152] The generation module 504 is configured to generate a correction instruction for the patient according to the error type, so that the patient performs rehabilitation training according to the correction instruction.

[0153] ​In some embodiments of the present application, the acquisition module 501 is further configured to perform grid division on the initial image and extract local features from the divided initial image; perform feature extraction on the local features through a backbone network to obtain a feature map; and perform human body detection through the feature map to obtain an initial human body bounding box and class information.

[0154] The determination module 502 is further configured to remove redundant boxes based on the initial human body bounding box and the class information to determine a human body bounding box; determine the target region image based on the human body bounding box, and determine the target region feature information including scale information, a rotation angle, and a detection confidence.

[0155] In some embodiments of the present application, the acquisition module 501 is further configured to perform region feature extraction on the target region image to obtain a target region feature; perform position encoding and channel normalization processing on the target region feature based on the scale information in the target region feature information to obtain a normalized feature.

[0156] The determination module 502 is further configured to perform pose regression processing on the normalized feature through a plurality of constraint conditions in the neural network based on the individualized calibration token to determine the key points and the confidence.

[0157] In some embodiments of the present application, the plurality of constraint conditions include a skeleton adjacency matrix, a left-right symmetry mapping matrix, and an upper and lower limit of an anatomical angle.

[0158] The acquisition module 501 is further configured to remove irrelevant points in the normalized feature through the skeleton adjacency matrix in the neural network based on the individualized calibration token to obtain screened points.

[0159] The determination module 502 is further configured to perform pose regression processing on the screened points based on the left-right symmetry mapping matrix and the upper and lower limit of the anatomical angle to determine a bone length consistency loss value, a penalty joint angle loss value, an uncertainty loss value, and a joint consistency loss value; perform weighted processing based on the bone length consistency loss value, the penalty joint angle loss value, the uncertainty loss value, and the joint consistency loss value to determine a total loss value; and determine the key points and the confidence based on the total loss value.

[0160] In some embodiments of the present application, the determining module 502 is further configured to calculate joint range of motion, angular velocity, and symmetry based on the key point trajectory; perform stage query based on the joint range of motion, the angular velocity, and the symmetry to determine a stage event; wherein the stage event includes preparation, acceleration, maintenance, deceleration, and recovery; and determine the joint range of motion, the angular velocity, the symmetry, and the stage event as the plurality of joint parameters; compare the plurality of joint parameters with the preset joint parameters to determine the error type.

[0161] In some embodiments of the present application, the obtaining module 501 is further configured to, before determining the key point and the confidence by performing pose regression processing on the target region image and the target region feature information through the neural network based on individual calibration token, obtain a baseline action of a patient before action recognition and evaluation of the patient for the first time of rehabilitation training.

[0162] The determining module 502 is further configured to determine a limb segment ratio, a joint range of motion, and a affected side label of the patient based on the baseline action; determine an individual calibration token of the patient based on the limb segment ratio, the joint range of motion, and the affected side label; and perform individual adjustment on a predetermined structure prior token based on the individual calibration token to obtain the neural network based on individual calibration token; wherein the structure prior token is an adjacency / Laplacian matrix or a structure prior token sequence of relative position bias containing a skeleton graph, which is constructed based on a preset key point set and a skeleton edge set.

[0163] In some embodiments of the present application, the determining module 502 is further configured to generate a correction instruction for the patient according to the error type, so that the patient performs rehabilitation training according to the correction instruction, and then determines a current rehabilitation training degree of the patient based on the error type of the patient and the correction instruction; and updates the neural network based on individual calibration token based on the current rehabilitation training degree to obtain an updated neural network based on individual calibration token, so that the patient is evaluated for rehabilitation again based on the updated neural network based on individual calibration token.

[0164] Based on the above-mentioned action recognition and evaluation method for rehabilitation training based on Transformer, the embodiments of the present application further provide an action recognition and evaluation device for rehabilitation training based on Transformer, as shown in Figure 6 Figure 6 ​A structural diagram of a rehabilitation training motion recognition and evaluation device based on a Transformer is provided in the embodiments of the present application, and the rehabilitation training motion recognition and evaluation device 6 includes a processor 601 and a memory 602. The memory 602 is configured to store a computer program, and the processor 601 is configured to call and run the computer program from the memory to perform the rehabilitation training motion recognition and evaluation method based on the Transformer, as described in the above embodiments.

[0165] In the embodiments of the present application, the processor 601 can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, or a microprocessor. It can be understood that, for different devices, the electronic device used to implement the functions of the processor can also be other devices, and the embodiments of the present application are not limited in this regard.

[0166] The embodiments of the present application provide a computer readable storage medium storing a computer program, which is used to implement the rehabilitation training motion recognition and evaluation method based on the Transformer when executed by a processor.

[0167] For example, the program instructions corresponding to the rehabilitation training motion recognition and evaluation method based on the Transformer in the embodiments of the present application can be stored on a storage medium such as an optical disc, a hard disk, a USB flash disk, etc. When the program instructions corresponding to the rehabilitation training motion recognition and evaluation method based on the Transformer in the storage medium are read by an electronic device or executed, the rehabilitation training motion recognition and evaluation method based on the Transformer can be implemented, as described in any of the above embodiments.

[0168] In addition, each functional module in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional module.

[0169] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer readable storage medium based on such understanding. The technical solutions of the embodiments can essentially or contribute to the prior art or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the embodiments. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0170] It should be understood that the "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiments are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" or "in some embodiments" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in various embodiments of the present application, the size of the sequence number of the above processes does not mean the execution order, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above sequence number of the embodiments of the present application is only for description, not representing the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between each embodiment, and the same or similar parts can be referred to each other. For the sake of brevity, this paper will not repeat here.

[0171] The above-mentioned modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical modules; they can be located in one place or distributed on multiple network units; part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0172] In addition, each functional module in each embodiment of the present application can be integrated in one processing unit, or each module can be a separate unit, or two or more modules can be integrated in one unit; the integrated module can be realized in the form of hardware or hardware plus software function unit.

[0173] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by relevant hardware of program instructions, the foregoing program can be stored in a computer readable storage medium, and the program executes the steps of the method embodiments when executed.

[0174] The methods disclosed in the several method embodiments provided by the embodiments of the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0175] The features disclosed in the several product embodiments provided by the embodiments of the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0176] The features disclosed in the several method or device embodiments provided by the embodiments of the present application can be combined arbitrarily without conflict to obtain new method or device embodiments.

[0177] The above is only the implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

1. A method for motion recognition and evaluation of rehabilitation training based on Transformer, characterized in that, The method comprises: acquiring an initial image containing a patient, performing human body detection on the initial image to obtain a target region image and target region feature information; performing pose regression processing on the target region image and the target region feature information through a pre-determined neural network based on individualized calibration tokens to determine key points and confidence; based on the key points, determining respective key points of continuous frames containing the patient, and performing filtering processing and time sequence prediction on the respective key points of the continuous frames to obtain key point trajectories and uncertainty of the key points; based on the key point trajectories, calculating a plurality of joint parameters, and comparing the plurality of joint parameters with preset joint parameters to determine an error type, wherein the plurality of joint parameters include joint range of motion, angular velocity, phase event, and symmetry; generating a correction instruction for the patient according to the error type, so that the patient performs rehabilitation training according to the correction instruction; wherein the pose regression processing on the target region image and the target region feature information through the pre-determined neural network based on individualized calibration tokens to determine key points and confidence comprises: performing region feature extraction on the target region image to obtain target region features; based on scale information in the target region feature information, performing position encoding and channel normalization processing on the target region features to obtain normalized features; performing pose regression processing on the normalized features through a plurality of constraint conditions in the neural network based on individualized calibration tokens to determine the key points and the confidence.

2. The method of claim 1, wherein, The human body detection on the initial image to obtain a target region image and target region feature information comprises: performing grid division on the initial image, and extracting local features from the divided initial image; performing feature extraction on the local features through a backbone network to obtain a feature map, and performing human body detection through the feature map to obtain an initial human body bounding box and class information; based on the initial human body bounding box and the class information, removing redundant boxes to determine a human body bounding box; based on the human body bounding box, determining the target region image, and the target region feature information containing scale information, rotation angle, and detection confidence.

3. The method of claim 1, wherein, The plurality of constraint conditions include a skeleton adjacency matrix, a left-right symmetry mapping matrix, and an upper and lower limit of an anatomical angle; The pose regression processing on the normalized features through the plurality of constraint conditions in the neural network based on individualized calibration tokens to determine the key points and the confidence comprises: removing irrelevant points in the normalized features through the skeleton adjacency matrix in the neural network based on individualized calibration tokens to obtain screened points; based on the left-right symmetry mapping matrix and the upper and lower limit of the anatomical angle, performing pose regression processing on the screened points to determine bone length consistency loss, penalty joint angle loss, uncertainty loss, and joint consistency loss. The total loss value is determined based on the bone consistency loss value, the penalty joint angle loss value, the uncertainty loss value, and the joint consistency loss value. The key points and the confidence are determined based on the total loss value.

4. The method of claim 1, wherein, A plurality of joint parameters are calculated based on the key point trajectory. The plurality of joint parameters are compared with preset joint parameters to determine an error type, including: Joint range of motion, angular velocity, and symmetry are calculated based on the key point trajectory. Stage events are determined by stage query based on the joint range of motion, angular velocity, and symmetry, including: preparation, acceleration, maintenance, deceleration, and recovery; and the joint range of motion, angular velocity, symmetry, and stage events are determined as the plurality of joint parameters. The plurality of joint parameters are compared with the preset joint parameters to determine the error type.

5. The method of claim 1, wherein, Before the key points and the confidence are determined by the pre-determined neural network based on the individualized calibration token for the posture regression processing of the target region image and the target region feature information, the method further includes: Before the action recognition and evaluation of the patient for the first time for rehabilitation training, the baseline action of the patient is obtained; Based on the baseline action, the limb segment ratio, joint range of motion, and affected side label of the patient are determined; and based on the limb segment ratio, joint range of motion, and affected side label, the individualized calibration token of the patient is determined. Based on the individualized calibration token, the pre-determined structure prior token is personalized adjusted to obtain the neural network based on the individualized calibration token; wherein the structure prior token is an adjacency / Laplacian matrix or a structure prior token sequence of relative position bias containing a skeleton graph, which is constructed based on a preset key point set and a skeleton edge set.

6. The method of claim 1, wherein, After the correction instruction for the patient is generated according to the error type, so that the patient performs rehabilitation training according to the correction instruction, the method further includes: Based on the error type of the patient and the correction instruction, the current rehabilitation training degree of the patient is determined; Based on the current rehabilitation training degree, the neural network based on the individualized calibration token is updated to obtain an updated neural network based on the individualized calibration token, so that the patient is evaluated for the next rehabilitation according to the updated neural network based on the individualized calibration token.

7. A motion recognition and evaluation system for rehabilitation training based on Transformer, characterized in that, The action recognition and evaluation system of the rehabilitation training based on the Transformer includes an acquisition module, a determination module, a prediction module, and a generation module, wherein, The acquisition module is configured to acquire an initial image containing a patient, perform human body detection on the initial image, and obtain a target region image and target region feature information; The determination module is configured to determine key points and confidence by pre-determined neural network based on individualized calibration token for posture regression processing of the target region feature information and target region feature information; The prediction module is configured to determine respective key points of continuous frames containing the patient based on the key points, and perform filtering processing and time series prediction on the respective key points of the continuous frames to obtain a key point trajectory and an uncertainty of the key points. The determination module is further configured to calculate a plurality of joint parameters based on the key point trajectory, and compare the plurality of joint parameters with preset joint parameters to determine an error type, wherein the plurality of joint parameters include joint range of motion, angular velocity, phase event, and symmetry. The generation module is configured to generate a correction instruction for the patient according to the error type, so that the patient performs rehabilitation training according to the correction instruction. The determination module is further configured to perform region feature extraction on the target region image to obtain target region features, perform position encoding and channel normalization processing on the target region features based on scale information in the target region feature information to obtain normalized features, and perform pose regression processing on the normalized features through a plurality of constraint conditions in the neural network based on the individualized calibration token to determine the key points and the confidence.

8. A motion recognition and evaluation device for rehabilitation training based on a Transformer, characterized in that Comprising: a processor and a memory, the memory is configured to store a computer program; the processor is configured to call and run the computer program from the memory to execute the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, executable instructions are stored for causing the processor to execute when the method of any one of claims 1 to 6 is implemented.

Images