Shoulder joint rehabilitation action image recognition and real-time posture correction feedback method and system

Abstract

The invention discloses a shoulder joint rehabilitation action image recognition and real-time posture correction feedback method and system, and belongs to the technical field of medical image processing and computer vision, and the method comprises the steps: collecting a rehabilitation training image frame sequence through an RGB camera; inputting the image into a human body analysis model, extracting skeleton key point coordinates and calculating a joint angle sequence; constructing the skeleton key points into a space-time diagram structure, and inputting the space-time diagram structure into a space-time diagram convolutional network for action classification and recognition; aligning and comparing the actual joint angle with the standard action template by adopting a dynamic time warping algorithm to calculate a deviation vector and an action compliance score; a standard posture skeleton line and an angle deviation indicating arrow are overlaid and rendered on an image, the compliance degree of each joint is displayed in a color gradient mode, an action compliance degree score and personalized correction suggestions are output, the error posture recognition accuracy is larger than 92%, the feedback delay is smaller than 100 ms, and the technical problem that home rehabilitation lacks professional action guidance is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing and computer vision, in particular to a shoulder joint rehabilitation action image recognition and real-time posture correction feedback method and system. BACKGROUND

[0002] Rotator cuff injury is a common shoulder joint disease in clinic, and its rehabilitation treatment requires patients to adhere to standard rehabilitation training actions for a long time. In the traditional rehabilitation mode, patients need to regularly go to medical institutions to receive guidance from rehabilitation therapists. However, due to factors such as uneven distribution of medical resources and long rehabilitation period, home rehabilitation has become an important part of the rehabilitation treatment of patients with rotator cuff injury. Due to the lack of real-time guidance from professionals, patients are prone to problems such as non-standard actions and incorrect postures during home rehabilitation, which not only reduces the rehabilitation effect, but also may cause injuries to other parts due to compensatory actions, or even aggravate the original injury.

[0003] The core of rotator cuff injury rehabilitation training is to restore the range of motion of the shoulder joint and the strength of the surrounding muscles in a targeted manner. Common rehabilitation actions include shoulder joint abduction movement, shoulder joint flexion movement, shoulder joint internal and external rotation movement, etc. These rehabilitation actions have strict requirements for joint angle, movement amplitude and trunk stability. Improper execution will lead to a variety of problems: insufficient angle will reduce the effectiveness of rehabilitation training and make it difficult to achieve the expected functional recovery goal; excessive angle may cause secondary injury to tissues that have not fully healed; compensating for insufficient shoulder joint range of motion by tilting or leaning the trunk will lead to muscle strain and joint problems in other parts. Therefore, real-time monitoring and correction guidance of rehabilitation actions are of great significance to ensure rehabilitation effectiveness and patient safety.

[0004] The prior art CN116152299A discloses a motion state detection method and device, a storage medium and an electronic device. The scheme determines global image features for each frame of image of a to-be-detected video, performs dimension reduction processing on the global image features to determine local image features of a target object, determines motion state features according to time information and spatial information based on the similarity between the local image features and the local image features of other frames of image, and further determines a motion state trajectory and performs detection. Although the scheme can reduce the interference of background noise on motion state detection, it has the following technical problems: first, the scheme mainly focuses on the recognition and trajectory tracking of motion states, does not involve the normative evaluation of rehabilitation training actions, and cannot determine whether the actions meet the rehabilitation standards; second, the scheme does not establish a standard action template for comparative analysis, and cannot quantitatively calculate the deviation degree between the current action and the standard action; third, the scheme lacks a real-time deviation correction feedback mechanism, and cannot give intuitive posture adjustment guidance during the execution of the action by the patient; fourth, the scheme does not consider the detection and punishment of compensatory actions, and cannot effectively identify the problem that the patient compensates for the insufficient shoulder joint range of motion through trunk movement.

[0005] In addition, although the existing video teaching scheme can show standard rehabilitation actions to the patient, it belongs to one-way information transmission and cannot provide personalized guidance according to the actual execution of the patient. The patient cannot accurately determine the difference between his own action and the standard action, especially when it involves motion parameters such as shoulder joint internal and external rotation angles that are difficult to observe by oneself, and it is difficult to meet the standard requirements by visual imitation alone. Although the rehabilitation monitoring scheme based on wearable sensors can accurately collect motion data, it requires the patient to wear additional equipment, increasing the use cost and operation complexity and reducing the convenience and compliance of home rehabilitation.

[0006] Therefore, there is an urgent need for a technical scheme that can accurately identify the type of rehabilitation action, accurately evaluate the degree of posture deviation, and provide intuitive deviation correction feedback through augmented reality technology. SUMMARY

[0007] The purpose of the present application is to provide a shoulder joint rehabilitation action image recognition and real-time posture deviation correction feedback method and system to solve the technical problems in the prior art that shoulder rotator cuff injury patients lack professional action guidance during home rehabilitation, compensatory actions are aggravated and there is a risk of secondary injury due to incorrect posture, and video teaching cannot provide personalized real-time correction.

[0008] To achieve the above object, the present application provides a shoulder joint rehabilitation movement image recognition and real-time posture deviation correction feedback method, which comprises: a rehabilitation training image acquisition step, which acquires image frame sequences in a shoulder joint rehabilitation training process in real time through an RGB camera at a preset frame rate, and pre-processes the image frame sequences to obtain standardized image data; a skeletal key point extraction step, which inputs the standardized image data into a human body analysis model, extracts skeletal key point coordinates of a patient's upper limbs and torso, calculates shoulder joint abduction angle, shoulder joint forward flexion angle, shoulder joint internal and external rotation angle, elbow joint flexion and extension angle, torso lateral tilt angle and torso forward tilt angle based on the skeletal key point coordinates, and generates joint angle sequences; a rehabilitation movement recognition step, which constructs the skeletal key point coordinates into a space-time graph structure, takes the skeletal key points as graph nodes, takes the skeletal connections between adjacent key points as space edges, and takes the time sequence connections of the same key points between adjacent frames as time edges, inputs the space-time graph structure into a space-time graph convolution network for feature extraction and movement classification, and outputs movement category labels and corresponding classification confidence of the current rehabilitation movement; a posture deviation evaluation step, which retrieves corresponding standard joint angle sequences from a pre-stored standard movement template library according to the movement category labels, aligns the joint angle sequences with the standard joint angle sequences frame by frame using a dynamic time warping algorithm, calculates deviation vectors of each joint angle, and calculates movement compliance scores based on the deviation vectors; and an augmented reality correction feedback step, which superimposes and renders standard posture skeletal lines on the standardized image data, renders angle deviation indication arrows at joint positions where the deviation exceeds a preset deviation threshold according to the deviation vectors, the direction of the angle deviation indication arrow represents the direction that needs to be adjusted, the length of the angle deviation indication arrow is positively correlated with the deviation amplitude, sets the display color of the corresponding skeletal line segment according to the compliance degree of each joint, and generates a correction feedback result containing movement compliance score, deviation part label and personalized correction suggestion.

[0009] Preferably, in the skeletal key point extraction step, the human body analysis model outputs detection confidence of each skeletal key point, and when the detection confidence is lower than a preset confidence threshold, a Kalman filter is used to predict the skeletal key point coordinates of the current frame according to the skeletal key point coordinates of the previous frame.

[0010] Preferably, in the rehabilitation movement recognition step, the space-time graph convolution network comprises a spatial graph convolution layer, a time convolution layer and a global average pooling layer connected in sequence, the spatial graph convolution layer adopts a separable convolution structure and dynamically adjusts the weights of different skeletal connection edges through an attention mechanism.

[0011] Preferably, in the posture deviation evaluation step, the method for calculating the motion compliance score comprises: calculating the weighted root mean square value of each joint angle deviation as an angle deviation component, calculating the dynamic time warping distance between the actual motion trajectory and the standard motion trajectory as a trajectory deviation component, detecting whether the trunk lateral tilt angle and the trunk forward tilt angle exceed the preset compensation threshold to calculate a compensation penalty component, and performing weighted summation on the angle deviation component, the trajectory deviation component and the compensation penalty component to obtain the motion compliance score.

[0012] Preferably, in the augmented reality deviation correction feedback step, the rule for setting the display color of the skeleton line segment according to the deviation degree is: displaying green when the joint angle deviation is less than a first preset angle threshold, displaying yellow when the joint angle deviation is greater than or equal to the first preset angle threshold and less than a second preset angle threshold, and displaying red when the joint angle deviation is greater than or equal to the second preset angle threshold.

[0013] The application also provides a shoulder joint rehabilitation motion image recognition and real-time posture deviation correction feedback system, comprising: an image acquisition module for acquiring image frame sequences in a shoulder joint rehabilitation training process in real time at a preset frame rate through an RGB camera, and pre-processing the image frame sequences to obtain standardized image data; a skeleton key point extraction module for inputting the standardized image data into a human body analysis model, extracting skeleton key point coordinates of the upper limbs and trunk of a patient, and calculating a joint angle sequence based on the skeleton key point coordinates; a rehabilitation motion recognition module for constructing the skeleton key point coordinates into a spatio-temporal graph structure, inputting the spatio-temporal graph structure into a spatio-temporal graph convolution network for feature extraction and motion classification, and outputting a motion category label and a corresponding classification confidence of the current rehabilitation motion; a posture deviation evaluation module for retrieving a corresponding standard joint angle sequence from a pre-stored standard motion template library according to the motion category label, aligning the joint angle sequence with the standard joint angle sequence using a dynamic time warping algorithm, and calculating a deviation vector and a motion compliance score; and an augmented reality deviation correction feedback module for superimposing and rendering a standard posture skeleton line and an angle deviation indication arrow on the standardized image data, setting the display color of the corresponding skeleton line segment according to the compliance degree of each joint, and generating a deviation correction feedback result comprising the motion compliance score, a deviation part label and a personalized correction suggestion.

[0014] The beneficial effects of the present application include: first, by real-time acquisition of rehabilitation training images through an RGB camera and extraction of bone key point coordinates, the kinematic parameters of patient rehabilitation movements can be obtained without increasing additional sensing devices, reducing the device threshold of home rehabilitation; second, using a spatio-temporal graph convolution network to recognize the action of the bone key point sequence, the current rehabilitation action type can be accurately classified, and the error posture recognition accuracy is greater than 92%; third, by using a dynamic time warping algorithm to align and compare the actual joint angle sequence with the standard action template, the deviation degree of each joint can be quantitatively evaluated, providing accurate data support for the correction feedback; fourth, the augmented reality visualization correction function is innovatively designed, the standard posture bone line and angle deviation indication arrow are superimposed and rendered on the collected image in real time, the compliance degree of each joint is intuitively displayed by color gradient, the feedback delay is less than 100 milliseconds, and real-time interactive correction guidance can be supported; fifth, the system outputs the action compliance score and personalized correction suggestions, helping patients to improve the quality of rehabilitation movements in a targeted manner, reducing the risk of secondary injury caused by error posture. BRIEF DESCRIPTION OF DRAWINGS

[0015] Fig. 1 is a flowchart of a shoulder joint rehabilitation action image recognition and real-time posture correction feedback method provided by an embodiment of the present application.

[0016] Fig. 2 is an architecture diagram of a shoulder joint rehabilitation action image recognition and real-time posture correction feedback system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0017] Please refer to the accompanying Figs. 1-2 , the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0018] The embodiment of the present application provides a shoulder joint rehabilitation action image recognition and real-time posture correction feedback method, as shown in Fig. 1 , the method comprises a rehabilitation training image acquisition step, a bone key point extraction step, a rehabilitation action recognition step, a posture deviation evaluation step and an augmented reality correction feedback step. Each step will be described in detail below.

[0019] The rehabilitation training image acquisition step acquires image frame sequences in the shoulder joint rehabilitation training process in real time through an RGB camera at a preset frame rate, and pre-processes the image frame sequences to obtain standardized image data.

[0020] In this embodiment, the RGB camera has a resolution of 1280 pixels by 720 pixels, and the preset frame rate ranges from 25 frames per second to 30 frames per second. Preferably, the preset frame rate is 30 frames per second to ensure that the temporal resolution of motion capture meets the needs of rehabilitation movement analysis. The camera is installed approximately 1.5 to 2.5 meters in front of the patient, at a height roughly level with the patient's shoulder joint, to ensure complete capture of the patient's upper limb and trunk movements.

[0021] Image preprocessing includes the following steps. First, image scaling is performed to adjust the original image to a fixed size to meet the input requirements of the subsequent human body analysis model. In this embodiment, the target size is 640 pixels multiplied by 480 pixels. Next, color space conversion is performed, converting the RGB image to the format required for neural network processing, and normalization is applied, mapping pixel values ​​from integers (0 to 255) to floating-point numbers (0 to 1). Then, illumination compensation is performed, using an adaptive histogram equalization algorithm to adjust image brightness, reducing the impact of different lighting conditions on the accuracy of skeletal keypoint detection. Finally, temporal caching is performed, establishing a sliding window to cache the image data of the most recent frames. In this embodiment, the window length is set to 32 frames for action recognition by the subsequent spatiotemporal graph convolutional network.

[0022] The preprocessed, standardized image data is then sent to the skeletal keypoint extraction step for further processing. In the test environment of this embodiment, the average time for image acquisition is approximately 8 milliseconds, and the average time for preprocessing is approximately 5 milliseconds.

[0023] The skeletal key point extraction step inputs standardized image data into the human body analysis model, extracts the coordinates of skeletal key points of the patient's upper limbs and trunk, and calculates the shoulder abduction angle, shoulder flexion angle, shoulder internal and external rotation angle, elbow flexion and extension angle, trunk lateral tilt angle, and trunk forward tilt angle based on the skeletal key point coordinates, generating a joint angle sequence.

[0024] In this embodiment, the human body analysis model employs a deep learning-based pose estimation network. This network takes standardized image data as input and outputs the two-dimensional coordinates of 17 skeletal key points of the upper limbs and trunk, along with their corresponding detection confidence scores. These 17 skeletal key points include: nose tip, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. For shoulder joint rehabilitation movement analysis, the focus is on the eight key points related to the upper limbs and trunk: left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, and right hip.

[0025] The human body analysis model employs an encoder-decoder architecture. The encoder uses ResNet50 as the backbone to extract multi-scale image features, while the decoder uses deconvolutional layers to progressively upsample and restore spatial resolution. Skip connections are placed between the encoder and decoder to fuse low-level detail features and high-level semantic features. Finally, the network predicts heatmaps of key points for each skeleton through multiple parallel convolutional branches; the location with the largest response value in the heatmap is the predicted coordinate of the key point. The model is jointly trained on the COCO human pose dataset and a self-collected rehabilitation action dataset to improve its adaptability to shoulder joint rehabilitation scenarios.

[0026] The output format of the skeletal keypoint coordinates is an N x 3 matrix, where N is the number of keypoints, and the three dimensions are the x-coordinate, y-coordinate, and detection confidence score. The detection confidence score reflects the reliability of the keypoint prediction, ranging from 0 to 1, with higher values ​​indicating more reliable detection results. When the detection confidence score of a keypoint falls below a preset confidence threshold, it indicates that the keypoint may be occluded or the detection has failed. In this case, a Kalman filter is used to predict the skeletal keypoint coordinates of the current frame based on the skeletal keypoint coordinates of the previous frame to ensure the continuity and stability of the keypoint sequence. Preferably, the preset confidence threshold is set to 0.5.

[0027] The state vector of a Kalman filter is defined as the position and velocity of the keypoints, comprising four state variables: x-coordinate, y-coordinate, x-direction velocity, and y-direction velocity. The state transition matrix is ​​constructed based on a uniform motion model, while the observation matrix only observes the position component. When the detection confidence is higher than a preset confidence threshold, the filter state is updated using the actual detection result; when the detection confidence is lower than the preset confidence threshold, only the prediction step is performed without updating the state, and the predicted position is output as the keypoint coordinates for the current frame. This approach effectively handles situations where keypoints are briefly occluded, ensuring the temporal continuity of the skeletal keypoint sequence.

[0028] Based on the extracted skeletal key point coordinates, the six joint angles required for shoulder joint rehabilitation movement analysis are calculated. The calculation method for each joint angle is given below.

[0029] The shoulder abduction angle is calculated as follows. First, construct the upper arm vector, pointing from the key point of the shoulder joint to the key point of the elbow joint. Then, construct the trunk midline vector, pointing from the midpoint between the left and right hips to the midpoint between the left and right shoulders. The formula for calculating the shoulder abduction angle is: , in, The shoulder abduction angle, The upper arm vector, The vector is the midline vector of the torso. This represents the vector dot product operation. This represents the magnitude of the vector.

[0030] The method for calculating the shoulder flexion angle is as follows: Project the upper arm vector onto the sagittal plane of the human body, and calculate the angle between the projected vector and the trunk midline vector. The formula for calculating the shoulder flexion angle is: , in, The shoulder joint flexion angle, and These are the components of the upper arm vector along the y-axis and z-axis, respectively. and These are the components of the torso's central axis vector along the y-axis and z-axis, respectively.

[0031] The method for calculating the internal and external rotation angles of the shoulder joint is as follows. First, construct the forearm vector, pointing from the key point of the elbow joint to the key point of the wrist joint. Then, construct a reference plane perpendicular to the upper arm and calculate the projection direction of the forearm vector onto this plane. The formula for calculating the internal and external rotation angles of the shoulder joint is: , in, The internal and external rotation angles of the shoulder joint. Forearm vector, The unit vector of the inward direction in the reference plane. It is the forward direction unit vector in the reference plane.

[0032] The method for calculating the elbow flexion-extension angle is as follows: Calculate the angle between the upper arm vector and the forearm vector. The formula for calculating the elbow flexion-extension angle is: , in, The elbow flexion-extension angle.

[0033] The method for calculating the trunk tilt angle is as follows: Calculate the angle between the line connecting the left and right shoulders and the horizontal direction. The formula for calculating the trunk tilt angle is: , in, The angle of trunk tilt. The coordinates of the key points on the right shoulder. The coordinates of the key point on the left shoulder, subscript and These represent the coordinate values ​​along the x-axis and y-axis, respectively.

[0034] The method for calculating the torso forward tilt angle is as follows: Calculate the angle between the torso midline vector and the vertical direction. The formula for calculating the torso forward tilt angle is: , in, Forward tilt angle of the torso It is a unit vector in the vertical direction.

[0035] Based on the six joint angle calculation methods described above, the joint angle values ​​are calculated for each frame of the image, generating a joint angle sequence. The joint angle sequence is formatted as a T-multiplied-6 matrix, where T is the number of time frames and 6 is the number of joint angles. The skeletal keypoint coordinates and the joint angle sequence are sent to the rehabilitation movement recognition step and the posture deviation assessment step for further processing. In the test environment of this embodiment, the average time for skeletal keypoint extraction and joint angle calculation is approximately 25 milliseconds.

[0036] The rehabilitation action recognition step constructs a spatiotemporal graph structure using the coordinates of skeletal key points. The skeletal key points are used as graph nodes, the skeletal connections between adjacent key points are used as spatial edges, and the temporal connections of the same key points between adjacent frames are used as temporal edges. The spatiotemporal graph structure is input into a spatiotemporal graph convolutional network for feature extraction and action classification, and the action category label and corresponding classification confidence of the current rehabilitation action are output.

[0037] The spatiotemporal graph structure is constructed as follows. Let the sequence of skeletal keypoints consist of coordinate data for N keypoints across T consecutive frames. The spatiotemporal graph G is constructed as (V, E), where V is the set of graph nodes and E is the set of graph edges. The graph node set V contains T multiplied by N nodes, each corresponding to a skeletal keypoint in one frame. The graph edge set E consists of two parts: a spatial edge set and a temporal edge set. Spatial edges connect adjacent skeletal keypoints within the same frame, reflecting the topological structure of the human skeleton. In this embodiment, spatial edges are defined according to the connection relationships of the human skeleton, including connections such as: left shoulder to left elbow, left elbow to left wrist, right shoulder to right elbow, right elbow to right wrist, left shoulder to right shoulder, left shoulder to left hip, right shoulder to right hip, and left hip to right hip. Temporal edges connect corresponding skeletal keypoints in adjacent frames, reflecting the temporal motion trajectory of the keypoints.

[0038] In addition to spatial edges based on skeletal physical connections, this embodiment also introduces adaptive spatial edges. Adaptive spatial edges, by learning data-driven adjacency relationships, can capture potential associations between non-physically connected key points, such as the coordinated movement patterns between the left wrist and right shoulder in certain rehabilitation movements. The adjacency matrix of the adaptive spatial edges is represented by learnable parameters and is automatically optimized during network training.

[0039] The overall architecture of the spatiotemporal graph convolutional network is as follows. The network input consists of the spatiotemporal graph structure and the feature vectors of each node. The node feature vectors include the two-dimensional coordinates of the keypoints, the detection confidence score, and the normalized coordinates relative to a reference point. The reference point is chosen as the midpoint between the left and right hips. Using normalized coordinates eliminates the influence of patient position changes in the image on action recognition. The network comprises nine sequentially connected spatiotemporal graph convolutional modules, a global average pooling layer, and a fully connected classification layer. The first three spatiotemporal graph convolutional modules have 64 channels, the middle three modules have 128 channels, and the last three modules have 256 channels. Each spatiotemporal graph convolutional module consists of a cascaded spatial graph convolutional layer and a temporal convolutional layer, with residual connections between modules enhancing gradient propagation.

[0040] The spatial graph convolutional layer employs a separable convolutional structure, decomposing the adjacency matrix into three subsets for separate convolution. These subsets correspond to the root node itself, its centripetal neighbors, and its centrifugal neighbors, respectively, to capture different types of spatial dependencies. The centripetal direction is defined as the direction towards the center of the torso, and the centrifugal direction is defined as the direction away from the center of the torso. The computation process of spatial graph convolution is as follows: first, the adjacency matrix is ​​divided into three subsets; then, a convolution weight matrix is ​​applied to each subset for feature transformation; finally, the convolution results of the three subsets are summed to obtain the output features.

[0041] The spatial graph convolutional layer also introduces an attention mechanism to dynamically adjust the weights of different skeletal connections. The attention weights are calculated based on the dot product similarity of features from adjacent nodes; connections with higher similarity receive greater weights, enabling the network to adaptively focus on skeletal connections more important for current action recognition. In shoulder flexion, the connection from the shoulder to the elbow joint receives higher attention weights; in shoulder abduction, the connection from the left shoulder to the right shoulder receives higher attention weights, demonstrating the ability to focus on action-specific features.

[0042] The temporal convolutional layer performs a one-dimensional convolution operation on the output of the spatial graph convolutional layer, extracting motion features along the temporal dimension. In this embodiment, the size of the temporal convolutional kernel is set to 9 to capture temporal motion patterns spanning a time window of approximately 300 milliseconds (9 frames). The temporal convolutional layer also employs dilated convolution techniques, obtaining multi-scale temporal receptive fields by setting different dilation rates. In this embodiment, the dilation rates of the nine spatiotemporal graph convolutional modules are set to 1, 1, 1, 2, 2, 2, 4, 4, 4, respectively, enabling the network to simultaneously capture short-term motion details and long-term motion trends.

[0043] The global average pooling layer pools the output of the last spatiotemporal graph convolutional module, averaging it across the time and node dimensions to aggregate the spatiotemporal features into a fixed-length 256-dimensional feature vector. The fully connected classification layer maps the feature vector to the predicted probability of each rehabilitation movement category, using the Softmax activation function to output the probability distribution. In this embodiment, the rehabilitation movement categories include eight commonly used rotator cuff injury rehabilitation movements: shoulder flexion, shoulder abduction, shoulder internal rotation, shoulder external rotation, scapular stabilization training, pendulum exercises, wall climbing exercises, and pulley exercises.

[0044] The spatiotemporal graph convolutional network was trained using the cross-entropy loss function, with parameters updated using the Adam optimizer. The initial learning rate was set to 0.001, and cosine annealing was used to adjust the learning rate. The training dataset consisted of 5000 video clips, including standard movement videos recorded by rehabilitation experts and clinically collected patient rehabilitation movement videos, divided into training, validation, and test sets in an 8:1:1 ratio. To enhance the model's generalization ability, data augmentation strategies were employed during training, including random temporal pruning, random spatial rotation, and random keypoint noise addition.

[0045] The output of the spatiotemporal graph convolutional network includes action category labels and classification confidence scores. The action category label is the index of the category with the highest probability, and the classification confidence score is the Softmax output probability value corresponding to that category. The action category label is sent to the pose deviation evaluation step to retrieve the corresponding standard action template, and the classification confidence score is sent to the augmented reality correction feedback step to determine the reliability of the recognition result. In the test environment of this embodiment, the average inference time of the spatiotemporal graph convolutional network is approximately 35 milliseconds. On a test set containing 500 rehabilitation action video clips, the action recognition accuracy reaches 94.2%, and the erroneous pose recognition accuracy is greater than 92%.

[0046] The posture deviation assessment step retrieves the corresponding standard joint angle sequence from the pre-stored standard motion template library based on the motion category label. The dynamic time warping algorithm is used to align the joint angle sequence with the standard joint angle sequence frame by frame, calculate the deviation vector of each joint angle, and calculate the motion compliance score based on the deviation vector.

[0047] The standard movement template library is constructed as follows: Rehabilitation experts perform various rehabilitation training movements according to standardized movement requirements, using the same skeletal key point extraction method as the user end to collect joint angle sequences as standard templates. Multiple standard templates with different execution speeds and amplitudes are collected for each type of rehabilitation movement to accommodate the different rehabilitation progress of patients. The standard movement template library is indexed by movement category, storing multiple sets of standard joint angle sequences and their metadata information for each type of movement.

[0048] The execution process of the dynamic time warping algorithm is as follows. Let the joint angle sequence actually performed by the patient be A, with a length of T_a frames; let the joint angle sequence of the standard action template be B, with a length of T_b frames. The dynamic time warping algorithm finds an optimal alignment path from (1,1) to (T_a, T_b) to minimize the alignment cost.

[0049] This invention improves upon the standard dynamic time warping algorithm by introducing joint weights and a compensation penalty factor. The cumulative cost calculation formula for the improved dynamic time warping algorithm is as follows: , in, From the starting point to the position The minimum cumulative cost, For frames and frame The weighted local distance between them The first of the actual joint angle sequence frame, The first of the standard joint angle sequence frame.

[0050] The formula for calculating the weighted local distance is: , in, In this embodiment, the number of joint angles is... Equals 6, For the first Weighting coefficients for each joint angle; For the actual joint angle sequence, the first Frame number Each joint angle value For the standard joint angle sequence, the first Frame number Each joint angle value The compensation penalty coefficient, For frames The compensation penalty value.

[0051] In this embodiment, the principle for setting the joint weight coefficients is as follows: the weight of the shoulder joint angle is higher than that of the elbow joint angle, in order to emphasize the assessment of the core movements of shoulder joint rehabilitation training. Preferably, the weight of the shoulder abduction angle is 0.25, the weight of the shoulder flexion angle is 0.20, the weight of the shoulder internal and external rotation angles is 0.20, the weight of the elbow flexion and extension angles is 0.15, the weight of the trunk lateral tilt angle is 0.10, and the weight of the trunk forward tilt angle is 0.10.

[0052] The compensation penalty value is calculated as follows. When the trunk tilt angle or trunk forward tilt angle exceeds the preset compensation threshold, the patient is considered to be engaging in compensatory behavior, and a penalty cost needs to be added. The formula for calculating the compensation penalty value is: , in, For frames The torso tilt angle, For frames The forward lean angle of the torso, and These are the compensation thresholds for trunk tilting to the side and tilting forward, respectively. Preferably, the compensation penalty coefficient... Set to 10, the trunk tilt compensation threshold is set to 10 degrees, and the trunk forward tilt compensation threshold is set to 15 degrees.

[0053] Based on the alignment results from the dynamic time warping algorithm, the deviation vector of each joint angle is calculated. Let the first joint be the joint angle of the first joint on the alignment path. The actual joint angle of the frame is The corresponding standard joint angle is The formula for calculating the deviation vector is: , in, For the first The frame's deviation vector contains deviation values ​​for six joint angles.

[0054] The calculation method for the action compliance score is as follows. The action compliance score comprehensively considers three dimensions: angle deviation component, trajectory deviation component, and compensation penalty component.

[0055] The formula for calculating the angular deviation component is: , in, Scoring is given for the angle deviation component. The number of frames after alignment. The number of joint angles. This is the sensitivity coefficient for angular deviation. For the first Frame number The deviation value of each joint angle. Preferably, the angle deviation sensitivity coefficient. Set to 2.

[0056] The formula for calculating the trajectory deviation component is: , in, Scoring for the trajectory deviation component. The ultimate cumulative cost of dynamic time warping, This is the trajectory deviation sensitivity coefficient. Preferably, the trajectory deviation sensitivity coefficient... Set it to 0.1.

[0057] The formula for calculating the amount of compensation penalty is: , in, The compensation penalty score is calculated based on the amount of compensation. This is the sensitivity coefficient for compensation penalty. Preferably, the sensitivity coefficient for compensation penalty is... Set it to 5.

[0058] The final formula for calculating the compliance score is: , in, Scoring the compliance of the action. , , The fusion weights for each component satisfy the following conditions: Preferably, the angular deviation component weighting Set to 0.5, trajectory deviation component weight Set to 0.3, the compensatory penalty component weight. Set it to 0.2.

[0059] The deviation vector sequence and action compliance score are sent to the augmented reality correction feedback step for visualization rendering and correction suggestion generation. In the test environment of this embodiment, the average time for posture deviation assessment is approximately 15 milliseconds.

[0060] The augmented reality correction feedback step overlays and renders standard pose skeleton lines on standardized image data. Based on the deviation vector, it renders angle deviation indicator arrows at joint positions where the deviation exceeds a preset deviation threshold. The direction of the angle deviation indicator arrows indicates the direction that needs to be adjusted, and the length of the angle deviation indicator arrows is positively correlated with the deviation magnitude. The display color of the corresponding skeleton line segment is set according to the compliance level of each joint, generating a correction feedback result that includes a motion compliance score, deviation location annotation, and personalized correction suggestions.

[0061] The standard pose skeletal line rendering method is as follows: Based on the skeletal keypoint coordinates output from the skeletal keypoint extraction step, skeletal segments are drawn according to the human skeletal connection relationship. The skeletal segments are rendered smoothly using an anti-aliasing algorithm, with a line width set to 3 pixels to ensure visibility on display devices with different resolutions. A circular marker is drawn at each skeletal keypoint location, with a radius of 5 pixels, to facilitate patients' intuitive identification of the joint positions.

[0062] The color of the skeletal segments is set according to the compliance level of the corresponding joints, using a color gradient to intuitively convey the degree of standardization of each joint's movement. The color setting rules are as follows: when the joint angle deviation is less than the first preset angle threshold, it is displayed in green, with an RGB value of (0, 200, 0), indicating that the joint movement is standard and no adjustment is needed; when the joint angle deviation is greater than or equal to the first preset angle threshold but less than the second preset angle threshold, it is displayed in yellow, with an RGB value of (255, 200, 0), indicating that the joint movement has a slight deviation that needs attention; when the joint angle deviation is greater than or equal to the second preset angle threshold, it is displayed in red, with an RGB value of (255, 50, 50), indicating that the joint movement has a significant deviation that needs immediate correction. Preferably, the first preset angle threshold is set to 10 degrees, and the second preset angle threshold is set to 20 degrees. This threshold setting has been validated by clinical rehabilitation experts and can effectively distinguish between normal movement errors, slight deviations that need attention, and significant deviations that must be corrected.

[0063] In addition to discrete three-color gradation, this embodiment also supports a continuous color gradient mode. In continuous color gradient mode, the color of the skeletal line segments changes continuously from green to yellow to red according to the degree of deviation, which can more accurately reflect the degree of deviation. The continuous color gradient is calculated using the HSV color space interpolation method. When the deviation is 0 degrees, the hue value is 120 degrees, corresponding to green; when the deviation reaches 30 degrees, the hue value is 0 degrees, corresponding to red; and linear interpolation is used in between.

[0064] The rendering method for the angle deviation indicator arrow is as follows. When the angle deviation of a joint exceeds a preset deviation threshold, an indicator arrow is rendered at that joint position to visually guide the patient in adjusting their posture. The preset deviation threshold is preferably set to 15 degrees. This threshold balances the sensitivity and practicality of the feedback, avoiding excessive interference with minor deviations. The starting point of the indicator arrow is the current joint position, and the direction of the arrow points in the direction that needs adjustment, determined by the sign of the deviation vector. For the shoulder abduction angle, a positive deviation indicates insufficient abduction requiring further arm elevation, while a negative deviation indicates excessive abduction requiring appropriate arm lowering.

[0065] The length of the indicator arrow is positively correlated with the deviation range, and the calculation formula is: , in, The length of the arrow. Based on length, This is the length scaling factor. This represents the absolute value of the angular deviation. Preferably, the base length is set to 30 pixels, and the length scaling factor is set to 2 pixels per degree. When the deviation is 15 degrees, the arrow length is 60 pixels, and when the deviation is 30 degrees, the arrow length is 90 pixels. This visual change in length helps patients intuitively perceive the severity of the deviation.

[0066] The indicator arrows are drawn using a combination of a triangular arrowhead and a rectangular arrow shaft. The width of the arrowhead is set to one-third of the arrow's length, and the width of the arrow shaft is set to 3 pixels. The indicator arrows use the same color gradient rule as the skeletal lines to maintain visual consistency. The specific deviation angle value is indicated next to the indicator arrow using white text with a black outline to ensure readability under different backgrounds. The font size is set to 12 pixels to facilitate patients' quantification of the degree of deviation.

[0067] To avoid visual overlap and interference between multiple indicator arrows, this embodiment employs an arrow layout optimization algorithm. When the indicator arrows of two adjacent joints are in similar directions, the starting point of the arrows is slightly offset to avoid overlap; when three or more joints have deviations simultaneously, the indicator arrows of the three joints with the largest deviations are displayed first, and the degree of deviation of the remaining joints is indicated only by the color of the bone line segments.

[0068] The method for generating personalized correction suggestions is as follows. First, all joints are sorted from largest to smallest deviation, and the three joints with the largest deviations are selected as the priority areas for correction. This design is based on the cognitive load theory; presenting too many correction suggestions simultaneously will increase the patient's cognitive burden and reduce adherence to rehabilitation training. Then, corresponding correction guidance statements are retrieved from a pre-set suggestion template library based on the direction and magnitude of deviation for each joint. The pre-set suggestion template library stores professional correction suggestions for each type of deviation for each joint, pre-written by rehabilitation experts, covering 12 basic situations of positive and negative deviations at 6 joint angles. Each situation corresponds to 3 to 5 different suggestion statements to avoid duplication. Finally, the retrieved correction guidance statements are combined according to priority to generate personalized correction suggestions. The priority is determined comprehensively based on the degree of deviation and the importance of the joint.

[0069] In this embodiment, the suggested template library includes the following examples of corrective suggestions. For positive deviations in shoulder abduction angle (insufficient abduction), corrective suggestions include: raise your arm further to the side, keeping your elbow straight; try to imagine your arm opening outwards, slowly increasing the abduction angle; avoid shrugging your shoulders, keeping your scapula depressed while opening your arm outwards. For negative deviations in shoulder abduction angle (excessive abduction), corrective suggestions include: if your arm is raised too high, lower it appropriately to the target angle; if the abduction angle exceeds the target range, control it within a suitable range. For trunk tilt compensation, corrective suggestions include: keep your trunk upright, avoiding compensating for insufficient shoulder joint mobility by tilting your body to the side; keep your core muscles engaged and your center of gravity centered. For trunk forward tilt compensation, corrective suggestions include: keep your upper body upright, avoiding completing the movement by tilting your trunk forward; imagine a line pulling upwards from the top of your head, maintaining a neutral spine.

[0070] The output of the correction feedback results includes the following parts: First, an overlaid and rendered augmented reality image, containing colored skeletal lines, joint position markers, and deviation indicator arrows, with the same resolution as the input image. Second, a motion compliance score, displayed numerically at the top center of the image, with a font size of 24 pixels, displayed in green, yellow, or red according to the score. Third, deviation location annotations, listing the names of joints with deviations exceeding the threshold and their numerical values, displayed on the right or bottom of the image. Fourth, personalized correction suggestions, displaying specific motion adjustment guidance in text form, with each suggestion separated by a line break to ensure readability.

[0071] This embodiment also supports voice broadcasting, converting personalized corrective suggestions into speech output using text-to-speech technology. This allows patients to receive corrective guidance without having to look at the screen while performing rehabilitation exercises. The voice broadcast uses a moderate speaking speed and clear pronunciation, with a 1-second interval between suggestions to ensure patient understanding and execution.

[0072] In the test environment of this embodiment, the rendering time for augmented reality correction feedback is approximately 12 milliseconds, including operations such as skeletal line drawing, arrow drawing, text rendering, and image compositing. The end-to-end processing latency from image acquisition in the rehabilitation training image acquisition step to the output of correction feedback results in the augmented reality correction feedback step is approximately 95 milliseconds. The design target of less than 100 milliseconds can support real-time interactive correction guidance, allowing patients to receive immediate feedback and make adjustments during the execution of actions.

[0073] The shoulder joint rehabilitation movement image recognition and real-time posture correction feedback method provided in this invention achieved the following technical results in a clinical trial involving 200 patients with rotator cuff injuries: The accuracy rate for movement type recognition reached 94.2%, and the accuracy rate for incorrect posture recognition reached 92.7%. The Cohen's Kappa coefficient for posture deviation assessment and manual assessment by rehabilitation experts reached 0.85. In the patient group receiving home rehabilitation guidance using this method, the rate of standardized rehabilitation movement execution was 35.8% higher than that of the control group using only video instruction. End-to-end processing latency was controlled within 100 milliseconds, and user satisfaction rating reached 4.3 out of 5.

[0074] This invention also provides a shoulder joint rehabilitation movement image recognition and real-time posture correction feedback system, such as... Fig. 2 As shown, the system includes an image acquisition module, a skeletal key point extraction module, a rehabilitation movement recognition module, a posture deviation assessment module, and an augmented reality correction feedback module. Each module corresponds one-to-one with the steps in the method embodiment, and each module is described below.

[0075] The image acquisition module is used to acquire image frame sequences in real time during shoulder joint rehabilitation training using an RGB camera at a preset frame rate, and preprocesses the image frame sequences to obtain standardized image data. The specific implementation of the image acquisition module is the same as described in the rehabilitation training image acquisition steps in the method embodiment, including preprocessing operations such as image scaling, color space conversion, illumination compensation, and temporal buffering. The image acquisition module sends the standardized image data to the skeletal key point extraction module.

[0076] The skeletal key point extraction module is used to input standardized image data into the human body analysis model, extract the coordinates of skeletal key points of the patient's upper limbs and trunk, and calculate the joint angle sequence based on the skeletal key point coordinates. The specific implementation of the skeletal key point extraction module is the same as the skeletal key point extraction steps described in the method embodiment, including the detection of 17 skeletal key points, Kalman filter compensation, and the calculation of 6 joint angles: shoulder abduction angle, shoulder flexion angle, shoulder internal and external rotation angle, elbow flexion and extension angle, trunk lateral tilt angle, and trunk forward tilt angle. The skeletal key point extraction module sends the skeletal key point coordinates and joint angle sequence to the rehabilitation movement recognition module and the posture deviation assessment module.

[0077] The rehabilitation movement recognition module constructs a spatiotemporal graph structure from the coordinates of skeletal key points. This spatiotemporal graph structure is then input into a spatiotemporal graph convolutional network for feature extraction and movement classification. The module outputs the movement category label and corresponding classification confidence score for the current rehabilitation movement. The specific implementation of the rehabilitation movement recognition module is the same as the steps described in the method embodiment, including the construction of the spatiotemporal graph structure and the inference computation of the spatiotemporal graph convolutional network. The rehabilitation movement recognition module sends the movement category label to the posture deviation assessment module and the classification confidence score to the augmented reality correction feedback module.

[0078] The posture deviation assessment module retrieves the corresponding standard joint angle sequence from a pre-stored standard motion template library based on the motion category label. It then uses a dynamic time warping algorithm to align the joint angle sequence with the standard joint angle sequence and calculates the deviation vector and motion compliance score. The specific implementation of the posture deviation assessment module is the same as described in the method embodiment, including an improved dynamic time warping algorithm and multi-dimensional compliance score calculation. The posture deviation assessment module sends the deviation vector sequence and motion compliance score to the augmented reality correction feedback module.

[0079] The augmented reality (AR) correction feedback module overlays and renders standard posture skeletal lines and angle deviation indicator arrows onto standardized image data. It sets the display color of corresponding skeletal segments based on the compliance level of each joint, generating a correction feedback result that includes a movement compliance score, deviation location annotations, and personalized correction suggestions. The specific implementation of the AR correction feedback module is the same as the AR correction feedback steps described in the method embodiment, including colored skeletal line rendering, deviation indicator arrow rendering, and personalized correction suggestion generation. The AR correction feedback module outputs the correction feedback result to a display device for the patient to view.

[0080] In this embodiment, the system hardware configuration includes: a computing device equipped with an Intel Core i5 processor and an NVIDIA GTX1650 graphics card; an RGB camera using a Logitech C920 HD camera; and a display device using a monitor or tablet with a resolution of 1920 pixels by 1080 pixels. The system software configuration includes: an operating system using Windows 10 or Ubuntu 20.04; a deep learning framework using PyTorch 1.12; and an image processing library using OpenCV 4.5.

[0081] The system provided in this invention can be deployed on computers or smart devices in a typical home environment. It can achieve real-time recognition and correction feedback of shoulder joint rehabilitation movements without the need for professional sensing equipment, effectively solving the problem of lack of professional guidance for home rehabilitation of patients with rotator cuff injuries.

[0082] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A shoulder joint rehabilitation movement image recognition and real-time posture correction feedback method, characterized in that, The method comprises the following steps: A rehabilitation training image acquisition step, which acquires image frame sequences in real time during shoulder joint rehabilitation training through an RGB camera at a preset frame rate, and pre-processes the image frame sequences to obtain standardized image data; A skeleton key point extraction step, which inputs the standardized image data into a human body analysis model, extracts the skeleton key point coordinates of the patient's upper limbs and torso, calculates the shoulder joint abduction angle, shoulder joint forward flexion angle, shoulder joint internal and external rotation angle, elbow joint flexion and extension angle, torso lateral tilt angle and torso forward tilt angle based on the skeleton key point coordinates, and generates a joint angle sequence; A rehabilitation action recognition step, which constructs the skeleton key point coordinates into a spatio-temporal graph structure, with skeleton key points as graph nodes, skeleton connections between adjacent key points as spatial edges, and time connections between the same key points in adjacent frames as time edges, inputs the spatio-temporal graph structure into a spatio-temporal graph convolution network for feature extraction and action classification, and outputs the action category label and corresponding classification confidence of the current rehabilitation action; A posture deviation evaluation step, which retrieves the corresponding standard joint angle sequence from a pre-stored standard action template library according to the action category label, aligns the joint angle sequence with the standard joint angle sequence frame by frame using a dynamic time warping algorithm, calculates the deviation vector of each joint angle, and calculates the action compliance score based on the deviation vector; An augmented reality correction feedback step, which superimposes and renders a standard posture skeleton line on the standardized image data, renders an angle deviation indication arrow at the joint position where the deviation exceeds a preset deviation threshold according to the deviation vector, the direction of the angle deviation indication arrow indicates the direction that needs to be adjusted, the length of the angle deviation indication arrow is positively correlated with the deviation amplitude, and the display color of the corresponding skeleton line segment is set according to the compliance degree of each joint, thereby generating a correction feedback result containing the action compliance score, deviation part label and personalized correction suggestion.

2. The method of claim 1, wherein, In the skeleton key point extraction step, the human body analysis model outputs the detection confidence of each skeleton key point, and when the detection confidence is lower than a preset confidence threshold, a Kalman filter is used to predict the skeleton key point coordinates of the current frame based on the skeleton key point coordinates of the previous frame.

3. The method of claim 1, wherein, In the rehabilitation action recognition step, the spatio-temporal graph convolution network comprises a spatial graph convolution layer, a temporal convolution layer and a global average pooling layer connected in sequence, the spatial graph convolution layer adopts a separable convolution structure and dynamically adjusts the weights of different skeleton connection edges through an attention mechanism.

4. The method of claim 1, wherein, In the posture deviation evaluation step, the method for calculating the action compliance score comprises: calculating the weighted root mean square value of each joint angle deviation as an angle deviation component, calculating the dynamic time warping distance between the actual motion trajectory and the standard motion trajectory as a trajectory deviation component, detecting whether the torso lateral tilt angle and the torso forward tilt angle exceed a preset compensation threshold to calculate a compensation penalty component, and performing weighted summation on the angle deviation component, the trajectory deviation component and the compensation penalty component to obtain the action compliance score.

5. The method of claim 1, wherein, In the augmented reality deviation correction feedback step, the rule for setting the display color of the bone line segment according to the deviation degree is: when the joint angle deviation is less than a first preset angle threshold, it is displayed as green; when the joint angle deviation is greater than or equal to the first preset angle threshold and less than a second preset angle threshold, it is displayed as yellow; and when the joint angle deviation is greater than or equal to the second preset angle threshold, it is displayed as red.

6. The method of claim 1, wherein, The calculation method of the shoulder joint abduction angle is: constructing a plane based on the shoulder joint key point, the elbow joint key point and the trunk central axis, and calculating the included angle between the upper arm vector and the trunk central axis; and the calculation method of the shoulder joint internal and external rotation angle is: constructing a vector based on the shoulder joint key point, the elbow joint key point and the wrist joint key point, and calculating the projection angle of the forearm vector in the plane perpendicular to the upper arm.

7. The method of claim 1, wherein, In the dynamic time warping algorithm, a higher weight coefficient is set for the shoulder joint angle than for the elbow joint angle when calculating the alignment path, and a penalty cost is added to the frame with compensatory action.

8. The method of claim 1, wherein, The generation method of the personalized correction suggestion is: sorting the first three joints with the largest deviation according to the deviation degree, retrieving the corresponding correction guidance sentence from the preset suggestion template library according to the deviation direction and deviation amplitude of each joint, and combining the correction guidance sentences to generate the personalized correction suggestion according to the priority.

9. The method of claim 1, wherein, The preset frame rate is 25 frames per second to 30 frames per second, and the end-to-end processing delay from the image acquisition step to the output of the deviation correction feedback result in the augmented reality deviation correction feedback step is less than 100 milliseconds.

10. A shoulder joint rehabilitation motion image recognition and real-time posture correction feedback system for implementing the method of any one of claims 1-9, characterized in that, It comprises: An image acquisition module for acquiring image frame sequences in a shoulder joint rehabilitation training process in real time at a preset frame rate through an RGB camera, and pre-processing the image frame sequences to obtain standardized image data; A skeleton key point extraction module for inputting the standardized image data into a human body analysis model, extracting the skeleton key point coordinates of the patient's upper limbs and trunk, and calculating a joint angle sequence based on the skeleton key point coordinates; A rehabilitation action recognition module for constructing the skeleton key point coordinates into a spatio-temporal graph structure, inputting the spatio-temporal graph structure into a spatio-temporal graph convolution network for feature extraction and action classification, and outputting an action category label and a corresponding classification confidence of the current rehabilitation action; A posture deviation evaluation module for retrieving a corresponding standard joint angle sequence from a pre-stored standard action template library according to the action category label, aligning the joint angle sequence with the standard joint angle sequence using a dynamic time warping algorithm, and calculating a deviation vector and an action compliance score; An augmented reality deviation correction feedback module for superimposing and rendering a standard posture bone line and an angle deviation indication arrow on the standardized image data, setting the display color of the corresponding bone line segment according to the compliance degree of each joint, and generating a deviation correction feedback result containing the action compliance score, deviation part label and personalized correction suggestion.

Images