Knee joint rehabilitation evaluation method based on human skeleton by using space-time diagram convolutional network

Abstract

The invention relates to the technical field of knee joint rehabilitation evaluation, in particular to a knee joint rehabilitation evaluation method based on a human skeleton by using a space-time diagram convolutional network, and the method comprises the following steps: obtaining a rehabilitation evaluation gait video set; extracting human body key point skeleton data, and slicing the human body key point skeleton data into gait samples according to the number of time frames; preprocessing the gait sample, and extracting joint, skeleton and joint angle features; a gait-link method is adopted to divide a space gait skeleton diagram, a space-time attention mechanism is added, and an improved CTR-GCN network model is constructed; training the improved CTR-GCN network model by using the gait sample to obtain a gait evaluation model; and processing a gait video to be evaluated, and inputting the processed gait video into the gait evaluation model to obtain a rehabilitation evaluation result of the patient. According to the method, the human body key point skeleton in the walking video of the patient is extracted and input into the space-time diagram convolutional network for training, the rehabilitation evaluation model is obtained, the rehabilitation condition of the patient is evaluated by using the model, and the requirement of rehabilitation evaluation is met.

Description

Technical Field

[0001] The present application relates to the technical field of knee joint rehabilitation assessment for patients with peri-knee fractures, and in particular to a knee joint rehabilitation assessment method based on the human skeleton using a spatiotemporal graph convolutional network. Background Art

[0002] Rehabilitation assessment runs through the entire rehabilitation treatment process. Doctors should be able to remotely intervene in rehabilitation management and improve rehabilitation plans at any time based on the results of rehabilitation assessment. In terms of rehabilitation assessment methods after fracture surgery, the current clinical practice mainly conducts assessments at three levels: body structure, mobility, and participation. Specifically, these include posture assessment, joint range of motion measurement, muscle strength measurement, pain assessment, sensory assessment, neuromuscular electrophysiology, balance, coordination, gait, ADL, occupational ability assessment, and some special orthopedic physical examinations. However, these rehabilitation assessment methods either rely on the doctor's subjective experience and lack objective quantitative indicators, or require expensive and bulky equipment and professional operator assessment personnel. They cannot be used for objective quantitative assessment of limb motor function in a remote home environment, and it is difficult to meet the demands of patients and medical staff for remote rehabilitation medical care.

[0003] Functional impairment caused by knee fractures can lead to motor dysfunction. By observing the patient's gait characteristics, we can assess the patient's recovery progress. There are two common methods for collecting gait information: wearable device-based gait analysis and vision-based gait analysis. Wearable device-based gait analysis requires sensors, such as accelerometers, to be installed on various parts of the body. Vision-based gait analysis, on the other hand, uses imaging sensors to capture gait data without the need for wearable devices, reducing the physical burden on the patient during the collection process, improving patient comfort, and facilitating home diagnosis. To implement a reliable vision-based gait assessment system, we require a sufficiently diverse dataset and an efficient human keypoint object detection algorithm. We also utilize human posture recognition information to extract key kinematic features. In the absence of a unified evaluation standard, we leverage machine learning techniques to establish an effective, objective, and clinically recognized quantitative assessment model. Therefore, a postoperative objective quantitative assessment method for limb motor function in fracture patients is urgently needed that can be performed remotely in the home or community hospital setting without the need for direct physician involvement. Summary of the Invention

[0004] An embodiment of the present application provides a knee joint rehabilitation assessment method based on the human skeleton using a spatiotemporal graph convolutional network. By extracting the skeleton of key human points in a patient's walking video and inputting it into the spatiotemporal graph convolutional network for training, a rehabilitation assessment model is obtained. This model is used to evaluate the patient's rehabilitation status, meeting the needs of rehabilitation assessment.

[0005] In order to solve the above technical problems, an embodiment of the present application provides a knee joint rehabilitation assessment method based on the human skeleton using a spatiotemporal graph convolutional network, comprising the following steps: step S1, obtaining a rehabilitation assessment gait video set; the rehabilitation assessment gait video set includes a rehabilitation assessment gait video and a corresponding rehabilitation assessment scale; step S2, extracting human key point skeleton data from the rehabilitation assessment gait video set, and slicing the human key point skeleton data into gait samples according to the number of time frames; step S3, preprocessing the gait samples, and extracting joint, bone and joint angle features; step S4, using the gait-link method to divide the spatial gait skeleton graph, and adding a spatiotemporal attention mechanism to the CTR-GCN network model to construct an improved CTR-GCN network model; step S5, using the gait samples to train the improved CTR-GCN network model to obtain a gait assessment model; step S6, inputting the gait video to be evaluated into the gait assessment model after processing from steps S2 to S3 to obtain the patient's rehabilitation assessment result.

[0006] In some exemplary embodiments, obtaining a rehabilitation assessment gait video set includes: filming the patient from the front and side while the patient walks forward naturally to obtain a rehabilitation assessment gait video; performing a clinical rehabilitation assessment on the patient whose gait video was filmed to obtain a rehabilitation assessment score and a rehabilitation assessment scale corresponding to the rehabilitation assessment gait video; wherein the items of the clinical rehabilitation assessment include: Lysholm knee joint function score, pain evaluation scoring standard and muscle strength assessment examination standard.

[0007] In some exemplary embodiments, the scoring criteria are weighted averaged according to the following formula to obtain the patient's rehabilitation assessment score;

[0008] S=0.7×S L +0.2×S P +0.1×S M

[0009] Among them, S L ,S P ,S M represent the normalized scores of the Lysholm knee function score, pain evaluation score standard, and muscle strength assessment examination standard, respectively.

[0010] In some exemplary embodiments, in step S2, mediapipe is used to extract human key point skeleton data from the rehabilitation assessment gait video set, and the human key point skeleton data is sliced ​​into gait samples according to the number of time frames, including: inputting the rehabilitation assessment gait video frame by frame into the posture recognition model of mediapipe, outputting the human key point results of each frame, and selecting world_pose_landmarks from the human key point results as the coordinates for human key point extraction; splicing the human joint point coordinates frame by frame to obtain a human joint point coordinate sequence consistent with the length of the rehabilitation assessment gait video; calculating the slicing step length according to the length of the human key point coordinate sequence, and slicing each human key point coordinate sequence into 20 gait samples with a length of 120 frames.

[0011] In some exemplary embodiments, world_pose_landmarks is selected from the human key point results as the coordinates for extracting human key points. Each coordinate point contains four pieces of information, namely, x-coordinate, y-coordinate, z-coordinate, and confidence level. The coordinates of the human joint points frame by frame are spliced ​​in chronological order to obtain a human joint point coordinate sequence consistent with the length of the rehabilitation assessment gait video. The structure of the coordinate sequence is a three-dimensional matrix of [T, N, C], where T represents the time length, N represents the number of key points, and C represents the number of channels. The formula for calculating the slice step length is:

[0012]

[0013] Where, Step is the step length, T all is the total length of the video, T cut is the length of the slice gait sample, and N is the number of slice samples.

[0014] In some exemplary embodiments, the gait samples are preprocessed and joint, bone and joint angle features are extracted, including: filtering the coordinates of the joint points using a one-dimensional Kalman filter, using a 3 times standard deviation method to remove key points detected incorrectly due to occlusion, and using a pchip interpolation method to fill data gaps; selecting parts in the video that do not change in length due to movement as the human body length benchmark, normalizing the coordinates of each key point to ensure that the size of the human body in the video is stable; simplifying the key points extracted by mediapipe, removing key points that affect the patient's gait assessment, and using the simplified key points as joint point features in the training sample; calculating the joint angles of the human body through the joint feature points, using the coordinate positions of the three joint points in space, calculating the joint angles of the human body related to gait, and obtaining the joint angle features of the human body; calculating the gait connection features of the human body through the joint feature points, and connecting the key points that can reflect the gait features according to the relevant experience knowledge of gait analysis to form gait connection features.

[0015] In some exemplary embodiments, the key points extracted by mediapipe are simplified, key points that affect the patient's gait assessment are removed, and the simplified key points are used as joint point features in the training sample, including: simplifying the key points extracted by mediapipe, selecting key points closely related to gait features from the extracted 33 key points, and using them as joint point features in the training sample, the selected key points being: left shoulder, right shoulder, left hip, right hip, left knee, right knee, left ankle, right ankle, left heel, right heel, left toe, and right toe; the formula for calculating the joint angles of the human body related to gait is:

[0016]

[0017] Among them, θ is the human joint angle feature, and A, B, and C are three joint points.

[0018] In some exemplary embodiments, the gait-link method is used to divide the spatial gait skeleton graph, including: using the gait-link method, dividing the spatial skeleton graph adjacency matrix into three adjacency matrices, respectively denoted as A root , A struc , A gait ; The three adjacency matrices are used as input hyperparameters of the CTR-GC module; Among them, the adjacency matrix A root It only contains the root node of the skeleton graph, which reflects the characteristics of the joint trajectory. Its expression is A root =I; adjacency matrix A struc Contains three joint points connected to the root node by only one bone. These three joint points represent the joint angle characteristics, and their expressions are:

[0019]

[0020] Among them, H bone The set of node pairs connected by only one bone;

[0021] A gait is a skeleton graph containing gait link information, reflecting step length and gait characteristics, and its expression is:

[0022]

[0023] Among them, H gait It is a collection of node pairs connected by gait-link lines.

[0024] In some exemplary embodiments, the improved CTR-GCN network model includes: three CTR-GCN basic blocks with a spatiotemporal attention mechanism, a Flatten layer, and a classification layer. The number of channels of the three CTR-GCN basic blocks with a spatiotemporal attention mechanism are 64, 128, and 256, respectively. In the CTR-GCN basic block with a spatiotemporal attention mechanism, the feature vector passes through a spatial graph convolution module, a temporal convolution module, and a spatiotemporal attention module in sequence and is connected through a residual connection.

[0025] In some exemplary embodiments, in the spatial graph convolution module, the feature vector passes through a network structure composed of three parallel CTR-GC modules, and the outputs of the three modules are respectively normalized and activated to output the output of the spatial graph convolution module; in the temporal convolution module, the feature vector is first compressed by three-branch 1×1 convolution operations for channel dimension, and then subjected to maximum pooling operations and temporal convolution operations respectively, and then connected with the results of the first three branches through a 1×1 convolution branch, and a residual connection is adopted; a spatiotemporal attention mechanism is introduced after the CTR-GCN module, and the temporal attention factor and the spatial attention factor are calculated respectively, and then the temporal attention factor is multiplied by the spatial attention factor and normalized to obtain a spatiotemporal attention map, and the attention map is multiplied by the feature vector and the feature vector output is obtained through a residual connection; the abnormal gait assessment problem is regarded as an ordinal multi-classification problem, and Ordinal Cross-Entropy is adopted as the loss function of the model.

[0026] The technical solution provided by the embodiments of the present application has at least the following advantages:

[0027] An embodiment of the present application provides a knee joint rehabilitation assessment method based on the human skeleton using a spatiotemporal graph convolutional network, comprising the following steps: step S1, obtaining a rehabilitation assessment gait video set; the rehabilitation assessment gait video set includes a rehabilitation assessment gait video and a corresponding rehabilitation assessment scale; step S2, extracting human key point skeleton data from the rehabilitation assessment gait video set, and slicing the human key point skeleton data into gait samples according to the number of time frames; step S3, preprocessing the gait samples, and extracting joint, bone and joint angle features; step S4, using the gait-link method to divide the spatial gait skeleton graph, and adding a spatiotemporal attention mechanism to the CTR-GCN network model to construct an improved CTR-GCN network model; step S5, using the gait samples to train the improved CTR-GCN network model to obtain a gait assessment model; step S6, inputting the gait video to be evaluated into the gait assessment model after processing in steps S2 to S3 to obtain the patient's rehabilitation assessment result.

[0028] The embodiments of this application provide a human skeleton-based knee joint rehabilitation assessment method using a spatiotemporal graph convolutional network. On the one hand, this application uses an ordinary camera to record gait, eliminating the need for expensive specialized equipment or complex operating procedures, reducing costs while also increasing accessibility. Medical staff can implement this assessment method in a variety of settings, including clinics, wards, and even patients' homes, making rehabilitation assessments more flexible and convenient, and facilitating long-term tracking and observation of patients' recovery processes.

[0029] Furthermore, by continuously recording and analyzing gait scores, this application enables doctors to develop personalized rehabilitation plans for each patient and flexibly adjust them based on recovery progress. This precise, data-driven rehabilitation management helps improve treatment outcomes, ensuring that each patient receives the most appropriate care and enabling personalized rehabilitation monitoring.

[0030] Furthermore, this application can leverage internet technology to upload gait videos recorded by ordinary cameras to the cloud for analysis and processing. This means that even in remote locations or with limited mobility, patients can receive professional rehabilitation assessment services. This is of great significance for expanding the coverage of medical services and improving the uneven distribution of medical resources. Furthermore, this also opens up the possibility of contactless medical services during epidemics or other special circumstances. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] One or more embodiments are exemplarily described by the pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Unless otherwise stated, the pictures in the drawings do not constitute proportional limitations.

[0032] Figure 1 A flowchart of a knee joint rehabilitation assessment method based on the human skeleton using a spatiotemporal graph convolutional network is provided in an embodiment of the present application.

[0033] Figures 2A to 2D This is a schematic representation of the rehabilitation clinical quantitative assessment provided in the embodiments of this application.

[0034] Figure 3 This is the human body posture key point diagram of the mediapipe provided in the embodiment of this application.

[0035] Figure 4 This is a structural diagram of the improved CTR-GCN network model provided in an embodiment of the present application.

[0036] Figure 5 A method for grouping spatial gait graphs provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] As can be seen from the background technology, fracture patients are currently in urgent need of an objective quantitative assessment method for limb motor function after surgery that does not require direct involvement of a doctor and can be performed remotely in a home or community hospital environment.

[0038] In order to solve the above technical problems, an embodiment of the present application provides a knee joint rehabilitation assessment method based on the human skeleton using a spatiotemporal graph convolutional network, comprising the following steps: step S1, obtaining a rehabilitation assessment gait video set; the rehabilitation assessment gait video set includes a rehabilitation assessment gait video and a corresponding rehabilitation assessment scale; step S2, extracting human key point skeleton data from the rehabilitation assessment gait video set, and slicing the human key point skeleton data into gait samples according to the number of time frames; step S3, preprocessing the gait samples, and extracting joint, bone and joint angle features; step S4, using the gait-link method to divide the spatial gait skeleton graph, and adding a spatiotemporal attention mechanism to the CTR-GCN network model to construct an improved CTR-GCN network model; step S5, using the gait samples to train the improved CTR-GCN network model to obtain a gait assessment model; step S6, inputting the gait video to be evaluated into the gait assessment model after processing from steps S2 to S3 to obtain the patient's rehabilitation assessment result. An embodiment of the present application provides a knee joint rehabilitation assessment method based on the human skeleton using a spatiotemporal graph convolutional network. By extracting the skeleton of key human points in a patient's walking video and inputting it into the spatiotemporal graph convolutional network for training, a rehabilitation assessment model is obtained. This model is used to evaluate the patient's rehabilitation status, meeting the needs of rehabilitation assessment.

[0039] The following detailed description of the various embodiments of the present application is provided in conjunction with the accompanying drawings. However, those skilled in the art will appreciate that many technical details are provided in the various embodiments of the present application to facilitate a better understanding of the present application. However, even without these technical details and the various variations and modifications based on the following embodiments, the technical solutions claimed in the present application can still be implemented.

[0040] See Figure 1 The present invention provides a method for knee joint rehabilitation assessment based on the human skeleton using a spatiotemporal graph convolutional network, comprising the following steps:

[0041] Step S1: Acquire a rehabilitation assessment gait video set; the rehabilitation assessment gait video set includes a rehabilitation assessment gait video and a corresponding rehabilitation assessment scale.

[0042] Step S2: extracting human key point skeleton data from the rehabilitation assessment gait video set, and slicing the human key point skeleton data into gait samples according to the number of time frames.

[0043] Step S3: pre-process the gait samples and extract joint, bone and joint angle features.

[0044] Step S4: Use the gait-link method to divide the spatial gait skeleton graph, and add the spatiotemporal attention mechanism to the CTR-GCN network model to construct an improved CTR-GCN network model.

[0045] Step S5: Use the gait samples to train the improved CTR-GCN network model to obtain a gait evaluation model.

[0046] Step S6: After the gait video to be evaluated is processed in steps S2 to S3, it is input into the gait evaluation model to obtain the patient rehabilitation evaluation result.

[0047] In some embodiments, obtaining a rehabilitation assessment gait video set in step S1 includes: filming the patient from the front and side while the patient is walking forward naturally to obtain a rehabilitation assessment gait video; performing a clinical rehabilitation assessment on the patient who filmed the gait video to obtain a rehabilitation assessment score and a rehabilitation assessment scale corresponding to the rehabilitation assessment gait video; wherein the items of the clinical rehabilitation assessment include: Lysholm knee joint function score, pain evaluation scoring standard and muscle strength assessment examination standard.

[0048] It should be noted that to ensure video quality, specific filming methods and angles are specified. The patient should walk naturally forward during filming, and the photographer should film from both the front and side of the patient. The distance between the patient and the photographer should be kept relatively still, ensuring that the patient's entire limbs are within the camera's field of view. The camera should be kept as stable as possible, and the walking distance should be approximately 5 to 6 meters.

[0049] Patients who took gait videos were also evaluated for clinical rehabilitation. The evaluation items were based on Figures 2A to 2D , including: Lysholm knee function score, pain evaluation scoring standard and muscle strength assessment examination standard.

[0050] In some embodiments, the scoring criteria are weighted averaged according to the following formula to obtain the patient's rehabilitation assessment score;

[0051] S=0.7×S L +0.2×S P +0.1×S M

[0052] Among them, S L ,S P ,S M represent the normalized scores of the Lysholm knee function score, pain evaluation score standard, and muscle strength assessment examination standard, respectively.

[0053] In some embodiments, step S2 uses mediapipe to extract human key point skeleton data from the rehabilitation assessment gait video set, and slices the human key point skeleton data into gait samples according to the number of time frames, including: inputting the rehabilitation assessment gait video frame by frame into the posture recognition model of mediapipe, selecting the model Pose Landmarker (Full) for training, and outputting the human key point results for each frame, wherein each key point represents a human body part such as Figure 3 As shown in the figure, world_pose_landmarks is selected from the human key point results as the coordinates for human key point extraction; the human joint point coordinates are spliced ​​frame by frame to obtain a human joint point coordinate sequence that is consistent with the length of the rehabilitation assessment gait video; based on the length of the human key point coordinate sequence, the slicing step length is calculated, and each human key point coordinate sequence is sliced ​​into 20 gait samples with a length of 120 frames.

[0054] In some exemplary embodiments, world_pose_landmarks is selected from the human key point results as the coordinates for extracting human key points. Each coordinate point contains four pieces of information, namely, x-coordinate, y-coordinate, z-coordinate, and confidence level. The coordinates of the human joint points frame by frame are spliced ​​in chronological order to obtain a human joint point coordinate sequence consistent with the length of the rehabilitation assessment gait video. The structure of the coordinate sequence is a three-dimensional matrix of [T, N, C], where T represents the time length, N represents the number of key points, and C represents the number of channels. The formula for calculating the slice step length is:

[0055]

[0056] Where, Step is the step length, T all is the total length of the video, T cut is the length of the slice gait sample, and N is the number of slice samples.

[0057] In some exemplary embodiments, the gait samples are preprocessed and joint, bone and joint angle features are extracted, including: filtering the coordinates of the joint points using a one-dimensional Kalman filter, using a 3 times standard deviation method to remove key points detected incorrectly due to occlusion, and using a pchip interpolation method to fill data gaps; selecting parts in the video that do not change in length due to movement as the human body length benchmark, normalizing the coordinates of each key point to ensure that the size of the human body in the video is stable; simplifying the key points extracted by mediapipe, removing key points that affect the patient's gait assessment, and using the simplified key points as joint point features in the training sample; calculating the joint angles of the human body through the joint feature points, using the coordinate positions of the three joint points in space, calculating the joint angles of the human body related to gait, and obtaining the joint angle features of the human body; calculating the gait connection features of the human body through the joint feature points, and connecting the key points that can reflect the gait features according to the relevant experience knowledge of gait analysis to form gait connection features.

[0058] In some exemplary embodiments, the key points extracted by mediapipe are simplified, key points that affect the patient's gait assessment are removed, and the simplified key points are used as joint point features in the training sample, including: simplifying the key points extracted by mediapipe, selecting key points closely related to gait features from the extracted 33 key points, and using them as joint point features in the training sample, the selected key points being: left shoulder, right shoulder, left hip, right hip, left knee, right knee, left ankle, right ankle, left heel, right heel, left toe, and right toe; the formula for calculating the joint angles of the human body related to gait is:

[0059]

[0060] Among them, θ is the human joint angle feature, and A, B, and C are three joint points.

[0061] like Figure 5 As shown, the skeleton graph is divided into three parts according to the gait-link method, namely, the joint point graph, the skeleton connection graph and the gait connection graph. In some embodiments, the gait-link method is used to divide the spatial gait skeleton graph, including: using the gait-link method, dividing the spatial skeleton graph adjacency matrix into three adjacency matrices, respectively denoted as A root , A struc , A gait ; The three adjacency matrices are used as input hyperparameters of the CTR-GC module; Among them, the adjacency matrix A root It only contains the root node of the skeleton graph, which reflects the characteristics of the joint trajectory. Its expression is A root =I; adjacency matrix A strucContains three joint points connected to the root node by only one bone. These three joint points represent the joint angle characteristics, and their expressions are:

[0062]

[0063] Among them, H bone The set of node pairs connected by only one bone;

[0064] A gait is a skeleton graph containing gait link information, reflecting step length and gait characteristics, and its expression is:

[0065]

[0066] Among them, H gait It is a collection of node pairs connected by gait-link lines.

[0067] like Figure 4 As shown, in some embodiments, the improved CTR-GCN network model includes: three ACTR-GCN basic blocks (ACTR-GCN Block) with added spatiotemporal attention mechanism, flattening layer (Flatten) and classification layer (ClassificationLayers), the number of channels of the three ACTR-GCN basic blocks with added spatiotemporal attention mechanism are 64, 128 and 256 respectively; in the ACTR-GCN basic block with added spatiotemporal attention mechanism, the feature vector passes through the spatial graph convolution module (CTR-GC), the temporal convolution module (Temporal Modeling) and the spatiotemporal attention module (Spatio-temporal Attention) and is connected through the residual connection (Residual). The classification layer includes: batch normalization layer and dropout layer (BN+Dropout), fully connected layer (FClayer) and normalized index layer (Softmax).

[0068] In some embodiments, in the spatial graph convolution module, the feature vector passes through a network structure composed of three parallel CTR-GC modules, and the outputs of the three modules are respectively normalized and activated to output the output of the spatial graph convolution module; in the temporal convolution module, the feature vector is first compressed by three-branch 1×1 convolution operations for channel dimension, and then subjected to maximum pooling operations and temporal convolution operations respectively, and then connected with the results of the first three branches through a 1×1 convolution branch, and a residual connection is adopted; a spatiotemporal attention mechanism is introduced after the CTR-GCN module, and the temporal attention factor and the spatial attention factor are calculated respectively, and then the temporal attention factor is multiplied by the spatial attention factor and normalized to obtain a spatiotemporal attention map, and the attention map is multiplied by the feature vector and the feature vector output is obtained through a residual connection; finally, the abnormal gait assessment problem is regarded as an ordinal multi-classification problem, and Ordinal Cross-Entropy is adopted as the loss function of the model.

[0069] The spatiotemporal attention mechanism is introduced after the CTR-GCN module to calculate the temporal attention factor and the spatial attention factor respectively. The spatiotemporal attention factor can be obtained through the fully connected layer, and its expression is:

[0070]

[0071] Among them, φ (l) is the temporal attention factor vector, θ (l) is the spatial attention factor vector, are learnable parameters.

[0072] The gait features are weighted using the above-mentioned spatiotemporal attention factors to obtain new gait features as follows:

[0073]

[0074] Among them, φ (l) θ (l)T It can be defined as a spatiotemporal attention map, which is normalized as:

[0075] Att (l) =softmax(φ (l) θ (l) ),

[0076] The final attention-enhanced gait feature map is shown below:

[0077]

[0078] After constructing the improved CTR-GCN network model, the improved CTR-GCN network model is trained using gait samples to obtain a gait evaluation model; the gait video to be evaluated is processed according to steps S2 to S3 and input into the gait evaluation model to obtain the patient's rehabilitation evaluation results.

[0079] An embodiment of the present application provides a knee joint rehabilitation assessment method based on the human skeleton using a spatiotemporal graph convolutional network, comprising the following steps: step S1, obtaining a rehabilitation assessment gait video set; the rehabilitation assessment gait video set includes a rehabilitation assessment gait video and a corresponding rehabilitation assessment scale; step S2, extracting human key point skeleton data from the rehabilitation assessment gait video set, and slicing the human key point skeleton data into gait samples according to the number of time frames; step S3, preprocessing the gait samples, and extracting joint, bone and joint angle features; step S4, using the gait-link method to divide the spatial gait skeleton graph, and adding a spatiotemporal attention mechanism to the CTR-GCN network model to construct an improved CTR-GCN network model; step S5, using the gait samples to train the improved CTR-GCN network model to obtain a gait assessment model; step S6, inputting the gait video to be evaluated into the gait assessment model after processing in steps S2 to S3 to obtain the patient's rehabilitation assessment result.

[0080] Based on the above technical solutions, the embodiment of the present application provides a knee joint rehabilitation assessment method based on the human skeleton using a spatiotemporal graph convolutional network. On the one hand, this application uses an ordinary camera to record gait, without the need for expensive professional equipment or complicated operating procedures, which reduces costs while also improving the accessibility of detection. Medical staff can implement this assessment method in a variety of environments, including clinics, wards, and even patients' homes, making rehabilitation assessment more flexible and convenient, and conducive to long-term tracking and observation of patients' recovery process.

[0081] Furthermore, by continuously recording and analyzing gait scores, this application enables doctors to develop personalized rehabilitation plans for each patient and flexibly adjust them based on recovery progress. This precise, data-driven rehabilitation management helps improve treatment outcomes, ensuring that each patient receives the most appropriate care and enabling personalized rehabilitation monitoring.

[0082] Furthermore, this application can leverage internet technology to upload gait videos recorded by ordinary cameras to the cloud for analysis and processing. This means that even in remote locations or with limited mobility, patients can receive professional rehabilitation assessment services. This is of great significance for expanding the coverage of medical services and improving the uneven distribution of medical resources. Furthermore, this also opens up the possibility of contactless medical services during epidemics or other special circumstances.

[0083] Those skilled in the art will appreciate that the above-described embodiments are specific examples for implementing the present application, and that in actual applications, various changes in form and detail may be made thereto without departing from the spirit and scope of the present application. Any person skilled in the art may make changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be subject to the scope defined in the claims.

Claims

1. A knee joint rehabilitation assessment method based on human skeleton using spatiotemporal graph convolutional network, characterized in that: The following steps are involved: Step S1, obtaining a rehabilitation assessment gait video set; the rehabilitation assessment gait video set includes rehabilitation assessment gait videos and corresponding rehabilitation assessment scales; Step S2, extracting human body key point skeleton data from the rehabilitation assessment gait video set, and slicing the human body key point skeleton data into gait samples according to the number of time frames; Step S3, preprocessing the gait sample and extracting joint, bone and joint angle features; Step S4: Use the gait-link method to divide the spatial gait skeleton graph, and add the spatiotemporal attention mechanism to the CTR-GCN network model to construct an improved CTR-GCN network model; Step S5: using the gait samples to train an improved CTR-GCN network model to obtain a gait evaluation model; Step S6: After the gait video to be evaluated is processed in steps S2 to S3, it is input into the gait evaluation model to obtain the patient rehabilitation evaluation result.

2. The knee joint rehabilitation assessment method based on human skeleton using spatiotemporal graph convolutional network according to claim 1 is characterized in that: The method of obtaining a gait video set for rehabilitation assessment includes: When the patient walks forward naturally, the video is taken from the front and side of the patient to obtain a rehabilitation assessment gait video; Patients whose gait videos were recorded were evaluated for clinical rehabilitation, and rehabilitation assessment scores and rehabilitation assessment scales corresponding to the rehabilitation assessment gait videos were obtained. The clinical rehabilitation assessment items included: Lysholm knee function score, pain evaluation scoring standard, and muscle strength assessment examination standard.

3. The knee joint rehabilitation assessment method based on human skeleton using spatiotemporal graph convolutional network according to claim 2 is characterized in that: The scoring criteria are weighted averaged according to the following formula to obtain the patient's rehabilitation assessment score; S=0.7×S L +0.2×S P +0.1×S M Among them, S L ,S P ,S M represent the normalized scores of the Lysholm knee function score, pain evaluation score standard, and muscle strength assessment examination standard, respectively.

4. The knee joint rehabilitation assessment method based on human skeleton using spatiotemporal graph convolutional network according to claim 1, characterized in that: In step S2, mediapipe is used to extract human key point skeleton data from the rehabilitation assessment gait video set, and the human key point skeleton data is sliced ​​into gait samples according to the number of time frames, including: Input the rehabilitation assessment gait video frame by frame into the posture recognition model of mediapipe, output the human body key point results for each frame, and select world_pose_landmarks from the human body key point results as the coordinates for human body key point extraction; The coordinates of the human joints are spliced ​​frame by frame to obtain a human joint coordinate sequence that is consistent with the length of the rehabilitation assessment gait video; According to the length of the human key point coordinate sequence, the slicing step length is calculated, and each human key point coordinate sequence is sliced ​​into 20 gait samples with a length of 120 frames.

5. The knee joint rehabilitation assessment method based on human skeleton using spatiotemporal graph convolutional network according to claim 4 is characterized in that: Select world_pose_landmarks from the human key point results as the coordinates for human key point extraction. Each coordinate point contains four pieces of information: x-coordinate, y-coordinate, z-coordinate, and confidence level. The coordinates of the human joint points frame by frame are spliced ​​in chronological order to obtain a human joint point coordinate sequence that is consistent with the length of the rehabilitation assessment gait video. The structure of the coordinate sequence is a three-dimensional matrix [T, N, C], where T represents the time length, N represents the number of key points, and C represents the number of channels; The formula for calculating the slice step size is: Where, Step is the step length, T all is the total length of the video, T cut is the length of the slice gait sample, and N is the number of slice samples.

6. The knee joint rehabilitation assessment method based on human skeleton using spatiotemporal graph convolutional network according to claim 1, characterized in that: The gait samples are preprocessed and joint, bone and joint angle features are extracted, including: The coordinates of the joint points are filtered using a one-dimensional Kalman filter, the 3-times standard deviation method is used to remove key points that are misdetected due to occlusion, and the pchip interpolation method is used to fill data gaps; The parts of the video that do not change in length due to movement are selected as the human body length benchmark, and the coordinates of each key point are normalized to ensure the size of the human body in the video is stable; Simplify the key points extracted by mediapipe, remove the key points that affect the patient's gait assessment, and use the simplified key points as the joint point features in the training sample; The joint angles of the human body are calculated through the joint feature points. The coordinate positions of the three joint points in space are used to calculate the joint angles of the human body related to gait and obtain the joint angle features of the human body. The gait connection features of the human body are calculated through joint feature points. According to the relevant experience knowledge of gait analysis, the key points that can reflect the gait characteristics are connected to form the gait connection features.

7. The knee joint rehabilitation assessment method based on human skeleton using spatiotemporal graph convolutional network according to claim 6, characterized in that: The key points extracted by mediapipe are simplified to remove the key points that affect the patient's gait assessment. The simplified key points are used as the joint point features in the training sample, including: The key points extracted by mediapipe are simplified, and the key points closely related to gait characteristics are selected from the extracted 33 key points. These key points are used as joint features in the training samples. The selected key points are: left shoulder, right shoulder, left hip, right hip, left knee, right knee, left ankle, right ankle, left heel, right heel, left toe, and right toe. The formula for calculating the joint angles of the human body related to gait is: Among them, θ is the human joint angle feature, and A, B, and C are three joint points.

8. The knee joint rehabilitation assessment method based on human skeleton using spatiotemporal graph convolutional network according to claim 1, characterized in that: The gait-link method is used to divide the spatial gait skeleton graph, including: Using the gait-link method, the spatial skeleton graph adjacency matrix is ​​divided into three adjacency matrices, denoted as A root , A struc , A gait ; The three adjacency matrices are used as input hyperparameters of the CTR-GC module; Among them, Adjacency matrix A root It only contains the root node of the skeleton graph, which reflects the characteristics of the joint trajectory. Its expression is A root =I; Adjacency matrix A struc Contains three joint points connected to the root node by only one bone. These three joint points represent the joint angle characteristics, and their expressions are: Among them, H bone The set of node pairs connected by only one bone; A gait is a skeleton graph containing gait link information, reflecting step length and gait characteristics, and its expression is: Among them, H gait It is a collection of node pairs connected by gait-link lines.

9. The knee joint rehabilitation assessment method based on human skeleton using spatiotemporal graph convolutional network according to claim 1, characterized in that: The improved CTR-GCN network model includes: three CTR-GCN basic blocks with spatiotemporal attention mechanism, a Flatten layer and a classification layer. The number of channels of the three CTR-GCN basic blocks with spatiotemporal attention mechanism are 64, 128 and 256 respectively. In the CTR-GCN basic block with spatiotemporal attention mechanism, the feature vector passes through the spatial graph convolution module, the temporal convolution module and the spatiotemporal attention module successively and is connected through the residual connection.

10. The knee joint rehabilitation assessment method based on human skeleton using spatiotemporal graph convolutional network according to claim 9, characterized in that: In the spatial graph convolution module, the feature vector passes through a network structure composed of three parallel CTR-GC modules, and the outputs of the three modules are normalized and activated respectively to output the output of the spatial graph convolution module; In the temporal convolution module, the feature vector is first compressed by three 1×1 convolution operations, followed by maximum pooling and temporal convolution operations, and then connected to the results of the first three branches through a 1×1 convolution branch, and a residual connection is used; The spatiotemporal attention mechanism is introduced after the CTR-GCN module. The temporal attention factor and spatial attention factor are calculated separately. The temporal attention factor and the spatial attention factor are then multiplied and normalized to obtain the spatiotemporal attention map. The attention map is multiplied by the feature vector and the feature vector output is obtained through residual connection. The abnormal gait assessment problem is regarded as an ordinal multi-classification problem, and Ordinal Cross-Entropy is used as the loss function of the model.

Images