A custom rehabilitation training monitoring and evaluation method based on Lite-HRNet

Through the combination of Lite-HRNet network and Kmeans/KNN algorithm, the environmental impact and low efficiency of pose estimation in rehabilitation training are solved, high-resolution and high-efficiency pose monitoring is achieved, customized needs are met, and quantified The evaluation provides real-time feedback, improving the robustness and adaptability of the system.

CN115661856BActive Publication Date: 2024-11-12SOUTHWEST JIAOTONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211237629.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-10-10
Publication Date
2024-11-12
Estimated Expiration
2042-10-10

AI Technical Summary

Technical Problem

The existing human posture estimation algorithms have problems such as environmental factors, low recognition efficiency, lack of customization of data sets and insufficient standardization in rehabilitation training, making it difficult to meet the needs of high resolution, high efficiency and customization.

Method used

Using a custom rehabilitation training monitoring and evaluation method based on Lite-HRNet, data clustering and KNN algorithm are classified through the Kmeans algorithm, and model training is carried out in combination with public open source data sets and custom data sets to achieve high resolution and high efficiency pose estimation, and action evaluation is performed through angle eigenvalue calculation and limb vector angle cosine value calculation.

Benefits of technology

It realizes fully automated data processing, improves the robustness and adaptability of the model, meets the customized needs of rehabilitation training, ensures high efficiency and high standard posture estimation, reduces manpower burden, and provides real-time through quantitative evaluation feedback.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115661856B_ABST
    Figure CN115661856B_ABST
Patent Text Reader

Abstract

The present invention discloses a custom rehabilitation training monitoring and evaluation method based on Lite-HRNet, specifically: using a camera to shoot pre-trained rehabilitation training action pictures, creating or updating a rehabilitation training action image dataset; constructing a Lite-HRnet network model, respectively using an open source COCO dataset and a custom dataset to train and fine-tune the network model; clustering and labeling through the Kmeans algorithm to construct a personalized dataset; extracting key frames to the Lite-HRnet network for training, performing KNN classification after obtaining key point information, and then completing similarity calculation and threshold detection. The present invention solves the manpower loss caused by complex data and realizes fully automated data processing to a certain extent; the algorithm model used takes into account operating efficiency while maintaining high-resolution performance, can provide strong guarantees for subsequent work, and ensures the reliability and convenience of rehabilitation training standard evaluation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of human posture estimation, and in particular relates to a custom rehabilitation training monitoring and evaluation method based on Lite-HRNet. Background Art

[0002] With the aging of the population, industry, traffic accidents, natural disasters, work pressure, diseases and other reasons, the number of people with rehabilitation needs has increased at an alarming rate year by year. The expansion of rehabilitation demand has led to a shortage of nursing staff and low rehabilitation efficiency in the rehabilitation field. In addition, in traditional medical rehabilitation training, professional and technical personnel are generally required for guidance to avoid physical injuries caused by improper exercise, and a specific environment is required for training. Therefore, there is a lot of room for research to achieve the goal of high-efficiency rehabilitation training anytime and anywhere. The development of machine vision technology has provided new development ideas for the field of rehabilitation. Combining machine vision technology with rehabilitation technology can assist rehabilitation personnel to complete training tasks independently, improve the active training participation and confidence of rehabilitation personnel, and use the Internet to conduct real-time effect evaluation and training adjustments for patients at home, improve rehabilitation efficiency and effectively reduce the burden on medical staff and family members.

[0003] At present, human posture estimation is mainly used in face recognition, intelligent security, unmanned driving and other fields. The research algorithms for multi-person human posture detection are mainly divided into two aspects: (1) Top-down method, in which the top-down method includes two parts: human body detection and single human key point detection. That is, the human body in the image is first detected by the target detection algorithm, and then the key points of each human body are detected based on the detected human body. The advantage of this type of algorithm is that the recognition efficiency is generally high, but because the algorithm needs to detect each individual in the image, the running speed is slow. (2) Bottom-up method, in which the bottom-up method includes two parts: key point detection and clustering of the detected key points. That is, the key points of all people in the image are first detected, and then the detected key points are clustered and combined into different individuals. The main advantage of this type of algorithm is that it is fast and is not affected by the number of people in the image, but due to the uncertainty of clustering itself, the recognition accuracy of the entire algorithm is low.

[0004] At present, most of the data sets relied on for posture detection algorithm verification are public open source data sets. However, the requirements for rehabilitation training models vary according to different application backgrounds and user tolerance. The current mainstream open source data sets are mainly COCO and MPII. COCO is a large-scale target detection, segmentation, key point detection and subtitle data set, which consists of 328,000 images including 17 human key points. The MPII data set includes 25,000 annotated images of more than 40k people, with 16 human key point features. These data sets are too large and waste unnecessary resources. Secondly, they lack pertinence and cannot meet the user's customized needs. However, for self-made data sets in other studies, either labels are missing or labels need to be added, and machine autonomy is lacking.

[0005] From the above background, we can clearly see that the five key points that must be solved when using deep learning algorithms for rehabilitation training monitoring and evaluation are: (1) The algorithm model must be able to effectively suppress the influence of environmental factors, such as light, occlusion, etc., have strong robustness, and overcome the limitations of ordinary images. (2) The algorithm model must ensure high resolution and high processing performance, so that it can replace the guidance of current nursing staff and ensure the standardization of evaluation. (3) The algorithm model must meet high efficiency. The duration and intensity of training are limited during use. During this data processing period, the entire detection video image sequence needs to be recognized and estimated. Therefore, the posture estimation efficiency has extremely high requirements, and the algorithm model needs to accurately complete the posture estimation in a short time. (4) The data set needs to be changeable and customizable, and the data can be clustered and grouped by itself. (5) The evaluation criteria need to be quantified, and can score the detected actions and alarm dangerous actions. Summary of the invention

[0006] In order to achieve that the algorithm model can continuously maintain high resolution and high-speed processing performance in the overall rehabilitation training monitoring and evaluation process; rehabilitation patients can meet customized needs and evaluate and judge by calculating relevant objective data. It can not only meet the customized needs of users, but also solve the problem of the contradiction between the actual performance and efficiency of posture estimation. The present invention provides a customized rehabilitation training monitoring and evaluation method based on Lite-HRNet.

[0007] A custom rehabilitation training monitoring and evaluation method based on Lite-HRNet of the present invention comprises the following steps:

[0008] Step 1: Use a camera to take pre-trained posture action pictures to create or update a rehabilitation training action image dataset.

[0009] Step 2: Build the Lite-HRnet network model and use two datasets to train and fine-tune the network model.

[0010] S21: Load the COCO dataset into the Lite-HRnet network model. Pre-train the constructed network to locate the human body area of ​​initial interest in the full image.

[0011] S22: Load the custom image library into the Lite-HRnet network model. Fine-tune the constructed dataset, crop the preliminary region of interest from the full image, and locate each key point to be detected in this preliminary region of interest.

[0012] Step 3: Use the Kmeans algorithm to perform clustering, grouping and labeling to build a personalized data set.

[0013] S31: In the above process, key point information such as the human body trunk, abdomen and arms is obtained, and feature values ​​such as angle values ​​are extracted.

[0014] S32: Load the data set and feature index into the Kmeans algorithm model. Select Euclidean distance to calculate the sample data and obtain the distance matrix for clustering. Distance formula:

[0015]

[0016] Where D represents the number of attributes of the data object.

[0017] For example: (vector angle) The vertical support phase when a person is walking, that is, the state where the supporting leg and the torso are in a straight line, is selected as the torso. At this time, the angle α between the horizontal line and the line connecting the heel to the shoulder joint is taken as the data of the torso posture. The angle β between the line connecting the highest point of the ear and the eye and the vertical line of the ground is taken as the data of the head posture. The angle γ between the vertical line of the highest point of the top of the head and the highest point and one of the arms is taken as the arm posture data in the front view of the walking person. The torso angle, head angle and two arm angle data are collected.

[0018] S33: Continue iterating, and the corresponding cluster center needs to be recalculated (updated):

[0019]

[0020] Where K represents the number of clusters. When the difference between two iterations J is less than a certain threshold, that is, ΔJ < δ, the iteration is terminated, and the cluster obtained at this time is the final clustering result.

[0021] Step 4: Extract key frames to the Lite-HRnet network for training, obtain key point information, perform KNN classification first, and then complete the similarity calculation.

[0022] S41: Read the video image to be detected, and use the frame difference algorithm to obtain the average inter-frame difference intensity of the read video frame image; then perform convolution smoothing operation on the sequence to obtain the key video key frame.

[0023] S42: Load the extracted key frame image into the Lite-HRNet network for posture estimation to obtain key point coordinate information.

[0024] S43: When evaluating the video to be tested, it is first classified using the KNN algorithm to determine the category of the data set to which it belongs, where the distance formula between two pixel points (x1, y1) and (x2, y2) in the two-dimensional space is:

[0025]

[0026] The distance between the predicted point and all points is calculated using the KNN algorithm, and then saved and sorted. The first K values ​​are selected to see which category is more numerous, thereby determining the category of the rehabilitation training action pattern.

[0027] S44: Define limb vectors: First, take the torso as the center, remove irrelevant key points (such as the nose, etc.), define the extension direction of the limbs as the direction of the human skeleton vector, and then define the limb segment vector according to the direction of the skeleton vector.

[0028] S45: Calculate the cosine value of the vector angle. The cosine value of the angle between the vectors of the same limb segment in the template posture and the posture to be tested is calculated using the coordinates of the skeletal joint points. The angle between the same limbs in the two comparison postures is indirectly determined by the cosine value of the angle, thereby obtaining the similarity between the template training action and the action to be tested, that is:

[0029]

[0030] in, is the body vector of the template image action; is the body vector of the user's video action to be detected.

[0031] Since the lengths of the joints of the human skeleton are different, when calculating the similarity, the longer joints have a more important influence on the calculation of the human posture similarity, and the shorter joints have a smaller influence on the calculation of the human posture similarity. Therefore, the weight value of each joint point is different. The weight corresponding to the key nodes not involved is set to 0. Therefore, after determining the weight of each joint point, the average value of the cosine similarity of the angles of the 14 joints of the extracted human skeleton is weighted and calculated, that is:

[0032]

[0033] Among them, the value of n is 14; the value range of i is [1,14].

[0034] S46: Calculate the cosine value of the limb segment vector angle and obtain the cosine similarity of the human body posture of the key frame image, convert the cosine similarity into a percentage system and output it, that is:

[0035]

[0036] Among them, score represents the score value of measuring the standard of training action.

[0037] S47: A threshold detection link is set, that is, when the score is lower than 70, an alarm is issued. At this time, we roughly believe that too much difference in movement will be harmful to physical health.

[0038]

[0039] In this way, the calculated data can be uploaded to the nursing staff via the network as a quantitative value for standard evaluation.

[0040] The beneficial technical effects of the present invention are:

[0041] 1. The present invention adopts Kmeans and KNN joint exercises to realize fully automated data processing. The main scheme is to first use a Kmeans algorithm model to cluster and group the input unlabeled data according to the angle feature value, then label the grouping, and then use a KNN algorithm model to classify the data to be detected. This method solves the manpower loss caused by complex data and realizes fully automated data processing to a certain extent.

[0042] 2. In this scenario, the present invention adopts a high-performance Lite-HRnet network algorithm model according to the needs of posture estimation. The main implementation scheme first uses a public open source data set for pre-training, and then uses a custom data set for retraining and fine-tuning. This network algorithm model solves the common drawbacks of the two types of existing posture estimation algorithms, that is, while meeting the requirements of continuous high-resolution and continuous output, it also takes into account the indicator of computational efficiency.

[0043] 3. The present invention realizes the customization of data sets and ensures the richness of data by combining the differences in the needs of rehabilitation patients. The training mode can be designed according to the patient's physical fitness, and the data set can be updated at any time to better meet the psychological needs of the patient. The image data of each posture action can be randomly processed with exposure, saturation, hue, etc. according to different environmental factors (such as lighting, etc.) to expand the data set. This method improves the generalization ability of the model and ensures the robustness of the entire system method. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1This is a flow chart of the custom rehabilitation training monitoring and evaluation method based on Lite-HRNet of the present invention.

[0045] Figure 2 To define the limb vector graph. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0047] A custom rehabilitation training monitoring and evaluation method based on Lite-HRNet of the present invention is as follows Figure 1 As shown, the following steps are included:

[0048] Step 1: Use a camera to take pre-trained rehabilitation training action pictures to create or update a training action image dataset.

[0049] The image data is collected by shooting with a camera or other equipment, and the collected data is an unobstructed high-definition action picture. In order to minimize the impact of shooting light on posture estimation, attention should be paid to shooting in multiple directions, angles, and without obstructions. Secondly, in order to improve the generalization ability of the network model, it is also necessary to randomly change the exposure, saturation, and hue of a single training action image to form image data under different lighting and colors, and expand the data set.

[0050] Step 2: Build the Lite-HRnet network model and use two datasets to train and fine-tune the network model.

[0051] In order to meet the requirements of high-resolution and high-performance posture estimation, the network model adopted is based on the conventional HRNet network model. By applying the shuffle block to HRNet, a lightweight native Lite HRNet can be realized. The shuffle block is used to replace the second 3×3 convolution of the handle of the lightweight HRNet, and all conventional residual blocks (composed of 2 3×3 convolutions) are replaced. The conventional convolution in multi-resolution fusion is replaced by separable convolution, thus forming an original Lite-HRNet. An efficient conditional channel weighted unit is introduced to replace the computationally expensive point-by-point (1*1) convolution in the shuffle block, and the weights are calculated across channels and resolutions to construct the required Lite-HRNet network model.

[0052] Next, in order to make the model robust, we used two datasets to train and fine-tune the network model. First, the public open source COCO dataset was loaded into the Lite-HRnet network model to pre-train the constructed network. Then, the custom image dataset was loaded into the Lite-HRnet network model to retrain and fine-tune the constructed dataset. This solution has better expressiveness than one-step dataset training. This is because the two-step dataset training makes the model more targeted, which enhances the adaptability to the subsequent video frame images to be detected.

[0053] Step 3: Use the Kmeans algorithm to perform clustering, grouping and labeling to build a personalized data set.

[0054] Since several people initially created labels for the data set, it would take a lot of time and effort when the data set is slightly larger. Therefore, we used the Kmeans algorithm for cluster analysis. Here we take the back muscle training action as an example: First, a series of image data was taken and the following groups were pre-established:

[0055] Group = {Flying Swallow Push Up, Three-point Push Up, Five-point Push Up, Plank Push Up}

[0056] Then we only need to set the K value to 4, and we can divide the data into four categories and add labels to them according to the needs. First, the coordinates of key points such as the human torso, abdomen and arms are extracted through the posture estimation algorithm to calculate the angle data. Then select the Euclidean distance to calculate the sample data and obtain the distance matrix for clustering. Distance formula:

[0057]

[0058] Where D represents the number of attributes of the data object. For example, (vector angle): Here, the vertical support phase when a person is walking, that is, the state in which the supporting leg and the torso are in a straight line, is selected as the torso. At this time, the angle α between the horizontal line and the line connecting the heel to the shoulder joint is taken as the data of the torso posture. From the side of the human head, the angle β between the line connecting the highest point of the ear and the eye and the vertical line of the ground is taken as the data of the head posture. From the front view of the human walking, the angle γ between the vertical line of the highest point of the top of the head and the highest point and one of the arms is taken as the arm posture data. The data of the torso angle, head angle and two arm angles are collected.

[0059] Continue to iterate, and the corresponding cluster center needs to be recalculated (updated): the mean of all data objects in the corresponding cluster is the cluster center of the updated cluster. Define the cluster center of the Kth cluster as Center k , then the cluster center is updated as follows:

[0060]

[0061] Where K represents the number of clusters. When the difference between two iterations J is less than a certain threshold, that is, ΔJ<δ, the iteration is terminated, and the cluster obtained at this time is the final clustering result. To prevent inevitable errors, some error data can be fine-tuned. This solution can effectively reduce manpower for medium and large data sets.

[0062] Step 4: Extract key frames to the Lite-HRnet network for training, obtain key point information, perform KNN classification first, and then complete the similarity calculation.

[0063] Read the video image to be tested, and use the frame difference algorithm to obtain the average inter-frame differential intensity of the read video frame image; then perform convolution smoothing operation on its sequence to obtain the key video key frame. Load the extracted key frame image into the Lite-HRNet network for posture estimation to obtain the key point coordinate information. When evaluating the video to be tested, first classify it with the help of the KNN algorithm to determine the category of the data set to which it belongs. The distance formula between two pixel points (x1, y1) and (x2, y2) in two-dimensional space is:

[0064]

[0065] The distance between the predicted point and all points is calculated using the KNN algorithm, and then saved and sorted. The first K values ​​are selected to see which category is more numerous, thereby determining the category of the rehabilitation training action.

[0066] Next, define the limb vector (such as Figure 2 As shown in the figure): First, with the torso as the center, remove irrelevant key points (such as the nose, etc.) to define the extension direction of the limbs as the direction of the human skeleton vector, and then define the limb segment vector according to the skeleton vector direction. Then, calculate the cosine value of the vector angle: use the skeleton joint point coordinates to calculate the cosine value of the angle between the template posture and the limb segment vector with the same posture to be detected, and indirectly judge the angle between the two limbs with the same posture through the cosine value of the angle, so as to obtain the similarity between the template action and the action to be tested, that is:

[0067]

[0068] in, is the body vector of the template image action; is the body vector of the user's video action to be detected.

[0069] Since the lengths of the joints of the human skeleton are different, when calculating the similarity, the longer joints have a more important influence on the calculation of the human posture similarity, and the shorter joints have a smaller influence on the calculation of the human posture similarity. Therefore, the weight value of each joint point is different. The weight corresponding to the key nodes not involved is set to 0. Therefore, after determining the weight of each joint point, the average value of the cosine similarity of the angles of the 14 joints of the extracted human skeleton is weighted and calculated, that is:

[0070]

[0071] The value of n is 14; the value range of i is [1,14]. Calculate the cosine value of the limb segment vector angle and obtain the cosine similarity of the human body posture of the key frame image. Convert the cosine similarity into a percentage and output it, that is:

[0072]

[0073] Among them, score represents the score value of measuring the action standard.

[0074] Here we set a threshold detection link, that is, when the score is lower than 70, an alarm is issued. At this time, we roughly believe that too much difference in movement will be harmful to health.

[0075]

[0076] In this way, the calculated data can be uploaded to the nursing staff via the network as a quantitative value for standard evaluation.

[0077] The overall solution of the present invention has a complete rehabilitation training monitoring and evaluation process, which realizes the customization of the data set and ensures the fullness of the data in combination with the differences in the needs of rehabilitation personnel. The exercise mode can be designed according to the patient's physical fitness, and the data set can be updated at any time to better meet the patient's psychological needs. The image data of each rehabilitation training action can be randomly processed with exposure, saturation, hue, etc. according to different environmental factors (such as light color, etc.) to expand the data set. This method improves the generalization ability of the model and ensures the robustness of the entire system method.

[0078] The present invention adopts Kmeans and KNN joint exercises to realize fully automated data processing. In the early stage, a Kmeans algorithm model is used to cluster and group the input unlabeled data according to the angle feature value, and then the grouping is labeled. In the later stage, a KNN algorithm model is used to classify the data to be detected. This method solves the manpower loss caused by complex data and realizes fully automated data processing to a certain extent.

[0079] According to the needs of posture estimation, the present invention adopts a high-performance Lite-HRnet network algorithm model. Through a deep learning model trained with a large number of samples, it has the advantages of strong generalization, strong adaptability, good portability, etc. This method first uses a public open source data set for pre-training, and then uses a custom data set for retraining and fine-tuning. The reasoning speed of the model is basically stable at 0.16s, flops is 0.42G, and the number of parameters is 1.76M. This network algorithm model solves the common drawbacks of the two types of existing posture estimation algorithms, that is, while meeting the requirements of continuous high-resolution and continuous output, it also takes into account the indicator of computational efficiency. It also has excellent performance in real-time algorithm detection, providing strong guarantee for subsequent work and ensuring the reliability and convenience of rehabilitation training standard evaluation.

Claims

1. A custom rehabilitation training monitoring and evaluation method based on Lite-HRNet, characterized in that: The following steps are involved: Step 1: Use a camera to take pre-trained rehabilitation training action pictures and create or update the action image dataset; Step 2: Build the Lite-HRnet network model and use two datasets to train and fine-tune the network model; S21: Load the COCO dataset into the Lite-HRnet network model, pre-train the constructed network, locate the human body area of ​​initial interest in the whole image, and obtain the first training set; S22: Load the custom image dataset into the Lite-HRnet network model; Fine-tune the constructed data set, crop the preliminary region of interest from the whole image, and locate each key point to be detected in this preliminary region of interest to obtain the second training set; Step 3: Use the Kmeans algorithm to cluster, group and label to build a personalized data set; S31: extracting feature values ​​from key point information; S32: Load the data set and feature indicators into the Kmeans algorithm model, select Euclidean distance to calculate the sample data and obtain the distance matrix for clustering. The distance formula is: Where D represents the number of attributes of the data object; S33: Continue iterating, and the corresponding cluster center needs to be recalculated or updated: Where K represents the number of clusters; when the difference between two iterations is less than a certain threshold, that is, ΔJ < δ, the iteration is terminated, and the cluster obtained at this time is the final clustering result; Step 4: Extract key frames to the Lite-HRnet network for training, obtain key point information, perform KNN classification first, and then complete the similarity calculation; S41: reading the video image to be detected, and using the frame difference algorithm to obtain the average inter-frame difference intensity of the read video frame image; then performing convolution smoothing operation on the sequence to obtain the key video key frame; S42: Load the extracted key frame image into the Lite-HRNet network for posture estimation to obtain key point coordinate information; S43: When evaluating the video to be tested, it is first classified using the KNN algorithm to determine the category of the data set to which it belongs, where the distance formula between two pixel points (x1, y1) and (x2, y2) in the two-dimensional space is: The KNN algorithm is used to calculate the distance between the predicted point and all points, and then the distances are saved and sorted. The first K values ​​are selected to see which category is more numerous, thereby determining the category of the rehabilitation training mode. S44: Define limb vectors: First, take the trunk as the center, remove irrelevant key points and define the extension direction of the limbs as the direction of the human skeleton vector, and then define the limb segment vector according to the direction of the skeleton vector; S45: Calculate the cosine value of the vector angle; use the coordinates of the skeletal joint points to calculate the cosine value of the angle between the vectors of the same limb segment of the template action posture and the action posture to be detected, and indirectly judge the angle between the two limbs with the same comparison posture through the cosine value of the angle, so as to obtain the similarity between the template posture and the posture to be tested, that is: in, is the limb vector of the action posture of the template image; The limb vector of the user's action posture to be detected in the video; Determine the weight W of each joint point i After that, the average value of the cosine similarity of the angles of the 14 joints of the human skeleton extracted is calculated by weighted calculation, that is: Among them, the value of n is 14; the value range of i is [1,14]; S46: Calculate the cosine value of the limb segment vector angle and obtain the cosine similarity of the human body posture of the key frame image, convert the cosine similarity into a percentage system and output it, that is: Among them, score represents the score value of measuring the action standard; S47: Set a threshold detection link, that is, when the score is lower than 70, an alarm is issued. At this time, we believe that the action difference is too large and will be harmful to health; The calculated data is sent to the nursing staff via the network as a quantitative value of the standard evaluation.