OpenPose-based body posture detection method and system

Abstract

The invention relates to a body posture detection method and system based on OpenPose, and the method comprises the following steps: constructing a data set, and carrying out the retraining of an OpenPose model, and obtaining an OpenPose precise standard model; inputting a to-be-detected picture into the OpenPose fine standard model, and outputting to obtain 21 fine standard mark points; 6 auxiliary mark points are determined on the basis of the 21 precise standard mark points in combination with a preset marking rule; according to the 21 precise standard mark points and the 6 auxiliary mark points, 25 preset body posture interpretation indexes are combined, and a body posture detection result is obtained through calculation and comparison. Compared with the prior art, the method has the advantages that an automatic whole-process standardized detection scheme is realized, and the posture detection and recognition can be efficiently, accurately and comprehensively carried out.

Description

Technical Field

[0001] The present invention relates to the technical field of human body data recognition, and in particular to a body posture detection method and system based on OpenPose. Background Art

[0002] In the process of human posture and body shape detection and analysis, most of them rely on manual detection, using original measurement tools and the experience of the testers to measure the posture and body shape of each person, so as to obtain the angle deviation of specific parts or joints of the human body, and then use it to correct the posture and body shape of each specific customer. However, this measurement method mainly relies on human subjective assumptions and daily experience, which not only has a large measurement error, but also cannot meet the needs of scientific analysis of posture and body shape and skeleton angles for specific customers.

[0003] In particular, in the current spinal screening service, it is necessary to manually mark key bone points, measure and calculate body posture and posture interpretation indicators, and interpret them. This process has a low degree of standardization, is time-consuming, has low manual efficiency, and cannot form effective data assets. In addition, the existing screening indicators are only single scoliosis, which makes it difficult to detect and identify the whole body posture. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a body posture detection method and system based on OpenPose, which can efficiently, accurately and comprehensively perform body posture detection and recognition.

[0005] The purpose of the present invention can be achieved by the following technical solution: A body posture detection method based on OpenPose, comprising the following steps:

[0006] S1. Build a data set and retrain the OpenPose model to obtain the OpenPose standard model.

[0007] S2. Input the image to be tested into the OpenPose precision standard model, and output 21 precision standard markers;

[0008] S3, based on the 21 precise standard marking points and combined with the preset marking rules, 6 auxiliary marking points are determined;

[0009] S4. Based on the 21 precise standard marking points and 6 auxiliary marking points, combined with the preset 25 posture judgment indicators, the posture and body shape detection results are obtained through calculation and comparison.

[0010] Furthermore, the step S1 includes the following process:

[0011] S11. Collect multiple groups of human posture images. The samples include adolescents aged 6-18 of different genders and adults aged 18 and above of different genders. Each group of human posture images includes body images of the front, back and side of the human body, and construct a data set.

[0012] S12, annotate 21 initial key points for the images in the dataset;

[0013] S13. Repeatedly train the OpenPose model using the labeled data set to obtain the OpenPose precision standard model, which can output 21 precision standard marker points by inputting a posture picture.

[0014] Furthermore, the 21 precise standard marking points include: left and right pupils, left and right ears, sternal notch, left and right acromioclavicular joints, left and right greater trochanters of the femur, left and right anterior superior iliac spines, left and right ankles, left and right lateral epicondyles of the femur, left and right patellar centers, left and right Achilles tendons, and left and right heels.

[0015] Furthermore, step S13 specifically selects a 25-point model from the pre-trained model provided by OpenPose as a basis, batch inputs the tuned 21 initial key points through a script, and tunes the trained data again to repeat the training, and obtains the OpenPose precision standard model by increasing the number of training iterations.

[0016] Furthermore, the six auxiliary marking points include: the left and right eighth ribs, the left and right posterior superior iliac spines on the back, the anterior superior iliac spine on the side, and the posterior superior iliac spine on the side.

[0017] Furthermore, the labeling rule preset in step S3 is specifically to determine the relevant weight data of each auxiliary marking point by calculating the distance and weight between each auxiliary marking point and its adjacent precise standard marking point.

[0018] Furthermore, the 25 posture interpretation indicators include 19 precise standard indicators and 6 auxiliary indicators, and the 19 precise standard indicators include: head and neck tilt left and right, head extension, shoulder height left and right, shoulder front and back tilt, pelvis height left and right, pelvis left and right displacement, pelvis front and back displacement, knee valgus (X type), knee valgus (O type), knee hyperextension, knee hyperflexion, foot valgus, foot varus;

[0019] The six auxiliary indicators include: left and right displacement of the thoracic spine, anterior and posterior tilt of the pelvis, and left and right displacement of the pelvis.

[0020] Furthermore, the step S4 comprises the following steps:

[0021] S41, calculating the offset angles of 25 posture judgment indicators based on 21 precise standard marking points and 6 auxiliary marking points;

[0022] S42, comparing the deviation angles of the 25 posture judgment indicators with the preset thresholds respectively to obtain the posture detection results.

[0023] Furthermore, the preset thresholds corresponding to the offset angles of the 25 body posture determination indicators in step S42 are:

[0024] The preset threshold corresponding to the lateral head extension is 4.95 degrees;

[0025] The preset threshold for left tilt of the head from the front and back is 2.05 degrees;

[0026] The preset threshold for the right tilt of the head from the front and back is 2.05 degrees;

[0027] The preset threshold corresponding to the left height of the front and back shoulders is 1.53 degrees;

[0028] The preset threshold corresponding to the right height of the front and back shoulders is 1.53 degrees;

[0029] The preset threshold corresponding to the lateral shoulder forward tilt is 3.11 degrees;

[0030] The preset threshold corresponding to the lateral shoulder tilt is 3.11 degrees;

[0031] The preset threshold corresponding to the lateral pelvic anterior displacement is 1.98 degrees;

[0032] The preset threshold corresponding to the lateral pelvic posterior displacement is 1.98 degrees;

[0033] The preset threshold corresponding to the left pelvic height of the front and back is 2.01 degrees;

[0034] The preset threshold corresponding to the right pelvic height of the front and back sides is 2.01 degrees;

[0035] The preset threshold corresponding to the leftward shift of the frontal pelvis is 2.05 degrees;

[0036] The preset threshold corresponding to the frontal pelvic right shift is 2.05 degrees;

[0037] The preset threshold corresponding to the lateral knee hyperextension was 1.96 degrees;

[0038] The preset threshold corresponding to the lateral knee hyperflexion is 1.96 degrees;

[0039] The preset threshold corresponding to the front and back knee varus (O type) is 3.03 degrees;

[0040] The preset threshold corresponding to the front and back knee valgus (X-type) is 3.03 degrees;

[0041] The preset threshold corresponding to the dorsal ankle inversion was 2.97 degrees;

[0042] The preset threshold corresponding to the dorsal ankle eversion was 2.97 degrees;

[0043] The preset threshold corresponding to the lateral pelvic anterior tilt is 9.81 degrees;

[0044] The preset threshold corresponding to the lateral pelvic posterior tilt is 3.09 degrees;

[0045] The preset threshold corresponding to the leftward shift of the frontal pelvis is 0.87 degrees;

[0046] The preset threshold corresponding to the frontal pelvic right shift is 0.87 degrees;

[0047] The preset threshold corresponding to the leftward shift of the front and back thoracic vertebrae is 0.99 degrees;

[0048] The preset threshold corresponding to the rightward shift of the front and back thoracic vertebrae is 0.99 degrees.

[0049] A body posture detection system based on OpenPose, comprising a mobile terminal and a server terminal connected thereto, wherein the mobile terminal is equipped with a camera device for collecting front, side and back images of a human body and transmitting the images to the server terminal;

[0050] The server is provided with an OpenPose fine standard model, and the front, side and back pictures of the human body uploaded by the mobile terminal are input into the OpenPose fine standard model, and 21 fine standard marking points are output;

[0051] The server determines 6 auxiliary marking points based on 21 precise standard marking points and preset marking rules; then, based on the 21 precise standard marking points and 6 auxiliary marking points and combined with 25 preset posture judgment indicators, the server calculates and compares the 21 precise standard marking points and 6 auxiliary marking points to output the posture detection results and transmit them to the mobile terminal.

[0052] Compared with the prior art, the present invention has the following advantages:

[0053] The present invention constructs a data set to retrain the OpenPose model to obtain an OpenPose fine standard model, uses the OpenPose fine standard model to obtain 21 fine standard markers, and combines the preset annotation rules to further determine 6 auxiliary markers (covering the head and neck, shoulders, thoracolumbar spine, pelvis, knee joints, ankles and other parts), and finally calculates and compares the 21 fine standard markers and 6 auxiliary markers in combination with the preset 25 posture interpretation indicators to obtain the posture detection results. This realizes an automatic and efficient full-process standardized detection solution that can accurately and comprehensively detect and identify human posture.

[0054] The present invention designs 25 posture judgment indicators and corresponding indicator abnormal thresholds, which greatly enriches the number of abnormal screening indicators, expanding from the previous single scoliosis to the whole body posture recognition of the head, neck, pelvis, leg shape, etc. (head forward tilt, pelvic forward tilt, XO-type legs, calf hyperextension / flexion, etc.), and can fully ensure the accuracy of detection and identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0056] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Example

[0058] like Figure 1 As shown, a body posture detection method based on OpenPose includes the following steps:

[0059] S1. Build a data set and retrain the OpenPose model to obtain the OpenPose standard model.

[0060] S2. Input the image to be tested into the OpenPose precision standard model, and output 21 precision standard markers;

[0061] S3, based on the 21 precise standard marking points and combined with the preset marking rules, 6 auxiliary marking points are determined;

[0062] S4. Based on the 21 precise standard marking points and 6 auxiliary marking points, combined with the preset 25 posture judgment indicators, the posture and body shape detection results are obtained through calculation and comparison.

[0063] Based on the above method, a posture detection system based on OpenPose is built, including a mobile terminal and a server connected thereto, wherein the mobile terminal is equipped with a camera device for collecting front, side and back images of the human body and transmitting them to the server;

[0064] The server is equipped with an OpenPose precision standard model. The front, side, and back images of the human body uploaded by the mobile terminal are input into the OpenPose precision standard model, and 21 precision standard markers are output;

[0065] The server also determines 6 auxiliary markers based on the 21 precise standard markers and the preset annotation rules; then, based on the 21 precise standard markers and 6 auxiliary markers, combined with the preset 25 posture judgment indicators, the server calculates and compares the 21 precise standard markers and 6 auxiliary markers to output the posture detection results and transmit them to the mobile terminal.

[0066] This embodiment applies the above scheme and first proposes a basic design of detection and recognition data of 27 marking points + 25 body posture judgment indicators, wherein the 27 marking points include 21 precise standard marking points + 6 auxiliary marking points; the 25 body posture judgment indicators include 19 precise standard indicators + 6 auxiliary indicators.

[0067] The details are as follows:

[0068] (1) Precision standard marking points: 21

[0069] Left and right pupils, left and right ears, sternal notch, left and right acromioclavicular joints, left and right greater trochanters of the femur, left and right anterior superior iliac spines, left and right ankles, left and right lateral epicondyles of the femur, left and right patellar centers, left and right Achilles tendons, and left and right heels.

[0070] (2) Precision standard indicators: 19

[0071] Head and neck tilt left and right, head extends forward, shoulders raised left and right, shoulders tilted forward and backward, pelvis raised left and right, pelvis moved left and right, pelvis moved forward and backward, knee valgus (X type), knee valgus (O type), knee hyperextension, knee hyperflexion, foot valgus, foot varus.

[0072] (3) Auxiliary marking points: 6 points

[0073] The eighth rib on the left and right, the left and right posterior superior iliac spines on the back, the anterior superior iliac spine on the side, and the posterior superior iliac spine on the side.

[0074] (4) Auxiliary indicators: 6

[0075] The thoracic spine moves left and right, the pelvis tilts forward and backward, and the pelvis moves left and right.

[0076] Among the 27 marker points, 21 precise standard marker points are first obtained based on the retrained OpenPose model: after the user uploads the front, back and side views on the mobile terminal, the server calls the OpenPose service to obtain the key points of each posture (data retrained by OpenPose); then, based on the 21 precise standard marker points, 6 auxiliary marker points are automatically drawn using the annotation rules.

[0077] This embodiment, based on the existing open source OpenPose algorithm model, uses data enhancement to train a more accurate model, namely the OpenPose precise standard model. The process is as follows:

[0078] 1. Dataset preparation: Based on a set of collected adolescent body image libraries, there are 1,200 groups of photos, each with front, back and side views, for a total of 3,600 human body posture images.

[0079] 2. Data annotation: Each picture is based on the 21 initial key points obtained by the OpenPose algorithm model, and these 21 initial points are drawn using the fabricjs canvas. Each point is tuned and adjusted to the most accurate position.

[0080] 3. Training model: Select a 25-point model from the pre-trained model provided by OpenPose as the basis, and batch input the 21 initial points after tuning through the script to accelerate convergence and improve results.

[0081] 4. Data enhancement: The trained data is fine-tuned again and the training is repeated. By increasing the number of training iterations, the final output result is close to the diagnostic threshold.

[0082] 5. Result output: The trained OpenPose precise standard model is obtained. Based on the model algorithm, 21 precise standard markers can be obtained from a body posture picture.

[0083] In order to accurately draw these six auxiliary marking points, this embodiment uses manual annotation + weighted average algorithm to obtain a set of drawing rules. The calculation process is as follows:

[0084] 1. Data input: constructed adolescent body image library.

[0085] 2. Labeling process: The 21 precise standard markers obtained based on the OpenPose precise standard model are displayed on the human body canvas, and then manually marked. For example, the missing eighth rib markers on the left and right sides of the human body are marked at the correct position on the human body diagram.

[0086] 3. Calculation process: Based on the manual marking point, the distance and weight between the manual marking point and its adjacent precision standard marking point are calculated through the weighted average algorithm, and the relevant weight data of the missing 6 auxiliary points are calculated.

[0087] 4. Result output: Get the labeling rules for each auxiliary marker point, then draw them according to their respective rules and display them on the human body canvas.

[0088] Therefore, according to the OpenPose precise standard model + auxiliary point labeling rules, the 27 precise marking points required for human spine assessment can be obtained, and then the abnormal indicator results of 25 indicators can be obtained based on these marking points.

[0089] This embodiment designs the judgment criteria of 25 indicators:

[0090] 1. Source of interpretation standards: Obtain the body posture interpretation index file, which lists in detail the key points and measurement methods of each index to derive the respective abnormal diagnosis standards.

[0091] 2. Drawing calculation: Based on the OpenPose standard model + auxiliary point annotation rules, 27 marking points are marked on the human body canvas, and according to the judgment criteria, the lines between the relevant marking points are drawn, and the corresponding measurement method is used to obtain the index offset angle.

[0092] 3. Enhanced interpretation standards: Determine the degree of deviation of abnormal indicators, combine the specific deviation angle, and customize enhanced interpretation standards through the k-means algorithm. For example, for the left and right tilt indicators of the head, according to the first and second steps above, the head tilted 2.2 degrees to the left, the corresponding level is primary left tilt, and the left tilt of 1.9 degrees corresponds to the normal range. Finally, the k-means algorithm calculates that the normal range of left and right tilt is less than 2.05 degrees, which means that as long as the tilt angle exceeds 2.05 degrees, it is an abnormal indicator.

[0093] The 25 indicator abnormal threshold data obtained in this embodiment are specifically:

[0094] Serial number Body shape direction Location index Abnormal Threshold 1 side head Reach 4.95 degrees 2 Front, back head Left-leaning 2.05 degrees 3 Front, back head Right 2.05 degrees 4 Front, back Shoulders Left High 1.53 degrees 5 Front, back Shoulders Right High 1.53 degrees 6 side Shoulders Lean Forward 3.11 degrees 7 side Shoulders Backward 3.11 degrees 8 side pelvis Move forward 1.98 degrees 9 side pelvis Move Back 1.98 degrees 10 Front, back pelvis Left High 2.01 degrees 11 Front, back pelvis Right High 2.01 degrees 12 front pelvis Shift Left 2.05 degrees 13 front pelvis Shift Right 2.05 degrees 14 side Knee Knee hyperextension 1.96 degrees 15 side Knee Knee hyperflexion 1.96 degrees 16 Front, back Knee Inversion (O type) 3.03 degrees 17 Front, back Knee Eversion (X-type) 3.03 degrees 18 back Ankle Inversion of the foot 2.97 degrees 19 back Ankle Foot eversion 2.97 degrees 20 side pelvis Lean Forward 9.81 degrees 21 side pelvis Backward 3.09 degrees 22 front pelvis Shift Left 0.87 degrees 23 front pelvis Shift Right 0.87 degrees 24 Front, back thoracic Shift Left 0.99 degrees 25 Front, back thoracic Shift Right 0.99 degrees

[0095] Finally, through the OpenPose precise standard model + auxiliary point annotation rules + indicator judgment standards, the final list of posture detection results can be automatically evaluated based on a body posture picture.

[0096] In summary, this solution realizes a highly accurate and automated posture detection method and system (skeletal key point identification and labeling, posture judgment index measurement and calculation, posture judgment index evaluation). On the one hand, it standardizes the screening and evaluation, changing the past manual dot measurement and subjective evaluation to full process standardization. On the other hand, it enriches the marker points (covering the head and neck, shoulders, thoracic and lumbar spine, pelvis, knee joints, ankles, etc.) and screening index content, expanding from the past single scoliosis to the head and neck, pelvis, leg type and other whole body posture recognition (head forward tilt, pelvic forward tilt, XO-type legs, calf hyperextension / bend, etc.). In addition, it can also realize data asset retention, based on screening data and through manual adjustment records, and supervised learning to train the OpenPose25 model to improve recognition accuracy and precision. At the same time, the data assets can also be used in other fields.

Claims

1. A body posture detection method based on OpenPose, characterized in that: The following steps are involved: S1. Build a data set and retrain the OpenPose model to obtain the OpenPose standard model. S2. Input the image to be tested into the OpenPose precision standard model, and output 21 precision standard markers; S3, based on the 21 precise standard marking points and combined with the preset marking rules, 6 auxiliary marking points are determined; S4. Based on the 21 precise standard marking points and 6 auxiliary marking points, combined with the preset 25 posture judgment indicators, the posture and body shape detection results are obtained through calculation and comparison.

2. The method for detecting body posture based on OpenPose according to claim 1, characterized in that: The step S1 includes the following process: S11. Collect multiple groups of human posture images. The samples include adolescents aged 6-18 of different genders and adults aged 18 and above of different genders. Each group of human posture images includes body images of the front, back and side of the human body, and construct a data set. S12, annotate 21 initial key points for the images in the dataset; S13. Repeatedly train the OpenPose model using the labeled data set to obtain the OpenPose precision standard model, which can output 21 precision standard marker points by inputting a posture picture.

3. The method for detecting body posture based on OpenPose according to claim 1, characterized in that: The 21 precise standard marking points include: left and right pupils, left and right ears, sternal notch, left and right acromioclavicular joints, left and right greater trochanters of femur, left and right anterior superior iliac spines, left and right ankles, left and right lateral epicondyles of femur, left and right patellar centers, left and right Achilles tendons, and left and right heels.

4. The method for detecting body posture based on OpenPose according to claim 2, characterized in that: The step S13 specifically selects a 25-point model from the pre-trained model provided by OpenPose as a basis, batch inputs the 21 initial key points after tuning through a script, and tunes the trained data again to perform repeated training, and obtains the OpenPose precise standard model by increasing the number of training iterations.

5. The method for detecting body posture based on OpenPose according to claim 1, characterized in that: The six auxiliary marking points include: the left and right eighth ribs, the left and right posterior superior iliac spines on the back, the anterior superior iliac spine on the side, and the posterior superior iliac spine on the side.

6. The method for detecting body posture based on OpenPose according to claim 1, characterized in that: The preset marking rule in step S3 is specifically to determine the relevant weight data of each auxiliary marking point by calculating the distance and weight between each auxiliary marking point and its adjacent precise standard marking point.

7. The method for detecting body posture based on OpenPose according to claim 1, characterized in that: The 25 posture interpretation indicators include 19 precise standard indicators and 6 auxiliary indicators. The 19 precise standard indicators include: head and neck tilt left and right, head extension, shoulder height left and right, shoulder front and back tilt, pelvis height left and right, pelvis left and right displacement, pelvis front and back displacement, knee valgus, knee varus, knee hyperextension, knee hyperflexion, foot valgus, foot varus; The six auxiliary indicators include: left and right displacement of the thoracic spine, anterior and posterior tilt of the pelvis, and left and right displacement of the pelvis.

8. The method for detecting body posture based on OpenPose according to claim 7, characterized in that: The step S4 comprises the following steps: S41, calculating the offset angles of 25 posture judgment indicators based on 21 precise standard marking points and 6 auxiliary marking points; S42, comparing the deviation angles of the 25 posture judgment indicators with the preset thresholds respectively to obtain the posture detection results.

9. The method for detecting body posture based on OpenPose according to claim 8, characterized in that: The preset thresholds corresponding to the offset angles of the 25 posture judgment indicators in step S42 are: The preset threshold corresponding to the lateral head extension is 4.95 degrees; The preset threshold for left tilt of the head from the front and back is 2.05 degrees; The preset threshold for the right tilt of the head from the front and back is 2.05 degrees; The preset threshold corresponding to the left height of the front and back shoulders is 1.53 degrees; The preset threshold corresponding to the right height of the front and back shoulders is 1.53 degrees; The preset threshold corresponding to the lateral shoulder forward tilt is 3.11 degrees; The preset threshold corresponding to the lateral shoulder tilt is 3.11 degrees; The preset threshold corresponding to the lateral pelvic anterior displacement is 1.98 degrees; The preset threshold corresponding to the lateral pelvic posterior displacement is 1.98 degrees; The preset threshold corresponding to the left pelvic height of the front and back is 2.01 degrees; The preset threshold corresponding to the right pelvic height of the front and back sides is 2.01 degrees; The preset threshold corresponding to the leftward shift of the frontal pelvis is 2.05 degrees; The preset threshold corresponding to the frontal pelvic right shift is 2.05 degrees; The preset threshold corresponding to the lateral knee hyperextension was 1.96 degrees; The preset threshold corresponding to the lateral knee hyperflexion is 1.96 degrees; The preset threshold corresponding to front and back knee varus is 3.03 degrees; The preset threshold corresponding to the front and back knee valgus is 3.03 degrees; The preset threshold corresponding to the dorsal ankle inversion was 2.97 degrees; The preset threshold corresponding to the dorsal ankle eversion was 2.97 degrees; The preset threshold corresponding to the lateral pelvic anterior tilt is 9.81 degrees; The preset threshold corresponding to the lateral pelvic posterior tilt is 3.09 degrees; The preset threshold corresponding to the leftward shift of the frontal pelvis is 0.87 degrees; The preset threshold corresponding to the frontal pelvic right shift is 0.87 degrees; The preset threshold corresponding to the leftward shift of the front and back thoracic spine is 0.99 degrees; The preset threshold corresponding to the rightward shift of the front and back thoracic vertebrae is 0.99 degrees.

10. A body posture detection system based on OpenPose, characterized in that: It includes a mobile terminal and a server terminal connected thereto, wherein the mobile terminal is equipped with a camera device for collecting front, side and back images of a human body and transmitting them to the server terminal; The server is provided with an OpenPose fine standard model, and the front, side and back pictures of the human body uploaded by the mobile terminal are input into the OpenPose fine standard model, and 21 fine standard marking points are output; The server determines 6 auxiliary marking points based on 21 precise standard marking points and preset marking rules; then, based on the 21 precise standard marking points and 6 auxiliary marking points and combined with 25 preset posture judgment indicators, the server calculates and compares the 21 precise standard marking points and 6 auxiliary marking points to output the posture detection results and transmit them to the mobile terminal.

Images