Non-standing body measurement method and device based on real-time key point detection

Abstract

The invention discloses a non-standing type body measurement method and device based on real-time key point detection. The method comprises the following steps: (1) collecting a non-standing type data set image; (2) marking joint key points: a left ear, a right ear, a left shoulder, a right shoulder, a left elbow, a right elbow, a left wrist, a right wrist, a left hip, a right hip, a left knee, a right knee, a left ankle and a right ankle; the labeling detection frame comprises a big arm, a small arm, a thigh, a shank, a body and a head; marking the longest edge key points of the local cross section: a big arm, a small arm, a thigh and a shank; (3) designing a composite network, merging all key points and detection frames, and sending the merged key points and detection frames into network training; and (4) the composite network supports three modes of video real-time detection, video uploading and picture uploading to obtain a body data result. The method has the advantages of being high in precision, low in delay and high in robustness, can be used for product function application such as body measurement of the physically disabled, assistance of rapid measurement of body data of healthy people and intelligent body measurement of clothing merchants, and has good commercialization prospects.

Description

Technical Field

[0001] The present invention relates to the field of computer vision real-time target detection, and in particular to a non-standing body measurement method and device based on real-time key point detection. Background Art

[0002] Traditional body measurement methods usually rely on door-to-door services or family members to help use measuring tools for measurement, which is not only cumbersome, but also has problems such as high service fees and lack of measuring tools. The current body measurement method is mainly suitable for standing healthy people to measure in a fixed position, and only supports measurement by taking photos, which makes it difficult for people with physical disabilities to obtain accurate body data. With the popularization of smart mobile devices, it is particularly important to develop a method or device that can be integrated into mobile phones, tablets and other devices, which has the ability to accurately measure the body data of people with physical disabilities and simplifies the user's operation steps to help people with physical disabilities achieve barrier-free body measurement. Summary of the invention

[0003] The present invention aims to overcome the above-mentioned deficiencies in the prior art and provides a non-standing body measurement method and device based on real-time key point detection, which uses a designed composite network to measure the body data of the person being measured from the whole to the part and then to the whole.

[0004] A first aspect of the present invention relates to a non-standing body measurement method based on real-time key point detection, comprising the following steps:

[0005] (1) Collecting images of non-standing datasets: The person being collected needs to place a card of regular specifications in the camera's field of view. The collector can take a picture of the person being collected or record a circle around the person being collected;

[0006] (1.1) The person being collected must have a card with a length of 85.6mm, a width of 54mm, and a thickness of less than 10mm;

[0007] (1.2) Place the card in step (1.1) within the camera’s field of view;

[0008] (1.3) Taking photos of the person being collected or recording a video around the person being collected;

[0009] (2) Label key points and detection frames: Label joint key points: left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle; label the longest edge key points of the local cross section of the upper arm, forearm, thigh, and calf; label the detection frames: upper arm, forearm, thigh, calf, body, head, and card;

[0010] (2.1) For the detection frame: upper arm, forearm, thigh, calf, body, head, card; use the smallest bounding box to encircle the target, and the length and width of the bounding box are parallel to the length and width of the image. Record the coordinate information (X, Y, W, H) of the bounding box of each part.

[0011]

[0012] Where x, y are the image coordinates of the center of the bounding box; Box_x, Box_y are the length and width of the bounding box respectively; image_x and image_y are the length and width of the image respectively.

[0013] (2.2) For the key joint points: left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle; the marking rules are marked according to the medical human structure rules. In the local area of ​​step (2.1), record the coordinates of each joint point in the local area (X, Y, 0, 0); the calculation of X, Y is the same as formula (2-1);

[0014] (2.3) The key points of the longest edge of the local cross section: upper arm, forearm, thigh, calf; the labeling rule is to label according to the widest cross section of the current area image. In the local area of ​​step (2.1), record the coordinates of the two points of the widest cross section (X1, Y1, 0, 0), (X2, Y2, 0, 0); the calculation of X1, Y1, X2, Y2 is the same as formula (2-1);

[0015] (3) Design a composite network and train data: After the image is input into the composite network, the upper arm, forearm, thigh, calf, body, and head regions are first detected based on the target detection model; key point detection and regional data analysis are then performed on each sub-region; finally, the data of each region are merged for training.

[0016] (3.1) The composite network structure includes input layer, region detection layer, key point detection layer, data analysis layer, data correction layer, update layer and output layer.

[0017] (3.1.1) Input layer: can input images; if input video, then each frame of the video is input in a loop;

[0018] (3.1.2) Region detection layer: This layer uses a target detection algorithm to detect upper arms, lower arms, thighs, lower legs, body, head, and cards. The target detection algorithm includes but is not limited to the YOLO series and the RCNN series. Its label is the detection box of step (2.1);

[0019] (3.1.3) Key point detection layer: This layer is based on the region detection results of step (3.1.2) and is fed into each branch of this layer to detect the key points contained in each branch. Each branch has its own weight in the format of step (3.1.2), and its label is the key point of step (2.2) and step (2.3).

[0020] (3.1.4) Data analysis layer: The length and width of the detection results of each area need to be mapped. The mapping rules are:

[0021]

[0022] Where R_h is calculated in the same way as formula (3-2). (x 1 ,y 1 ), (x 2 ,y 2 ) are the coordinates of the key points detected in the area. C_w, C_h are the length and width of the card bounding box detected in step (3.1.2); T_x, T_y are the real mapping lengths in each area.

[0023] (3.1.5) Data correction layer: introduce hyperparameter coefficients to correct each indicator: long arm length = λ 1 *T_x 1 、Arm width = λ 2 *T_y 1 , forearm length = λ 3 *T_x 2 、Forearm width = λ 4 *T_y 2 , thigh length = λ 5 *T_x 3 、Thigh width = λ 6 *T_y 3 , calf length = λ 7 *T_x 4 , calf width = λ 8 *T_y 4 , head length = λ 9 *T_x 5 , body length = λ 10 *T_x 6 , height = head length + torso length + thigh length + forearm length; 1 -λ 10 Corrections are made at this layer. Correction updates are made in the back propagation of this layer.

[0024] (3.1.6) Update layer: This layer records the maximum value of each indicator data of each image frame, and continuously updates the maximum value of each indicator data during the loop.

[0025] (3.1.7) Output layer: At the end of step (3.1.6), the final body index data is output.

[0026] (3.2) Repeat step (3.1) for all the images and video data collected in step (1.3) to train and obtain the optimal model;

[0027] (4) Video stream, upload video, upload image into the composite network to obtain body data: the result of a single image test is the final body measurement data; the video stream and the uploaded video are input into the composite network to obtain the maximum index data in each frame image. (4.1) The user performs real-time video flow. The user places the card that meets step (1.1) at any position in the camera's field of view, and records a circle around the person being tested, waiting for step (3.1.7) to stop updating data in real time. Or stop in advance at any time.

[0028] (4.2) To upload videos and images, you must follow the shooting rules of step (4.1) and wait for the final body data measured in step (3.1.7).

[0029] The second aspect of the present invention relates to a non-standing body measuring device based on real-time key point detection, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a non-standing body measuring method based on real-time key point detection of the present invention.

[0030] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a non-standing body measurement method based on real-time key point detection of the present invention.

[0031] The beneficial effect of the present invention is that the real-time physical data measurement of the person being tested can be measured through a composite network, which can greatly improve the robustness of the physical data measurement. Since cards of commonly used specifications are used as mapping references, the present invention can measure the person being measured at various angles without being limited to the shooting position, thereby improving the measurement accuracy. At the same time, the data update layer in the composite network can reduce the mapping errors generated by the camera and the environment, thereby improving the measurement accuracy of the present invention. The present invention simultaneously supports three methods: video stream detection, uploaded video detection, and uploaded picture detection, which will greatly reduce the threshold and operational difficulty of physical data measurement for people with physical disabilities. The effect of the present invention is better than the existing method, and has the characteristics of versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of steps for using the method of the present invention;

[0033] Figure 2is a bounding box annotation diagram of the target by the method of the present invention;

[0034] Figure 3 It is a joint key point labeling diagram of the target by the method of the present invention;

[0035] Figure 4 It is a key point marking diagram of the target area by the method of the present invention;

[0036] Figure 5 It is a composite network structure diagram of the method of the present invention;

[0037] Figure 6 It is a video updating body data map of the method of the present invention;

[0038] Figure 7 The method of the present invention obtains a body data map by uploading an image. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0040] Example 1

[0041] The core technology of the present invention is to use a composite network to perform real-time detection of the measured person from the whole to the part and then to the whole.

[0042] This embodiment proposes a non-standing body measurement method based on real-time key point detection, including the following steps: Figure 1 ): (1) Collecting non-standing dataset images: The person being collected needs to place a regular-sized card in the camera's field of view, and the collector can take a picture of the person being collected or record a circle around the person being collected;

[0043] (1.1) The person being collected has a card with a length of 85.6 mm, a width of 54 mm, and a thickness of 0.9 mm;

[0044] (1.2) Place the card in step (1.1) within the camera’s field of view;

[0045] (1.3) Taking photos of the person being collected or recording a video around the person being collected;

[0046] (2) Label key points and detection frames: Label joint key points: left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle; label the longest edge key points of the local cross section of the upper arm, forearm, thigh, and calf; label the detection frames: upper arm, forearm, thigh, calf, body, head, and card;

[0047] (2.1) For the detection frame: upper arm, lower arm, thigh, calf, body, head, card; use the smallest bounding box to encircle the target ( Figure 2), the length and width of the bounding box are parallel to the length and width of the image. Record the coordinate information (X, Y, W, H) of the bounding box of each part.

[0048]

[0049] Where x, y are the image coordinates of the center of the bounding box; Box_x, Box_y are the length and width of the bounding box respectively; image_x and image_y are the length and width of the image respectively.

[0050] (2.2) For the key joint points: left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle; the annotation rules are based on the medical human structure rules. In the local area of ​​step (2.1), record the coordinates of each joint point in the local area (X, Y, 0, 0) ( Figure 3 ); The calculation of X, Y is the same as formula (2-1); (2.3) The key points of the longest edge of the local cross section: upper arm, lower arm, thigh, lower leg; The labeling rule is to label according to the widest cross section of the current area image ( Figure 4 ), in the local area of ​​step (2.1), record the coordinates of the two points of the widest cross section (X1, Y1, 0, 0), (X2, Y2, 0, 0); the calculation of X1, Y1, X2, Y2 is the same as formula (2-1);

[0051] (3) Design a composite network and train data: After the image is input into the composite network, the upper arm, forearm, thigh, calf, body, and head regions are first detected based on the target detection model; key point detection and regional data analysis are then performed on each sub-region; finally, the data of each region are merged for training.

[0052] (3.1) The composite network structure includes input layer, region detection layer, key point detection layer, data analysis layer, data correction layer, update layer, output layer ( Figure 5 ).

[0053] (3.1.1) Input layer: can input images; if input video, then each frame of the video is input in a loop;

[0054] (3.1.2) Region detection layer: This layer uses the target detection algorithm to detect the upper arm, forearm, thigh, calf, body, head, and card. The target detection algorithm uses YOLO V6.1;

[0055] (3.1.3) Key point detection layer: This layer is based on the region detection results of step (3.1.2) and is fed into each branch of this layer to detect the key points contained in each branch. Each branch has its own weight in the format of step (3.1.2), and its label is the key point of step (2.2) and step (2.3).

[0056] (3.1.4) Data analysis layer: The length and width of the detection results of each area need to be mapped. The mapping rules are:

[0057]

[0058] Where R_h is calculated in the same way as formula (3-2). (x 1 ,y 1 ), (x 2 ,y 2 ) are the coordinates of the key points detected in the area. C_w, C_h are the length and width of the card bounding box detected in step (3.1.2); T_x, T_y are the real mapping lengths in each area.

[0059] (3.1.5) Data correction layer: introduce hyperparameter coefficients to correct each indicator: long arm length = λ 1 *T_x 1 、Arm width = λ 2 *T_y 1 , forearm length = λ 3 *T_x 2 、Forearm width = λ 4 *T_y 2 , thigh length = λ 5 *T_x 3 、Thigh width = λ 6 *T_y 3 , calf length = λ 7 *T_x 4 , calf width = λ 8 *T_y 4 , head length = λ 9 *T_x 5 , body length = λ 10 *T_x 6 , height = head length + torso length + thigh length + forearm length; 1 -λ 10 Corrections are made at this layer. Correction updates are made in the back propagation of this layer.

[0060] (3.1.6) Update layer: This layer records the maximum value of each indicator data of each image frame, and continuously updates the maximum value of each indicator data during the loop.

[0061] (3.1.7) Output layer: At the end of step (3.1.6), the final body index data is output.

[0062] (3.2) Repeat step (3.1) for all the images and video data collected in step (1.3) to train and obtain the optimal model;

[0063] (4) Video stream, upload video, upload image into the composite network to obtain body data: the single image detection result is the final body measurement data; video stream and uploaded video are input into the composite network to obtain the maximum index data in each frame image. (4.1) The user performs real-time video flow, places the card that meets step (1.1) at any position in the camera's field of view, and records a circle around the person being tested, waiting for step (3.1.7) to stop updating data in real time ( Figure 6 ).

[0064] (4.2) Upload the final body data measured by the image ( Figure 7 ).

[0065] The present invention has the characteristics of high precision, low latency and strong robustness. It can be used for body measurement of people with physical disabilities, assisting healthy people to quickly measure body data, intelligent body measurement for clothing merchants, and other product functional applications, and has good commercial prospects.

[0066] Example 2

[0067] The present embodiment relates to a non-standing body measuring device based on real-time key point detection, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a non-standing body measuring method based on real-time key point detection in embodiment 1.

[0068] Example 3

[0069] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a non-standing body measurement method based on real-time key point detection of embodiment 1 is implemented.

[0070] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A non-standing body measurement method based on real-time key point detection, characterized in that: The steps include: (1) Collecting images of non-standing datasets: The person being collected needs to place a card of regular specifications in the camera's field of view. The collector can take a picture of the person being collected or record a circle around the person being collected; (2) Label key points and detection frames: Label joint key points: left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle; label the longest edge key points of the local cross section of the upper arm, forearm, thigh, and calf; label the detection frames: upper arm, forearm, thigh, calf, body, head, and card; (3) Design a composite network and train data: After the image is input into the composite network, the upper arm, forearm, thigh, calf, body, and head regions are first detected based on the target detection model; key point detection and regional data analysis are then performed on each sub-region; finally, the data of each region is merged for training; (4) Video stream, uploaded video, and uploaded image are input into the composite network to obtain body data: the single image detection result is the final body measurement data; the video stream and uploaded video are input into the composite network to obtain the maximum indicator data in each frame image.

2. The non-standing body measurement method based on real-time key point detection according to claim 1, characterized in that: The step (1) of collecting non-standing dataset images includes the following sub-steps: (1.1) The person being collected must have a card with a length of 85.6mm, a width of 54mm, and a thickness of less than 10mm; (1.2) Place the card in step (1.1) within the camera’s field of view; (1.3) Take a photo of the person being collected or record a video around the person being collected.

3. The non-standing body measurement method based on real-time key point detection according to claim 1, characterized in that: The step (2) of marking key points and detection frames includes the following sub-steps: (2.1) For the detection frame: upper arm, forearm, thigh, calf, body, head, card; use the smallest bounding box to encircle the target, the length and width of the bounding box are parallel to the length and width of the image, and record the coordinate information (X, Y, W, H) of the bounding box of each part: Where x, y are the image coordinates of the center of the bounding box; Box_x, Box_y are the length and width of the bounding box respectively; image_x and image_y are the length and width of the image respectively; (2.2) For the key joint points: left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle; the marking rules are marked according to the medical human structure rules. In the local area of ​​step (2.1), record the coordinates of each joint point in the local area (X, Y, 0, 0); the calculation of X, Y is the same as formula (2-1); (2.3) The key points of the longest edge of the local cross section: upper arm, forearm, thigh, calf; the labeling rule is to label according to the widest cross section of the current area image. In the local area of ​​step (2.1), record the coordinates of the two points of the widest cross section (X1, Y1, 0, 0), (X2, Y2, 0, 0); the calculation of X1, Y1, X2, Y2 is the same as formula (2-1).

4. The non-standing body measurement method based on real-time key point detection according to claim 1, characterized in that: The step (3) of combining the network and training the data includes the following sub-steps: (3.1) The composite network structure includes input layer, region detection layer, key point detection layer, data analysis layer, data correction layer, update layer, and output layer; (3.1.1) Input layer: can input images; if input video, then each frame of the video is input in a loop; (3.1.2) Region detection layer: This layer uses a target detection algorithm to detect upper arms, lower arms, thighs, lower legs, body, head, and cards. The target detection algorithm includes but is not limited to the YOLO series and the RCNN series. Its label is the detection box of step (2.1); (3.1.3) Key point detection layer: This layer is based on the regional detection results of step (3.1.2) and is sent to each branch of this layer to detect the key points contained in each branch; each branch has its own weight in the format of step (3.1.2), and its label is the key point of step (2.2) and step (2.3); (3.1.4) Data analysis layer: The length and width of the detection results of each area need to be mapped. The mapping rules are: Where R_h is calculated the same as formula (3-2). (x1, y1), (x2, y2) are the coordinates of the key points detected in the area. C_w, C_h are the length and width of the card bounding box detected in step (3.1.2); T_x, T_y are the real mapping lengths in each area; (3.1.5) Data correction layer: introduce hyperparameter coefficients to correct each indicator: upper arm length = λ1*T_x1, upper arm width = λ2*T_y1, forearm length = λ3*T_x2, forearm width = λ4*T_y2, thigh length = λ5*T_x3, thigh width = λ6*T_y3, calf length = λ7*T_x4, calf width = λ8*T_y4, head length = λ9*T_x5, body length = λ 10 *T_x6, height = head length + torso length + thigh length + forearm length; λ1-λ 10 Correction is performed at this layer. Correction updates are back-propagated at this layer; (3.1.6) Update layer: This layer records the maximum value of each indicator data of each image frame, and continuously updates the maximum value of each indicator data during the loop; (3.1.7) Output layer: At the end of step (3.1.6), the final body index data is output; (3.2) Repeat step (3.1) for all the images and video data collected in step (1.3) to train and obtain the optimal model.

5. The non-standing body measurement method based on real-time key point detection according to claim 1, characterized in that: The step (4) of streaming video, uploading video, and uploading images into the composite network includes the following sub-steps: (4.1) The user conducts real-time video flow. The user places the card that meets step (1.1) at any position within the camera's field of view, and records a circle around the person being tested, waiting for step (3.1.7) to stop updating data in real time. Or stop in advance at any time; (4.2) To upload videos and images, you must follow the shooting rules of step (4.1) and wait for the final body data measured in step (3.1.7).

6. A non-standing body measurement device based on real-time key point detection, characterized in that: The method comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, the method is used to implement a non-standing body measurement method based on real-time key point detection as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a non-standing body measurement method based on real-time key point detection as described in any one of claims 1 to 5 is implemented.

Images