A non-standing body measurement method and device based on real-time key point detection

Abstract

A non-standing body measurement method and device based on real-time key point detection, the method comprising: (1) collecting non-standing data set images; (2) marking joint key points: left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle; marking detection frames: upper arm, forearm, thigh, calf, body trunk, head; marking the longest edge key points of the local cross section: upper arm, forearm, thigh, calf; (3) designing a composite network, merging all key points and detection frames into network training; (4) the composite network supports three methods of video real-time detection, video upload, and picture upload to obtain body data results. The present invention has the characteristics of high precision, low latency, and strong robustness, and can be used for body measurement of people with physical disabilities, assisting healthy people in rapid body data measurement, and intelligent body measurement of clothing merchants, etc., and has good commercial prospects.

Description

Technical Field

[0001] The present invention relates to the field of real-time target detection using computer vision, and in particular to a non-standing body measurement method and device based on real-time key point detection. Background Art

[0002] Traditional body measurement methods usually rely on door-to-door services or family members to assist in measuring using measuring tools. This process is not only cumbersome, but also has problems such as high service fees and lack of measuring tools. Current body measurement methods are mainly suitable for standing healthy people to measure in a fixed position, and only support photo measurement, which makes it difficult for people with physical disabilities to obtain accurate body data. With the popularization of smart mobile devices, it is particularly important to develop a method or device that can be integrated into devices such as mobile phones and tablets, which has the ability to accurately measure the body data of people with physical disabilities and simplifies the user operation steps to help people with physical disabilities achieve barrier-free body measurement. Summary of the Invention

[0003] The present invention aims to overcome the above-mentioned deficiencies in the prior art and provides a non-standing body measurement method and device based on real-time key point detection. The designed composite network is used to measure the body data of the person being measured from the whole to the part and then to the whole.

[0004] A first aspect of the present invention relates to a non-standing body measurement method based on real-time key point detection, comprising the following steps:

[0005] (1) Collecting images of non-standing datasets: The person being collected needs to place a regular-sized card in the camera's field of view. The collector can take a picture of the person being collected or record a video around the person;

[0006] (1.1) The person being collected must have a card with a length of 85.6mm, a width of 54mm, and a thickness of less than 10mm;

[0007] (1.2) Place the card from step (1.1) within the camera’s field of view;

[0008] (1.3) Taking photos of the person being collected or recording a video around the person being collected;

[0009] (2) Mark key points and detection frames: Mark joint key points: left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle; mark the longest edge key points of the local cross section of the upper arm, forearm, thigh, and calf; mark the detection frames: upper arm, forearm, thigh, calf, body, head, and card;

[0010] (2.1) For the detection frames: upper arm, forearm, thigh, calf, torso, head, and card, use the smallest bounding box possible to enclose the target, with the length and width of the bounding box parallel to the length and width of the image. Record the coordinates (X, Y, W, H) of each bounding box.

[0011]

[0012] Where x and y are the image coordinates of the center of the bounding box; Box_x and Box_y are the length and width of the bounding box respectively; image_x and image_y are the length and width of the image respectively.

[0013] (2.2) For the key joint points: left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle; the annotation rules are based on the medical human body structure rules. In the local area of ​​step (2.1), record the coordinates of each joint point in the local area (X, Y, 0, 0); the calculation of X, Y is the same as formula (2-1);

[0014] (2.3) Key points of the longest edge of the local cross section: upper arm, forearm, thigh, and calf. Labeling rules: Label according to the widest cross section of the current area image. In the local area of ​​step (2.1), record the coordinates of the two points of the widest cross section (X1, Y1, 0, 0) and (X2, Y2, 0, 0). The calculation of X1, Y1, X2, and Y2 is the same as formula (2-1).

[0015] (3) Design a composite network and train data: After the image is input into the composite network, the upper arm, forearm, thigh, calf, body torso, and head regions are first detected based on the target detection model; key point detection and regional data analysis are then performed on each sub-region; finally, the data of each region are merged for training.

[0016] (3.1) The composite network structure includes input layer, region detection layer, key point detection layer, data analysis layer, data correction layer, update layer, and output layer.

[0017] (3.1.1) Input layer: can input images; if input is a video, it will loop and input each frame of the video in sequence;

[0018] (3.1.2) Region Detection Layer: This layer uses target detection algorithms to detect upper arms, lower arms, thighs, lower legs, body parts, heads, and cards. Target detection algorithms include but are not limited to YOLO and RCNN. The labels are the detection boxes from step (2.1).

[0019] (3.1.3) Keypoint Detection Layer: This layer feeds the region detection results from step (3.1.2) into each branch of the layer to detect the keypoints contained in each branch. Each branch has its own weight in the format of step (3.1.2), and its label is the keypoint of step (2.2) and step (2.3).

[0020] (3.1.4) Data analysis layer: The length and width of the detection results of each area need to be mapped. The mapping rules are as follows:

[0021]

[0022] R_h is calculated the same way as in formula (3-2). (x1, y1) and (x2, y2) are the coordinates of the key points detected in the region. C_w and C_h are the length and width of the card's bounding box detected in step (3.1.2); T_x and T_y are the actual mapping lengths within each region.

[0023] (3.1.5) Data correction layer: introduce hyperparameter coefficients to correct each indicator: upper arm length = λ1*T_x1, upper arm width = λ2*T_y1, forearm length = λ3*T_x2, forearm width = λ4*T_y2, thigh length = λ5*T_x3, thigh width = λ6*T_y3, calf length = λ7*T_x4, calf width = λ8*T_y4, head length = λ9*T_x5, trunk length = λ 10 *T_x6, height = head length + torso length + thigh length + forearm length; λ1-λ 10 Corrections are made at this layer. Correction updates are made at this layer through back propagation.

[0024] (3.1.6) Update layer: This layer records the maximum value of each indicator data of each image frame and continuously updates the maximum value of each indicator data during the loop.

[0025] (3.1.7) Output layer: At the end of step (3.1.6), the final body index data is output.

[0026] (3.2) Repeat step (3.1) for all the images and video data collected in step (1.3) to train and obtain the optimal model;

[0027] (4) Video streams, uploaded videos, and uploaded images are fed into a composite network to obtain body data: The result of a single image test is the final body measurement data; the video stream and uploaded video are both input into the composite network, and the maximum indicator data in each frame of the image will be obtained. (4.1) The user performs real-time video flow. The user places the card that meets the requirements of step (1.1) at any position within the camera's field of view, and records a circle around the person being tested. Wait until step (3.1.7) stops updating data in real time. Or stop at any time.

[0028] (4.2) Uploading videos and images must follow the shooting rules of step (4.1) and wait for the final body data measured in step (3.1.7).

[0029] The second aspect of the present invention relates to a non-standing body measurement device based on real-time key point detection, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a non-standing body measurement method based on real-time key point detection of the present invention.

[0030] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a non-standing body measurement method based on real-time key point detection according to the present invention.

[0031] The beneficial effect of the present invention is that the real-time physical data measurement of the person being tested can be measured through a composite network, which can greatly improve the robustness of the physical data measurement. Since cards of commonly used specifications are used as mapping references, the present invention can measure the person being measured at various angles without being limited to the shooting position, thereby improving the measurement accuracy. At the same time, the data update layer in the composite network can reduce the mapping error generated by the camera and the environment, thereby improving the measurement accuracy of the present invention. The present invention supports three methods: video stream detection, uploaded video detection, and uploaded picture detection. This will greatly reduce the threshold and operational difficulty of physical data measurement for people with physical disabilities. The effect of the present invention is better than the existing method, and has the characteristics of versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the steps of using the method of the present invention;

[0033] Figure 2 It is the bounding box annotation diagram of the target by the method of the present invention;

[0034] Figure 3 It is a joint key point labeling diagram of the target by the method of the present invention;

[0035] Figure 4 It is a key point marking map of the target area by the method of the present invention;

[0036] Figure 5 It is a composite network structure diagram of the method of the present invention;

[0037] Figure 6 It is a video update body data map of the method of the present invention;

[0038] Figure 7 The method of the present invention is to obtain a body data map by uploading an image. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0040] Example 1

[0041] The core technology of the present invention is to use a composite network to perform real-time detection on the measured person from the whole to the part and then to the whole.

[0042] This embodiment proposes a non-standing body measurement method based on real-time key point detection, including the following steps: Figure 1 ): (1) Collecting non-standing dataset images: The person being collected needs to place a regular-sized card in the camera's field of view, and the collector can take a picture of the person being collected or record a circle around the person being collected;

[0043] (1.1) The person being collected has a card with a length of 85.6mm, a width of 54mm, and a thickness of 0.9mm;

[0044] (1.2) Place the card from step (1.1) within the camera’s field of view;

[0045] (1.3) Taking photos of the person being collected or recording a video around the person being collected;

[0046] (2) Mark key points and detection frames: Mark joint key points: left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle; mark the longest edge key points of the local cross section of the upper arm, forearm, thigh, and calf; mark the detection frames: upper arm, forearm, thigh, calf, body, head, and card;

[0047] (2.1) For the detection frame: upper arm, lower arm, thigh, calf, body, head, card; use the smallest bounding box to encircle the target ( Figure 2 ), the length and width of the bounding box are parallel to the length and width of the image. Record the coordinate information (X, Y, W, H) of the bounding box of each part.

[0048]

[0049] Where x and y are the image coordinates of the center of the bounding box; Box_x and Box_y are the length and width of the bounding box respectively; image_x and image_y are the length and width of the image respectively.

[0050] (2.2) For the key points of joints: left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle; the annotation rules are based on the medical human body structure rules. In the local area of ​​step (2.1), record the coordinates of each joint point in the local area (X, Y, 0, 0) ( Figure 3); The calculation of X, Y is the same as formula (2-1); (2.3) The key points of the longest edge of the local cross section: upper arm, lower arm, thigh, and lower leg; the annotation rule is to mark according to the widest cross section of the current area image ( Figure 4 ), in the local area of ​​step (2.1), record the coordinates of the two points of the widest cross section (X1, Y1, 0, 0) and (X2, Y2, 0, 0); the calculation of X1, Y1, X2, Y2 is the same as formula (2-1);

[0051] (3) Design a composite network and train data: After the image is input into the composite network, the upper arm, forearm, thigh, calf, body torso, and head regions are first detected based on the target detection model; key point detection and regional data analysis are then performed on each sub-region; finally, the data of each region are merged for training.

[0052] (3.1) The composite network structure includes input layer, region detection layer, key point detection layer, data analysis layer, data correction layer, update layer, output layer ( Figure 5 ).

[0053] (3.1.1) Input layer: can input images; if input is a video, it will loop and input each frame of the video in sequence;

[0054] (3.1.2) Region Detection Layer: This layer uses the target detection algorithm to detect the upper arm, forearm, thigh, calf, body, head, and card. The target detection algorithm uses YOLO V6.1;

[0055] (3.1.3) Keypoint Detection Layer: This layer feeds the region detection results from step (3.1.2) into each branch of the layer to detect the keypoints contained in each branch. Each branch has its own weight in the format of step (3.1.2), and its label is the keypoint of step (2.2) and step (2.3).

[0056] (3.1.4) Data analysis layer: The length and width of the detection results of each area need to be mapped. The mapping rules are as follows:

[0057]

[0058] R_h is calculated the same way as in formula (3-2). (x1, y1) and (x2, y2) are the coordinates of the key points detected in the region. C_w and C_h are the length and width of the card's bounding box detected in step (3.1.2); T_x and T_y are the actual mapping lengths within each region.

[0059] (3.1.5) Data correction layer: introduce hyperparameter coefficients to correct each indicator: upper arm length = λ1*T_x1, upper arm width = λ2*T_y1, forearm length = λ3*T_x2, forearm width = λ4*T_y2, thigh length = λ5*T_x3, thigh width = λ6*T_y3, calf length = λ7*T_x4, calf width = λ8*T_y4, head length = λ9*T_x5, trunk length = λ 10 *T_x6, height = head length + torso length + thigh length + forearm length; λ1-λ 10 Corrections are made at this layer. Correction updates are made at this layer through back propagation.

[0060] (3.1.6) Update layer: This layer records the maximum value of each indicator data of each image frame and continuously updates the maximum value of each indicator data during the loop.

[0061] (3.1.7) Output layer: At the end of step (3.1.6), the final body index data is output.

[0062] (3.2) Repeat step (3.1) for all the images and video data collected in step (1.3) to train and obtain the optimal model;

[0063] (4) Video stream, uploaded video, uploaded image into the composite network to obtain body data: the result of a single image test is the final body measurement data; the video stream and uploaded video are input into the composite network, and the maximum index data in each frame image will be obtained. (4.1) The user performs real-time video flow, places the card that meets step (1.1) at any position within the camera's field of view, and records a circle around the person being tested, waiting for step (3.1.7) to stop updating data in real time ( Figure 6 ).

[0064] (4.2) Upload the final body data measured by image ( Figure 7 ).

[0065] The present invention has the characteristics of high precision, low latency and strong robustness. It can be used for body measurement of people with physical disabilities, assisting healthy people in quickly measuring body data, and intelligent body measurement for clothing merchants. It has good commercial prospects.

[0066] Example 2

[0067] This embodiment relates to a non-standing body measurement device based on real-time key point detection, including a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement a non-standing body measurement method based on real-time key point detection in Example 1.

[0068] Example 3

[0069] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a non-standing body measurement method based on real-time key point detection according to embodiment 1 is implemented.

[0070] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A non-standing body measurement method based on real-time key point detection, characterized in that: The steps include: (1) Collecting images of non-standing datasets: The person being collected needs to place a regular-sized card in the camera's field of view. The collector can take a picture of the person being collected or record a video around the person; (2) Mark key points and detection frames: Mark joint key points: left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle; mark the longest edge key points of the local cross section of the upper arm, forearm, thigh, and calf; mark the detection frames: upper arm, forearm, thigh, calf, body, head, card; specifically including: (2.1) For the detection frames: upper arm, forearm, thigh, calf, body, head, card; use the smallest bounding box to encircle the target, and the length and width of the bounding box are parallel to the length and width of the image. Record the coordinate information (X, Y, W, H) of the bounding box of each part: Where x, y are the image coordinates of the center of the bounding box; Box_x, Box_y are the length and width of the bounding box respectively; image_x and image_y are the length and width of the image respectively; (2.2) For the key joint points: left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle; the annotation rules are based on the medical human body structure rules. In the local area of ​​step (2.1), record the coordinates of each joint point in the local area (X, Y, 0, 0); the calculation of X, Y is the same as formula (2-1); (2.3) Key points of the longest edge of the local cross section: upper arm, forearm, thigh, and calf. Labeling rules: Label according to the widest cross section of the current area image. In the local area of ​​step (2.1), record the coordinates of the two points of the widest cross section (X1, Y1, 0, 0) and (X2, Y2, 0, 0). The calculation of X1, Y1, X2, and Y2 is the same as formula (2-1). (3) Design a composite network and train data: After the image is input into the composite network, the upper arm, forearm, thigh, calf, body, and head regions are first detected based on the target detection model; key point detection and regional data analysis are then performed on each sub-region; finally, the data of each region are merged for training; specifically, (3.1) The composite network structure includes input layer, region detection layer, key point detection layer, data analysis layer, data correction layer, update layer, and output layer; (3.1.1) Input layer: can input images; if input is a video, it will loop and input each frame of the video in sequence; (3.1.2) Region Detection Layer: This layer uses target detection algorithms to detect upper arms, lower arms, thighs, lower legs, body parts, heads, and cards. Target detection algorithms include but are not limited to YOLO and RCNN. The labels are the detection boxes from step (2.1). (3.1.3) Keypoint detection layer: This layer is based on the region detection results of step (3.1.2) and is fed into each branch of this layer to detect the keypoints contained in each branch; each branch has its own weight in the format of step (3.1.2), and its label is the keypoint of step (2.2) and step (2.3); (3.1.4) Data analysis layer: The length and width of the detection results of each area need to be mapped. The mapping rules are as follows: Where R_h is calculated the same as formula (3-2); (x1, y1), (x2, y2) are the coordinates of the key points detected in the area; C_w, C_h are the length and width of the card bounding box detected in step (3.1.2); T_x, T_y are the actual mapping lengths in each area; (3.1.5) Data correction layer: introduce hyperparameter coefficients to correct each indicator: upper arm length = λ1*T_x1, upper arm width = λ2*T_y1, forearm length = λ3*T_x2, forearm width = λ4*T_y2, thigh length = λ5*T_x3, thigh width = λ6*T_y3, calf length = λ7*T_x4, calf width = λ8*T_y4, head length = λ9*T_x5, trunk length = λ 10 *T_x6, height = head length + torso length + thigh length + forearm length; λ1-λ 10 Correction is performed at this layer; correction updates are performed in reverse propagation at this layer; (3.1.6) Update layer: This layer records the maximum value of each indicator data in each image frame and continuously updates the maximum value of each indicator data during the loop; (3.1.7) Output layer: At the end of step (3.1.6), the final body index data is output; (3.2) Repeat step (3.1) for all the images and video data collected in step (1.3) to train and obtain the optimal model; (4) Video stream, uploaded video, and uploaded image are fed into the composite network to obtain body data: the result of a single image detection is the final body measurement data; the video stream and uploaded video are input into the composite network to obtain the maximum index data in each frame image.

2. The non-standing body measurement method based on real-time key point detection according to claim 1, characterized in that: The step (1) of collecting non-standing dataset images includes the following sub-steps: (1.1) The person being collected must have a card with a length of 85.6mm, a width of 54mm, and a thickness of less than 10mm; (1.2) Place the card from step (1.1) within the camera’s field of view; (1.3) Take a photo of the person being collected or record a video around the person being collected.

3. The non-standing body measurement method based on real-time key point detection according to claim 1, characterized in that: The step (4) of streaming video, uploading video, and uploading images into the composite network includes the following sub-steps: (4.1) The user performs real-time video streaming. The user places the card that meets the requirements of step (1.1) at any position within the camera's field of view, and records a circle around the person being tested. The user waits until step (3.1.7) stops updating data in real time, or stops the recording at any time. (4.2) Uploading videos and images must follow the shooting rules of step (4.1) and wait for the final body data measured in step (3.1.7).

4. A non-standing body measurement device based on real-time key point detection, characterized in that: The method comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the method is used to implement a non-standing body measurement method based on real-time key point detection according to any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, a non-standing body measurement method based on real-time key point detection according to any one of claims 1 to 3 is implemented.

Images