A method for continuous human posture estimation based on key point motion estimation

Abstract

The present invention discloses a method for estimating continuous human posture based on key point motion estimation, which applies a motion estimation block matching algorithm to human key point tracking to obtain a continuous human posture result, and uses a human posture estimator based on a deep neural network to continuously correct the result. The present invention can realize the estimation of continuous human posture in a video stream, wherein the human posture is specifically reflected in the coordinate position of the key points of the human body in the video frame. Compared with the posture estimation method that completely relies on a deep neural network, the posture estimation method proposed by the present invention has the advantages of high frame rate, low hardware requirements, and the recognition result has the advantages of time series continuity; compared with the posture estimation method that completely relies on the motion estimation algorithm, the present invention can correct the accumulated error and improve the estimation accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, and in particular to a method for estimating continuous posture of a human body based on key point motion estimation. Background Art

[0002] In recent years, human posture estimation in images / videos is generally achieved using deep neural networks. However, deep neural networks have high requirements for computer hardware and consume a lot of computing resources. Although the calculation results are highly accurate, the calculation speed is slow and the real-time performance is poor. These defects limit the application of human posture estimation and make it more difficult to promote. In addition, using deep neural networks to estimate human posture in videos essentially splits the video into each frame and calculates it separately. It does not take advantage of the prior knowledge that the human posture information between each frame in the video is continuous. This results in the information of each frame being independent and fragmented, and prone to jumps.

[0003] Based on the above deep neural network to extract human posture scheme, there are the following references:

[0004] (1) In the invention patent with authorization announcement number CN110334607B and invention name “A method and system for identifying human interactive behavior in videos”, a YOLOv3 network is used for pedestrian detection. The YOLOv3 network is a convolutional neural network.

[0005] (2) In the invention patent application with publication number CN112686097A and invention name “A method for estimating the posture of key points in human body images”, a deep neural network model capable of estimating the posture of human body images to obtain the human body firmware key point images is obtained through training.

[0006] (3) In the invention patent application with publication number CN110638461A and invention name “A method and system for recognizing human posture on an electric hospital bed”, the Stacked Hourglass algorithm is used to recognize human posture. Stacked Hourglass is a deep neural network.

[0007] (4) In the invention patent application with publication number CN110197123A and invention name “A human posture recognition method based on Mask R-CNN”, the Mask R-CNN network is used to extract posture features. The Mask R-CNN network is a deep neural network.

[0008] (5) In the invention patent application with publication number CN109919122A and invention name “A method for detecting temporal behavior based on 3D human key points”, a multi-layer CNN network is used to extract the bounding box of a person target, and the coordinates of 2D key points are obtained by body part positioning and correlation analysis, and then a key point regression network is constructed to achieve the mapping of 2D key points to 3D key points. Both the CNN and the key point regression network are deep neural networks.

[0009] In the broad field of image processing and video coding, there are industry-recognized coding standards and protocols such as H.264 and H.265. In these standards, the ultimate goal of motion estimation is to achieve compression of continuous picture frames in the video. It is achieved by dividing the entire picture into many small areas (macroblocks) and then searching and estimating the most similar areas of these macroblocks. The above-mentioned motion estimation algorithm based on block matching is called block matching algorithm. Summary of the invention

[0010] In view of the above-mentioned shortcomings in the prior art of using deep neural networks for human posture estimation, the present invention proposes a human body continuous posture estimation algorithm that integrates a deep neural network human body posture estimation algorithm and a block matching motion estimation algorithm. The proposed algorithm can give full play to the advantages of the two technical routes, while avoiding and making up for their disadvantages, and realize fast and accurate human body continuous posture estimation.

[0011] A method for estimating continuous posture of a human body based on key point motion estimation, wherein the system for estimating includes two estimators,

[0012] Estimator 1: a pre-trained deep neural network pose estimator,

[0013] Estimator 2: Motion estimator based on the H.264 video coding standard,

[0014] The human body continuous posture estimation method based on key point motion estimation includes three stages:

[0015] Phase 1: After the system starts running, the first frame of the video frame is used as the key frame I0, and the human body posture is recognized by using the estimator 1 to obtain the initial coordinates of the human body key points; during the operation of the estimator 1, the system continues to collect multiple frames of video frames and temporarily stores them in the computer memory queue; after the operation of the estimator 1 is completed, the key point coordinate data obtained and all the temporarily stored video frames are passed to the estimator 2 for sequential operation to obtain the human body key point estimation results of each video frame;

[0016] Phase 2: After the calculation of estimator 1 in the first phase is completed, the system immediately collects the next video frame I1 and uses estimator 1 and estimator 2 to calculate it at the same time. When estimator 1 has not yet completed the calculation, estimator 2 continues to perform real-time calculations on the new video frames collected subsequently.

[0017] The third stage: after the operation of estimator 1 in the second stage is completed, compare the estimation results of the key points in the video frame I1 by estimator 1 and estimator 2. If the sum of the coordinate errors of all the key points of the two is less than the set threshold ε, repeat the steps of the second stage for the subsequent video frames; if the errors of the two are greater than the threshold, the estimation results of the key points of the video frame I1 by estimator 1 shall prevail, and estimator 2 shall be used to update the video frame key point estimation results obtained by estimator 2 during the operation of estimator 1. After these results are updated, repeat the steps of the second stage for the subsequent video frames.

[0018] The value of the threshold ε in the third stage can be set as needed, and there is no unified standard.

[0019] Preferably, the estimator 1 uses a VNect, DeepPose, Stacked Hourglass or RMPE neural network model trained by the MPI-INF-3DHP dataset.

[0020] Preferably, the algorithm in the estimator 2 is an algorithm that uses a block matching algorithm to detect changes in the coordinates of key points. In the block matching algorithm, it is necessary to find the block most similar to the macroblock to be matched in the previous frame in the current frame based on a given matching criterion. The macroblock is a small rectangular area centered on the selected key point. The range of block matching is called the search window, which is a larger rectangular area centered on the selected key point. The macroblock with the smallest error with the macroblock to be matched in the search window is taken as the matching result.

[0021] More preferably, the matching criterion in the block matching algorithm uses the minimum mean square error (MSE), the minimum mean absolute error (MAD) or the minimum sum of differences (SAD) criterion,

[0022] The definitions are as follows:

[0023]

[0024]

[0025]

[0026] Where p represents the pixel point in the macroblock B to be matched, v represents the motion vector corresponding to the two macroblocks being matched, f(a) represents the pixel value at position a in the current video frame, and f last(a) represents the pixel value at position a in the previous video frame, that is, if f(p+v) represents the pixel value at position p+v in the current video frame, f last (p) represents the pixel value at position p in the previous video frame.

[0027] More preferably, after determining the matching criteria, it is also necessary to match the actual macroblocks, and when the block matching algorithm selects the macroblocks to be matched in the current frame, the search template is used to selectively determine the macroblocks to be selected. Further preferably, when the search template is used to selectively determine the macroblocks to be selected, the motion estimation search algorithm used is a three-step search method, a diamond search method, or a four-step search method.

[0028] Further preferably, the three-step search method steps are as follows:

[0029] Step 1: Set a larger area in the video frame that completely contains the macroblock as the search window, and the center of the macroblock as the center point of the search window. Take the center point as the search starting point, and start the search step from a value equal to or slightly larger than half of the search window radius. In each step of the search, calculate the matching index of the center point of the current square and the eight points around it, a total of nine points, and select the point with the best index as the center point of the next search;

[0030] Step 2: With the point obtained in the previous step as the center, reduce the current search step to half of the previous search step, and then perform a similar search to obtain the best matching point;

[0031] The third step is to continue reducing the search step size to half of the previous one and search for the best matching position.

[0032] Further preferably, the diamond search method has two different matching templates, a large diamond and a small diamond. The large diamond has 9 search points, and the small diamond has only 5 search points. First, a large diamond search template with a larger step size is used for a rough search, and then a small diamond template is used for a fine search. The search steps of the diamond search method are as follows:

[0033] Step 1: Using the diamond as a template, calculate the matching index values ​​of the center point and the eight points around it, a total of nine points, and compare to get the point with the best index;

[0034] Step 2: If the center point of the search is the point with the best index, jump to step 3 and use the small diamond search template, otherwise return to the search in step 1;

[0035] Step 3: Use a small diamond search template with only 5 search points to calculate the matching index values ​​of these 5 points, and take the point with the best index as the best matching point.

[0036] Further preferably, the four-step search method steps are as follows:

[0037] Step 1: The initial setting is the same as the three-step search method. The matching index values ​​of the nine selected points centered on the key point are calculated. If the point with the best index is the center point, jump to step 4, otherwise go to step 2;

[0038] Step 2: Take the pixel point searched in the previous step as the center point, keep the interval of the selected points unchanged, and continue to calculate the matching index values ​​of the newly obtained nine points. If the center point is the point with the best index, jump to step 4, otherwise go to step 3;

[0039] Step 3: If the searched center point reaches the edge of the search window, the interval between selected points must be reduced to 3×3, and then go to step 4, otherwise repeat step 2;

[0040] Step 4: Reduce the interval of selected points to 3×3, search, and obtain the best matching point.

[0041] Preferably, when defining the key points of the human body, a total of 21 key points are defined, which are:

[0042]

[0043]

[0044] The basic idea of ​​the present invention is to apply the motion estimation block matching algorithm to human key point tracking, thereby obtaining the continuous posture results of the human body, and using the human posture estimator based on the deep neural network to continuously correct the results. For human key point tracking, it is only necessary to divide the corresponding number of macroblocks with the selected key point to be tracked as the center, without dividing the entire picture into multiple macroblocks as in picture compression, and then searching for the changes of macroblocks in the two adjacent picture frames according to a certain strategy to find the best estimation result of the motion of the macroblock.

[0045] The present invention can estimate the continuous posture of a human body in a video stream, wherein the human body posture is specifically reflected as the coordinate position of the key points of the human body in the video frame. Compared with the posture estimation method that completely relies on a deep neural network, the posture estimation method proposed by the present invention has the advantages of high frame rate, low hardware requirements, and the recognition result has the characteristics of time series continuity; compared with the posture estimation method that completely relies on the motion estimation algorithm, the present invention can correct the accumulated error and improve the estimation accuracy.

[0046] The video stream processed by the technical solution of the present application can be from reading the video stored in the hard disk, or from the real-time video captured by the camera. Moreover, when processing the real-time video captured by the camera, the advantages of the present invention can be more prominent due to the high requirements for real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1It is a schematic diagram of the positions of key points of the human body in the present invention.

[0048] Figure 2 The present invention is a flow chart of a method for estimating continuous posture of a human body based on key point motion estimation.

[0049] Figure 3 Schematic diagram of the motion estimation block matching algorithm.

[0050] Figure 4 Schematic diagrams of three classic motion estimation algorithms, including (a) three-step search, (b) four-step search, and (c) diamond search. DETAILED DESCRIPTION

[0051] The key points of the human body tracked in the embodiment are defined as follows. A total of 21 key points are defined, and the name and number of each key point are shown in Table 1 below. The position of each key point is as follows: Figure 1 shown.

[0052] Table 1

[0053]

[0054]

[0055] The flowchart of the method for estimating continuous posture of human body based on key point motion estimation of the present invention is as follows: Figure 2 shown.

[0056] The core part of the algorithm is divided into two modules: a pre-trained deep neural network pose estimator, namely, estimator (1), and a motion estimator based on the video coding H.264 standard, namely, estimator (2).

[0057] For the estimator (1), the VNect neural network model trained by the MPI-INF-3DHP dataset is used in the embodiment (other feasible network models include DeepPose, Stacked Hourglass, RMPE, etc.). The frame rate of the estimator (1) can reach about 30 Hz in the environment of Intel Core i5-8400 CPU and NVIDIA GeForce GTX 1060 6GB GPU, and the average coordinate error is 82.5 mm.

[0058] As for estimator (2), estimator (2) is an algorithm that uses a block matching algorithm to detect changes in key point coordinates.

[0059] For block matching algorithms, such as Figure 3As shown in the figure, in the block matching algorithm, it is necessary to find the block that is most similar to the macroblock to be matched in the previous frame in the current frame based on the given matching criteria. A macroblock is a small rectangular area centered on a selected key point. The range of block matching is called a search window, which is a larger rectangular area centered on a selected key point. The macroblock with the smallest error with the macroblock to be matched in the search window is taken as the matching result.

[0060] The commonly used matching criteria of block matching algorithms include minimum mean square error (MSE), minimum mean absolute error (MAD), minimum difference sum (SAD) criteria, etc. Their definitions are as follows:

[0061]

[0062]

[0063]

[0064] Where p represents the pixel point in the macroblock B to be matched, v represents the motion vector (relative position) corresponding to the two macroblocks being matched. f(a) represents the pixel value at position a in the current video frame, f last (a) represents the pixel value at position a in the previous video frame. Among the above matching criteria, the SAD criterion is the most widely used.

[0065] After determining the matching criteria, the actual macroblocks need to be matched. When the block matching algorithm selects the macroblock to be matched in the current frame, if all macroblocks in the area are matched in sequence, a global best matching macroblock in the area can be found eventually. However, the amount of calculation required to do so is too large and does not meet the lightweight requirements. This method is rarely used in the field of video coding. In contrast, a variety of search templates can be used to selectively determine the macroblocks to be selected. The more classic motion estimation search algorithms include the three-step search method, the diamond search method, and the four-step search method. Figure 4 (Image source: Yang Xiaoyu, "Optimization of Motion Estimation Algorithm Based on HEVC Video Coding," Master's Degree, Nanjing University of Posts and Telecommunications, 2018.) Schematic diagrams of three classic motion estimation algorithms, including (a) three-step search, (b) four-step search, and (c) diamond search.

[0066] The three-step search method steps are as follows:

[0067] Step 1: Set a larger area in the image that completely contains the macroblock as the search window, and the center of the macroblock (i.e., the position of the key point) as the center point of the search window. Take the center point as the search starting point, and start the search step (i.e., the center distance between the selected macroblocks to be matched) from being equal to or slightly greater than half of the search window radius. In each step of the search, calculate the matching index (such as SAD) of the center point of the current square and the eight points around it, a total of nine points, and select the point with the best index (such as the smallest SAD value) as the center point of the next search;

[0068] Step 2: With the point obtained in the previous step as the center, reduce the current search step to half of the previous search step, and then perform a similar search to obtain the best matching point.

[0069] The third step is to continue reducing the search step size to half of the previous one and search for the best matching position.

[0070] The four-step search method steps are as follows:

[0071] Step 1: The initial setting is the same as the three-step search method. The matching index values ​​of the nine selected points centered on the key point are calculated. If the point with the best index is the center point, jump to step 4, otherwise go to step 2;

[0072] Step 2: Take the pixel point searched in the previous step as the center point, keep the interval of the selected points unchanged, and continue to calculate the matching index values ​​of the newly obtained nine points. If the center point is the point with the best index, jump to step 4, otherwise go to step 3;

[0073] Step 3: If the searched center point reaches the edge of the search window, the interval between selected points must be reduced to 3×3, and then go to step 4, otherwise repeat step 2;

[0074] Step 4: Reduce the interval of selected points to 3×3, search, and obtain the best matching point.

[0075] Diamond search is also called diamond search. There are two different matching templates: large diamond and small diamond. The large diamond has 9 search points, and the small diamond has only 5 search points. First, use the large diamond search template with a larger step size for a rough search, and then use the small diamond template for a fine search. The search steps of the diamond search method are as follows:

[0076] Step 1: Using the diamond as a template, calculate the matching index values ​​of the center point and the eight points around it, a total of nine points, and compare to get the point with the best index;

[0077] Step 2: If the center point of the search is the point with the best index, jump to step 3 and use the small diamond search template, otherwise return to the search in step 1;

[0078] Step 3: Use a small diamond search template with only 5 search points to calculate the matching index values ​​of these 5 points, and take the point with the best index as the best matching point.

[0079] In the embodiment of the present application, a three-step search method based on the minimum difference and criterion is preferably adopted.

[0080] The estimator (2) is simple to calculate and has a very fast operation speed. In the environment of Intel Core i5-8400 CPU and NVIDIA GeForce GTX 1060 6GB GPU, the frame rate can reach about 50,000 Hz, which far exceeds the frame rate requirement for real-time processing of video streams. However, the estimator may experience serious drift over time, gradually deviating from and losing the tracking target.

[0081] The processed video stream may come from a video stored in a storage device read frame by frame, or may come from a real-time video captured by a camera (in this case, the real-time requirement is higher, which can highlight the advantages of the present invention).

[0082] The algorithm process is divided into the following three stages.

[0083] Phase 1:

[0084] After the program starts running, the first imported video frame is used as the key frame I0, and the human body posture is identified using the estimator (1) to obtain the initial human body key point coordinates. The operation of the estimator (1) takes a long time. During this process, the system will continue to collect multiple video frames and temporarily store them in the computer memory queue. After the operation of the estimator (1) is completed, the key point coordinate data obtained and all the temporarily stored video frames are passed to the estimator (2) for sequential operation to obtain the human body key point estimation results for each video frame. The operation of the estimator (2) takes a very short time and can be considered to be completed in real time.

[0085] Phase 2:

[0086] After the above-mentioned estimator (1) has completed the operation, the system immediately collects the next frame of video frame I1 and uses estimator (1) and estimator (2) to calculate it at the same time. When the estimator (1) has not yet completed the operation, the estimator (2) continues to perform real-time operations on the new video frames that are subsequently collected.

[0087] Phase 3:

[0088] After the operation of the estimator (1) in the second stage is completed, the estimation results of the key points in I1 by the estimator (1) and the estimator (2) are compared. If the sum of the coordinate errors of all the key points of the two is less than the set threshold ε, the steps of the second stage are repeated for the subsequent video frames; if the errors of the two are greater than the threshold, the estimation results of the key points of I1 by the estimator (1) are used as the basis, and the video frame key point estimation results obtained by the estimator (2) during the operation of the estimator (1) are updated again by the estimator (2). After these results are updated, the steps of the second stage are repeated for the subsequent video frames. Similarly, the operation of the estimator (2) takes a very short time and can be considered to be completed in real time.

Claims

1. A method for estimating continuous human posture based on key point motion estimation, characterized in that: The system used to perform the estimation includes two estimators, Estimator 1: a pre-trained deep neural network pose estimator, Estimator 2: Motion estimator based on the H.264 video coding standard, The human body continuous posture estimation method based on key point motion estimation includes three stages: Phase 1: After the system starts running, the first frame of the video frame is used as the key frame I0, and the human posture is recognized by using the estimator 1 to obtain the initial coordinates of the key points of the human body; during the operation of the estimator 1, the system continues to collect multiple frames of video frames and temporarily stores them in the computer memory queue; After the calculation of estimator 1 is completed, the obtained key point coordinate data and all temporarily stored video frames are passed to estimator 2 for sequential calculation to obtain the human body key point estimation results of each video frame; Phase 2: After the calculation of estimator 1 in the first phase is completed, the system immediately collects the next video frame I1 and uses estimator 1 and estimator 2 to calculate it at the same time. When estimator 1 has not yet completed the calculation, estimator 2 continues to perform real-time calculations on the new video frames collected subsequently. Stage 3: After the calculation of estimator 1 in stage 2 is completed, the estimation results of the key points in the video frame I1 by estimator 1 and estimator 2 are compared. If the sum of the coordinate errors of all the key points of the two is less than the set threshold ε, the steps of stage 2 are repeated for the following video frames. If the error between the two is greater than the threshold, the estimation result of the key point of video frame I1 by estimator 1 is used as the basis, and estimator 2 is used to update the video frame key point estimation result obtained by estimator 2 during the operation of estimator 1. After these results are updated, the steps of the second stage are repeated for the subsequent video frames.

2. The method for estimating continuous posture of a human body based on key point motion estimation according to claim 1, characterized in that: Estimator 1 uses VNect, DeepPose, Stacked Hourglass or RMPE neural network models trained with the MPI-INF-3DHP dataset.

3. The method for estimating continuous posture of a human body based on key point motion estimation according to claim 1, characterized in that: The algorithm in estimator 2 is an algorithm that uses a block matching algorithm to detect changes in key point coordinates. In the block matching algorithm, it is necessary to find the block that is most similar to the macroblock to be matched in the previous frame based on a given matching criterion in the current frame. The macroblock is a small rectangular area centered on the selected key point. The range of block matching is called the search window, which is a larger rectangular area centered on the selected key point. The macroblock with the smallest error with the macroblock to be matched in the search window is taken as the matching result.

4. The method for estimating continuous posture of a human body based on key point motion estimation according to claim 3, characterized in that: The matching criteria in the block matching algorithm use the minimum mean square error function MSE, the minimum mean absolute error MAD or the minimum difference and SAD criteria. The definitions are as follows: Where p represents the pixel point in the macroblock B to be matched, v represents the motion vector corresponding to the two macroblocks being matched, f(a) represents the pixel value at position a in the current video frame, and f last (a) represents the pixel value at position a in the previous video frame.

5. The method for estimating continuous posture of a human body based on key point motion estimation according to claim 4, characterized in that: After determining the matching criteria, the actual macroblocks need to be matched. When the block matching algorithm selects the macroblocks to be matched in the current frame, it uses a search template to selectively determine the macroblocks to be selected.

6. The method for estimating continuous posture of a human body based on key point motion estimation according to claim 5, characterized in that: When the search template is used to selectively determine the macroblock to be selected, the motion estimation search algorithm used is a three-step search method, a diamond search method, or a four-step search method.

7. The method for estimating continuous posture of a human body based on key point motion estimation according to claim 6, characterized in that: The three-step search method steps are as follows: Step 1: Set a larger area in the video frame that completely contains the macroblock as the search window, and the center of the macroblock as the center point of the search window. Take the center point as the search starting point, and start the search step from a value equal to or slightly larger than half of the search window radius. In each step of the search, calculate the matching index of the center point of the current square and the eight points around it, a total of nine points, and select the point with the best index as the center point of the next search; Step 2: With the point obtained in the previous step as the center, reduce the current search step to half of the previous search step, and then perform a similar search to obtain the best matching point; The third step is to continue reducing the search step size to half of the previous one and search for the best matching position.

8. The method for estimating continuous posture of a human body based on key point motion estimation according to claim 6, characterized in that: The diamond search method has two different matching templates: large diamond and small diamond. The large diamond has 9 search points, and the small diamond has only 5 search points. First, use the large diamond search template with a larger step size for a rough search, and then use the small diamond template for a fine search. The search steps of the diamond search method are as follows: Step 1: Using the diamond as a template, calculate the matching index values ​​of the center point and the eight points around it, a total of nine points, and compare to get the point with the best index; Step 2: If the center point of the search is the point with the best index, jump to step 3 and use the small diamond search template, otherwise return to the search in step 1; Step 3: Use a small diamond search template with only 5 search points to calculate the matching index values ​​of these 5 points, and take the point with the best index as the best matching point.

9. The method for estimating continuous posture of a human body based on key point motion estimation according to claim 6, characterized in that: The four-step search method steps are as follows: Step 1: The initial setting is the same as the three-step search method. The matching index values ​​of the nine selected points centered on the key point are calculated. If the point with the best index is the center point, jump to step 4, otherwise go to step 2; Step 2: Take the pixel point searched in the previous step as the center point, keep the interval of the selected points unchanged, and continue to calculate the matching index values ​​of the newly obtained nine points. If the center point is the point with the best index, jump to step 4, otherwise go to step 3; Step 3: If the searched center point reaches the edge of the search window, the selected point interval must be reduced to 3×3, then go to step 4, otherwise repeat step 2; Step 4: Reduce the interval of selected points to 3×3, search, and obtain the best matching point.

10. The method for estimating continuous posture of a human body based on key point motion estimation according to claim 1, characterized in that: When defining the key points of the human body, a total of 21 key points are defined, namely:

Images