Human pose estimation method based on progressive filtering fusion
Through two Azure Kinect DK cameras and progressive filtering fusion processing, the problem of high cost and reduced posture data accuracy in traditional methods is solved, and low-cost and high-precision human posture estimation is achieved, which is suitable for scenes where no joint marking is worn.
Patent Information
- Application Number
- CN202311572460.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-23
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2043-11-23
AI Technical Summary
The traditional human posture estimation method is costly and inconvenient, and when the tester crosses his arms or turns sideways, some of the human body joints in the pictures taken by the RGB-D camera are blocked, resulting in a reduced accuracy of the posture data.
Two Azure Kinect DK cameras are used for data acquisition, and the camera data affected by occlusion is filtered using the Mahayana distance, and the camera data affected by occlusion is performed to improve the accuracy of posture estimation, and without wearing the joint mark. Achieve low-cost and high-precision human posture estimation.
Low-cost and high-precision human posture estimation is achieved, which avoids the inconvenience of the tester to wear joint marks, improves the overall accuracy of human posture data, and reduces cumulative position errors.
Smart Images

Figure CN117689713B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a human body posture estimation method, in particular to a human body posture estimation method based on progressive filtering fusion. Background Art
[0002] With the development of machine learning and computer vision, AI fitness coaches have gradually been widely used in the field of fitness. In the development and use of AI fitness coaches, human posture estimation is needed to perceive the tester's fitness posture data. The traditional human posture estimation method is to obtain human posture data through a human posture capture system. Although the human posture capture system can obtain human posture data with high accuracy, it not only requires more than ten professional cameras to shoot, but also has relatively expensive hardware and high cost, and requires the tester to wear many joint point markers, which causes inconvenience to the tester.
[0003] In order to solve the problems of traditional human posture estimation methods, a human posture estimation method has emerged that uses a low-cost RGB-D camera to take a single picture of the test subject, and then estimates the test subject's current three-dimensional posture based on the single picture and detects the corresponding 3D joint point position data (i.e., human posture data). Although the human posture estimation method is low-cost and does not cause inconvenience to the test subject, in the human posture estimation method, when the test subject crosses his arms or turns sideways, some of the human body joints in the picture taken by the RGB-D camera will be blocked, and the blocked joint point position data estimated based on these pictures is not accurate, resulting in a reduction in the overall accuracy of the human posture data. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a human posture estimation method based on progressive filtering fusion which has both low cost and high precision and does not require the tester to wear joint point markers and does not cause inconvenience to the tester.
[0005] The technical solution adopted by the present invention to solve the above technical problems is: a human body posture estimation method based on progressive filtering fusion, comprising the following steps:
[0006] Step 1: Perform initial data collection. The specific process is as follows:
[0007] S1.1. Place two Azure Kinect DK cameras on the test bench, where the two Azure Kinect DK cameras are on the same horizontal plane, the field of view overlap rate reaches more than 80%, and the field of view of the two Azure Kinect DK cameras can fully cover the tester in the center of the test bench;
[0008] S1.2. Ask the tester to stand upright in the center of the test bench, holding a checkerboard calibration plate with a square length of 10 mm and a pattern array of 8*6 in both hands, so that the checkerboard pattern on the front of the 8*6 checkerboard calibration plate can be clearly seen in both Azure Kinect DK cameras;
[0009] S1.3, first start two Azure Kinect DK cameras at the same time to shoot a frame of color image data, then close the two Azure Kinect DK cameras, and then obtain a frame of color image data shot by each of the two Azure Kinect DK cameras, so that two frames of color image data are obtained;
[0010] S1.4, inputting the two frames of color image data into the OpenCV software toolkit, and processing the two frames of color image data to obtain a transformation matrix between the two Azure Kinect DK cameras, and recording the transformation matrix as W;
[0011] S1.5. Take the checkerboard calibration plate away from the tester, and the tester enters the ready state;
[0012] S1.6. The tester starts to perform the test action. At the same time, two Azure Kinect DK cameras are started to record video at the same time. Until the tester completes all the test actions, the video recording of the two Azure Kinect DK cameras is completed, and the two Azure Kinect DK cameras are turned off. In which, each frame of data in the video recorded by each Azure Kinect DK camera includes image data and human skeleton data corresponding to the image data. The human skeleton data is the three-dimensional position vector of 32 human skeleton joints in the coordinate system of the Azure Kinect DK camera. The 32 human skeleton joints are pelvis, spine, spine chest, neck, left clavicle, left shoulder, left elbow, left wrist, left hand, left hand tip, left thumb, right clavicle, right shoulder, right elbow, right wrist, right hand, right hand tip, right thumb, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, head, nose, left eye, left ear, right eye and right ear; in order from 1 to 32 The pelvis, spine, spine chest, neck, left clavicle, left shoulder, left elbow, left wrist, left hand, left hand tip, left thumb, right clavicle, right shoulder, right elbow, right wrist, right hand, right hand tip, right thumb, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, head, nose, left eye, left ear, right eye and right ear are numbered respectively, and the human skeleton joint numbered as l is called l joint point, l = 1, 2, ...., 32; the total number of frames of data in the video recorded by the two Azure Kinect DK cameras is recorded as Y, that is, each Azure Kinect DK camera obtains Y frames of data respectively;
[0013] S1.7. Use any one of the two Azure Kinect DK cameras as the master camera and the other as the slave camera; multiply the three-dimensional position vector of the l-th joint point of the a-th frame data obtained from the slave camera by the transformation matrix W to obtain the mapping vector of the l-th joint point in the main camera coordinate system, which is called the mapping vector of the l-th joint point, and is recorded as The three-dimensional position vector of the l-th joint point of the a-th frame data obtained by the main camera is recorded as
[0014] Step 2: Set the estimated global 3D position vector of the joint point l in the a-th frame data to set up The covariance matrix of The estimated value of the global 3D position vector of the l joint point in the first frame of data To initialize, right The covariance matrix of To initialize, is equal to the 3*3 identity matrix, which is denoted as I; set The measurement noise covariance is make Equal to 3*I; set The measurement noise covariance is make =3*I; Set the arbitrary human skeleton joint point measurement matrix of the a-th frame data to H a , let H a =I, where * is the multiplication symbol;
[0015] Step 3: Perform data filtering and fusion processing. The specific process is as follows:
[0016] S3.1, set the frame number variable to k, initialize k, and set k=2;
[0017] S3.2. The three-dimensional position vector of the l-th joint point of the k-th frame data obtained by the main camera and the mapping vector of the l joint point Processing is performed to obtain the estimated global three-dimensional position vector of the l joint point of the k-th frame data The specific process is:
[0018] S3.2.1. Set the state transfer matrix of the k-th frame data to F k , let F k =I; Set the process noise covariance matrix of the k-th frame data to Q k , let Q k =0.2*I; Set the global three-dimensional position vector prediction value of the l-th joint point of the k-th frame data to set up The covariance matrix of
[0019] S3.2.2, using formula (1) and (2) to calculate and
[0020]
[0021]
[0022] In formula (2), the subscript T represents the transpose symbol of the matrix;
[0023] S3.2.3. Setting and The square of the Mahalanobis distance is Using formula (3) to calculate
[0024]
[0025] In formula (3),
[0026] S3.2.4. Setting and The square of the Mahalanobis distance is set up and The square of the Mahalanobis distance is Using formula (4) and (5) respectively, we can get and
[0027]
[0028]
[0029] In formula (4), In formula (5),
[0030] S3.2.5. Set the confidence threshold of the l joint point to χ l , where χ l The value of is greater than or equal to 10 and less than or equal to 20;
[0031] S3.2.6, first and Respectively with χ l Compare and then process them according to the comparison results, specifically:
[0032] When the comparison result satisfies or and When , the processing process is:
[0033] A1. Use equations (6), (7) and (8) to calculate the local 3D position vector estimate of the l-th joint point from the k-th frame data of the camera. and The covariance matrix of
[0034]
[0035]
[0036]
[0037] A2. Use equations (9), (10) and (11) to calculate the local 3D position vector estimate of the l joint point of the kth frame data of the main camera and The covariance matrix of
[0038]
[0039]
[0040]
[0041] A3, proceed to step S3.2.7;
[0042] When the comparison result satisfies and and The specific processing process is:
[0043] B1. Use equations (6), (7) and (8) to calculate the local 3D position vector estimate of the l-th joint point of the k-th frame data collected from the camera. and The covariance matrix of
[0044] B2. Judgment Is it true? If so, let the local 3D position vector estimate of the l joint point of the k-th frame data of the main camera be The covariance matrix of Then go to step S3.2.7; if it is not true, the three-dimensional position vector of the l-th joint point of the k-th frame data collected by the main camera is Continue with progressive filtering. The specific progressive filtering process is as follows:
[0045] B2.1. Set the iteration variable of the progressive filter to t, set the maximum iteration step number M = 10, and set the state variable in the progressive filter to and
[0046] B2.2, initialize t, let t = 1, and To initialize,
[0047] B2.3, perform the tth iteration filtering, the specific process is:
[0048] B2.3.1. Calculate the intermediate state variable of the tth generation using equations (12) to (16): The covariance matrix of the intermediate state of the tth generation and the tth generation intermediate judgment variable
[0049]
[0050]
[0051]
[0052]
[0053]
[0054] In the above formula,
[0055] B2.3.2. Judgment Is it established? If so, let Then go to step S3.2.7; if not, further determine whether the current value of t is equal to M. If not, use the sum of the current value of t plus 1 to update the value of t, and then return to step B2.3 for the next iterative filtering. If it is equal, set Then proceed to step S3.2.7;
[0056] When the comparison result satisfies and and The specific processing process is:
[0057] C1, using equations (9), (10) and (11) to calculate the local 3D position vector estimate of the l joint point of the k-th frame data collected by the main camera and The covariance matrix of
[0058] C2. Judgment Is it true? If so, let the local 3D position vector estimate of the l joint point from the kth frame data of the camera be The covariance matrix of Then go to step S3.2.7; otherwise, the mapping vector of the l joint point from the kth frame data of the camera Continue with progressive filtering. The specific progressive filtering process is as follows:
[0059] C2.1. Set the iteration variable in the progressive filtering to n, the maximum number of iteration steps N = 10, and set the state variable in the progressive filtering to and
[0060] C2.2, initialize n, set n = 1; and To initialize,
[0061] C2.3, perform the nth iteration filtering, the specific process is:
[0062] C2.3.1. Using equations (17) to (21), calculate the nth generation intermediate state variable in the progressive filtering from the camera: The nth generation intermediate state covariance matrix and the nth generation intermediate judgment variable
[0063]
[0064]
[0065]
[0066]
[0067]
[0068] In the above formula,
[0069] C2.3.2. Judgment Is it established? If so, let Then go to step S3.2.7; if not, further determine whether the current value of n is equal to N. If not, first use the current value of n plus 1 to update the value of n, then return to step C2.3 for the next iterative filtering. If it is equal, set Then proceed to step S3.2.7;
[0070] When the comparison result satisfies and and When directly Then proceed to step S3.2.7;
[0071] S3.2.7, using formula (22) and formula (23) to calculate the global three-dimensional position vector estimate of the l joint point of the k-th frame data and The covariance matrix of
[0072]
[0073]
[0074] S3.3, determine whether the current value of k is equal to Y. If not, use the sum of the current value of k plus 1 to update the value of k, and then return to step S3.2 to process the next frame of data. If it is equal, the human body posture evaluation is completed. That is the obtained human body posture data.
[0075] Compared with the prior art, the advantage of the present invention is that data is first collected by two Azure Kinect DK cameras to obtain the three-dimensional position vectors of 32 human skeleton joints in the coordinate system of each Azure Kinect DK camera, that is, the measurement information of each Azure Kinect DK camera, and then the initial data of each joint point obtained by each camera is determined based on the measurement information of each Azure Kinect DK camera (the initial data of each joint point obtained from the camera is represented by the mapping vector of each joint point obtained from the camera in the coordinate system of the main camera, and the initial data of each joint point obtained by the main camera is represented by the three-dimensional position vector of each joint point obtained by the main camera), and then data filtering and fusion processing is performed based on the initial data of each joint point obtained by each Azure Kinect DK camera. During the data filtering and fusion processing, the initial data of each joint point obtained by each Azure Kinect DK camera is classified by using the Mahalanobis distance. When the square of the Mahalanobis distance between the initial data of a certain joint point obtained from the camera and the initial data of the corresponding joint point obtained from the main camera exceeds the set confidence threshold, it means that the measurement information of the two Azure Kinect DK cameras is inconsistent, indicating that a certain Azure Kinect The DK camera may be occluded. In this case, the square of the Mahalanobis distance between the predicted global 3D position vector of the corresponding joint point and the initial data of the corresponding joint point obtained by each Azure Kinect DK camera is measured to screen out the cameras affected by visual occlusion. If the square of the Mahalanobis distance between the predicted global 3D position vector of the corresponding joint point and the initial data of the corresponding joint point obtained by the screened Azure Kinect DK camera is too large, it means that the screened Azure Kinect DK camera is greatly affected by the occlusion and its measurement information is unreliable. Therefore, the fusion of the initial data of the corresponding joint point obtained by the Azure Kinect DK camera is rejected to avoid affecting the final result.If the square of the Mahalanobis distance between the predicted value of the global three-dimensional position vector of the corresponding joint point and the initial data of the corresponding joint point obtained by the selected Azure KinectDK camera is lower than the confidence threshold, the initial data of the corresponding joint point obtained by the selected Azure KinectDK camera can be guided by the initial data of the corresponding joint point obtained by another Azure Kinect DK camera to perform progressive filtering fusion, and the judgment variable is used to control the stop of the progressive filtering iteration to achieve the effect of implicit compensation. Finally, the initial data of the corresponding joint points obtained by different Azure Kinect DK cameras are fully utilized, and the local estimation results obtained after local filtering of each Azure Kinect DK camera are globally fused to further improve the accuracy of human posture estimation. Therefore, the present invention realizes initial data acquisition only by two Azure Kinect DK cameras and the human posture capture function contained therein, which greatly reduces the cost, has both low cost and high precision, and does not require the tester to wear joint point markers, which will not cause inconvenience to the tester. ; BRIEF DESCRIPTION OF THE DRAWINGS
[0076] Figure 1 Human skeleton joint hierarchy graph captured by Azure Kinect DK camera;
[0077] Figure 2 A schematic diagram of scene equipment arrangement of a method for estimating human posture based on progressive filtering fusion of the present invention;
[0078] Figure 3 A flowchart of a method for estimating human posture based on progressive filtering fusion of the present invention;
[0079] Figure 4 It is a comparison diagram of position errors between the method of the present invention and the traditional method under the same experimental environment. DETAILED DESCRIPTION
[0080] The present invention is further described in detail below with reference to the accompanying drawings.
[0081] Example: Figure 2 As shown, a method for estimating human posture based on progressive filtering fusion comprises the following steps:
[0082] Step 1: Figure 3 As shown, initial data collection is performed, and the specific process is as follows:
[0083] S1.1. Place two Azure Kinect DK cameras on the test bench, where the two Azure Kinect DK cameras are on the same horizontal plane, the field of view overlap rate reaches more than 80%, and the field of view of the two Azure Kinect DK cameras can fully cover the tester in the center of the test bench;
[0084] S1.2. Ask the tester to stand upright in the center of the test bench, holding a checkerboard calibration plate with a square length of 10 mm and a pattern array of 8*6 in both hands, so that the checkerboard pattern on the front of the 8*6 checkerboard calibration plate can be clearly seen in both Azure Kinect DK cameras;
[0085] S1.3, first start two Azure Kinect DK cameras at the same time to shoot a frame of color image data, then close the two Azure Kinect DK cameras, and then obtain a frame of color image data shot by each of the two Azure Kinect DK cameras, so that two frames of color image data are obtained;
[0086] S1.4, inputting the two frames of color image data into the OpenCV software toolkit, and processing the two frames of color image data to obtain a transformation matrix between the two Azure Kinect DK cameras, and recording the transformation matrix as W;
[0087] S1.5. Take the checkerboard calibration plate away from the tester, and the tester enters the ready state;
[0088] S1.6. The tester starts to perform the test action. At the same time, two Azure Kinect DK cameras are started to record video at the same time. Until the tester completes all the test actions, the video recording of the two Azure Kinect DK cameras is completed, and the two Azure Kinect DK cameras are turned off. In which, each frame of data in the video recorded by each Azure Kinect DK camera includes image data and human skeleton data corresponding to the image data. The human skeleton data is the three-dimensional position vector of 32 human skeleton joints in the coordinate system of the Azure Kinect DK camera. The 32 human skeleton joints are pelvis, spine, spine chest, neck, left clavicle, left shoulder, left elbow, left wrist, left hand, left hand tip, left thumb, right clavicle, right shoulder, right elbow, right wrist, right hand, right hand tip, right thumb, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, head, nose, left eye, left ear, right eye and right ear; in order from 1 to 32 The pelvis, spine, spine chest, neck, left clavicle, left shoulder, left elbow, left wrist, left hand, left hand tip, left thumb, right clavicle, right shoulder, right elbow, right wrist, right hand, right hand tip, right thumb, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, head, nose, left eye, left ear, right eye and right ear are numbered respectively, and the human skeletal joint numbered l is called l joint point, l = 1, 2, ...., 32; the total number of frames of data in the video recorded by the two Azure Kinect DK cameras is recorded as Y, that is, each Azure Kinect DK camera obtains Y frames of data respectively; the three-dimensional position vectors of the 32 human skeletal joints in the coordinate system of each Azure Kinect DK camera are the measurement information of the Azure Kinect DK camera, where the hierarchical structure diagram of the 32 human skeletal joints is as follows Figure 1 As shown;
[0089] S1.7. Use any one of the two Azure Kinect DK cameras as the master camera and the other as the slave camera; multiply the three-dimensional position vector of the l-th joint point of the a-th frame data obtained from the slave camera by the transformation matrix W to obtain the mapping vector of the l-th joint point in the main camera coordinate system, which is called the mapping vector of the l-th joint point, and is recorded as The three-dimensional position vector of the l-th joint point of the a-th frame data obtained by the main camera is recorded as
[0090] Step 2: Set the estimated global 3D position vector of the joint point l in the a-th frame data to set up The covariance matrix of The estimated value of the global 3D position vector of the l joint point in the first frame of data To initialize, right The covariance matrix of To initialize, is equal to the 3*3 identity matrix, which is denoted as I; set The measurement noise covariance is make Equal to 3*I; set The measurement noise covariance is make =3*I; Set the arbitrary human skeleton joint point measurement matrix of the a-th frame data to H a , let H a =I, where * is the multiplication symbol;
[0091] Step 3: Perform data filtering and fusion processing. The specific process is as follows:
[0092] S3.1, set the frame number variable to k, initialize k, and set k=2;
[0093] S3.2. The three-dimensional position vector of the l-th joint point of the k-th frame data obtained by the main camera and the mapping vector of the l joint point Processing is performed to obtain the estimated global three-dimensional position vector of the l joint point of the k-th frame data The specific process is:
[0094] S3.2.1. Set the state transfer matrix of the k-th frame data to F k , let F k =I; Set the process noise covariance matrix of the k-th frame data to Q k , let Q k =0.2*I; Set the global three-dimensional position vector prediction value of the l-th joint point of the k-th frame data to set up The covariance matrix of
[0095] S3.2.2, using formula (1) and (2) to calculate and
[0096]
[0097]
[0098] In formula (2), the subscript T represents the transpose symbol of the matrix;
[0099] S3.2.3. Setting and The square of the Mahalanobis distance is Using formula (3) to calculate
[0100]
[0101] In formula (3),
[0102] S3.2.4. Setting and The square of the Mahalanobis distance is set up and The square of the Mahalanobis distance is Using formula (4) and (5) respectively, we can get and
[0103]
[0104]
[0105] In formula (4), In formula (5),
[0106] S3.2.5. Set the confidence threshold of the l joint point to χ l , where χ l The value of is greater than or equal to 10 and less than or equal to 20;
[0107] S3.2.6, first and Respectively with χ l Compare and then process them according to the comparison results, specifically:
[0108] When the comparison result satisfies or and When , the processing process is:
[0109] A1. Use equations (6), (7) and (8) to calculate the local 3D position vector estimate of the l-th joint point from the k-th frame data of the camera. and The covariance matrix of
[0110]
[0111]
[0112]
[0113] A2. Use equations (9), (10) and (11) to calculate the local 3D position vector estimate of the l joint point of the kth frame data of the main camera and The covariance matrix of
[0114]
[0115]
[0116]
[0117] A3, proceed to step S3.2.7;
[0118] When the comparison result satisfies and and The specific processing process is:
[0119] B1. Use equations (6), (7) and (8) to calculate the local 3D position vector estimate of the l-th joint point of the k-th frame data collected from the camera. and The covariance matrix of
[0120] B2. Judgment Is it true? If so, let the local 3D position vector estimate of the l joint point of the k-th frame data of the main camera be The covariance matrix of Then go to step S3.2.7; if it is not true, the three-dimensional position vector of the l-th joint point of the k-th frame data collected by the main camera is Continue with progressive filtering. The specific progressive filtering process is as follows:
[0121] B2.1. Set the iteration variable of the progressive filter to t, set the maximum iteration step number M = 10, and set the state variable in the progressive filter to and
[0122] B2.2, initialize t, let t = 1, and To initialize,
[0123] B2.3, perform the tth iteration filtering, the specific process is:
[0124] B2.3.1. Calculate the intermediate state variable of the tth generation using equations (12) to (16): The covariance matrix of the intermediate state of the tth generation and the tth generation intermediate judgment variable
[0125]
[0126]
[0127]
[0128]
[0129]
[0130] In the above formula,
[0131] B2.3.2. Judgment Is it established? If so, let Then go to step S3.2.7; if not, further determine whether the current value of t is equal to M. If not, use the sum of the current value of t plus 1 to update the value of t, and then return to step B2.3 for the next iterative filtering. If it is equal, set Then proceed to step S3.2.7;
[0132] When the comparison result satisfies and and The specific processing process is:
[0133] C1, using equations (9), (10) and (11) to calculate the local 3D position vector estimate of the l joint point of the k-th frame data collected by the main camera and The covariance matrix of
[0134] C2. Judgment Is it true? If so, let the local 3D position vector estimate of the l joint point from the kth frame data of the camera be The covariance matrix of Then go to step S3.2.7; otherwise, the mapping vector of the l-th joint point from the k-th frame data of the camera is Continue with progressive filtering. The specific progressive filtering process is as follows:
[0135] C2.1, set the iteration variable in the progressive filtering to n, the maximum number of iteration steps N = 10, set the state variable in the progressive filtering and
[0136] C2.2, initialize n, set n = 1; and To initialize,
[0137] C2.3, perform the nth iteration filtering, the specific process is:
[0138] C2.3.1. Using equations (17) to (21), calculate the nth generation intermediate state variable in the progressive filtering from the camera: The nth generation intermediate state covariance matrix and the nth generation intermediate judgment variable
[0139]
[0140]
[0141]
[0142]
[0143]
[0144] In the above formula,
[0145] C2.3.2. Judgment Is it established? If so, let Then go to step S3.2.7; if not, further determine whether the current value of n is equal to N. If not, first use the current value of n plus 1 to update the value of n, then return to step C2.3 for the next iterative filtering. If it is equal, set Then proceed to step S3.2.7;
[0146] When the comparison result satisfies and and When directly Then proceed to step S3.2.7;
[0147] S3.2.7, using formula (22) and formula (23) to calculate the global three-dimensional position vector estimate of the l joint point of the k-th frame data and The covariance matrix of
[0148]
[0149]
[0150] S3.3, determine whether the current value of k is equal to Y. If not, use the sum of the current value of k plus 1 to update the value of k, and then return to step S3.2 to process the next frame of data. If it is equal, the human body posture evaluation is completed. That is the obtained human body posture data.
[0151] In order to verify the rationality and effectiveness of the human body posture estimation method based on progressive filtering fusion of the present invention, a human body posture estimation experiment consisting of two Azure Kinect DK cameras was designed. The experimental process specifically refers to steps S1.1-S3.3 of the human body posture estimation method based on progressive filtering fusion of the present invention. In addition, the human body posture detection value of the OptiTrack human motion capture system is used as the true value, and the cumulative position error between the human body posture estimation results under different methods and the true value is used as a measurement standard. The human body posture estimation method based on progressive filtering fusion of the present invention is compared with the human body posture estimation method based on observation fusion disclosed in the existing document "Multiple Kinect Sensor Fusion for Human Skeleton Tracking Using Kalman Filtering" and the human body posture estimation method based on centralized fusion disclosed in the document "Human Action Recognition Using a Distributed RGB-Depth Camera Network" and the human body posture estimation method based on information weighted consensus filtering. The experimental results are as follows. Figure 4 Analysis Figure 4 It can be seen that when Y=146, the cumulative error of the human body posture estimation method based on progressive filtering fusion of the present invention is 17.44 meters, the cumulative error of the human body posture estimation method based on information weighted consensus filtering is 22.36 meters, the cumulative error of the human body posture estimation method based on centralized fusion is 22.75 meters, and the cumulative error of the human body posture estimation method based on observation fusion is 24.3 meters. It can be seen that compared with the three existing human body posture estimation methods, the cumulative error of the human body posture estimation method based on progressive filtering fusion of the present invention is reduced by 4.92 to 6.86 meters, and the accuracy is improved.
Claims
1. A human body posture estimation method based on progressive filtering fusion, characterized in that The following steps are involved: Step 1: Perform initial data collection. The specific process is as follows: S1.
1. Place two Azure Kinect DK cameras on the test bench, where the two Azure Kinect DK cameras are on the same horizontal plane, the field of view overlap rate reaches more than 80%, and the field of view of the two Azure Kinect DK cameras can fully cover the tester in the center of the test bench; S1.
2. Ask the tester to stand upright in the center of the test bench, holding a checkerboard calibration plate with a square length of 10 mm and a pattern array of 8*6 in both hands, so that the checkerboard pattern on the front of the 8*6 checkerboard calibration plate can be clearly seen in both Azure Kinect DK cameras; S1.3, first start two Azure Kinect DK cameras at the same time to shoot a frame of color image data, then close the two Azure Kinect DK cameras, and then obtain a frame of color image data shot by each of the two Azure Kinect DK cameras, so that two frames of color image data are obtained; S1.4, inputting the two frames of color image data into the OpenCV software toolkit, and processing the two frames of color image data to obtain a transformation matrix between the two Azure Kinect DK cameras, and recording the transformation matrix as W; S1.
5. Take the checkerboard calibration plate away from the tester, and the tester enters the ready state; S1.
6. The tester starts to perform the test action and starts two Azure Kinect DK cameras to start recording video at the same time. The tester completes all the test actions. At this time, the video recording of the two Azure Kinect DK cameras is completed and the two Azure Kinect DK cameras are turned off. Each frame of data in the video recorded by each Azure Kinect DK camera includes image data and human skeleton data corresponding to the image data. The human skeleton data is the three-dimensional position vector of 32 human skeleton joints in the coordinate system of the Azure KinectDK camera. The 32 human skeleton joints are pelvis, spine, spine chest, neck, left clavicle, left shoulder, left elbow, left wrist, left hand, left hand tip, left thumb, right clavicle, right shoulder, right elbow, right wrist, right hand, right hand tip, right thumb, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, head, nose, left eye, left ear, right eye and right ear. According to 1-3 2. Number the pelvis, spine, spine chest, neck, left clavicle, left shoulder, left elbow, left wrist, left hand, left hand tip, left thumb, right clavicle, right shoulder, right elbow, right wrist, right hand, right hand tip, right thumb, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, head, nose, left eye, left ear, right eye and right ear in sequence, and call the human skeleton joint numbered l as l joint point, l = 1, 2, ...., 32; record the total number of frames of data in the video recorded by the two Azure Kinect DK cameras as Y, that is, each Azure KinectDK camera obtains Y frames of data; S1.
7. Use any one of the two Azure Kinect DK cameras as the master camera and the other as the slave camera; multiply the three-dimensional position vector of the l-th joint point of the a-th frame data obtained from the slave camera by the transformation matrix W to obtain the mapping vector of the l-th joint point in the main camera coordinate system, which is called the mapping vector of the l-th joint point, and is recorded as The three-dimensional position vector of the l-th joint point of the a-th frame data obtained by the main camera is recorded as Step 2: Set the estimated global 3D position vector of the joint point l in the a-th frame data to set up The covariance matrix of The estimated value of the global 3D position vector of the l joint point in the first frame of data To initialize, right The covariance matrix of To initialize, is equal to the 3*3 identity matrix, which is denoted as I; set The measurement noise covariance is make Equal to 3*I; set The measurement noise covariance is make =3*I; Set the arbitrary human skeleton joint point measurement matrix of the a-th frame data to H a , let H a =I, where * is the multiplication symbol; Step 3: Perform data filtering and fusion processing. The specific process is as follows: S3.1, set the frame number variable to k, initialize k, and set k=2; S3.
2. The three-dimensional position vector of the l-th joint point of the k-th frame data obtained by the main camera and the mapping vector of the l joint point Processing is performed to obtain the estimated global three-dimensional position vector of the l joint point of the k-th frame data The specific process is: S3.2.
1. Set the state transfer matrix of the kth frame data to F k , let F k =I; Set the process noise covariance matrix of the k-th frame data to Q k , let Q k =0.2*I; Set the global 3D position vector prediction value of the l joint point of the k-th frame data to set up The covariance matrix of S3.2.2, using formula (1) and (2) to calculate and In formula (2), the subscript T represents the transpose symbol of the matrix; S3.2.
3. Setting and The square of the Mahalanobis distance is Calculated using formula (3) In formula (3), S3.2.
4. Setting and The square of the Mahalanobis distance is set up and The square of the Mahalanobis distance is Using formula (4) and (5) respectively, we can get and In formula (4), In formula (5), S3.2.
5. Set the confidence threshold of the l joint point to χ l , χ l The value of is greater than or equal to 10 and less than or equal to 20; S3.2.6, first and Respectively with χ l Compare and then process them according to the comparison results, specifically: When the comparison result satisfies or and When , the processing process is: A1. Use equations (6), (7) and (8) to calculate the local 3D position vector estimate of the l-th joint point from the k-th frame data of the camera. and The covariance matrix of A2. Use equations (9), (10) and (11) to calculate the local 3D position vector estimate of the l joint point of the kth frame data of the main camera and The covariance matrix of A3, proceed to step S3.2.7; When the comparison result satisfies and and The specific processing process is: B1. Use equations (6), (7) and (8) to calculate the local 3D position vector estimate of the l-th joint point of the k-th frame data collected from the camera. and The covariance matrix of B2. Judgment Is it true? If so, let the local 3D position vector estimate of the l joint point of the k-th frame data of the main camera be The covariance matrix of Then go to step S3.2.7; if it is not true, the three-dimensional position vector of the l-th joint point of the k-th frame data collected by the main camera is Continue with progressive filtering. The specific progressive filtering process is as follows: B2.
1. Set the iteration variable of the progressive filter to t, set the maximum iteration step number M = 10, and set the state variable in the progressive filter to and B2.2, initialize t, let t = 1, and To initialize, B2.3, perform the tth iteration filtering, the specific process is: B2.3.
1. Calculate the intermediate state variable of the tth generation using equations (12) to (16): The covariance matrix of the intermediate state of the tth generation and the tth generation intermediate judgment variable In the above formula, B2.3.
2. Judgment Is it established? If so, let Then go to step S3.2.7; if not, further determine whether the current value of t is equal to M. If not, use the sum of the current value of t plus 1 to update the value of t, and then return to step B2.3 for the next iterative filtering. If it is equal, set Then proceed to step S3.2.7; When the comparison result satisfies and and The specific processing process is: C1, using equations (9), (10) and (11) to calculate the local 3D position vector estimate of the l joint point of the k-th frame data collected by the main camera and The covariance matrix of C2. Judgment Is it true? If so, let the local 3D position vector estimate of the l joint point from the kth frame data of the camera be The covariance matrix of Then go to step S3.2.7; otherwise, the mapping vector of the l-th joint point from the k-th frame data of the camera is Continue with progressive filtering. The specific progressive filtering process is as follows: C2.1, set the iteration variable in the progressive filtering to n, the maximum number of iteration steps N = 10, set the state variable in the progressive filtering and C2.2, initialize n, set n = 1; and To initialize, C2.3, perform the nth iteration filtering, the specific process is: C2.3.
1. Using equations (17) to (21), calculate the nth generation intermediate state variable in the progressive filtering from the camera: The nth generation intermediate state covariance matrix and the nth generation intermediate judgment variable In the above formula, C2.3.
2. Judgment Is it established? If so, let Then go to step S3.2.7; if not, further determine whether the current value of n is equal to N. If not, first use the current value of n plus 1 to update the value of n, then return to step C2.3 for the next iterative filtering. If it is equal, set Then proceed to step S3.2.7; When the comparison result satisfies and and When directly Then proceed to step S3.2.7; S3.2.7, using formula (22) and formula (23) to calculate the global three-dimensional position vector estimate of the l joint point of the k-th frame data and The covariance matrix of S3.3, determine whether the current value of k is equal to Y. If not, use the sum of the current value of k plus 1 to update the value of k, and then return to step S3.2 to process the next frame of data. If it is equal, the human body posture evaluation is completed. That is the obtained human body posture data.
