Multi-view fusion human motion estimation method based on distributed progressive gaussian filtering

Abstract

Disclosed in the present invention is a multi-view fusion human motion estimation method based on distributed progressive Gaussian filtering. The method comprises: firstly, performing data collection by means of two Azure Kinect DK cameras, and determining initial data; then, performing data filtering fusion processing; performing classification processing on the initial data of the two Azure Kinect DK cameras by using a Mahalanobis distance, and screening and discarding measurement information of the Azure Kinect DK camera highly susceptible to visual occlusion; for measurement information of the Azure Kinect DK camera which is determined to be less affected by visual occlusion, guiding, by means of measurement information of another Azure Kinect DK camera, to perform progressive filtering fusion on the measurement information of the Azure Kinect DK camera, so as to achieve the effect of implicit compensation; and finally, performing global fusion, so as to further improve the precision of human body pose estimation. The method has the advantages of low cost and high precision, and it is not necessary for a tester to wear a joint point marker, and thus, no inconvenience is caused to the tester.

Description

Multi-view fusion human motion estimation method based on distributed progressive Gaussian filtering Technical Field

[0001] The present invention relates to a human motion estimation method, in particular to a multi-view fusion human motion estimation method based on distributed progressive Gaussian filtering. Background Art

[0002] With the development of machine learning and computer vision, AI fitness trainers are gradually being widely used in the fitness field. In the development and use of AI fitness trainers, human motion (i.e., posture) estimation is needed to perceive the test subject's fitness posture data. The traditional method of human motion estimation is to obtain human posture data through a human posture capture system. Although the human posture capture system can obtain human posture data with high accuracy, it not only requires the use of more than ten professional cameras for shooting, but also has relatively expensive hardware and high costs. In addition, it requires the test subject to wear many joint markers, which causes inconvenience to the test subject.

[0003] To address the challenges of traditional human motion estimation methods, a new approach has emerged. This approach uses a low-cost RGB-D camera to capture a single image of the subject. Based on this single image, the subject's current three-dimensional pose is estimated and the corresponding 3D joint position data (i.e., human pose data) is detected. While this method is inexpensive and does not inconvenience the subject, it does suffer from the problem that when the subject's arms are crossed or they are sideways, some of their joints are obscured in the image captured by the RGB-D camera. Consequently, the estimated position data for these obscured joints is inaccurate, resulting in a reduction in the overall accuracy of the human pose data.

[0004] Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a multi-view fusion human motion estimation method based on distributed progressive Gaussian filtering, which has both low cost and high precision and does not require the tester to wear joint point markers and does not cause inconvenience to the tester.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a multi-view fusion human motion estimation method based on distributed progressive Gaussian filtering, comprising the following steps:

[0007] Step 1: Perform initial data collection. The specific process is as follows:

[0008] S1.1. Place two Azure Kinect DK cameras on the test bench, ensuring that they are on the same horizontal plane, have a field of view overlap of at least 80%, and have a field of view that fully covers the subject in the center of the test bench.

[0009] S1.2. Have the tester stand upright in the center of the test bench, holding a 10mm square checkerboard calibration plate with an 8x6 checkerboard pattern. Ensure that both Azure Kinect DK cameras clearly see the checkerboard pattern on the front of the 8x6 checkerboard calibration plate.

[0010] S1.3. First, simultaneously start two Azure Kinect DK cameras to capture a frame of color image data, then close the two Azure Kinect DK cameras, and then obtain a frame of color image data captured by each of the two Azure Kinect DK cameras. In this case, two frames of color image data are obtained.

[0011] S1.4. Input the two frames of color image data into the OpenCV software toolkit. The OpenCV software toolkit processes the two frames of color image data to obtain a transformation matrix between the two Azure Kinect DK cameras. The transformation matrix is ​​denoted as W.

[0012] S1.5. Remove the checkerboard calibration plate from the tester, and the tester enters the ready state;

[0013] S1.6. The tester begins to perform the test action. At the same time, two Azure Kinect DK cameras are started to record video. Until the tester completes all test actions, the video recording of the two Azure Kinect DK cameras is completed, and the two Azure Kinect DK cameras are turned off. Each frame of data in the video recorded by each Azure Kinect DK camera includes image data and human skeleton data corresponding to the image data. The human skeleton data is the three-dimensional position vector of 32 human skeletal joints in the coordinate system of the Azure Kinect DK camera. The 32 human skeletal joints are pelvis, spine, spine chest, neck, left clavicle, left shoulder, left elbow, left wrist, left hand, left hand tip, left thumb, right clavicle, right shoulder, right elbow, right wrist, right hand, right hand tip, right thumb, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, head, nose, left eye, left ear, right eye and right ear; in order from 1 to 32. Number the pelvis, spine, spinal chest, neck, left clavicle, left shoulder, left elbow, left wrist, left hand, left hand tip, left thumb, right clavicle, right shoulder, right elbow, right wrist, right hand, right hand tip, right thumb, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, head, nose, left eye, left ear, right eye, and right ear respectively. The human skeletal joint numbered l is called the l-joint, where l = 1, 2, ..., 32. The total number of frames of data in the video recorded by the two Azure Kinect DK cameras is recorded as Y, that is, each Azure Kinect DK camera obtains Y frames of data.

[0014] S1.7. Use one of the two Azure Kinect DK cameras as the master camera and the other as the slave camera. Multiply the 3D position vector of the joint point l in the a-th frame data obtained from the slave camera by the transformation matrix W to obtain the mapping vector of the joint point l in the master camera coordinate system. This is called the mapping vector of the joint point l and is denoted as The three-dimensional position vector of the l-th joint point of the a-th frame data obtained by the main camera is recorded as

[0015] Step 2: Set the estimated global 3D position vector of the joint point l in the a-th frame data to be set up The covariance matrix of Estimated global 3D position vector of the l joint point in the first frame of data Initialize, let right The covariance matrix of Initialize, let is equal to the 3*3 identity matrix, which is denoted as I; set The measurement noise covariance is make Equal to 3*I; set The measurement noise covariance is make =3*I; Set the measurement matrix of any human skeleton joint point of the a-th frame data to H a , let H a =I, where * is the multiplication operator;

[0016] Step 3: Perform data filtering and fusion processing. The specific process is as follows:

[0017] S3.1. Set the frame number variable to k and initialize k to 2;

[0018] S3.2. The three-dimensional position vector of the joint point l in the k-th frame data obtained by the main camera and the mapping vector of the l joint point Processing is performed to obtain the global three-dimensional position vector estimate of the l joint point of the k-th frame data The specific process is:

[0019] S3.2.1. Set the state transition matrix of the k-th frame data to F k , let F k Equal to I; set the process noise covariance matrix of the k-th frame data to Q k , let Q k = 0.2*I; set the global three-dimensional position vector prediction value of the joint point l in the k-th frame data to be set up The covariance matrix of

[0020] S3.2.2, using formula (1) and (2) to calculate and

[0021] In formula (2), the subscript T represents the transpose symbol of the matrix;

[0022] S3.2.3. Setting and The square of the Mahalanobis distance is Calculated using formula (3)

[0023] In formula (3),

[0024] S3.2.4. Setting and The square of the Mahalanobis distance is set up and The square of the Mahalanobis distance is Using formula (4) and (5) respectively, we can get and

[0025] In formula (4), In formula (5),

[0026] S3.2.5. Set the confidence threshold of joint point l to χ l , where χ l The value of is greater than or equal to 10 and less than or equal to 20;

[0027] S3.2.6, first and Respectively with χ l Compare and then process according to the comparison results, specifically:

[0028] When the comparison result satisfies or and When , the processing process is:

[0029] A1. Use equations (6), (7) and (8) to calculate the local three-dimensional position vector estimate of the l-th joint point from the k-th frame data of the camera. and The covariance matrix of

[0030] A2. Use equations (9), (10) and (11) to calculate the local three-dimensional position vector estimate of the joint point l of the k-th frame data of the main camera and The covariance matrix of

[0031] A3. Go to step S3.2.7;

[0032] When the comparison result satisfies and and The specific processing process is:

[0033] B1. Use equations (6), (7) and (8) to calculate the local three-dimensional position vector estimate of the joint point l of the k-th frame data collected from the camera. and The covariance matrix of

[0034] B2. Judgment Is it true? If so, let the local 3D position vector estimate of the l joint point of the k-th frame data of the main camera be The covariance matrix of Then go to step S3.2.7; if it is not true, the three-dimensional position vector of the l-th joint point of the k-th frame data collected by the main camera is Continue with progressive filtering. The specific progressive filtering process is as follows:

[0035] B2.1. Set the iteration variable of the progressive filter to t, set the maximum number of iteration steps M = 10, and set the state variable in the progressive filter to and

[0036] B2.2, initialize t, let t = 1, and Initialize, let

[0037] B2.3, perform the t-th iterative filtering, the specific process is:

[0038] B2.3.1. Calculate the intermediate state variable of the tth generation using equations (12) to (16): The covariance matrix of the intermediate state of the tth generation and the tth generation intermediate judgment variable

[0039] In the above formula,

[0040] B2.3.2 Judgment Is it established? If so, let Then go to step S3.2.7; if not, further determine whether the current value of t is equal to M. If not, use the sum of the current value of t plus 1 to update the value of t, and then return to step B2.3 for the next iterative filtering. If it is equal, set Then proceed to step S3.2.7;

[0041] When the comparison result satisfies and and The specific processing process is:

[0042] C1, using equations (9), (10) and (11) to calculate the local three-dimensional position vector estimate of the l-th joint point of the k-th frame data collected by the main camera and The covariance matrix of

[0043] C2. Judgment Is it true? If so, let the local 3D position vector estimate of the l joint point from the kth frame data of the camera be The covariance matrix of Then go to step S3.2.7; otherwise, the mapping vector of the l-th joint point from the k-th frame data of the camera is Continue with progressive filtering. The specific progressive filtering process is as follows:

[0044] C2.1. Set the iteration variable in the progressive filtering to n, the maximum number of iteration steps N = 10, and set the state variable in the progressive filtering to and

[0045] C2.2, initialize n, set n = 1; and Initialize, let

[0046] C2.3, perform the nth iterative filtering, the specific process is as follows:

[0047] C2.3.1. Using equations (17) to (21), calculate the nth generation intermediate state variable in the progressive filtering from the camera The covariance matrix of the intermediate state of the nth generation and the nth generation intermediate judgment variable

[0048] In the above formula,

[0049] C2.3.2 Judgment Is it established? If so, let Then go to step S3.2.7; if not, further determine whether the current value of n is equal to N. If not, first use the current value of n plus 1 to update the value of n, then return to step C2.3 for the next iterative filtering. If it is equal, then set Then proceed to step S3.2.7;

[0050] When the comparison result satisfies and and When directly Then proceed to step S3.2.7;

[0051] S3.2.7, use formula (22) and formula (23) to calculate the global three-dimensional position vector estimate of the joint point l of the k-th frame data and The covariance matrix of

[0052] S3.3, determine whether the current value of k is equal to Y. If not, update the value of k by adding 1 to the current value of k, and then return to step S3.2 to process the next frame of data. If it is equal, the human body posture evaluation is completed. This is the obtained human body posture data.

[0053] Compared with the prior art, the advantage of the present invention is that it first collects data through two Azure Kinect DK cameras to obtain the three-dimensional position vectors of 32 human skeletal joints in the coordinate system of each Azure Kinect DK camera, that is, the measurement information of each Azure Kinect DK camera, and then determines the initial data of each joint point obtained by each camera based on the measurement information of each Azure Kinect DK camera (the initial data of each joint point obtained from the camera is represented by the mapping vector of each joint point obtained from the camera in the coordinate system of the main camera, and the initial data of each joint point obtained from the main camera is represented by the three-dimensional position vector of each joint point obtained by the main camera), and then performs data filtering and fusion processing based on the initial data of each joint point obtained by each Azure Kinect DK camera. During the data filtering and fusion processing, the initial data of each joint point obtained by each Azure Kinect DK camera is classified using the Mahalanobis distance. When the square of the Mahalanobis distance between the initial data of a joint point obtained from the camera and the initial data of the corresponding joint point obtained from the main camera exceeds the set confidence threshold, it means that the measurement information of the two Azure Kinect DK cameras is inconsistent, indicating that a certain Azure Kinect The DK camera may be occluded. In this case, the square of the Mahalanobis distance between the predicted global 3D position vector of the corresponding joint point and the initial data of the corresponding joint point obtained by each Azure Kinect DK camera is measured to filter out the cameras affected by visual occlusion. If the square of the Mahalanobis distance between the predicted global 3D position vector of the corresponding joint point and the initial data of the corresponding joint point obtained by the screened Azure Kinect DK camera is too large, it means that the screened Azure Kinect DK camera is greatly affected by the occlusion and its measurement information is unreliable. Therefore, the fusion of the initial data of the corresponding joint point obtained by the Azure Kinect DK camera is rejected to avoid affecting the final result.If the square of the Mahalanobis distance between the predicted global 3D position vector of the corresponding joint point and the initial data of the corresponding joint point obtained by the selected Azure Kinect DK camera is below a confidence threshold, the initial data of the corresponding joint point obtained by another Azure Kinect DK camera can be used to guide the initial data of the corresponding joint point obtained by the selected Azure Kinect DK camera to perform progressive filtering fusion. The judgment variable is used to control the stopping of the progressive filtering iteration to achieve the effect of implicit compensation. Finally, the initial data of the corresponding joint points obtained by different Azure Kinect DK cameras are fully utilized. The local estimation results obtained after local filtering for each Azure Kinect DK camera are globally fused, further improving the accuracy of human pose estimation. Therefore, the present invention achieves initial data acquisition using only two Azure Kinect DK cameras and the human pose capture function contained therein, significantly reducing costs and achieving both low cost and high accuracy. It also does not require the test subject to wear joint markers, thus avoiding inconvenience to the test subject. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 shows the human skeleton joint hierarchy structure captured by the Azure Kinect DK camera;

[0055] FIG2 is a schematic diagram of scene equipment arrangement of a multi-view fusion human motion estimation method based on distributed progressive Gaussian filtering according to the present invention;

[0056] FIG3 is a flowchart of a multi-view fusion human motion estimation method based on distributed progressive Gaussian filtering according to the present invention;

[0057] FIG4 is a comparison diagram of position errors between the method of the present invention and the traditional method under the same experimental environment. DETAILED DESCRIPTION

[0058] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0059] Embodiment: As shown in FIG2 , a multi-view fusion human motion estimation method based on distributed progressive Gaussian filtering includes the following steps:

[0060] Step 1: As shown in Figure 3, perform initial data collection. The specific process is as follows:

[0061] S1.1. Place two Azure Kinect DK cameras on the test bench, ensuring that they are on the same horizontal plane, have a field of view overlap of at least 80%, and have a field of view that fully covers the subject in the center of the test bench.

[0062] S1.2. Have the tester stand upright in the center of the test bench, holding a 10mm square checkerboard calibration plate with an 8x6 checkerboard pattern. Ensure that both Azure Kinect DK cameras clearly see the checkerboard pattern on the front of the 8x6 checkerboard calibration plate.

[0063] S1.3. First, simultaneously start two Azure Kinect DK cameras to capture a frame of color image data, then close the two Azure Kinect DK cameras, and then obtain a frame of color image data captured by each of the two Azure Kinect DK cameras. In this case, two frames of color image data are obtained.

[0064] S1.4. Input the two frames of color image data into the OpenCV software toolkit. The OpenCV software toolkit processes the two frames of color image data to obtain a transformation matrix between the two Azure Kinect DK cameras. The transformation matrix is ​​denoted as W.

[0065] S1.5. Remove the checkerboard calibration plate from the tester, and the tester enters the ready state;

[0066] S1.6. The tester begins to perform the test action. At the same time, two Azure Kinect DK cameras are started to record video. Until the tester completes all test actions, the video recording of the two Azure Kinect DK cameras is completed, and the two Azure Kinect DK cameras are turned off. Each frame of data in the video recorded by each Azure Kinect DK camera includes image data and human skeleton data corresponding to the image data. The human skeleton data is the three-dimensional position vector of 32 human skeletal joints in the coordinate system of the Azure Kinect DK camera. The 32 human skeletal joints are pelvis, spine, spine chest, neck, left clavicle, left shoulder, left elbow, left wrist, left hand, left hand tip, left thumb, right clavicle, right shoulder, right elbow, right wrist, right hand, right hand tip, right thumb, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, head, nose, left eye, left ear, right eye and right ear; in order from 1 to 32. The pelvis, spine, spinal chest, neck, left clavicle, left shoulder, left elbow, left wrist, left hand, left hand tip, left thumb, right clavicle, right shoulder, right elbow, right wrist, right hand, right hand tip, right thumb, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, head, nose, left eye, left ear, right eye, and right ear are numbered respectively. The human skeletal joint numbered l is called l joint point, where l = 1, 2, ..., 32. The total number of frames of data in the video recorded by the two Azure Kinect DK cameras is recorded as Y, that is, each Azure Kinect DK camera obtains Y frames of data. The three-dimensional position vectors of the 32 human skeletal joint points in the coordinate system of each Azure Kinect DK camera are the measurement information of the Azure Kinect DK camera. The hierarchical structure diagram of the 32 human skeletal joint points is shown in Figure 1.

[0067] S1.7. Use one of the two Azure Kinect DK cameras as the master camera and the other as the slave camera. Multiply the 3D position vector of the joint point l in the a-th frame data obtained from the slave camera by the transformation matrix W to obtain the mapping vector of the joint point l in the master camera coordinate system. This is called the mapping vector of the joint point l and is denoted as The three-dimensional position vector of the l-th joint point of the a-th frame data obtained by the main camera is recorded as

[0068] Step 2: Set the estimated global 3D position vector of the joint point l in the a-th frame data to be set up The covariance matrix of Estimated global 3D position vector of the l joint point in the first frame of data Initialize, let right The covariance matrix of Initialize, let is equal to the 3*3 identity matrix, which is denoted as I; set The measurement noise covariance is make Equal to 3*I; set The measurement noise covariance is make =3*I; Set the measurement matrix of any human skeleton joint point of the a-th frame data to H a , let H a =I, where * is the multiplication operator;

[0069] Step 3: Perform data filtering and fusion processing. The specific process is as follows:

[0070] S3.1. Set the frame number variable to k and initialize k to 2;

[0071] S3.2. The three-dimensional position vector of the joint point l in the k-th frame data obtained by the main camera and the mapping vector of the l joint point Processing is performed to obtain the global three-dimensional position vector estimate of the l joint point of the k-th frame data The specific process is:

[0072] S3.2.1. Set the state transition matrix of the k-th frame data to F k , let F k Equal to I; set the process noise covariance matrix of the k-th frame data to Q k , let Q k = 0.2*I; set the global three-dimensional position vector prediction value of the joint point l in the k-th frame data to be set up The covariance matrix of

[0073] S3.2.2, using formula (1) and (2) to calculate and

[0074] In formula (2), the subscript T represents the transpose symbol of the matrix;

[0075] S3.2.3. Setting and The square of the Mahalanobis distance is Calculated using formula (3)

[0076] In formula (3),

[0077] S3.2.4. Setting and The square of the Mahalanobis distance is set up and The square of the Mahalanobis distance is Using formula (4) and (5) respectively, we can get and

[0078] In formula (4), In formula (5),

[0079] S3.2.5. Set the confidence threshold of joint point l to χ l , where χ l The value of is greater than or equal to 10 and less than or equal to 20;

[0080] S3.2.6, first and Respectively with χ l Compare and then process according to the comparison results, specifically:

[0081] When the comparison result satisfies or and When , the processing process is:

[0082] A1. Use equations (6), (7) and (8) to calculate the local three-dimensional position vector estimate of the l-th joint point from the k-th frame data of the camera. and The covariance matrix of

[0083] A2. Use equations (9), (10) and (11) to calculate the local three-dimensional position vector estimate of the joint point l of the k-th frame data of the main camera and The covariance matrix of

[0084] A3. Go to step S3.2.7;

[0085] When the comparison result satisfies and and The specific processing process is:

[0086] B1. Use equations (6), (7) and (8) to calculate the local three-dimensional position vector estimate of the joint point l of the k-th frame data collected from the camera. and The covariance matrix of

[0087] B2. Judgment Is it true? If so, let the local 3D position vector estimate of the l joint point of the k-th frame data of the main camera be The covariance matrix of Then go to step S3.2.7; if it is not true, the three-dimensional position vector of the l-th joint point of the k-th frame data collected by the main camera is Continue with progressive filtering. The specific progressive filtering process is as follows:

[0088] B2.1. Set the iteration variable of the progressive filter to t, set the maximum number of iteration steps M = 10, and set the state variable in the progressive filter to and

[0089] B2.2, initialize t, let t = 1, and Initialize, let

[0090] B2.3, perform the t-th iterative filtering, the specific process is:

[0091] B2.3.1. Calculate the intermediate state variable of the tth generation using equations (12) to (16): The covariance matrix of the intermediate state of the tth generation and the tth generation intermediate judgment variable

[0092] In the above formula,

[0093] B2.3.2 Judgment Is it established? If so, let Then go to step S3.2.7; if not, further determine whether the current value of t is equal to M. If not, use the sum of the current value of t plus 1 to update the value of t, and then return to step B2.3 for the next iterative filtering. If it is equal, set Then proceed to step S3.2.7;

[0094] When the comparison result satisfies and and The specific processing process is:

[0095] C1, using equations (9), (10) and (11) to calculate the local three-dimensional position vector estimate of the l-th joint point of the k-th frame data collected by the main camera and The covariance matrix of

[0096] C2. Judgment Is it true? If so, let the local 3D position vector estimate of the l joint point from the kth frame data of the camera be The covariance matrix of Then go to step S3.2.7; otherwise, the mapping vector of the l-th joint point from the k-th frame data of the camera is Continue with progressive filtering. The specific progressive filtering process is as follows:

[0097] C2.1. Set the iteration variable in the progressive filtering to n, the maximum number of iteration steps N = 10, and set the state variable in the progressive filtering to and

[0098] C2.2, initialize n, set n = 1; and Initialize, let

[0099] C2.3, perform the nth iterative filtering, the specific process is as follows:

[0100] C2.3.1. Using equations (17) to (21), calculate the nth generation intermediate state variable in the progressive filtering from the camera The covariance matrix of the intermediate state of the nth generation and the nth generation intermediate judgment variable

[0101] In the above formula,

[0102] C2.3.2 Judgment Is it established? If so, let Then go to step S3.2.7; if not, further determine whether the current value of n is equal to N. If not, first use the current value of n plus 1 to update the value of n, then return to step C2.3 for the next iterative filtering. If it is equal, then set Then proceed to step S3.2.7;

[0103] When the comparison result satisfies and and When directly Then proceed to step S3.2.7;

[0104] S3.2.7, use formula (22) and formula (23) to calculate the global three-dimensional position vector estimate of the joint point l of the k-th frame data and The covariance matrix of

[0105] S3.3, determine whether the current value of k is equal to Y. If not, update the value of k by adding 1 to the current value of k, and then return to step S3.2 to process the next frame of data. If it is equal, the human body posture evaluation is completed. This is the obtained human body posture data.

[0106] In order to verify the rationality and effectiveness of the multi-view fusion human motion estimation method based on distributed progressive Gaussian filtering of the present invention, a human posture estimation experiment was designed in an environment consisting of two Azure Kinect DK cameras. The experimental process specifically refers to steps S1.1-S3.3 of the multi-view fusion human motion estimation method based on distributed progressive Gaussian filtering of the present invention. In addition, the human posture detection value of the OptiTrack human motion capture system is used as the true value, and the cumulative position error between the human posture estimation results under different methods and the true value is used as a measurement standard. The multi-view fusion human motion estimation method based on distributed progressive Gaussian filtering of the present invention is compared with the human motion estimation method based on observation fusion disclosed in the existing document "Multiple Kinect Sensor Fusion for Human Skeleton Tracking Using Kalman Filtering" and the human motion estimation method based on centralized fusion disclosed in the document "Human Action Recognition Using a Distributed RGB-Depth Camera Network" and the human motion estimation method based on information weighted consensus filtering. The experimental results are shown in Figure 4. Analysis of Figure 4 shows that when Y = 146, the cumulative error of the multi-perspective fusion human motion estimation method based on distributed progressive Gaussian filtering of the present invention is 17.44 meters, the cumulative error of the human motion estimation method based on information weighted consensus filtering is 22.36 meters, the cumulative error of the human motion estimation method based on centralized fusion is 22.75 meters, and the cumulative error of the human motion estimation method based on observation fusion is 24.3 meters. It can be seen that the cumulative error of the multi-perspective fusion human motion estimation method based on distributed progressive Gaussian filtering of the present invention is reduced by 4.92 to 6.86 meters compared to the three existing human motion estimation methods, and the accuracy is improved.

Claims

1. A multi-view fusion human motion estimation method based on distributed progressive Gaussian filtering, characterized in that it includes the following steps: Step 1: Perform initial data collection. The specific process is as follows: S1.1: Place two Azure Kinect DK cameras on the test bench. Among them, the two Azure Kinect DK cameras are on the same horizontal plane, the field of view overlap rate reaches more than 80%, and the field of view ranges of the two Azure Kinect DK cameras can completely cover the tester in the center of the test bench; S1.2: Let the tester stand upright in the center of the test bench with a checkerboard calibration board with a grid side length of 10 mm and a pattern array of 8 * 6 in both hands, so that the checkerboard pattern on the front of the 8 * 6 checkerboard calibration board can be clearly seen in both Azure Kinect DK cameras; S1.3: First, start both Azure Kinect DK cameras to capture a frame of color image data at the same time, and then turn off both Azure Kinect DK cameras. Then, obtain a frame of color image data captured by each of the two Azure Kinect DK cameras. At this time, two frames of color image data are obtained; S1.4: Input the two frames of color image data into the OpenCV software toolkit. The OpenCV software toolkit processes the two frames of color image data to obtain the transformation matrix between the two Azure Kinect DK cameras, and denote this transformation matrix as W; S1.5: Take away the checkerboard calibration board from the tester, and the tester enters the preparation state; S1.6: The tester starts to perform the test actions. At the same time, start both Azure Kinect DK cameras to record videos until the tester completes all test actions. At this time, the video recording of both Azure Kinect DK cameras is completed, and turn off both Azure Kinect DK cameras; among them, each frame of data in the video recorded by each Azure Kinect DK camera includes image data and the human skeleton data corresponding to this image data. The human skeleton data is the three-dimensional position vectors of 32 human skeletal joints in the coordinate system of this Azure Kinect DK camera. The 32 human skeletal joints are the pelvis, spine, thoracic spine, neck, left clavicle, left shoulder, left elbow, left wrist, left hand, left hand tip, left thumb, right clavicle, right shoulder, right elbow, right wrist, right hand, right hand tip, right thumb, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, head, nose, left eye, left ear, right eye and right ear; sequentially number the pelvis, spine, thoracic spine, neck, left clavicle, left shoulder, left elbow, left wrist, left hand, left hand tip, left thumb, right clavicle, right shoulder, right elbow, right wrist, right hand, right hand tip, right thumb, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle, right foot, head, nose, left eye, left The ears, right eye, and right ear are numbered, and the human body bone joint point numbered l is called the l joint point, where l = 1, 2,...., 32; the total number of frames of data in the videos recorded by two Azure Kinect DK cameras is denoted as Y, that is, each Azure Kinect DK camera obtains Y frames of data; S1.

7. Take any one of the two Azure Kinect DK cameras as the master camera and the other as the slave camera; left-multiply the 3D position vector of the l-th joint point of the a-th frame data obtained from the slave camera by the transformation matrix W to obtain the mapping vector of the l-th joint point in the master camera coordinate system, which is called the mapping vector of the l-th joint point and is denoted as Denote the three-dimensional position vector of the l joint point of the a-th frame data obtained by the main camera as a = 1, 2,...., Y; Step 2. Set the estimated value of the global three-dimensional position vector of the l joint point of the a-th frame data to be Settings The covariance matrix of Estimated value of the global three-dimensional position vector of the l joint point for the first frame of data Initialize and set To Covariance matrix Initialize and set The unit matrix equal to 3*3, and this 3*3 unit matrix is denoted as I; set The measurement noise covariance is Let Equal to 3*I; set The measurement noise covariance is Let Equal to 3*I; set the measurement matrix of any human body bone joint point of the a-th frame data as H a , let H a = I, where * is the multiplication operator; Step 3: Perform data filtering and fusion processing. The specific process is as follows: S3.1: Set the frame number variable as k, initialize k, and let k = 2; S3.

2. Three-dimensional position vector of the l-th joint point of the k-th frame data obtained by the main camera Mapping vector of the l joint point Processed to obtain the estimated value of the global three-dimensional position vector of the l joint points of the k-th frame data The specific process is as follows: S3.2.

1. Set the state transition matrix of the k-th frame data as F k , and let F k be equal to I; Set the process noise covariance matrix of the k-th frame data as Q k , and let Q k be equal to 0.2 * I; Set the predicted value of the global three-dimensional position vector of the l-joint point of the k-th frame data as Settings The covariance matrix is S3.2.

2. Calculate respectively using formulas (1) and (2) to obtain With In Equation (2), the subscript T represents the transpose symbol of the matrix; S3.2.

3. Setting With The square of the Mahalanobis distance is Calculated using Equation (3) In formula (3), S3.2.

4. Setting With The square of the Mahalanobis distance is Settings and The square of the Mahalanobis distance is Calculated respectively using equations (4) and (5) to obtain And In formula (4), In formula (5), S3.2.

5. Set the confidence threshold of the l joint point to χ l , χ l takes a value greater than or equal to 10 and less than or equal to 20; S3.2.

6. First, and Compare with χ respectively l Then process them respectively according to the comparison results, specifically: When the comparison result satisfies or And When..., the processing process is: A1. Calculate the estimated value of the local three-dimensional position vector of the l joint point from the k-th frame data of the camera by using equations (6), (7) and (8) respectively With Covariance matrix A2. Use equations (9), (10), and (11) to calculate the estimated value of the local three-dimensional position vector of the l joint point of the k-th frame data of the main camera respectively With Covariance matrix A3: Enter step S3.2.7; When the comparison result satisfies and And When..., the specific processing process is: B1. Calculate the estimated value of the local three-dimensional position vector of the l joint point of the k-th frame data collected from the camera by using equations (6), (7), and (8) respectively With Covariance matrix B2. Judgment Whether it holds. If it holds, let the estimated value of the local three-dimensional position vector of the l joint point of the k-th frame data of the main camera Covariance matrix Then, proceed to step S3.2.7; if not, the three-dimensional position vector of the l joint point of the k-th frame data collected by the main camera Continue with progressive filtering. The specific progressive filtering process is: B2.

1. Set the iterative variable of the progressive filtering as t, set the maximum number of iterative steps M = 10, and set the state variables in the progressive filtering and B2.

2. Initialize t, set t = 1, for And Perform initialization and set B2.3: Perform the t-th iterative filtering. The specific process is: B2.3.

1. Calculate the intermediate state variables of the t-th generation using equations (12) to (16). The covariance matrix of the intermediate state of the t-th generation and the intermediate judgment variable of the t-th generation In the above formula, B2.3.

2. Judgment Whether it holds. If it holds, then let Then proceed to step S3.2.7; if not, further determine whether the current value of t is equal to M. If not, update the value of t with the sum of the current value of t plus 1, and then return to step B2.3 for the next iterative filtering. If equal, then let Then enter step S3.2.7; When the comparison result satisfies And and When..., the specific processing process is: C1. Calculate the estimated value of the local three-dimensional position vector of the l joint point of the k-th frame data collected by the main camera by using equations (9), (10) and (11) respectively And Covariance matrix C2. Judgment Whether it holds. If it holds, let the estimated value of the local three-dimensional position vector of the l-th joint point of the k-th frame data of the camera Covariance matrix Then, proceed to step S3.2.7; otherwise, for the mapping vector of the l joint points of the k-th frame data from the camera Continue with progressive filtering. The specific progressive filtering process is: C2.

1. Set the iteration variable in the progressive filtering as n, the maximum number of iteration steps N = 10, and set the state variable in the progressive filtering And C2.

2. Initialize n and set n = 1; For and Perform initialization and set C2.3: Perform the n-th iterative filtering. The specific process is: C2.3.

1. Using equations (17) to (21), the n-th generation intermediate state variables in the progressive filtering from the camera are calculated The covariance matrix of the n-th generation intermediate state and the nth generation intermediate judgment variable In the above formula, C2.3.

2. Judgment Is it established? If so, then let Then go to step S3.2.7; if not, further determine whether the current value of n is equal to N. If not, first update the value of n with the sum of the current value of n plus 1, and then return to step C2.3 for the next iterative filtering. If it is equal, then let Then enter step S3.2.7; When the comparison result meets and And When, directly let Then enter step S3.2.7; S3.2.

7. Calculate the estimated value of the global three-dimensional position vector of the l joint point of the k-th frame data respectively using Equations (22) and (23). With Covariance matrix S3.

3. Determine whether the current value of k is equal to Y. If not, update the value of k with the sum of the current value of k plus 1, and then return to step S3.2 to process the next frame of data. If it is equal, the human body posture evaluation ends. That is the obtained human body pose data.

Images