Short-baseline binocular three-dimensional human body posture reconstruction method and system

Abstract

The invention provides a short-baseline binocular three-dimensional human body posture reconstruction method and system, and the method comprises the steps: collecting original left and right image pairs, and carrying out the detection of a human body region; carrying out human body image pair matching and scale transformation on the detected human body region; parallax features are obtained based on the processed left and right human body area image pairs, feature fusion is carried out on the parallax features and human body features extracted by the three-dimensional human body posture estimation sub-network, a parallax image initial result is obtained based on feature fusion information, a loss function is established with the human body features to carry out parallax optimization, and an optimized parallax image and a scene depth image are obtained; human body features are obtained based on left and right human body area image pairs, feature fusion is carried out on the human body features and parallax features, two-dimensional human body joint points are obtained through two-dimensional posture estimation according to feature fusion results, and a three-dimensional human body posture initial result is obtained based on binocular camera parameters and a triangulation calculation method; and establishing a loss function with the scene depth map and human body prior supervision, and performing three-dimensional human body posture optimization to obtain an optimized three-dimensional human body posture.

Description

Technical Field

[0001] The present invention relates to the technical effect of three-dimensional human pose reconstruction. Specifically, it relates to a short-baseline binocular three-dimensional human pose reconstruction method and system. More specifically, it relates to a short-baseline binocular three-dimensional human pose reconstruction method based on stereo matching and human appearance feature fusion. Background Art

[0002] With the rapid development of related technologies in the field of artificial intelligence in recent years, stereo matching, as one of the key technologies for multiple computer vision tasks such as scene understanding and 3D scene reconstruction, and autonomous driving perception systems, has been widely studied and applied. The stereo matching algorithm is a scene depth estimation algorithm based on binocular vision sensing. Its core principle is to establish a mathematical relationship between the offset of the same pixel points in the left and right images and the depth of the pixel point based on the epipolar geometric constraint of the images captured by the binocular vision system. Given a pair of stereo images, the key to stereo matching is to calculate the horizontal offset, i.e., the disparity d, between a pixel on the left image and the corresponding pixel on the right image. Then, the depth Z of the pixel can be obtained through formula (1), where f is the focal length of the binocular camera and B is the distance between the centers of the binocular cameras:

[0003]

[0004] Traditional depth estimation methods are mainly divided into four modules: matching cost calculation, matching cost aggregation, disparity calculation and optimization, and disparity improvement. Among them, the matching cost calculation is a process of calculating the similarity cost between all pixel points within a certain offset in the right image and the left pixel point with the left pixel point as the reference. The smaller the cost, the more likely it is a matching point. The matching cost aggregation is to consider the matching cost situation of adjacent pixel points in the neighborhood of the pixel point in the left image, and aggregate a better matching cost matrix based on constraints such as depth continuity in the real scene. The disparity calculation and optimization is a process of calculating the disparity of each pixel point in the left image based on the result of the matching cost aggregation. The disparity improvement is a process of denoising, smoothing, etc. according to the global effect of the obtained disparity map to make it closer to the actual scene depth distribution.

[0005] With the emergence of deep learning algorithms that can extract image features from the original image more effectively and utilize data more fully, the binocular scene depth estimation method based on deep learning has gradually replaced the leading position of traditional methods in terms of performance. In the prior art, it is proposed to first use a convolutional neural network to replace the matching cost calculation and cost aggregation modules of traditional binocular depth estimation. By generating positive and negative samples from the left image patch and the corresponding right image patch and its neighborhood pixel patches for training, during the depth estimation process, the pre-network first divides the left and right images into small image patches, calculates the matching cost between the left image patch and the right image patches within a certain range respectively, and takes the point with the lowest cost as the corresponding matching point.

[0006] The binocular scene depth estimation method based on deep learning has formed a transition from the four-step traditional method to the method of feature extraction by 2D convolutional network, disparity feature aggregation, and disparity calculation and optimization by 3D convolutional network. Due to the limited scenes in the currently public datasets, the training data of deep learning networks mostly focus on virtual scenes and real driving scenes, and the generalization performance for indoor scenes dominated by human movements is poor.

[0007] Three-dimensional human pose estimation is an important task in the field of computer vision and has a wide range of applications in fields such as human-computer interaction, intelligent monitoring, intelligent motion analysis, and medical rehabilitation. Three-dimensional human pose estimation is to recover the three-dimensional pose information of the human body from the given input image or video. Traditional methods are mainly divided into generative algorithms and discriminative algorithms based on template matching. The model-based generative method mainly considers using a model to represent the human body structure and models the spatial relationship between adjacent parts through the constraints between human body structures; the discriminative method regards the pose estimation task as a regression problem, maps the input image to the feature space through a manually designed feature extraction algorithm, and then learns the mapping from the feature space to the three-dimensional human pose space.

[0008] Three-dimensional human pose estimation based on deep learning can be divided into one-stage and two-stage algorithms in terms of algorithm structure. The one-stage algorithm directly estimates the coordinates of three-dimensional joint points from the input RGB image to achieve end-to-end three-dimensional pose estimation; the two-stage algorithm usually extracts features from the input image first, predicts the two-dimensional coordinates or other information of the joint points in the pixel space in the image, and then further realizes the transformation from the intermediate result to the three-dimensional space, so as to finally estimate the three-dimensional pose.

[0009] Three-dimensional human pose estimation based on deep learning can be divided into monocular, binocular, and multi-view algorithms in terms of the number of sensors. Compared with the monocular input algorithm, the multi-view input algorithm can make more full use of the geometric information of multi-view images and obtain more accurate results in the three-dimensional human pose estimation. As the simplest case of multi-view input, the three-dimensional human pose estimation based on binocular input is easier to deploy and use in practical application scenarios such as embedded systems on the basis of having certain geometric information of the input image and due to the relatively simple sensor structure.

[0010] Most existing stereo matching methods based on deep learning adopt a data-driven training method, using public datasets for training. Currently, the main public datasets for stereo matching are virtual scenes and real driving scenes, resulting in poor generalization performance of existing methods for indoor motion scenes dominated by humans. To solve this problem, the present invention fuses stereo matching and human appearance features, and uses two-dimensional human pose estimation results and human prior information to correct and optimize stereo matching, so that the present invention meets the actual usage requirements in indoor motion scenes dominated by humans: the stereo matching performance with bad disparity matching points (pixel points with a matching error greater than 3 pixels) less than 10% of all pixel points. Summary of the Invention

[0011] Aiming at the defects in the prior art, the purpose of the present invention is to provide a short-baseline binocular three-dimensional human pose reconstruction method and system.

[0012] A short-baseline binocular three-dimensional human pose reconstruction method provided by the present invention includes:

[0013] Step S1: Collect the original left and right image pairs through a binocular vision sensor, and perform human region detection on the collected original left and right image pairs through a target detection network;

[0014] Step S2: Perform human image pair matching and scale transformation on the human regions detected from the original left and right image pairs to obtain the processed left and right human region image pairs;

[0015] Step S3: Based on the left and right human region image pairs, obtain disparity features through the feature extraction module of the stereo matching sub-network, fuse the features with the human features extracted by the three-dimensional human pose estimation sub-network to obtain feature fusion information, the feature fusion information obtains the initial result of the disparity map through the disparity calculation module, and establishes a loss function with the human features extracted by the three-dimensional human pose estimation sub-network for disparity optimization training to obtain the optimized disparity map and scene depth map;

[0016] Step S4: Based on the left and right human region image pairs, obtain human features through the feature extraction module of the three-dimensional human pose estimation sub-network, fuse the features with the disparity features extracted by the stereo matching sub-network to obtain a feature fusion result, the feature fusion result obtains two-dimensional human joint points through two-dimensional pose estimation, and obtains the initial result of the three-dimensional human pose based on the binocular camera parameters and the triangulation calculation method, and establishes a loss function with the scene depth map and human prior supervision for three-dimensional human pose optimization training to obtain the optimized three-dimensional human pose.

[0017] Preferably, the step S2 includes:

[0018] Step S2.1: If multiple human body regions are detected in the original left and right images, pair them according to the vertex coordinate detection results of different human body regions in the original left and right images to form multiple groups of human body image pairs;

[0019] Step S2.2: For the human body region object detection results of a group of human body image pairs, calculate the pixel areas of the corresponding human body regions in the left image and the right image in the two-dimensional pixel plane respectively. The human body region object detection result of the image with the smaller area is covered by the human body region object detection result of the image with the larger area;

[0020] Step S2.3: Calculate the length and width of the rectangular vertices of the human body regions in the left and right images, and align the shorter side of the rectangle with the longer side in pixel units, so that the human body detection regions in the left and right images become squares;

[0021] Step S2.4: Scale the regions cropped from the original images according to the human body detection vertex coordinates of the left and right images to the fixed size required by the input of the stereo matching sub-network and the three-dimensional human pose estimation sub-network to obtain the scaled left and right human body region image pairs.

[0022] Preferably, the said step S3 includes:

[0023] Step S3.1: The processed left and right human body region image pairs are subjected to feature extraction through the feature extraction module of the stereo matching sub-network with shared weights to obtain the disparity features of the left and right human body regions; among them, the processed left and right human body region image pairs are denoted in the matrix data format of (2, w, h, 3); where 2 represents two images, one left and one right; w = h, representing the pixel unit length of the length and width of the square human body image pair; 3 represents the three-channel data of the color RGB image; the disparity feature information of the left and right human body regions is denoted in the matrix data format of (2, w / 2, h / 2, C), where w / 2 and h / 2 represent that the length and width pixel unit lengths of the disparity feature information are 1 / 2 of the input left and right human body region image pixels, and C is the number of channels after the disparity feature information passes through multiple two-dimensional convolutional neural networks;

[0024] Step S3.2: Concatenate the matrix data of the human body features (2, w / 2, h / 2, C) extracted by the three-dimensional human pose estimation sub-network with the disparity feature information in the channel number dimension to obtain the matrix data of (2, w / 2, h / 2, 2C), and then reduce the number of channels through a layer of two-dimensional convolutional neural network to perform further feature fusion to obtain the feature fusion information of (2, w / 2, h / 2, C);

[0025] Step S3.3: Divide the feature fusion information (2, w / 2, h / 2, C) into the left image feature fusion information (w / 2, h / 2, C) and the right image feature fusion information (w / 2, h / 2, C). Perform a pixel translation operation on the right image feature fusion information in the pixel w dimension with pixel units ranging from 0 to (D / 2 - 1); where D is the disparity search range of the left and right human body image regions. The translation operation is performed D / 2 times to obtain matrix data of the right image feature fusion information as (w / 2, h / 2, C, D / 2). Subtract the left image feature fusion information from the right image feature fusion information to obtain the disparity cost aggregation information of (w / 2, h / 2, C, D / 2). Further reduce the dimensionality of the channel number dimension and the disparity D dimension of the disparity cost aggregation information to obtain matrix data of (w / 2, h / 2, C×D / 2).

[0026] Step S3.4: Use the (w / 2, h / 2, C×D / 2) disparity cost aggregation information as the input, pass through a multi-layer two-dimensional convolutional neural network to obtain the disparity calculation result of (w / 2, h / 2, D / 2). After bilinear interpolation upsampling, obtain the original resolution disparity calculation result of (w, h, D). Normalize and perform weighted averaging on the values in the D disparity dimension of this result to finally obtain the disparity calculation map of (w, h). Each value in the matrix represents the disparity value of the pixel point in the left image relative to the pixel matching point in the right image at the original resolution.

[0027] Step S3.5: Use the result of two-dimensional human pose estimation combined with the human prior in the real scene to optimize the disparity calculation map.

[0028] Preferably, the step S3.5 includes: During the process of using the result of two-dimensional human pose estimation combined with the human prior in the real scene to optimize the disparity calculation map, regard a certain limb L of the two-dimensional human skeleton constructed by the two-dimensional human pose estimation key points as a rigid body structure, and the depth change gradient along the human bone direction should remain unchanged. Construct the network loss function according to the gradient distribution of the disparity estimation result in the human bone region and the gradient distribution of the ideal bone as follows:

[0029] Loss = abs(P x1 - P x2 ) (x1, x2 ∈)

[0030] where abs represents taking the absolute value, P is the first-order derivative map of the disparity calculation map, R is the pixel region composed of pixel points in the left image human detection region whose pixel distance from the two-dimensional human skeleton constructed by the two-dimensional human pose estimation key points is less than a certain threshold, and x1 and x2 are any two adjacent points in the w direction of the image within the R pixel region;

[0031] During the training process of the sub-network, this loss function is used as the disparity optimization loss function for supervised learning, achieving the effect of optimizing the disparity calculation graph using the results of two-dimensional human pose estimation; finally, the optimized disparity calculation graph (w, h) is output as the calculation result of the scene depth map.

[0032] Preferably, step S4 includes:

[0033] Step S4.1: The processed left and right human body region image pairs are subjected to feature extraction through the feature extraction module of the three-dimensional human pose estimation sub-network with shared weights to obtain the key point feature information of the left and right two-dimensional human poses; among them, the processed left and right human body region image pairs are denoted as matrix data in the format of (2, w, h, 3); where 2 represents two images, one on the left and one on the right; w = h, representing the pixel unit length of the length and width of the square human body image pair; 3 represents the three-channel data of the color RGB image; the key point feature information of the left and right two-dimensional human poses is denoted as matrix data in the format of (2, w / 2, h / 2, C), 2 represents two images, one on the left and one on the right; w / 2 and h / 2 represent that the length and width pixel unit length of the disparity feature information is 1 / 2 of the input left and right human body region image pixels, and C is the number of channels after the disparity feature information passes through multiple two-dimensional convolutional neural networks;

[0034] Step S4.2: The matrix data of the feature extraction result (2, w / 2, h / 2, C) of the two-dimensional human pose estimation is concatenated with the disparity feature information of (2, w / 2, h / 2, C) in the channel number dimension to obtain a matrix data format of (2, w / 2, h / 2, 2C), and then the number of channels is reduced through a layer of two-dimensional convolutional neural network for further feature fusion to obtain the feature fusion information of (2, w / 2, h / 2, C);

[0035] Step S4.3: The feature fusion information of (2, w / 2, h / 2, C) is passed through multiple two-dimensional convolutional neural networks to obtain a joint heat map with a size of (2, w / 4, h / 4, N), where N is the number of human skeleton joints, and each value on it represents the probability that the pixel point is the corresponding human skeleton joint at 1 / 4 of the original image size. Subsequently, the joint heat map is normalized and weighted averaged in the w dimension and h dimension of the image respectively to obtain the human skeleton joints of (2, 2, N), representing the (x, y) coordinates of N human key points in the image coordinate system of the left and right human body image pairs;

[0036] Step S4.4: Using the human skeleton key point coordinates of (2, 2, N) and the binocular camera calibration parameters as inputs, the (3, N) three-dimensional human pose coordinates are calculated through triangulation, representing the (x, y, z) three-dimensional coordinates of the real human key points in the world coordinate system during binocular camera calibration;

[0037] Step S4.5: Using the scene depth map and human prior information, optimize the 3D human key point coordinates using the 3D human pose estimation sub-network.

[0038] Preferably, the step S4.5 includes:

[0039] During the training process, construct a loss function using the depth dimension coordinate of the 3D human pose coordinate and the corresponding depth value of the optimized scene depth map as follows:

[0040] Loss1 = abs(Z x -V x ) (x ∈ N)

[0041] where x is any one of the N 3D human bone key points, Z is the Z-axis coordinate value of this key point in the world coordinate system, and V is the depth value of the corresponding coordinate of the 2D key point of the left image of this key point in the scene depth map;

[0042] In addition, the human bone has the prior that the lengths of symmetric limbs such as the left and right arms and the left and right legs are equal. Therefore, construct the following loss function:

[0043] Loss2 = abs(L m -L n ) (m, n ∈ K)

[0044] where K is the set composed of human symmetric limbs, m and n are any two symmetric limbs in the human symmetric limb set, and L is the limb length calculated according to the 3D human key point coordinates;

[0045] The above loss function is weighted and added to the loss function formed by the ground truth provided by the training set during training to obtain the final loss function; the final network output is the 3D human pose optimization result;

[0046] During training, the stereo matching sub-network and the 3D human pose estimation sub-network are respectively pre-trained based on a public dataset, and then co-trained to achieve the effect of optimizing the 3D human key point coordinates using the scene depth map and human prior information.

[0047] A short-baseline binocular 3D human pose reconstruction system according to the present invention includes:

[0048] Module M1: Collect the original left and right image pairs through a binocular vision sensor, and perform human region detection on the collected original left and right image pairs through a target detection network;

[0049] Module M2: Perform human image pair matching and scale transformation on the human regions detected from the original left and right image pairs to obtain the processed left and right human region image pairs;

[0050] Module M3: Based on the left and right human body region image pairs, obtain the disparity features through the feature extraction module of the stereo matching sub-network, perform feature fusion with the human body features extracted by the three-dimensional human body pose estimation sub-network to obtain the feature fusion information. The feature fusion information obtains the initial result of the disparity map through the disparity calculation module, establishes a loss function with the human body features extracted by the three-dimensional human body pose estimation sub-network for disparity optimization training, and obtains the optimized disparity map and scene depth map;

[0051] Module M4: Based on the left and right human body region image pairs, obtain the human body features through the feature extraction module of the three-dimensional human body pose estimation sub-network, perform feature fusion with the disparity features extracted by the stereo matching sub-network to obtain the feature fusion result. The feature fusion result obtains the two-dimensional human body joint points through two-dimensional pose estimation, and obtains the initial result of the three-dimensional human body pose based on the binocular camera parameters and the triangulation calculation method. Establish a loss function with the scene depth map and human body prior supervision for three-dimensional human body pose optimization training, and obtain the optimized three-dimensional human body pose.

[0052] Preferably, the module M2 includes:

[0053] Module M2.1: If multiple human body regions are detected in the original left and right images, pair them according to the vertex coordinate detection results of different human body regions in the original left and right images to form multiple groups of human body image pairs;

[0054] Module M2.2: For the human body region object detection results of a group of human body image pairs, calculate the pixel areas of the corresponding human body regions in the left image and the right image in the two-dimensional pixel plane respectively. The human body region object detection result of the image with the smaller area is covered by the human body region object detection result of the image with the larger area;

[0055] Module M2.3: Calculate the length and width of the rectangular vertices of the human body regions in the left and right images, and align the shorter side of the rectangle with the longer side in pixel units to make the human body detection regions in the left and right images into squares;

[0056] Module M2.4: Scale the regions cropped from the original images according to the human body detection vertex coordinates of the left and right images to the fixed size required by the input of the stereo matching sub-network and the three-dimensional human body pose estimation sub-network to obtain the scaled left and right human body region image pairs.

[0057] Preferably, the module M3 includes:

[0058] Module M3.1: The processed left and right human body region image pair undergoes feature extraction through the feature extraction module of the stereo matching sub-network with shared weights to obtain the disparity features of the left and right human body regions. Among them, the processed left and right human body region image pair is denoted as a matrix data format of (2, w, h, 3). Here, 2 represents two images, one on the left and one on the right; w = h represents the pixel unit length of the length and width of the square human body image pair; 3 represents the three-channel data of the color RGB image. The disparity feature information of the left and right human body regions is denoted as a matrix data format of (2, w / 2, h / 2, C), where w / 2 and h / 2 represent that the length and width pixel unit length of the disparity feature information is 1 / 2 of the input left and right human body region image pixel pair, and C is the number of channels after the disparity feature information passes through multiple two-dimensional convolutional neural networks.

[0059] Module M3.2: Concatenate the human body features (2, w / 2, h / 2, C) in matrix data format extracted by the three-dimensional human body pose estimation sub-network with the disparity feature information in the channel number dimension to obtain matrix data of (2, w / 2, h / 2, 2C), and then reduce the number of channels through a layer of two-dimensional convolutional neural network for further feature fusion to obtain the feature fusion information of (2, w / 2, h / 2, C).

[0060] Module M3.3: Divide the feature fusion information (2, w / 2, h / 2, C) into the left image feature fusion information (w / 2, h / 2, C) and the right image feature fusion information (w / 2, h / 2, C). Perform a pixel translation operation on the right image feature fusion information in the pixel w dimension with pixel units ranging from 0 to (D / 2 - 1). Here, D is the disparity search range of the left and right human body image regions. The translation operation is performed D / 2 times to obtain the right image feature fusion information as a matrix data of (w / 2, h / 2, C, D / 2). Subtract the left image feature fusion information from the right image feature fusion information to obtain the disparity cost aggregation information of (w / 2, h / 2, C, D / 2). Further reduce the dimension of the channel number dimension and the disparity D dimension of the disparity cost aggregation information to obtain a matrix data of (w / 2, h / 2, C×D / 2).

[0061] Module M3.4: Use the (w / 2, h / 2, C×D / 2) disparity cost aggregation information as the input, pass through multiple two-dimensional convolutional neural networks to obtain the disparity calculation result of (w / 2, h / 2, D / 2), and through bilinear interpolation upsampling, obtain the original resolution disparity calculation result of (w, h, D). Normalize and perform weighted average on the values in the D disparity dimension of this result to finally obtain the disparity calculation map of (w, h). Each value in the matrix represents the disparity value of the pixel point in the left image relative to the pixel matching point in the right image at the original resolution.

[0062] Module M3.5: Optimize the disparity calculation map by combining the results of two-dimensional human pose estimation with human priors in the real scene;

[0063] The said module M3.5 includes: In the process of optimizing the disparity calculation map by combining the results of two-dimensional human pose estimation with human priors in the real scene, a certain limb L of the two-dimensional human skeleton constructed by the key points of two-dimensional human pose estimation is regarded as a rigid body structure, and the depth change gradient along the human bone direction should remain unchanged. According to the gradient distribution of the disparity estimation results in the human bone area and the gradient distribution of the ideal bone, the network loss function is constructed as follows:

[0064] Loss = abs(P x1 -P x2 ) (x1,x2 ∈ R)

[0065] where abs represents taking the absolute value, P is the first-order derivative map of the disparity calculation map, R is the pixel area composed of pixel points in the human detection area of the left image whose pixel distance from the two-dimensional human skeleton constructed by the key points of two-dimensional human pose estimation is less than a certain threshold, and x1 and x2 are any two adjacent points in the w direction of the image within the R pixel area;

[0066] During the training process of the sub-network, use this loss function as the disparity optimization loss function for supervised learning to achieve the effect of optimizing the disparity calculation map using the results of two-dimensional human pose estimation; finally, output the optimized disparity calculation map (w, h) as the calculation result of the scene depth map.

[0067] Preferably, the said module M4 includes:

[0068] Module M4.1: The processed left and right human region image pairs are subjected to feature extraction through the feature extraction module of the three-dimensional human pose estimation sub-network with shared weights to obtain the key point feature information of the left and right two-dimensional human poses; among them, the processed left and right human region image pairs are recorded in the matrix data format of (2, w, h, 3); where 2 represents two images, one left and one right; w = h, representing the pixel unit length of the length and width of the square human image pair; 3 represents the three-channel data of the color RGB image; the key point feature information of the left and right two-dimensional human poses is recorded in the matrix of (2, w / 2, h / 2, C), 2 represents two images, one left and one right; w / 2 and h / 2 represent that the length and width pixel unit length of the disparity feature information is 1 / 2 of the input left and right human region image pixels, and C is the number of channels after the disparity feature information passes through multiple two-dimensional convolutional neural networks;

[0069] Module M4.2: Concatenate the matrix data of the feature extraction results (2, w / 2, h / 2, C) of 2D human pose estimation with the disparity feature information of (2, w / 2, h / 2, C) in the channel dimension to obtain a matrix data format of (2, w / 2, h / 2, 2C), and then reduce the number of channels through a two-dimensional convolutional neural network for further feature fusion to obtain the feature fusion information of (2, w / 2, h / 2, C);

[0070] Module M4.3: Pass the feature fusion information of (2, w / 2, h / 2, C) through a multi-layer two-dimensional convolutional neural network to obtain a joint heatmap with a size of (2, w / 4, h / 4, N), where N is the number of human skeleton joints. Each value on it represents the probability that the pixel point is the corresponding human skeleton joint at 1 / 4 of the original image size. Subsequently, perform normalization and weighted averaging on the joint heatmap in the w dimension and h dimension of the image respectively to obtain the human skeleton joints of (2, 2, N), representing the (x, y) coordinates of N human key points in the image coordinate system of the left and right human image pairs;

[0071] Module M4.4: Using the human skeleton key point coordinates of (2, 2, N) and the binocular camera calibration parameters as inputs, calculate the three-dimensional human pose coordinates of (3, N) through triangulation, representing the three-dimensional coordinates (x, y, z) of the real human key points in the world coordinate system during binocular camera calibration;

[0072] Module M4.5: Use the scene depth map and human prior information to optimize the three-dimensional human key point coordinates using the three-dimensional human pose estimation sub-network;

[0073] The said module M4.5 includes:

[0074] During the training process, construct a loss function using the depth dimension coordinate of the three-dimensional human pose coordinates and the corresponding depth value of the optimized scene depth map as follows:

[0075] Loss1 = abs(Z x - V x ) (x ∈ N)

[0076] where x is any one of the N three-dimensional human skeleton key points, Z is the Z-axis coordinate value of this key point in the world coordinate system, and V is the depth value of the corresponding coordinate of the two-dimensional key point of the left image of this key point in the scene depth map;

[0077] In addition, since the human skeleton has the prior that the lengths of symmetric limbs such as the left and right arms and the left and right legs are equal, the following loss function is constructed:

[0078] Loss2 = abs(L m - L n ) (m, n ∈ K)

[0079] Among them, \(K\) is the set composed of symmetric limbs of the human body, \(m\) and \(n\) are any two symmetric limbs in the set of symmetric limbs of the human body, and \(L\) is the limb length calculated according to the three-dimensional human key point coordinates;

[0080] The above loss function is weighted and added to the loss function formed by providing the true value according to the training set during training to obtain the final loss function; the final network output is the optimized result of the three-dimensional human body pose;

[0081] During training, the stereo matching sub-network and the three-dimensional human body pose estimation sub-network are pre-trained based on the public data set respectively, and then co-trained to achieve the effect of optimizing the three-dimensional human body key point coordinates using the depth map of the usage scenario and the human body prior information.

[0082] Compared with the prior art, the present invention has the following beneficial effects:

[0083] 1. By fusing stereo matching and human body appearance features, the present invention uses the two-dimensional human body pose estimation result and the human body prior information to correct and optimize the stereo matching, so as to meet the actual use requirements in the indoor sports scenario mainly composed of the human body: the stereo matching performance with the bad points of disparity matching (pixel points with a matching error greater than 3 pixels) less than 10% of all pixel points;

[0084] 2. By combining stereo matching with the three-dimensional human body pose reconstruction method, the present invention uses the depth estimation result of stereo matching and the human body prior information to correct and optimize the three-dimensional human body pose estimation result, so as to meet the actual use requirements in the application scenario of short-baseline binocular three-dimensional human body pose estimation: the performance with an average error of the whole body degrees of freedom less than 5 cm.

[0085] 3. The present invention designs a joint network structure for the stereo matching task and the three-dimensional human body pose estimation task, and makes relatively full use of the association and constraint between the depth information and the three-dimensional pose information in the human body motion scenario; taking the binocular vision sensor that is convenient for deployment as the input, in the indoor scenario mainly composed of human body motion, the method of the present invention can quickly and accurately estimate the depth information of human body motion and the three-dimensional human body pose estimation information. Brief Description of the Drawings

[0086] By reading the detailed description of the non-restrictive embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0087] Figure 1 It is a flow chart of the short-baseline binocular three-dimensional human body pose reconstruction method.

[0088] Figure 2 It is a schematic diagram of the overall structure of the stereo matching sub-network.

[0089] Figure 3 It is a schematic diagram of the overall structure of the 3D human pose estimation sub-network. Specific implementation manners

[0090] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those of ordinary skill in the art can make several changes and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0091] Embodiment 1

[0092] According to a short-baseline binocular 3D human pose reconstruction method and system provided by the present invention, based on the research of stereo matching and 3D human pose reconstruction, the pre-stage 2D human pose estimation of stereo matching (binocular scene depth estimation) and 3D human pose reconstruction has similarities, including: both are essentially pixel classification tasks in the image 2D pixel coordinate system. Stereo matching calculates the disparity by extracting features and calculating which pixel point in the left image and which pixel point in the right image belong to the same class (i.e., the same point). 2D human pose estimation calculates which human key point category or background category each pixel point in the image belongs to by extracting features. The structures of their feature extraction modules are also similar; the feature extraction information and the final calculation results of the two can theoretically be complementary. The disparity calculation result can be used to calculate the depth result through a formula, and the 2D human pose estimation result can be used to calculate the 3D human pose reconstruction result through triangulation. One of the three dimensions of the 3D pose is also the depth information in the camera coordinate system. Therefore, when there are individual pixel point estimation errors in one of them, information can be obtained from the estimation result of the other to correct.

[0093] Therefore, in this embodiment, the essence of the short-baseline binocular three-dimensional human pose reconstruction method and system is to establish the collaboration between the stereo matching network and the three-dimensional human pose reconstruction network. First, it is to establish the feature information fusion of the feature extraction results of the stereo matching network and the feature extraction results of the two-dimensional human pose estimation. Because the feature extraction results of the stereo matching network pay more attention to the global and local color features and texture features of the pixel points in the image, while the feature extraction results of the two-dimensional human pose estimation pay more attention to whether the pixel points show the features of the human-like structure locally. On the image with the human body as the main image content, the feature extraction results of the two can complement each other in terms of feature information. Secondly, it is to establish the mutual verification between the stereo matching result and the three-dimensional human pose reconstruction result. In the optimization training of the network, a loss function is established with the relatively stable two-dimensional human pose estimation ground truth and the disparity optimization result to reduce the error of the disparity optimization result. In addition, since the disparity optimization result is the depth result obtained by fusing the global and local features of the pixel points in the depth dimension, a loss function can be established with the depth dimension of the three-dimensional human pose reconstruction result to reduce the error of the three-dimensional human pose reconstruction result in the depth dimension.

[0094] The short-baseline binocular three-dimensional human pose reconstruction method includes:

[0095] Step S1: Collect the original left and right image pairs through a binocular vision sensor, and perform human region detection on the collected original left and right image pairs through an object detection network;

[0096] Step S2: Perform human image pair matching and scale transformation on the human regions detected from the original left and right image pairs to obtain the processed left and right human region image pairs;

[0097] Step S3: Obtain the disparity feature through the feature extraction module of the stereo matching sub-network based on the left and right human region image pairs, fuse the feature with the human feature extracted by the three-dimensional human pose estimation sub-network to obtain the feature fusion information. The feature fusion information obtains the initial result of the disparity map through the disparity calculation module, and a loss function is established with the human feature extracted by the three-dimensional human pose estimation sub-network for disparity optimization training to obtain the optimized disparity map and the scene depth map;

[0098] Step S4: Obtain the human feature through the feature extraction module of the three-dimensional human pose estimation sub-network based on the left and right human region image pairs, fuse the feature with the disparity feature extracted by the stereo matching sub-network to obtain the feature fusion result. The feature fusion result obtains the two-dimensional human joint points through two-dimensional pose estimation, and obtains the initial result of the three-dimensional human pose based on the binocular camera parameters and the triangulation calculation method. A loss function is established with the scene depth map and the human prior supervision for three-dimensional human pose optimization training to obtain the optimized three-dimensional human pose.

[0099] Among them, the object detection network includes:

[0100] Taking the original left and right image pairs collected by the binocular vision sensor of the object detection network as input, using a general residual network to perform object detection on the human body regions in the images, and outputting the rectangular vertex coordinates in the two-dimensional image coordinate system of the human body regions in the left and right images.

[0101] The general residual network has a certain network depth and is easy to converge during training. Therefore, the general residual network is used as the object detection sub-network. During training, public object detection datasets such as coco are used, and the data labels are divided into two types of objects, namely people and the background, for separate training.

[0102] Specifically, the step S2 includes:

[0103] Step S2.1: Matching. If there are multiple human body regions in the left and right images, then according to the vertex coordinate detection results of different human body regions in the left and right images, they are paired to form multiple groups of human body image pairs, and the subsequent steps calculate with each group of human body image pairs as input.

[0104] Step S2.2: Scale transformation. For the object detection results of the human body regions of a group of human body image pairs, calculate the pixel areas of the corresponding human body regions in the left image and the right image in the two-dimensional pixel plane respectively. The object detection result of the human body region in the image with the smaller area is covered by the object detection result of the human body region in the image with the larger area. For example, if the pixel area of the human detection region in the left image is larger than that in the right image, then replace the rectangular vertex coordinates in the image coordinate system of the human detection region in the right image with the rectangular vertex coordinates of the human body region in the left image.

[0105] Subsequently, calculate the length and width of the rectangular vertex coordinates of the human body regions in the left and right images, and align the shorter side of the rectangle with the longer side in pixel units. Specifically, for example, if the upper left corner of the rectangular vertex coordinates is (x1, y1) and the lower right corner is (x2, y2), if the value of (x2 - x1) is greater than the value of (y2 - y1), then keep the center coordinates of the rectangle unchanged, and y1 and y2 are respectively decreased and increased by the value of ((x2 - x1) - (y2 - y1)) / 2, so that the human body detection regions in the left and right images become squares.

[0106] Finally, scale the regions cropped from the original images according to the human body detection vertex coordinates in the left and right images to the fixed size required for the input of the stereo matching sub-network and the 3D human pose estimation sub-network, such as a pair of human body images with both length and width of 256 pixels, as the subsequent input.

[0107] Specifically, the stereo matching sub-network includes: The overall structure of the stereo matching sub-network is as Figure 2As shown in the figure, compared with the fully connected network, the two-dimensional convolutional network has the advantage of fully extracting the multi-scale feature information of the image in image feature extraction and visual task calculation, and has a smaller computational cost compared with higher-dimensional networks such as three-dimensional convolutional networks. Therefore, the sub-networks of the algorithm of the present invention are all based on the two-dimensional convolutional neural network as the main body of the algorithm.

[0108] Specifically, the step S3 includes:

[0109] Step S3.1: Feature extraction. Take the left and right human body region images with fixed length and width obtained in step S2 as the input, denoted as a matrix data format of (2, w, h, 3), where 2 represents two images, one on the left and one on the right; w = h, representing the pixel unit length of the length and width of the square human body image pair; 3 represents the three-channel data of the color RGB image.

[0110] The left and right human body region data pass through the feature extraction module with shared weights to obtain the disparity feature information of the left and right human body regions, denoted as a matrix data format of (2, w / 2, h / 2, C), where w / 2 and h / 2 represent that the length and width pixel unit length of the disparity feature information is 1 / 2 of the input left and right human body region image pixel pairs, and C is the number of channels after the disparity feature information passes through multiple two-dimensional convolutional neural networks.

[0111] In this embodiment, the dimension of the feature extraction result of two-dimensional human pose estimation is the same as that of the disparity feature information. This is because the structures of the stereo matching feature extraction network and the two-dimensional human pose estimation feature extraction network are highly similar, but the information extracted from the original image by the two is different. The stereo matching feature extraction network extracts the global and local color and texture feature information of the pixels in the image, and the two-dimensional human pose estimation feature extraction network extracts the feature information of whether the pixels show a human-like structure locally. This is due to the use of different ground truths for verification (the disparity ground truth map and the two-dimensional human pose estimation ground truth map respectively) and different loss functions in the pre-training process of the two feature networks.

[0112] Step S3.2: Feature fusion: Concatenate the matrix data of the feature extraction result (2, w / 2, h / 2, C) of two-dimensional human pose estimation with the disparity feature information obtained in step 3.1 in the dimension of the number of channels to obtain a matrix data format of (2, w / 2, h / 2, 2C), and then reduce the number of channels through a layer of two-dimensional convolutional neural network for further feature fusion to obtain the feature fusion information of (2, w / 2, h / 2, C) as the input for the subsequent steps.

[0113] Step S3.3: Cost aggregation: Divide the disparity and human feature fusion information (2, w / 2, h / 2, C) into left image feature fusion information (w / 2, h / 2, C) and right image feature fusion information (w / 2, h / 2, C). Perform pixel translation operations on the right image feature fusion information in the pixel w dimension with pixel units ranging from 0 to (D / 2 - 1), where D is the disparity search range of the left and right human image regions, that is, the default maximum disparity of the left and right image pixels is D, and this value is determined according to the original resolution of the left and right images and the baseline distance of the binocular camera sensor.

[0114] The translation operation is performed D / 2 times to obtain the matrix data format of the right image feature fusion information as (w / 2, h / 2, C, D / 2). At the same time, the left image feature fusion information is copied D / 2 times to obtain the matrix data format of the left image feature fusion information as (w / 2, h / 2, C, D / 2). Subtract the left image feature fusion information from the right image feature fusion information to obtain the disparity cost aggregation information of (w / 2, h / 2, C, D / 2). Further, reduce the dimensionality of the channel number dimension and the disparity D dimension of the disparity cost aggregation information to obtain the matrix data format of (w / 2, h / 2, C×D / 2).

[0115] Step S3.4: Disparity calculation: Using the (w / 2, h / 2, C×D / 2) disparity cost aggregation information obtained in step S3.3 as the input, through a multi-layer two-dimensional convolutional neural network, obtain the disparity calculation result of (w / 2, h / 2, D / 2). After bilinear interpolation upsampling, obtain the original resolution disparity calculation result of (w, h, D). Normalize and weighted average the values in the D disparity dimension of this result to finally obtain the disparity calculation map of (w, h). Each value in the matrix represents the disparity value of the pixel point in the left image relative to the pixel matching point in the right image at the original resolution.

[0116] Step S3.5: Disparity optimization: Use the results of two-dimensional human pose estimation combined with the human prior in the real scene to optimize the disparity result. Specifically, during the training process of the sub-network, regard a certain limb L of the two-dimensional human skeleton constructed by the two-dimensional human pose estimation key points as a rigid body structure, and the depth change gradient along the human bone direction should remain unchanged. Construct the network loss function according to the gradient distribution of the disparity estimation result in the human bone region and the gradient distribution of the ideal bone as follows:

[0117] Loss = abs(P x1 - P x2 ) (x1, x2 ∈ R)

[0118] Where abs represents taking the absolute value, P is the first-order derivative map of the disparity calculation map, R is the pixel region composed of pixel points whose pixel distances from the two-dimensional human body skeleton constructed by the key points of the human body detection region in the left image are less than a certain threshold (such as a certain number of pixels), and x1 and x2 are any two adjacent points in the w direction of the image within the R pixel region.

[0119] During the training process of the sub-network, this loss function is used as the disparity optimization loss function for supervised learning to achieve the effect of optimizing the disparity calculation map using the results of two-dimensional human pose estimation. Finally, the optimized disparity calculation map (w, h) is output as the calculation result of the scene depth map.

[0120] The overall structure of the three-dimensional human pose estimation sub-network is as Figure 3 shown;

[0121] Specifically, step S4 includes:

[0122] Step S4.1: Feature extraction. The two-dimensional human pose estimation takes the left and right human body region images with fixed dimensions of length and width obtained in step S2 as inputs, denoted as a matrix data format of (2, w, h, 3), and through a feature extraction module with shared weights, the key point feature information of the left and right two-dimensional human poses is obtained, denoted as a matrix data format of (2, w / 2, h / 2, C), and the dimensions of each dimension are the same as the corresponding dimensions in step S3.

[0123] Step S4.2: Feature fusion. The matrix data of the feature extraction result (2, w / 2, h / 2, C) of the two-dimensional human pose estimation is concatenated with the disparity feature information of (2, w / 2, h / 2, C) in the channel number dimension to obtain a matrix data format of (2, w / 2, h / 2, 2C), and then the channel number is reduced through a two-dimensional convolutional neural network layer for further feature fusion to obtain the feature fusion information of (2, w / 2, h / 2, C) as the input for the subsequent steps.

[0124] Step S4.3: Two-dimensional human pose estimation. The feature fusion information of (2, w / 2, h / 2, C) is passed through a multi-layer two-dimensional convolutional neural network to obtain a joint point heat map with a size of (2, w / 4, h / 4, N), where N is the number of human body skeleton joint points, and each value on it represents the probability that the pixel point is the corresponding human body skeleton joint point at 1 / 4 of the original image size. Subsequently, the joint point heat map is normalized and weighted averaged in the w dimension and h dimension of the image respectively to obtain the human body skeleton joint points of (2, 2, N), representing the (x, y) coordinates of N human body key points in the image coordinate system of the left and right human body image pairs.

[0125] Step S4.4: 3D human body pose coordinate calculation. Using the human body skeleton key point coordinates of (2, 2, N) and the binocular camera calibration parameters as inputs, the (3, N) 3D human body pose coordinates are calculated through triangulation, representing the (x, y, z) 3D coordinates of the real human body key points in the world coordinate system during binocular camera calibration.

[0126] Step S4.5: 3D human body pose optimization. Use the scene depth map obtained in Step S3 and the human body prior information to optimize the 3D human body key point coordinates obtained in Step S4.4. Specifically, during training, a loss function is constructed using the depth dimension coordinate of the 3D human body pose coordinates and the corresponding depth value of the optimized scene depth map as follows:

[0127] Loss1 = abs(Z x -V x ) (x ∈ N)

[0128] where x is any one of the N 3D human body skeleton key points, Z is the Z-axis coordinate value of this key point in the world coordinate system, and V is the depth value of the corresponding coordinate of the 2D key point of the left image of this key point in the scene depth map.

[0129] In addition, the human body skeleton has the prior that the lengths of symmetric limbs such as the left and right arms and the left and right legs are equal. Therefore, the following loss function is constructed:

[0130] Loss2 = abs(L m -L n ) (m, n ∈ K)

[0131] where K is the set composed of human body symmetric limbs, m and n are any two symmetric limbs in the human body symmetric limb set, and L is the limb length calculated according to the 3D human body key point coordinates.

[0132] The above loss function is weighted and added to the loss function formed by the ground truth provided by the training set during training to obtain the final loss function. The final network output is the 3D human body pose optimization result.

[0133] During training, the stereo matching sub-network and the 3D human body pose estimation sub-network are pre-trained based on public datasets respectively, and then co-trained to achieve the effect of optimizing the 3D human body key point coordinates using the scene depth map and the human body prior information.

[0134] In this embodiment, the stereo matching sub-network and the 3D human pose estimation sub-network can use a two-dimensional convolutional neural network composed of multiple-level encoders and decoders as the network implementation method. Each level of the encoder structure consists of a 3×3 depth convolution + BN layer + ReLU activation function + max pooling layer, and each level of the decoder structure consists of a 3×3 depth convolution + BN layer + ReLU activation function + upsampling layer.

[0135] The KITTI dataset and the publicly available Human3.6m dataset are used for network training. The size of the input left and right image pairs is 256×256 pixels, and the default maximum disparity is set to 192 pixels under the KITTI dataset. During the training process, the weighting coefficient of the true value error loss function formed by the ground truth provided by the dataset and the network output result is 0.8, and the weighting coefficients of the loss functions in steps 3.5 and 4.5 above are 0.2. During training, the stereo matching sub-network and the 3D human pose estimation sub-network are pre-trained based on the publicly available dataset respectively, and then co-trained. Finally, after the training is completed, the network outputs the scene depth map and the 3D human pose estimation result, achieving a stereo matching performance with the bad disparity matching points (pixel points with a matching error greater than 3 pixels) less than 10% of all pixel points and a 3D human pose estimation performance with an average error of the whole body degrees of freedom less than 5 cm on the publicly available dataset.

[0136] The stereo matching sub-network and the 3D human pose estimation sub-network can use a shared feature extraction module with an encoding and decoding structure according to actual application needs, so as to achieve the effect of reducing the network parameter quantity and the consumption of algorithm computing resources.

[0137] In the method for optimizing the stereo matching output disparity map according to the 2D human pose estimation result, in a scenario where the training data is insufficient, post-processing methods such as linear regression and smoothing of the disparity change curve in the human bone region can be used for replacement.

[0138] In the method for optimizing the 3D human pose estimation result according to the stereo matching output disparity map, in a scenario where the training data is insufficient, post-processing methods such as correction of the corresponding points of the 2D human pose estimation and correction of the human prior can be used for replacement.

[0139] The present invention also provides a short-baseline binocular 3D human pose reconstruction system. The short-baseline binocular 3D human pose reconstruction system can be implemented by executing the process steps of the short-baseline binocular 3D human pose reconstruction method, that is, those skilled in the art can understand the short-baseline binocular 3D human pose reconstruction method as the preferred implementation manner of the short-baseline binocular 3D human pose reconstruction system.

[0140] Those skilled in the art know that, in addition to implementing the systems, devices, and their respective modules provided by the present invention in the form of pure computer-readable program code, it is entirely possible to logically program the method steps so that the systems, devices, and their respective modules provided by the present invention are implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. to achieve the same program. Therefore, the systems, devices, and their respective modules provided by the present invention can be considered as a kind of hardware component, and the modules included therein for implementing various programs can also be regarded as the structures within the hardware component; the modules for implementing various functions can also be regarded as either software programs for implementing the method or the structures within the hardware component.

[0141] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A short-baseline binocular three-dimensional human pose reconstruction method, characterized in that Including: Step S1: Collect the original left and right image pairs through a binocular vision sensor, and perform human region detection on the collected original left and right image pairs through a target detection network; Step S2: Perform human image pair matching and scale transformation on the human regions detected from the original left and right image pairs to obtain the processed left and right human region image pairs; Step S3: Obtain a disparity feature through the feature extraction module of the stereo matching sub-network based on the left and right human region image pairs, perform feature fusion with the human feature extracted by the three-dimensional human pose estimation sub-network to obtain feature fusion information, and obtain an initial disparity map result through the disparity calculation module. Establish a loss function with the human feature extracted by the three-dimensional human pose estimation sub-network for disparity optimization training to obtain the optimized disparity map and scene depth map; Step S4: Obtain a human feature through the feature extraction module of the three-dimensional human pose estimation sub-network based on the left and right human region image pairs, perform feature fusion with the disparity feature extracted by the stereo matching sub-network to obtain a feature fusion result, obtain two-dimensional human joint points through two-dimensional pose estimation based on the feature fusion result, and obtain an initial three-dimensional human pose result based on the binocular camera parameters and the triangulation calculation method. Establish a loss function with the scene depth map and human prior supervision for three-dimensional human pose optimization training to obtain the optimized three-dimensional human pose.

2. The short baseline binocular three-dimensional human body pose reconstruction method according to claim 1, wherein The said step S2 includes: Step S2.1: If multiple human regions are detected in the original left and right images, pair them according to the vertex coordinate detection results of different human regions in the original left and right images to form multiple groups of human image pairs; Step S2.2: For the human region target detection results of a group of human image pairs, calculate the pixel areas of the corresponding human regions in the left image and the right image in the two-dimensional pixel plane respectively, and the human region target detection result of the image with the smaller area is covered by the human region target detection result of the image with the larger area; Step S2.3: Calculate the length and width of the rectangular vertices of the human regions in the left and right images, and align the shorter side of the rectangle with the longer side in pixel units to make the human detection regions in the left and right images become squares; Step S2.4: Scale the regions cropped from the original images according to the human detection vertex coordinates of the left and right images to the fixed size required for the input of the stereo matching sub-network and the three-dimensional human pose estimation sub-network to obtain the scaled left and right human region image pairs.

3. The short-baseline binocular three-dimensional human pose reconstruction method according to claim 1, characterized in that The said step S3 includes: Step S3.1: The processed left and right human body region image pairs are subjected to feature extraction through the feature extraction module of the stereo matching sub-network with shared weights to obtain the disparity features of the left and right human body regions. Among them, the processed left and right human body region image pairs are denoted in the matrix data format of (2, w, h, 3). Here, 2 represents two images, one left and one right; w = h, representing the pixel unit length of the length and width of the square human body image pair; 3 represents the three-channel data of the color RGB image. The disparity feature information of the left and right human body regions is denoted in the matrix data format of (2, w / 2, h / 2, C), where w / 2 and h / 2 represent that the length and width pixel unit lengths of the disparity feature information are 1 / 2 of the input left and right human body region image pixels, and C is the number of channels after the disparity feature information passes through multiple two-dimensional convolutional neural networks. Step S3.2: The matrix data of the human body features (2, w / 2, h / 2, C) extracted by the three-dimensional human body pose estimation sub-network is concatenated with the disparity feature information in the channel number dimension to obtain the matrix data of (2, w / 2, h / 2, 2C), and then the number of channels is reduced through a layer of two-dimensional convolutional neural network for further feature fusion to obtain the feature fusion information of (2, w / 2, h / 2, C). Step S3.3: The feature fusion information (2, w / 2, h / 2, C) is divided into the left image feature fusion information (w / 2, h / 2, C) and the right image feature fusion information (w / 2, h / 2, C). The right image feature fusion information is subjected to a pixel translation operation with pixel units of 0 to (D / 2 - 1) in the pixel w dimension. Here, D is the disparity search range of the left and right human body image regions. The translation operation is performed D / 2 times to obtain the matrix data of the right image feature fusion information as (w / 2, h / 2, C, D / 2). The left image feature fusion information is subtracted from the right image feature fusion information to obtain the disparity cost aggregation information of (w / 2, h / 2, C, D / 2). Further, the channel number dimension and the disparity D dimension of the disparity cost aggregation information are reduced in dimension to obtain the matrix data of (w / 2, h / 2, C×D / 2). Step S3.4: Taking the (w / 2, h / 2, C×D / 2) disparity cost aggregation information as the input, through multiple two-dimensional convolutional neural networks, the disparity calculation result of (w / 2, h / 2, D / 2) is obtained. After bilinear interpolation upsampling, the original resolution disparity calculation result of (w, h, D) is obtained. The values in the D disparity dimension of this result are normalized and weighted averaged to finally obtain the disparity calculation map of (w, h). Each value in the matrix represents the disparity value of the pixel point in the left image relative to the pixel matching point in the right image at the original resolution. Step S3.5: Use the result of two-dimensional human body pose estimation combined with the human body prior in the real scene to optimize the disparity calculation map.

4. The short baseline binocular three-dimensional human body pose reconstruction method according to claim 3, wherein, The step S3.5 includes: in the process of optimizing the disparity calculation map by using the results of two-dimensional human pose estimation combined with human priors in the real scene, a certain limb L of the two-dimensional human skeleton constructed by the key points of two-dimensional human pose estimation is regarded as a rigid body structure, and the depth change gradient along the human bone direction should remain unchanged. According to the gradient distribution of the disparity estimation results in the human bone region and the gradient distribution of the ideal bone, the network loss function is constructed as follows: Loss = abs(P x1 - P x2 )(x1, x2 ∈ R) Among them, abs represents taking the absolute value, P is the first-order derivative map of the disparity calculation map, R is the pixel region composed of pixel points in the human detection region of the left image whose pixel distance from the two-dimensional human skeleton constructed by the key points of two-dimensional human pose estimation is less than a certain threshold, and x1 and x2 are any two adjacent points in the w direction of the image within the R pixel region; During the training process of the sub-network, this loss function is used as the disparity optimization loss function for supervised learning to achieve the effect of optimizing the disparity calculation map by using the results of two-dimensional human pose estimation; finally, the optimized disparity calculation map (w, h) is output as the calculation result of the scene depth map.

5. The short baseline binocular three-dimensional human pose reconstruction method according to claim 1, characterized in that, The step S4 includes: Step S4.1: The processed left and right human region image pairs are subjected to feature extraction through the feature extraction module of the three-dimensional human pose estimation sub-network with shared weights to obtain the key point feature information of the left and right two-dimensional human poses; among them, the processed left and right human region image pairs are denoted in the matrix data format of (2, w, h, 3); where 2 represents two images, one left and one right; w = h represents the pixel unit length of the length and width of the square human image pair; 3 represents the three-channel data of the color RGB image; the key point feature information of the left and right two-dimensional human poses is denoted as a matrix of (2, w / 2, h / 2, C), 2 represents two images, one left and one right; w / 2 and h / 2 represent that the length and width pixel unit lengths of the disparity feature information are 1 / 2 of the input left and right human region image pixels, and C is the number of channels after the disparity feature information passes through a multi-layer two-dimensional convolutional neural network; Step S4.2: The matrix data of the feature extraction result (2, w / 2, h / 2, C) of two-dimensional human pose estimation is concatenated with the disparity feature information of (2, w / 2, h / 2, C) in the channel number dimension to obtain a matrix data format of (2, w / 2, h / 2, 2C), and then the number of channels is reduced through a layer of two-dimensional convolutional neural network for further feature fusion to obtain the feature fusion information of (2, w / 2, h / 2, C); Step S4.3: The feature fusion information of (2, w / 2, h / 2, C) passes through a multi-layer two-dimensional convolutional neural network to obtain a joint heat map with a size of (2, w / 4, h / 4, N), where N is the number of human skeleton joints, and each value on it represents the probability that the pixel point is the corresponding human skeleton joint at 1 / 4 of the original image size. Subsequently, the joint heat map is normalized and weighted averaged in the w dimension and h dimension of the image respectively to obtain the human skeleton joints of (2, 2, N), representing the (x, y) coordinates of N human key points in the image coordinate system in the left and right human image pairs; Step S4.4: Using the human skeleton key point coordinates of (2, 2, N) and the binocular camera calibration parameters as inputs, calculate the 3D human pose coordinates of (3, N) through triangulation, which represent the (x, y, z) 3D coordinates of the real human key points in the world coordinate system during binocular camera calibration; Step S4.5: Use the scene depth map and human prior information, and utilize the 3D human pose estimation sub-network to optimize the 3D human key point coordinates.

6. The short baseline binocular three-dimensional human pose reconstruction method according to claim 5, characterized in that The said Step S4.5 includes: During the training process, construct the loss function using the depth dimension coordinate of the 3D human pose coordinates and the corresponding depth value of the optimized scene depth map as follows: Loss1 = abs(Z x -V x )(x ∈ N) where x is any one of the N 3D human skeleton key points, Z is the Z-axis coordinate value of this key point in the world coordinate system, and V is the depth value of the corresponding coordinate of the 2D key point of this key point in the left image in the scene depth map; In addition, the human skeleton has the prior that the symmetric limbs such as the left and right arms and the left and right legs are of equal length. Therefore, construct the following loss function: Loss2 = abs(L m - L n )(m, n ∈ K) where K is the set composed of human symmetric limbs, m and n are any two symmetric limbs in the set of human symmetric limbs, and L is the limb length calculated according to the 3D human key point coordinates; The above loss function and the loss function formed according to the ground truth provided by the training set during training are weighted and added to obtain the final loss function; the final network output is the optimized result of the 3D human pose; During training, the stereo matching sub-network and the 3D human pose estimation sub-network are respectively pre-trained based on the public data set, and then co-trained to achieve the effect of optimizing the 3D human key point coordinates using the scene depth map and human prior information.

7. A short baseline binocular three-dimensional human body pose reconstruction system, characterized in that, It includes: Module M1: Collect the original left and right image pairs through the binocular vision sensor, and perform human region detection on the collected original left and right image pairs through the target detection network; Module M2: Perform human image pair matching and scale transformation on the human regions detected based on the original left and right image pairs to obtain the processed left and right human region image pairs; Module M3: Obtain the disparity feature through the feature extraction module of the stereo matching sub-network based on the left and right human region image pairs, perform feature fusion with the human feature extracted by the 3D human pose estimation sub-network to obtain the feature fusion information, the feature fusion information obtains the initial result of the disparity map through the disparity calculation module, establish a loss function with the human feature extracted based on the 3D human pose estimation sub-network for disparity optimization training, and obtain the optimized disparity map and scene depth map; Module M4: Based on the left and right human region image pairs, obtain the human feature through the feature extraction module of the 3D human pose estimation sub-network, perform feature fusion with the disparity feature extracted by the stereo matching sub-network to obtain the feature fusion result, the feature fusion result obtains the 2D human joint points through 2D pose estimation, obtain the initial result of the 3D human pose based on the binocular camera parameters and the triangulation calculation method, and establish a loss function with the scene depth map and human prior supervision for 3D human pose optimization training to obtain the optimized 3D human pose.

8. The short baseline binocular three-dimensional human body pose reconstruction system according to claim 7, wherein, The said Module M2 includes: Module M2.1: If multiple human body regions are detected in the original left and right images, pair them according to the vertex coordinate detection results of different human body regions in the original left and right images to form multiple groups of human body image pairs; Module M2.2: For the human body region object detection results of a group of human body image pairs, calculate the pixel areas of the corresponding human body regions in the left image and the right image in the two-dimensional pixel plane respectively, and the human body region object detection result of the image with the smaller area is covered by the human body region object detection result of the image with the larger area; Module M2.3: Calculate the length and width of the rectangular vertex coordinates of the human body regions in the left and right images, and align the shorter side of the rectangle with the longer side in pixel units, so that the human body detection regions in the left and right images become squares; Module M2.4: Scale the regions cropped from the original images according to the human body detection vertex coordinates of the left and right images to the fixed size required by the input of the stereo matching sub-network and the three-dimensional human body pose estimation sub-network to obtain the scaled left and right human body region image pairs.

9. The short-baseline binocular three-dimensional human body pose reconstruction system according to claim 7, wherein, The said module M3 includes: Module M3.1: The processed left and right human body region image pairs are subjected to feature extraction through the feature extraction module of the stereo matching sub-network with shared weights to obtain the disparity features of the left and right human body regions; among them, the processed left and right human body region image pairs are recorded in the matrix data format of (2, w, h, 3); where 2 represents two images, one left and one right; w = h, representing the pixel unit length of the length and width of the square human body image pair; 3 represents the three-channel data of the color RGB image; the disparity feature information of the left and right human body regions is recorded in the matrix data format of (2, w / 2, h / 2, C), where w / 2 and h / 2 represent that the length and width pixel unit lengths of the disparity feature information are 1 / 2 of the input left and right human body region image pixels, and C is the number of channels after the disparity feature information passes through multiple two-dimensional convolutional neural networks; Module M3.2: Concatenate the matrix data of the human body features (2, w / 2, h / 2, C) extracted by the three-dimensional human body pose estimation sub-network with the disparity feature information in the dimension of the number of channels to obtain the matrix data of (2, w / 2, h / 2, 2C), and then reduce the number of channels through a layer of two-dimensional convolutional neural network for further feature fusion to obtain the feature fusion information of (2, w / 2, h / 2, C); Module M3.3: Divide the feature fusion information (2, w / 2, h / 2, C) into the left image feature fusion information (w / 2, h / 2, C) and the right image feature fusion information (w / 2, h / 2, C), and perform a pixel translation operation on the right image feature fusion information in the pixel w dimension with pixel units from 0 to (D / 2 - 1); where D is the disparity search range of the left and right human body image regions; the translation operation is performed D / 2 times to obtain matrix data of the right image feature fusion information as (w / 2, h / 2, C, D / 2); subtract the left image feature fusion information from the right image feature fusion information to obtain the disparity cost aggregation information of (w / 2, h / 2, C, D / 2), and further reduce the dimensions of the channel number dimension and the disparity D dimension of the disparity cost aggregation information to obtain matrix data of (w / 2, h / 2, C×D / 2). Module M3.4: Take the (w / 2, h / 2, C×D / 2) disparity cost aggregation information as the input, pass through a multi-layer two-dimensional convolutional neural network to obtain the disparity calculation result of (w / 2, h / 2, D / 2), and through bilinear interpolation upsampling, obtain the original resolution disparity calculation result of (w, h, D). Normalize and weighted average the values in the D disparity dimension of this result to finally obtain the disparity calculation map of (w, h). Each value in the matrix represents the disparity value of the pixel point in the left image at the original resolution relative to the pixel matching point in the right image. Module M3.5: Use the result of two-dimensional human pose estimation combined with the human prior in the real scene to optimize the disparity calculation map. The said Module M3.5 includes: During the process of using the result of two-dimensional human pose estimation combined with the human prior in the real scene to optimize the disparity calculation map, regard a certain limb L of the two-dimensional human skeleton constructed by the two-dimensional human pose estimation key points as a rigid body structure. The depth change gradient along the human bone direction should remain unchanged. Construct the network loss function according to the gradient distribution of the disparity estimation result in the human bone region and the gradient distribution of the ideal bone as follows: Loss = abs(P x1 - P x2 )(x1, x2 ∈ R) Where abs represents taking the absolute value, P is the first-order derivative map of the disparity calculation map, R is the pixel region composed of pixel points in the left image human detection region whose pixel distance from the two-dimensional human skeleton constructed by the two-dimensional human pose estimation key points is less than a certain threshold, and x1 and x2 are any two adjacent points in the w direction of the image within the R pixel region. During the training process of the sub-network, use this loss function as the disparity optimization loss function for supervised learning to achieve the effect of optimizing the disparity calculation map using the result of two-dimensional human pose estimation; finally output the optimized disparity calculation map (w, h) as the scene depth map calculation result.

10. The short baseline binocular three-dimensional human body pose reconstruction system according to claim 7, wherein The said Module M4 includes: Module M4.1: The processed left and right human body region image pair undergoes feature extraction through the feature extraction module of the 3D human pose estimation sub-network with shared weights to obtain the key point feature information of the left and right 2D human poses. Among them, the processed left and right human body region image pair is denoted as a matrix data format of (2, w, h, 3). Here, 2 represents two images, one left and one right; w = h, representing the pixel unit length of the length and width of the square human body image pair; 3 represents the three-channel data of the color RGB image. The key point feature information of the left and right 2D human poses is denoted as a matrix of (2, w / 2, h / 2, C). 2 represents two images, one left and one right; w / 2 and h / 2 represent that the length and width pixel unit length of the disparity feature information is 1 / 2 of the input left and right human body region image pixel pair, and C is the number of channels after the multi-layer 2D convolutional neural network for the disparity feature information. Module M4.2: The matrix data of the feature extraction result (2, w / 2, h / 2, C) of the 2D human pose estimation is concatenated with the disparity feature information of (2, w / 2, h / 2, C) in the channel number dimension to obtain a matrix data format of (2, w / 2, h / 2, 2C), and then the number of channels is reduced through a layer of 2D convolutional neural network for further feature fusion to obtain the feature fusion information of (2, w / 2, h / 2, C). Module M4.3: The feature fusion information of (2, w / 2, h / 2, C) passes through a multi-layer 2D convolutional neural network to obtain a joint heat map with a size of (2, w / 4, h / 4, N), where N is the number of human skeleton joints. Each value on it represents the probability that the pixel point at 1 / 4 of the original image size is the corresponding human skeleton joint. Subsequently, the joint heat map is normalized and weighted averaged in the w dimension and h dimension of the image respectively to obtain the human skeleton joints of (2, 2, N), representing the (x, y) coordinates of N human key points in the image coordinate system of the left and right human body image pair. Module M4.4: Using the human skeleton key point coordinates of (2, 2, N) and the binocular camera calibration parameters as inputs, the 3D human pose coordinates of (3, N) are calculated through triangulation, representing the (x, y, z) three-dimensional coordinates of the real human key points in the world coordinate system during binocular camera calibration. Module M4.5: Using the scene depth map and human prior information, the 3D human pose estimation sub-network is used to optimize the 3D human key point coordinates. The Module M4.5 includes: During the training process, a loss function is constructed using the depth dimension coordinate of the 3D human pose coordinates and the corresponding depth value of the optimized scene depth map as follows: Loss1 = abs(Z x -V x )(x ∈ N) Among them, x is any one of the N 3D human skeleton key points, Z is the Z-axis coordinate value of this key point in the world coordinate system, and V is the depth value of the corresponding coordinate of the 2D key point of the left image of this key point in the scene depth map. In addition, since the human skeleton has the prior that the lengths of symmetric limbs such as the left and right arms and the left and right legs are equal, the following loss function is constructed: Loss2 = abs(L m - L n )(m, n ∈ K) Among them, \(K\) is the set composed of symmetric limbs of the human body, \(m\) and \(n\) are any two symmetric limbs in the set of symmetric limbs of the human body, and \(L\) is the limb length calculated according to the three-dimensional human key point coordinates; The above loss function is weighted and added to the loss function formed by providing the ground truth according to the training set during training to obtain the final loss function; the final network output is the optimized result of the three-dimensional human pose; During training, the stereo matching sub-network and the three-dimensional human pose estimation sub-network are respectively pre-trained based on the public dataset, and then co-trained to achieve the effect of optimizing the three-dimensional human key point coordinates using the depth map of the usage scenario and the human prior information pair.

Images