A human body three-dimensional posture reconstruction method and system based on multi-view human body images

Abstract

The application provides a human body three-dimensional posture reconstruction method and system based on multi-view human body images, and belongs to the field of computer vision.The method comprises the following steps: collecting human body images of multiple views through multiple cameras; determining the depth values of each human body surface point by using a pre-trained encoding and decoding network according to the human body images of multiple views, the minimum depth value, the maximum depth value, the internal parameter matrix and the external parameter matrix of each camera; determining human body point cloud data according to the depth values of each human body surface point, the internal parameter matrix and the external parameter matrix of each camera; determining human body three-dimensional key point coordinates in a directional bounding box coordinate system by using a pre-trained feature extraction network based on the human body point cloud data; and converting the human body key point coordinates in the directional bounding box coordinate system to the camera coordinate system to obtain human body three-dimensional key point coordinates in the camera coordinate system, so as to determine the human body three-dimensional posture.The application improves the accuracy and generalization of human body three-dimensional posture reconstruction.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and system for reconstructing human three-dimensional pose based on multi-view human images. Background Technology

[0002] Human 3D pose estimation aims to estimate the 3D coordinates of key points in a human skeleton from images or videos, and then connect these key points in a specific order to form the skeleton. It is a classic problem in computer vision. Human 3D pose estimation can be applied in motion capture simulation, pedestrian recognition, and human-computer interaction. Two-dimensional pose estimation is similar to 3D pose estimation, both estimating the coordinates of key points in a human skeleton, the difference being whether the coordinates are in 2D or 3D space. In recent years, with the widespread application of deep learning, many deep learning-based methods have emerged in human pose estimation. 2D pose estimation often achieves good results using supervised methods because datasets are easier to create. However, 3D pose estimation is more difficult due to the difficulty of dataset annotation, and the lack of sufficient 3D spatial information in images. Previous methods for human 3D pose estimation often directly extracted features from images to predict 2D or 3D key points. If 3D annotations were available, 3D poses were used for supervised training; otherwise, 2D poses were used for weakly supervised training. This method of directly extracting features from images and regressing to predict key points is easily affected by occlusion or insufficient spatial information, thus existing methods cannot simultaneously achieve ideal results in terms of accuracy in 3D pose and generalization across different scene data. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for reconstructing human three-dimensional pose based on multi-view human images, which can improve the accuracy and generalization of human three-dimensional pose reconstruction.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] A method for reconstructing human 3D pose based on multi-view human images, comprising:

[0006] Human images are captured from multiple perspectives using multiple cameras, and the intrinsic and extrinsic parameter matrices of each camera, as well as the minimum and maximum depth values ​​of the human body's position relative to the camera, are determined.

[0007] Based on human body images from multiple perspectives, the minimum depth value, the maximum depth value, the intrinsic parameter matrix and extrinsic parameter matrix of each camera, a pre-trained encoding and decoding network is used to determine the depth value of each point on the human body surface.

[0008] Human point cloud data is determined based on the depth values ​​of each human body surface point, the intrinsic parameter matrix of each camera, and the extrinsic parameter matrix; the human point cloud data includes the three-dimensional coordinates of each human body surface point in the camera coordinate system.

[0009] Based on the human point cloud data, a pre-trained feature extraction network is used to determine the coordinates of the three-dimensional key points of the human body in the directed bounding box coordinate system.

[0010] The coordinates of human body key points in the directed bounding box coordinate system are transformed to the camera coordinate system to obtain the coordinates of human body 3D key points in the camera coordinate system, so as to determine the human body 3D pose.

[0011] Optionally, based on human images from multiple perspectives, the minimum depth value, the maximum depth value, and the intrinsic and extrinsic parameter matrices of each camera, a pre-trained encoder-decoder network is used to determine the depth value of each point on the human body surface, specifically including:

[0012] Convolutional neural networks are used to extract features from each human body image, obtain feature maps corresponding to each viewpoint, and determine the feature vector of each pixel in each feature map;

[0013] For any pixel to be estimated in the feature map of the viewpoint to be estimated, multiple initial depth values ​​of the pixel to be estimated are determined based on the minimum depth value and the maximum depth value; the viewpoint to be estimated is any one of the multiple viewpoints, and the pixel to be estimated is any pixel in the feature map of the viewpoint to be estimated.

[0014] For any initial depth value of the pixel to be estimated, the coordinates of the target pixel in the feature map of each target view are determined based on the initial depth value, the intrinsic parameter matrix of each camera, the extrinsic parameter matrix of each camera, and the coordinates of the pixel to be estimated in the feature map of the view to be estimated; the target view is any view other than the view to be estimated; the target pixel is the pixel in the feature map of the target view when the depth of the pixel to be estimated is the initial depth value.

[0015] For any target viewpoint, the feature vector of each target pixel is determined based on the feature map of the target viewpoint and the coordinates of each target pixel in the feature map of the target viewpoint.

[0016] Based on the feature vectors of each target pixel in the feature map of the target view, the key vector of the target view is determined;

[0017] Based on the key vector of the target viewpoint and the feature vector of the pixel to be estimated, the attention weights and cost volume of the target viewpoint are determined.

[0018] Based on the feature vector of the pixel to be estimated, the attention weights and cost volumes of each target viewpoint, a pre-trained encoder-decoder network is used to determine the depth values ​​of human body surface points in the viewpoint to be estimated.

[0019] Optionally, the initial depth value of the pixel to be estimated can be determined using the following formula:

[0020] d j =d min +(d max -d min ) / D*j, 0≤j≤D-1;

[0021] Where, d j Let d be the j-th initial depth value of the pixel to be estimated. min d is the minimum depth value. max Where is the maximum depth value, and D is the number of preset initial depth values.

[0022] Optionally, the coordinates of the corresponding pixel in the feature map of the target viewpoint i when the depth of the pixel to be estimated is the j-th initial depth value are determined using the following formula:

[0023]

[0024] Among them, (u i,j ,v i,j K represents the coordinates of the pixel in the feature map corresponding to the target viewpoint i when the depth of the pixel to be estimated is the j-th initial depth value. i Let R be the intrinsic parameter matrix of the camera corresponding to the target viewpoint i. i Let be the extrinsic parameter matrix of the camera corresponding to the target viewpoint i, K0 be the intrinsic parameter matrix of the camera corresponding to the viewpoint to be estimated, R0 be the extrinsic parameter matrix of the camera corresponding to the viewpoint to be estimated, (u0, v0) be the coordinates of the pixel to be estimated in the feature map of the viewpoint to be estimated, and d be the extrinsic parameter matrix of the camera corresponding to the target viewpoint i. j Let be the j-th initial depth value of the pixel to be estimated.

[0025] Optionally, based on the key vector of the target viewpoint and the feature vector of the pixel to be estimated, the attention weights and cost body of the target viewpoint are determined, specifically including:

[0026] Based on the key vector of the target viewpoint and the feature vector of the pixel to be estimated, the attention weight of the target viewpoint i is determined using the following formula:

[0027]

[0028] Among them, w i Let v be the attention weight for the target viewpoint i. i Let f be the key vector of the target viewpoint i. rLet t be the feature vector of the pixel to be estimated. e Here, C represents the temperature coefficient, and C represents the number of channels in the feature map.

[0029] The cost volume of the target viewpoint is obtained by taking the inner product of the key vector of the target viewpoint and the feature vector of the pixel to be estimated.

[0030] Optionally, based on the human point cloud data, a pre-trained feature extraction network is used to determine the coordinates of the three-dimensional key points of the human body in the directed bounding box coordinate system, specifically including:

[0031] The human body point cloud data is downsampled to determine multiple reference points;

[0032] Principal component analysis was performed on the three-dimensional coordinates of each reference point to calculate the covariance matrix;

[0033] The three coordinate axis vectors of the directed bounding box are determined based on the covariance matrix to establish the directed bounding box;

[0034] Based on the directed bounding box, the three-dimensional coordinates of each reference point are transformed into the directed bounding box coordinate system and normalized to obtain normalized point cloud data.

[0035] Based on the normalized point cloud data, a pre-trained feature extraction network is used to determine low-dimensional feature vectors.

[0036] The coordinates of the three-dimensional key points of the human body in the directed bounding box coordinate system are determined based on the low-dimensional feature vectors.

[0037] Optionally, based on the normalized point cloud data, a pre-trained feature extraction network is used to determine low-dimensional feature vectors, specifically including:

[0038] Extract the features from the normalized point cloud data to obtain point cloud features;

[0039] Max pooling is performed on the point cloud features to obtain global features;

[0040] The global features are then reduced in dimensionality to obtain a low-dimensional feature vector.

[0041] Optionally, the coordinates of the 3D key points of the human body in the directed bounding box coordinate system can be determined using the following formula:

[0042]

[0043] in, Let P be the coordinates of the 3D key points of the human body in a directed bounding box coordinate system, F() be the trained feature extraction network, and P be the coordinates of the key points of the human body in a directed bounding box coordinate system. noris a low-dimensional feature vector, w is the network parameter of the feature extraction network, I is the pre-defined principal component of the three-dimensional key points of the human body, and u is the pre-defined empirical average value.

[0044] Optionally, the coordinates of human keypoints in the directed bounding box coordinate system can be transformed to the camera coordinate system using the following formula:

[0045]

[0046] in, The coordinates of the three-dimensional key points of the human body in the camera coordinate system. Let L be the rotation matrix of the directed bounding box coordinate system relative to the camera coordinate system. obb Let be the longest axis length of the directed bounding box. The coordinates of the 3D key points of the human body in the directed bounding box coordinate system. The coordinates of the center point of the point cloud in the directed bounding box coordinate system.

[0047] To achieve the above objectives, the present invention also provides the following solution:

[0048] A human body 3D pose reconstruction system based on multi-view human images, comprising:

[0049] The image acquisition unit is used to acquire human images from multiple perspectives through multiple cameras, and to determine the intrinsic parameter matrix, extrinsic parameter matrix, minimum depth value and maximum depth value of the human body position relative to the camera for each camera.

[0050] The depth determination unit, connected to the image acquisition unit, is used to determine the depth value of each human body surface point based on human body images from multiple perspectives, the minimum depth value, the maximum depth value, the intrinsic parameter matrix and extrinsic parameter matrix of each camera, and a pre-trained encoding and decoding network.

[0051] A point cloud determination unit, connected to the depth determination unit, is used to determine human point cloud data based on the depth values ​​of each human body surface point, the intrinsic parameter matrix and the extrinsic parameter matrix of each camera; the human point cloud data includes the three-dimensional coordinates of each human body surface point in the camera coordinate system.

[0052] The key point determination unit, connected to the point cloud determination unit, is used to determine the coordinates of three-dimensional key points of the human body in the directed bounding box coordinate system based on the human body point cloud data and using a pre-trained feature extraction network.

[0053] The pose determination unit, connected to the key point determination unit, is used to transform the coordinates of human body key points in the directed bounding box coordinate system to the camera coordinate system to obtain the coordinates of human body three-dimensional key points in the camera coordinate system, so as to determine the human body three-dimensional pose.

[0054] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0055] This invention first estimates human point cloud data from multi-view human images, and then estimates the human three-dimensional pose based on the human point cloud data. Multi-view images make up for the lack of depth information in single-view images, improving the accuracy of pose reconstruction. Furthermore, by first determining the depth of points on the human surface, the human surface point cloud is obtained, which preserves the spatial information of the human three-dimensional pose to the greatest extent and improves the generalization of pose reconstruction. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart of the human body three-dimensional pose reconstruction method of the present invention;

[0058] Figure 2 A flowchart of a method for determining depth values ​​of points on the human body surface;

[0059] Figure 3 A flowchart of a method for determining the coordinates of three-dimensional key points of the human body;

[0060] Figure 4 A schematic diagram illustrating the principle of human three-dimensional pose reconstruction;

[0061] Figure 5 This is a schematic diagram of the human body three-dimensional pose reconstruction system based on multi-view human body images according to the present invention.

[0062] Symbol explanation:

[0063] Image acquisition unit-1, depth determination unit-2, point cloud determination unit-3, key point determination unit-4, pose determination unit-5. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] The purpose of this invention is to provide a method and system for human three-dimensional pose reconstruction based on multi-view human images. This method does not require supervised training with three-dimensional pose labels, and improves generalization while ensuring the accuracy of three-dimensional pose reconstruction. This solves the problem that the limited number of datasets with real human three-dimensional pose labels leads to unsatisfactory training results for human pose estimation networks.

[0066] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] Example 1

[0068] like Figure 1 As shown, this embodiment provides a method for reconstructing human three-dimensional pose based on multi-view human images, including:

[0069] S1: Collect human images from multiple perspectives using multiple cameras, and determine the intrinsic parameter matrix, extrinsic parameter matrix, minimum depth value and maximum depth value of the human body position relative to the camera for each camera.

[0070] In this embodiment, multiple KinectV2 cameras are used to acquire human RGB images from multiple perspectives, and the intrinsic and extrinsic parameters of the multiple cameras are calibrated using the Zhang Zhengyou camera calibration method.

[0071] Specifically, three depth cameras are placed in different positions. Since the KinectV2 projects infrared light through its infrared cameras to form reflected light, and calculates the depth of an object's surface based on the time-of-flight of the light, simultaneous depth acquisition by all three cameras would cause interference. In actual shooting, the camera in the center, directly facing the human, acquires both depth and RGB images, while the infrared cameras of the other two are covered by black plastic sheets, acquiring only RGB images. The three cameras are triggered sequentially by a program. Although the cameras are not synchronized by hardware, the time it takes for each camera to acquire and transmit a single frame is relatively short, so it can be assumed that the human's posture does not change during the image acquisition process.

[0072] During image acquisition, the three cameras remained stationary, and the subject was positioned in the center of the scene while performing actions. The central camera continuously captured RGB and depth images, while the left and right cameras continuously captured RGB images. From the captured video, one frame was selected every five frames as the dataset. A 2D keypoint detection network was used to detect human keypoints J. Based on these keypoints, the image was cropped to center the human figure, and the image resolution was adjusted to 1024*1024.

[0073] S2: Based on human body images from multiple perspectives, the minimum depth value, the maximum depth value, the intrinsic parameter matrix and extrinsic parameter matrix of each camera, a pre-trained encoding and decoding network is used to determine the depth value of each point on the human body surface.

[0074] This invention estimates the depth of points on the human body surface by using depth search to find pixel matching points from different viewpoints and introducing an attention mechanism to improve accuracy. Then, it uses camera intrinsics to calculate the point cloud coordinates of the human body surface. Specifically, this invention does not require additional preprocessing. Instead, it determines the minimum and maximum depths based on the range of human activity in the scene, transforming the original process of searching for pixel matching points along epipolar lines into sampling depths between the minimum and maximum depths. Then, it performs feature sampling on feature maps from other viewpoints based on the sampled depths.

[0075] Furthermore, such as Figure 2 As shown, S2 specifically includes:

[0076] S21: Use a convolutional neural network to extract features from each human body image to obtain feature maps corresponding to each viewpoint. And determine the feature vector of each pixel in each feature map. Where, f i Let i be the feature map of viewpoint i.

[0077] S22: For any pixel to be estimated in the feature map of the viewpoint to be estimated, determine multiple initial depth values ​​for the pixel to be estimated based on the minimum depth value and the maximum depth value. Wherein, the viewpoint to be estimated is any one of the multiple viewpoints, and the pixel to be estimated is any pixel in the feature map of the viewpoint to be estimated.

[0078] Specifically, the initial depth value of the pixel to be estimated is determined using the following formula:

[0079] d j =d min +(d max -d min ) / D*j, 0≤j≤D-1;

[0080] Where, d j Let d be the j-th initial depth value of the pixel to be estimated. min d is the minimum depth value. max Where is the maximum depth value, and D is the number of preset initial depth values.

[0081] For each initial depth value, its corresponding pixel coordinates on the feature map of other views can be calculated. The number of initial depth values ​​is equal to the number of pixels in the feature map of any other view.

[0082] S23: For any initial depth value of the pixel to be estimated, determine the coordinates of the target pixel in the feature map of each target viewpoint based on the initial depth value, the intrinsic parameter matrix of each camera, the extrinsic parameter matrix of each camera, and the coordinates of the pixel to be estimated in the feature map of the viewpoint to be estimated.

[0083] Wherein, the target viewpoint is any viewpoint other than the viewpoint to be estimated. The target pixel is the pixel in the feature map of the target viewpoint when the depth of the pixel to be estimated is the initial depth value.

[0084] Specifically, the initial depth value d based on the above assumptions j The pixel depth d at position (u0, v0) in the feature map of the viewpoint to be estimated can be calculated using the camera's intrinsic and extrinsic parameters. j At that time, the pixel coordinates (u) corresponding to the feature map of viewpoint i i,j ,v i,j ):

[0085]

[0086] Among them, (u i,j ,v i,j K represents the coordinates of the pixel in the feature map corresponding to the target viewpoint i when the depth of the pixel to be estimated is the j-th initial depth value. i Let R be the intrinsic parameter matrix of the camera corresponding to the target viewpoint i. i Let be the extrinsic parameter matrix of the camera corresponding to the target viewpoint i, K0 be the intrinsic parameter matrix of the camera corresponding to the viewpoint to be estimated, and R0 be the extrinsic parameter matrix of the camera corresponding to the viewpoint to be estimated. Let be the inverse matrix of the intrinsic parameters of the camera corresponding to the viewpoint to be estimated. Let be the inverse extrinsic parameter matrix of the camera corresponding to the viewpoint to be estimated, and (u0, v0) be the coordinates of the pixel to be estimated in the feature map of the viewpoint to be estimated. j Let be the j-th initial depth value of the pixel to be estimated.

[0087] S24: For any target viewpoint, determine the feature vector of each target pixel based on the feature map of the target viewpoint and the coordinates of each target pixel in the feature map of the target viewpoint.

[0088] Specifically, based on the target pixel coordinates (u) calculated above i,j ,v i,j Using bilinear interpolation, the sampling depth value d is obtained. j At that time, the feature map f of target viewpoint i i The upper coordinate is (u i,j ,v i,，j The feature vector of the target pixel Here, BL() represents the bilinear interpolation operation.

[0089] S25: Determine the key vector of the target viewpoint based on the feature vectors of each target pixel in the feature map of the target viewpoint.

[0090] Specifically, the D initial depth values ​​correspond to D feature vectors. Merge the D eigenvectors into a new tensor. Its shape and size are [C,D]. Perform a softmax operation on the tensor along the depth dimension to obtain the key vector v of the target viewpoint in the attention mechanism. i :

[0091] S26: Based on the key vector of the target viewpoint and the feature vector of the pixel to be estimated, determine the attention weight and cost body of the target viewpoint.

[0092] Specifically, the feature vector f of the pixel to be estimated r As the query vector in the attention mechanism, the attention weights for the target view i are determined using the following formula:

[0093]

[0094] Among them, w i Let v be the attention weight for the target viewpoint i. i Let f be the key vector of the target viewpoint i. r Let f be the feature vector of the pixel to be estimated. r The shape and size are [C], t e Where is the temperature coefficient, and C is the number of channels in the feature map.

[0095] The cost volume s of the target viewpoint is obtained by taking the inner product of the key vector of the target viewpoint and the feature vector of the pixel to be estimated. i :s i =(v i ,f r ), (,) denote the inner product operation. The cost volume is used to measure the degree of matching between the feature to be estimated and other features.

[0096] S27: Based on the feature vector of the pixel to be estimated, the attention weights and cost bodies of each target viewpoint, a pre-trained encoding and decoding network is used to determine the depth value of the human body surface point in the viewpoint to be estimated.

[0097] Specifically, the input data of the encoder-decoder network is calculated based on the attention weights and cost volumes of each target perspective. Where N is the number of target viewpoints. Based on the input data c and the feature vector f of the pixel to be estimated. r The codec network is used to predict the depth values ​​of points on the human body surface in the viewpoint to be estimated.

[0098] During the training of the encoder-decoder network, based on the estimated depth value Construct a depth-supervised loss function using the ground truth depth d of human body surface points acquired by the camera: Here, l represents the value of the deep supervised loss function. By minimizing the loss function through gradient descent and using the Adam optimization algorithm to iteratively solve for the optimal parameters of the network using samples in the training set, the trained encoder-decoder network can be obtained.

[0099] This invention introduces an attention mechanism when fusing the results related to the features of the viewpoint to be estimated and other viewpoints. r As the query vector in the attention mechanism, and as feature vectors from other perspectives Perform concat and softmax operations along the depth dimension to obtain the key vector v in the attention mechanism. i The attention weights w for features from different viewpoints are calculated. i Cost aggregation is performed using information from three-dimensional space without introducing new network parameters.

[0100] S3: Determine the human body point cloud data based on the depth values ​​of each point on the human body surface, and the intrinsic and extrinsic parameter matrices of each camera. The human body point cloud data includes the three-dimensional coordinates of each point on the human body surface in the camera coordinate system. Where (u,v) are the coordinates of a point on the human body surface. P represents the depth value of a point (u,v) on the human body surface. cam Let (u,v) be the three-dimensional coordinates of a point (u,v) on the human body surface in the camera coordinate system, and K be the camera intrinsic parameter matrix.

[0101] S4: Based on the human body point cloud data, a pre-trained feature extraction network is used to determine the coordinates of the three-dimensional key points of the human body in the directed bounding box coordinate system.

[0102] Specifically, the human point cloud data is first normalized in the OBB (Oriented Bounding Box) coordinate system. Then, feature vectors that can represent the three-dimensional posture of the human body are extracted through a feature extraction network. The mapping relationship between the feature vectors and the coordinates of the three-dimensional key points is established by using principal component analysis. Supervised and self-supervised constraints are constructed to train the feature extraction network.

[0103] Furthermore, such as Figure 3 As shown, S4 specifically includes:

[0104] S41: Downsample the human body point cloud data to determine M reference points.

[0105] S42: To obtain the OBB bounding box that is closely attached to M reference points, the 3D coordinates (x, y) of the M reference points are... 1a ,x 2a ,x 3aPrincipal component analysis is performed on a = 1, 2, ..., M to calculate the covariance matrix A.

[0106]

[0107]

[0108]

[0109] S43: Determine the three coordinate axis vectors of the directed bounding box based on the covariance matrix to establish the directed bounding box.

[0110] Specifically, based on the covariance matrix A, the λ values ​​satisfying the equation |A-λE|=0 are solved using linear algebra. These are the eigenvalues ​​λ of the covariance matrix A, where E is a 3*3 identity matrix. The three largest eigenvalues ​​are taken and substituted into the equation (A-λE)x=0 to solve for x, resulting in three three-dimensional eigenvectors, which serve as the three coordinate axis vectors of the OBB.

[0111] S44: Based on the directed bounding box, the three-dimensional coordinates of each reference point are transformed to the directed bounding box coordinate system and normalized to obtain normalized point cloud data.

[0112] Specifically, in order for the feature extraction network to process human point clouds with arbitrary orientation and arbitrary number of point clouds, the original point cloud data is first downsampled to a fixed number, and then the three-dimensional coordinates P in the camera coordinate system are... cam Transform to the OBB coordinate system closely aligned with the point cloud and perform normalization. Use the formula... Transform the 3D coordinates of the reference point to the directed bounding box coordinate system using the formula. Normalize the coordinates of the reference point in the directed bounding box coordinate system, where P obb The coordinates of the reference point in the directed bounding box coordinate system. Let be the rotation matrix of the bounding box coordinate system relative to the camera coordinate system. Let L be the coordinates of the center point of the point cloud in the directed bounding box coordinate system. obb The longest axis length of the directed bounding box.

[0113] S45: Based on the normalized point cloud data, a pre-trained feature extraction network is used to determine low-dimensional feature vectors.

[0114] Specifically, firstly, a multilayer perceptron is used to extract features from the normalized point cloud data, obtaining M*C dimensional point cloud features. Then, max pooling is performed on the M*C dimensional point cloud features to obtain global features. Finally, a multilayer perceptron is used to reduce the dimensionality of the global features, obtaining a low-dimensional feature vector.

[0115] This invention uses a multilayer perceptron and a max pooling layer to construct a feature extraction network, extracts human point cloud feature vectors, and obtains the principal components of human pose in the dataset by performing principal component analysis on the coordinates of real 3D key points in the training dataset, thus establishing a mapping between feature vectors and 3D key point coordinates.

[0116] S46: Determine the coordinates of the three-dimensional key points of the human body in the directed bounding box coordinate system based on the low-dimensional feature vectors. For example... Figure 4 The diagram shown illustrates the principle of human three-dimensional pose reconstruction.

[0117] Specifically, the coordinates of the three-dimensional key points of the human body in the directed bounding box coordinate system are determined using the following formula:

[0118]

[0119] in, Let P be the coordinates of the 3D key points of the human body in a directed bounding box coordinate system, F() be the trained feature extraction network, and P be the coordinates of the key points of the human body in a directed bounding box coordinate system. nor denoted as a low-dimensional feature vector, w represents the network parameters of the feature extraction network, I represents the pre-defined principal components of the three-dimensional key points of the human body, and u represents the empirical average value.

[0120] S5: Transform the coordinates of human body key points in the directed bounding box coordinate system to the camera coordinate system to obtain the coordinates of human body 3D key points in the camera coordinate system, so as to determine the human body 3D pose.

[0121] Specifically, the following formula is used to transform the coordinates of human keypoints in the directed bounding box coordinate system to the camera coordinate system:

[0122]

[0123] in, The coordinates of the three-dimensional key points of the human body in the camera coordinate system. Let L be the rotation matrix of the directed bounding box coordinate system relative to the camera coordinate system. obb Let be the longest axis length of the directed bounding box. The coordinates of the 3D key points of the human body in the directed bounding box coordinate system. The coordinates of the center point of the point cloud in the directed bounding box coordinate system.

[0124] During the training of the feature extraction network, a supervision constraint term is constructed based on the low-dimensional feature vector: Among them, L fs Here, θ represents the supervision constraint value, and θ is the regularization coefficient. The F-dimensional projection vector of the real 3D keypoints is obtained by performing principal component analysis on the coordinates of the real 3D keypoints in the training set. X nor These are the normalized coordinates of real 3D key points on the human body.

[0125] Suppose there are H images in the training set, and G are the number of 3D keypoints on the human body in each image. Each 3D keypoint includes three coordinate values. We can obtain a keypoint coordinate matrix B for all images in the dataset. B has dimensions [3*G, H], that is, 3*G rows and H columns, with each column representing 3*G coordinate values ​​from one image. Calculating the mean of each row of matrix B yields 3*G values, which serve as the empirical average u of the human body keypoints. Then, subtract the empirical average from each element of each row of matrix B to calculate the covariance matrix. The λ′ values ​​that satisfy the equation |Q-λE|=0 are obtained by linear algebra, which are the eigenvalues ​​λ′ of matrix Q, where E is the identity matrix. The largest S λ′ values ​​are taken and substituted into the equation (Q-λ′E)x=0 to solve for x. The S eigenvectors are then used as the principal components I of the three-dimensional key points of the human body.

[0126] Based on the 3D human body key points in the camera coordinate system, they are projected onto the image coordinate system according to the camera intrinsic parameter K to obtain the predicted 2D human body key points.

[0127] The actual two-dimensional human key points J in the human image were obtained using OpenPose detection.

[0128] Weakly supervised constraints are constructed based on actual and predicted 2D human body key points: Among them, L ws This is a weakly supervised constraint value.

[0129] Based on the supervision constraint term L fs and weak supervision constraint L ws Construct the total constraints for the training phase of the human point cloud feature extraction network:

[0130] By minimizing the total constraint term through gradient descent, the Adam optimization algorithm is used to iteratively solve for the network parameters w using samples in the training set, in order to obtain a well-trained feature extraction network.

[0131] This invention combines supervised and self-supervised methods. Supervised methods constrain the feature vector corresponding to the human body's 3D pose, while self-supervised methods constrain the 3D coordinates mapped from the 3D pose feature vectors and projected onto the 2D keypoint coordinates of the image. It can be trained and tested effectively even on datasets lacking real 3D labels, alleviating the difficulty of 3D pose annotation to some extent and improving the generalization of pose reconstruction.

[0132] In summary, this invention uses multiple KinectV2 cameras to acquire multi-view human RGBD images; it trains a network model to estimate the depth of human surface points from multi-view human images using the Kinect V2 RGBD dataset, including using a depth-based search method to determine matching points between different views and using an attention mechanism to distinguish the importance of features from other views at different sampling depths; and it estimates the positions of three-dimensional key points of the human body based on the depth-reconstructed human point cloud data, including using a feature extraction network to extract human point cloud features and using principal component analysis to establish the mapping relationship between human point cloud feature vectors and three-dimensional key points. This invention can be applied to fields such as human-computer interaction, pedestrian recognition, and virtual reality.

[0133] Example 2

[0134] In order to implement the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, a human body three-dimensional pose reconstruction system based on multi-view human body images is provided below.

[0135] like Figure 5 As shown, the human body three-dimensional pose reconstruction system based on multi-view human body images provided in this embodiment includes: image acquisition unit 1, depth determination unit 2, point cloud determination unit 3, key point determination unit 4, and pose determination unit 5.

[0136] The image acquisition unit 1 is used to acquire human images from multiple perspectives through multiple cameras, and to determine the intrinsic parameter matrix, extrinsic parameter matrix, minimum depth value and maximum depth value of the human body position distance from the camera for each camera.

[0137] The depth determination unit 2 is connected to the image acquisition unit 1. The depth determination unit 2 is used to determine the depth value of each human body surface point based on human body images from multiple perspectives, the minimum depth value, the maximum depth value, the intrinsic parameter matrix and extrinsic parameter matrix of each camera, and a pre-trained encoding and decoding network.

[0138] Point cloud determination unit 3 is connected to depth determination unit 2. Point cloud determination unit 3 is used to determine human body point cloud data based on the depth values ​​of each human body surface point, the intrinsic parameter matrix of each camera, and the extrinsic parameter matrix. The human body point cloud data includes the three-dimensional coordinates of each human body surface point in the camera coordinate system.

[0139] The key point determination unit 4 is connected to the point cloud determination unit 3. The key point determination unit 4 is used to determine the coordinates of the three-dimensional key points of the human body in the directed bounding box coordinate system based on the human body point cloud data and using a pre-trained feature extraction network.

[0140] The pose determination unit 5 is connected to the key point determination unit 4. The pose determination unit 5 is used to transform the coordinates of human body key points in the directed bounding box coordinate system to the camera coordinate system to obtain the coordinates of human body three-dimensional key points in the camera coordinate system, so as to determine the human body three-dimensional pose.

[0141] Compared with the prior art, the human body three-dimensional pose reconstruction system based on multi-view human body images provided in this embodiment has the same beneficial effects as the human body three-dimensional pose reconstruction method based on multi-view human body images provided in Embodiment 1, and will not be repeated here.

[0142] Example 3

[0143] This embodiment provides an electronic device, including a memory and a processor. The memory is used to store computer programs, and the processor runs the computer programs to enable the electronic device to perform the human body three-dimensional pose reconstruction method based on multi-view human body images according to Embodiment 1.

[0144] Alternatively, the aforementioned electronic device may be a server.

[0145] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the human body three-dimensional pose reconstruction method based on multi-view human body images of Embodiment 1.

[0146] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0147] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A human body three-dimensional pose reconstruction method based on multi-view human body images, characterized in that, The human body three-dimensional posture reconstruction method based on multi-view human body images comprises the following steps: Collecting human body images from multiple views through multiple cameras, and determining the intrinsic matrix, extrinsic matrix, minimum depth value and maximum depth value of the human body relative to the cameras; According to the human body images from multiple views, the minimum depth value, the maximum depth value, the intrinsic matrix and the extrinsic matrix of each camera, a pre-trained encoding and decoding network is used to determine the depth value of each human body surface point; According to the depth value of each human body surface point, the intrinsic matrix and the extrinsic matrix of each camera, human body point cloud data is determined; the human body point cloud data comprises three-dimensional coordinates of each human body surface point in the camera coordinate system; Based on the human body point cloud data, a pre-trained feature extraction network is used to determine the human body three-dimensional key point coordinates in the oriented bounding box coordinate system, specifically including: performing down-sampling on the human body point cloud data to determine multiple reference points; performing principal component analysis on the three-dimensional coordinates of each reference point to calculate a covariance matrix; determining three coordinate axis vectors of the oriented bounding box based on the covariance matrix to establish the oriented bounding box; based on the oriented bounding box, the three-dimensional coordinates of each reference point are converted to the oriented bounding box coordinate system and normalized to obtain normalized point cloud data; according to the normalized point cloud data, a pre-trained feature extraction network is used to determine a low-dimensional feature vector; according to the low-dimensional feature vector, the human body three-dimensional key point coordinates in the oriented bounding box coordinate system are determined; The human body key point coordinates in the oriented bounding box coordinate system are converted to the camera coordinate system to obtain the human body three-dimensional key point coordinates in the camera coordinate system to determine the human body three-dimensional posture.

2. The method of claim 1, wherein the method further comprises: According to the human body images from multiple views, the minimum depth value, the maximum depth value, the intrinsic matrix and the extrinsic matrix of each camera, a pre-trained encoding and decoding network is used to determine the depth value of each human body surface point, specifically including: A convolutional neural network is used to extract the features of each human body image to obtain the feature maps corresponding to each view, and to determine the feature vectors of each pixel in each feature map; For any to-be-estimated pixel in the feature map of a to-be-estimated view, multiple initial depth values of the to-be-estimated pixel are determined according to the minimum depth value and the maximum depth value; the to-be-estimated view is any view in the multiple views, and the to-be-estimated pixel is any pixel in the feature map of the to-be-estimated view; For any initial depth value of the to-be-estimated pixel, the coordinates of a target pixel in the feature map of each target view are determined according to the initial depth value, the intrinsic matrix of each camera, the extrinsic matrix of each camera and the coordinates of the to-be-estimated pixel in the feature map of the to-be-estimated view; the target view is any view other than the to-be-estimated view; the target pixel is the corresponding pixel in the feature map of the target view when the depth of the to-be-estimated pixel is the initial depth value; For any target view, the feature vectors of each target pixel are determined according to the feature map of the target view and the coordinates of each target pixel in the feature map of the target view; The key vector of the target view is determined according to the feature vectors of each target pixel in the feature map of the target view; determine the attention weight and the cost volume of the target view based on the key vector of the target view and the feature vector of the pixel to be estimated; determine the depth value of the surface point of the human body in the target view to be estimated by using a pre-trained encoding-decoding network based on the feature vector of the pixel to be estimated, the attention weight and the cost volume of each target view.

3. The method of claim 2, wherein the method further comprises: determine the initial depth value of the pixel to be estimated by using the following formula: d j = d min + (d max - d min ) / D*j, 0≤j≤D-1; wherein d j is the jthinitial depth value of the pixel to be estimated, d min is the minimum depth value, d max is the maximum depth value, and D is the number of pre-set initial depth values.

4. The method of claim 2, wherein the method further comprises: determine the coordinates of the pixel in the feature map of the target view i when the depth of the pixel to be estimated is the jth initial depth value by using the following formula: wherein (u i,j ,v i,j ) is the coordinate of the corresponding pixel in the feature map of the target view i when the depth of the pixel to be estimated is the jth initial depth value, K i is the intrinsic matrix of the camera corresponding to the target view i, R i is the extrinsic matrix of the camera corresponding to the target view i, K0 is the intrinsic matrix of the camera corresponding to the view to be estimated, R0 is the extrinsic matrix of the camera corresponding to the view to be estimated, (u0,v0) is the coordinate of the pixel to be estimated in the feature map of the view to be estimated, and d j is the jth initial depth value of the pixel to be estimated.

5. The method of claim 2, wherein the method further comprises: determine the attention weight and the cost volume of the target view based on the key vector of the target view and the feature vector of the pixel to be estimated, specifically including: determine the attention weight of the target view i by using the following formula according to the key vector of the target view and the feature vector of the pixel to be estimated: wherein w i is the attention weight of the target view i, v i is the key vector of the target view i, f r is the feature vector of the pixel to be estimated, t e is the temperature coefficient, and C is the number of channels of the feature map. obtain the cost volume of the target view by calculating the inner product of the key vector of the target view and the feature vector of the pixel to be estimated.

6. The method of claim 1, wherein the method further comprises: determine the low-dimensional feature vector by using a pre-trained feature extraction network according to the normalized point cloud data, specifically including: extract the features of the normalized point cloud data to obtain point cloud features; perform maximum pooling operation on the point cloud features to obtain global features; perform dimension reduction on the global features to obtain a low-dimensional feature vector.

7. The method of claim 1, wherein the method further comprises: determine the coordinates of the three-dimensional key points of the human body in the oriented bounding box coordinate system by using the following formula: wherein, are three-dimensional key point coordinates of a human body in a directional bounding box coordinate system, F() is a trained feature extraction network, P nor is a low-dimensional feature vector, w is a network parameter of the feature extraction network, I is a principal component of three-dimensional key points of a human body preset in advance, and u is an empirical average value preset in advance.

8. The method of claim 1, wherein the method further comprises: convert the coordinates of the key points of the human body in the oriented bounding box coordinate system to the camera coordinate system by using the following formula: wherein, is the 3D key point coordinate of the human body in the camera coordinate system, is the rotation matrix of the oriented bounding box coordinate system relative to the camera coordinate system, L obb is the length of the longest axis of the oriented bounding box, is the 3D key point coordinate of the human body in the oriented bounding box coordinate system, is the point cloud center point coordinate in the oriented bounding box coordinate system. 9.A human body three-dimensional pose reconstruction system based on multi-view human body images, applied to the human body three-dimensional pose reconstruction method based on multi-view human body images in any one of claims 1-8, characterized in that, The human body three-dimensional pose reconstruction system based on multi-view human body images comprises: An image acquisition unit is configured to acquire human body images in multiple views by multiple cameras and determine intrinsic matrices, extrinsic matrices, minimum depth values and maximum depth values of the human body relative to the cameras of the cameras. A depth determination unit is connected to the image acquisition unit and configured to determine depth values of the surface points of the human body by using a pre-trained encoding-decoding network based on the human body images in multiple views, the minimum depth values, the maximum depth values, the intrinsic matrices and the extrinsic matrices of the cameras. A point cloud determination unit is connected to the depth determination unit and configured to determine human body point cloud data based on the depth values of the surface points of the human body, the intrinsic matrices and the extrinsic matrices of the cameras. The human body point cloud data comprises three-dimensional coordinates of the surface points of the human body in the camera coordinate system. A key point determination unit is connected to the point cloud determination unit and configured to determine the coordinates of the three-dimensional key points of the human body in the oriented bounding box coordinate system by using a pre-trained feature extraction network based on the human body point cloud data. A pose determination unit is connected to the key point determination unit and configured to convert the coordinates of the key points of the human body in the oriented bounding box coordinate system to the camera coordinate system to obtain the coordinates of the three-dimensional key points of the human body in the camera coordinate system and determine the three-dimensional pose of the human body.

Images