Fusion multi-view three-dimensional human pose estimation method and system based on voxel space projection

Abstract

The application discloses a kind of fusion multi-view three-dimensional human posture estimation method and system based on voxel space projection.The method inputs the image collected by multiple cameras into CNN to generate initial heat map, utilizes stacked hourglass network multi-scale processing and executes multi-view feature fusion based on epipolar constraint between adjacent sub-networks;Call camera internal and external parameter to construct public three-dimensional voxel space, generate probability feature body by back projection and average after fusing heat map;Along the height axis maximum pooling dimension reduction obtains the feature mapping of bird's eye view, extracts human position anchor point by two-dimensional convolution network;The average three-dimensional Euclidean distance between each single-view posture result and anchor point is calculated to group matching, finally executes triangulation reconstruction or 3DPS model to output three-dimensional posture.The application embeds epipolar fusion into multi-stage network to realize iterative cross-view supervision, replaces three-dimensional convolution by bird's eye view projection to reduce calculation amount, effectively improves the estimation accuracy and matching efficiency in occlusion scene.

Description

Technical Field

[0001] This invention belongs to the fields of photogrammetry, computer vision and artificial intelligence, and specifically relates to a method and system for fusion of multi-view three-dimensional human pose estimation based on voxel space projection. Background Technology

[0002] In recent years, with the continuous improvement of computer technology and information technology, and the rise of concepts such as the metaverse, computer vision has been integrated into all aspects of people's lives. Humans, as the main users of technology and the primary recipients of computer services, have always had human body localization, recognition, and modeling as central topics in computer vision research. Furthermore, due to the increasing prevalence of terminals such as smartphones, the traditional keyboard and mouse mode is gradually being replaced by diverse human-computer interaction technologies such as motion sensors and posture sensors. The emergence of these new hardware technologies is also promoting the continuous development and innovation of computer vision-related technical theories.

[0003] Human pose estimation can be divided into two-dimensional pose estimation and three-dimensional pose estimation. Generally, the results of two-dimensional pose estimation are used to reconstruct the three-dimensional pose through triangulation. Because backprojection using a single camera is an ill-posed problem, meaning that multiple three-dimensional poses share the same two-dimensional projection, a calibrated multi-camera system is often required for three-dimensional pose estimation. However, existing technologies in the field of multi-view three-dimensional human pose estimation mainly suffer from the following three technical problems.

[0004] First, there is the problem of a significant decrease in 2D pose estimation accuracy under occlusion. While multi-camera systems can improve the accuracy of 3D pose estimation, they cannot fundamentally solve the problem of 2D pose estimation errors caused by occlusion. Most existing 2D human pose estimation methods use monocular estimation algorithms, which produce many erroneous results when occlusion occurs and cannot be effectively supervised. Although methods such as Cross View Fusion (Qiu et al., ICCV 2019), Epipolar Transformers (He et al., CVPR 2020), and AdaFuse (Zhang et al., IJCV 2021) utilize epipolar geometric constraints to fuse multi-view features to improve 2D pose estimation accuracy, these methods treat fusion as a single post-processing operation after the backbone network. The fused information cannot be fed back to the intermediate stage of feature extraction, limiting the full potential of the fusion effect. Especially in stacked multi-stage network architectures, the intermediate prediction results of each stage do not receive supervision from cross-view information, causing the network to be unable to gradually accumulate cross-view correction information during multi-stage refinement.

[0005] Second, there are issues with the accuracy and efficiency of multi-view matching. When estimating the 3D pose of multiple targets, the problem of multi-view matching arises. Specifically, it's difficult to find a reliable algorithm to cluster the results of the 2D pose estimation. If the 2D pose estimation results of targets not belonging to a particular individual are used for the triangulation reconstruction of that individual, the 3D pose estimation result will inevitably be incorrect. Existing methods, such as MVPose (Dong et al., CVPR 2019), use a similarity matrix based on appearance re-identification features and epipolar constraints for view matching. However, this method requires prior knowledge of the number of people in the scene, and the matching process has high computational complexity. When the total number of targets in the scene is unknown, this type of similarity matrix-based method is difficult to work effectively.

[0006] Third, there is the problem of excessive computational complexity in voxel space-based methods. Existing voxel space-based methods for multi-person 3D pose estimation, such as VoxelPose (Tu et al., ECCV 2020), implicitly solve the multi-view matching problem by constructing a shared 3D voxel space and using a 3D convolutional neural network for human detection and pose estimation. However, the computational cost of the 3D convolutional neural network is enormous, severely limiting the real-time performance of the method. Although Faster VoxelPose (Ye et al., ECCV 2022) significantly reduces computational complexity by projecting the 3D voxel space along the height axis onto a 2D bird's-eye view plane and using a 2D convolutional network for human detection, its voxel space construction is still based on heatmaps estimated independently from each viewpoint, failing to fully utilize cross-view fusion information to improve the quality of voxel features.

[0007] In summary, the main shortcomings of existing technologies include: multi-view epipolar constraint fusion is only executed once after the backbone network output, and cannot play a cross-view supervision role in the intermediate stages of multi-stage networks; multi-view matching relies on similarity matrices or voxel subspace segmentation, which is inefficient or cannot work in scenarios where the number of people is unknown; voxel space-based methods use heatmaps estimated independently by each viewpoint to construct feature volumes, failing to introduce the high-quality heatmaps after epipolar fusion into the voxel construction process, and the computational cost of 3D convolution is too large. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of the aforementioned background technologies. This invention proposes a multi-view 3D human pose estimation method and system based on voxel space projection, which solves the technical problems of improving the accuracy of 2D pose estimation and achieving accurate multi-view matching in existing technologies. This method integrates three technical modules—episole-constrained multi-view feature fusion, voxel space top-view human anchor point extraction, and 3D Euclidean distance-based cross-view matching—into a complete technical pipeline, achieving significant improvements in 2D pose estimation accuracy, multi-view matching efficiency, and computational complexity.

[0009] The technical solution adopted in this invention is: a method for estimating 3D human pose based on voxel space projection and fusion of multiple perspectives, comprising:

[0010] Acquire multi-view image data of the research object synchronously acquired by a pre-calibrated multi-camera system;

[0011] The multi-view image data is input into a CNN network to extract features and generate initial two-dimensional heatmaps of key points for each camera view in the multi-camera system.

[0012] The initial heatmap is processed at multiple scales using a stacked hourglass network containing multiple cascaded single hourglass subnetworks. Each single hourglass subnetwork outputs a corresponding intermediate-level heatmap. A multi-view feature fusion module is set between adjacent single hourglass subnetworks. Utilizing the epipolar constraint relationship between different camera views constructed based on camera intrinsic and extrinsic parameters, for each camera view, the intermediate-level heatmap under that camera view is used as the current main heatmap to be fused. Feature fusion processing is performed with the corresponding intermediate-level heatmaps under all other camera views, so that each camera view obtains a corresponding fused two-dimensional joint heatmap.

[0013] A common three-dimensional voxel space is constructed by calling the intrinsic and extrinsic parameters of the camera. The fused two-dimensional joint heatmaps corresponding to each camera viewpoint are back-projected onto the common three-dimensional voxel space, and the average value of the projection results of each camera viewpoint is taken to generate a three-dimensional voxel space probability feature.

[0014] The three-dimensional voxel space probability feature volume is projected and dimension-reduced along a two-dimensional top view plane orthogonal to the height axis to obtain a two-dimensional top view feature map, and a two-dimensional convolutional neural network is called to extract human body position anchor points on the two-dimensional top view feature map.

[0015] The single-view two-dimensional human pose estimation result set corresponding to each detected individual is obtained by decoding the fused two-dimensional joint point heat map from each camera view. For each obtained single-view two-dimensional human pose estimation result set corresponding to each detected individual, the discrete joint points in the result set are back-projected to three-dimensional space, and the average three-dimensional Euclidean distance between each back-projected discrete joint point and each human position anchor point is calculated.

[0016] Based on the average three-dimensional Euclidean distance between each single-view two-dimensional human pose estimation result set and each human position anchor point, each single-view two-dimensional human pose estimation result set is grouped and assigned to the corresponding human position anchor point, and aggregated to form a cross-view matching result set for the same target individual.

[0017] Based on the cross-view 2D poses of the same individual in the matching result set, triangulation reconstruction or input of a 3DPS model is performed to output the final 3D human pose space coordinates of each target individual in the scene.

[0018] In the above technical solution, the training process of the stacked hourglass network includes:

[0019] Based on the human joint coordinates labeled in the training dataset, the discrete joint coordinates are transformed into continuous training heatmap labels using a Gaussian mixture model. The response distribution of each body joint in the heatmap is described by the Gaussian mixture model, and its probability density function is determined by the total number of Gaussian components, the normalized weight of each component, the probability density function of each component, the mean, and the covariance matrix.

[0020] After the feature fusion based on epipolar constraints is completed at each stage of the stacked hourglass network, the fused two-dimensional joint heatmaps output by each hourglass sub-network are used as intermediate supervision objects. The mean square error is used as the loss function to calculate the spatial distribution difference between the fused two-dimensional joint heatmaps and the training heatmap labels, and backpropagation is performed based on the spatial distribution difference to update the network parameters.

[0021] In the above technical solution, each single hourglass sub-network is composed of a downsampling first half network for feature dimensionality reduction and an upsampling second half network for feature recovery.

[0022] When calculating the feature map output of each layer of the upsampled second half network, it is required that the feature output of the downsampled features of the corresponding scale layer of the downsampled first half network after processing by the residual module be added and merged element by element with the recovery result of the upsampled features of the previous scale layer of the second half network after deconvolution operation, and finally generate the intermediate heatmap with the same scale as the input feature map of the hourglass sub-network.

[0023] In the above technical solution, the step of performing feature fusion processing on the intermediate-level heatmap of each camera view and the corresponding intermediate-level heatmaps of all other camera views includes:

[0024] Extract the intermediate heatmap output from the current camera view as the main feature to be enhanced, and calculate the epipolar coordinates of all other camera views projected onto the current camera view by calling the pre-stored camera intrinsic and extrinsic parameters.

[0025] Along the calculated epipolar coordinate region, extract the response feature distribution from the intermediate-level heatmaps of all other camera views;

[0026] The extracted response feature distribution is fused with the subject feature to be enhanced to generate the fused two-dimensional joint heatmap corresponding to the camera viewpoint.

[0027] In the above technical solution, the process of generating a three-dimensional voxel space probabilistic feature volume includes:

[0028] The continuous physical coordinate range corresponding to the public three-dimensional voxel space is equally dispersed into a three-dimensional feature cube array defined by the center of the orthogonal grid.

[0029] For each discrete voxel anchor point in the array, the three-dimensional spatial coordinates of the voxel anchor point are back-projected onto the image plane of each camera viewpoint using the camera intrinsic and extrinsic parameters to obtain the corresponding projection position. The response probability value of the fused two-dimensional joint heat map of each camera viewpoint at the projection position is read, and the average of the response probability values ​​of all participating camera viewpoints is taken as the spatial thermal probability value of the voxel anchor point.

[0030] The three-dimensional voxel spatial probability feature is composed of the spatial heat probability values ​​of all discrete voxel anchor points.

[0031] In the above technical solution, the process of projecting the three-dimensional voxel space probabilistic feature volume onto an orthogonal two-dimensional top-view plane for dimensionality reduction includes:

[0032] A top-view projection transformation matrix is ​​predefined, and this matrix is ​​used to map the coordinates of the common three-dimensional voxel space to the basic two-dimensional coordinate system, wherein the top-view projection transformation matrix transforms the three-dimensional vector into a two-dimensional vector with zero depth coordinates;

[0033] Based on the mapping direction defined by the top-view projection transformation matrix, the three-dimensional voxel space probability feature volume is collapsed and projected onto a two-dimensional horizontal plane along the height axis using a maximum pooling mathematical operation operator, generating the two-dimensional top-view feature map where the extreme values ​​of the human body key point response intensity at the corresponding position are consistent with the original three-dimensional distribution.

[0034] In the above technical solution, the step of calling a two-dimensional convolutional neural network to extract human body position anchor points on the two-dimensional top view feature map includes the following training process of the two-dimensional convolutional neural network:

[0035] During the training phase, the three-dimensional Euclidean distance between the geometric coordinates of the real human pose position provided by the training dataset and each discrete voxel anchor point in the common three-dimensional voxel space is calculated.

[0036] Based on the calculated three-dimensional Euclidean distance, the shortest distance value between each voxel anchor point and the real human posture of all target individuals in the scene is extracted as a regression benchmark.

[0037] Based on the extracted shortest distance value, the true confidence probability score of the voxel anchor point as the center point of the human body is calculated by using the two-dimensional Gaussian distribution probability density function.

[0038] The calculated true confidence probability score array of each voxel anchor point is also subjected to maximum pooling dimensionality reduction on the height axis to obtain the true value heatmap of the human body anchor point. This heatmap, along with the predicted heatmap output by the two-dimensional convolutional neural network, is then substituted into the loss function for supervised iterative training. This allows the trained two-dimensional convolutional neural network to extract the human body position anchor point from the two-dimensional top view feature map.

[0039] In the above technical solution, the step of calculating the average three-dimensional Euclidean distance between each discrete joint point after back projection and each of the human body position anchor points includes:

[0040] For each individual being detected, the single-view two-dimensional human pose estimation result set is obtained. After back-projecting the discrete joints in the result set using the camera intrinsic and extrinsic parameters, the three-dimensional Euclidean distance between the joints of each body part and the anchor point of the human body position is calculated.

[0041] The arithmetic mean of the summation of the three-dimensional Euclidean distances of all different joint points is used to obtain the average three-dimensional Euclidean distance used as the basis for matching judgment. Specifically, it is calculated by the total number of different body posture key points contained in the single-view two-dimensional human posture estimation result set, and the three-dimensional Euclidean distance of each specific posture key point from the human position anchor point.

[0042] This invention also provides a fusion multi-view 3D human pose estimation system based on voxel space projection, used to implement the method described in the above technical solution, including:

[0043] The data acquisition module is used to acquire multi-view image data of the research object synchronously collected by a pre-calibrated multi-camera system;

[0044] The two-dimensional feature extraction module is used to input the multi-view image data into the CNN network to extract features and generate initial two-dimensional joint point heatmaps for each camera view in the multi-camera system.

[0045] The multi-scale processing and fusion module is used to perform multi-scale processing on the initial heatmap using a stacked hourglass network containing multiple cascaded single hourglass sub-networks. Each single hourglass sub-network outputs a corresponding intermediate-level heatmap. A multi-view feature fusion module is set between adjacent single hourglass sub-networks. Utilizing the epipolar constraint relationship between different camera views constructed based on camera intrinsic and extrinsic parameters, for each camera view, the intermediate-level heatmap under that camera view is used as the current main heatmap to be fused, and feature fusion processing is performed with the corresponding intermediate-level heatmaps under all other camera views, so that each camera view obtains a corresponding fused two-dimensional joint heatmap.

[0046] The voxel space construction module is used to call the camera intrinsic and extrinsic parameters to construct a common three-dimensional voxel space, back-project the fused two-dimensional joint heatmaps corresponding to each camera viewpoint to the common three-dimensional voxel space, and take the average value of the projection results of each camera viewpoint to generate a three-dimensional voxel space probability feature.

[0047] The dimension reduction and anchor point extraction module is used to project the three-dimensional voxel space probability feature volume along a two-dimensional top view plane orthogonal to the height axis to reduce the dimension, obtain the two-dimensional top view feature map, and call a two-dimensional convolutional neural network to extract the human body position anchor points on the two-dimensional top view feature map.

[0048] The distance calculation module is used to decode and obtain the single-view two-dimensional human pose estimation result set corresponding to each detected individual from the fused two-dimensional joint point heat map of each camera view. For the single-view two-dimensional human pose estimation result set corresponding to each detected individual, the discrete joint points in the result set are back-projected to three-dimensional space, and the average three-dimensional Euclidean distance between each discrete joint point after back-projection and each human position anchor point is calculated.

[0049] The clustering and matching module is used to group each set of two-dimensional human pose estimation results into corresponding human position anchors based on the average three-dimensional Euclidean distance between each set of two-dimensional human pose estimation results and each set of human position anchors, and to aggregate them into a cross-view matching result set for the same target individual.

[0050] The 3D reconstruction module is used to perform triangulation reconstruction or input a 3DPS model based on the cross-view 2D pose of the same individual in the matching result set to output the final 3D human pose space coordinates of each target individual in the scene.

[0051] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the voxel space projection-based fusion multi-view three-dimensional human pose estimation method as described in the above technical solution.

[0052] The beneficial effects of this invention are as follows: At the overall technical solution level, this invention generates an initial heatmap by inputting multi-view image data into a CNN network, and then uses a stacked hourglass network composed of multiple single hourglass sub-networks for multi-scale processing. This effectively utilizes the implicit relationships between joints throughout the body to improve the prediction performance of the next sub-network. Within each single hourglass sub-network, the symmetrical structure of the downsampling first half and the upsampling second half, combined with the skip connections after residual module processing and the element-wise addition and merging mechanism of deconvolution upsampling, ensures that the final output feature map is consistent with the input scale and retains information from all layers, providing a solid feature foundation for generating high-quality intermediate-level heatmaps.

[0053] At the training strategy level, a Gaussian mixture model is used to transform the discrete joint coordinates labeled in the training dataset into continuous training heatmap labels. The mean squared error is used as the loss function to calculate the spatial distribution difference between the fused heatmap and the training labels, providing accurate intermediate supervision signals for each stage of the stacked hourglass network. This training strategy allows the spatial distribution deviation between the output of each hourglass sub-network and the true labels to be effectively measured and optimized. This enables continuous updating of network parameters through backpropagation, ensuring the quality of the fused two-dimensional joint heatmap output by the stacked hourglass network.

[0054] At the multi-view feature fusion level, the core innovation of this invention lies in inserting a multi-view epipolar constraint fusion module between the stages of a stacked hourglass network, creating an iterative refinement loop. All existing multi-view epipolar constraint fusion methods treat fusion as a single post-processing operation after the backbone network, while this method allows each hourglass sub-network to benefit from the cross-view information accumulated in all preceding stages. By calling camera intrinsic and extrinsic parameters to calculate epipolar coordinates and sampling the response feature distribution of the reference viewpoint along the epipolar direction, accurate cross-view feature correspondence is achieved. When the human pose is occluded, this method can effectively utilize different camera views to further improve the accuracy of 2D human pose estimation, exhibiting strong robustness. On the Human3.6M dataset, the average correct joint prediction percentage reaches 96.47%, an improvement of 3.67% compared to HRNet.

[0055] At the voxel space construction level, this method equally disperses the continuous physical coordinate range corresponding to the common 3D voxel space into a 3D feature cube array. It then backprojects the high-quality heatmaps obtained from epipolar constraint fusion of each viewpoint into the voxel space and averages them to generate a 3D voxel space probabilistic feature volume. Compared to existing technologies like VoxelPose and FasterVoxelPose, which directly use independently estimated heatmaps from each viewpoint, this method feeds the epipolar constraint-fused heatmaps into the voxel construction pipeline, resulting in higher-quality 3D feature volumes. This is particularly effective in occluded scenes, generating cleaner voxel feature volumes and more reliable human detection results.

[0056] At the dimensionality reduction level of the top-view projection, this method transforms the 3D vector into a 2D vector with zero depth coordinates through a predefined top-view projection transformation matrix, and then uses max pooling to shrink the projection along the height axis, taking advantage of the characteristics of less interaction and less occlusion between human figures in the top-view projection. This design ensures that the extreme values ​​of the response intensity of key points on the top-view plane are consistent with the original 3D distribution, providing a reliable 2D feature input for subsequent human position anchor point extraction. At the same time, using a 2D convolutional neural network instead of a 3D convolutional neural network for feature processing significantly reduces computational complexity.

[0057] At the training level of the human anchor point extraction network, this method calculates the 3D Euclidean distance between the real human pose position and each voxel anchor point, selects the shortest distance, calculates the confidence probability score using a 2D Gaussian distribution, and then performs max pooling dimensionality reduction on the height axis to obtain a heatmap of the true human anchor point values, thus constructing a complete supervised training process. This training strategy enables the 2D convolutional neural network to accurately extract human position anchor points from the top-view feature map, implicitly solving the problem of unknown number of people in multi-view matching, because the number of anchor points naturally corresponds to the number of people in the scene.

[0058] At the cross-view matching level, this method calculates the average 3D Euclidean distance between the back-projected joints of each individual's single-view pose set and the human body's anchor point, and then groups the single-view results sets into corresponding anchor points to form a cross-view matching result set. Compared to methods like MVPose that use similarity matrices for view matching, this method uses anchor points extracted from the voxel space top view as cluster centers, organically combining 2D pose decoding with volumetric human detection, improving the efficiency of view matching and solving the problem of unknown numbers in multi-view matching. Furthermore, 2D pose decoding can utilize the full resolution of the fused heatmap without being limited by the coarse voxel mesh resolution, while still leveraging the robust human detection capabilities provided by volumetric aggregation.

[0059] At the system implementation level, this invention also provides a corresponding multi-view 3D human pose estimation system based on voxel space projection and a computer-readable storage medium. The various steps of the above method are organized in a modular manner into a data acquisition module, a 2D feature extraction module, a multi-scale processing and fusion module, a voxel space construction module, a dimensionality reduction and anchor point extraction module, a distance calculation module, a clustering and matching module, and a 3D reconstruction module. This system architecture allows each module to be flexibly deployed on GPUs or CPUs according to their computational characteristics, facilitating engineering implementation and performance optimization.

[0060] In summary, the experimental results of this invention on publicly available datasets demonstrate that the average correct joint prediction percentage reaches 96.47% on the Human3.6M dataset, a 3.67% improvement compared to HRNet, proving the feasibility of the 2D pose estimation method based on stacked multi-view fusion. Experiments on the Shelf and CMU Panoptic datasets show that this method has lower computational complexity than MVPose and significantly lower computational complexity than VoxelPose. In particular, compared to VoxelPose, within an average joint position error increase of only 5.08 mm, the computational cost of this method is reduced to half that of VoxelPose, achieving a good balance between accuracy and computational complexity in 3D keypoints. Qualitative analysis using visualized results shows that it also achieves good results in occlusion scenarios, further demonstrating the feasibility and effectiveness of the method. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating the overall process of a multi-view 3D human pose estimation method based on voxel space projection provided in this invention. The flowchart outlines the complete technical route from multi-view image input, CNN feature extraction, epipolar-constrained multi-view fusion, voxel space construction and top-view projection, human anchor point extraction and multi-view matching to the final 3D pose output. The left side of the diagram shows the epipolar-constrained supervised 2D pose estimation regression network, and the right side shows the orthogonal projection-based multi-view matching part.

[0062] Figure 2 This is a schematic diagram of a single hourglass network structure, showing the symmetrical structure of the first half of the downsampling network (C1 to C6) and the second half of the upsampling network (C3b to C1b), as well as the skip connections between each layer achieved through residual modules and deconvolution operations.

[0063] Figure 3 This diagram illustrates the intermediate fusion method of stacked multi-view fusion, showing the architecture of hourglass networks with multiple views connected in parallel through a multi-view fusion module during the serial process. The heatmaps output by the hourglass networks are fused to generate a fused heatmap, which is then passed to the hourglass structure in the next stage. Detailed Implementation

[0064] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments to facilitate a clear understanding of the present invention, but these descriptions do not constitute a limitation on the present invention.

[0065] Human pose estimation can be divided into 2D pose estimation and 3D pose estimation. Generally, the 2D pose estimation result is used to reconstruct the 3D pose through triangulation. Because backprojection using a single camera is an ill-posed problem (i.e., multiple 3D poses share the same 2D projection), a calibrated multi-camera system is often required for 3D pose estimation. While multi-camera systems can improve the accuracy of 3D pose estimation, they cannot fundamentally solve the problem of 2D pose estimation errors caused by occlusion. This is because when performing 3D pose estimation on multiple targets, the problem of multi-view matching is introduced. It is difficult to find a reliable algorithm to cluster the 2D pose estimation results. If the 2D pose estimation result that does not belong to the individual is used for the triangulation reconstruction of that individual, the 3D pose estimation result will inevitably be incorrect. Therefore, the key to 3D pose estimation lies in improving the accuracy of 2D pose estimation and performing accurate multi-view matching.

[0066] Based on this, the present invention provides a method for fusion of multi-view 3D human pose estimation based on voxel space projection. Its core technical approach includes three closely related technical modules: first, 2D human pose estimation based on stacked multi-view fusion; second, multi-view matching based on voxel space top view; and third, cross-view matching and 3D reconstruction based on 3D Euclidean distance. The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0067] Please see Figure 1 The method provided in this embodiment of the invention includes the following steps:

[0068] S1: Acquire multi-view image data synchronously acquired by a pre-calibrated multi-camera system.

[0069] S2: Input multi-view image data into a CNN network to extract features and generate initial heatmaps of two-dimensional key points from each camera viewpoint.

[0070] S3: Multi-scale processing is performed using a stacked hourglass network, and multi-view feature fusion based on epipolar constraints is performed between adjacent hourglass sub-networks, so that each viewpoint obtains a fused two-dimensional joint heatmap.

[0071] S4: Call the camera's intrinsic and extrinsic parameters to construct a common three-dimensional voxel space, and take the average of the back-projected heatmaps after fusing the various viewpoints to generate a three-dimensional voxel space probability feature volume.

[0072] S5: Project the 3D probabilistic feature volume along the height axis to reduce its dimensionality, obtain the 2D top view feature map, and extract the human body position anchor points through a 2D convolutional network.

[0073] S6: Decode and obtain the single-view pose results of each detected individual, calculate the average three-dimensional Euclidean distance, and group them into the corresponding anchor points.

[0074] S7: Perform triangulation reconstruction or output three-dimensional pose of the 3DPS model based on the matching results.

[0075] In step S1, the multi-camera system consists of multiple synchronously triggered cameras, and the intrinsic and extrinsic parameter matrices of each camera have been pre-calibrated. The intrinsic parameter matrix describes the camera's optical characteristics, including focal length, principal point coordinates, and distortion coefficients; the extrinsic parameter matrix describes the camera's position and orientation in the world coordinate system, including the rotation matrix R and translation vector t. The camera's intrinsic and extrinsic parameters are prerequisites for subsequent steps such as epipolar constraint calculation, voxel space construction, and back projection.

[0076] In step S2, the multi-view image data is input into a CNN network to extract features, generating initial 2D heatmaps of key points for each camera view in the multi-camera system. Specifically, mainstream backbone networks such as HRNet-W32 and ResNet-152 can be used as feature extractors, and the recommended input image resolution is 256×256 or 384×384 pixels. To balance memory and quantization error, the heatmap size is usually set to one-quarter of the input image size; for example, when the input image resolution is 256×256, the heatmap resolution is 64×64. The number of channels in the heatmap corresponds to the number of human key points to be detected; for example, on the Human3.6M dataset, 17 key points are typically used, so the heatmap output has 17 channels.

[0077] Preferably, during the training process of the stacked hourglass network, it is necessary to convert the discrete joint coordinates into continuous training heatmap labels using a Gaussian mixture model based on the human joint coordinates labeled in the training dataset. The method for solving the value at any position on the heatmap corresponding to a certain feature point is shown in formula (1).

[0078] (1)

[0080] In the formula, The result of calculating the training heatmap label corresponding to the feature point is a Gaussian distribution centered at that feature point. The value on, For any position coordinate on the heatmap, Let σ be the coordinates of the labeled center of the i-th feature point, and σ be a manually selected fixed value, which is generally proportional to the size of the heatmap.

[0081] When there are K Gaussian models mixed into a Gaussian mixture model, i.e. a heatmap containing multiple key points, the response distribution of each body joint in the heatmap is described by the Gaussian mixture model, and its probability density function is shown in Equation (2).

[0082] (2)

[0084] Where K is the total number of Gaussian components. To select the weights of the k-th Gaussian model, For the probability density function, a further representation is... Choose the weights of the k-th Gaussian model. It is the first The probability density function of a Gaussian distribution. The mean, Let be the covariance matrix.

[0085] After completing the feature fusion based on epipolar constraints at each stage of the stacked hourglass network, the fused two-dimensional joint heatmaps output by each hourglass subnetwork are used as intermediate supervision objects, and the average mean square error is used as the loss function, as shown in formula (3).

[0086] (3)

[0088] In the formula, K is the number of Gaussian models, and m and n are the height and width of the heatmap. The training heatmap label corresponding to the i-th joint point generated by formula (1) is located at... The value on the image is the actual heatmap value.

[0089] To predict heatmap values, the loss function calculates the spatial distribution difference between the fused heatmap and the training heatmap labels, and performs backpropagation based on this difference to update the network parameters.

[0090] In step S3, as Figure 2 As shown, the stacked hourglass network is composed of multiple single hourglass subnetworks connected in series. Its main feature is that it uses features at multiple scales for attitude estimation.

[0091] Preferably, each individual hourglass sub-network consists of a downsampling first half and an upsampling second half. During the feature map output calculation at each level of the upsampling second half network, the output of the features from the downsampling corresponding scale level, processed by the residual module, is element-wise added to the result of the upsampling of the features from the previous level of the second half through deconvolution. Taking C3b as an example, layer C6 achieves the same resolution as C3a through deconvolution; C3a can be understood as the output of C3 after processing by the residual module. Adding the two together yields C3b. The final feature map has the same scale as the input and retains information from all layers, generating an intermediate-level heatmap.

[0092] Preferably, such as Figure 3 As shown, in the multi-view feature fusion processing of step S3, the fusion module is set between adjacent hourglass sub-networks, utilizing the epipolar constraint relationship constructed based on camera intrinsic and extrinsic parameters. For each camera viewpoint, the intermediate-level heatmap under that viewpoint is used as the main heatmap and fused with the intermediate-level heatmaps under all other camera views. Specifically, the intermediate-level heatmap of the current viewpoint is extracted as the main feature to be enhanced; the epipolar coordinates projected from the other views to the current viewpoint are calculated using camera intrinsic and extrinsic parameters; the response feature distribution in the intermediate-level heatmaps of the other views is extracted along the epipolar coordinate region; the response features are fused with the main features to generate the fused two-dimensional joint heatmap of that viewpoint. The fusion operation can be implemented using methods such as stitching, weighted summation, or attention mechanisms.

[0093] Specifically, for each pixel position on the current viewpoint heatmap, its corresponding epipolar position on the reference viewpoint image plane is determined using a fundamental matrix. The fundamental matrix is ​​calculated from the intrinsic and extrinsic parameter matrices of the two cameras. Let the intrinsic parameter matrix of camera a be... The extrinsic parameter matrix is The intrinsic parameter matrix of camera b is The extrinsic parameter matrix is Then the fundamental matrix F can be derived from the projection matrices of the two cameras, and its core property is: if point p a If is a pixel on the image plane of camera a, then An epipolar line is defined on the image plane of camera b, and all points on this epipolar line are... The projection of the corresponding 3D spatial point onto camera b. Feature sampling of the intermediate-level heatmap of the reference viewpoint along this epipolar direction can accurately locate cross-viewpoint feature correspondences, avoiding the computational overhead of a global search across the entire image plane. In the specific implementation of the fusion operation, the fusion weights can be automatically learned through network training, enabling the network to adaptively determine the contribution level of features from different viewpoints, automatically reducing weights in occluded viewpoints and automatically increasing weights in clear viewpoints.

[0094] Specifically, in the concatenation fusion method, the main features of the current viewpoint and the response features of the reference viewpoint sampled along the epipolar line are concatenated along the channel dimension. Then, a 1×1 convolutional layer is used to reduce the dimensionality of the concatenated features back to the original number of channels. The parameters of this 1×1 convolutional layer are jointly optimized with the entire network during training. In the weighted summation fusion method, a lightweight attention subnetwork (e.g., composed of two fully connected layers and a sigmoid activation function) automatically generates a channel-wise fusion weight vector based on the main features of the current viewpoint and the response features of the reference viewpoint. The reference viewpoint response features are then multiplied element-wise with this weight vector and added to the main features. In the attention mechanism fusion method, a Transformer-style cross-attention module can be used, with the current viewpoint features as the query and the reference viewpoint epipolar features as the key and value for attention calculation. The attention weights are automatically learned during training through end-to-end backpropagation.

[0095] This design, which adds a fusion layer to each hourglass structure and connects sub-networks in parallel during the serial process, allows for supervision from another perspective while preserving the extracted key point features. In effect, it strengthens intermediate supervision, enriching the loss function and resulting in more spatially accurate predicted heatmaps while significantly increasing robustness. This is one of the core innovations of this invention.

[0096] After completing the multi-view fusion in step S3, proceed to step S4. A voxel, also known as a stereo pixel, is the smallest unit of digital data in three-dimensional space; it is a set of feature cubes distributed at the center of an orthogonal grid.

[0097] When constructing a common 3D voxel space, the first step is to determine its coordinate range. Specifically, based on the extrinsic parameter matrices of all cameras in the multi-camera system, the intersection region of the field of view of all cameras in 3D space is calculated. The bounding box of this intersection region is used as the x, y, and z-axis coordinate range of the common 3D voxel space. The coordinate range of the xy plane covers the horizontal activity area of ​​the scene, and the coordinate range of the z-axis is determined based on the possible height range of the human figure in the scene, typically set to the ground height to the maximum standing height of the human figure. The selection of the discretization spacing needs to strike a balance between spatial resolution and computational resources; a smaller spacing results in a larger number of voxels, and a higher spatial resolution but also a greater computational burden.

[0098] Preferably, in step S4, the continuous physical coordinate range corresponding to the common three-dimensional voxel space is equally dispersed into a three-dimensional feature cube array defined by the center of the orthogonal grid, typically configured as 80×80×20 voxels. For each discrete voxel anchor point, its three-dimensional coordinates are back-projected onto the image plane of each viewpoint using the camera's intrinsic and extrinsic parameters. The response probability value of the fused heatmap at the projection position is read, and the response probability values ​​of all viewpoints are averaged. The calculation process is shown in formula (4).

[0099] (4)

[0101] In the formula, It is the heat of a voxel anchor point. The total number of camera viewpoints participating in the observation.

[0102] This represents the probability value of the heatmap at the projected location, calculated using the intrinsic and extrinsic parameters of the v-th camera. The spatial thermal probability values ​​of all discrete voxel anchor points collectively constitute the three-dimensional voxel spatial probability feature.

[0103] After completing the voxel space construction, proceed to step S5. Preferably, a top-view projection transformation matrix P is predefined to map the common three-dimensional voxel space coordinates to a two-dimensional coordinate system. This matrix transforms the three-dimensional vector into a two-dimensional vector with zero depth coordinates, as shown in formulas (5) and (6).

[0104] (5)

[0106] (6)

[0108] In the formula, P is the transformation matrix. The vector before the transformation. The transformed vector sets the z-component of the 3D vector to zero, achieving a mapping from 3D space to a 2D top-view plane. Based on the mapping direction defined by this transformation matrix, the max-pooling mathematical operator is used to shrink and project the 3D voxel space probability feature volume along the height axis onto the 2D horizontal plane, i.e. V x,y The top view features a plane. The voxel anchor point heat value is defined in formula (4). This operation maximizes the response intensity along the height axis (z-axis). This operation ensures that the extreme values ​​of the keypoint response intensity on the top view plane remain consistent with the original 3D distribution. This top view feature plane... Generating top-view heatmaps using a 2D convolutional network Extract human body location anchor points from them.

[0109] Preferably, the training process of the two-dimensional convolutional neural network in step S5 includes: calculating the three-dimensional Euclidean distance between the actual human pose position coordinates provided by the training dataset and each discrete voxel anchor point in the common three-dimensional voxel space, as shown in formula (7).

[0110] (7)

[0112] In the formula, This represents the three-dimensional Euclidean distance between the current voxel anchor point and the actual human pose position. To determine the coordinates of the geometric center of the real human pose labeled in the training dataset in a common 3D voxel space, The spatial coordinates of the current voxel anchor point. This represents the three-dimensional Euclidean distance norm.

[0113] Since there may be multiple targets in the scene, the distance value of the target with the shortest distance to the anchor point is selected as the final true value, as shown in formula (8), where n is the index of the nth target human body in the scene, and p is the spatial position coordinate of the current voxel anchor point.

[0114] (8)

[0116] Based on the shortest distance value, the confidence probability score is calculated using a two-dimensional Gaussian distribution, which causes the confidence to decrease rapidly as the distance increases. The loss function of the human anchor point estimation network (TAN) in the top view is shown in formula (9).

[0117] (9)

[0119] After reducing the dimensionality of the confidence array of each voxel anchor point by pooling the maximum value on the height axis, a heatmap of the true values ​​of human body anchor points on the xy plane is obtained. This value represents the confidence probability that each anchor point may be the center point of the human body. This heatmap, along with the predicted heatmap output by the 2D convolutional network, is substituted into the loss function for supervised training, enabling the network to extract human body position anchor points from the top-view feature map.

[0120] After the anchor point extraction is completed, proceed to step S6. Decode the single-view pose result set of each detected individual from the heatmap after fusion of each viewpoint, and group them using Euclidean distance, as shown in formula (10).

[0121] Specifically, the decoding process involves performing peak detection on each joint channel of the fused heatmap. This can be achieved by using the argmax operation to directly extract the position with the largest response value from the heatmap as the joint's 2D coordinates, or by using a soft-argmax operation to perform a weighted average of the response values ​​to obtain sub-pixel accuracy coordinates. After obtaining the 2D coordinates of each joint, joints belonging to the same individual are grouped together using the detection box information provided by the human detector or a top-down joint grouping strategy, forming a single-view 2D human pose estimation result set for that individual. Each result set contains the 2D pixel coordinates of all detected body joints for that individual.

[0122] (10)

[0124] Preferably, the average value of all joint points is used as the final judgment basis, as shown in formula (11).

[0125] (11)

[0127] In the formula, This indicates the number of different pose points. The three-dimensional Euclidean distance from each attitude point to the anchor point is calculated. Based on the average three-dimensional Euclidean distance, the single-view result sets are grouped and assigned to their corresponding anchor points, forming a cross-view matching result set.

[0128] The specific grouping strategy is as follows: For each human body position anchor point, in the set of all single-view 2D human pose estimation results, select the set of results with the smallest average 3D Euclidean distance to that anchor point and assign it to the target individual cluster corresponding to that anchor point. When multiple anchor points compete for the same set of pose results, assign the result to the anchor point with the smallest distance, ensuring that each set of pose results belongs to at most one anchor point.

[0129] Furthermore, a preset distance threshold can be set. When the minimum distance between a set of pose results and all anchor points still exceeds the preset distance threshold, the result is considered a noise detection result and is not included in any anchor point cluster. This threshold can be adjusted according to the spatial scale of the scene and the voxel spatial resolution. For example, when the voxel spatial coverage is 8m×8m×2m and the voxel resolution is 80×80×20, the side length of a single voxel is approximately 0.1m, and the distance threshold can be set between 0.5m and 1.0m.

[0130] Furthermore, when the same anchor point receives multiple pose results from the same viewpoint, only the set with the smallest distance is retained, ensuring that each anchor point corresponds to at most one individual's detection result in each viewpoint. Through the above grouping strategy, a set of cross-viewpoint matching results for the same target individual is formed. Each set contains 2D pose estimation results of the same individual from multiple different viewpoints, providing a reliable cross-viewpoint correspondence for subsequent 3D reconstruction.

[0131] After completing multi-view matching, proceed to step S7. Two methods can be used to achieve 3D reconstruction: the first is direct triangulation reconstruction, which utilizes multi-view geometric relationships to back-project the 2D coordinates of the same joint point under different viewpoints and solve for the 3D coordinates; the second is inputting a 3D Pictorial Structures (3DPS) model, which fills the 2D joint candidates from each viewpoint into the 3D state space through triangulation, and then uses a graph model for inference to output the final 3D pose. The graph model inference process includes a univariate potential function (describing the likelihood of a single joint point at a specific 3D position), a binary potential function (modeling the spatial constraint relationship between adjacent joint points), and a ternary potential function (modeling the angular constraint relationship between three joint points).

[0132] To verify the effectiveness of the method of the present invention, tests were conducted on multiple public datasets, and comparisons were made with several existing methods.

[0133] In terms of 2D pose estimation accuracy, the average percentage of correct joint predictions (PCK) reaches 96.47% on the Human3.6M dataset, which is 3.67 percentage points higher than HRNet, demonstrating the feasibility of the 2D pose estimation method based on stacked multi-view fusion. Ablation experiments were also conducted on different fusion methods, including splicing fusion, weighted summation fusion, and attention mechanism fusion. Experimental results show that when human pose is occluded, this method can effectively utilize different viewpoints to improve accuracy and robustness.

[0134] In multi-view matching and 3D reconstruction, the Shelf dataset was primarily used to verify the effectiveness of accuracy evaluation and to conduct quantitative index analysis. Considering the large size of the CMU Panoptic dataset, qualitative analysis was conducted in addition to comparing computational speed and complexity. Experimental results show that, within a relatively small range of accuracy degradation compared to other methods, this method has lower computational complexity than MVPose and significantly lower computational complexity than VoxelPose. In particular, compared to VoxelPose, within a range where the average position error per joint (MPJPE) increases by only 5.08 mm, the computational load of this method is reduced to half that of VoxelPose, achieving a good balance between accuracy and computational complexity in 3D keypoints.

[0135] Furthermore, qualitative analysis using visualized results shows that this method also achieves good results when occlusion occurs. This is due to the multi-view fusion module's ability to obtain supplementary information from other unoccluded perspectives, and the fact that there is less interaction between human figures in the top-view projection, which effectively avoids mismatch problems caused by occluded perspectives.

[0136] This invention has at least the following technical effects:

[0137] First, the core innovation of this invention lies in inserting a multi-view epipolar constraint fusion module between the stages of a stacked hourglass network, creating an iterative refinement loop. All existing multi-view epipolar constraint fusion methods treat fusion as a single post-processing step after the backbone network, while this method allows each hourglass sub-network to benefit from the cross-view information accumulated in the preceding stages, significantly improving the accuracy and robustness of 2D pose estimation in occluded scenarios.

[0138] Second, this method solves the problem of inaccurate correspondence between people's positions in traditional multi-view matching by constructing a voxelized space, and uses the characteristics of less interaction and less occlusion between people in the top view projection to extract human body anchor points. Compared with the similarity matrix method used by MVPose, this method improves the efficiency of view matching and solves the problem of unknown number of people.

[0139] Third, this method uses a two-dimensional convolutional neural network instead of a three-dimensional convolutional neural network for top view feature processing, achieving a good balance between spatial discretization and computational complexity. In particular, compared with VoxelPose, the computational load is halved within the range of MPJPE increasing by only 5.08mm.

[0140] Fourth, this method uniquely bridges two technical paradigms: epipolar constraint fusion and volume detection. It feeds the fused heatmap into a voxel construction pipeline to generate higher quality 3D feature volumes. At the same time, through an explicit distance matching mechanism, 2D pose decoding can utilize the full resolution of the fused heatmap without being limited by the voxel mesh resolution.

[0141] This invention also provides a fusion multi-view 3D human pose estimation system based on voxel space projection, the system comprising:

[0142] The data acquisition module is used to perform the function of step S1 above to acquire multi-view image data synchronously acquired by a pre-calibrated multi-camera system.

[0143] The two-dimensional feature extraction module is used to perform the function of step S2 above, inputting multi-view image data into the CNN network to extract features and generate initial two-dimensional joint heatmaps for each camera view.

[0144] The multi-scale processing and fusion module is used to perform the functions of step S3 above. It uses a stacked hourglass network to perform multi-scale processing on the initial heat map and performs multi-view feature fusion processing between adjacent hourglass sub-networks based on the epipolar constraint relationship, so that each viewpoint obtains a fused two-dimensional joint heat map.

[0145] The voxel space construction module is used to perform the function of step S4 above. It calls the camera intrinsic and extrinsic parameters to construct a common three-dimensional voxel space, and takes the average of the back projection of the fused heat map to generate a three-dimensional voxel space probability feature.

[0146] The dimension reduction and anchor point extraction module is used to perform the function of step S5 above, which reduces the dimension of the three-dimensional probabilistic feature volume by projecting it along the height axis, obtains the two-dimensional top view feature map, and extracts human body position anchor points through a two-dimensional convolutional network.

[0147] The distance calculation module is used to perform the decoding and distance calculation functions in step S6 above. It decodes the single-view pose result set of each detected individual from the heat map after fusion of various viewpoints and calculates the average three-dimensional Euclidean distance between it and each human body position anchor point.

[0148] The clustering and matching module is used to perform the grouping and classification function in step S6 above. Based on the average three-dimensional Euclidean distance, it groups and classifies each single-view result set into the corresponding anchor point to form a cross-view matching result set. The three-dimensional reconstruction module is used to perform the function in step S7 above. Based on the matching result set, it performs triangulation reconstruction or outputs the three-dimensional pose of the 3DPS model.

[0149] The modules described above can be deployed on GPU-equipped computing servers, with data transfer between modules via shared memory or message queues. The 2D feature extraction module and the multi-scale processing and fusion module are computationally intensive and are recommended for deployment on GPUs to fully utilize parallel computing capabilities. The voxel space construction module and the dimensionality reduction and anchor point extraction module involve 3D-to-2D projection operations and are also suitable for GPU acceleration. The distance calculation module and the clustering matching module have relatively low computational requirements and can be executed on a CPU. Synchronous acquisition of the multi-camera system can be achieved through hardware trigger signals or network time protocols (such as PTP / IEEE 1588).

[0150] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned method for fusing multi-view 3D human pose estimation based on voxel space projection. The computer-readable storage medium includes, but is not limited to, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices; computing devices equipped with graphics processing units (GPUs) are particularly suitable for performing the deep learning computational tasks involved in this invention.

[0151] In practical implementation, mainstream backbone networks such as HRNet-W32, HRNet-W48, ResNet-50, ResNet-152, or VGG-19 can be used for CNN networks, among which HRNet-W32 offers a good balance between accuracy and speed. The recommended input image resolution is 256×256 or 384×384 pixels, and the corresponding heatmap output resolution is 64×64 or 96×96 pixels. The number of heatmap channels corresponds to the number of detected joints; for example, the Human3.6M dataset uses 17 joints, and the CMU Panoptic dataset uses 15 joints. The number of subnetworks in the stacked hourglass network can be set to 2-8, with 4 recommended. The voxel space resolution can be adjusted according to the scene size and computational resources; a typical configuration is 80×80×20 voxels. Public datasets such as Human3.6M, Shelf, Campus, and CMU Panoptic can be used for training.

[0152] The above embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0153] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A method for estimating 3D human pose based on voxel space projection and fusion of multiple perspectives, characterized in that, include: Acquire multi-view image data of the research object synchronously acquired by a pre-calibrated multi-camera system; The multi-view image data is input into a CNN network to extract features and generate initial two-dimensional heatmaps of key points for each camera view in the multi-camera system. The initial heatmap is processed at multiple scales using a stacked hourglass network containing multiple cascaded single hourglass subnetworks. Each single hourglass subnetwork outputs a corresponding intermediate-level heatmap. A multi-view feature fusion module is set between adjacent single hourglass subnetworks. Utilizing the epipolar constraint relationship between different camera views constructed based on camera intrinsic and extrinsic parameters, for each camera view, the intermediate-level heatmap under that camera view is used as the current main heatmap to be fused. Feature fusion processing is performed with the corresponding intermediate-level heatmaps under all other camera views, so that each camera view obtains a corresponding fused two-dimensional joint heatmap. A common three-dimensional voxel space is constructed by calling the intrinsic and extrinsic parameters of the camera. The fused two-dimensional joint heatmaps corresponding to each camera viewpoint are back-projected onto the common three-dimensional voxel space, and the average value of the projection results of each camera viewpoint is taken to generate a three-dimensional voxel space probability feature. The three-dimensional voxel space probability feature volume is projected and dimension-reduced along a two-dimensional top view plane orthogonal to the height axis to obtain a two-dimensional top view feature map, and a two-dimensional convolutional neural network is called to extract human body position anchor points on the two-dimensional top view feature map. The single-view two-dimensional human pose estimation result set corresponding to each detected individual is obtained by decoding the fused two-dimensional joint point heat map from each camera view. For each obtained single-view two-dimensional human pose estimation result set corresponding to each detected individual, the discrete joint points in the result set are back-projected to three-dimensional space, and the average three-dimensional Euclidean distance between each back-projected discrete joint point and each human position anchor point is calculated. Based on the average three-dimensional Euclidean distance between each single-view two-dimensional human pose estimation result set and each human position anchor point, each single-view two-dimensional human pose estimation result set is grouped and assigned to the corresponding human position anchor point, and aggregated to form a cross-view matching result set for the same target individual. Based on the cross-view 2D poses of the same individual in the matching result set, triangulation reconstruction or input of a 3DPS model is performed to output the final 3D human pose space coordinates of each target individual in the scene.

2. The method for fusion of multi-view 3D human pose estimation based on voxel space projection according to claim 1, characterized in that, The training process of the stacked hourglass network includes: Based on the human joint coordinates labeled in the training dataset, the discrete joint coordinates are transformed into continuous training heatmap labels using a Gaussian mixture model. The response distribution of each body joint in the heatmap is described by the Gaussian mixture model, and its probability density function is determined by the total number of Gaussian components, the normalized weight of each component, the probability density function of each component, the mean, and the covariance matrix. After the feature fusion based on epipolar constraints is completed at each stage of the stacked hourglass network, the fused two-dimensional joint heatmaps output by each hourglass sub-network are used as intermediate supervision objects. The mean square error is used as the loss function to calculate the spatial distribution difference between the fused two-dimensional joint heatmaps and the training heatmap labels, and backpropagation is performed based on the spatial distribution difference to update the network parameters.

3. The method for fusion of multi-view 3D human pose estimation based on voxel space projection according to claim 1, characterized in that, Each single hourglass sub-network consists of a downsampling first half network for feature dimensionality reduction and an upsampling second half network for feature recovery. When calculating the feature map output of each layer of the upsampled second half network, it is required that the feature output of the downsampled features of the corresponding scale layer of the downsampled first half network after processing by the residual module be added and merged element by element with the recovery result of the upsampled features of the previous scale layer of the second half network after deconvolution operation, and finally generate the intermediate heatmap with the same scale as the input feature map of the hourglass sub-network.

4. The method for fusion of multi-view 3D human pose estimation based on voxel space projection according to claim 1, characterized in that, The step of performing feature fusion processing on the intermediate-level heatmap of each camera viewpoint and the corresponding intermediate-level heatmaps of all other camera views includes: Extract the intermediate heatmap output from the current camera view as the main feature to be enhanced, and calculate the epipolar coordinates of all other camera views projected onto the current camera view by calling the pre-stored camera intrinsic and extrinsic parameters. Along the calculated epipolar coordinate region, extract the response feature distribution from the intermediate-level heatmaps of all other camera views; The extracted response feature distribution is fused with the subject feature to be enhanced to generate the fused two-dimensional joint heatmap corresponding to the camera viewpoint.

5. The method for fusion of multi-view 3D human pose estimation based on voxel space projection according to claim 1, characterized in that, The process of generating a three-dimensional voxel space probabilistic feature volume includes: The continuous physical coordinate range corresponding to the public three-dimensional voxel space is equally dispersed into a three-dimensional feature cube array defined by the center of the orthogonal grid. For each discrete voxel anchor point in the array, the three-dimensional spatial coordinates of the voxel anchor point are back-projected onto the image plane of each camera viewpoint using the camera intrinsic and extrinsic parameters to obtain the corresponding projection position. The response probability value of the fused two-dimensional joint heat map of each camera viewpoint at the projection position is read, and the average of the response probability values ​​of all participating camera viewpoints is taken as the spatial thermal probability value of the voxel anchor point. The three-dimensional voxel spatial probability feature is composed of the spatial heat probability values ​​of all discrete voxel anchor points.

6. The method for fusion of multi-view 3D human pose estimation based on voxel space projection according to claim 1, characterized in that, The process of projecting the three-dimensional voxel space probabilistic feature volume onto an orthogonal two-dimensional top-view plane for dimensionality reduction includes: A top-view projection transformation matrix is ​​predefined, and this matrix is ​​used to map the coordinates of the common three-dimensional voxel space to the basic two-dimensional coordinate system, wherein the top-view projection transformation matrix transforms the three-dimensional vector into a two-dimensional vector with zero depth coordinates; Based on the mapping direction defined by the top-view projection transformation matrix, the three-dimensional voxel space probability feature volume is collapsed and projected onto a two-dimensional horizontal plane along the height axis using a maximum pooling mathematical operation operator, generating the two-dimensional top-view feature map where the extreme values ​​of the human body key point response intensity at the corresponding position are consistent with the original three-dimensional distribution.

7. The method for fusion of multi-view 3D human pose estimation based on voxel space projection according to claim 1, characterized in that, In the step of calling the two-dimensional convolutional neural network to extract human body position anchor points on the two-dimensional top view feature map, the training process of the two-dimensional convolutional neural network includes: During the training phase, the three-dimensional Euclidean distance between the geometric coordinates of the real human pose position provided by the training dataset and each discrete voxel anchor point in the common three-dimensional voxel space is calculated. Based on the calculated three-dimensional Euclidean distance, the shortest distance value between each voxel anchor point and the real human posture of all target individuals in the scene is extracted as a regression benchmark. Based on the extracted shortest distance value, the true confidence probability score of the voxel anchor point as the center point of the human body is calculated by using the two-dimensional Gaussian distribution probability density function. The calculated true confidence probability score array of each voxel anchor point is also subjected to maximum pooling dimensionality reduction on the height axis to obtain the true value heatmap of the human body anchor point. This heatmap, along with the predicted heatmap output by the two-dimensional convolutional neural network, is then substituted into the loss function for supervised iterative training. This allows the trained two-dimensional convolutional neural network to extract the human body position anchor point from the two-dimensional top view feature map.

8. The method for fusion of multi-view 3D human pose estimation based on voxel space projection according to claim 1, characterized in that, The step of calculating the average three-dimensional Euclidean distance between each discrete joint point after back projection and each of the human body position anchor points includes: For each individual being detected, the single-view two-dimensional human pose estimation result set is obtained. After back-projecting the discrete joints in the result set using the camera intrinsic and extrinsic parameters, the three-dimensional Euclidean distance between the joints of each body part and the anchor point of the human body position is calculated. The arithmetic mean of the summation of the three-dimensional Euclidean distances of all different joint points is used to obtain the average three-dimensional Euclidean distance used as the basis for matching judgment. Specifically, it is calculated by the total number of different body posture key points contained in the single-view two-dimensional human posture estimation result set, and the three-dimensional Euclidean distance of each specific posture key point from the human position anchor point.

9. A fusion multi-view 3D human pose estimation system based on voxel space projection, characterized in that, To implement the method according to any one of claims 1-8, comprising: The data acquisition module is used to acquire multi-view image data of the research object synchronously collected by a pre-calibrated multi-camera system; The two-dimensional feature extraction module is used to input the multi-view image data into the CNN network to extract features and generate initial two-dimensional joint point heatmaps for each camera view in the multi-camera system. The multi-scale processing and fusion module is used to perform multi-scale processing on the initial heatmap using a stacked hourglass network containing multiple cascaded single hourglass sub-networks. Each single hourglass sub-network outputs a corresponding intermediate-level heatmap. A multi-view feature fusion module is set between adjacent single hourglass sub-networks. Utilizing the epipolar constraint relationship between different camera views constructed based on camera intrinsic and extrinsic parameters, for each camera view, the intermediate-level heatmap under that camera view is used as the current main heatmap to be fused, and feature fusion processing is performed with the corresponding intermediate-level heatmaps under all other camera views, so that each camera view obtains a corresponding fused two-dimensional joint heatmap. The voxel space construction module is used to call the camera intrinsic and extrinsic parameters to construct a common three-dimensional voxel space, back-project the fused two-dimensional joint heatmaps corresponding to each camera viewpoint to the common three-dimensional voxel space, and take the average value of the projection results of each camera viewpoint to generate a three-dimensional voxel space probability feature. The dimension reduction and anchor point extraction module is used to project the three-dimensional voxel space probability feature volume along a two-dimensional top view plane orthogonal to the height axis to reduce the dimension, obtain the two-dimensional top view feature map, and call a two-dimensional convolutional neural network to extract the human body position anchor points on the two-dimensional top view feature map. The distance calculation module is used to decode and obtain the single-view two-dimensional human pose estimation result set corresponding to each detected individual from the fused two-dimensional joint point heat map of each camera view. For the single-view two-dimensional human pose estimation result set corresponding to each detected individual, the discrete joint points in the result set are back-projected to three-dimensional space, and the average three-dimensional Euclidean distance between each discrete joint point after back-projection and each human position anchor point is calculated. The clustering and matching module is used to group each set of two-dimensional human pose estimation results into corresponding human position anchors based on the average three-dimensional Euclidean distance between each set of two-dimensional human pose estimation results and each set of human position anchors, and to aggregate them into a cross-view matching result set for the same target individual. The 3D reconstruction module is used to perform triangulation reconstruction or input a 3DPS model based on the cross-view 2D pose of the same individual in the matching result set to output the final 3D human pose space coordinates of each target individual in the scene.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the fusion multi-view three-dimensional human pose estimation method based on voxel space projection as described in any one of claims 1 to 8.

Images