Human pose estimation method based on adaptive multi-view feature fusion

Abstract

This invention discloses a human pose estimation method based on adaptive multi-view feature fusion. Specifically, it involves: acquiring synchronized multi-view images and preprocessing them to form a dataset, which is then divided into a training set, a validation set, and a test set; designing a multi-view feature fusion network, including a 2D pose estimation backbone network, an appearance embedding network, a geometric embedding network, a weight prediction and learning network, and a multi-view feature weighted fusion network; training and validating the multi-view feature fusion network using the training and validation sets; inputting the test set into the trained multi-view feature fusion network for testing; and outputting a pose estimation image. This method can enhance the features of occluded views by borrowing features from visible views, while adaptively learning view weights to prevent low-quality views from contaminating global features, significantly improving the accuracy and robustness of 3D pose estimation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to a human pose estimation method based on adaptive multi-view feature fusion. Background Technology

[0002] Accurately estimating 3D human pose from multiple camera perspectives has always been one of the core tasks in computer vision. Its ultimate goal is to recover the absolute 3D position of human joints in the world coordinate system from multiple calibrated cameras placed in a natural environment. This task has wide applications in augmented reality, virtual reality, human-computer interaction, and intelligent athlete analysis of sports videos. However, in real-world environments, occlusion is the biggest challenge in human pose estimation. Existing solutions to the occlusion problem can be broadly categorized, but all have significant limitations: The first category relies on additional sensors, such as inertial measurement units, to detect occluded joints. However, sensors suffer from time-varying behavior, and wearing sensors on the human body is invasive, making it unacceptable in many real-world scenarios such as football matches. The second category utilizes the ability of radio signals to penetrate walls for pose estimation, but this approach cannot handle self-occlusion, limiting its application scenarios. In the pure vision-based multi-view 3D pose estimation framework, existing mainstream methods typically rely on a simple two-step approach: first, independently detecting the 2D pose of each viewpoint, and then recovering the 3D pose through triangulation or graph structure models. This method is highly dependent on the accuracy of 2D detection.

[0003] To handle outliers in multi-sensor or multi-view fusion, traditional methods often employ consensus learning algorithms such as RANSAC. However, in multi-view human pose estimation tasks, the number of cameras is typically small, such as only 4 to 8, making the probability of RANSAC detecting genuine outliers extremely low, resulting in less than ideal performance. Other feature fusion methods based on spatial correspondence lack flexibility in practical applications, often requiring the training of a separate fusion network for each possible new camera placement, leading to poor generalization ability.

[0004] Furthermore, in narrow-baseline multi-view fusion, accurate point-to-point matching is particularly challenging for textureless human body regions. Therefore, there is an urgent need for an efficient multi-view adaptive fusion method that can adapt to new camera configurations without retraining and robustly handle occlusion. Summary of the Invention

[0005] The purpose of this invention is to provide a human pose estimation method based on adaptive multi-view feature fusion, which significantly improves the accuracy and robustness of 3D pose estimation.

[0006] The technical solution adopted in this invention is a human pose estimation method based on adaptive multi-view feature fusion, which is implemented according to the following steps:

[0007] Step 1: Acquire multi-view synchronized images and preprocess them to form a dataset, which is then divided into a training set, a validation set, and a test set. Step 2: Design a multi-view feature fusion network, including a 2D pose estimation backbone network, an appearance embedding network, a geometric embedding network, a weight prediction and learning network, and a multi-view feature weighted fusion network. Step 3: Train and validate the multi-view feature fusion network using the training set and validation set, input the test set into the trained multi-view feature fusion network for testing, and output the pose estimation image.

[0008] The invention is further characterized in that, Step 1 specifically involves: given N cameras with known intrinsic and extrinsic parameters, acquiring synchronized RGB images from multiple viewpoints at the same time; using an object detection algorithm to extract human body bounding boxes, then cropping each extracted human body image and scaling it to a uniform size. The pixels are used to form a dataset, which is then divided into a training set, a validation set, and a test set.

[0009] Step 2 is detailed as follows: Step 2.1: Generate an initial 2D pose heatmap using a 2D pose estimation backbone network: Input each preprocessed single-view image frame into a deep convolutional neural network for pose estimation. The network independently processes the image from each viewpoint and outputs the response probability distribution of each joint point on the 2D image. For each joint point, the network outputs a single-channel image with a spatial resolution of [missing information]. Initial 2D attitude heatmap; Step 2.2: Input the initial 2D pose heatmap into the appearance embedding network to extract appearance embedding features based on single-view quality; Step 2.3: Input the initial 2D pose heatmap into the geometric embedding network to extract geometric embedding features based on cross-view consistency; Step 2.4: Input the above features into the weight prediction and learning network to integrate the above appearance and geometric information, and assign a scalar weight between 0 and 1 to each viewpoint; Step 2.5, Multi-view Feature Weighted Fusion Network: This network combines the adaptive quality weights of each viewpoint obtained in Step 2.4 with the physical spatial geometric constraints between cameras to efficiently fuse the initial 2D pose heatmaps from multiple views.

[0010] Step 2.2 specifically involves: Step 2.21, Feature downsampling and spatial feature extraction: Extracting individual key points from the output of Step 2.1... The initial 2D pose heatmap is input into the appearance embedding network; firstly, through convolutional layers with a ReLU activation function, local spatial features and texture information are extracted, and the size is downsampled to... Subsequently, using a kernel size of A global average pooling layer is used to reduce the spatial dimension, compress the spatial feature map, and extract the global distribution information of the heatmap, ultimately obtaining a result of size [size missing]. Feature map; Step 2.22, Generate appearance embedding: The image obtained in step 2.21... The feature map is fed into a fully connected layer for non-linear mapping, and finally outputs a fixed 128-dimensional vector.

[0011] Step 2.3 specifically involves: Step 2.31, Extracting Continuous Two-Dimensional Coordinates: Soft-argmax calculates the expected value of all pixel coordinates on the initial 2D pose heatmap. First, the initial 2D pose heatmap is transformed into a global two-dimensional probability distribution using the SoftMax function. Then, the coordinate values ​​of the two-dimensional grid in the image are multiplied by the corresponding probability values ​​and integrated globally to extract the sub-pixel level continuous two-dimensional coordinates of the target joint. At the same time, the corresponding maximum response confidence level is extracted. ; Step 2.32, calculate Sampson distance: using the known intrinsic and extrinsic parameters of the two cameras, derive the fundamental matrix F between the two views;

[0012] in, This is the predicted two-dimensional homogeneous coordinate column vector from the current reference viewpoint; For the corresponding two-dimensional homogeneous coordinate column vectors predicted from other auxiliary perspectives; This is the fundamental matrix from the auxiliary perspective to the current baseline perspective; , Representing the polar equation vectors respectively The first and second components; Calculate the predicted two-dimensional coordinates from the current viewpoint using the formula. Coordinates predicted by other auxiliary perspectives Sampson distance between them; Step 2.33, Generate geometric embedding: The scalar Sampson distance calculated above is mapped to a 256-dimensional high-dimensional feature through a sub-network containing two fully connected layers; the 256-dimensional distance features generated by the current viewpoint and all its paired auxiliary viewpoints are averaged to finally obtain a geometric embedding vector with viewpoint invariance and a fixed 256-dimensional dimension.

[0013] Step 2.4 specifically involves: Step 2.41, Feature concatenation: The 128-dimensional appearance embedding vector obtained in Step 2.2 and the 256-dimensional geometric embedding vector obtained in Step 2.3 are concatenated along the feature channel dimension to obtain a 384-dimensional fused feature vector; Step 2.42, Regression and Fusion Weights: Input the concatenated 384-dimensional fusion features into a lightweight weight learning network consisting of three fully connected layers. The network nodes are as follows: The last output node, after passing through the Sigmoid activation function, is compressed to... The interval is used to regress and output the adaptive fusion weight scalar for each camera view.

[0014] Step 2.5 specifically involves: Step 2.51, Calculate the epipolar mapping: Based on the principle of multi-view epipolar geometry, for the target baseline view that needs to be feature-enhanced... Iterate through every pixel coordinate point on its initial 2D pose heatmap. Using the system-calibrated view and auxiliary view The fundamental matrix F between them is used to analytically calculate the pixel points. In the target auxiliary view The strictly corresponding polar lines ; Step 2.52, One-dimensional maximum response search: along the target auxiliary view Middle polar line Within the covered set of discrete coordinates, a one-dimensional search with dimensionality reduction is performed to find and extract the maximum response value. As an auxiliary view, it compensates for the features of the current base view; Step 2.53, Perform adaptive feature weighted fusion: Adaptively fuse weight scalars using the auxiliary view from the regression output in Step 2.4. The maximum response values ​​of the extracted epipolar lines are weighted and added to the initial response value of the baseline view itself to complete the cross-view information fusion at the feature level, resulting in a fused global heatmap. The specific multi-view adaptive weighted fusion formula is as follows:

[0015] in, This represents the final heatmap response value after enhanced fusion; Adaptive fusion weights for the baseline view; The initial thermal map response value for the baseline view; To assist in the total number of perspectives to be merged; For auxiliary views in polar coordinates The response value at the location; Step 2.54: Apply the SoftMax exponential normalization operator to the fused global heatmap to finally output the refined 2D fused heatmap; Step 2.55, 3D pose reconstruction: Based on the refined 2D fused heatmap, the Soft-argmax operator is used again to extract the final refined 2D sub-pixel extreme coordinates of each viewpoint; then, using multi-view triangulation, combined with the intrinsic and extrinsic parameters of each camera, the highly accurate 3D spatial coordinates of the human body key points in the world coordinate system are reconstructed and output.

[0016] Step 3 specifically involves: Step 3.1, train the loss function, which is defined by the formula:

[0017] in, The total number of joints. To predict heat maps, This is a heatmap of real labels; Step 3.2, Training conditions and backpropagation: The error gradient starts from the MSE loss function mentioned above and backpropagates through the multi-view feature fusion network; Step 3.3, Training Termination Condition: Update parameters using the Adam optimizer, with an initial learning rate of 0.001; perform iterative training on the training set and monitor the mean joint position error on the validation set; terminate training when the MPJPE on the validation set no longer decreases for 10 consecutive epochs, or when the preset maximum number of iterations is reached, save the current optimal network weight model, input the test set into the trained multi-view feature fusion network for testing, and output the pose estimation image.

[0018] The beneficial effects of this invention are: (1) Through adaptive fusion, the position of the occluded joint can be accurately located by utilizing the high-quality features of other perspectives, thereby avoiding reliance on invasive sensors. (2) A coarse-to-fine approach is adopted. First, the correspondence between points and epipolar lines is established through epipolar geometry. Then, the maximum response value on the epipolar line is directly selected by utilizing the natural sparsity of the feature heatmap. This avoids the matching problem in textureless regions. (3) This invention combines appearance distribution and cross-view geometric consistency to jointly infer view quality. For viewpoints whose features are damaged due to severe occlusion or false detection, a smaller weight is assigned, thereby ensuring that high-quality viewpoints dominate the fusion process and solving the defect of traditional heuristic average fusion being susceptible to noise interference.

[0019] (4) The network weights of this invention are fully shared for different joint types and camera viewpoints, and because the averaging operation is used to integrate geometric features, the model is insensitive to the number of input cameras. For deployment environments with different numbers of cameras and new camera extrinsics, AdaFuse can be applied directly without retraining.

[0020] (5) In simple epipolar-based cross-fusion, background pixels may be inexplicably enhanced. This invention finds that erroneous locations can only be enhanced by a maximum of one additional viewpoint, while correct locations are enhanced by multiple viewpoints. Therefore, applying the SoftMax operation directly after fusion can efficiently eliminate small interference noise and ensure accuracy. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the structure of the multi-view feature fusion network in the human pose estimation method based on adaptive multi-view feature fusion of the present invention; Figure 2 This is a schematic diagram of the appearance embedding network in the human pose estimation method based on adaptive multi-view feature fusion of the present invention; Figure 3 This is a schematic diagram of the geometric embedding network structure in the human pose estimation method based on adaptive multi-view feature fusion of the present invention. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0023] Example 1 This invention is a human pose estimation method based on adaptive multi-view feature fusion, which is implemented according to the following steps: Step 1: Acquire and preprocess multi-view synchronized images; Given N cameras with known intrinsic and extrinsic parameters, acquire synchronized RGB images of multiple views at the same time. Suppose the system contains N cameras with known intrinsic parameters (focal length, principal point, etc.) and extrinsic parameters (rotation matrix R and translation vector t), synchronously acquiring multi-view images for each frame. To reduce background noise interference and computational complexity, a target detection algorithm (such as YOLO) is used to extract human bounding boxes. Subsequently, each extracted human image frame is cropped and uniformly scaled to a larger size. The pixels are used to form a dataset, which is then divided into a training set, a validation set, and a test set. Step 2: Design a multi-view feature fusion network, including a 2D pose estimation backbone network, an appearance embedding network, a geometric embedding network, a weight prediction and learning network, and a multi-view feature weighted fusion network, such as... Figure 1 As shown, the details are as follows: Step 2.1: Generate an initial 2D pose heatmap using a 2D pose estimation backbone network: Input each preprocessed single-view image frame into a deep convolutional neural network (such as HRNet or SimpleBaseline) for pose estimation. The network independently processes the image from each viewpoint and outputs the response probability distribution of each joint point on the 2D image. For each joint point, the network outputs a single-channel image with a spatial resolution of [missing information]. 2D heatmap (i.e., size is) This is the initial 2D attitude heatmap. Physically, the heatmap represents a two-dimensional Gaussian distribution; a higher response value (closer to 1) indicates a greater probability that the pixel is a target joint. Step 2.2: The appearance embedding network is used to extract appearance embedding features based on single-view quality. This aims to preliminarily assess the reliability level of joint features from that viewpoint by utilizing the appearance distribution characteristics of the heatmap. Figure 2 As shown, specifically: Step 2.21, Feature downsampling and spatial feature extraction: Extracting individual key points from the output of Step 2.1... The initial 2D pose heatmap is input to the appearance embedding network. First, local spatial features and texture information are extracted through convolutional layers (typically using convolutional kernels with a stride of 4, or consisting of consecutive lightweight residual blocks, expanding the number of channels to 256), coupled with the ReLU activation function, and the size is downsampled to... Subsequently, using a kernel size of A global average pooling layer is used to reduce the spatial dimension, compress the spatial feature map, and extract the global distribution information of the heatmap, ultimately obtaining a result of size [size missing]. The feature map.

[0024] Step 2.22, Generate appearance embedding: The image obtained in step 2.21... The feature map is fed into a fully connected layer (FC layer, with an input dimension of 256 and an output dimension of 128) for non-linear mapping, ultimately outputting a fixed 128-dimensional vector, which is the appearance embedding vector reflecting the distribution characteristics of the heatmap. In this step, different joint types and different camera views share the exact same network parameters, which not only greatly reduces the number of model parameters but also ensures the translation and viewpoint invariance of feature extraction.

[0025] Step 2.3, the geometric embedding network is used to extract geometric embedding features based on cross-view consistency; such as Figure 3 As shown, specifically: Step 2.31, Extracting Continuous Two-Dimensional Coordinates: Soft-argmax calculates the expected value by considering all pixel coordinates on the initial 2D pose heatmap. The calculation process of Soft-argmax is as follows: First, the initial 2D pose heatmap is transformed into a global two-dimensional probability distribution using the SoftMax function. Then, the coordinate values ​​of the image's two-dimensional grid ( and The values ​​are multiplied by their corresponding probability values ​​and then integrated globally (i.e., the expected value is calculated) to extract the sub-pixel level continuous two-dimensional coordinates of the target joint. At the same time, the corresponding maximum response confidence level is extracted. This operation makes the entire coordinate extraction process differentiable, supporting end-to-end training.

[0026] Step 2.32, calculate the Sampson distance: The Sampson distance is an efficient first-order approximation method for calculating reprojection errors in epipolar geometry of two views. Using the known intrinsic and extrinsic parameters of the two cameras, the fundamental matrix F between the two views is derived.

[0027]

[0028] in, This is the predicted two-dimensional homogeneous coordinate column vector from the current reference viewpoint; For the corresponding two-dimensional homogeneous coordinate column vectors predicted from other auxiliary perspectives; The fundamental matrix from the auxiliary viewpoint to the current reference viewpoint ( (matrix) , Representing the polar equation vectors respectively The first and second components; This formula calculates the Sampson distance between the 2D coordinates predicted from the current viewpoint and the coordinates predicted from other auxiliary viewpoints. This distance intuitively reflects the geometric correspondence error across viewpoints; the larger the distance, the more likely the prediction from that viewpoint is to violate the true 3D structure of the physical world.

[0029] Calculate the predicted two-dimensional coordinates from the current viewpoint using the formula. Coordinates predicted by other auxiliary perspectives The Sampson distance between them. This distance intuitively reflects the geometric correspondence error across viewpoints; the larger the distance, the more likely the prediction from that viewpoint is to violate the true three-dimensional structure of the physical world.

[0030] Step 2.33, Generate Geometric Embedding: The scalar Sampson distance calculated above is mapped to a 256-dimensional high-dimensional feature through a sub-network containing two fully connected layers (e.g., input dimension 1, hidden layer dimension 64, output dimension 256). To enable the model to adapt to any number of camera environments, the 256-dimensional distance features generated by the current viewpoint and all its paired auxiliary viewpoints are averaged. This results in a geometric embedding vector with a fixed 256-dimensional dimension and viewpoint invariance.

[0031] Step 2.4: The weight prediction and learning network integrates the aforementioned appearance and geometric information, assigning a scalar weight between 0 and 1 to each viewpoint to accurately reflect its feature quality, thereby mitigating the negative impact of low-quality views in subsequent fusion. Specifically: Step 2.41, Feature concatenation: The 128-dimensional appearance embedding vector obtained in Step 2.2 and the 256-dimensional geometric embedding vector obtained in Step 2.3 are concatenated along the feature channel dimension to obtain a 384-dimensional fused feature vector; Step 2.42, Regression and Fusion Weights: Input the concatenated 384-dimensional fusion features into a lightweight weight learning network consisting of three fully connected layers (network nodes are as follows). The last output node, after passing through the Sigmoid activation function, is compressed to... The network iterates through the range of camera views to generate adaptive fusion weight scalars for each view. When a view has severe object occlusion (damaged appearance distribution) or false positives in quadratic counting (violation of geometric consistency), the network will produce an output close to 0 based on the low-quality input features; conversely, it assigns high weights close to 1 to high-quality views.

[0032] Step 2.5, Multi-view Feature Weighted Fusion Network: This network combines the adaptive quality weights obtained in Step 2.4 with the physical spatial geometric constraints between cameras to efficiently fuse the initial 2D pose heatmaps from multiple views. Specifically: Step 2.51, Calculate the epipolar mapping: Based on the principle of multi-view epipolar geometry, for the target baseline view that needs to be feature-enhanced... Iterate through every pixel coordinate point on its initial 2D pose heatmap. Using the system-calibrated view and auxiliary view The fundamental matrix F between them is used to analytically calculate the pixel points. In the target auxiliary view The strictly corresponding polar lines .

[0033] Step 2.52, One-dimensional maximum response search: Due to the extremely strong spatial sparsity of the feature heatmap, directly and strictly follow the target auxiliary view. Middle polar line Within the covered set of discrete coordinates, a one-dimensional search with dimensionality reduction is performed to find and extract the maximum response value. This serves as an auxiliary view for feature compensation of the current baseline view.

[0034] Step 2.53, Perform adaptive feature weighted fusion: Adaptively fuse weight scalars using the auxiliary view from the regression output in Step 2.4. The maximum response values ​​of the extracted epipolar lines are weighted and added to the initial response value of the baseline view itself to complete the cross-view information fusion at the feature level, resulting in a fused global heatmap. The specific multi-view adaptive weighted fusion formula is as follows:

[0035] in, This represents the final heatmap response value after enhanced fusion; Adaptive fusion weights for the baseline view; The initial thermal map response value for the baseline view; To assist in the total number of fusion perspectives; For auxiliary views in polar coordinates The response value at that location.

[0036] Step 2.54, Eliminating Background Ambiguity and Outputting a Refined Heatmap: To eliminate the "background ghosting" noise generated during epipolar cross-mapping, the SoftMax exponential normalization operator is applied to the fused global heatmap. This operation exponentially widens the gap between the true peak values ​​of multiple line intersections and the false peak values ​​accidentally swept by a single line, filtering out interference noise and finally outputting a refined 2D fused heatmap.

[0037] Step 2.55, 3D pose reconstruction: Based on the refined 2D fused heatmap, the Soft-argmax operator is used again to extract the final refined 2D sub-pixel extreme coordinates of each viewpoint. Then, using multi-view triangulation (such as Direct Linear Transform DLT) and combining the intrinsic and extrinsic parameters of each camera, the highly accurate 3D spatial coordinates of the human body key points in the world coordinate system are reconstructed and output. Step 3, train the multi-view feature fusion network: using an end-to-end deep learning training paradigm; Example 2 Further, step 3 specifically involves: Step 3.1, Training the Loss Function: The mean squared error between the refined 2D fused heatmap (feature extraction by the network front-end (Step 2.1), adaptive weighted fusion by the mid-end (Step 2.53), and the refined 2D fused heatmap (output by the back-end (Step 2.54)) and the 2D real Gaussian heatmap projected from manually labeled real 3D coordinates) is used as the sole objective loss function for optimization. The loss function formula is defined as:

[0038] in, The total number of joints. To predict heat maps, This is a heatmap of real labels.

[0039] Step 3.2, Training Conditions and Backpropagation: The error gradient starts from the MSE loss function mentioned above and propagates back through the fusion module, the weight learning network, and the front-end pose feature extraction network. Since the Soft-argmax, fully connected layers, and SoftMax normalization in the network are all differentiable operators, there are no non-differentiable hard truncation operations in the entire chain. During this process, the weight learning network is implicitly optimized by sensing changes in the fusion error, without the need to apply any intermediate ground truth supervision signals.

[0040] Step 3.3, Training Termination Condition: Update parameters using the Adam optimizer, with an initial learning rate of 0.001. Iteratively train on the training set and monitor the mean joint position error on the validation set. Terminate training when the MPJPE on the validation set stops decreasing for 10 consecutive epochs (early stopping mechanism), or when the preset maximum number of iterations (e.g., 100 epochs) is reached. Save the current optimal network weight model, fuse multi-view features from the test set into the network for testing, and output the pose estimation image.

[0041] Example 3 Using the UnrealCV engine, we render multi-view images and depth maps in virtual indoor scenes (living room, bedroom, etc.) to construct a synthetic dataset (Occlusion-Person) with severe occlusion labels. The dataset contains 13 human models performing various everyday actions, with objects such as sofas and tables intentionally used to occlude the human joints, resulting in an occlusion rate of approximately 20%. Eight virtual cameras are evenly arranged in a ring around the scene. The dataset is divided into training and test sets at 75% and 25% of the total area, respectively.

[0042] The mean joint position error (MPJPE) is used as the accuracy evaluation index for 3D pose estimation. The smaller the MPJPE, the closer the 3D coordinates estimated by the algorithm are to the true coordinates, and the higher the accuracy.

[0043] The test set was input into the trained multi-view feature fusion network for testing. Experimental results show that the method of the present invention achieves an average MPJPE of 12.6 mm across all joints. Especially when dealing with severely occluded joints (such as knees, ankles, and wrists), the method of the present invention can adaptively assign extremely low fusion weights to them and utilize features from other clear viewpoints to achieve high-precision localization of occluded areas.

[0044] Example 4 To verify the superiority of the method of this invention, it was compared with the following baseline methods on the Occlusion-Person dataset: No-fusion method: 2D prediction is performed using only a single viewpoint, followed by direct triangulation. Its MPJPE is as high as 48.1 mm because a single viewpoint cannot handle severe occlusion.

[0045] Heuristic averaging fusion: This method performs a simple, equal-weighted fusion of features from all viewpoints. Its MPJPE is 18.0 mm. However, because it doesn't differentiate between view quality, low-quality occluded views contaminate the overall features.

[0046] Traditional RANSAC filtering method: Utilizes the RANSAC algorithm to remove outliers during 3D reconstruction. Its MPJPE is 15.5mm. Due to the limited number of cameras (8), RANSAC struggles to reliably find true interior points.

[0047] The method of this invention: MPJPE is 12.6mm.

[0048] Compared to non-fusion methods, this invention reduces 3D errors by approximately 73.8%. Compared to traditional RANSAC methods and simple average fusion methods, this invention effectively blocks the contamination of low-quality views by adaptively learning view weights, demonstrating superior accuracy and robustness in complex natural environments containing a large number of occlusions and outliers.

[0049] Example 5 Accurate multi-view 3D human pose estimation is performed in a complex indoor environment. The scene contains numerous everyday objects such as sofas, tables, and chairs that could severely obscure human figures. Eight calibrated RGB cameras are evenly arranged in a ring within the system. The acquired video data is divided into training and validation sets, and the skeleton to be estimated contains 15 key human body nodes. After preprocessing such as cropping each frame of image, it is input into the 2D pose estimation backbone network in parallel to initially extract 2D initial heatmaps of 15 core human joints in 8 independent viewpoints and output the initial 2D pose heatmap.

[0050] Appearance and Geometric Embedding Feature Extraction: For a specific joint (e.g., the right wrist joint), in viewpoint 1, this joint is severely obscured by the sofa in the room. At this time, the appearance embedding network detects that the heatmap response of viewpoint 1 shows a scattered and small non-Gaussian distribution, and then outputs appearance features with low confidence; simultaneously, the geometric embedding network calculates that the coordinates extracted by viewpoint 1 through Soft-argmax have a large Sampson reprojection error with the coordinates of the other 7 clear viewpoints, and therefore outputs geometric features with low consistency.

[0051] Adaptive weight allocation: After receiving the two spliced ​​features mentioned above, the weight learning network accurately identifies that the local features of viewpoint 1 have been severely damaged, and thus assigns it a very low fusion weight (e.g., 0.08). Conversely, for viewpoint 3, which is unoccluded, has a perfect Gaussian distribution in its heatmap, and exhibits extremely consistent geometric correspondence across multiple views, the network adaptively assigns a high fusion weight (e.g., 0.96).

[0052] Multi-view fusion and denoising: Using the occluded viewpoint 1 as the target reference view, the pixels on its image plane are mapped to epipolar lines from the other seven auxiliary viewpoints using the epipolar geometry of the calibrated camera. Response peaks are extracted along these epipolar line trajectories and weighted compensation is performed strictly according to the assigned weights. By leveraging the high response features of unoccluded, high-quality views such as viewpoint 3, the right wrist joint features lost due to occlusion in viewpoint 1 are significantly and accurately enhanced. Subsequently, a SoftMax operation is applied to the fused heatmap to completely remove background ghosting noise caused by mismatched epipolar line intersections.

[0053] 3D Pose Output: After the adaptive weighted fusion described above, the 2D estimation results from the eight perspectives were all protected from contamination by inferior views, resulting in significant correction and refinement. Finally, these eight highly consistent and accurate 2D coordinates were input into the triangulation algorithm module to geometrically calculate the accurate 3D spatial coordinates of the right wrist joint. This significantly reduced the average joint position error of the occluded joint, overcoming the challenge of multiple occlusions in the natural environment.

[0054] Through this series of steps, the absolute 3D pose coordinates can be inferred with strong robustness when there is human self-occlusion and object occlusion, and it can adapt to any multi-view environment without retraining.

[0055] Example 6 The human pose estimation method based on adaptive multi-view feature fusion of the present invention can enhance the features of the occluded view by borrowing the features of the visible view, and at the same time adaptively learn the view weights to block the pollution of global features by low-quality views, thus significantly improving the accuracy and robustness of 3D pose estimation.

Claims

1. A human pose estimation method based on adaptive multi-view feature fusion, characterized in that, The specific steps are as follows: Step 1: Acquire multi-view synchronized images and preprocess them to form a dataset, which is then divided into a training set, a validation set, and a test set. Step 2: Design a multi-view feature fusion network, including a 2D pose estimation backbone network, an appearance embedding network, a geometric embedding network, a weight prediction and learning network, and a multi-view feature weighted fusion network. Step 3: Train and validate the multi-view feature fusion network using the training set and validation set, input the test set into the trained multi-view feature fusion network for testing, and output the pose estimation image.

2. The human pose estimation method based on adaptive multi-view feature fusion as described in claim 1, characterized in that, In the step 1, specifically, for a camera with N known internal and external parameters, multiple multi-view synchronous RGB images at the same time are obtained; a target detection algorithm is used to obtain a human body bounding box, and then each extracted human body image is cropped and uniformly scaled to pixels to form a data set, which is divided into a training set, a validation set and a test set.

3. The human pose estimation method based on adaptive multi-view feature fusion as described in claim 1, characterized in that, Step 2 is specifically as follows: Step 2.1: Generate an initial 2D pose heatmap using a 2D pose estimation backbone network: Input each preprocessed single-view image frame into a deep convolutional neural network for pose estimation. The network independently processes the image from each viewpoint and outputs the response probability distribution of each joint point on the 2D image. For each joint point, the network outputs a single-channel image with a spatial resolution of [missing information]. Initial 2D attitude heatmap; Step 2.2: Input the initial 2D pose heatmap into the appearance embedding network to extract appearance embedding features based on single-view quality; Step 2.3: Input the initial 2D pose heatmap into the geometric embedding network to extract geometric embedding features based on cross-view consistency; Step 2.4: Input the above features into the weight prediction and learning network to integrate the above appearance and geometric information, and assign a scalar weight between 0 and 1 to each viewpoint; Step 2.5, Multi-view Feature Weighted Fusion Network: This network combines the adaptive quality weights of each viewpoint obtained in Step 2.4 with the physical spatial geometric constraints between cameras to efficiently fuse the initial 2D pose heatmaps from multiple views.

4. The human pose estimation method based on adaptive multi-view feature fusion as described in claim 3, characterized in that, In step 2.2, specifically: Step 2.21, Feature downsampling and spatial feature extraction: Extracting individual key points from the output of Step 2.1... The initial 2D pose heatmap is input into the appearance embedding network; firstly, through convolutional layers with a ReLU activation function, local spatial features and texture information are extracted, and the size is downsampled to... ; Subsequently, using a kernel size of A global average pooling layer is used to reduce the spatial dimension, compress the spatial feature map, and extract the global distribution information of the heatmap, ultimately obtaining a result of size [size missing]. Feature map; Step 2.22, Generate appearance embedding: The image obtained in step 2.21... The feature map is fed into a fully connected layer for non-linear mapping, and finally outputs a fixed 128-dimensional vector.

5. The human pose estimation method based on adaptive multi-view feature fusion as described in claim 4, characterized in that, In step 2.3, specifically: Step 2.31, Extracting Continuous Two-Dimensional Coordinates: Soft-argmax calculates the expected value of all pixel coordinates on the initial 2D pose heatmap. First, the initial 2D pose heatmap is transformed into a global two-dimensional probability distribution using the SoftMax function. Then, the coordinate values ​​of the two-dimensional grid in the image are multiplied by the corresponding probability values ​​and integrated globally to extract the sub-pixel level continuous two-dimensional coordinates of the target joint. At the same time, the corresponding maximum response confidence level is extracted. ; Step 2.32, calculate Sampson distance: using the known intrinsic and extrinsic parameters of the two cameras, derive the fundamental matrix F between the two views; in, This is the predicted two-dimensional homogeneous coordinate column vector from the current reference viewpoint; For the corresponding two-dimensional homogeneous coordinate column vectors predicted from other auxiliary perspectives; This is the fundamental matrix from the auxiliary perspective to the current baseline perspective; , Representing the polar equation vectors respectively The first and second components; Calculate the predicted two-dimensional coordinates from the current viewpoint using the formula. Coordinates predicted by other auxiliary perspectives Sampson distance between them; Step 2.33, Generate geometric embedding: The scalar Sampson distance calculated above is mapped to a 256-dimensional high-dimensional feature through a sub-network containing two fully connected layers; the 256-dimensional distance features generated by the current viewpoint and all its paired auxiliary viewpoints are averaged to finally obtain a geometric embedding vector with viewpoint invariance and a fixed 256-dimensional dimension.

6. The human pose estimation method based on adaptive multi-view feature fusion as described in claim 5, characterized in that, In step 2.4, specifically: Step 2.41, Feature concatenation: The 128-dimensional appearance embedding vector obtained in Step 2.2 and the 256-dimensional geometric embedding vector obtained in Step 2.3 are concatenated along the feature channel dimension to obtain a 384-dimensional fused feature vector; Step 2.42, Regression and Fusion Weights: Input the concatenated 384-dimensional fusion features into a lightweight weight learning network consisting of three fully connected layers. The network nodes are as follows: The last output node, after passing through the Sigmoid activation function, is compressed to... The interval is used to regress and output the adaptive fusion weight scalar for each camera view.

7. The human pose estimation method based on adaptive multi-view feature fusion as described in claim 6, characterized in that, Step 2.5 specifically includes: Step 2.51, Calculate the epipolar mapping: Based on the principle of multi-view epipolar geometry, for the target baseline view that needs to be feature-enhanced... Iterate through every pixel coordinate point on its initial 2D pose heatmap. Using the system-calibrated view and auxiliary view The fundamental matrix F between them is used to analytically calculate the pixel points. In the target auxiliary view The strictly corresponding polar lines ; Step 2.52, One-dimensional maximum response search: along the target auxiliary view Middle polar line Within the covered set of discrete coordinates, a one-dimensional search with dimensionality reduction is performed to find and extract the maximum response value. As an auxiliary view, it compensates for the features of the current base view; Step 2.53, Perform adaptive feature weighted fusion: Adaptively fuse weight scalars using the auxiliary view from the regression output in Step 2.

4. The maximum response values ​​of the extracted epipolar lines are weighted and added to the initial response value of the baseline view itself to complete the cross-view information fusion at the feature level, resulting in a fused global heatmap. The specific multi-view adaptive weighted fusion formula is as follows: in, This represents the final heatmap response value after enhanced fusion; Adaptive fusion weights for the baseline view; The initial thermal map response value for the baseline view; To assist in the total number of fusion perspectives; For auxiliary views in polar coordinates The response value at the location; Step 2.54: Apply the SoftMax exponential normalization operator to the fused global heatmap to finally output the refined 2D fused heatmap; Step 2.55, 3D pose reconstruction: Based on the refined 2D fused heatmap, the Soft-argmax operator is used again to extract the final refined 2D sub-pixel extreme coordinates of each viewpoint; then, using multi-view triangulation, combined with the intrinsic and extrinsic parameters of each camera, the highly accurate 3D spatial coordinates of the human body key points in the world coordinate system are reconstructed and output.

8. The human pose estimation method based on adaptive multi-view feature fusion as described in claim 7, characterized in that, Step 3 specifically includes: Step 3.1, train the loss function, which is defined by the formula: in, The total number of joints. To predict heat maps, This is a heatmap of real labels; Step 3.2, Training conditions and backpropagation: The error gradient starts from the MSE loss function mentioned above and backpropagates through the multi-view feature fusion network; Step 3.3, Training Termination Condition: Update parameters using the Adam optimizer, with an initial learning rate of 0.001; perform iterative training on the training set and monitor the mean joint position error on the validation set; terminate training when the MPJPE on the validation set no longer decreases for 10 consecutive epochs, or when the preset maximum number of iterations is reached, save the current optimal network weight model, input the test set into the trained multi-view feature fusion network for testing, and output the pose estimation image.

Images