A method, system, and storage medium for 3D human pose estimation based on multi-view visual information fusion.

Abstract

This invention provides a method, system, and storage medium for 3D human pose estimation based on multi-view visual information fusion. The method includes: Step 1: At the multi-view feature fusion level, the first half of the model is designed as Encoder Layers with the same number of layers as the Transformer decoder. The input multi-view original images are data augmented and then input into a CNN Backbone with shared weights to extract initial feature maps; Step 2: A differentiable reprojection optimization mechanism is introduced to deeply integrate multi-view geometric consistency constraints into the model training process, guiding the model to predict 3D pose end-to-end; Step 3: A dynamic projection compensation module is constructed. The beneficial effects of this invention are: it is particularly suitable for capturing human pose information in multi-person interactive scenarios, effectively overcoming the occlusion problem and depth estimation ambiguity under single-view conditions, and significantly improving the robustness, accuracy, and efficiency of 3D human pose estimation.

Description

Technical Field

[0001] This invention relates to the intersection of computer vision and artificial intelligence technologies, and in particular to a method, system and storage medium for three-dimensional human pose estimation based on multi-view visual information fusion. Background Technology

[0002] 3D human pose estimation is a key technology for understanding human movement and behavior, with significant application value in numerous scenarios such as motion capture, human-computer interaction, virtual / augmented reality, intelligent security, sports analytics, and medical rehabilitation. Existing methods mainly include monocular vision methods, inertial sensor methods, and multi-view fusion methods. However, each of these three estimation methods has its limitations. For example, monocular vision methods utilize only the visual information from a single camera, directly regressing 3D joint coordinates or parameterized human model parameters through deep learning models (such as CNNs and Transformers). These methods have low deployment costs, but are limited by inherent depth ambiguity and occlusion issues in single-view methods, often resulting in insufficient estimation accuracy and robustness. Furthermore, inertial sensor-based methods require inertial sensors to be worn on key parts of the human body, calculating pose through sensor fusion. While offering high accuracy and being unaffected by visual occlusion, these methods require wearable devices, leading to a poor user experience, high cost, and issues with accumulated errors and magnetic interference.

[0003] In contrast, multi-view fusion method uses multiple spatially distributed cameras to simultaneously capture target human information, which can theoretically effectively overcome the depth ambiguity and occlusion problems of single view. It is an important approach and main research direction for improving the accuracy and robustness of pose estimation.

[0004] The current mainstream three-dimensional human pose estimation methods based on multi-view fusion usually follow the following process: First, feature extraction is performed on images from each independent view (such as 2D key point detection and semantic feature map generation); then, different strategies are used to fuse cross-view information. Common methods include: (1) Reconstruction-based methods: First, two-dimensional human pose key points are detected independently in each single view, and then three-dimensional pose is reconstructed through cross-view matching (such as triangulation and three-dimensional graph structure model); (2) Feature projection and aggregation based on 3D spatial representation (such as voxels and point clouds): The features of each view are mapped to a unified 3D space and then fused; (3) Feature-level fusion based on deep learning models: The correlation between views is learned and the features are weighted and fused; finally, the fused information is used to regress the coordinates of 3D key points or the parameters of the human model. However, reconstruction-based methods heavily rely on the accuracy of intermediate 2D detection results, cross-view matching is computationally complex, and they are prone to failure in occluded or low-texture areas, leading to the accumulation of reconstruction errors. Methods based on 3D spatial representation require alignment of features from different viewpoints in a unified space, resulting in significant computational and memory overhead, and insufficient alignment accuracy can lead to poor fusion results. In feature-level fusion methods based on deep learning models, simple feature concatenation or average pooling may lose important information, and a large amount of background noise may be introduced during the fusion process. Designing an efficient fusion mechanism that can fully exploit complementary information between multiple viewpoints, strengthen consistency constraints, and suppress noise and erroneous information remains a challenge.

[0005] The Transformer-based Multi-view Pose Transformer (MvP) model represents a more advanced approach. It abandons intermediate tasks (such as 2D detection and voxel construction), modeling multi-view, multi-person 3D pose estimation as a direct regression problem. The core idea of ​​MvP is to represent human skeletal joints as learnable joint query embeddings. Through a Transformer decoder, these queries progressively focus on multi-view input features, directly predicting 3D joint coordinates. Key techniques include: hierarchical joint query embedding (sharing joint-level knowledge and introducing person-level queries), projective attention (using geometric projection to guide local feature fusion), and RayConv operations (encoding camera ray geometry information), significantly surpassing previous methods (such as VoxelPose) in both accuracy and efficiency.

[0006] Despite the significant achievements of the MvP model, it still faces the following key challenges in feature fusion, optimization mechanisms, and dynamic adaptability:

[0007] Multi-view feature fusion is insufficient and inefficient: MvP mainly relies on projective attention to sample features near the estimated 3D point projection for fusion. However, the first half of its model handles multi-view features relatively simply (pooling sampling + concatenation), failing to fully utilize the rich complementary information between multiple views. In the original scheme, the feature map is only downsampled to... The feature maps are simply stitched together, resulting in low information utilization efficiency and a lack of effective cross-view interaction, which limits the model's ability to represent complex poses and occluded scenes.

[0008] Lack of deep integration of geometric constraint optimization: Although the training loss function of MvP includes a two-dimensional projection loss term (L1 loss for projected 2D joints), it is essentially still a supervisory signal. It fails to deeply embed multi-view geometric consistency constraints (such as reprojection error) as a core, differentiable optimization mechanism into the model's learning process to actively guide the model to learn a 3D pose that is more in line with physical constraints.

[0009] Insufficient robustness to dynamic scenes: MvP's projection attention heavily relies on the currently estimated 3D point positions for 2D projection localization. In cases of temporary occlusion, sudden motion causing instantaneous deviations in pose estimation, or extreme changes in viewpoint, feature sampling based on projected points at the deviation locations can introduce noise or even erroneous information, hindering subsequent optimization and leading to unstable or decreased pose estimation results. Existing models lack the ability to proactively compensate for such dynamic disturbances. Summary of the Invention

[0010] To address the problems in existing technologies, particularly the limitations of the MvP model, this invention provides a 3D human pose estimation method based on multi-view visual information fusion, comprising:

[0011] Step 1, Iterative Multi-View Feature Fusion: At the multi-view feature fusion level, the first half of the model is designed as an encoder layer with the same number of layers as the Transformer decoder. After data augmentation, the input multi-view original images are input into the backbone of a convolutional neural network with shared weights to extract the initial feature maps. ,in The number of viewpoints is represented by an iterative feature fusion algorithm, which fuses multi-view visual feature information from coarse to fine to generate feature vectors for the decoder to use.

[0012] Step 2, Introduction of Differentiable Reprojection Optimization Mechanism: The differentiable reprojection optimization mechanism is introduced to deeply integrate multi-view geometric consistency constraints into the model training process, guiding the model to predict 3D pose end-to-end.

[0013] Step 3, Construct a dynamic projection compensation module: Construct a dynamic projection compensation module to fine-tune camera parameters online and perceive and compensate for projection point positioning errors caused by occlusion, motion blur, or estimation deviation in real time.

[0014] As a further improvement of the present invention, step 1 further includes:

[0015] Step S1, coarse fusion stage: In the first encoder layer, the feature maps of each view are processed... Perform standard adaptive average pooling downsampling to the target size get Subsequently, the sampled feature maps from all perspectives are fed into the feature fusion module. The core operation of the feature fusion module is the cross-attention mechanism: in the first... Layer encoder layer, for each viewpoint Using its feature map as the query vector Q, while taking all other perspectives The feature maps are concatenated and used as the key vector K and value vector V, respectively. The query vector is then calculated. With key vector Attention weights and weighted fusion of viewpoints To obtain this perspective Features after fusion Ultimately, all perspectives These elements are pieced together in a visual dimension to form the output of this layer. ;

[0016] Step S2, Fine fusion stage: In the subsequent encoder layer, the first... layer, Deformable pooling is introduced for each viewpoint. Utilizing the previous level, that is, the first Layer, preliminary 3D joint positions predicted by the Transformer decoder That is, the initial position and camera parameters Project it onto the viewpoint On the two-dimensional image plane, obtain the coordinates of the projection anchor point. Drawing inspiration from deformable convolution, a lightweight quantum network is designed, taking the current viewpoint features as input. Integration features with the previous layer Predict a set of spatial offsets and the corresponding attention weights At the projection anchor point Around, based on the predicted offset sampling Features of each point and using the predicted attention weights These sampled features are weighted and summed to obtain the viewpoint. Pooling features Each perspective The input is then fed back into the feature fusion module for fusion to obtain the fused features. Features after fusion of all perspectives The concatenation is used as the output of the first encoder layer. .

[0017] As a further improvement of the present invention, step 2 further includes:

[0018] Step a1: For the first 3D joint positions predicted by layer decoder Using camera parameters Combined with bilinear interpolation, it can be projected differentially onto each viewpoint. On the 2D image plane, obtain the coordinates of the predicted projection point. ;

[0019] Step a2: Calculate the coordinate distance loss and heatmap distance loss, and finally obtain the differentiable reprojection loss for the layer. The weighted sum of these two losses over all joints, views, and samples is expressed as: ,in This is an adjustable parameter.

[0020] As a further improvement of the present invention, the process of calculating the coordinate distance loss in step a2 is as follows:

[0021] Calculate each joint in each view Upper Predicted Projection Point With real 2D annotation points The L2 distance between them is expressed by the mathematical formula as follows: ;

[0022] The process of calculating the distance loss in the heatmap is as follows:

[0023] Render the predicted projection points as a Gaussian heatmap And calculate its Gaussian heatmap rendered with the actual annotation points. The L2 distance between them is expressed by the mathematical formula as follows: .

[0024] As a further improvement of the present invention, the dynamic projection compensation module includes three lightweight MLP branches, all of which share the following input: current joint query embedding. 3D position of the current layer (preliminary prediction) and the current perspective Local feature information near the projection point .

[0025] As a further improvement to the present invention, the three lightweight branches include:

[0026] Branch 1: Output an offset vector Used for dynamically fine-tuning camera parameters To compensate for calibration errors or changes in the scene;

[0027] Branch 2: Output a 2D offset vector Using compensated camera parameters The preliminary 3D position of the current layer Projection yields the basic projection point Then, compensation is applied to obtain the anchor point and the final prediction result. ;

[0028] Branch 3: Output a set of confidence scores Represents the current view The reliability or confidence level of the data is aggregated from the view within the projection attention. When sampling features, the vector corresponding to the viewpoint is multiplied by the confidence score of that view, thereby reducing the contribution of low-confidence views, expressed as: ,in To compensate for the feature vectors of each viewpoint, To compensate for the feature vectors of each viewpoint.

[0029] The present invention also discloses a three-dimensional human pose estimation system based on multi-view visual information fusion, comprising: a memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the method described in the present invention when called by the processor.

[0030] The present invention also discloses a computer-readable storage medium storing a computer program configured to implement the steps of the method described in the present invention when invoked by a processor.

[0031] The beneficial effects of this invention are: this invention is particularly suitable for capturing human pose information in multi-person interactive scenarios, and can effectively overcome the occlusion problem and depth estimation ambiguity under single viewpoint, and significantly improve the robustness, accuracy and efficiency of three-dimensional human pose estimation. Attached Figure Description

[0032] Figure 1 This is a flowchart of the three-dimensional human pose estimation method of the present invention;

[0033] Figure 2 This is a block diagram illustrating the principle of the iterative multi-view feature fusion algorithm of this invention;

[0034] Figure 3 This is a block diagram illustrating the principle of the differential reprojection optimization mechanism of this invention;

[0035] Figure 4 This is a block diagram illustrating the principle of the dynamic projection compensation module of this invention; Detailed Implementation

[0036] This invention specifically relates to a method for end-to-end human keypoint detection and pose reconstruction based on a multi-view feature fusion algorithm, a differentiable reprojection optimization mechanism, and a dynamic projection compensation module. The technical solution proposed in this invention is particularly suitable for capturing human pose information in multi-person interactive scenarios, and can effectively overcome the occlusion problem and depth estimation ambiguity under single viewpoint, significantly improving the robustness, accuracy, and efficiency of 3D human pose estimation.

[0037] Currently, the known application areas and application methods of the technology / product of this invention include: (1) Sports training and analysis: applied to capturing the details of athletes' movement techniques, evaluating sports performance and optimizing training programs, quantifying joint angles and movement trajectories, and assisting in scientific training. (2) Medical rehabilitation assessment and monitoring: applied to the quantitative assessment of patients' motor function disorders (such as gait analysis and joint range of motion measurement), monitoring of rehabilitation training processes and objective quantification of therapeutic effects. (3) Intelligent monitoring and security: deployed in public places for abnormal behavior recognition, fall risk detection, etc., to improve the level of intelligent safety management.

[0038] Potential future applications of this invention include: (1) Virtual Reality / Augmented Reality (VR / AR): Providing high-precision, low-latency posture input for immersive virtual avatar-driven, natural human-computer interaction, enhancing the realism of the user experience. (2) Film and Animation and Game Production: Serving low-cost, high-efficiency motion capture and 3D character animation generation. (3) Robot Navigation and Human-Computer Collaboration: Providing service robots and collaborative robots with the ability to understand the postures and intentions of people around them, enabling safer and more natural interaction.

[0039] This invention proposes and integrates three technical improvements based on MvP, thereby achieving more robust and accurate 3D human pose estimation, while ensuring faster loss convergence and more stable accuracy improvement during training. The overall process is attached. Figure 1 As shown.

[0040] The technical process of this invention includes the following three key modules:

[0041] (1) Iterative Multi-View Feature Fusion Algorithm (I-MVFF) (Appendix) Figure 2 )

[0042] At the multi-view feature fusion level, this invention abandons the simple concatenation approach in the original MVP, which only performs adaptive average pooling downsampling on the feature maps extracted from the backbone. Instead, it adopts a coarse-to-fine, iterative feature fusion strategy. Specifically, the first half of the model is designed with the same number of Encoder Layers as the Transformer Decoder. After data augmentation, the input multi-view original images are input into a CNN Backbone (convolutional neural backbone network, such as ResNet50) with shared weights to extract initial feature maps. ,in The number of viewpoints is represented by an iterative feature fusion algorithm, which fuses multi-view visual feature information from coarse to fine to generate feature vectors for the decoder to use, as detailed below.

[0043] Coarse fusion stage: In the first encoder layer, the feature maps of each view are processed. Perform standard adaptive average pooling downsampling to the target size get This approach reduces the vector length in subsequent attention operations, thus lowering computational cost, while also preserving feature information from each perspective. Subsequently, the downsampled feature maps from all perspectives are fed into the Feature Fusion Module (FFM). The core operation of FFM is the Cross-Attention mechanism: in the... Encoder Layer, for each viewpoint It uses its feature map as the query vector Q (Query), while taking all other perspectives The feature maps are concatenated and used as the key vector K and value vector V, respectively. The query vector is then calculated. With key vector K Attention weights and weighted fusion To obtain this perspective Features after fusion Ultimately, all perspectives These elements are pieced together in a visual dimension to form the output of this layer. Its mathematical formula is expressed as:

[0044] ,

[0045] in Indicates the viewpoint number. Indicates the encoder layer number. T This indicates the transpose operation.

[0046] Fine-grained blending stage: In the subsequent Encoder Layer (the... layer, This introduces a more refined form of deformable pooling. For each viewpoint... Utilizing the previous level (the first (Layer) TransformerDecoder predicts the initial 3D joint positions (or initial position) and camera parameters Project it onto the viewpoint On the two-dimensional image plane, obtain the coordinates of the projection anchor point. Drawing inspiration from deformable convolution, a lightweight quantum network (MLP) is designed, taking the current viewpoint features as input. Integration features with the previous layer Predict a set of spatial offsets and the corresponding attention weights At the projection anchor point Around, based on the predicted offset sampling Features of each point and using the predicted attention weights These sampled features are weighted and summed to obtain the viewpoint. Pooling features Each perspective The input is then fed back into the Feature Fusion Module (FFM) (with the same structure as the first layer) for fusion, resulting in... All perspectives The concatenated result serves as the output of the l-th EncoderLayer. I-MVFF overcomes the shortcomings of simple stitching by forcing information interaction between different perspectives through a cross-attention mechanism; it achieves iterative optimization (from coarse to fine) of feature extraction and pose estimation by using deformable pooling (centered on the reprojection anchor point), which significantly improves the quality and efficiency of multi-view feature fusion; at the same time, this method of fusing and predicting at the same time can make the most of multi-view information, guide the model to pay more attention to key point regions, and help the model better predict the 3D human joint coordinates.

[0047] (2) Differentiable reprojection optimization mechanism (DROM) (Appendix) Figure 3 )

[0048] To address the issue of reprojection errors not being deeply integrated into end-to-end training in existing methods, this invention introduces a differentiable reprojection optimization mechanism (DROM). This mechanism is applied to the output of each Transformer Decoder layer. Specifically, for the... 3D joint positions predicted by layer decoder Using camera parameters Combined with bilinear interpolation, it can be projected differentially onto each viewpoint. On the 2D image plane, obtain the coordinates of the predicted projection point. Then, two differentiable losses are calculated: one is the coordinate distance loss (CoordLoss), calculated for each joint in each view. Upper Predicted Projection Point With real 2D annotation points The L2 distance between them is expressed by the mathematical formula as follows: The second is heatmap loss, which renders the predicted projection points as a Gaussian heatmap. And calculate its Gaussian heatmap rendered with the actual annotation points. The L2 distance between them is expressed by the mathematical formula as follows: The final layer-differentiable reprojection loss It is a weighted sum of these two losses across all joints, views, and samples, expressed as: ,in These are adjustable parameters. The core function of DROM lies in: its loss Directly applied to the predicted 3D coordinates The gradient updates the model parameters through backpropagation, thereby explicitly forcing the model to learn a 3D pose that satisfies multi-view geometric consistency; simultaneously, the gradient... Layer prediction Will be used for the The deformable pooling anchor point generation of the Encoder Layer forms a closed-loop optimization for feature fusion and pose estimation. This results in more accurate... This allows the next layer to sample features at more precise locations, further improving the fusion effect. Therefore, by deeply integrating multi-view geometric constraints, DROM effectively reduces the ambiguity of 3D pose estimation and improves prediction accuracy, especially for poses that are prone to ambiguity.

[0049] (3) Dynamic Projection Compensation Module (DPCM) (attached) Figure 4 )

[0050] Current mainstream multi-view human pose estimation methods based on geometric measurements heavily rely on calibrated camera parameters during multi-view information interaction and fusion. However, due to measurement noise, model simplification, operational factors, and algorithm limitations, errors in camera calibration parameters are unavoidable. The MvP model repeatedly uses camera parameters for vector calculations, such as camera ray acquisition and projection / backprojection transformations between 2D and 3D coordinates, leading to inaccurate projection point positioning and error accumulation. Furthermore, the reliability of information contained in 3D coordinates projected onto different viewpoints varies, resulting in different confidence levels in feature maps obtained from different viewpoints. To address the issue of insufficient accuracy caused by parameter errors and computational biases, this invention introduces a dynamic projection compensation module.

[0051] Specifically, DPCM comprises three lightweight MLP branches that share the input (current joint query embedding). 3D position of the current layer (preliminary prediction) and the current perspective Local feature information near the projection point The lightweight MLP branch refers to a multilayer perceptron subnetwork structure that is computationally efficient and has a small number of parameters.

[0052] Branch 1 (MLP1) outputs an offset vector. Used for dynamically fine-tuning camera parameters This compensates for calibration errors or changes in the scene.

[0053] Branch 2 (MLP2) outputs a 2D offset vector. Using compensated camera parameters Will Projection yields the basic projection point Then, compensation is applied to obtain more robust anchor points and final prediction results. .

[0054] Branch 3 (MLP3) outputs a set of confidence scores. Represents the current view The reliability or confidence level of the data is aggregated from the view within the projection attention. When sampling features, the vector corresponding to the viewpoint is multiplied by the confidence score of that view, thereby reducing the contribution of low-confidence views, expressed as: .

[0055] DPCM effectively mitigates the problem of incorrect projection point localization caused by calibration parameter errors, temporary occlusion, motion blur, or instantaneous deviations in pose estimation by dynamically adjusting the projection reference point (camera parameter fine-tuning and position compensation) and adaptive fusion weights (confidence weighting). This significantly enhances the robustness of the model in dynamic and complex environments and the continuity of pose estimation.

[0056] The present invention also discloses a three-dimensional human pose estimation system based on multi-view visual information fusion, comprising: a memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the method described in the present invention when called by the processor.

[0057] The present invention also discloses a computer-readable storage medium storing a computer program configured to implement the steps of the method described in the present invention when invoked by a processor.

[0058] This invention proposes an improved method, system, and storage medium for 3D human pose estimation based on multi-view visual information fusion, aiming to:

[0059] (1) Design an iterative multi-view feature fusion algorithm from coarse to fine to significantly improve the utilization efficiency and fusion quality of multi-view feature information and enhance the model’s ability to represent complex poses and occlusions.

[0060] (2) A differentiable reprojection optimization mechanism is introduced to deeply integrate multi-view geometric consistency constraints into the model training process, guiding the model to predict three-dimensional poses that are more in line with physical laws from end to end.

[0061] (3) Construct a dynamic projection compensation module to fine-tune camera parameters online, perceive and compensate for projection point positioning errors caused by occlusion, motion blur or estimation deviation in real time, and greatly improve the robustness and attitude continuity of the model in dynamic and complex scenes.

[0062] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A three-dimensional human pose estimation method based on multi-view visual information fusion, characterized in that, include: Step 1, Iterative Multi-View Feature Fusion: At the multi-view feature fusion level, the first half of the model is designed as an encoder layer with the same number of layers as the Transformer decoder. After data augmentation, the input multi-view original images are input into the backbone of a convolutional neural network with shared weights to extract the initial feature maps. ,in The number of viewpoints is represented by an iterative feature fusion algorithm, which fuses multi-view visual feature information from coarse to fine to generate feature vectors for the decoder to use. Step 2, Introduction of Differentiable Reprojection Optimization Mechanism: The differentiable reprojection optimization mechanism is introduced to deeply integrate multi-view geometric consistency constraints into the model training process, guiding the model to predict 3D pose end-to-end. Step 3, Construct a dynamic projection compensation module: Construct a dynamic projection compensation module to fine-tune camera parameters online and perceive and compensate for projection point positioning errors caused by occlusion, motion blur or estimation deviation in real time; The dynamic projection compensation module comprises three lightweight MLP branches, all of which share the following input: current joint query embedding. 3D position of the current layer (preliminary prediction) and the current perspective Local feature information near the projection point .

2. The three-dimensional human pose estimation method according to claim 1, characterized in that, Step 1 also includes: Step S1, coarse fusion stage: In the first encoder layer, the feature maps of each view are processed... Perform standard adaptive average pooling downsampling to the target size get Subsequently, the sampled feature maps from all perspectives are fed into the feature fusion module. The core operation of the feature fusion module is the cross-attention mechanism: in the first... Layer encoder layer, for each viewpoint Using its feature map as the query vector Q, while taking all other perspectives The feature maps are concatenated and used as the key vector K and value vector V, respectively. The query vector is then calculated. With key vector Attention weights and weighted fusion of viewpoints To obtain this perspective Features after fusion Ultimately, all perspectives These elements are pieced together in a visual dimension to form the output of this layer. ; Step S2, Fine fusion stage: In the subsequent encoder layer, the first... layer, Deformable pooling is introduced for each viewpoint. Utilizing the previous level, i.e., the Layer, preliminary 3D joint positions predicted by the Transformer decoder That is, the initial position and camera parameters Project it onto the viewpoint On the two-dimensional image plane, obtain the coordinates of the projection anchor point. Design a lightweight quantum network, taking the current viewpoint features as input. Integration features with the previous layer Predict a set of spatial offsets and the corresponding attention weights At the projection anchor point Around, based on the predicted offset sampling Features of each point and using the predicted attention weights These sampled features are weighted and summed to obtain the viewpoint. Pooling features Each perspective The input is then fed back into the feature fusion module for fusion to obtain the fused features. Features after fusion of all perspectives The concatenation is used as the output of the first encoder layer. .

3. The three-dimensional human pose estimation method according to claim 2, characterized in that, Step 2 also includes: Step a1: For the first 3D joint positions predicted by layer decoder Using camera parameters Combined with bilinear interpolation, it can be projected differentially onto each viewpoint. On the 2D image plane, obtain the coordinates of the predicted projection point. ; Step a2: Calculate the coordinate distance loss and heatmap distance loss, and finally obtain the differentiable reprojection loss for the layer. The weighted sum of these two losses over all joints, views, and samples is expressed as: , Represent each joint in each view Upper predicted projection point coordinates With real 2D annotation points L2 distance between them This indicates that the predicted projection points are rendered as a Gaussian heatmap. Gaussian heatmap rendered with real-world labeled points The L2 distance between them.

4. The three-dimensional human pose estimation method according to claim 3, characterized in that, In step a2, the process of calculating the coordinate distance loss is as follows: Calculate each joint in each view Upper predicted projection point coordinates With real 2D annotation points The L2 distance between them is expressed by the mathematical formula as follows: ; The process of calculating the distance loss in the heatmap is as follows: Render the predicted projection points as a Gaussian heatmap And calculate its Gaussian heatmap rendered with respect to the actual labeled points. The L2 distance between them is expressed by the mathematical formula as follows: .

5. The three-dimensional human pose estimation method according to claim 1, characterized in that, The three lightweight branches include: Branch 1: Output an offset vector Used for dynamically fine-tuning camera parameters To compensate for calibration errors or changes in the scene, among which Indicates camera parameters; Branch 2: Output a 2D offset vector Using compensated camera parameters The preliminary 3D position of the current layer Projection yields the basic projection point Then, compensation is applied to obtain the anchor point and the final prediction result. ; Branch 3: Output a set of confidence scores Represents the current view The reliability or confidence level of the data is aggregated from the view within the projection attention. When sampling features, the vector corresponding to the viewpoint is multiplied by the confidence score of that view, expressed as: ,in To compensate for the feature vectors of each viewpoint, To compensate for the feature vectors of each viewpoint.

6. A three-dimensional human pose estimation system based on multi-view visual information fusion, characterized in that, include: A memory, a processor, and a computer program stored on the memory, the computer program being configured to implement the steps of the method of any one of claims 1-5 when invoked by the processor.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to implement the steps of the method according to any one of claims 1-5 when invoked by a processor.

Images