Three-dimensional human body posture estimation method and system based on multi-view visual information fusion and storage medium

Abstract

The invention provides a three-dimensional human body posture estimation method and system based on multi-view visual information fusion and a storage medium, and the method comprises the steps: 1, designing the front half part of a model into Ender Layers with the same layer number as a Transform decoder at a multi-view feature fusion layer, carrying out the data enhancement of an input multi-view original image, and carrying out the reconstruction of the Ender Layers in the multi-view feature fusion layer; inputting the CNN Backbone with the shared weight to extract an initial feature map; 2, introducing a micro-reprojection optimization mechanism, deeply fusing the multi-view geometric consistency constraint into a model training process, and guiding the model to predict a three-dimensional attitude end to end; and step 3, constructing a dynamic projection compensation module. The method has the beneficial effects that the method is particularly suitable for capturing human body posture information in a multi-person interaction scene, the shielding problem and depth estimation ambiguity in a single view angle can be effectively overcome, and the robustness, precision and efficiency of three-dimensional human body posture estimation are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the intersection of computer vision and artificial intelligence technology, and in particular to a three-dimensional human posture estimation method, system and storage medium based on multi-view visual information fusion. Background Art

[0002] Three-dimensional human pose estimation is a key technology for understanding human motion and behavior, with significant applications in numerous scenarios, including motion capture, human-computer interaction, virtual / augmented reality, intelligent security, sports analysis, and medical rehabilitation. Existing methods primarily include monocular vision, inertial sensor methods, and multi-view fusion methods. However, each of these three estimation methods has its limitations. For example, monocular vision methods utilize only visual information from a single camera, directly regressing 3D joint coordinates or parametric human model parameters through deep learning models (such as CNNs and Transformers). While these methods offer low deployment costs, they are often limited by the inherent depth ambiguity and occlusion issues inherent in a single view, resulting in insufficient estimation accuracy and robustness. Furthermore, inertial sensor-based methods require wearing inertial sensors at key locations on the body. Sensor fusion is used to calculate pose, offering high accuracy and immunity to visual occlusion. However, these methods require wearing a device, resulting in a poor user experience, high cost, and potential for cumulative errors and magnetic interference.

[0003] In contrast, the multi-view fusion method uses multiple spatially distributed cameras to synchronously capture the target human body information. In theory, it can effectively overcome the depth ambiguity and occlusion problems of a single view. It is an important way and main research direction to improve the accuracy and robustness of posture estimation.

[0004] The current mainstream 3D human pose estimation methods based on multi-view fusion usually follow the following process: first, feature extraction is performed on each independent view image (such as 2D key point detection and semantic feature map generation); then different strategies are used to fuse cross-view information. Common methods include: (1) reconstruction-based methods: first, 2D human pose key points are detected independently in each single view, and then 3D pose is reconstructed through cross-view matching (such as triangulation, 3D graph structure model); (2) feature projection and aggregation based on 3D spatial representation (such as voxels, point clouds): the features of each view are mapped to a unified 3D space and then fused; (3) feature-level fusion based on deep learning models: the correlation between views is learned and the fused features are weighted; finally, the fused information is used to regress the 3D joint coordinates or human model parameters. However, reconstruction-based methods are highly dependent on the accuracy of intermediate two-dimensional detection results, cross-view matching is computationally complex, and is prone to failure in occluded or low-texture areas, resulting in accumulated reconstruction errors; methods based on 3D spatial representation require alignment of features from each perspective in a unified space, which has high computational and memory overhead, and insufficient alignment accuracy leads to poor fusion results; simple feature splicing or average pooling in feature-level fusion methods based on deep learning models may lose important information and may introduce a large amount of background noise during the fusion process. How to design an efficient fusion mechanism that can fully mine the complementary information between multiple perspectives, strengthen consistency constraints, and suppress noise and erroneous information at the same time remains a challenge.

[0005] The Transformer-based MvP (Multi-view Pose Transformer) model represents a more advanced approach. It eliminates intermediate tasks (such as 2D detection and voxel construction) and models multi-view multi-person 3D pose estimation as a direct regression problem. The core idea of ​​MvP is to represent human skeleton joints as learnable joint query embeddings. Through the Transformer decoder, these queries are gradually tuned to multi-view input features, directly predicting 3D joint coordinates. Key technical approaches include hierarchical joint query embeddings (sharing joint-level knowledge and introducing person-level queries), a projective attention mechanism (using geometric projections to guide local feature fusion), and a RayConv operation (encoding camera ray geometry). These methods significantly surpass previous approaches (such as VoxelPose) in accuracy and efficiency.

[0006] Although the MvP model has achieved remarkable results, it still has the following key issues in feature fusion, optimization mechanism, and dynamic adaptability: Multi-view feature fusion is not sufficient and efficient: MvP mainly relies on projective attention to sample features near the estimated 3D point projection for fusion. However, the first half of the model handles multi-view features in a relatively simple way (pooling sampling + splicing), failing to fully utilize the rich complementary information between multiple views. In the original solution, the feature map is only downsampled to The feature maps are simply spliced ​​together, which has low information utilization efficiency and lacks effective cross-view interaction, limiting the model's ability to represent complex postures and occluded scenes.

[0007] Lack of deeply integrated geometric constraint optimization: Although the training loss function of MvP includes a two-dimensional projection loss term (L1 loss for projected 2D joints), its essence is still a supervisory signal. It fails to deeply embed multi-view geometric consistency constraints (such as reprojection error) as a core, differentiable optimization mechanism into the model's learning process to actively guide the model to learn a 3D pose that is more in line with physical constraints.

[0008] Lack of robustness in dynamic scenes: MvP's projected attention relies heavily on the current estimated 3D point position for 2D projection positioning. In the event of temporary occlusion, intense motion leading to instantaneous deviations in pose estimation, or extreme changes in viewpoint, feature sampling based on projected points at these deviations can introduce noise or even erroneous information, hindering subsequent optimization and causing unstable or degraded pose estimation results. Existing models lack the ability to actively compensate for these dynamic interference factors. Summary of the Invention

[0009] In order to solve the problems in the prior art, especially the limitations of the MvP model, the present invention provides a 3D human pose estimation method based on multi-view visual information fusion, comprising: Step 1, iterative multi-view feature fusion: At the multi-view feature fusion level, the first half of the model is designed as an encoder layer with the same number of layers as the Transformer decoder. After data enhancement, the input multi-view original image is input into the shared weight convolutional neural network backbone to extract the initial feature map. ,in Indicates the number of viewpoints. Through an iterative feature fusion algorithm, multi-view visual feature information is fused from coarse to fine to generate a feature vector for use by the decoder. Step 2: Introducing a differentiable reprojection optimization mechanism: This mechanism deeply integrates multi-view geometric consistency constraints into the model training process, guiding the model to predict 3D poses end-to-end. Step 3: Build a dynamic projection compensation module: Build a dynamic projection compensation module to fine-tune camera parameters online, and perceive and compensate for projection point positioning errors caused by occlusion, motion blur, or estimation bias in real time.

[0010] As a further improvement of the present invention, in step 1, the following is further included: Step S1, coarse fusion stage: At the first encoder layer, the feature map of each view Perform standard adaptive average pooling downsampling to the target size get ,Then, the sampled feature maps under all view angles are fed into the feature fusion module. The core operation of the feature fusion module is the cross attention mechanism: Layer encoder layer, for each view , using its feature map as the query vector Q, and all other viewpoints The feature maps are concatenated as key vector K and value vector V, respectively, and the query vector is calculated by With key vector Attention weights and weighted fusion view counts , get the perspective The fusion features , ultimately, all perspectives Stitched together in the visual dimension to form the output of this layer ; Step S2, fine fusion stage: In the subsequent encoder layer, the layer, , introduce deformable pooling, for each perspective , using the previous level, that is, Layer, preliminary 3D joint positions predicted by Transformer decoder , that is, the initial position and camera parameters , projecting it onto the viewing angle On the two-dimensional image plane, the projection anchor point coordinates are obtained , drawing on the idea of ​​deformable convolution, we design a light quantum network and input the current perspective feature and the previous layer fusion features , predicting a set of spatial offsets and the corresponding attention weights , at the projection anchor point Around, according to the predicted offset sampling Features of each point , and use the predicted attention weights Perform weighted summation on these sampling features to obtain the viewing angle The pooled features , each perspective Input it into the feature fusion module again for fusion to obtain the fused feature , the fused features of all views After concatenation, it is used as the output of the first encoder layer .

[0011] As a further improvement of the present invention, in step 2, the following is further included: Step a1: For 3D joint positions predicted by the layer decoder , using camera parameters Combined with bilinear interpolation, it can be projected to each perspective differently On the 2D image plane, the predicted projection point coordinates are obtained ; Step a2: Calculate the coordinate distance loss and heat map distance loss, and the final layer obtains the differentiable reprojection loss is the weighted sum of these two losses over all joints, views, and samples, expressed as: ,in It is an adjustable parameter.

[0012] As a further improvement of the present invention, in step a2, the process of calculating the coordinate distance loss is as follows: Calculate for each joint in each view Upper predicted projection point and real 2D annotation points The L2 distance between them is expressed as ; The process of calculating the heatmap distance loss is as follows: Render the predicted projection points into a Gaussian heat map , and calculate the Gaussian heat map rendered with the real annotation points The L2 distance between them is expressed as .

[0013] As a further improvement of the present invention, the dynamic projection compensation module includes three lightweight MLP branches, which share the following inputs: the current joint query embedding , the initial predicted 3D position of the current layer , and the current perspective Local feature information near the projection point .

[0014] As a further improvement of the present invention, the three lightweight branches include: Branch 1: Output an offset vector , used to dynamically fine-tune camera parameters ,compensate for calibration errors or scene changes; Branch 2: Output a 2D offset vector , using the compensated camera parameters The 3D position of the current layer is initially predicted Projection to get the basic projection point , and then apply compensation to get the anchor point and the final prediction result ; Branch 3: Output a set of confidence scores , representing the current view The reliability or confidence of the aggregated data from the view When sampling features, the vector corresponding to the perspective is multiplied by the confidence score of the view, thereby reducing the contribution of low-confidence views, which is expressed as: ,in is the feature vector of each view after compensation, To compensate for the feature vectors of each previous view.

[0015] The present invention also discloses a three-dimensional human posture estimation system based on multi-view visual information fusion, comprising: a memory, a processor, and a computer program stored on the memory, wherein the computer program is configured to implement the steps of the method described in the present invention when called by the processor.

[0016] The present invention further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the method described in the present invention when called by a processor.

[0017] The beneficial effects of the present invention are: the present invention is particularly suitable for capturing human posture information in multi-person interaction scenarios, can effectively overcome the occlusion problem and depth estimation ambiguity under single-view, and significantly improve the robustness, accuracy and efficiency of three-dimensional human posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a technical flow chart of the three-dimensional human body posture estimation method of the present invention; Figure 2 This is a principle block diagram of the iterative multi-view feature fusion algorithm of the present invention; Figure 3 This is a block diagram of the principle of the micro-reprojection optimization mechanism of the present invention; Figure 4 This is a block diagram of the principle of the dynamic projection compensation module of the present invention; DETAILED DESCRIPTION

[0019] The present invention specifically relates to a method for achieving end-to-end human key point detection and posture reconstruction based on a multi-view feature fusion algorithm, a differentiable reprojection optimization mechanism, and a dynamic projection compensation module. The technical solution proposed in the present invention is particularly suitable for capturing human posture information in multi-person interaction scenarios. It can effectively overcome the occlusion problem and depth estimation ambiguity under a single view, and significantly improve the robustness, accuracy, and efficiency of three-dimensional human posture estimation.

[0020] Currently, the known application areas and methods of the technology / products of the present invention include: (1) Sports training and analysis: used to capture the technical details of athletes' movements, evaluate sports performance, and optimize training plans. It can quantify joint angles and movement trajectories to assist in scientific training. (2) Medical rehabilitation assessment and monitoring: used to quantitatively evaluate patients' motor dysfunction (such as gait analysis and joint range of motion measurement), monitor the rehabilitation training process, and objectively quantify the therapeutic effect. (3) Intelligent monitoring and security: deployed in public places for abnormal behavior recognition, fall risk detection, etc., to improve the intelligent level of safety management.

[0021] Potential future applications of this invention include: (1) Virtual Reality / Augmented Reality (VR / AR): Providing high-precision, low-latency gesture input for immersive avatar-driven, natural human-machine interaction, enhancing the realism of the user experience. (2) Film and Television Animation and Game Production: Serving low-cost, high-efficiency motion capture and 3D character animation generation. (3) Robotic Navigation and Human-Robot Collaboration: Providing service robots and collaborative robots with the ability to understand the gestures and intentions of surrounding humans, enabling safer and more natural interactions.

[0022] This paper proposes and integrates three technical improvements based on MvP, thereby achieving more robust and accurate 3D human pose estimation, while ensuring faster loss convergence and more stable accuracy improvement during training. The overall process is shown in the attached figure. Figure 1 shown.

[0023] The technical process of the present invention includes the following three key modules: (1) Iterative multi-view feature fusion algorithm (I-MVFF) (Appendix Figure 2 ) In terms of multi-view feature fusion, this paper abandons the original MvP approach of simply splicing the feature maps extracted by the backbone after adaptive average pooling and downsampling, and instead adopts a coarse-to-fine, iterative feature fusion strategy. Specifically, the first half of the model is designed to have the same number of EncoderLayers as the Transformer Decoder layers. After data enhancement, the input multi-view original image is input into the weight-shared CNN Backbone (convolutional neural network, such as ResNet50) to extract the initial feature map. ,in Represents the number of viewpoints. Through an iterative feature fusion algorithm, multi-view visual feature information is fused from coarse to fine to generate a feature vector for use by the decoder, as follows.

[0024] Coarse fusion stage: In the first Encoder Layer, the feature map of each view is Perform standard Adaptive Average Pooling downsampling to the target size get This not only reduces the vector length in subsequent attention operations to reduce the amount of computation, but also better preserves the feature information of each view. Subsequently, the downsampled feature maps of all views are sent to the Feature Fusion Module (FFM). The core operation of FFM is the Cross-Attention Mechanism: Encoder Layer, for each view , using its feature map as the query vector Q (Query), and all other viewpoints The feature maps of are concatenated (Concat) as the key vector K (Key) and the value vector V (Value). By calculating the query vector With key vector K Attention weights and weighted fusion , get the perspective The fusion features . Ultimately, all perspectives Stitched together in the visual dimension to form the output of this layer Its mathematical formula is: , in Indicates the view angle number, Indicates the Encoder layer number, T Represents a transpose operation.

[0025] Fine fusion stage: In the subsequent Encoder Layer ( layer, ), a more sophisticated deformable pooling is introduced. For each view , using the previous level ( Layer) TransformerDecoder predicts preliminary 3D joint positions (or initial position) and camera parameters , projecting it onto the viewing angle On the two-dimensional image plane, the projection anchor point coordinates are obtained Drawing on the idea of ​​deformable convolution, we design a lightweight quantum network (MLP) that inputs the current viewing angle features. and the previous layer fusion features , predicting a set of spatial offsets and the corresponding attention weights At the projection anchor point Around, according to the predicted offset sampling Features of each point , and use the predicted attention weights Perform weighted summation on these sampling features to obtain the viewing angle The pooled features . Every perspective Input it again into the feature fusion module (FFM) (structure is the same as the first layer) for fusion, and get , all perspectives After splicing, it is used as the output of the lth layer EncoderLayer I-MVFF overcomes the shortcomings of simple splicing by forcing information interaction between different viewpoints through a cross-attention mechanism. It also uses deformable pooling (centered around the reprojection anchor point) to achieve iterative optimization (from coarse to fine) of feature extraction and pose estimation, significantly improving the quality and efficiency of multi-view feature fusion. Furthermore, this simultaneous fusion and prediction approach maximizes the use of multi-view information, guiding the model to focus more on key point areas and helping it better predict 3D human joint coordinates.

[0026] (2) Differentiable Reprojection Optimization Mechanism (DROM) (Appendix Figure 3 ) To address the problem that the reprojection error in existing methods is not deeply integrated into end-to-end training, this paper introduces a differentiable reprojection optimization mechanism (DROM). This mechanism is applied to the output of each Transformer Decoder layer. Specifically, for the first 3D joint positions predicted by layer Decoder , using camera parameters Combined with bilinear interpolation, it can be projected to each perspective differently On the 2D image plane, the predicted projection point coordinates are obtained Then two differentiable losses are calculated: one is the coordinate distance loss (CoordLoss), which calculates the Upper predicted projection point and real 2D annotation points The L2 distance between them is expressed as ; The second is the heatmap distance loss (HeatmapLoss), which renders the predicted projection points into a Gaussian heatmap , and calculate the Gaussian heat map rendered with the real annotation points The L2 distance between them is expressed as The final layer differentiable reprojection loss is the weighted sum of these two losses over all joints, views, and samples, expressed as: ,in It is an adjustable parameter. The core function of DROM is: Acting directly on the predicted 3D coordinates , the gradient updates the model parameters through back propagation, thereby explicitly forcing the model to learn 3D poses that meet multi-view geometric consistency; at the same time, Layer prediction Will be used for The deformable pooling anchor point generation of the Encoder Layer forms a closed-loop optimization of feature fusion and pose estimation. More accurate This can guide the next layer to sample features at more precise locations, further improving the fusion effect. Therefore, DROM effectively reduces the ambiguity of 3D pose estimation by deeply integrating multi-view geometric constraints, improving prediction accuracy, especially for poses prone to ambiguity.

[0027] (3) Dynamic Projection Compensation Module (DPCM) (Appendix Figure 4 ) Current mainstream multi-view human pose estimation methods based on geometric measurements rely heavily on calibrated camera parameters during the interactive fusion of multi-view information. However, due to measurement noise, model simplification, operational factors, and algorithmic limitations, errors in camera calibration parameters are inevitable. The Multi-View-Performance (MvP) model uses camera parameters multiple times for vector calculations, such as camera ray acquisition and projection / backprojection conversion between 2D and 3D coordinates. This leads to inaccurate projection point positioning and error accumulation. Furthermore, the reliability of 3D coordinates projected to different viewpoints varies, resulting in varying confidence levels in feature maps derived from different viewpoints. To address the lack of accuracy caused by parameter errors and computational bias, the present invention introduces a dynamic projection compensation module.

[0028] Specifically, DPCM consists of three lightweight MLP branches that share the input (current joint query embedding , the initial predicted 3D position of the current layer , and the current perspective Local feature information near the projection point ), the lightweight MLP branch refers to a multilayer perceptron subnetwork structure with high computational efficiency and small number of parameters.

[0029] Branch 1 (MLP1) outputs an offset vector , used to dynamically fine-tune camera parameters , compensating for calibration errors or scene changes.

[0030] Branch 2 (MLP2) outputs a 2D offset vector , using the compensated camera parameters Will Projection to get the basic projection point , and then apply compensation to obtain more robust anchor points and final prediction results .

[0031] Branch 3 (MLP3) outputs a set of confidence scores , representing the current view The reliability or confidence of the aggregated data from the view When sampling features, the vector corresponding to the perspective is multiplied by the confidence score of the view, thereby reducing the contribution of low-confidence views, which is expressed as: .

[0032] DPCM effectively alleviates the problem of projection point positioning errors caused by calibration parameter errors, temporary occlusion, motion blur or instantaneous deviations in attitude estimation by dynamically adjusting the projection reference point (camera parameter fine-tuning, position compensation) and adaptive fusion weights (confidence weighting), significantly enhancing the robustness of the model in dynamic and complex environments and the continuity of attitude estimation.

[0033] The present invention also discloses a three-dimensional human posture estimation system based on multi-view visual information fusion, comprising: a memory, a processor, and a computer program stored on the memory, wherein the computer program is configured to implement the steps of the method described in the present invention when called by the processor.

[0034] The present invention further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the method described in the present invention when called by a processor.

[0035] The present invention proposes an improved 3D human pose estimation method, system and storage medium based on multi-view visual information fusion, aiming to: (1) Design an iterative multi-view feature fusion algorithm from coarse to fine, significantly improve the utilization efficiency and fusion quality of multi-view feature information, and enhance the model's ability to represent complex postures and occlusions; (2) A differentiable reprojection optimization mechanism is introduced to deeply integrate multi-view geometric consistency constraints into the model training process, guiding the model to predict 3D poses that are more consistent with physical laws in an end-to-end manner; (3) Build a dynamic projection compensation module to fine-tune camera parameters online, perceive and compensate in real time the projection point positioning error caused by occlusion, motion blur or estimation bias, and significantly improve the robustness and posture continuity of the model in dynamic and complex scenes.

[0036] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A 3D human pose estimation method based on multi-view visual information fusion, characterized in that: include: Step 1, iterative multi-view feature fusion: At the multi-view feature fusion level, the first half of the model is designed as an encoder layer with the same number of layers as the Transformer decoder. After data enhancement, the input multi-view original image is input into the shared weight convolutional neural network backbone to extract the initial feature map. ,in Indicates the number of viewpoints. Through an iterative feature fusion algorithm, multi-view visual feature information is fused from coarse to fine to generate a feature vector for use by the decoder. Step 2: Introducing a differentiable reprojection optimization mechanism: This mechanism deeply integrates multi-view geometric consistency constraints into the model training process, guiding the model to predict 3D poses end-to-end. Step 3: Build a dynamic projection compensation module: Build a dynamic projection compensation module to fine-tune camera parameters online, and perceive and compensate for projection point positioning errors caused by occlusion, motion blur, or estimation bias in real time.

2. The 3D human body posture estimation method according to claim 1, wherein: In step 1, the method further includes: Step S1, coarse fusion stage: At the first encoder layer, the feature map of each view Perform standard adaptive average pooling downsampling to the target size get ,Then, the sampled feature maps under all view angles are fed into the feature fusion module. The core operation of the feature fusion module is the cross attention mechanism: Layer encoder layer, for each view , using its feature map as the query vector Q, and all other viewpoints The feature maps are concatenated as key vector K and value vector V, respectively, and the query vector is calculated by With key vector Attention weights and weighted fusion view counts , get the perspective The fusion features , ultimately, all perspectives Stitched together in the visual dimension to form the output of this layer ; Step S2, fine fusion stage: In the subsequent encoder layer, the layer, , introduce deformable pooling, for each perspective Using the previous level, that is, Layer, preliminary 3D joint positions predicted by Transformer decoder , that is, the initial position and camera parameters , projecting it onto the viewing angle On the two-dimensional image plane, the projection anchor point coordinates are obtained , design a light quantum network, input the current perspective feature and the previous layer fusion features , predicting a set of spatial offsets and the corresponding attention weights , at the projection anchor point Around, according to the predicted offset sampling Features of each point , and use the predicted attention weights Perform weighted summation on these sampling features to obtain the viewing angle The pooled features , each perspective Input it into the feature fusion module again for fusion to obtain the fused feature Fusion features from all perspectives After concatenation, it is used as the output of the first encoder layer .

3. The 3D human body posture estimation method according to claim 2, wherein: In step 2, the method further includes: Step a1: For 3D joint positions predicted by the layer decoder , using camera parameters Combined with bilinear interpolation, it can be projected to each perspective differently On the 2D image plane, the predicted projection point coordinates are obtained ; Step a2: Calculate the coordinate distance loss and heat map distance loss, and the final layer obtains the differentiable reprojection loss is the weighted sum of these two losses over all joints, views, and samples, expressed as: ,in It is an adjustable parameter.

4. The 3D human body posture estimation method according to claim 3, wherein: In step a2, the process of calculating the coordinate distance loss is as follows: Calculate for each joint in each view Upper predicted projection point and real 2D annotation points The L2 distance between them is expressed as ; The process of calculating the heatmap distance loss is as follows: Render the predicted projection points into a Gaussian heat map , and calculate the Gaussian heat map rendered with the real annotation points The L2 distance between them is expressed as .

5. The 3D human body posture estimation method according to claim 1, wherein: The dynamic projection compensation module consists of three lightweight MLP branches, which share the following inputs: the current joint query embedding , the initial predicted 3D position of the current layer , and the current perspective Local feature information near the projection point .

6. The 3D human body posture estimation method according to claim 5, characterized in that: The three lightweight branches include: Branch 1: Output an offset vector , used to dynamically fine-tune camera parameters ,compensate for calibration errors or scene changes; Branch 2: Output a 2D offset vector , using the compensated camera parameters The 3D position of the current layer is initially predicted Projection to get the basic projection point , and then apply compensation to get the anchor point and final prediction result ; Branch 3: Output a set of confidence scores , representing the current view The reliability or confidence of the aggregated data from the view When sampling features, the vector corresponding to the view angle is multiplied by the confidence score of the view, which is expressed as: ,in is the feature vector of each view after compensation, To compensate for the feature vectors of each previous view.

7. A 3D human pose estimation system based on multi-view visual information fusion, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the method according to any one of claims 1 to 6 when called by the processor.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of any one of claims 1 to 6 when called by a processor.

Images