Three-dimensional human pose estimation method and device based on multi-modal and occlusion perception

Abstract

This invention discloses a method and apparatus for 3D human pose estimation based on multimodal and occlusion perception, relating to the field of computer vision. The method includes: S1, acquiring a multimodal video sequence including RGB, depth, and infrared sequences; S2, preprocessing to obtain a data-aligned multimodal video sequence; S3, constructing a 3D human pose estimation model including a spatiotemporal feature extraction network, an occlusion detection network, a temporal coordinate classification module, a Z-axis depth prediction module, and a 3D coordinate stitching module; S4, training the 3D human pose estimation model using the data-aligned multimodal video sequence; and S5, using the trained model to perform 3D human pose estimation. This invention effectively handles the 3D human pose estimation problem in complex occlusion scenarios by fusing RGB, depth, and infrared three-modality data, using an occlusion-aware temporal coordinate classification method to predict X and Y axis coordinates, and employing Z-axis depth prediction based on binning and multimodal multi-resolution probabilistic fusion.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and particularly to a three-dimensional human pose estimation method and device based on multi-modal and occlusion perception. Background Art

[0002] As one of the core technologies in the field of computer vision, three-dimensional human pose estimation aims to accurately locate the three-dimensional spatial coordinates of human key points from visual inputs. This technology has made remarkable progress in the past decade, evolving from traditional machine learning-based methods to deep learning-based end-to-end solutions. However, existing technologies still face many challenges in complex occlusion scenarios.

[0003] The technological development process of three-dimensional human pose estimation includes: the traditional method stage, early methods based on handcrafted features and geometric models, which rely on complex human prior knowledge and have limited robustness in real scenarios; the two-dimensional to three-dimensional mapping stage, where three-dimensional coordinates are deduced through two-dimensional key point detection combined with geometric constraints, but it is difficult to obtain depth information; the end-to-end deep learning stage, directly regressing three-dimensional coordinates from images, but the single-modal information limits the accuracy improvement space; the multi-modal fusion exploration stage, which begins to attempt to fuse multiple modalities such as RGB, depth, and infrared, but lacks effective fusion strategies and occlusion processing mechanisms.

[0004] The main limitations of existing technologies are insufficient single-modal information, insufficient ability to handle occlusion scenarios, insufficient utilization of temporal information, and immature multi-modal fusion strategies.

[0005] Insufficient single-modal information problem; existing three-dimensional human pose estimation methods mainly rely on single RGB image inputs, with the following technical defects:

[0006] Missing depth information: RGB images are essentially two-dimensional projections of the three-dimensional world, lacking direct depth information, resulting in severely insufficient prediction accuracy of the Z-axis coordinates. Especially in scenarios such as human self-occlusion and multi-person interaction, it is difficult to accurately distinguish the front-back relationship and depth levels based solely on RGB information.

[0007] Light sensitivity: RGB images are extremely sensitive to changes in lighting conditions. In complex lighting environments such as strong light, backlight, and shadows, the human contour and texture information severely degrades, leading to the failure of key point detection.

[0008] Texture dependence: Traditional RGB methods rely heavily on the texture and color information of the human body surface. In cases where the clothing color is single, the texture is missing, or similar to the background, the key point localization accuracy significantly decreases.

[0009] Insufficient ability to handle occlusion scenarios problem; complex occlusion scenarios are the core challenges in three-dimensional human pose estimation, and existing technologies have significant deficiencies in this regard:

[0010] Lack of occlusion detection mechanism: Most existing methods lack effective occlusion detection and quantification mechanisms, making it impossible to accurately identify which key points are occluded, resulting in the application of the same processing strategy to occluded and visible areas.

[0011] The occlusion prediction strategy is too simplistic: traditional methods mainly rely on simple linear interpolation, symmetry assumptions or historical frame extrapolation for occlusion key points, and lack intelligent reasoning capabilities based on multimodal information.

[0012] Insufficient precision in joint-level occlusion processing: Existing methods mostly use overall occlusion judgment, which cannot achieve fine occlusion analysis at the joint level, resulting in some visible key points being incorrectly processed.

[0013] Poor adaptability to dynamic occlusion: In dynamic occlusion scenarios, existing methods are unable to adjust processing strategies in real time and lack an adaptive response mechanism to changes in the degree of occlusion.

[0014] The problem of insufficient utilization of time series information; existing methods have the following shortcomings in time series information processing:

[0015] Limitations of single-frame processing: Most existing methods are based on single-frame image processing, which ignores the temporal continuity constraint of human motion, and are prone to problems such as drastic jumps in key point positions and unstable prediction.

[0016] Insufficient depth in temporal modeling: Most methods that consider temporal information use simple temporal smoothing or Kalman filtering, lacking deep temporal semantic modeling capabilities.

[0017] Long-term modeling difficulties: In long-term occlusion scenarios, existing methods struggle to maintain long-term tracking and prediction of key points, and are prone to tracking loss issues.

[0018] Insufficient temporal consistency constraints: The lack of an effective temporal consistency constraint mechanism may lead to abrupt changes in pose prediction results between adjacent frames that do not conform to the laws of human motion.

[0019] The immaturity of multimodal fusion strategies: Although multimodal fusion is an important way to improve pose estimation performance, existing technologies still have significant shortcomings in this regard.

[0020] The modal fusion strategy is simple: existing multimodal methods mostly use simple feature splicing or weighted averaging, lacking deep modal interaction and collaborative modeling mechanisms.

[0021] Lack of modal reliability assessment: There is a lack of dynamic assessment mechanism for the reliability of different modes in different scenarios, and it is impossible to adaptively adjust the contribution weight of each mode according to environmental changes.

[0022] Insufficient spatial registration accuracy: The limited accuracy of spatial registration and temporal synchronization of multimodal data restricts the fusion effect.

[0023] Computational complexity control is difficult: Multimodal processing often leads to a significant increase in computational complexity, and existing methods struggle to meet real-time requirements while maintaining accuracy.

[0024] In industrial human-robot collaborative environments, 3D human pose estimation technology faces unique challenges. Complex occlusion environments, including production equipment, workpieces, and multi-person collaboration, frequently obscure human figures, requiring algorithms with robust occlusion handling capabilities. Multimodal sensing environments, such as industrial sites equipped with RGB cameras, depth sensors, and infrared thermal imagers, necessitate the effective fusion of multimodal information. High precision requirements are paramount; industrial operations demand extremely high accuracy, and pose estimation errors directly impact production quality and safety, requiring centimeter-level precision in key point localization. Real-time requirements are also crucial; human-robot collaboration demands millisecond-level response times, imposing stringent real-time requirements on the algorithms.

[0025] Other application scenarios also have additional requirements for 3D human pose estimation technology. Medical rehabilitation scenarios require precise joint angle and motion trajectory analysis, with extremely high requirements for temporal consistency and accuracy. Sports training scenarios need to capture fine movements during rapid motion, requiring high levels of temporal modeling and dynamic adaptability. Virtual reality interaction scenarios require natural and smooth pose tracking, with prominent requirements for temporal continuity and real-time performance.

[0026] Based on the limitations of existing technologies and application requirements, the development trends of 3D human pose estimation technology include: deep multimodal fusion, requiring the development of more intelligent multimodal fusion strategies to fully leverage the complementary advantages of each modality; intelligent occlusion perception processing, requiring the construction of accurate occlusion detection and adaptive processing mechanisms to improve robustness in complex scenes; enhanced temporal modeling capabilities, requiring deeper modeling and utilization of temporal information to ensure the temporal continuity and stability of prediction results; and a balance between real-time performance and accuracy, requiring lightweight and real-time algorithms while maintaining high accuracy. To address these technical challenges and development needs, there is an urgent need to develop an occlusion-aware 3D human pose estimation method based on multimodal video sequences to overcome the shortcomings of existing technologies and meet the needs of practical applications. Summary of the Invention

[0027] To address the above problems, this invention proposes a three-dimensional human pose estimation method and device based on multimodal and occlusion perception. By fusing RGB, depth, and infrared three-modal data, it uses a temporal coordinate classification method based on occlusion perception to predict the X and Y axis coordinates, and employs a Z-axis depth prediction method based on bucketing and multimodal multi-resolution probability fusion. This method can improve perception capabilities in complex scenes, solve the problem of insufficient depth prediction accuracy, and effectively handle the problem of three-dimensional human pose estimation in complex occlusion scenes.

[0028] On the one hand, the 3D human pose estimation method based on multimodal and occlusion perception has the following specific steps:

[0029] S1, acquire a multimodal video sequence including RGB sequence, depth sequence and infrared sequence;

[0030] S2, preprocess the multimodal video sequence to obtain a data-aligned multimodal video sequence;

[0031] S3, construct a 3D human pose estimation model including a spatiotemporal feature extraction network, an occlusion detection network, a temporal coordinate classification module, a Z-axis depth prediction module, and a 3D coordinate stitching module;

[0032] The spatiotemporal feature extraction network extracts spatiotemporal features from data-aligned multimodal video sequences to obtain multi-level spatiotemporal features;

[0033] The occlusion detection network fuses multi-level spatiotemporal features to obtain multimodal fusion features, and uses the multimodal fusion features to predict the occlusion probability.

[0034] The temporal coordinate classification module predicts coordinates based on multi-level spatiotemporal features and occlusion probability to obtain X-axis and Y-axis coordinates.

[0035] The Z-axis depth prediction module discretizes the depth range of human joints into buckets to obtain the center depth value of each bucket; it extracts depth features at both coarse and fine scales from multi-level spatiotemporal features to obtain a depth probability distribution; and it weights and sums the center depth value with the depth probability distribution to obtain the Z-axis depth coordinate.

[0036] The 3D coordinate stitching module stitches together the X-axis coordinates, Y-axis coordinates, and Z-axis depth coordinates to obtain complete 3D human body key point coordinates;

[0037] S4. Use data-aligned multimodal video sequences to train a 3D human pose estimation model to obtain a trained 3D human pose estimation model.

[0038] S5 uses the trained model to estimate the 3D human pose and obtain the coordinates of the 3D human key points.

[0039] Preferably, the preprocessing of the multimodal video sequence to obtain a data-aligned multimodal video sequence is as follows:

[0040] Spatial registration was performed, and a unified coordinate system for the multimodal video sequence was established using the calibration parameters of the RGB camera, depth sensor, and infrared thermal imager.

[0041] Time synchronization is performed using the Precise Time Protocol (PTP) to achieve hardware-level timestamp synchronization, and time delay between sensors is compensated by cubic spline interpolation.

[0042] Standardize the data, adjust the resolution of the RGB sequence, depth sequence, and infrared sequence, and unify the frame rate to obtain a data-aligned multimodal video sequence.

[0043] Preferably, the spatiotemporal feature extraction network is a 3D convolutional network; the kernel size of the 3D convolutional network is 3×3×3 or 5×5×5; the convolutional layers of the 3D convolutional network are separable convolutions to reduce computational cost, and a spatiotemporal attention mechanism is used for feature enhancement; the activation function of the 3D convolutional network is the Swish activation function; the multi-level spatiotemporal features include: The resolution is 1 / 4 of the input, the number of channels is 256, and the temporal receptive field coverage is... Used for precise spatial positioning; The resolution is 1 / 8 of the input, the number of channels is 512, and the temporal receptive field coverage is... Used for mid-level semantic understanding; The resolution is 1 / 16 of the input, the number of channels is 1024, and the temporal receptive field coverage is... , used for global context modeling; where, Indicates a time step.

[0044] Preferably, the occlusion detection network fuses multi-level spatiotemporal features to obtain multimodal fusion features, and uses these multimodal fusion features to predict occlusion probability, as detailed below:

[0045] 3D convolutional layers are used to fuse multi-level spatiotemporal features to obtain fused features;

[0046] Occlusion probability is predicted using fused features, and the occlusion probability is expressed as:

[0047] ;

[0048] in, Indicates the probability of occlusion; This represents the Sigmoid activation function; This represents a 2D convolution classifier; Indicates fusion characteristics; Indicates a joint index; Indicates a time step.

[0049] Preferably, the temporal coordinate classification module predicts coordinates based on multi-level spatiotemporal features and occlusion probability to obtain X-axis and Y-axis coordinates, as follows:

[0050] Feature map projection is performed, and the multi-level spatiotemporal features are subjected to two-dimensional max pooling along the height dimension to obtain pooled features in the X-axis direction. The multi-level spatiotemporal features are subjected to two-dimensional max pooling along the width dimension to obtain pooled features in the Y-axis direction. ;

[0051] The occlusion probabilities aggregated along the Y-axis are used as the occlusion sensing mask along the X-axis; the occlusion probabilities aggregated along the X-axis are used as the occlusion sensing mask along the Y-axis; the occlusion sensing masks are weighted and represented as follows:

[0052] ;

[0053] ;

[0054] in, This represents the X-axis output of the classifier after weighting by the occlusion-aware mask. This represents the Y-axis output of the classifier after weighting by the occlusion-aware mask. This represents a 1D convolution classifier; An occlusion-aware mask representing the X-axis direction; This represents an occlusion-aware mask along the Y-axis. and These represent the horizontal and vertical coordinate indices, respectively. Indicates a time step;

[0055] Timing information fusion is performed using a gated recurrent unit (GRU), represented as follows:

[0056] ;

[0057] ;

[0058] in, Represents temporal fusion features; Represents temporal fusion features; Indicates the weight of the current frame; Indicates the temporal fusion weights; Indicates the motion prediction weights; This represents the timing fusion function of the gated loop unit; Indicates the features of the current frame; This represents the predicted features based on the motion model; and They represent At time step and The value at time; Indicates the features of the current frame; and They represent At time step and The value at time; This represents the predicted features based on the motion model;

[0059] Continuous coordinates are obtained by calculating the expected value, and are represented as follows:

[0060] ;

[0061] ;

[0062] in, Represents the probability distribution of the X-axis coordinates; Represents the probability distribution of the Y-axis coordinate; Indicates width; Indicates altitude; Represents the Softmax function;

[0063] The X-axis coordinate is obtained by weighted summation using the probability distribution of the X-axis coordinate; the Y-axis coordinate is obtained by weighted summation using the probability distribution of the Y-axis coordinate.

[0064] Preferably, the depth feature extraction of multi-level spatiotemporal features at both coarse and fine scales is performed to obtain a depth probability distribution; the Z-axis depth coordinate is obtained by weighted summation of the center depth value and the depth probability distribution; specifically as follows:

[0065] The reliability of each modality is dynamically evaluated through an attention mechanism to obtain the modality weights.

[0066] Coarse-scale and fine-scale depth feature extraction is performed to obtain the coarse-grained depth probability distribution and fine-grained depth probability distribution for each modality.

[0067] Perform the final depth probabilistic fusion to obtain the Logit vector corresponding to each depth bucket, represented as:

[0068] ;

[0069] in, This represents the Logit vector corresponding to each depth bucket; Representing modes The weights; Indicates the weight at the coarse scale; Represents the weights at a finer scale; Representing modes Coarse-scale depth probability distribution; Representing modes Fine-scale depth probability distribution; Indicates the time step; RGB represents the mode corresponding to the RGB sequence, D represents the mode corresponding to the depth sequence, and IR represents the mode corresponding to the infrared sequence;

[0070] The density function for each bin is calculated as follows:

[0071] ;

[0072] in, Indicates the Gaussian amplitude parameter; Indicates the Gaussian standard deviation; Indicates a uniformly distributed base value; Indicates depth prior weights; This represents the prior distribution provided by the depth sensor; This indicates the depth value to be evaluated; Indicates the depth center;

[0073] The binning density function and the deep feature extraction network output logit are used. After multiplication and normalization, it can be represented as:

[0074] ;

[0075] in, Represents the depth probability distribution; and They represent the first and The logit vector of each bucket; and They represent the first The and the first The density function of each bucket; Indicates the number of buckets;

[0076] The final depth is calculated and expressed as:

[0077] ;

[0078] in, Indicates the center depth value; Indicates the final depth.

[0079] Preferably, the depth range of the human joint is discretized into buckets to obtain the center depth value of each bucket, as follows:

[0080] The human body detection bounding box is obtained using object detection, and the size of the bounding box is calculated.

[0081] Within the mask of the detection box region in the current frame, take all valid pixels of the depth map and calculate the median depth as the center;

[0082] The comprehensive occlusion degree, which characterizes the degree of spatial occlusion and the degree of temporal occlusion, is calculated based on the occlusion probability.

[0083] Based on overall occlusion Dynamically adjusting the depth prediction range is expressed as:

[0084] ;

[0085] in, Indicates the dynamic search range for depth prediction; Indicates the depth coefficient; Indicates the current size of the human detection frame; Represents the joint type coefficient; Indicates the occlusion spread factor; Indicates the overall occlusion level;

[0086] Different bin densities are applied based on the overall occlusion level; occlusion levels less than or equal to the first occlusion threshold are classified as slight occlusion; occlusion levels greater than or equal to the first occlusion threshold but less than the second occlusion threshold are classified as moderate occlusion; occlusion levels greater than or equal to the second occlusion threshold are classified as severe occlusion; specifically:

[0087] Slight shading is represented by 64 evenly distributed bins:

[0088] ;

[0089] in, Indicates the first The center depth value of each depth bucket; For depth center; Indicates the dynamic search range for depth prediction;

[0090] Moderate occlusion employs a hybrid strategy of 32 core dense buckets and 32 edge sparse buckets, represented as:

[0091] ;

[0092] in, This represents the depth step size for the core dense buckets; Let the depth step be represented as the edge sparse bucketing depth; then By sequentially increasing the step size within the search range and get;

[0093] Severe occlusion is addressed using 16 large-area buckets and depth prior guidance, represented as follows:

[0094] .

[0095] Preferably, the overall occlusion degree is as follows:

[0096] Calculate spatial occlusion and evaluate the degree of occlusion of joints within a single frame, expressed as:

[0097] ;

[0098] in, Indicates spatial occlusion; Indicates joint The sensory field region; Indicates characteristic functions; Indicates the visibility threshold; Indicates the total number of pixels in the region; Indicating the receptive field region Occlusion probability of coordinates;

[0099] The temporal occlusion degree is calculated by considering the persistence of occlusion over time, and is expressed as follows:

[0100] ;

[0101] in, Indicates temporal occlusion degree; Indicates the timing window radius; Indicates time step as Spatial occlusion during the time;

[0102] The overall occlusion degree is calculated by fusing spatial and temporal information and expressed as follows:

[0103] ;

[0104] in, Indicates the overall occlusion level; Indicates spatial weight.

[0105] On the other hand, a 3D human pose estimation device based on multimodal and occlusion perception includes the following:

[0106] A multimodal video sequence acquisition module is used to acquire multimodal video sequences including RGB sequences, depth sequences, and infrared sequences;

[0107] The preprocessing module is used to preprocess the multimodal video sequence to obtain a data-aligned multimodal video sequence;

[0108] The model building module is used to build a 3D human pose estimation model, which includes a spatiotemporal feature extraction network, an occlusion detection network, a temporal coordinate classification module, a Z-axis depth prediction module, and a 3D coordinate stitching module.

[0109] The spatiotemporal feature extraction network extracts spatiotemporal features from data-aligned multimodal video sequences to obtain multi-level spatiotemporal features;

[0110] The occlusion detection network fuses multi-level spatiotemporal features to obtain multimodal fusion features, and uses the multimodal fusion features to predict the occlusion probability.

[0111] The temporal coordinate classification module predicts coordinates based on multi-level spatiotemporal features and occlusion probability to obtain X-axis and Y-axis coordinates.

[0112] The Z-axis depth prediction module discretizes the depth range of human joints into buckets to obtain the center depth value of each bucket; it extracts depth features at both coarse and fine scales from multi-level spatiotemporal features to obtain a depth probability distribution; and it weights and sums the center depth value with the depth probability distribution to obtain the Z-axis depth coordinate.

[0113] The 3D coordinate stitching module stitches together the X-axis coordinates, Y-axis coordinates, and Z-axis depth coordinates to obtain complete 3D human body key point coordinates;

[0114] The model training module is used to train a 3D human pose estimation model using data-aligned multimodal video sequences to obtain a trained 3D human pose estimation model.

[0115] The 3D human pose estimation module is used to estimate the 3D human pose using a trained model and obtain the coordinates of the 3D human key points.

[0116] Compared with the prior art, the present invention has the following beneficial effects:

[0117] (1) This invention significantly improves perception capabilities in complex scenes by fusing RGB, depth and infrared three-modal data;

[0118] (2) This invention achieves accurate identification and adaptive processing of joint-level occlusion states through innovative occlusion detection and quantification technology;

[0119] (3) This invention significantly improves the temporal continuity and stability of attitude estimation by using GRU temporal modeling and multi-frame information fusion;

[0120] (4) This invention effectively solves the problem of insufficient depth prediction accuracy by using occlusion perception layering and binning technology;

[0121] (5) This invention uses RGB, depth and infrared three-modal fusion; occlusion detection and quantization technology; GRU temporal modeling and multi-frame information fusion; occlusion perception layering and binning technology. In complex scenarios such as occlusion, lighting changes and fast movement, this invention has stronger robustness and higher accuracy than existing technologies. Attached Figure Description

[0122] The present invention will now be described in further detail with reference to the accompanying drawings;

[0123] Figure 1 This is a flowchart of a three-dimensional human pose estimation method based on multimodal and occlusion perception according to an embodiment of the present invention;

[0124] Figure 2 This is a schematic diagram illustrating the technical path of the three-dimensional human pose estimation method based on multimodal and occlusion perception in an embodiment of the present invention.

[0125] Figure 3 This is a 3D human pose estimation framework diagram of the 3D human pose estimation method based on multimodal and occlusion perception according to an embodiment of the present invention.

[0126] Figure 4 This is a flowchart of the SimCC-X / Y axis prediction process for the 3D human pose estimation method based on multimodal and occlusion perception, according to an embodiment of the present invention.

[0127] Figure 5 This is a flowchart of occlusion perception binning and density-weighted normalization for Z-axis depth prediction in a 3D human pose estimation method based on multimodal and occlusion perception, according to an embodiment of the present invention.

[0128] Figure 6 This is a structural block diagram of a three-dimensional human pose estimation device based on multimodal and occlusion perception according to an embodiment of the present invention. Detailed Implementation

[0129] The present invention will be further described below through specific embodiments.

[0130] like Figure 1 and Figure 2 As shown, the specific steps of the 3D human pose estimation method based on multimodal and occlusion perception are as follows:

[0131] S1, acquire a multimodal video sequence including RGB sequence, depth sequence and infrared sequence.

[0132] The multimodal video sequences used as input data include: RGB sequences. Depth sequence and infrared sequence .in, This is the length of the timing window, typically set to 8-16 frames to balance computational efficiency and timing modeling capability; , These are the image height and width, respectively, with a standard resolution of 256×192 or 384×288 pixels.

[0133] S2, preprocess the multimodal video sequence to obtain a data-aligned multimodal video sequence.

[0134] To ensure the spatial and temporal consistency of multimodal data, precise data alignment is required. This includes spatial registration, temporal synchronization, and resolution unification.

[0135] Spatial registration involves calibrating the internal and external parameters of the RGB camera, depth sensor, and infrared thermal imager using the Zhang Zhengyou calibration method, establishing a unified coordinate system, and requiring sub-pixel level registration accuracy.

[0136] Specifically, the Zhang Zhengyou calibration method was used for camera calibration. Calibration parameters included: intrinsic parameter matrix; focal length. Main point Distortion coefficient ;External parameter matrix: rotation matrix Translation vector Accuracy requirements: sub-pixel level (reprojection error < 0.5 pixels); Coordinate system establishment: establish a unified world coordinate system based on the RGB camera.

[0137] The spatial registration transformation formula is expressed as:

[0138] ;

[0139] in, For modality Aligned pixel coordinates This is the intrinsic parameter matrix. It is an extrinsic parameter matrix. Used as world coordinates.

[0140] Time synchronization employs a hardware timestamp synchronization mechanism with software-level interpolation compensation, achieving a time synchronization accuracy of ≤1ms.

[0141] Specifically, hardware synchronization uses Precise Time Protocol (PTP) to achieve hardware-level timestamp synchronization; software compensation uses cubic spline interpolation to compensate for time delays between different sensors; synchronization accuracy is set to timestamp error ≤ 1ms and frame rate synchronization error ≤ 1 frame.

[0142] The time synchronization interpolation formula is expressed as:

[0143] ;

[0144] in, This represents the newly generated image frame of mode m, aligned with other modes, at time t. These are the cubic spline interpolation coefficients. Indicates time as Image frames of time modality m, As the base time, For time intervals.

[0145] Set the resolution to be uniform, adjust all modes to the same resolution, and set the frame rate to 30fps.

[0146] S3 constructs a 3D human pose estimation model that includes a spatiotemporal feature extraction network, an occlusion detection network, a temporal coordinate classification module, a Z-axis depth prediction module, and a 3D coordinate stitching module.

[0147] The framework diagram of the 3D human pose estimation model is as follows: Figure 3 As shown. The network architecture of the spatiotemporal feature extraction network is as follows:

[0148] Adopting a lightweight spatiotemporal backbone network To obtain multi-level spatiotemporal features, the network structure is based on 3D convolution and spatiotemporal attention mechanisms, and is represented as follows:

[0149] ;

[0150] in, The radius of the time window. For 3D spatiotemporal convolution operators, Represents multimodal spatiotemporal features. For time window arrive RGB image sequence within, For time window arrive Depth map sequence within, For time window arrive Infrared thermal image sequence within.

[0151] The multi-level feature specification of multi-level spatiotemporal features is as follows: The resolution is 1 / 4 of the input, the number of channels is 256, and the temporal receptive field coverage is... Used for precise spatial positioning; The resolution is 1 / 8 of the input, the number of channels is 512, and the temporal receptive field coverage is... Used for mid-level semantic understanding; The resolution is 1 / 16 of the input, the number of channels is 1024, and the temporal receptive field coverage is... It is used for global context modeling.

[0152] The 3D spatiotemporal convolution kernel configuration is as follows: in the temporal dimension, the kernel size is typically 3×3×3 or 5×5×5; in the spatial dimension, separable convolution is used to reduce computational complexity; the activation function uses the Swish activation function to enhance nonlinear expressiveness; and regularization employs GroupNorm and Dropout to prevent overfitting.

[0153] The network architecture of the occlusion detection network is as follows:

[0154] Occlusion detection network based on multimodal feature fusion It achieves accurate recognition of joint-level occlusion states; its input is multimodal spatiotemporal features. ,in, This is the radius of the temporal context window, typically set to 2-4 frames to capture short-term motion patterns. Its output is the joint-level occlusion probability. ,in, Indicates a joint index. Indicates a time step.

[0155] The specific implementation of the occlusion detection calculation process is as follows:

[0156] Multimodal feature fusion is represented as:

[0157] ;

[0158] in, It is a 3D convolutional layer used to fuse spatiotemporal information from different modalities.

[0159] The occlusion probability prediction is expressed as:

[0160] ;

[0161] in, It is the Sigmoid activation function. For 2D convolution classifiers, subscript Indicates the first Prediction results for each joint.

[0162] A multi-dimensional quantification strategy was used to quantify the degree of occlusion. To comprehensively assess the degree of occlusion, three quantitative indicators were designed: spatial, temporal, and comprehensive.

[0163] Spatial occlusion, which evaluates the degree of occlusion of joints within a single frame, is expressed as:

[0164] ;

[0165] in, For joints The sensory field area For indicator functions, This is the visibility threshold (usually set to 0.5). This represents the total number of pixels in the region.

[0166] Temporal occlusion degree, taking into account the duration of occlusion over time, is expressed as:

[0167] ;

[0168] in, The time window radius is the average degree of occlusion of the joint within the time window.

[0169] The final occlusion assessment, which integrates spatial and temporal information based on the overall occlusion degree, is expressed as:

[0170] ;

[0171] in, This represents the spatial weight, reflecting the importance of the occlusion state in the current frame.

[0172] Occlusion-aware temporal SimCC (Simple Coordinate Classification) three-axis extension.

[0173] The temporal coordinate classification module is used for predicting X and Y axis occlusion sensing coordinates.

[0174] This embodiment uses an improved SimCC coordinate classification, such as Figure 4 As shown, an occlusion-aware temporal coordinate classification method is used to extend the traditional SimCC to occlusion scenes. The specific implementation is as follows:

[0175] Feature map projection is represented as:

[0176] ;

[0177] ;

[0178] in, For batch size, The number of feature channels is used to project the 2D feature map into a 1D coordinate space through max pooling.

[0179] Occlusion-aware masking applications are represented as follows:

[0180] ;

[0181] ;

[0182] in, For 1D convolution classifiers, and Occlusion-aware masks in the X and Y directions.

[0183] To enhance the temporal continuity of predictions, GRU is used for temporal information fusion to achieve temporal consistency fusion, as shown below:

[0184] ;

[0185] ;

[0186] in, The weight of the current frame; For time-series fusion weights; Weights for motion prediction; This is the timing fusion function for gated loop units; Features of the current frame; These are prediction features based on motion models.

[0187] To perform the final coordinate prediction, continuous coordinates are obtained by calculating the expected value, as follows:

[0188] ;

[0189] ;

[0190] The Z-axis depth prediction module is used to perform Z-axis depth prediction.

[0191] Z-axis depth prediction is a core challenge in 3D pose estimation. This embodiment employs an occlusion awareness and multi-resolution fusion strategy, such as... Figure 5 As shown, it includes occlusion-aware layered depth binning and multimodal, multi-resolution probabilistic fusion.

[0192] Occlusion perception is layered and depth-based binning, as detailed below:

[0193] Perform detection frame analysis:

[0194] First, the detection frame parameters are calculated based on the human body detection results, and are expressed as follows:

[0195] ;

[0196] in, and These are the coordinates of the top left and bottom right corners of the detection box, respectively.

[0197] Within the mask of the detection bounding box in the current frame, take all valid pixels from the depth map, calculate the median depth as the center, and represent it as follows:

[0198] ;

[0199] in, Represents pixel coordinates The depth value at that location; Indicates the first Frame joint The detection frame area; Median operator;

[0200] Perform depth range adaptive calculation:

[0201] The depth prediction range is dynamically adjusted based on the degree of occlusion, as shown below:

[0202] ;

[0203] in, This is the depth coefficient, typically ranging from 0.8 to 1.2. Joint type coefficient, trunk joints: ; Limb joints: Endpoint joint: ; This is the occlusion spread factor, typically ranging from 0.3 to 0.5.

[0204] Use a bucketing strategy:

[0205] Different bin densities are used based on the degree of occlusion, which includes slight occlusion, moderate occlusion, and severe occlusion, as detailed below:

[0206] Slight obstruction ( The system uses 64 evenly distributed buckets to ensure high-precision prediction, as shown below:

[0207] ;

[0208] Moderate occlusion ( The strategy employs a hybrid approach of 32 core dense buckets and 32 edge sparse buckets, expressed as follows:

[0209] ;

[0210] in, This represents the depth step size for the core dense buckets; Let the depth step be represented as the edge sparse bucketing depth; then By sequentially increasing the step size within the search range (Edge area) and (Core area obtained)

[0211] Severe obstruction ( Using 16 large-range buckets + depth-first guidance reduces computational complexity, as shown below:

[0212] ;

[0213] In this embodiment, the bucketing density function adopts a Gaussian weighted density distribution to optimize the bucketing strategy, expressed as:

[0214] ;

[0215] in, The Gaussian amplitude parameter; For depth center; The standard deviation is Gaussian. The base value is uniformly distributed; For depth prior weights; Prior distributions provided for depth sensors.

[0216] Network output logit Multiplying by density and then normalizing, it can be expressed as:

[0217] ;

[0218] The final depth and confidence level are expressed as follows:

[0219] ;

[0220] Multimodal, multi-resolution probabilistic fusion, as detailed below:

[0221] A dual-scale prediction structure is used; two prediction branches are designed: a coarse-scale branch and a fine-scale branch. Global depth range prediction is performed at 1 / 16 resolution, resulting in low computational complexity and wide coverage; fine-scale branching is based on... (1 / 4 resolution) performs local fine depth prediction with high accuracy and rich detail.

[0222] Adaptive calculation of multimodal weights:

[0223] Dynamically assess the reliability of different modalities using an attention mechanism:

[0224] ;

[0225] in, The fusion weights for the RGB modes; The attention score is for the RGB modality; the attention score is calculated as follows:

[0226] ;

[0227] in, This is a multilayer perceptron attention network that outputs importance scores for each modality.

[0228] The final deep probabilistic fusion, which integrates multimodal and multi-resolution information, is represented as:

[0229] ;

[0230] in, This represents the Logit vector corresponding to each depth bucket; For modality The weights; , The weights for the coarse and fine scales are 0.4 and 0.6, respectively. , Modal The coarse-scale and fine-scale depth probability distributions.

[0231] The 3D coordinate stitching module is used for 3D coordinate stitching.

[0232] The triaxial expectation for the same joint j is expressed as:

[0233] ;

[0234] The final three-dimensional attitude coordinates are represented as follows:

[0235] ;

[0236] S4. Use data-aligned multimodal video sequences to train a 3D human pose estimation model to obtain a trained 3D human pose estimation model.

[0237] S5 uses the trained model to estimate the 3D human pose and obtain the coordinates of the 3D human key points.

[0238] In summary, this method first performs precise spatial registration and temporal synchronization preprocessing on video sequences of three modalities: RGB, depth, and infrared thermal imaging. Zhang Zhengyou's calibration method is used to achieve sub-pixel-level multimodal data alignment. Then, a lightweight 3D spatiotemporal convolutional network is used to extract multi-level spatiotemporal features, obtaining multi-scale feature representations at different resolutions. Next, an occlusion detection network based on multimodal feature fusion is constructed to achieve joint-level occlusion state recognition and multi-dimensional quantitative evaluation. In the coordinate prediction stage, an occlusion-aware temporal coordinate classification method is used to predict the X and Y axis coordinates, combined with gated recurrent units to achieve temporal consistency fusion. For Z-axis depth prediction, an occlusion-aware hierarchical depth binning strategy is adopted, dynamically adjusting the bin density according to the degree of occlusion. Multimodal multi-resolution probabilistic fusion technology is used to integrate coarse-scale global prediction and fine-scale local prediction, ultimately outputting complete 3D human keypoint coordinates. This method can effectively handle the 3D human pose estimation problem in complex occlusion scenarios, and is expected to improve estimation accuracy and robustness. This method is applicable to the following application scenarios: (1) Human pose estimation in complex occlusion environments, including mutual occlusion between people, object occlusion, self-occlusion, etc.; (2) Pose tracking in multi-person scenarios, improving recognition accuracy through multimodal information fusion; (3) Environments with large changes in illumination, using infrared thermal imaging information to supplement the deficiencies of RGB information; (4) Applications that require high-precision three-dimensional pose information, such as motion analysis, medical rehabilitation, human-computer interaction, etc.

[0239] like Figure 6As shown, the present invention also discloses a three-dimensional human pose estimation device based on multimodal and occlusion perception, comprising:

[0240] The multimodal video sequence acquisition module 601 is used to acquire multimodal video sequences including RGB sequences, depth sequences, and infrared sequences;

[0241] The preprocessing module 602 is used to preprocess the multimodal video sequence to obtain a data-aligned multimodal video sequence;

[0242] The model building module 603 is used to build a three-dimensional human pose estimation model, including a spatiotemporal feature extraction network, an occlusion detection network, a temporal coordinate classification module, a Z-axis depth prediction module, and a three-dimensional coordinate stitching module.

[0243] The spatiotemporal feature extraction network extracts spatiotemporal features from data-aligned multimodal video sequences to obtain multi-level spatiotemporal features;

[0244] The occlusion detection network fuses multi-level spatiotemporal features to obtain multimodal fusion features, and uses the multimodal fusion features to predict the occlusion probability.

[0245] The temporal coordinate classification module predicts coordinates based on multi-level spatiotemporal features and occlusion probability to obtain X-axis and Y-axis coordinates.

[0246] The Z-axis depth prediction module discretizes the depth range of human joints into buckets to obtain the center depth value of each bucket; it extracts depth features at both coarse and fine scales from multi-level spatiotemporal features to obtain a depth probability distribution; and it weights and sums the center depth value with the depth probability distribution to obtain the Z-axis depth coordinate.

[0247] The 3D coordinate stitching module stitches together the X-axis coordinates, Y-axis coordinates, and Z-axis depth coordinates to obtain complete 3D human body key point coordinates;

[0248] The model training module 604 is used to train a 3D human pose estimation model using data-aligned multimodal video sequences to obtain a trained 3D human pose estimation model.

[0249] The 3D human pose estimation module 605 is used to perform 3D human pose estimation using a trained model to obtain the coordinates of 3D human key points.

[0250] The specific implementation of the 3D human pose estimation device based on multimodal and occlusion perception is the same as that of the 3D human pose estimation method based on multimodal and occlusion perception, and will not be described again in this embodiment.

[0251] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A three-dimensional human pose estimation method based on multimodal and occlusion perception, characterized in that, Includes the following steps: S1, acquire a multimodal video sequence including RGB sequence, depth sequence and infrared sequence; S2, preprocess the multimodal video sequence to obtain a data-aligned multimodal video sequence; S3, construct a 3D human pose estimation model including a spatiotemporal feature extraction network, an occlusion detection network, a temporal coordinate classification module, a Z-axis depth prediction module, and a 3D coordinate stitching module; The spatiotemporal feature extraction network extracts spatiotemporal features from data-aligned multimodal video sequences to obtain multi-level spatiotemporal features; The occlusion detection network fuses multi-level spatiotemporal features to obtain multimodal fusion features, and uses the multimodal fusion features to predict the occlusion probability. The temporal coordinate classification module predicts coordinates based on multi-level spatiotemporal features and occlusion probability to obtain X-axis and Y-axis coordinates. The Z-axis depth prediction module discretizes the depth range of human joints into buckets to obtain the center depth value of each bucket; it extracts depth features at both coarse and fine scales from multi-level spatiotemporal features to obtain a depth probability distribution; and it weights and sums the center depth value with the depth probability distribution to obtain the Z-axis depth coordinate. The 3D coordinate stitching module stitches together the X-axis coordinates, Y-axis coordinates, and Z-axis depth coordinates to obtain complete 3D human body key point coordinates; S4. Use data-aligned multimodal video sequences to train a 3D human pose estimation model to obtain a trained 3D human pose estimation model. S5. Use the trained model to perform 3D human pose estimation and obtain the coordinates of 3D human key points. The process involves extracting depth features at both coarse and fine scales from multi-level spatiotemporal features to obtain a depth probability distribution; the Z-axis depth coordinates are then obtained by weighted summation of the center depth value and the depth probability distribution; specifically as follows: The reliability of each modality is dynamically evaluated through an attention mechanism to obtain the modality weights. Coarse-scale and fine-scale depth feature extraction is performed to obtain the coarse-grained depth probability distribution and fine-grained depth probability distribution for each modality. Perform the final depth probabilistic fusion to obtain the Logit vector corresponding to each depth bucket, represented as: ； in, This represents the Logit vector corresponding to each depth bucket; Representing modes The weights; Indicates the weight at the coarse scale; Represents the weights at a finer scale; Representing modes Coarse-scale depth probability distribution; Representing modes Fine-scale depth probability distribution; Indicates the time step; RGB represents the mode corresponding to the RGB sequence, D represents the mode corresponding to the depth sequence, and IR represents the mode corresponding to the infrared sequence; The density function for each bin is calculated as follows: ； in, Indicates the Gaussian amplitude parameter; Indicates the Gaussian standard deviation; Indicates a uniformly distributed base value; Indicates depth prior weights; This represents the prior distribution provided by the depth sensor; This indicates the depth value to be evaluated; Indicates the depth center; The binning density function and the deep feature extraction network output logit are used. After multiplication and normalization, it can be represented as: ； in, Represents the depth probability distribution; and They represent the first and The logit vector of each bucket; and They represent the first The and the first The density function of each bucket; Indicates the number of buckets; The final depth is calculated and expressed as: ； in, Indicates the center depth value; Indicates the final depth.

2. The three-dimensional human pose estimation method based on multimodal and occlusion perception according to claim 1, characterized in that, The preprocessing of the multimodal video sequence to obtain a data-aligned multimodal video sequence is as follows: Spatial registration was performed, and a unified coordinate system for the multimodal video sequence was established using the calibration parameters of the RGB camera, depth sensor, and infrared thermal imager. Time synchronization is performed using the Precise Time Protocol (PTP) to achieve hardware-level timestamp synchronization, and time delay between sensors is compensated by cubic spline interpolation. Standardize the data, adjust the resolution of the RGB sequence, depth sequence, and infrared sequence, and unify the frame rate to obtain a data-aligned multimodal video sequence.

3. The three-dimensional human pose estimation method based on multimodal and occlusion perception according to claim 1, characterized in that, The spatiotemporal feature extraction network is a 3D convolutional network; the kernel size of the 3D convolutional network is 3×3×3 or 5×5×5; the convolutional layers of the 3D convolutional network are separable convolutions to reduce computational cost, and feature enhancement is performed using a spatiotemporal attention mechanism; the activation function of the 3D convolutional network is the Swish activation function. The multi-level spatiotemporal features include: The resolution is 1 / 4 of the input, the number of channels is 256, and the temporal receptive field coverage is... Used for precise spatial positioning; The resolution is 1 / 8 of the input, the number of channels is 512, and the temporal receptive field coverage is... Used for mid-level semantic understanding; The resolution is 1 / 16 of the input, the number of channels is 1024, and the temporal receptive field coverage is... , used for global context modeling; where, Indicates a time step.

4. The three-dimensional human pose estimation method based on multimodal and occlusion perception according to claim 1, characterized in that, The occlusion detection network fuses multi-level spatiotemporal features to obtain multimodal fusion features, and uses these multimodal fusion features to predict occlusion probability, as detailed below: 3D convolutional layers are used to fuse multi-level spatiotemporal features to obtain fused features; Occlusion probability is predicted using fused features, and the occlusion probability is expressed as: ； in, Indicates the probability of occlusion; This represents the Sigmoid activation function; This represents a 2D convolution classifier; Indicates fusion characteristics; Indicates a joint index; Indicates a time step.

5. The three-dimensional human pose estimation method based on multimodal and occlusion perception according to claim 1, characterized in that, The temporal coordinate classification module predicts coordinates based on multi-level spatiotemporal features and occlusion probability to obtain X-axis and Y-axis coordinates, as detailed below: Feature map projection is performed, and the multi-level spatiotemporal features are subjected to two-dimensional max pooling along the height dimension to obtain pooled features in the X-axis direction. The multi-level spatiotemporal features are subjected to two-dimensional max pooling along the width dimension to obtain pooled features in the Y-axis direction. ; Use the occlusion probabilities aggregated along the Y-axis as the occlusion perception mask in the X-axis direction; use the occlusion probabilities aggregated along the X-axis as the occlusion perception mask in the Y-axis direction. Weighted using occlusion-aware masks, represented as: ； ； in, This represents the X-axis output of the classifier after weighting by the occlusion-aware mask. This represents the Y-axis output of the classifier after weighting by the occlusion-aware mask. This represents a 1D convolution classifier; An occlusion-aware mask representing the X-axis direction; This represents an occlusion-aware mask along the Y-axis. and These represent the horizontal and vertical coordinate indices, respectively. Indicates a time step; Timing information fusion is performed using a gated recurrent unit (GRU), represented as follows: ； ； in, Represents temporal fusion features; Represents temporal fusion features; Indicates the weight of the current frame; Indicates the temporal fusion weights; Indicates the motion prediction weights; This represents the timing fusion function of the gated loop unit; Indicates the features of the current frame; This represents the predicted features based on the motion model; and They represent At time step and The value at time; Indicates the features of the current frame; and They represent At time step and The value at time; This represents the predicted features based on the motion model; Continuous coordinates are obtained by calculating the expected value, and are represented as follows: ； ； in, Represents the probability distribution of the X-axis coordinates; Represents the probability distribution of the Y-axis coordinate; Indicates width; Indicates altitude; Represents the Softmax function; The X-axis coordinate is obtained by weighted summation using the probability distribution of the X-axis coordinate; the Y-axis coordinate is obtained by weighted summation using the probability distribution of the Y-axis coordinate.

6. The three-dimensional human pose estimation method based on multimodal and occlusion perception according to claim 1, characterized in that, The depth range of human joints is discretized into buckets to obtain the center depth value of each bucket, as detailed below: The human body detection bounding box is obtained using object detection, and the size of the bounding box is calculated. Within the mask of the detection box region in the current frame, take all valid pixels of the depth map and calculate the median depth as the center; The comprehensive occlusion degree, which characterizes the degree of spatial occlusion and the degree of temporal occlusion, is calculated based on the occlusion probability. Based on overall occlusion Dynamically adjusting the depth prediction range is expressed as: ； in, Indicates the dynamic search range for depth prediction; Indicates the depth coefficient; Indicates the current size of the human detection frame; Represents the joint type coefficient; Indicates the occlusion spread factor; Indicates the overall occlusion level; Different bin densities are applied based on the overall occlusion level; occlusion levels less than or equal to the first occlusion threshold are classified as slight occlusion; occlusion levels greater than or equal to the first occlusion threshold but less than the second occlusion threshold are classified as moderate occlusion; occlusion levels greater than or equal to the second occlusion threshold are classified as severe occlusion; specifically: Slight shading is represented by 64 evenly distributed bins: ； in, Indicates the first The center depth value of each depth bucket; For depth center; Indicates the dynamic search range for depth prediction; Moderate occlusion employs a hybrid strategy of 32 core dense buckets and 32 edge sparse buckets, represented as: ； in, This represents the depth step size for the core dense buckets; Let the depth step be represented as the edge sparse bucketing depth; then By sequentially increasing the step size within the search range and get; Severe occlusion is addressed using 16 large-area buckets and depth prior guidance, represented as follows: 。 7. The three-dimensional human pose estimation method based on multimodal and occlusion perception according to claim 6, characterized in that, The overall occlusion level is as follows: Calculate spatial occlusion and evaluate the degree of occlusion of joints within a single frame, expressed as: ； in, Indicates spatial occlusion; Indicates joint The sensory field region; Indicates characteristic functions; Indicates the visibility threshold; Indicates the total number of pixels in the region; Indicating the receptive field region Occlusion probability of coordinates; The temporal occlusion degree is calculated by considering the persistence of occlusion over time, and is expressed as follows: ； in, Indicates temporal occlusion degree; Indicates the timing window radius; Indicates time step as Spatial occlusion during the time; The overall occlusion degree is calculated by fusing spatial and temporal information and expressed as follows: ； in, Indicates the overall occlusion level; Indicates spatial weight.

8. A three-dimensional human pose estimation device based on multimodal and occlusion perception, characterized in that, Including the following: A multimodal video sequence acquisition module is used to acquire multimodal video sequences including RGB sequences, depth sequences, and infrared sequences; The preprocessing module is used to preprocess the multimodal video sequence to obtain a data-aligned multimodal video sequence; The model building module is used to build a 3D human pose estimation model, which includes a spatiotemporal feature extraction network, an occlusion detection network, a temporal coordinate classification module, a Z-axis depth prediction module, and a 3D coordinate stitching module. The spatiotemporal feature extraction network extracts spatiotemporal features from data-aligned multimodal video sequences to obtain multi-level spatiotemporal features; The occlusion detection network fuses multi-level spatiotemporal features to obtain multimodal fusion features, and uses the multimodal fusion features to predict the occlusion probability. The temporal coordinate classification module predicts coordinates based on multi-level spatiotemporal features and occlusion probability to obtain X-axis and Y-axis coordinates. The Z-axis depth prediction module discretizes the depth range of human joints into buckets to obtain the center depth value of each bucket; it extracts depth features at both coarse and fine scales from multi-level spatiotemporal features to obtain a depth probability distribution; and it weights and sums the center depth value with the depth probability distribution to obtain the Z-axis depth coordinate. The 3D coordinate stitching module stitches together the X-axis coordinates, Y-axis coordinates, and Z-axis depth coordinates to obtain complete 3D human body key point coordinates; The model training module is used to train a 3D human pose estimation model using data-aligned multimodal video sequences to obtain a trained 3D human pose estimation model. The 3D human pose estimation module is used to perform 3D human pose estimation using a trained model and obtain the coordinates of 3D human key points. The process involves extracting depth features at both coarse and fine scales from multi-level spatiotemporal features to obtain a depth probability distribution; the Z-axis depth coordinates are then obtained by weighted summation of the center depth value and the depth probability distribution; specifically as follows: The reliability of each modality is dynamically evaluated through an attention mechanism to obtain the modality weights. Coarse-scale and fine-scale depth feature extraction is performed to obtain the coarse-grained depth probability distribution and fine-grained depth probability distribution for each modality. Perform the final depth probabilistic fusion to obtain the Logit vector corresponding to each depth bucket, represented as: ； in, This represents the Logit vector corresponding to each depth bucket; Representing modes The weights; Indicates the weight at the coarse scale; Represents the weights at a finer scale; Representing modes Coarse-scale depth probability distribution; Representing modes Fine-scale depth probability distribution; Indicates the time step; RGB represents the mode corresponding to the RGB sequence, D represents the mode corresponding to the depth sequence, and IR represents the mode corresponding to the infrared sequence; The density function for each bin is calculated as follows: ； in, Indicates the Gaussian amplitude parameter; Indicates the Gaussian standard deviation; Indicates a uniformly distributed base value; Indicates depth prior weights; This represents the prior distribution provided by the depth sensor; This indicates the depth value to be evaluated; Indicates the depth center; The binning density function and the deep feature extraction network output logit are used. After multiplication and normalization, it can be represented as: ； in, Represents the depth probability distribution; and They represent the first and The logit vector of each bucket; and They represent the first The and the first The density function of each bucket; Indicates the number of buckets; The final depth is calculated and expressed as: ； in, Indicates the center depth value; Indicates the final depth.

Images