Two-dimensional human pose estimation method for complex scenes based on double-branch hybrid architecture

Abstract

The application discloses a complex scene two-dimensional human body posture estimation method based on a double-branch mixed architecture, constructs a double-branch mixed posture estimation network, obtains joint heat maps and fused joint heat maps, constructs a detection-posture feature fusion subnetwork, divides a human body posture data set into a training set and a verification set, jointly trains the network by using the training set, simultaneously updates learnable parameters of human body detection, two heat map branches, occlusion attention and fusion weights in one-time back propagation, and selects a model according to a verification set index. The application inputs a to-be-processed image into the trained network, and outputs multi-human two-dimensional posture estimation results, so that the accuracy and robustness of multi-human posture estimation under the conditions of congestion, occlusion and detection regression error cascade propagation are improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to human pose estimation technology, and more particularly to a two-dimensional human pose estimation method for complex scenes with occlusion perception dual-branch hybrid, which is used to simultaneously estimate the two-dimensional positions of multiple human bounding boxes and joints from a single image in situations where multiple people overlap, occlusion and background interference coexist. Background Technology

[0002] Human pose estimation is a fundamental task in computer vision. Its goal is to estimate the two-dimensional or three-dimensional positions of human joints from images and form a skeleton representation. It has wide applications in behavior analysis, intelligent monitoring, and human-computer interaction. Heatmap-based two-dimensional pose estimation, by predicting spatial response maps for each joint and decoding coordinates on them, has achieved high accuracy on public benchmarks and has become a common technical approach.

[0003] In multi-person scenarios, existing methods can be broadly categorized into top-down and bottom-up approaches based on their main workflow. Top-down methods first obtain the human detection bounding box, then perform keypoint or heatmap regression on individual people within the box. This helps highlight the appearance of individual people and suppress some background, but aliasing within the bounding box, missed detections, and box positioning errors can still easily cause the regression stage to absorb detection errors. Bottom-up methods directly regress the joint heatmaps on the entire input image, facilitating the use of global context and skeletal structure clues. However, in cases of strong overlap, difficulties such as peak attribution and correspondence among multiple people may still arise.

[0004] Furthermore, occlusion reduces the visibility of some joints and weakens the texture and edge information used for precise localization within the neighborhood. Without compensation for occlusion and interference, responses between heatmap channels are prone to mutual interference. Some studies have introduced attention or conditional heatmaps into single-path networks, but end-to-end solutions that jointly model human detection, whole-map heatmap regression, and cross-branch consistency are still relatively insufficient. Therefore, there is an urgent need for a complex scene 2D human pose estimation method that combines top-down detection priors with bottom-up whole-map heatmaps and incorporates occlusion-aware feature reweighting, enabling parallel detection and multi-human pose inference in a single forward propagation. Summary of the Invention

[0005] The purpose of this invention is to solve the problem of insufficient accuracy and robustness of existing multi-person pose estimation under conditions of crowding, occlusion, and cascading propagation of detection regression errors.

[0006] To address the aforementioned problems, this invention provides a method for estimating two-dimensional human pose in complex scenes based on a dual-branch hybrid architecture, comprising the following steps:

[0007] S1: Construct a dual-branch hybrid pose estimation network; the network integrates the human detection branch, the top-down joint heatmap branch, the bottom-up full-frame input joint heatmap branch, the dual-branch heatmap fusion module, and the detection-pose feature fusion subnetwork into a single forward propagation computable end-to-end model;

[0008] S2: The top-down joint heatmap branch regresses the joint heatmap corresponding to the detected human body on the human body sub-image cropped by the bounding box obtained from human body detection, denoted as H. TD ;

[0009] S3: The bottom-up full-image input joint heatmap branch obtains a joint heatmap covering the entire image by passing through the backbone network, conditional heatmap fusion, occlusion perception attention, and heatmap head on the full input, denoted as H. BU ;

[0010] S4: Construct a dual-branch heatmap fusion module; after the two branch heatmaps are consistent in terms of mesh and joint channels, they are fused element-wise using learnable weights to obtain the fused joint heatmap H. fuse The joint plane coordinates are obtained by decoding the differentiable coordinates;

[0011] S5: Construct a detection-pose feature fusion subnetwork; the bounding box, category and confidence output by the human detection branch are mapped to obtain detection features, which are then concatenated with pose features and passed through the fusion subnetwork to obtain fusion features, and participate in cross-branch consistency or fusion-related supervision in the total loss;

[0012] S6: Prepare a human pose dataset containing multiple people, occlusion, and crowding scenarios. Divide the data into training and validation sets. Perform consistent preprocessing on the joint topology, input resolution, and heatmap grid size used in annotation and network output according to the selected evaluation protocol and network structure.

[0013] S7: The network is jointly trained using the training set. The total loss is obtained by weighted summation of at least two of the following: human detection loss, pose regression loss based on fused joint heatmap, and fusion or cross-branch constraint loss.

[0014] S8: In one backpropagation, the learnable parameters of human detection, two heatmap branches, occlusion attention and fusion weights are updated simultaneously, and the model is selected based on the validation set metrics.

[0015] S9: Input the image to be processed into the trained network, and obtain the human body detection boxes, fused joint heatmaps and joint plane coordinates in parallel during one forward propagation, as the result of multi-human two-dimensional pose estimation.

[0016] In the preferred embodiment, the bottom-up branching backbone network adopts a convolutional neural network. The input image is processed by the first convolutional layer, batch normalization and activation, and then by multiple convolutions and pooling to obtain multi-scale features. The conditional heatmap is bilinearly interpolated in height and width to be the same height and width as the output features of the backbone network, and then concatenated with the features in the channel dimension and fused by 1×1 convolution to obtain fused features.

[0017] In a preferred embodiment, the occlusion perception attention module includes an occlusion probability estimation unit, a spatial attention sub-branch, and a channel attention sub-branch; the occlusion probability estimation unit performs several layers of convolution and activation on the fused features to generate a single-channel occlusion probability map, and each element takes a value between 0 and 1 after being processed by Sigmoid, indicating the possibility of occlusion at a spatial location.

[0018] In the preferred embodiment, the spatial attention sub-branch performs a 1×1 convolution on the features to obtain the query and the key. Multiple dilated convolutions with distinguishably set dilation rates are applied in parallel to the key. Each layer is summed element-wise with the initial key to obtain an enhanced key incorporating multi-scale context. The query is then processed by global average pooling and correlation operations to obtain spatial attention coefficients, which are combined with the enhanced key. The combined result is further adaptively adjusted by multiplicative and additive terms generated from the occlusion probability map through two 1×1 convolutions to adjust the effective receptive field according to the degree of occlusion.

[0019] In the preferred embodiment, the channel attention sub-branch maps the features to queries and multi-channel keys respectively using 1×1 convolutions. The channel attention weights are obtained through matrix operations and Sigmoid, and share the same adaptive parameters for generating the self-occlusion probability map with the spatial branch. The final output of the module is the result of multiplying the fused features and the original fused features after 1×1 convolutions element-wise with the spatial attention weight map and the channel attention weight vector respectively. Then, the residual is added to the unweighted fused features to obtain the enhanced features. When this module is turned off, it degenerates into directly passing the previous-level features without attention calculations.

[0020] In the preferred embodiment, the enhanced features are further upsampled, convolved, and bilinearly interpolated to obtain a bottom-up branch joint heatmap H. BU The top-down branch obtains H on the human body sub-graph via a heatmap prediction head. TD If the height and width of two heatmaps are different, one of them should be bilinearly interpolated to match the height and width of the other before weighted fusion.

[0021] In the preferred embodiment, the conditional heatmap used by the detection branch as a bottom-up branch includes a single-channel or multi-channel prior heatmap that is coplanar with the backbone features, formed by constructing or updating the conditional quantities obtained by encoding the normalized position, scale, and confidence of each human candidate box on the entire input, based on the normalized position, scale, and confidence of each human candidate box. The conditional heatmap is obtained from the heatmap generated by annotation, historical iteration estimation, or other offline results during the training phase, and is constructed or updated by encoding the human candidate box information output by the detection branch during the inference phase.

[0022] In the preferred embodiment, the learnable scalar w takes a value between 0 and 1, and the fused joint heatmap is represented as H. fuse = w·H BU + (1-w)·H TD And, together with the backbone network and the two heatmap heads, it is updated synchronously with the total loss during training; the joint coordinates are updated by adjusting the H-axis of each joint channel. fuse The above soft-argmax decoding method is used to obtain the multi-person pose, which is then combined with the detection box and multi-person instance matching strategy to output the multi-person pose.

[0023] In the preferred embodiment, the top-down human detection sub-network, after preprocessing the entire image at a fixed spatial resolution, is input to a convolutional backbone, 1×1 channel width adjustable convolution, and a multi-scale detection head. The output includes normalized bounding boxes, human and background categories, and target confidence scores for each human candidate. For each human candidate, a sub-image is cropped from the original image within the normalized bounding box. The sub-image is then interpolated in height and width to match the agreed-upon input size of the heatmap prediction head. This sub-image is then sequentially processed through convolution, pooling, multi-step deconvolution upsampling, and a final 1×1 convolution to obtain the top-down joint heatmap H corresponding to the candidate. TD During training, if a sample has a bounding box, the first human body bounding box in the bounding box is used as the cropping criterion; otherwise, the bounding box currently output by the detection sub-network is used for joint optimization.

[0024] In the preferred embodiment, the total loss described in S7 is L = λ d ·L det + λ p ·L pose + λ f ·L fusion The weighted form, L det For the inspection supervision items relative to the inspection head, L pose To integrate the mean square regression error between the joint heatmap and the ground truth heatmap, L fusion λ is a term constraining the consistency of bi-branch or fusion features. d , λ f With λ p It is a non-negative constant;

[0025] The human body detection loss L detThe loss includes bounding box coordinates, category classification, and object confidence, employing a multi-task weighted approach compatible with detection networks; the pose regression loss L... pose The pixel-wise mean squared error is calculated between the fused joint heatmap and the ground Gaussian heatmap, and averaged over a batch of samples; the fusion or cross-branch constraint loss L... fusion Including at least one of the following: similarity constraints between detection features and pose features in the output of the fusion sub-network, response consistency constraints between the two heatmaps at corresponding spatial locations, or geometric consistency constraints between the decoded coordinates and the detection box;

[0026] The training strategy employs a two-stage progressive approach: pre-training on an occlusion-focused dataset and then continuing training on a crowded and overlapping-focused dataset, combined with data augmentation techniques such as horizontal flipping, random rotation, and color perturbation. The optimizer uses an adaptive learning rate method with weight decay, which can enable automatic mixed precision, cosine learning rate annealing, parametric moving average, and curriculum-based hard sample sampling.

[0027] The dataset mentioned in S6 is at least one or a mixture of COCO, OHuman, and CrowdPose, and the number of key points is consistent with the annotation specifications of the dataset. The data preprocessing includes scaling the input image to a fixed spatial resolution, subtracting the mean used for pre-training from each channel and dividing by the corresponding standard deviation to complete normalization, and mapping the 14-point annotation of CrowdPose to the 17-point channel of COCO and handling zero coordinates and visibility markers at missing joints.

[0028] The spatial resolution of the joint heatmap output by the network is fixed at a height of 96 and a width of 72, which is consistent with the grid size defined by the supervised heatmap. When the height and width dimensions of the two branch heatmaps are different, the dual-branch heatmap fusion module first bilinearly scales one branch to the same height and width as the other branch and unifies the joint channel order before performing weighted fusion.

[0029] The detection-pose feature fusion subnetwork consists of three fully connected layers stacked sequentially. The output of each fully connected layer is followed by a corrected linear unit for element-wise nonlinear feeding into the next layer. The multi-scale dilated convolution in the occlusion perception attention module uses three parallel dilated convolutions with dilation rates of 6, 12, and 18. The output of each layer is added element-wise to the original key to obtain the enhanced key.

[0030] The beneficial effects of this invention are as follows: This invention places the top-down detection branch and the bottom-up overall heatmap branch in the same network, enabling the prior information such as the spatial range and detection confidence provided by the human candidate box to guide the heatmap regression process on the entire input. The two types of complementary information work synergistically in a single forward propagation, thereby mitigating the problem of detection errors being progressively propagated to the keypoint regression stage in traditional staged methods. The occlusion perception attention module adaptively expands the receptive field and adjusts the spatial-channel weights in the occluded region by estimating the occlusion probability map, suppressing false heatmap responses that do not conform to the skeleton geometry constraints. The bi-branch heatmap is adaptively weighted and fused by a learnable scalar w according to the sample difficulty. The total loss of the multi-task process constrains the consistency of detection, fused heatmaps, and cross-branch features, achieving end-to-end joint optimization. Attached Figure Description

[0031] Figure 1 This is a diagram of the overall architecture of the dual-branch hybrid network of the present invention;

[0032] Figure 2 This is a top-down branch structure diagram;

[0033] Figure 3 This is a bottom-up heatmap branch structure diagram. Detailed Implementation

[0034] Example 1:

[0035] A method for 2D human pose estimation in complex scenes based on a dual-branch hybrid architecture includes the following steps:

[0036] S1: Construct a dual-branch hybrid pose estimation network; the network integrates the human detection branch, the top-down joint heatmap branch, the bottom-up full-frame input joint heatmap branch, the dual-branch heatmap fusion module, and the detection-pose feature fusion subnetwork into a single forward propagation computable end-to-end model;

[0037] S2: The top-down joint heatmap branch regresses the joint heatmap corresponding to the detected human body on the human body sub-image cropped by the bounding box obtained from human body detection, denoted as H. TD ;

[0038] S3: The bottom-up full-image input joint heatmap branch obtains a joint heatmap covering the entire image by passing through the backbone network, conditional heatmap fusion, occlusion perception attention, and heatmap head on the full input, denoted as H. BU ;

[0039] S4: Construct a dual-branch heatmap fusion module; after the two branch heatmaps are consistent in terms of mesh and joint channels, they are fused element-wise using learnable weights to obtain the fused joint heatmap H. fuse The joint plane coordinates are obtained by decoding the differentiable coordinates;

[0040] S5: Construct a detection-pose feature fusion subnetwork; the bounding box, category and confidence output by the human detection branch are mapped to obtain detection features, which are then concatenated with pose features and passed through the fusion subnetwork to obtain fusion features, and participate in cross-branch consistency or fusion-related supervision in the total loss;

[0041] S6: Prepare a human pose dataset containing multiple people, occlusion, and crowding scenarios. Divide the data into training and validation sets. Perform consistent preprocessing on the joint topology, input resolution, and heatmap grid size used in annotation and network output according to the selected evaluation protocol and network structure.

[0042] S7: The network is jointly trained using the training set. The total loss is obtained by weighted summation of at least two of the following: human detection loss, pose regression loss based on fused joint heatmap, and fusion or cross-branch constraint loss.

[0043] S8: In one backpropagation, the learnable parameters of human detection, two heatmap branches, occlusion attention and fusion weights are updated simultaneously, and the model is selected based on the validation set metrics.

[0044] S9: Input the image to be processed into the trained network, and obtain the human body detection boxes, fused joint heatmaps and joint plane coordinates in parallel during one forward propagation, as the result of multi-human two-dimensional pose estimation.

[0045] The bottom-up branching backbone network adopts a convolutional neural network. The input image is processed by the first convolutional layer, batch normalization and activation, and then by multiple convolutions and pooling to obtain multi-scale features. The conditional heatmap is bilinearly interpolated in height and width to be the same height and width as the output features of the backbone network. Then, it is concatenated with the features in the channel dimension and fused by 1×1 convolution to obtain fused features.

[0046] The occlusion perception attention module includes an occlusion probability estimation unit, a spatial attention sub-branch, and a channel attention sub-branch. The occlusion probability estimation unit performs several layers of convolution and activation on the fused features to generate a single-channel occlusion probability map. Each element takes a value between 0 and 1 after being processed by Sigmoid, indicating the possibility of occlusion at a spatial location.

[0047] The spatial attention sub-branch performs a 1×1 convolution on the features to obtain the query and the key. Multiple dilated convolutions with distinguishable dilation rates are applied to the key in parallel. Each layer is summed element-wise with the initial key to obtain an enhanced key that incorporates multi-scale context. The query is processed by global average pooling and correlation operations to obtain spatial attention coefficients, which are then combined with the enhanced key. The combination result is further adaptively adjusted by multiplicative and additive terms generated from the occlusion probability map through two 1×1 convolutions to adjust the effective receptive field according to the degree of occlusion.

[0048] The channel attention sub-branch maps features to queries and multi-channel keys using 1×1 convolutions, and obtains channel attention weights through matrix operations and sigmoid, sharing the same adaptive parameters for generating the self-occlusion probability map as the spatial branch. The final output of the module is the result of multiplying the fused feature and the original fused feature after 1×1 convolution with the spatial attention weight map and the channel attention weight vector element by element, and then adding the residual to the unweighted fused feature to obtain the enhanced feature. When this module is turned off, it degenerates into directly passing the previous-level features without attention calculation.

[0049] The enhanced features are then upsampled, convolutional, and bilinearly interpolated to obtain a bottom-up branch joint heatmap H. BU The top-down branch obtains H on the human body sub-graph via a heatmap prediction head. TD If the height and width of two heatmaps are different, one of them should be bilinearly interpolated to match the height and width of the other before weighted fusion.

[0050] The conditional heatmap used by the detection branch as a bottom-up branch includes a single-channel or multi-channel prior heatmap that is coplanar with the backbone features, formed by constructing or updating the conditional quantities obtained by encoding the normalized position, scale and confidence of each human candidate box on the whole input and then updating them. The conditional heatmap is obtained from the heatmap generated by annotation, historical iteration estimation or other offline results during the training phase, and is constructed or updated by encoding the human candidate box information output by the detection branch during the inference phase.

[0051] The learnable scalar w takes values ​​between 0 and 1, and is represented by H in the fused joint heatmap. fuse = w·H BU + (1-w)·H TD And, together with the backbone network and the two heatmap heads, it is updated synchronously with the total loss during training; the joint coordinates are updated by adjusting the H-axis of each joint channel. fuse The above soft-argmax decoding method is used to obtain the multi-person pose, which is then combined with the detection box and multi-person instance matching strategy to output the multi-person pose.

[0052] The top-down human detection subnetwork, after being preprocessed to a fixed spatial resolution, is input to a convolutional backbone, 1×1 channel width-adjustable convolution, and a multi-scale detection head. The output includes normalized bounding boxes, human and background categories, and target confidence scores for each human candidate. For each human candidate, a sub-image is cropped from the original image within the normalized bounding box. The sub-image is then interpolated in height and width to match the agreed-upon input size of the heatmap prediction head. This sub-image is then sequentially processed through convolution, pooling, multi-step deconvolution upsampling, and a final 1×1 convolution to obtain the top-down joint heatmap H corresponding to the candidate. TDDuring training, if a sample has a bounding box, the first human body bounding box in the bounding box is used as the cropping criterion; otherwise, the bounding box currently output by the detection sub-network is used for joint optimization.

[0053] The total loss described in S7 is L = λ d ·L det + λ p ·L pose + λ f ·L fusion The weighted form, L det For the inspection supervision items relative to the inspection head, L pose To integrate the mean square regression error between the joint heatmap and the ground truth heatmap, L fusion λ is a term constraining the consistency of bi-branch or fusion features. d , λ f With λ p It is a non-negative constant;

[0054] The human body detection loss L det The loss includes bounding box coordinates, category classification, and object confidence, employing a multi-task weighted approach compatible with detection networks; the pose regression loss L... pose The pixel-wise mean squared error is calculated between the fused joint heatmap and the ground Gaussian heatmap, and averaged over a batch of samples; the fusion or cross-branch constraint loss L... fusion Including at least one of the following: similarity constraints between detection features and pose features in the output of the fusion sub-network, response consistency constraints between the two heatmaps at corresponding spatial locations, or geometric consistency constraints between the decoded coordinates and the detection box;

[0055] The training strategy employs a two-stage progressive approach: pre-training on an occlusion-focused dataset and then continuing training on a crowded and overlapping-focused dataset, combined with data augmentation techniques such as horizontal flipping, random rotation, and color perturbation. The optimizer uses an adaptive learning rate method with weight decay, which can enable automatic mixed precision, cosine learning rate annealing, parametric moving average, and curriculum-based hard sample sampling.

[0056] The dataset mentioned in S6 is at least one or a mixture of COCO, OHuman, and CrowdPose, and the number of key points is consistent with the annotation specifications of the dataset. The data preprocessing includes scaling the input image to a fixed spatial resolution, subtracting the mean used for pre-training from each channel and dividing by the corresponding standard deviation to complete normalization, and mapping the 14-point annotation of CrowdPose to the 17-point channel of COCO and handling zero coordinates and visibility markers at missing joints.

[0057] The spatial resolution of the joint heatmap output by the network is fixed at a height of 96 and a width of 72, which is consistent with the grid size defined by the supervised heatmap. When the height and width dimensions of the two branch heatmaps are different, the dual-branch heatmap fusion module first bilinearly scales one branch to the same height and width as the other branch and unifies the joint channel order before performing weighted fusion.

[0058] The detection-pose feature fusion subnetwork consists of three fully connected layers stacked sequentially. The output of each fully connected layer is followed by a corrected linear unit for element-wise nonlinear feeding into the next layer. The multi-scale dilated convolution in the occlusion perception attention module uses three parallel dilated convolutions with dilation rates of 6, 12, and 18. The output of each layer is added element-wise to the original key to obtain the enhanced key.

[0059] Example 2:

[0060] In this embodiment, human detection, bottom-up full-frame heatmap, and top-down bounding box heatmap are computed in parallel during a single forward propagation, and the final pose estimate is obtained by fusing the occlusion-aware dual-branch heatmaps. To facilitate comparison with publicly available evaluation protocols, the number of joints adopts a 17-point format consistent with the COCO dataset; if using 14-point annotations such as CrowdPose, it can be mapped to a unified channel during data reading, and the coordinates and visibility markers of missing joints can be set to zero.

[0061] The input to this example is an RGB three-channel image. During preprocessing, the image is scaled to 416×416 resolution and normalized according to mean and variance. The network output includes the bounding boxes of detected human bodies and the coordinates of key points corresponding to each human body. The number of detections, K, is determined by the number of human bodies detected in the current forward propagation, and can also be filtered and processed by a confidence threshold. The network parameters are updated synchronously through a weighted sum of three losses: detection, pose, and fusion. This differs from the phased process of first training the detection branch and then fixing the detection and only training the regression branch.

[0062] One implementation of this method includes: S1 constructing and initializing the dual-branch hybrid network; S2 preparing and preprocessing the multi-person pose dataset; S3 jointly training and validating; and S4 inferring multi-person poses on the image to be tested. The structure of each module is described in detail below with reference to the accompanying drawings.

[0063] like Figure 1 As shown, the connection relationships of the human detection branch, the top-down heatmap branch, the bottom-up heatmap branch, and the fusion module are illustrated. The network structure of this invention includes a human detection branch, a top-down heatmap branch, a bottom-up heatmap branch, and a fusion module. After preprocessing the input image, the three processing paths run in parallel during the same forward propagation: the detection branch outputs candidate bounding boxes, categories, and confidence scores for each human body on the entire image; the top-down branch crops sub-images from the original image according to the candidate bounding boxes and regresses the heatmap H. TDThe bottom-up branch extracts features F from the RGB convolutional backbone, which is then concatenated with the conditional heatmap constructed from the detection information along the channel dimension and fused by a 1×1 convolution. This fusion is then fed into the occlusion probability estimation and spatial-channel dual-path attention module, followed by deconvolution and bilinear upsampling to obtain H. BU The two branch heatmaps are then uniformly sized using bilinear interpolation and weighted by learnable weights w to obtain H. fuse The joint coordinates are obtained through soft-argmax decoding. The detection features F output by the detection head are... d With pose feature F p F is obtained through the fusion subnetwork fus This is used for fusion loss constraints during the training phase.

[0064] like Figure 2 The diagram illustrates the processing flow from a convolutional backbone, a detection head, and human sub-image cropping to a heatmap prediction head. The top-down detection sub-network consists of a multi-scale convolutional backbone, a 1×1 channel width-adjustable convolution, and a multi-scale detection head. After the backbone extracts multi-scale two-dimensional features, the detection head outputs the normalized coordinates (x1, y1, x2, y2) of the bounding box. For each candidate box, the sub-image is bilinearly interpolated to the input size of the heatmap head, followed by convolution, pooling, multi-stride deconvolution upsampling, and a 1×1 convolution to obtain the corresponding human body H-axis. TD During training, the labeled bounding boxes should be used for cropping first; otherwise, the detection output boxes should be used.

[0065] like Figure 3 The diagram illustrates the processing flow of conditional heatmap fusion, occlusion probability estimation, spatial-channel attention modulation, and heatmap output. The bottom-up heatmap branch first extracts backbone features F through multi-layer convolution. The conditional heatmap is adjusted to the same size as F via bilinear interpolation, concatenated with F along the channel dimension, and then fused through convolution to obtain the initial features. The occlusion probability map is generated from the fused features through convolution and sigmoid activation. In the spatial attention branch, features are convolved to obtain query and key vectors; the key vector expands the receptive field through three dilated convolutions with dilation rates of 6, 12, and 18, and is added to the original key vector to obtain the enhancement key; the query vector is combined with the enhancement key after global average pooling, and adjusted by adaptive parameters generated from the occlusion probability map to obtain spatial weights. The channel attention branch uses a similar structure to generate channel weights. The final output is the result of the initial features after residual connection and attention weighting, followed by upsampling to obtain the heatmap H. BU .

[0066] This method can be trained and evaluated using datasets such as CrowdPose, OCHuman, and COCO, which contain occlusion or dense crowd scenes. For different joint number annotation systems, they can be mapped to a unified channel during data reading. Training adopts a progressive strategy, first training on datasets focusing on occlusion, and then further optimizing on scenarios focusing on dense overlap, combined with conventional data augmentation and adaptive learning rate adjustment.

[0067] The technical advantages of this method are as follows: the dual-branch hybrid architecture complements the detection prior and the overall image context information, and can maintain both human body separability and heatmap response stability in human body overlapping and occlusion scenarios; occlusion perception attention expands the effective receptive field and suppresses false responses in the occluded area through probability-guided feature modulation, thereby improving the robustness of key point localization; the progressive training strategy enables the network to gradually adapt to complex scene changes from occlusion to dense overlap; and the architecture design is compatible with various backbone networks, making it easy to deploy flexibly in actual systems.

[0068] The above description is merely a preferred embodiment of the present invention. Without departing from the scope of protection of the present invention, those skilled in the art can make improvements regarding module replacement, numerical selection, operator equivalence, and training formulas, all of which should be considered within the scope of protection of the present invention.

Claims

1. A method for estimating two-dimensional human pose in complex scenes based on a dual-branch hybrid architecture, characterized in that, Includes the following steps: S1: Construct a dual-branch hybrid pose estimation network; the network integrates the human detection branch, the top-down joint heatmap branch, the bottom-up full-frame input joint heatmap branch, the dual-branch heatmap fusion module, and the detection-pose feature fusion subnetwork into a single forward propagation computable end-to-end model; S2: The top-down joint heatmap branch regresses the joint heatmap corresponding to the detected human body on the human body sub-image cropped by the bounding box obtained from human body detection, denoted as H. TD ; S3: The bottom-up full-image input joint heatmap branch obtains a joint heatmap covering the entire image by passing through the backbone network, conditional heatmap fusion, occlusion perception attention, and heatmap head on the full input, denoted as H. BU ; S4: Construct a dual-branch heatmap fusion module; after the two branch heatmaps are consistent in terms of mesh and joint channels, they are fused element-wise using learnable weights to obtain the fused joint heatmap H. fuse The joint plane coordinates are obtained by decoding the differentiable coordinates; S5: Construct a detection-pose feature fusion subnetwork; the bounding box, category and confidence output by the human detection branch are mapped to obtain detection features, which are then concatenated with pose features and passed through the fusion subnetwork to obtain fusion features, and participate in cross-branch consistency or fusion-related supervision in the total loss; S6: Prepare a human pose dataset containing multiple people, occlusion, and crowding scenarios. Divide the data into training and validation sets. Perform consistent preprocessing on the joint topology, input resolution, and heatmap grid size used in annotation and network output according to the selected evaluation protocol and network structure. S7: The network is jointly trained using the training set. The total loss is obtained by weighted summation of at least two of the following: human detection loss, pose regression loss based on fused joint heatmap, and fusion or cross-branch constraint loss. S8: In one backpropagation, the learnable parameters of human detection, two heatmap branches, occlusion attention and fusion weights are updated simultaneously, and the model is selected based on the validation set metrics. S9: Input the image to be processed into the trained network, and obtain the human body detection boxes, fused joint heatmaps and joint plane coordinates in parallel during one forward propagation, as the result of multi-human two-dimensional pose estimation.

2. The method for 2D human pose estimation in complex scenes based on a dual-branch hybrid architecture according to claim 1, characterized in that, The bottom-up branching backbone network adopts a convolutional neural network. The input image is processed by the first convolutional layer, batch normalization and activation, and then by multiple convolutions and pooling to obtain multi-scale features. The conditional heatmap is bilinearly interpolated in height and width to be the same height and width as the output features of the backbone network. Then, it is concatenated with the features in the channel dimension and fused by 1×1 convolution to obtain fused features.

3. The method for 2D human pose estimation in complex scenes based on a dual-branch hybrid architecture according to claim 1, characterized in that, The occlusion perception attention module includes an occlusion probability estimation unit, a spatial attention sub-branch, and a channel attention sub-branch. The occlusion probability estimation unit performs several layers of convolution and activation on the fused features to generate a single-channel occlusion probability map. Each element takes a value between 0 and 1 after being processed by Sigmoid, indicating the possibility of occlusion at a spatial location.

4. The method for 2D human pose estimation in complex scenes based on a dual-branch hybrid architecture according to claim 1, characterized in that, The spatial attention sub-branch performs a 1×1 convolution on the features to obtain the query and the key. Multiple dilated convolutions with distinguishable dilation rates are applied to the key in parallel. Each layer is summed element-wise with the initial key to obtain an enhanced key that incorporates multi-scale context. The query is processed by global average pooling and correlation operations to obtain spatial attention coefficients, which are then combined with the enhanced key. The combination result is further adaptively adjusted by multiplicative and additive terms generated from the occlusion probability map through two 1×1 convolutions to adjust the effective receptive field according to the degree of occlusion.

5. The method for 2D human pose estimation in complex scenes based on a dual-branch hybrid architecture according to claim 1, characterized in that, The channel attention sub-branch maps features to queries and multi-channel keys using 1×1 convolutions, and obtains channel attention weights through matrix operations and sigmoid, sharing the same adaptive parameters for generating the self-occlusion probability map as the spatial branch. The final output of the module is the result of multiplying the fused feature and the original fused feature after 1×1 convolution with the spatial attention weight map and the channel attention weight vector element by element, and then adding the residual to the unweighted fused feature to obtain the enhanced feature. When this module is turned off, it degenerates into directly passing previous features without attention calculations.

6. The method for estimating two-dimensional human pose in complex scenes based on a dual-branch hybrid architecture according to claim 1, characterized in that, The enhanced features are then upsampled, convolutional, and bilinearly interpolated to obtain a bottom-up branch joint heatmap H. BU The top-down branch obtains H on the human body sub-graph via a heatmap prediction head. TD If the height and width of two heatmaps are different, one of them should be bilinearly interpolated to match the height and width of the other before weighted fusion.

7. The method for 2D human pose estimation in complex scenes based on a dual-branch hybrid architecture according to claim 1, characterized in that, The conditional heatmap used by the detection branch as a bottom-up branch includes a single-channel or multi-channel prior heatmap that is coplanar with the backbone features, formed by constructing or updating the conditional quantities obtained by encoding the normalized position, scale and confidence of each human candidate box on the whole input and then updating them. The conditional heatmap is obtained from the heatmap generated by annotation, historical iteration estimation or other offline results during the training phase, and is constructed or updated by encoding the human candidate box information output by the detection branch during the inference phase.

8. The method for 2D human pose estimation in complex scenes based on a dual-branch hybrid architecture according to claim 1, characterized in that, The learnable scalar w takes values ​​between 0 and 1, and is represented by H in the fused joint heatmap. fuse = w·H BU + (1-w)·H TD And, together with the backbone network and the two heatmap heads, it is updated synchronously with the total loss during training; the joint coordinates are updated by adjusting the H-axis of each joint channel. fuse The above soft-argmax decoding method is used to obtain the multi-person pose, which is then combined with the detection box and multi-person instance matching strategy to output the multi-person pose.

9. The method for 2D human pose estimation in complex scenes based on a dual-branch hybrid architecture according to claim 1, characterized in that, The top-down human detection subnetwork, after being preprocessed to a fixed spatial resolution, is input to a convolutional backbone, 1×1 channel width-adjustable convolution, and a multi-scale detection head. The output includes normalized bounding boxes, human and background categories, and target confidence scores for each human candidate. For each human candidate, a sub-image is cropped from the original image within the normalized bounding box. The sub-image is then interpolated in height and width to match the agreed-upon input size of the heatmap prediction head. This sub-image is then sequentially processed through convolution, pooling, multi-step deconvolution upsampling, and a final 1×1 convolution to obtain the top-down joint heatmap H corresponding to the candidate. TD During training, if a sample has a bounding box, the first human body bounding box in the bounding box is used as the cropping criterion; otherwise, the bounding box currently output by the detection sub-network is used for joint optimization.

10. The method for estimating two-dimensional human pose in complex scenes based on a dual-branch hybrid architecture according to claim 1, characterized in that, The total loss described in S7 is L = λ d ·L det + λ p ·L pose + λ f ·L fusion The weighted form, L det For the inspection supervision items relative to the inspection head, L pose To integrate the mean square regression error between the joint heatmap and the ground truth heatmap, L fusion λ is a term constraining the consistency of bi-branch or fusion features. d , λ f With λ p It is a non-negative constant; The human body detection loss L det The loss includes bounding box coordinates, category classification, and object confidence, employing a multi-task weighted approach compatible with detection networks; the pose regression loss L... pose The pixel-wise mean squared error is calculated between the fused joint heatmap and the ground Gaussian heatmap, and averaged over a batch of samples; the fusion or cross-branch constraint loss L... fusion Including at least one of the following: similarity constraints between detection features and pose features in the output of the fusion sub-network, response consistency constraints between the two heatmaps at corresponding spatial locations, or geometric consistency constraints between the decoded coordinates and the detection box; The training strategy employs a two-stage progressive approach: pre-training on an occlusion-focused dataset and then continuing training on a crowded and overlapping-focused dataset, combined with data augmentation techniques such as horizontal flipping, random rotation, and color perturbation. The optimizer uses an adaptive learning rate method with weight decay, which can enable automatic mixed precision, cosine learning rate annealing, parametric moving average, and curriculum-based hard sample sampling. The dataset mentioned in S6 is at least one or a mixture of COCO, OHuman, and CrowdPose, and the number of key points is consistent with the annotation specifications of the dataset. The data preprocessing includes scaling the input image to a fixed spatial resolution, subtracting the mean used for pre-training from each channel and dividing by the corresponding standard deviation to complete normalization, and mapping the 14-point annotation of CrowdPose to the 17-point channel of COCO and handling zero coordinates and visibility markers at missing joints. The spatial resolution of the joint heatmap output by the network is fixed at 96 in height and 72 in width, which is consistent with the grid size defined in the supervised heatmap. When the height and width dimensions of the two branch heatmaps are different, the dual-branch heatmap fusion module first bilinearly scales one of the branches to the same height and width as the other branch and unifies the joint channel order before performing weighted fusion. The detection-pose feature fusion subnetwork consists of three fully connected layers stacked sequentially. The output of each fully connected layer is followed by a correction linear unit for element-wise nonlinearity before being fed into the next layer. The multi-scale dilated convolution in the occlusion perception attention module uses three parallel dilated convolution layers with dilation rates of 6, 12, and 18. The output of each layer is added element-wise to the original bond to obtain the enhanced bond.

Images