A deep learning-based end-to-end robust pose estimation method in a complex crowd scene

Abstract

The application discloses a kind of robust pose estimation methods under complex crowd scene based on end-to-end deep learning, comprising: 1) using CrowdPose dataset as the data source of model training, the dataset is divided into training set, verification set and test set;2) construct deep convolutional neural network, including preprocessing part, backbone network part, feature enhancement part, feature fusion part and head part;3) set training parameters, including batch size, learning rate and early stopping strategy, and train the deep convolutional neural network accordingly until the model converges;4) input the image containing the person into the trained deep convolutional neural network, and the deep convolutional neural network outputs the key point coordinates of the person in the image, and the corresponding human pose graph is obtained according to the connection relationship of key points.The technical scheme of the present application can improve the pose estimation accuracy and robustness in complex scene, and is suitable for further popularization and application.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a robust pose estimation method for complex crowd scenes based on deep learning, providing an end-to-end approach. Background Technology

[0002] In the field of computer vision, pose estimation technology is mainly used to identify and locate key points of the human body in images or videos. With the rapid development of deep learning, methods based on deep neural networks have gradually become the mainstream of pose estimation, especially in applications in complex scenes.

[0003] However, traditional pose estimation methods still face numerous challenges in environments with multiple people and dense crowds. These challenges include occlusion, pose crossing, fine-grained target loss, and multi-scale issues, resulting in poor accuracy and robustness of pose estimation in complex backgrounds. Traditional methods often rely on hand-designed features and shallow models, making it difficult to adapt to these ever-changing environments, leading to inaccurate keypoint localization, or even misjudgment or missed detection. Although some methods in recent years, such as YOLO (YouOnly Look Once), have improved estimation accuracy to some extent, they still struggle to completely solve the problems of pose crossing and occlusion in complex scenes. Especially in dense crowds or highly occluded scenes, accurately locating each keypoint remains a critical challenge that needs to be overcome.

[0004] Therefore, improving the robustness of models in complex crowd scenarios and maintaining efficient and high-precision pose estimation in various environments has become an important research direction. Summary of the Invention

[0005] To address the aforementioned problems, the present invention aims to overcome issues such as occlusion, pose overlap, and dense crowds in existing pose estimation techniques, and proposes an end-to-end robust pose estimation method based on deep learning to improve the accuracy and robustness of pose estimation in complex scenes.

[0006] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows: This invention provides a robust pose estimation method for complex crowd scenarios based on deep learning, comprising the following steps: 1) Use the CrowdPose dataset as the data source for model training. Divide the dataset into training set, validation set and test set. Use the training set and validation set together for model training, while the test set is used for final model performance evaluation. 2) Construct a deep convolutional neural network, including a preprocessing part, a backbone network part, a feature enhancement part, a feature fusion part, and a head part; The preprocessing section is used to process materials with a size of [missing information]. The 3-channel image data is input into the SAMP-Stem (structure-aware multi-path feature extraction module) module for preprocessing, which realizes the initial downsampling of the image, reduces the image size and adjusts the number of channels to reduce the amount of computation, while retaining shallow features for subsequent efficient feature extraction; The backbone network is used to feed the shallow feature maps passed from the preprocessing part into the CSP-MSLFA (Multi-Scale Local Feature Aggregation Module) module for feature extraction, and combine it with downsampled convolutional blocks to extract more abstract and higher-level features at four different scales. The feature enhancement part improves the semantics of the high-level features extracted from the backbone network through the SMSFE (Shared Multi-Scale Feature Extraction) module in the feature enhancement module, and combines the C2PSA attention module to enhance the features of key regions to obtain features at the fourth scale. The feature fusion part uses a feature fusion module to fuse the image features at the second and third different scales extracted from the backbone network and the features at the fourth scale after feature enhancement, using CSP-MSLFA blocks combined with the PANet structure to supplement semantics and details; The head part obtains the final bounding box and key points by inputting the features at three different scales output by the feature fusion part into the head module; 3) Set the training parameters, including batch size, learning rate, and early stopping strategy, and train the deep convolutional neural network accordingly until the model converges; 4) Input the image containing the person into the trained deep convolutional neural network. The deep convolutional neural network outputs the coordinates of the key points of the person in the image and obtains the corresponding human pose map according to the human body connection relationship of the key points.

[0007] Furthermore, in step 1), the dataset is divided into the training set, validation set, and test set in a ratio of 5:1:4.

[0008] Furthermore, the SAMP-Stem module described in step 2) first uses a step size of 2. The convolutional block will reduce the image size (size is...) The 3-channel image was downsampled to its original state. The channel is adjusted to C; Then use via with The three branches of the convolutional block and SE attention perform efficient feature extraction on the initially downsampled features to achieve multi-path perceptual fusion. The first Sobel branch is used to extract edge information in the image; the second max pooling branch is used to enhance the feature representation of key point regions, thereby extracting salient features; and the third dilated convolutional branch expands the receptive field to obtain richer contextual information. At the same time, each branch has After adjusting the channels in the convolutional blocks for consistency, SE attention is used to adaptively adjust the weights of the feature channels. Then, rich information is obtained through summation and fusion. Convolutional blocks enhance the flow of information, and finally, downsampling using convolutional blocks restores the image size to its original value. The channel has been adjusted to 2C.

[0009] Furthermore, in step 2), the backbone network first extracts features through a CSP-MSLFA module to obtain features at the first scale, at which point the size is the original size. Adjust the channel to 4C; Next, a convolutional block with a kernel size of 3 and a stride of 2 is used for downsampling to reduce the size. This is then fed into a CSP-MSLFA module to obtain the extracted features at the second scale, at which point the size is the same as the original. The channel has been adjusted to 8C; The convolutional block with a stride of 2 for downsampling is combined with the CSP-MSLFA module to extract features at the second scale. This entire set of features at the second scale is then input into this feature extraction combination to obtain features at the third scale, which is now the same size as the original. 1. Adjust the channel to 8C; then input the features from the third scale into the combination of the features extracted above to obtain the features from the fourth scale, at which point the size is the original. And adjust the channel to 8C.

[0010] Furthermore, the CSP-MSLFA module equally segments the input image features into two paths, A and B, along the channels: Branches A and B each retain their original features to obtain an output. Branch B then passes through two sequentially connected MSLFA modules to obtain another output. The feature map of the branch output retaining the original features is then concatenated with the branch output after passing through the first MSLFA module and the output features after passing through the two MSLFA modules. Finally, point convolution is used to achieve information fusion and interaction as well as channel adjustment. The MSLFA module first takes the features from the input B branch, retaining the original features in one path and then using a convolutional block with a kernel of 3 in the other path to obtain the original features with the same number of channels. The output; Next, the features output from branch B are evenly divided into two branches, C and D, according to channels. Branch C retains the original features, while branch D uses 5-group convolutional blocks to obtain the original features. Features; Then, the feature map output from branch D is evenly divided into two branches, E and F, by channel. Branch E retains the original features, while branch F uses grouped convolutional blocks with a kernel of 7 to obtain the original features. The features are then combined, and finally the outputs of C, E, and the F branch after grouped convolution are concatenated for use. The convolutional blocks are fused together; Finally, an addition operation is used to further merge them.

[0011] Furthermore, the feature enhancement module described in step 2) further enhances the semantic features and key information at the fourth scale by sequentially connecting the SMSFE module and the introduced C2PSA module; The SMSFE module extracts global features through dilated convolution and shared weights, expanding the receptive field and enhancing the semantics of the features, while the C2PSA module further focuses on important regions in the image through channel attention mechanism, thereby strengthening the feature representation of key regions. Among them, the SMSFE module first passes through The convolution will compress the channels to their original size. Then, it passes through three convolutional layers with dilation rates of 1, 3, and 5 respectively, thereby expanding the receptive field and extracting global features at multiple scales; at the same time, the three convolutions reduce the number of model parameters by sharing convolutions, thus maintaining the effectiveness of feature extraction.

[0012] Furthermore, in step 2), the feature fusion part inputs the second and third scale image features extracted from the backbone network part and the fourth scale feature map after feature enhancement into the feature fusion module. The semantic and detail fusion is achieved by using the MSLFA module in a top-down and bottom-up manner, and finally three feature maps of different scales are obtained.

[0013] The top-down approach uses upsampling and concatenation combined with the MSLFA module to achieve semantic transfer, while the bottom-up approach uses convolutional blocks with a stride of 2 to achieve downsampling and concatenation.

[0014] Furthermore, the head portion obtains the final bounding box and keypoints by inputting the features at three different scales output from the feature fusion part into the head module; the specific steps are as follows: The feature fusion part outputs three feature maps at different scales, denoted as P2, P3, and P4, corresponding to the detection of small, medium, and large human targets, respectively. These feature maps at different scales are input to the head module. The head module adopts a decoupled structure, passing the input features to three independent prediction branches to achieve mutual independence between tasks and feature optimization. The prediction branch includes a detection branch and a keypoint branch: I) The detection branch establishes a classification tower and a regression tower on each scale feature map respectively; the regression tower is used to predict the spatial distribution of the target boundary at each grid position and outputs the discrete probability distribution of the four boundary distances; it is decoded into continuous boundary distance values ​​by distributed focus regression, and combined with the anchor point coordinates generated by the feature map size and stride, the predicted boundary distances are converted into the absolute coordinates of the candidate boxes using geometric relationships; Meanwhile, the classification tower outputs the human category confidence score at each location, which, after Sigmoid activation, represents the probability of the detected target's existence. The detection results at three scales, P2, P3, and P4, are generated in parallel. Then, overlapping candidate boxes are removed through confidence thresholding and non-maximum suppression to finally obtain high-confidence human detection boxes. II) The keypoint branches are generated on feature maps at various scales. Key point prediction results for the channel, among which K Where D is the number of keypoints, and D is the dimension of each keypoint; the keypoint coordinates are decoded using the following formula:

[0015] in, σ This represents the Sigmoid function, where anchor is the coordinate of the anchor point and stride is the stride of the current feature layer; this process converts relative coordinates into absolute coordinates at the scale of the input image. The keypoint branch output and the target box predicted by the detection branch are independently predicted by the model at multiple scale feature layers and the results are fused. This allows shallow features to focus on detailed textures and deep features to capture global structure. In the post-processing stage, the model first filters low-confidence targets based on a confidence threshold, then uses the NMS algorithm to remove overlapping boxes. Each remaining detection box carries the corresponding coordinates of 14 keypoints and a visibility score. Finally, an affine transformation is used to map the predicted boxes and keypoint coordinates back to the original image size to form the final human pose estimation result.

[0016] Furthermore, in step 3), the deep convolutional neural network model is trained using the PyTorch framework. During the training process, the training model and the validation model results are generated in each iteration, and the deep convolutional neural network is trained until the model converges. During the training of the deep convolutional neural network, the batch size was set to 16, the optimizer used the stochastic gradient descent (SGD) algorithm, and the initial learning rate was set to 1×10⁻⁶. -2 .

[0017] Furthermore, Object Keypoint Similarity (OKS) is used as an evaluation metric to assess the performance of the trained model. The expression is as follows:

[0018] in, This represents the Euclidean distance between the detected keypoints and their corresponding ground truth labels. A visibility indicator that represents the actual annotation; Indicates the scale of an object; This represents a constant at each key point, controlling the decay.

[0019] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art: 1. The deep convolutional neural network constructed in this invention uses the SAMP-Stem module to adopt a multi-path perception method. Through edge extraction, max pooling and dilated convolution branches, it enhances the perception of local details, especially in the case of dense crowds and occlusion, and improves the accuracy of key point localization.

[0020] 2. The deep convolutional neural network of this invention uses the CSP-MSLFA module to combine multi-scale convolution and feature aggregation, which solves the problem of multi-scale target recognition and accurately locates the key points of instances at different scales.

[0021] 3. This invention uses the SMSFE module to improve global context awareness through dilated convolution and shared convolution, thereby increasing robustness to complex backgrounds and improving computational efficiency.

[0022] 4. This invention promotes the overall optimization of the network structure through the collaborative work of SAMP-Stem, CSP-MSLFA and SMSFE modules, which can accurately locate key points and efficiently distinguish individuals and perform accurate pose estimation even in complex scenarios such as pose crossing and occlusion environments. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a diagram of the overall network architecture for attitude estimation in this invention. Figure 2 Here is a diagram of the SAMP-Stem module structure; Figure 3 Here is a structural diagram of the CSP-MSLFA module; Figure 4Here is a structural diagram of the feature enhancement module; Figure 5 A simplified flowchart for attitude estimation; Figure 6 The original image to be processed; Figure 7 The image is processed using the YOLO pose estimation scheme; Figure 8 The image is processed using the technical solution of this invention; Figure 9 This is a comparison of the heatmaps using the YOLO solution and the heatmaps of the present invention. Detailed Implementation

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] See attached document Figure 1 and 5 As shown, this embodiment provides a robust pose estimation method for complex crowd scenarios based on deep learning, from end to end, including: 1) The CrowdPose dataset is used as the data source for model training. The dataset is divided into a training set, a validation set, and a test set. The training set and the validation set are used together for model training, while the test set is used for final model performance evaluation. In this embodiment, the CrowdPose dataset includes 20,000 images, and the dataset is divided into training, validation, and test sets in a 5:1:4 ratio.

[0027] 2) Construct a deep convolutional neural network, including a preprocessing part, a backbone network part, a feature enhancement part, a feature fusion part, and a head part.

[0028] In this embodiment, the preprocessing section is used to process the image data (size is...) The 3-channel image is input into the SAMP-Stem (structure-aware multipath feature extraction module) module for preprocessing, which achieves initial downsampling of the image and adjusts the image size to its original value. The number of channels was adjusted to 2C. The purpose of this preprocessing is to reduce the computational load by reducing the image size and adjusting the number of channels, while preserving shallow features for efficient feature extraction later.

[0029] See attached document Figure 2 As shown, the SAMP-Stem module first uses a step size of 2. Convolutional blocks downsample the image size to its original value. The channel is adjusted to C; then use the channel with... The three branches of the convolutional block and SE attention perform efficient feature extraction on the initially downsampled features to achieve multi-path perception fusion. Specifically, the first Sobel branch extracts edge information from the image, helping to enhance the perception of local details, especially in complex scenes such as crowd occlusion and pose intersections, enabling more effective differentiation of different individuals; the second max-pooling branch enhances the feature representation of key point regions, extracting salient features to enhance the perception of global information; and the third dilated convolution branch expands the receptive field to obtain richer contextual information, effectively mitigating the problem of local information loss.

[0030] At the same time, each branch has After adjusting the channel consistency of the convolutional blocks, SE attention is used to adaptively adjust the weights of the feature channels, thereby enhancing attention to important features while suppressing irrelevant or useless features. Then, rich information is obtained through summation and fusion. Convolutional blocks enhance the flow of information, and finally, downsampling using convolutional blocks restores the image size to its original value. The channel has been adjusted to 2C.

[0031] In this embodiment, the backbone network is responsible for feeding the shallow feature maps from the preprocessing part into the CSP-MSLFA (Multi-Scale Local Feature Aggregation Module) module for feature extraction, which combines the subsampled convolutional blocks to extract more abstract and higher-level features at four scales; the specific steps are as follows: After the preprocessed initial features are input into the backbone network, they first pass through a CSP-MSLFA module to extract features at the first scale (the size is the same as the original). (And adjust the channels to 4C); then use a convolutional block with a kernel size of 3 and a stride of 2 to downsample and reduce the size, and then feed it into a CSP-MSLFA module to obtain the extracted features at the second scale (the size is the original). And adjust the channel to 8C). The convolutional block with a stride of 2 for downsampling is combined with the CSP-MSLFA module to extract features at the second scale. This entire set of features is then used as a feature extraction combination. The features at the second scale are then input into this feature extraction combination, resulting in features at the third scale (with the same size as the original). (And adjust the channel to 8C); then input the features of the third scale into the combination of the above feature extractions to obtain the features of the fourth scale (the size is the original). And adjust the channel to 8C).

[0032] See attached document Figure 3 As shown, the CSP-MSLFA module combines the idea of ​​Cross-Stage Partial (CSP) to equally divide the input image features into two paths, A and B, along the channels. Branches A and B each retain their original features and produce an output. Branch B then passes through two sequentially connected MSLFA modules to produce another output. The feature map output from the branch that retains the original features is then concatenated with the output from the branch that passed through the first MSLFA module and the output features from the two MSLFA modules. Finally, point convolution is used to achieve information fusion and channel adjustment, thereby realizing cross-stage information transfer and fusion to improve the efficiency and performance of the network.

[0033] The MSLFA module utilizes multi-scale convolutions (such as 3×3, 5×5, 7×7), grouped convolutions, and cross-scale concatenation fusion. It first takes the features from the input B branch, retaining the original features in one path and then using a convolutional block with a 3-kernel to obtain the original features in the other. The output of branch B is then divided into two branches, C and D, with the features from branch B evenly divided by channels. Branch C retains the original features, while branch D uses 5-group convolutional blocks with 5 kernels to obtain the original features. The features are then divided into two branches, E and F, based on the number of channels. Branch E retains the original features, while branch F uses a grouped convolutional block with a kernel of 7 to obtain the features with the original number of channels. The features, C and E, and the output of the F branch after grouped convolution are concatenated for use. The convolutional blocks are fused, and then further fused using an addition operation, effectively handling multi-scale target problems and preserving more detailed information through skip connections. Through this process, the network can handle multi-scale features, especially when dealing with keypoints of different scales such as the head, hands, and knees, exhibiting stronger robustness and higher accuracy.

[0034] See attached document Figure 4As shown, the feature enhancement section improves the high-level features (features at the fourth scale) extracted from the backbone network through a feature enhancement module, particularly enhancing key region information in complex backgrounds of the image. To this end, this module further enhances the semantic features and key information at the fourth scale by sequentially connecting the Multi-Scale Feature Extraction (SMSFE) module and the introduced C2PSA module. Specifically, the SMSFE module extracts global features through dilated convolution and shared weights, expanding the receptive field and enhancing the semantics of the features, which helps to further capture image information at different scales; while the C2PSA module, through a channel attention mechanism, further focuses on important regions in the image, thereby strengthening the feature representation of key regions (at this point, the size is the same as the original). (Channel 8C).

[0035] Among them, the SMSFE module first passes through The convolution compresses the channels back to their original size. Then, three convolutional layers with dilation rates of 1, 3, and 5 are sequentially applied to expand the receptive field and achieve global feature extraction at multiple scales. Simultaneously, to reduce the number of parameters, the three convolutions share a common convolutional layer, ensuring the fusion of features at different scales. This makes the feature extraction process more efficient, reduces the number of model parameters, and maintains the effectiveness of feature extraction. The feature fusion (Neck) section fuses the image features at the second and third scales extracted from the backbone network, along with the last feature after feature enhancement. Specifically, the image features at the second and third scales extracted from the backbone network, along with the feature map at the fourth scale after feature enhancement, are input into the feature fusion module. The module uses the MSLFA module to achieve semantic and detail fusion through both top-down (using upsampling and concatenation) and bottom-up (using convolutional blocks with a stride of 2 for downsampling and concatenation) approaches, ultimately resulting in three feature maps at different scales.

[0036] The head section obtains the final bounding box and key points by inputting the features at three different scales output from the feature fusion part into the head module; the specific steps are as follows: The feature fusion part outputs three feature maps at different scales, denoted as P2, P3, and P4, corresponding to small, medium, and large-sized human targets, respectively. In the pose estimation evaluation of the CrowdPose dataset, the small, medium, and large sizes of human instances are defined based on the area of ​​their bounding boxes, completely following the COCO keypoint detection standard: small size corresponds to a bounding box area of ​​less than 32. 2 pixels, i.e., less than 1024 pixels 2Medium size corresponds to an area greater than 32. 2 pixels and less than 96 2 pixels, i.e., more than 1024 pixels 2 And less than 9216 pixels 2 Large size corresponds to an area greater than or equal to 96. 2 pixels, that is, 9216 pixels or more 2 For example, in an image of a crowded scene, a small, standing figure in the background might have a bounding box of 28 x 36 pixels, with an area of ​​approximately 1008 pixels. 2 It is a small-sized object; a person running in the mid-ground has a bounding box of approximately 75 x 110 pixels, with an area of ​​approximately 8250 pixels. 2 The size is medium; while the large figures with clearly visible foregrounds have a bounding box of 200×300 pixels, covering an area of ​​60,000 pixels. 2 Then it belongs to the large size category.

[0037] These feature maps at different scales are processed by convolutional layers and then input into the head module. The head module adopts a decoupled structure, which feeds the input features into three independent prediction branches to achieve mutual independence between tasks and feature optimization. The prediction branch includes a detection branch and a keypoint branch: I) The detection branch builds classification towers and regression towers on feature maps at each scale. The regression tower is used to predict the spatial distribution of the target boundary at each grid position, and outputs the discrete probability distribution of the four boundary distances (left, top, right, bottom). It is decoded into continuous boundary distance values ​​by using distributed focal loss (DFL). Combined with the anchor coordinates generated by the feature map size and stride, the predicted boundary distances are converted into the absolute coordinates of the candidate boxes using geometric relationships.

[0038] Meanwhile, the classification tower outputs the human category confidence score at each location, which, after being activated by the Sigmoid function, represents the probability of the detected target's presence. The detection results at the three scales (P2, P3, P4) are generated in parallel, and then overlapping candidate boxes are removed through confidence thresholding and non-maximum suppression (NMS) to finally obtain high-confidence human detection boxes.

[0039] II) Keypoint branches are generated on feature maps at various scales. Key point prediction results for the channel, among which K Where D is the number of keypoints (e.g., 17 keypoints according to the COCO standard), and D is the dimension of each keypoint (usually 3, corresponding to the x, y coordinates and visibility score); keypoint coordinates are decoded using the following formula:

[0040] in, σ This represents the Sigmoid function, where anchor is the coordinate of the anchor point and stride is the stride of the current feature layer. This process converts relative coordinates into absolute coordinates at the scale of the input image. When D=3, the third dimension outputs the visibility confidence of the keypoints via the Sigmoid function.

[0041] The output of the keypoint branch corresponds to the bounding box predicted by the detection branch. Each detection instance contains a complete set of keypoints, achieving end-to-end pose estimation. To improve robustness in complex crowd scenes, the output of the keypoint branch and the bounding box predicted by the detection branch independently predict keypoints and fuse the results at multi-scale feature layers. This allows shallow features to focus on detailed textures, while deep features capture the global structure.

[0042] In the post-processing stage, the model first filters low-confidence targets based on a confidence threshold, then uses the NMS algorithm to remove overlapping boxes. Each remaining detection box carries the corresponding coordinates of 14 key points and a visibility score. Finally, the predicted boxes and key point coordinates are mapped back to the original image size through an affine transformation to form the final human pose estimation result.

[0043] In summary, the head module achieves high-precision, real-time human pose estimation in complex crowd scenarios by performing target box regression and keypoint regression in parallel on multi-scale features, combined with strategies such as distributed regression, anchor point geometric decoding, multi-scale fusion, and non-maximum suppression.

[0044] 3) Set the training parameters, including batch size, learning rate, and early stopping strategy, and train the deep convolutional neural network accordingly until the model converges.

[0045] In this embodiment, a deep convolutional neural network model is trained using the PyTorch framework. During each iteration of the training process, training and validation model results are generated, and the deep convolutional neural network is trained until convergence. During the training of the deep convolutional neural network, the batch size is set to 16, the optimizer uses the stochastic gradient descent (SGD) algorithm, and the initial learning rate is set to 1×10⁻⁶. -2 .

[0046] Object keypoint similarity (OKS) is used as the evaluation metric to assess the performance of the trained model. The expression is as follows:

[0047] in, This represents the Euclidean distance between the detected keypoints and their corresponding ground truth labels. A visibility indicator that represents the actual annotation; Indicates the scale of an object; This represents a constant for each key point, controlling the decay, and is typically set to [0.025, 0.107].

[0048] 4) Input the image containing the person into the trained deep convolutional neural network. The deep convolutional neural network outputs the coordinates of the key points of the person in the image and obtains the corresponding human pose map according to the human body connection relationship of the key points.

[0049] Performance testing: In this experiment, the test set of the CrowdPose dataset was selected to systematically compare and evaluate the performance of the pose estimation method based on YOLOv11 and the method proposed in this invention. The testing procedure is as follows: First, the pre-divided test set in the dataset was used as a unified evaluation sample to ensure that all methods ran under identical input conditions. For the YOLO-based approach, the same strategy was used to train the model and obtain the weight model, which was then loaded to obtain the test set and keypoint prediction results. In the method of this invention, the pose estimation module was optimized into a more tightly integrated structure by utilizing improved feature extraction and optimized overall structure, thereby reducing the accumulation of errors between detection and pose estimation. The weights trained by the method of this invention were loaded, and the keypoint prediction file was output. Subsequently, the same OKS metric was used to calculate AP and AP11 using the CrowdPose evaluation tool. 50 The performance of the two methods was quantitatively compared using multiple metrics (as shown in Table 1 below).

[0050] Table 1 Results on the CrowdPose test set

[0051] Wherein, AP (represents the mean of the average precision values ​​at 10 locations with OKS thresholds ranging from 0.50 to 0.95 (intervals of 0.05), comprehensively evaluating the model's performance under different levels of stringency) 50 (This represents the average accuracy when the OKS threshold is 0.50, measuring the model's detection performance under more lenient matching requirements.) AP 75(This represents the average accuracy when the OKS threshold is 0.75, measuring the model's detection performance under strict matching requirements); To better measure the model's performance in different application scenarios, CrowdPose also introduces the concept of crowd density, and divides images into three categories based on the range of crowd density: Easy (Crowd Index = 0~0.1), Medium (Crowd Index = 0.1~0.8), and Difficult (Crowd Index = 0.8~1.0), corresponding to AP respectively. Easy AP Medium and AP Hard .

[0052] To measure the degree of crowding in an image, a Crowd Index is defined. While the number of people in an image intuitively seems like a good indicator of crowding, the main challenge in crowded scenes is not the sheer number of people, but rather the occlusion problem between them. Therefore, the Crowd Index is used to represent the degree of crowding in an image. Within the bounding box of an individual instance, the elements belonging to that instance will be... The number of key points for an individual (not others) is denoted as The number of key points belonging to others is recorded as , That is, the first The crowd ratio of an individual instance. The Crowd Index is obtained by averaging the crowd ratios of all individuals in the image.

[0053] in, This represents the total number of people in the image.

[0054] As can be seen from the data in Table 1, the method of this invention outperforms the YOLO-based approach in all relevant AP metrics, especially in high-density populations (AP). Hard The advantages are even more pronounced in crowded scenarios. This indicates that the method of the present invention can perform pose estimation tasks more accurately and stably in crowded environments.

[0055] See attached document Figures 6-8 As shown in the visualization, the proposed method exhibits better robustness to background noise, better differentiation of multiple human bodies, and more accurate and human-structure-compliant prediction of key point locations. Comprehensive analysis reveals that the network structure of this invention achieves excellent performance.

[0056] See attached document Figure 9As shown in the heatmap comparison, the heatmap distribution of the YOLOv11 solution is concentrated in one area, while the heatmap of the present invention is evenly distributed across each person. This indicates that the present invention has better attention to multiple people, is more robust to background noise, and has a better overall effect.

[0057] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A robust pose estimation method for complex crowd scenarios based on deep learning, characterized in that, Includes the following steps: 1) Use the CrowdPose dataset as the data source for model training. Divide the dataset into training set, validation set and test set. Use the training set and validation set together for model training, and use the test set for final model performance evaluation. 2) Construct a deep convolutional neural network, including a preprocessing part, a backbone network part, a feature enhancement part, a feature fusion part, and a head part; The preprocessing section is used to process materials with a size of [missing information]. The 3-channel image data is input into the SAMP-Stem module for preprocessing, which achieves initial downsampling of the image, reduces the image size and adjusts the number of channels, while preserving shallow features for subsequent efficient feature extraction. The backbone network is used to feed the shallow feature maps passed from the preprocessing part into the CSP-MSLFA module for feature extraction, and combine them with downsampled convolutional blocks to extract four more abstract and higher-level features at different scales. The feature enhancement part improves the semantics of high-level features extracted from the backbone network through the SMSFE module in the feature enhancement module, and combines the C2PSA attention module to enhance the features of key regions to obtain features at the fourth scale. The feature fusion part achieves semantic and detail enhancement by fusing the image features at the second and third different scales extracted from the backbone network and the features at the fourth scale after feature enhancement using CSP-MSLFA blocks combined with the PANet structure. The head part obtains the final bounding box and key points by inputting the features at three different scales output by the feature fusion part into the head module; 3) Set the training parameters, including batch size, learning rate, and early stopping strategy, and train the deep convolutional neural network accordingly until the model converges; 4) Input the image containing the person into the trained deep convolutional neural network. The deep convolutional neural network outputs the coordinates of the key points of the person in the image and obtains the corresponding human pose map according to the human body connection relationship of the key points.

2. The robust pose estimation method for complex crowd scenarios based on deep learning in end-to-end application as described in claim 1, characterized in that, In step 1), the dataset is divided into the training set, validation set, and test set in a ratio of 5:1:

4.

3. The robust pose estimation method for complex crowd scenarios based on deep learning in end-to-end application as described in claim 1, characterized in that, The SAMP-Stem module described in step 2) first uses a step size of 2. Convolutional blocks downsample the image size to its original value. The channel is adjusted to C; Then use via with The three branches of the convolutional block and SE attention perform efficient feature extraction on the initially downsampled features to achieve multi-path perceptual fusion. The first Sobel branch is used to extract edge information in the image; the second max pooling branch is used to enhance the feature representation of key point regions, thereby extracting salient features; and the third dilated convolutional branch expands the receptive field to obtain richer contextual information. At the same time, each branch has After adjusting the channels in the convolutional blocks for consistency, SE attention is used to adaptively adjust the weights of the feature channels. Then, rich information is obtained through summation and fusion. Convolutional blocks enhance the flow of information, and finally, downsampling using convolutional blocks restores the image size to its original value. The channel has been adjusted to 2C.

4. The robust pose estimation method for complex crowd scenarios based on deep learning in end-to-end application as described in claim 1, characterized in that, In step 2), the backbone network first extracts features through a CSP-MSLFA module to obtain features at the first scale, at which point the size is the same as the original. Adjust the channel to 4C; Next, a convolutional block with a kernel size of 3 and a stride of 2 is used for downsampling to reduce the size. This is then fed into a CSP-MSLFA module to obtain the extracted features at the second scale, at which point the size is the same as the original. The channel has been adjusted to 8C; The convolutional block with a stride of 2 for downsampling is combined with the CSP-MSLFA module to extract features at the second scale. This entire set of features at the second scale is then input into this feature extraction combination to obtain features at the third scale, with the size remaining the same as the original. Then adjust the channel to 8C; then input the features of the third scale into the combination of the above feature extractions to obtain the features of the fourth scale, at which point the size is the original. Adjust the channel to 8C.

5. The robust pose estimation method for complex crowd scenarios based on deep learning in end-to-end application as described in claim 4, characterized in that, The CSP-MSLFA module equally divides the input image features into two paths, A and B, along the channels: Branches A and B each retain their original features to obtain an output. Branch B then passes through two sequentially connected MSLFA modules to obtain another output. The feature map of the branch output retaining the original features is then concatenated with the branch output after passing through the first MSLFA module and the output features after passing through the two MSLFA modules. Finally, point convolution is used to achieve information fusion and interaction as well as channel adjustment. The MSLFA module first takes the features from the input B branch, retains the original features in one path, and then uses a convolutional block with a kernel of 3 to obtain the channels that are the original features in the other path. The output; Next, the features output from branch B are evenly divided into two branches, C and D, according to channels. Branch C retains the original features, while branch D uses 5-group convolutional blocks to obtain the original features. Features; Then, the feature map output from branch D is evenly divided into two branches, E and F, by channel. Branch E retains the original features, while branch F uses grouped convolutional blocks with a kernel of 7 to obtain the original features. Features are then used to concatenate the outputs of C and E, as well as the output of the F branch after grouped convolution. The convolutional blocks are fused together; Finally, an addition operation is used to further merge them.

6. The robust pose estimation method for complex crowd scenarios based on deep learning in end-to-end application as described in claim 1, characterized in that, The feature enhancement module described in step 2) further enhances the semantic features and key information at the fourth scale by sequentially connecting the SMSFE module and the introduced C2PSA module; The SMSFE module extracts global features through dilated convolution and shared weights, expanding the receptive field and enhancing the semantics of the features, while the C2PSA module further focuses on important regions in the image through channel attention mechanism, thereby strengthening the feature representation of key regions. Among them, the SMSFE module first passes through The convolution compresses the channels to their original size. Then, the model is passed through three convolutional layers with dilation rates of 1, 3, and 5, respectively, to expand the receptive field and achieve global feature extraction at multiple scales. At the same time, the three convolutions reduce the number of model parameters by sharing convolutions, while maintaining the effectiveness of feature extraction.

7. The robust pose estimation method for complex crowd scenarios based on deep learning in end-to-end application as described in claim 1, characterized in that, In step 2), the feature fusion part inputs the second and third scale image features extracted from the backbone network part and the fourth scale feature map after feature enhancement into the feature fusion module. The semantic and detail fusion is achieved by using the MSLFA module in a top-down and bottom-up manner, and finally three feature maps of different scales are obtained. The top-down approach uses upsampling and concatenation combined with the MSLFA module to achieve semantic transfer, while the bottom-up approach uses convolutional blocks with a stride of 2 to achieve downsampling and concatenation.

8. The robust pose estimation method for complex crowd scenarios based on deep learning in end-to-end application as described in claim 1, characterized in that, The final bounding box and keypoints are obtained by inputting the features at three different scales output from the feature fusion part into the head module; the specific steps are as follows: The feature fusion section outputs three feature maps at different scales, denoted as P2, P3, and P4, corresponding to the detection of small, medium, and large human targets. These feature maps at different scales are input into the head module. The head module adopts a decoupled structure, which feeds the input features into three independent prediction branches to achieve mutual independence between tasks and feature optimization. The prediction branch includes a detection branch and a keypoint branch: I) The detection branch establishes a classification tower and a regression tower on each scale feature map respectively; the regression tower is used to predict the spatial distribution of the target boundary at each grid position and outputs the discrete probability distribution of the four boundary distances; it is decoded into continuous boundary distance values ​​by distributed focus regression, and combined with the anchor point coordinates generated by the feature map size and stride, the predicted boundary distances are converted into the absolute coordinates of the candidate boxes using geometric relationships; Meanwhile, the classification tower outputs the human category confidence score at each location, which, after being activated by the Sigmoid, represents the probability of the detected target's presence. The detection results of three scales, P2, P3, and P4, are generated in parallel. Then, overlapping candidate boxes are removed by confidence thresholding and non-maximum suppression to finally obtain high-confidence human detection boxes. II) The keypoint branches are generated on feature maps at various scales. Key point prediction results for the channel, among which K Where D is the number of keypoints, and D is the dimension of each keypoint; the keypoint coordinates are decoded using the following formula: in, σ This represents the Sigmoid function, where anchor is the coordinate of the anchor point and stride is the stride of the current feature layer; this process converts relative coordinates into absolute coordinates at the scale of the input image. The keypoint branch output and the target box predicted by the detection branch are independently predicted by the model at multiple scale feature layers and the results are fused. This allows shallow features to focus on detailed textures and deep features to capture global structure. In the post-processing stage, the model first filters low-confidence targets based on a confidence threshold, then uses the NMS algorithm to remove overlapping boxes. Each remaining detection box carries the corresponding coordinates of 14 keypoints and a visibility score. Finally, an affine transformation is used to map the predicted boxes and keypoint coordinates back to the original image size to form the final human pose estimation result.

9. The robust pose estimation method for complex crowd scenarios based on deep learning in end-to-end application as described in claim 1, characterized in that, In step 3), the deep convolutional neural network model is trained using the PyTorch framework. During the training process, the training model and the validation model results are generated in each iteration. The deep convolutional neural network is trained until the model converges. During the training of the deep convolutional neural network, the batch size was set to 16, the optimizer used the stochastic gradient descent algorithm, and the initial learning rate was set to 1×10⁻⁶. -2 .

10. The robust pose estimation method for complex crowd scenarios based on deep learning in end-to-end application as described in claim 9, characterized in that, Object keypoint similarity (OKS) is used as the evaluation metric to assess the performance of the trained model. The expression is as follows: in, This represents the Euclidean distance between the detected keypoints and their corresponding ground truth labels. A visibility indicator that represents the actual annotation; Indicates the scale of an object; This represents a constant at each key point, controlling the decay.

Images