Congested multi-person attitude estimation method based on multi-attention mechanism HRNet
By introducing a multi-attention mechanism module into the HRNet model, the problem of traditional multi-person pose estimation method accurately identifying individual key points in complex and crowded environments is solved, and efficient and accurate pose estimation effect is achieved.
Patent Information
- Application Number
- CN202411957593.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-29
AI Technical Summary
Traditional multi-person pose estimation methods are difficult to accurately identify individual key points in complex and crowded environments, especially in occlusion, resulting in a decrease in the accuracy of pose estimation.
The HRNet model based on multi-attention mechanism is adopted, and the channel attention mechanism module and cross-attention module are introduced to enhance feature representation capabilities and global feature correlations, and the detection accuracy of the model in crowded scenarios is improved.
The parameter amount of the model is effectively reduced, and the detection accuracy is improved under occlusion, achieving accurate recognition of postures in complex environments, demonstrating excellent robustness.
Smart Images
Figure CN119904892A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and specifically relates to a crowded multi-person posture estimation method based on multi-attention mechanism HRNet. Background Art
[0002] Multi-person pose estimation is an important research direction in the field of computer vision, which aims to identify and locate the key points (such as head, wrist, knee, etc.) of multiple individuals in an image or video. This technology is widely used in scenarios such as sports analysis, intelligent monitoring, human-computer interaction, and augmented reality. With the development of deep learning, especially the introduction of convolutional neural networks, the accuracy and real-time performance of pose estimation have been significantly improved. Researchers have proposed a variety of algorithms and models that use contextual information in images and spatial relationships between individuals to optimize the detection and connection of key points.
[0003] The challenges faced by traditional multi-person pose estimation methods mainly include occlusion and interaction between individuals. These problems make it more difficult to accurately estimate the pose of each person in a complex and crowded environment. When the key points of an individual (such as joint positions) are occluded by others, it is difficult for the detection algorithm to accurately identify them, resulting in a decrease in the accuracy of pose estimation. To improve the performance of multi-person pose estimation, two issues need to be addressed: one is to reduce the number of model parameters to improve computational efficiency; the other is to improve the detection accuracy in the case of occlusion, so that the pose can still be accurately identified in a complex environment.
[0004] To address these challenges, deep learning can be used to learn features in the data, enabling the model to extract richer information and thus achieve higher accuracy in crowded scenes. Summary of the invention
[0005] This application provides a crowded multi-person pose estimation method based on the multi-attention mechanism HRNet. The core advantage of HRNet (High Resolution Net) is to maintain high-resolution feature maps and fuse these features between different levels, which enables it to effectively identify small-sized and detailed joints and parts in the image to solve the above technical problems.
[0006] To solve the above technical problems, a technical solution adopted in this application is: a crowded multi-person posture estimation method based on multi-attention mechanism HRNet, comprising the following steps:
[0007] Step S1. Collect human body images and convert them into a standard data set format to obtain a first data set;
[0008] Step S2. Add a multi-attention mechanism module based on the HRNet (High ResolutionNet) model to build an improved HRNet model;
[0009] Step S3. Based on the first data set, training an improved HRNet model;
[0010] Step S4. Use the trained improved HRNet model to test the crowded multi-person scene.
[0011] Further, the method of step S2 includes:
[0012] Step S21. Introduce a channel attention mechanism module based on the bottleneck module in the HRNet model, perform preliminary feature extraction on the images in the first data set, and obtain feature images;
[0013] Step S22. Based on the parallel structure and feature images in the HRNet model, generate branch feature images of different resolutions while retaining the high-resolution feature image;
[0014] Step S23. Perform multi-scale feature fusion on the low-resolution feature image and introduce a cross-attention mechanism module to encode one-dimensional features in two spatial directions in parallel.
[0015] Further, the method of step S21 includes:
[0016] The HRNet model includes a bottleneck module and a basic module; the bottleneck module includes a 1×1 convolution layer for reducing the number of channels, a 3×3 convolution layer for feature extraction, and a 1×1 convolution layer for restoring the number of channels;
[0017] The 3×3 convolutional layer in the bottleneck module is replaced by a 3×3 separation convolutional layer and the channel attention mechanism module is introduced to obtain the channel attention bottleneck module;
[0018] Based on the channel attention bottleneck module, feature images are obtained.
[0019] Further, based on formula (1), the channel attention mechanism module is obtained; wherein formula (1) is:
[0020]
[0021] Where σ is the activation function, is the aggregated feature without dimensionality reduction, w i Represents y i The channel attention, w j Represents y i The channel attention weight of k represents the coverage of local cross-channel interactions, and Represents y i The set of k adjacent channels.
[0022] Further, the method for obtaining the channel attention bottleneck module includes:
[0023] Based on formula (2), a one-dimensional global channel tensor is obtained; wherein formula (2) is:
[0024]
[0025] where F in represents the input feature image, GAP(.) represents the global average pooling function, X c (i, j) represents the value of the cth channel at position (i, j), W and H are the width and height of the spatial size;
[0026] Based on formula (3), the channel attention bottleneck module is obtained; where formula (3) is:
[0027] output = σ(Conv1D K (g c )) (3)
[0028] Among them, σ represents the activation function, ConvlD K It is a one-dimensional convolution, k represents the coverage of local cross-channel interaction; after batch normalization and activation, the 1×1 convolution layer is used to extract different range features of different input feature images and learn the importance of different channels.
[0029] Further, the method of step S23 includes:
[0030] Step S231. Based on the cross attention module, the 1×1 branch feature image encodes each channel in the horizontal and vertical directions to obtain a feature image containing spatial direction information;
[0031] Step S232. Based on the pooling kernel, encode each channel in the horizontal and vertical directions to obtain branch feature images of different resolutions.
[0032] Further, the method of step S232 includes:
[0033] Based on formulas (4)-(5), the output of the channel at the height and width of the feature image is obtained; where formulas (4)-(5) are:
[0034]
[0035] where z c (.) is the output associated with the cth channel, W is the width of the spatial dimension H×W, H is the height of the spatial dimension H×W, x c (.) represents the value of the cth channel position, G represents the G sub-features into which the input is divided and the G sub-features are much smaller than the number of c channels;
[0036] Based on formula (6), the nonlinear output is obtained; wherein formula (6) is:
[0037]
[0038] in, represents the connection operation along the spatial dimension, δ is the nonlinear activation function, and Conv is the convolution operation;
[0039] Based on formula (7), the feature image is divided into two independent tensors in the horizontal and vertical directions respectively; wherein formula (7) is:
[0040]
[0041] Where σ is the sigmoid function, G h and G w Represents the horizontal and vertical tensors f h and f w After convolution, it is transformed into a tensor with the same number of channels as the input;
[0042] Based on formula (8), the output of the 1×1 branch feature image is obtained; wherein formula (8) is:
[0043]
[0044] where O(i,j) is the output of the 1×1 branch at position (i,j), G h and G w The table is expanded and used as attention weights respectively; x(.) is the original input, is matrix multiplication.
[0045] The beneficial effects of the present application are as follows: the channel attention bottleneck module of the present application enhances the representation capability of features by utilizing the interdependence of feature relationships between channels, enabling the model to pay more attention to the main features and improving the exchange capability of local information. The cross attention module encodes channel dimension relationships and long-range dependencies, captures directional perception and position-sensitive information, and thus enhances the relevance of global feature information. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 It is a flowchart of an embodiment of a crowded multi-person posture estimation method based on a multi-attention mechanism HRNet of the present application;
[0047] Figure 2 yes Figure 1 A schematic flow chart of an embodiment of step S2 in FIG.
[0048] Figure 31 is a schematic diagram of the overall structure of a network of an embodiment of a crowded multi-person posture estimation method based on a multi-attention mechanism HRNet of the present application;
[0049] Figure 4 It is a structural diagram of a channel attention bottleneck module of an embodiment of a crowded multi-person posture estimation method based on a multi-attention mechanism HRNet of the present application;
[0050] Figure 5 It is a structural diagram of a cross-attention module of an embodiment of a crowded multi-person posture estimation method based on a multi-attention mechanism HRNet of the present application;
[0051] Figure 6 This is a first experimental result diagram of an embodiment of a crowded multi-person posture estimation method based on a multi-attention mechanism HRNet of the present application;
[0052] Figure 7 This is a second experimental result diagram of an embodiment of a crowded multi-person posture estimation method based on a multi-attention mechanism HRNet of the present application;
[0053] Figure 8 This is the third experimental result diagram of an embodiment of a crowded multi-person posture estimation method based on multi-attention mechanism HRNet of the present application. DETAILED DESCRIPTION
[0054] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific embodiments.
[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.
[0056] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a crowded multi-person posture estimation method based on a multi-attention mechanism High ResolutionNet (HRNet) of the present application. The method comprises:
[0057] Step S1. Collect human body images and convert them into a standard data set format to obtain a first data set.
[0058] Step S2. Add a multi-attention mechanism module based on the HRNet model to build an improved HRNet model.
[0059] For details, see Figure 2-3 , step S2 comprises:
[0060] Step S21. Introduce a channel attention mechanism module based on the bottleneck module in the HRNet model, perform preliminary feature extraction on the images in the first data set, and obtain feature images.
[0061] Specifically, the traditional HRNet model is mainly composed of bottleneck modules and base modules, where the bottleneck module contains three convolutional layers: first a 1×1 convolutional layer for reducing the number of channels, followed by a 3×3 convolutional layer for feature extraction, and finally another 1×1 convolutional layer for restoring the number of channels. The base module consists of two consecutive 3×3 convolutional layers.
[0062] See also Figure 4 ,In the early stage of feature extraction, the HRNet model is improved by replacing the traditional 3×3 convolution layer with a 3×3 depthwise separable convolution layer to reduce the repetition of complex modules. At the same time, a channel attention mechanism is introduced to ensure the prediction accuracy of the lightweight network.
[0063] In this process, the redesigned channel attention bottleneck module combines depth-separable convolution and channel attention to effectively capture cross-channel information without reducing the dimension. In addition, by considering each channel and its adjacent channels, the model performance is further improved by sharing the same learning parameters between all channels, which can be expressed as:
[0064]
[0065] Where σ is the activation function, is the aggregated feature without dimensionality reduction, w i Represents y i The channel attention, w j Represents y i The channel attention weight of k represents the coverage of local cross-channel interactions, and Represents y i The set of k adjacent channels.
[0066] First, the extracted convolutional features are converted into a one-dimensional global channel tensor g through a global average pooling operation. c :
[0067]
[0068] where F in represents the input feature image, GAP(.) represents the global average pooling function, X c (i, j) represents the value of the cth channel at position (i, j), W and H are the width and height of the spatial size. The channel attention bottleneck module can be expressed as:
[0069] output = σ(Conv1D K (g c )) (3)
[0070] Among them, σ represents the activation function, ConvlD K It is a one-dimensional convolution, k represents the coverage of local cross-channel interaction; after batch normalization and activation, the 1×1 convolution layer is used to extract different range features of different input feature images and learn the importance of different channels.
[0071] Step S22. Based on the parallel structure and feature images in the HRNet model, generate branch feature images of different resolutions while retaining the high-resolution feature images.
[0072] Step S23. Perform multi-scale feature fusion on the low-resolution feature image and introduce a cross-attention mechanism module to encode one-dimensional features in two spatial directions in parallel.
[0073] For details, see Figure 5 , step S23 comprises:
[0074] Step S231. Based on the cross attention module, the 1×1 branch feature image encodes each channel in the horizontal and vertical directions to obtain a feature image containing spatial direction information.
[0075] Specifically, the criss-cross attention module captures direction-aware and position-sensitive information by encoding channel-dimensional relationships and long-range dependencies. For a given input X∈R C×H×W , the cross-attention module encodes each channel of the 1×1 branch feature image in the horizontal and vertical directions to obtain a feature map containing spatial direction information, so that it can capture long-range dependencies in one spatial direction while accurately retaining the position information in the other direction to compensate for the loss of important position information during feature fusion. In addition, the module places a 3×1 convolution kernel in parallel on the 1×1 branch to aggregate multi-scale spatial structure information.
[0076] Step S232. Based on the pooling kernel, encode each channel in the horizontal and vertical directions to obtain branch feature images of different resolutions.
[0077] Specifically, in order to further enrich the feature aggregation, the cross-attention module also aggregates the results of the 1×1 branch and the 3×3 branch across space. For the 1×1 branch, two pooling kernels of size (1, H) and (1, W) are used to encode each channel in the horizontal and vertical directions. The output of c channels at height h and the output of width w It can be expressed as:
[0078]
[0079] Among them, z c (.) is the output associated with the cth channel, W is the width of the spatial dimension H×W, H is the height of the spatial dimension H×W, x c (.) represents the value of the cth channel position, G represents the G sub-features into which the input is divided, and the G sub-features are much smaller than the number of c channels. This step enables the module to capture long-term dependencies in spatial directions while retaining position information in another direction, thereby more accurately locating the target of interest. The aggregated features are generated by the equation and then input into the shared 1×1 convolution transformation function. Therefore, the nonlinear output can be expressed as:
[0080]
[0081] in, represents the connection operation along the spatial dimension, δ is a nonlinear activation function, and Conv is a convolution operation; the next step is to split the feature image f into two independent tensors in the horizontal and vertical directions, which can be expressed as:
[0082]
[0083] Where σ is the sigmoid function, G h and G w Represents the horizontal and vertical tensors f h and f w After convolution, it is transformed into a tensor with the same number of channels as the input; then the output G h and G w are expanded and used as attention weights respectively. Finally, the output of the 1×1 branch can be written as:
[0084]
[0085] where O(i,j) is the output of the 1×1 branch at position (i,j), G h and G w The table is expanded and used as attention weights respectively; x(.) is the original input, is matrix multiplication.
[0086] On the other hand, the global spatial information of the 1×1 branch output is encoded using a two-dimensional global average pooling operation. The nonlinear function softmax is used at the two-dimensional global output, and then multiplied with the unactivated result in the 3×3 branch. Finally, the output is converted into the corresponding dimensional shape, expressed as:
[0087]
[0088] Similarly, the global spatial information is encoded in the 3×3 branch through a 2D global average pooling operation and directly converted to the corresponding dimensional shape before the 1×1 branch is activated, expressed as:
[0090] Finally, the output of the group is calculated as the aggregation of the two generated spatial attention weight values, and then the result is output through the sigmoid function. The size of the final output is the same as the input X∈R C×H×W Be consistent.
[0091] Step S3: Based on the first data set, train an improved HRNet model.
[0092] Specifically, the improved HRNet model is trained, the weight decay rate is set to 0.05, the initial learning rate is 0.001, and the batch size is set to 32. The experimental results show that using the basic high-resolution network requires 2.9×10 7 parameters and 7.1 Gflops achieved an AP of 73.4%, while the improved high-resolution network can be used with only 9.3×10 6 The network has 2 parameters and 1.58 Gigaflops, achieving an AP of 72.6%, which means that only 32% of the parameters are needed to achieve similar accuracy to the original network, reflecting its balance in efficiency.
[0093] Step S4. Use the trained improved HRNet model to test the crowded multi-person scene.
[0094] See also Figure 6-8 , Figure 6-8 This is an example experimental result diagram of the present invention. It can be seen that a relatively accurate estimation result can be achieved for situations such as occlusion by a crowded crowd.
[0095] This application adopts a lightweight network based on a multi-attention mechanism, which effectively solves the problem of pursuing high precision while ignoring the reduction of parameters. The channel attention bottleneck module enhances the representation ability of features by utilizing the interdependence of feature relationships between channels, allowing the model to pay more attention to the main features and improve the exchange ability of local information. The cross-attention module encodes channel dimension relationships and long-range dependencies, captures directional perception and position-sensitive information, and thus enhances the relevance of global feature information. The organic combination of the two improves the efficiency of the model in posture estimation, and can achieve good detection results using only 32% of the parameters, while also showing excellent robustness in crowded scenes.
[0096] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A crowded multi-person posture estimation method based on multi-attention mechanism HRNet, characterized in that: The following steps are involved: Step S1. Collect human body images and convert them into a standard data set format to obtain a first data set; Step S2. Add a multi-attention mechanism module based on the HRNet model to build an improved HRNet model; Step S3. Based on the first data set, training the improved HRNet model; Step S4: Use the trained improved HRNet model to test a crowded multi-person scene.
2. The method according to claim 1, characterized in that: The method of step S2 comprises: Step S21. Introduce a channel attention mechanism module based on the bottleneck module in the HRNet model, perform preliminary feature extraction on the images in the first data set, and obtain feature images; Step S22. Based on the parallel structure in the HRNet model and the feature image, generate branch feature images of different resolutions while retaining the high-resolution feature image; Step S23. Perform multi-scale feature fusion on the low-resolution feature image and introduce a cross-attention mechanism module to encode one-dimensional features in two spatial directions in parallel.
3. The method according to claim 2, characterized in that The method of step S21 comprises: The HRNet model includes a bottleneck module and a basic module; the bottleneck module includes a 1×1 convolution layer for reducing the number of channels, a 3×3 convolution layer for feature extraction, and a 1×1 convolution layer for restoring the number of channels; The 3×3 convolution layer in the bottleneck module is replaced by a 3×3 separation convolution layer and a channel attention mechanism module is introduced to obtain a channel attention bottleneck module; Based on the channel attention bottleneck module, the feature image is obtained.
4. The method according to claim 3, characterized in that Based on formula (1), the channel attention mechanism module is obtained; wherein, formula (1) is: Where σ is the activation function, is the aggregated feature without dimensionality reduction, w i Represents y i The channel attention, w j Represents y i The channel attention weight of k represents the coverage of local cross-channel interactions, and Represents y i The set of k adjacent channels.
5. The method according to claim 3, characterized in that: The method for obtaining the channel attention bottleneck module includes: Based on formula (2), a one-dimensional global channel tensor is obtained; wherein the formula (2) is: where F in represents the input feature image, GAP(.) represents the global average pooling function, X c (i, j) represents the value of the cth channel at position (i, j), W and H are the width and height of the spatial size; Based on formula (3), the channel attention bottleneck module is obtained; wherein, the formula (3) is: output=σ(ConvlD K (g c )) (3) Among them, σ represents the activation function, ConvlD K It is a one-dimensional convolution, k represents the coverage of local cross-channel interaction; after batch normalization and activation, the 1×1 convolution layer is used to extract different range features of different input feature images and learn the importance of different channels.
6. The method according to claim 2, characterized in that The method of step S23 comprises: Step S231. Based on the cross attention module, the 1×1 branch feature image encodes each channel in the horizontal and vertical directions to obtain a feature map containing spatial direction information; Step S232: Based on the pooling kernel, encode each of the channels in the horizontal and vertical directions.
7. The method according to claim 5, characterized in that The method of step S232 includes: Based on formulas (4)-(5), the output of the channel at the height and width of the feature image is obtained; wherein the formulas (4)-(5) are: where z c (.) is the output associated with the cth channel, W is the width of the spatial dimension H×W, H is the height of the spatial dimension H×W, x c (.) represents the value of the c-th channel position, G represents the G sub-features into which the input is divided and the G sub-features are much smaller than the number of c channels; Based on formula (6), a nonlinear output is obtained; wherein the formula (6) is: in, represents the connection operation along the spatial dimension, δ is the nonlinear activation function, and Conv is the convolution operation; Based on formula (7), the feature image is obtained by dividing it into two independent tensors in the horizontal and vertical directions respectively; wherein the formula (7) is: Among them, σ is the sigmoid function, G h and G w Represents the horizontal and vertical tensors f h and f w After convolution, it is transformed into a tensor with the same number of channels as the input; Based on formula (8), the output of the 1×1 branch feature image is obtained; wherein the formula (8) is: Among them, O(i,j) is the output of the 1×1 branch at position (i,j), G h and G w The table is expanded and used as attention weights respectively; x(.) is the original input, is matrix multiplication.