Two-dimensional human arm pose estimation method based on feature fusion and attention mechanism

Abstract

The present application relates to the field of computer vision, especially to a two-dimensional human arm posture estimation method based on feature fusion and attention mechanism, comprising the following steps: step one, collecting human action images, and performing image cropping through human boundary range; step two, performing human arm joint feature extraction on the cropped images through a basic network module; step three, performing feature fusion and focusing through a feature fusion and attention combination module, so as to reduce feature loss and improve effective information concentration, and obtaining an arm joint heat map; step four, obtaining position coordinates from the obtained arm joint heat map. The present application can obtain accurate two-dimensional arm joint position coordinates from human action RGB images.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a two-dimensional human arm pose estimation method based on feature fusion and attention mechanisms. Background Technology

[0002] Two-dimensional human arm pose estimation involves using specific algorithms to obtain the positional information of the joints of the human arm in an image or video; it can also be understood as detecting feature points on the image representation. Early researchers established a joint pose estimation method based on a human structural model, but this model suffers from poor robustness against interference. In recent years, the development of Convolutional Neural Networks (CNNs) has propelled pose estimation to new heights, allowing researchers to achieve strong low-level feature extraction and visual structure modeling without manually building cumbersome pre-built models, through a shared convolutional kernel weight mechanism. However, precisely because of this structural characteristic, its receptive field is limited, making it difficult to capture global information. In practical pose estimation tasks, high-level semantic information focuses more on how to correlate feature elements to form a whole, which alleviates the limitations of CNNs to some extent. Therefore, proposing a two-dimensional arm joint pose estimation model that can learn low-level detailed features and high-level semantic features and fully utilize these features has significant research and application value. Summary of the Invention

[0003] To address the aforementioned technical problems in existing technologies, this invention proposes a two-dimensional human arm pose estimation method based on feature fusion and attention mechanisms, the specific technical solution of which is as follows:

[0004] A two-dimensional human arm pose estimation method based on feature fusion and attention mechanism includes the following steps:

[0005] Step 1: Acquire images of human movement and crop the images according to the human body's boundary range;

[0006] Step 2: Extract human arm joint features from the cropped image using the basic network module;

[0007] Step 3: Feature fusion and focusing are performed through the feature fusion and attention combination module to obtain the arm joint heat map;

[0008] Step 4: Obtain the position coordinates from the obtained arm joint heat map.

[0009] Furthermore, step one specifically involves: capturing a color image of human movement using an RGB camera, obtaining the bounding box coordinates of a single human body through manual annotation or by using a bounding box regression detection algorithm, and then using the obtained bounding box to crop the original RGB image to obtain a color image containing only a single person.

[0010] Furthermore, the basic network module adopts a parallel model combining CNN and Transformer, using CNN convolutional neural network to extract low-level detailed features of arm joints, and using Transformer's self-attention mechanism to learn high-level semantic features.

[0011] Furthermore, step three specifically includes:

[0012] Step 3.1: The feature fusion module in the improved feature fusion and attention combination module has a horizontally connected pyramid architecture, which receives the multi-resolution branch outputs of the base network module;

[0013] Step 3.2: After feature fusion in step 3.1, perform convolution operation using a 3x3 convolution kernel;

[0014] Step 3.3: Focus on feature information by combining the multi-dimensional attention mechanism algorithm module in the feature fusion and attention combination module.

[0015] Furthermore, step 3.1 specifically includes:

[0016] Step 3.1.1: Design the header of the feature fusion module as a point convolution structure to make the number of feature channels in each input branch consistent;

[0017] Step 3.1.2: Traverse from bottom to top. Small feature maps are upsampled by a factor of two using the nearest neighbor interpolation algorithm to make them the same size as the adjacent layers.

[0018] Step 3.1.3: Feature fusion is completed after horizontal addition;

[0019] Step 3.1.4: If there are output feature branches that have not been fused, then jump to step 3.1.2.

[0020] Furthermore, step 3.3 specifically includes:

[0021] Step 3.3.1: Determine the importance of each feature channel through the channel attention module, then enhance the learning of important feature channels, and suppress information from invalid feature channels;

[0022] Step 3.3.2: Input the results of the channel attention module into the coordinate attention module, decompose it into a feature aggregation process in two orthogonal directions, and always retain the precise position information in the X or Y direction during encoding, so as to obtain direction-aware and position-sensitive feature results;

[0023] Step 3.3.3: The spatial attention module uses channel weights and coordinate weights to strengthen the weighting of the key point location region;

[0024] Step 3.3.4: Add residual connections to break the symmetry of the network algorithm and improve the degradation problem of the network weight matrix.

[0025] Furthermore, step four specifically involves: obtaining the coordinates of the maximum value point of a certain joint heatmap, then mapping it onto the original cropped human body image, and finally determining the coordinates of the current joint point based on the position of the cropped image in the original image.

[0026] Beneficial effects: This invention proposes a network model based on feature fusion and multi-dimensional attention mechanism. It absorbs the advantages of the Transformer mechanism in high-level semantic information extraction, avoids information loss by fusing multi-scale features, and increases the model's attention to effective feature information by using multi-dimensional attention mechanism, thus effectively improving the detection capability of overall joint position, including occluded joints. Attached Figure Description

[0027] Figure 1 This is a schematic diagram illustrating the specific process of the two-dimensional human arm pose estimation method according to an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the Transformer structure in the basic network module used in this invention.

[0029] Figure 3 This is a general framework diagram of the two-dimensional human arm pose model based on feature fusion and attention mechanism of the present invention;

[0030] Figure 4 This is a schematic diagram of the multi-scale feature fusion of the present invention;

[0031] Figure 5 This is a schematic diagram of the multidimensional attention mechanism combination module of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0033] like Figure 1 As shown in the figure, an embodiment of the present invention provides a two-dimensional human arm pose estimation method based on feature fusion and attention mechanism. The method involves cropping the human action image using a human bounding box obtained by a detection algorithm and inputting it into the designed HRTP-Net two-dimensional arm pose estimation model. The basic network captures low-level detail features and high-level semantic features of the arm joints, and the FFCAM algorithm module is used to achieve feature fusion and focusing, thereby reducing feature loss and improving the focus on effective information. After processing the heatmaps of each joint output by the algorithm, the coordinates of the two-dimensional joint points of the human arm are obtained. The method specifically includes the following steps:

[0034] Step 1: Acquire images of human movement and crop the images according to the human body's boundary range.

[0035] Specifically, color images of human motion are captured using an RGB camera, and the bounding box coordinates of individual human figures are obtained through manual annotation or by using mature bounding box regression detection algorithms such as Faster R-CNN. The obtained bounding boxes are then used to crop the original RGB images, resulting in color images containing only a single person. The cropping result is shown below. Figure 1 The image shown is a cropped human body bounding box.

[0036] Step 2: Extract human arm joint features from the cropped image using the basic network module.

[0037] The basic network modules employ a parallel model to preserve high-resolution feature branches. For example, the HRFormer model combines the advantages of the structural characteristics of CNN and Transformer, utilizing the characteristics of convolutional neural networks to extract low-level detail features of the arm joints and leveraging the self-attention mechanism of Transformer to learn high-level semantic features. Figure 2 As shown, multi-head attention is used to learn independent subspace features, and a convolutional structure is used to fuse the subspaces, thereby achieving comprehensive and detailed feature capture.

[0038] Step 3: Feature fusion and focusing are performed using the Feature Fusion and Attention Combination Module (FFCAM module) to obtain a heatmap of the arm joints, such as... Figure 3 As shown, it includes the following sub-steps:

[0039] Step 3.1: Improve the feature fusion and attention combination module. The feature fusion module has a horizontally connected pyramid architecture to support the multi-resolution branch outputs of the base network modules, specifically including:

[0040] Step 3.1.1: Figure 3 The beginning of the FFCAM module is a point convolution structure. Through 1x1 convolution, the number of feature channels in the four different dimensions of the input branches is adjusted to be consistent.

[0041] Step 3.1.2: Figure 4 The bottom-up traversal process is clearly described. At the beginning, the feature map of the one-eighth size is upsampled by two times using the nearest neighbor interpolation algorithm so that it is the same as the branch size of the adjacent one-quarter size feature map. When returning to this step, the one-quarter size feature map is upsampled by two times, and so on, until only the highest resolution feature map exists.

[0042] Step 3.1.3: Horizontal addition completes feature fusion;

[0043] Step 3.1.4: If there are output feature branches that have not been fused, jump to step 3.1.2;

[0044] Step 3.2: After feature fusion, a 3x3 convolution kernel is used to perform further convolution operations to eliminate aliasing.

[0045] Step 3.3: By combining algorithm modules through the CASM multidimensional attention mechanism, the model can focus more on effective feature information and reduce the learning burden, such as... Figure 5 As shown, it includes the following steps:

[0046] Step 3.3.1: Determine the importance of each feature channel through the channel attention module, then enhance the learning of important channels and suppress invalid information;

[0047] Step 3.3.2: Input the results of the channel attention module into the coordinate attention module, decompose it into a feature aggregation process in two orthogonal directions, and always retain the precise position information in the X or Y direction during encoding, so as to obtain direction-aware and position-sensitive feature results;

[0048] Step 3.3.3: The spatial attention module makes full use of channel weights and coordinate weights to strengthen the weights on the key point location region and improve the model's expressive power;

[0049] Step 3.3.4: Add residual connections to break the symmetry of the network algorithm, improve the degradation problem of the weight matrix, and enhance the expressive power of the model.

[0050] Step 4: Obtain the position coordinates from the obtained arm joint heat map.

[0051] Specifically, the coordinates of the maximum value point of a certain joint heatmap are obtained and then mapped onto the original cropped human body image. The coordinates of the current joint point are finally determined based on the position of the cropped image in the original image.

[0052] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A two-dimensional human arm pose estimation method based on feature fusion and attention mechanism, characterized in that, Includes the following steps: Step 1: Acquire images of human movement and crop the images according to the human body's boundary range; Step 2: Extract human arm joint features from the cropped image using the basic network module; Step 3: Feature fusion and focusing are performed through the feature fusion and attention combination module to obtain the arm joint heatmap, specifically including: Step 3.1: Improve the feature fusion and attention combination module. The feature fusion module has a horizontally connected pyramid architecture, receiving the multi-resolution branch outputs of the base network module, specifically including: Step 3.1.1: Design the header of the feature fusion module as a point convolution structure to make the number of feature channels in each input branch consistent; Step 3.1.2: Traverse from bottom to top. Small feature maps are upsampled by a factor of two using the nearest neighbor interpolation algorithm to make them the same size as the adjacent layers. Step 3.1.3: Feature fusion is completed after horizontal addition; Step 3.1.4: If there are output feature branches that have not been fused, then proceed to step 3.1.2; Step 3.2: After feature fusion in step 3.1, perform convolution operation using a 3x3 convolution kernel; Step 3.3: Focus on feature information through the multi-dimensional attention mechanism combination algorithm module in the feature fusion and attention combination module, specifically including: Step 3.3.1: Determine the importance of each feature channel through the channel attention module, then enhance the learning of important feature channels, and suppress information from invalid feature channels; Step 3.3.2: Input the results of the channel attention module into the coordinate attention module, decompose it into a feature aggregation process in two orthogonal directions, and always retain the precise position information in the X or Y direction during encoding, so as to obtain direction-aware and position-sensitive feature results; Step 3.3.3: The spatial attention module uses channel weights and coordinate weights to strengthen the weighting of the key point location region; Step 3.3.4: Add residual connections to break the symmetry of the network algorithm and improve the degradation problem of the network weight matrix; Step 4: Obtain the position coordinates from the obtained arm joint heat map.

2. The two-dimensional human arm pose estimation method based on feature fusion and attention mechanism according to claim 1, characterized in that, Step one specifically involves: capturing color images of human motion using an RGB camera, obtaining the bounding box coordinates of a single human body through manual annotation or by using a bounding box regression detection algorithm, and then using the obtained bounding box to crop the original RGB image to obtain a color image containing only a single person.

3. The two-dimensional human arm pose estimation method based on feature fusion and attention mechanism according to claim 1, characterized in that, The basic network module adopts a parallel model combining CNN and Transformer. It uses CNN convolutional neural network to extract low-level detailed features of arm joints and uses the self-attention mechanism of Transformer to learn high-level semantic features.

4. The two-dimensional human arm pose estimation method based on feature fusion and attention mechanism according to claim 1, characterized in that, Step four specifically involves: obtaining the coordinates of the maximum value point of a certain joint heatmap, mapping it onto the original cropped human body image, and finally determining the coordinates of the current joint point based on the position of the cropped image in the original image.

Images