A drama character pose estimation method combining attention and parallel branch structure

Abstract

The application discloses a drama character posture estimation method combining attention parallel branch structure, which comprises the following steps: step 1, preprocessing the public data set MS COCO data set, and the preprocessing is specifically data enhancement operation, and the preprocessed data set is used as a training set; step 2, constructing a parallel branch high-resolution network model based on an attention mechanism; step 3, training the parallel branch high-resolution network model based on the attention mechanism by using the training set, and obtaining the trained parallel branch high-resolution network model based on the attention mechanism; step 4, preprocessing a drama character posture image to be estimated, and the preprocessing is specifically a uniform size operation; and step 5, inputting the preprocessed image to be estimated into the trained parallel branch high-resolution network model based on the attention mechanism, and obtaining an estimated posture result. The method enhances the feature expression capability of the network model, and realizes accurate positioning of a hidden joint node.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and key point estimation, and relates to a drama character pose estimation method combining attention parallel branch structure. BACKGROUND

[0002] With the comprehensive development of computer technology in medical treatment, transportation, agriculture and many other fields, human pose estimation, as one of the most basic downstream tasks in the field of computer vision, has become a research hotspot. Traditional human pose estimation is usually based on machine learning to extract features, which is limited in real-time performance and accuracy. Since 2014, deep learning has received more attention in the field of computer vision, and various pose estimation networks based on deep learning have been proposed, such as stacked hourglass network, convolution pose machine network and high-resolution network. Each module in the stacked hourglass network uses convolution and pooling operations for downsampling, bilinear interpolation for upsampling, and residual links for feature fusion for the same size feature map during downsampling and upsampling. However, due to the scale transformation of the feature map during upsampling and downsampling, local feature information is lost, resulting in low accuracy of human pose estimation. The convolution pose machine network models long-range spatial dependencies using larger convolution kernels and uses sequential convolution architecture to express image space and texture information, which is a multi-stage pose estimation network. However, due to its large parameter quantity and computational complexity, it is difficult to train on ordinary machines. The high-resolution network maintains high-resolution feature maps throughout the feature extraction process, gradually adding low-resolution subnetworks in parallel in the high-resolution feature map main network, thereby realizing multi-scale feature fusion and extraction. However, due to the use of 3x3 size convolution kernels throughout, there are defects in obtaining long-range spatial dependencies, and when the image has a complex background or the person in the image has complex deformation and distortion, the network joint estimation accuracy will be greatly lost.

[0003] Drama characters usually wear loose robe-style clothes, which can blur the specific position of human joints and make it difficult to accurately locate the human joint points. The stacked hourglass network and the high-resolution network cannot model the spatial dependency of human joints well, so they cannot perform high-precision pose estimation for the clothing occlusion problem in drama character images. The convolution pose machine network can effectively model some occluded human joint points due to the use of larger convolution kernels, but its large parameters and computational complexity make it difficult to implement.

[0004] In summary, the existing technology has the following problems:

[0005] (1) The traditional machine learning method for human pose estimation has low positioning efficiency and estimation accuracy.

[0006] (2) The stacked hourglass network and high-resolution network feature extraction is insufficient, and high-precision pose estimation cannot be performed on the costume occlusion problem in the drama character image.

[0007] (3) The convolutional pose machine network needs large computing power hardware support, and is difficult to implement. SUMMARY

[0008] The purpose of the present application is to provide a drama character pose estimation method combining attention parallel branch structure, which solves the problems of insufficient feature extraction and inaccurate occlusion node positioning in the prior art under the premise of not easily obtaining large computing power hardware devices.

[0009] In order to achieve the above purpose, the present application adopts the following technical solutions to solve it:

[0010] On the one hand, the present application provides a parallel branch high-resolution network model construction method based on attention mechanism, comprising the following steps:

[0011] Step 1, constructing a hybrid attention module based on channel domain and spatial domain;

[0012] The hybrid attention module is realized by using a serial channel domain attention module and two same and continuous spatial domain attention modules;

[0013] The input of the channel domain attention module is a local feature map S, and the spatial dimension satisfies S∈R c×w×h , c is the channel number of the local feature map S, w and h are the width and height of the local feature map S respectively; first, the local feature map S is respectively used for global maximum pooling and global average pooling, and the results are respectively subjected to 1×1 convolution, Relu activation, 1×1 convolution, and then the feature is aggregated by using the corresponding element addition method to obtain a channel correlation relationship matrix, the dimension of which is c×1×1; then the Sigmoid activation function is used to map the values in the channel correlation relationship matrix to 0-1 to obtain a channel correlation weight matrix W, and then the channel correlation weight matrix W is multiplied with the initial local feature map S in the form of corresponding elements to obtain an initial channel domain attention output I; finally, I and S are connected in residual to obtain the local feature map SS output by the final channel domain attention module;

[0014] The input of the spatial domain attention module is the local feature map SS output by the channel domain attention module, and the spatial size satisfies SS∈R c×w×h ; first, the local feature map SS is subjected to channel number adjustment using 1×1 convolution to obtain Q and K feature matrices, {Q,K}∈R c'×w×hwhere c' is the number of nodes of the corresponding data set; then an initial attention graph D is generated through Affinity Affine transformation; the initial attention graph D obtains a cross-attention matrix A after passing through a Softmax activation function; meanwhile, a local feature map SS output by a channel domain attention module obtains a feature map V through one 1*1 convolution;

[0015] Then, the feature map V and the cross-attention matrix A are aggregated according to the following formula:

[0016]

[0017] SS' = [SS' u ]

[0018] where Φ u is a two-dimensional vector composed of feature values of the feature map V at a row and a column where an arbitrary position u is located, and the values in the row and column directions are sequentially numbered as a Φ u column vector, and the values in the channel direction are a Φ u row vector; A u is a channel direction vector of the cross-attention matrix A at the position u; SS u is a channel dimension vector component of the input SS feature map of the spatial attention module at the position u;

[0019] Finally, the feature map SS' and the feature map SS are fused to obtain a feature map M of the output of the first spatial attention module;

[0020] Two consecutive spatial attention modules are used in the hybrid attention module, and the output feature map M of the first spatial attention module is taken as the input of the second spatial attention module, which is replaced by the feature map SS to enter the second spatial attention module, and the same operation as the first spatial attention module is performed to obtain a final output feature map N of the channel attention module;

[0021] Step 2, according to the result of step 1, a parallel branch high-resolution network based on an attention mechanism is constructed.

[0022] Further, step 2 includes the following sub-steps:

[0023] Step 21, taking the output of the first convolutional layer of the second stage of the HRNet as the input of the first hybrid attention module, to obtain the output of the first hybrid attention module;

[0024] Taking the output of the first convolutional layer of the third stage of the HRNet as the input of the second hybrid attention module, to obtain the output of the second hybrid attention module;

[0025] The output of the first convolutional layer of the fourth stage of the HRNet is taken as the input of the third mixed attention module, and the output of the third mixed attention module is obtained;

[0026] The channel number of the output feature map of each mixed attention module is consistent with the channel number of the highest resolution subnetwork;

[0027] In step 22, assuming that the channel number of the highest resolution subnetwork is c, the output feature map of the first channel attention module is subjected to twice feature refinement through residual units, and the output feature map of the second channel attention module is subjected to feature aggregation through a concate operation, at this time, the channel number of the feature map is 2c, then a 1*1 convolution is used for channel number adjustment to obtain a feature map with a channel number of c, and then a residual unit is used for feature refinement, and the output of the third attention block is subjected to feature aggregation through a concate operation, at this time, the channel number of the feature map is 2c, then a 1*1 convolution is used for channel number adjustment to obtain a feature map with a channel number of c; and then a residual unit is used for feature refinement;

[0028] In step 23, the final output feature map of the original HRNet network is taken as the input feature map of the fourth mixed attention module, and the feature map obtained in step 22 is subjected to feature aggregation through a concate operation, at this time, the channel number of the feature map is 2c, then a 1*1 convolution is used for channel number adjustment to obtain a feature map with a channel number of c, and then a residual unit is used for feature refinement to obtain a final feature map, which is taken as the output feature map of the drama character pose estimation network.

[0029] In another aspect, the present application provides a drama character pose estimation method combining attention parallel branch structure, comprising the following steps:

[0030] In step one, the public dataset MS COCO dataset is preprocessed, and the preprocessing is a data enhancement operation; and the preprocessed dataset is taken as a training set;

[0031] In step two, the attention mechanism-based parallel branch high-resolution network model is constructed as claimed in claim 1 or 2;

[0032] In step three, the attention mechanism-based parallel branch high-resolution network model is trained by using the training set, and a trained attention mechanism-based parallel branch high-resolution network model is obtained;

[0033] In step four, the drama character pose image to be estimated is preprocessed, and the preprocessing is a uniform size operation;

[0034] Step five, input the pre-processed image to be estimated into the trained attention mechanism based parallel branch high resolution network model to obtain the estimated pose result.

[0035] Compared with the prior art, the present application has the following advantages:

[0036] (1) By designing a hybrid attention mechanism based on channel domain and spatial domain, the key features of the image are rebalanced in the channel and spatial dimensions to extract more rich context relationships at a small cost, thereby enhancing the feature expression capability of the network model.

[0037] (2) By designing a high-resolution parallel branch structure network, the above attention mechanism is used in the parallel branch structure, which improves the feature processing capability of the network model and realizes accurate positioning of the occluded key nodes. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is the overall structure diagram of the hybrid attention module;

[0039] Figure 2 is the channel domain attention module diagram in the hybrid attention module;

[0040] Figure 3 is the structure diagram of the attention mechanism based parallel branch high resolution network model of the present application;

[0041] Figure 4 is the spatial domain attention module diagram in the hybrid attention module;

[0042] Figure 5 is the Q u vector diagram in the channel dimension of the feature map;

[0043] Figure 6 is the Q i,u vector diagram in the channel dimension of the feature map;

[0044] Figure 7 is the matrix construction process diagram of Phi u ;

[0045] Figure 8 is the (a) training Acc curve and (b) AP curve of each epoch on the validation set during training;

[0046] Figure 9 is the qualitative comparison result of each pose estimation network on the drama character image; (a) and (b) represent two different drama characters.

[0047] Figure 10 is the effect of the present application on pose estimation of drama characters on more images;

[0048] Figure 11 is the effect of the application on multi-person pose estimation of theatrical characters in a top-down manner;

[0049] Figure 12 is the effect of the application on single-person pose estimation of the COCO dataset;

[0050] Figure 13 is the effect of the application on multi-person pose estimation of the COCO dataset in a top-down manner. DETAILED DESCRIPTION

[0051] The application will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0052] The embodiment gives a theatrical character pose estimation method combining attention parallel branch structure, including the following steps:

[0053] Step one, pre-process the public dataset MS COCO dataset, the pre-processing is specifically a data enhancement operation, and the pre-processed dataset is used as a training set;

[0054] Step two, build an attention mechanism-based parallel branch high-resolution network model;

[0055] Step three, train the attention mechanism-based parallel branch high-resolution network model (APHRNet) using the training set, to obtain a trained attention mechanism-based parallel branch high-resolution network model;

[0056] Step four, pre-process the theatrical character pose image to be estimated, the pre-processing is specifically a uniform size operation;

[0057] Step five, input the pre-processed image to be estimated into the trained attention mechanism-based parallel branch high-resolution network model, to obtain the estimated pose result.

[0058] As shown in Figure 3 , step two of building an attention mechanism-based parallel branch high-resolution network model includes the following sub-steps:

[0059] Step 1, build a hybrid attention module based on channel domain and spatial domain.

[0060] As shown in Figure 1 , the hybrid attention module is implemented using a serial channel domain attention module and two identical and consecutive spatial domain attention modules. The input and output remain consistent in terms of channels and scales.

[0061] As shown in Figure 2 , the input of the channel domain attention module is a local feature map S, and the spatial dimension satisfies S∈R c×w×h, c is the number of channels of the local feature map S, w and h are the width and height of the local feature map S respectively. First, the global maximum pooling and the global average pooling are used on the local feature map S respectively, and the results are respectively subjected to 1x1 convolution, Relu activation, 1x1 convolution, and then the feature aggregation is performed in the manner of corresponding element addition to obtain a channel correlation relationship matrix, the dimension of which is c x 1 x 1; then the values in the channel correlation relationship matrix are mapped to 0-1 by using the Sigmoid activation function to obtain a channel correlation weight matrix W, and then the channel correlation weight matrix W is used in the manner of corresponding element multiplication on the initial local feature map S to obtain an initial channel domain attention output I. Finally, I and S are connected in residual to obtain the local feature map SS output by the final channel domain attention module. The channel domain attention module is represented by the following formula:

[0062]

[0063] MLP(X)=Conv(Relu(Conv(X))) (1.2)

[0064] As shown in Figure 4 , the spatial domain attention module is different from the Non-local spatial attention block in that one spatial domain attention block can only obtain the context information of one position in the vertical and horizontal directions at a time. By cyclically stacking two spatial attention blocks, each position in the spatial domain can obtain the feature correlation relationship of any other position. Compared with the time and space complexity of the Non-local module O(N 2 ), the time and space complexity of the spatial domain attention module used in the application are both , where N=w x h represents the feature map size.

[0065] The input of the spatial domain attention module is the local feature map SS output by the channel domain attention module of formula (1.1), and the spatial size satisfies SS∈R c×w×h . In specific implementation, first, a 1x1 convolution is used to adjust the channel number of the local feature map SS to obtain Q and K feature matrices, {Q, K}∈R c'×w×h , where c' is the number of channels corresponding to the data set, and in the COCO data set, its value is 17, and in the MPII data set, its value is 16. Generally, c' < c, that is, the adjusted channel number is less than the initial channel number, so that the operation amount of the subsequent calculation of the cross attention matrix is reduced; then an initial attention map D is generated through Affinity affine transformation. The Affinity correlation calculation formula is defined as follows:

[0066] d i,u =Q u ·Ω i,u (1.3)

[0067] D u =[d i,u (1.4)

[0068] D = [D u (1.5)

[0069] In the formula, Q u Let u be a vector extracted from any position u in the Q feature map along the channel direction, such as Figure 5 As shown, Q∈R c' Ω u Let Ω represent the features of the row and column corresponding to position u in the K feature map in the channel dimension. u ∈R (w+h-1)×c' Ω i,u Ω u The component at position i is as follows Figure 6 As shown, Ω i,u ∈R c' , i = 1, 2, ..., w + h - 1. d i,u Ω u and Ω i,u The dot product of two vectors is a scalar representing the degree of correlation between the two vectors. (D) u For [d] 1,u ,d 2,u ,…,d w+h-1 Vector, D u ∈R w+h-1 , representing the relevance of position u to other positions in its row and column. D is the initial attention map, D∈R. (w+h-1)×w×h The initial attention map D is processed by the Softmax activation function to obtain the cross-attention matrix A, where A∈R. (w+h-1)×w×h .

[0070] Simultaneously, the local feature map SS output by the channel domain attention module is convolved once to obtain the feature map V, where V∈R. c×w×h .

[0071] Then, the feature map V and the cross-attention matrix A are aggregated according to the following formula:

[0072]

[0073] SS'=[SS' u (1.7)

[0074] Where Φ u Φ is a two-dimensional vector consisting of the eigenvalues ​​of the row and column where u is located in the feature map V. The row and column direction values ​​are numbered sequentially and then used as Φ. u Column vector, channel direction value as Φ u Row vectors, such asFigure 7 Φu,v u ∈R (w+h-1)×c ; A u is the cross-attention matrix A at position u in the channel direction vector, A u ∈R w+h-1 SS u is the channel dimension vector component of the input SS feature map of the spatial attention module at position u, SS u ∈R c SS' is obtained from equation 1.6 u SS' u ∈R c represents the correlation vector between any position u of the input feature map of the spatial attention module and other positions in the row and column where it is located; the feature map SS' is obtained from equation 1.7, SS'∈R c×w×h ;

[0075] Finally, the feature map SS' is fused with the feature map SS to obtain the output feature map M of the first spatial attention module.

[0076] Two consecutive spatial attention modules are used in the mixed attention module, and the output feature map M of the first spatial attention module is used as the input of the second spatial attention module, which replaces the feature map SS into the second spatial attention module and performs the same operation as the first spatial attention module to obtain the final output feature map N of the channel attention module. The feature map N contains the correlation between all positions, enhancing the expression ability of image information and enabling the model to achieve better performance.

[0077] Step 2: Construct a parallel branch high-resolution network based on attention mechanism.

[0078] The parallel branch structure in the convolutional neural network refers to multiple branches composed of multiple independent convolutional layers or fully connected layers, which process the input data in parallel and then fuse the results of each branch output together. The neural network using the parallel branch structure has the following advantages: ① Improve model processing capacity, parallel multi-branch can process different features at the same time, thereby improving network processing speed; ② Improve model task accuracy, different branches can capture different scale features, and by fusing different scale features, the network model feature expression ability is enhanced, thereby improving the accuracy of the model task; ③ Reduce the risk of network overfitting, the multi-branch structure in the network can reduce the model depth and model complexity, thereby effectively reducing the risk of overfitting.

[0079] HRNet is a typical parallel branch structure network, which always maintains a high resolution feature branch, and adds low resolution subnetwork branches layer by layer, each branch has a different resolution, so it can capture different scale information. These different resolution subnetwork branches finally fuse features on the highest resolution subnetwork to form a global feature map that can reflect image details and global structure information.

[0080] The parallel branch structure proposed in the present application adds an attention parallel branch to the highest resolution subnetwork based on the existing HRNet network structure, and the overall network architecture is as shown in the figure. Figure 1 The specific construction process includes the following sub-steps:

[0081] Step 21, taking the output of the first convolutional layer of the second stage of HRNet as the input of the first mixed attention module, obtaining the output of the first mixed attention module;

[0082] Taking the output of the first convolutional layer of the third stage of HRNet as the input of the second mixed attention module, obtaining the output of the second mixed attention module;

[0083] Taking the output of the first convolutional layer of the fourth stage of HRNet as the input of the third mixed attention module, obtaining the output of the third mixed attention module;

[0084] The channel number of the output feature map of each mixed attention module is consistent with the channel number of the highest resolution subnetwork.

[0085] Step 22, assuming that the channel number of the highest resolution subnetwork is c, the output feature map of the first channel attention module is refined by two residual units, and the output feature map of the second channel attention module is aggregated by using the concate operation, at this time the channel number of the feature map is 2c, then adjust the channel number by using a 1×1 convolution, obtain a feature map with a channel number of c, and then use a residual unit for feature refinement, and then use the concate operation with the output of the third attention block for feature aggregation, at this time the channel number of the feature map is 2c, then adjust the channel number by using a 1×1 convolution, obtain a feature map with a channel number of c; and then use a residual unit for feature refinement.

[0086] Step 23, the final output feature map of the original HRNet network is taken as the input feature map of the fourth hybrid attention module, and the feature map obtained in step 22 is used for feature aggregation by using the concate operation, at this time, the channel number of the feature map is 2c, then a 1x1 convolution is used for channel number adjustment to obtain a feature map with a channel number of c, and then a residual unit is used for feature refinement to obtain a final feature map, which is taken as the output feature map of the drama character pose estimation network.

[0087] In order to prove the effectiveness of the present application, experiments are carried out on the COCO dataset, and quantitative and qualitative analyses are carried out. The hardware environment of the experiment is CPU: Intel Core i9-10900K, and the graphics card uses NVIDIA GeForce RTX 3080Ti, the software environment is Ubuntu 18.04.6LTS system, CUDA11.1 graphics card computing power acceleration driver, and data processing is carried out through Python3.6 and Pytorch1.8.0. The parameter table of the network model during training is shown in Table 1.

[0088] Table 1 Network parameter setting

[0089]

[0090]

[0091] First, the MS COCO dataset is trained, and the human body detection frame with a height-width ratio of 4:3 is used to crop the single human body instance in the image, which is adjusted to two different fixed size resolutions of 256x192 and 384x288 as network input. In terms of data enhancement, -45°-45° random rotation, 0.65-1.35 random scale transformation, random flipping and half-body image data enhancement strategy are used. During the training process, the Adam optimizer is used, the basic learning rate is set to 0.001, and is attenuated to 0.0001 at the 170th round and to 0.00001 at the 200th round. A total of 210 rounds of training are carried out, and then the model effect is verified on the COCO val2017 verification set, and the results are shown in Table 2. Among them, AP, AP 50 , AP 75 , AP M , AP L Five evaluation indexes are used as the measurement standard. In order to facilitate comparison, the network APHRNet proposed in the present application uses a light gray undercoat.

[0092] Table 2 Comparison of quantitative experimental results of the present method

[0093]

[0094] The upper half of the table inputs a network image size of 256x192. As can be seen from the table, the APHRNet network using the attention-based parallel branch module can achieve an AP of 75.1, which is improved by 8.2 percentage points compared with the early 8-stage stacked hourglass network model, indicating that the model structure has stronger feature processing capability compared with the stacked hourglass structure. Compared with the baseline scheme of the HRNet network, it is improved by 0.7 points, indicating that the attention mechanism and the parallel branch structure improve the model expression capability of the pose estimation task. The main benefit comes from the AP M index, reaching 1.1 percentage points. The AP L index also has a larger improvement, with a benefit of 0.7 percentage points. It shows that the network attention parallel branch structure can enhance the model feature information acquisition capability, thereby improving the accuracy of pose estimation.

[0095] The lower half of the table uses 384x288 resolution images as input. As can be seen from the table, compared with the baseline scheme of the HRNet_384 network, the APHRNet_384 model AP index can obtain a benefit of 0.9, and the main benefit is also from the AP M , indicating that the attention parallel branch structure can enhance the network's attention to the human joint points, thereby improving the pose estimation effect.

[0096] At the same time, in order to verify the convergence speed of the model, the training accuracy of each round in the training process of HRNet_256 and APHRNet_256 and the AP value on the validation set are made into training curves, as shown in Figure 8 (a) and (b). As can be seen from the figure, the training Acc curve of the HRNet network is higher than that of the APHRNet in the first few epochs, and then it is obviously below the APHRNet training Acc curve. The average accuracy of each epoch on the validation set in the training process of the HRNet network is always lower than that of the APHRNet. It shows that the attention parallel branch structure proposed in this chapter can accelerate the network convergence and improve the pose estimation effect of the network.

[0097] Then the qualitative results are compared. The model trained using the COCO training set is used to select the images of people wearing theatrical costumes for qualitative comparison. The results are shown in Figure 9 The first column is the estimation effect diagram using the stacked hourglass network, the second column is the estimation effect diagram using the high-resolution network, and the third column is the estimation effect diagram of the method of the present application, as can be seen from Figure 9 Compared with the previous high-resolution network and stacked hourglass network, the present application can better predict the body parts with partial occlusion.

[0098] Figure 10The effect of pose estimation of the application on more single-person drama character images is shown. As shown in the figure, for the character instance with a robe type costume, the application can accurately estimate the drama character pose.

[0099] Figure 11 The effect of using the application as a pose estimator in top-down multi-person pose estimation using images containing multiple drama characters is shown. As shown in the figure, for the challenges of joint occlusion, self-occlusion, and distortion deformation existing in drama characters, the application can accurately estimate the pose, indicating the effectiveness of the application.

[0100] Figure 12 The single-person pose estimation results of the application on some images in the COCO dataset are shown. As shown in the figure, the network can effectively realize high-precision joint pose estimation for various body poses such as sitting, standing, lying, and motion distortion in natural scenes.

[0101] Figure 13 The application is used as a pose estimator in top-down multi-person pose estimation, and the verification effect on the COCO dataset is shown. As shown in the figure, the application can accurately estimate the pose of a human body with complex deformation in a natural scene.

[0102] Finally, in order to prove the real effectiveness of the method of the application, the contributions of the parallel branch and the attention module to the network benefits are verified respectively, and four kinds of algorithm comparison experiments are designed. The resolution of 256x192 is used as the image input for training on the COCO training set, and the COCO validation set is used for testing, and the results are shown in Table 3.

[0103] In the table, HRNet-W32+Pb represents a parallel branch high-resolution network, which only adds a parallel branch to the HRNet and does not use an attention module, i.e. the attention module in the attention branch module is replaced with a residual unit. HRNet-W32+Pb+Attention is a parallel branch high-resolution network based on attention mechanism, which uses Figure 1The network of the structure, and from the second stage, the attention module is added, all the attention module weights remain consistent, no longer additional weight ratio is applied, that is, s2:s3:s4:fin=1:1:1:1, wherein s2 is the attention additional weight applied to stage2 stage, s3 is the attention additional weight applied to stage3 stage, s4 is the attention additional weight applied to stage4 stage, and fin is the additional weight applied to the final output attention block. APHRNet is the final network of this chapter, and the only difference between it and the HRNet-32+Pb+Attention structure is that the network applies an additional weight to each attention block, and the weight ratio satisfies s2:s3:s4:fin=1:1:3:10, that is, the attention weight output in the stage2 and stage3 stages uses a weight coefficient of 0.1, the attention weight output in the stage4 stage is multiplied by a weight coefficient of 0.3 as the final attention block weight output in the stage4 stage, and then the convolution output in the previous attention branch is fused using the concate feature to participate in subsequent convolution operation. The final attention block output weight is multiplied by a weight coefficient of 1, and then participates in subsequent feature fusion and convolution operation.

[0104] Table 3 Ablation experiment results of the method

[0105]

[0106] As can be seen from the data in the table, the HRNet-W32+Pb scheme with only the addition of the residual module parallel branch module makes the network consistent with the APHRNet structure, and only 0.1 point improvement can be obtained on the baseline scheme. After replacing the residual module in the newly added parallel branch module with an attention mechanism, the network can obtain a performance improvement of 0.5 points, indicating that the final benefit of the network mainly comes from the attention module in the parallel branch structure. And after adding the attention parallel branch, the AP M , AP L The indicators relative to the baseline scheme have a large amplitude of improvement, which are 0.8 points and 0.5 points respectively, indicating that the attention mechanism proposed in this chapter can make the network pay more attention to the key node estimation task, so as to extract more key node feature information and improve the final pose estimation accuracy. In addition, since the features contained in the shallow network are not sufficient, and the features extracted by the deep network often contain global abstract features and local detailed features, therefore, the weight parameters of the attention mechanism applied in different parts will affect the final accuracy of the network. After parameter fine-tuning, the APHRNet network uses 1:1:3:10 as the final attention block weight parameter, and the highest estimation accuracy is obtained.

Claims

1. A method for constructing a parallel branch high-resolution network model based on an attention mechanism, characterized in that, Comprising the following steps: Step 1, constructing a mixed attention module based on channel domain and spatial domain; The mixed attention module is implemented using a serial channel domain attention module and two identical and consecutive spatial domain attention modules; The input of the channel domain attention module is a local feature map S, and the spatial dimension satisfies , c is the number of channels of the local feature map S, w and h are the width and height of the local feature map S respectively; first, the global maximum pooling and the global average pooling are used on the local feature map S respectively, and then the results are respectively subjected to 1x1 convolution, Relu activation and 1x1 convolution, and then the feature aggregation is performed by using the corresponding element addition, to obtain a channel correlation matrix, and the dimension of the channel correlation matrix is c x 1 x 1; then the values in the channel correlation matrix are mapped to 0-1 by using a Sigmoid activation function, to obtain a channel correlation weight matrix W, and then the channel correlation weight matrix W is used to act on the initial local feature map S by using the corresponding element multiplication, to obtain an initial channel domain attention output I; finally, the I and the S are connected in a residual manner to obtain a local feature map SS output by the final channel domain attention module. The input of the spatial domain attention module is the local feature map SS output by the channel domain attention module, and the spatial size satisfies First, the channel number of the local feature map SS is adjusted using a 1×1 convolution to obtain Q and K feature matrices, , wherein is the number of corresponding data sets; then an initial attention map D is generated through an Affinity affine transformation; after the initial attention map D is activated by a Softmax activation function, a cross attention matrix A is obtained; at the same time, the local feature map SS output by the channel domain attention module is subjected to a 1×1 convolution to obtain a feature map V; Then, the feature map V and the cross-attention matrix A are aggregated according to the following formula: wherein is a two-dimensional vector of feature values of the feature map V at the row and column of an arbitrary position u, the row and column direction values being sequentially numbered as a column vector, and the channel direction values as a row vector; is a channel direction vector of the cross-attention matrix A at the position u; is a channel dimension vector component of the input SS feature map of the spatial attention module at the position u; Finally, the feature map M is obtained by performing feature fusion on the feature map M and the feature map M. The feature map M is obtained by performing feature fusion on the feature map M and the feature map M. SS The feature map M is obtained by performing feature fusion on the feature map M and the feature map M. The second spatial attention module in the mixed attention module takes the output feature map M of the first spatial attention module as input, replaces the feature map SS into the second spatial attention module, and performs the same operation as the first spatial attention module to obtain the final output feature map N of the channel attention module; Step 2, constructing an attention mechanism-based parallel branch high-resolution network according to the result of step 1, Comprising the following sub-steps: Step 21, taking the output of the first convolutional layer of the second stage of the HRNet as the input of the first mixed attention module, and obtaining the output of the first mixed attention module; Taking the output of the first convolutional layer of the third stage of the HRNet as the input of the second mixed attention module, and obtaining the output of the second mixed attention module; Taking the output of the first convolutional layer of the fourth stage of the HRNet as the input of the third mixed attention module, and obtaining the output of the third mixed attention module; The channel number of the output feature map of each mixed attention module is consistent with the channel number of the highest resolution sub-network; Step 22, assuming that the channel number of the highest resolution sub-network is c, the output feature map of the first channel attention module is refined by two residual units, and the output feature map of the second channel attention module is aggregated by using the concate operation, at this time the channel number of the feature map is 2c, then the channel number is adjusted by using a 1x1 convolution, and a feature map with a channel number of c is obtained, and then the feature is refined by using a residual unit, and the output of the third attention block is aggregated by using the concate operation, at this time the channel number of the feature map is 2c, then the channel number is adjusted by using a 1x1 convolution, and a feature map with a channel number of c is obtained; and then the feature is refined by using a residual unit; Step 23, taking the final output feature map of the original HRNet network as the input feature map of the fourth mixed attention module, and aggregating the feature map obtained in step 22 by using the concate operation, at this time the channel number of the feature map is 2c, then the channel number is adjusted by using a 1x1 convolution, and a feature map with a channel number of c is obtained, and then the feature is refined by using a residual unit to obtain the final feature map, which is used as the output feature map of the drama character pose estimation network.

2. A drama character pose estimation method combining attention and parallel branch structure, characterized in that, Comprising the following steps: Step one, preprocessing the public dataset MS COCO dataset, and the preprocessing specifically includes data enhancement operation, and the preprocessed dataset is used as the training set; Step two, constructing the attention mechanism-based parallel branch high-resolution network model as claimed in claim 1; Step three, training the attention mechanism based parallel branch high resolution network model by using the training set to obtain the trained attention mechanism based parallel branch high resolution network model; Step four, preprocessing the to-be-estimated drama character pose image, and the preprocessing specifically includes a uniform size operation; Step five, inputting the preprocessed to-be-estimated image into the trained attention mechanism based parallel branch high resolution network model to obtain an estimated pose result.

Images