HRNet human posture recognition method based on attention mechanism optimization

Abstract

The present invention discloses a HRNet human posture recognition method based on attention mechanism optimization, which belongs to the field of human posture recognition and deep learning processing. The method first adds dilated convolution in the cross-channel fusion process of feature maps with different resolutions, which can increase the receptive field without changing the size of the low-resolution feature map without generating additional parameters and calculation amount, ensuring that important information is not ignored when making decisions; secondly, a new feature fusion strategy is proposed, which introduces a channel attention mechanism to perform weighted fusion of feature maps with different resolutions, and adaptively recalibrates another direction of the feature map to enhance meaningful features while suppressing weak features, thereby accelerating the convergence speed, optimizing the human posture recognition performance, and further improving the detection accuracy.

Description

Technical Field

[0001] The present invention relates to a HRNet human posture recognition method based on attention mechanism optimization, and belongs to the field of human posture recognition and deep learning processing. Background Art

[0002] Human posture estimation is an important research topic in computer vision. Currently, human posture can be inferred through the posture detection method Part Affinity Field (PAF) by combining key points with human graph structure. The algorithm first decomposes the human structure into many nodes, uses the human structure model to model the relationship between nodes, and finally divides each node and connects it into a whole to form a complete human posture. However, in the case of complex backgrounds and highly flexible human postures, the accuracy and efficiency of the graph structure model for similar actions will drop sharply, making it difficult to achieve the level of practical application.

[0003] In recent years, many methods that directly learn and recognize human behavior from image measurements have explored this field in depth, but they still face a series of challenges, including foreground occlusion, background chaos, lighting, posture complexity, multi-person overlap, and similar computational complexity. In 2013, Coredelia matched features by superphoton descriptors and dense optical flow, and calculated the optical flow of all pixels on the image, which requires a lot of calculations and is difficult to use in real-time detection; in 2016, Wang et al. improved the visual word bag and fusion method to divide human posture, but could not grasp the overall characteristics of the human body for similar actions; from the development of human posture estimation, the addition of deep convolutional neural networks has achieved considerable results; in 2020, Hiroaki et al. achieved an accuracy of 96.1% in UCF-50 through the method of linear dynamical systems (LDSs); in 2021, many deep convolutional networks were added to detect human posture. Most existing methods preprocess the video frame and input it into the network, pass the input through a network, consisting of high-resolution to low-resolution subnetworks connected in series, and then increase the resolution. For example, the hourglass network restores high resolution through a symmetrical low-to-high process; SimpleBaseline uses some transformed convolutional layers to generate high-resolution representations. Deep learning has been a hot research direction in the field of machine learning in recent years. Many major achievements have been successfully applied to various image processing tasks. Vision-based intelligent human action recognition is the most challenging direction in the field of computer vision in recent years. It detects human actions in video sequences, extracts action features, and learns action features to identify human behavior in the video. Therefore, the research on human action recognition methods based on deep learning is particularly important.

[0004] In the existing human posture recognition scheme based on the HRNet network, feature extraction is achieved by stacking convolutional layers and pooling layers. After the image is output from the network, the convolutional layer extracts features, while the pooling layer aggregates features and allows the model to have a certain degree of translation invariance to reduce the computing power of the subsequent convolutional layers, and finally outputs the classification results to the fully connected layer. However, stacking will lead to an increase in parameters and calculations, increasing the computing cost. The feature map is often predicted by the output of the last layer. Then, the number of pixels of the original image that a point on the feature map can be mapped to determines the upper limit of the size that the network can detect, and downsampling is required to ensure the receptive field. The result of downsampling is that small targets are not easily detected. Therefore, the existing human posture recognition scheme based on the HRNet network has problems such as high computing cost and low accuracy. Summary of the invention

[0005] In order to solve the problems of high computational cost and low precision of current human posture recognition methods, the present invention provides a HRNet human posture recognition method based on attention mechanism optimization.

[0006] The first object of the present invention is to provide a human posture recognition system based on an optimized HRNet network, characterized in that the recognition system comprises: a video stream acquisition module, an optimized HRNet network module, and a classification result output module; the video stream acquisition module, the optimized HRNet network module, and the classification result output module are connected in sequence;

[0007] The optimized HRNet network module includes: a basic HRNet module, a dilated convolution module, and an attention mechanism module; the dilated convolution module is located between subnets of different resolutions in the basic HRNet module, and is used to perform dilated convolution in the upsampling or downsampling process between subnets of different resolutions to increase the receptive field; the attention mechanism module is used to perform weighted fusion of feature maps of different resolutions.

[0008] The second object of the present invention is to provide a human posture recognition method based on an optimized HRNet network, characterized in that the method is implemented based on the human posture recognition system according to claim 1, comprising:

[0009] Step 1: Get the video stream;

[0010] Step 2: The optimized HRNet network module obtains the human posture image to be recognized in the video stream, and inputs the image into the channel of the original resolution to obtain a high-resolution subnet feature map, which is the output feature map of the first stage;

[0011] Step 3: downsample the high-resolution feature map to obtain a low-resolution subnetwork feature map with a resolution of 1 / 2 times the original one, and use dilated convolution to increase the receptive field during the sampling process;

[0012] Step 4: Introduce the attention mechanism to perform cross-resolution feature fusion on the feature maps of subnetworks with different resolutions, perform global average pooling on the feature map of each channel, shrink the spatial dimension to obtain the corresponding weight value, and obtain the weighted feature map, which is the output feature map of the second stage;

[0013] Step 5: Use the weighted feature map obtained in step 4 as the input of the next stage, and repeat steps 2 to 4 until the output feature map of the fourth stage is output;

[0014] Step 6: Obtain the final human posture recognition result according to the output feature map of the fourth stage.

[0015] Optionally, the method for calculating the dilated convolution input and output feature maps is:

[0016]

[0017] Among them, W1 is the input feature map, W2 is the output feature map, d is the expansion rate, p is the number of pixels filled in the convolution, k is the size of the channel, and s is the step size.

[0018] Optionally, the calculation process of step 4 includes:

[0019] The feature maps obtained by the dilated convolution are uniformly upsampled or downsampled, and the feature maps with the same resolution are obtained and then fused. The feature map fusion strategy adopts the channel attention model, and introduces the compression excitation module SE to perform global average pooling on the feature map of each channel and then shrink the spatial dimension to obtain the corresponding weight value;

[0020] The SE module performs cross-channel feature fusion in a weighted manner. When cross-resolution interaction is performed from the n-1 stage to the n stage, b represents the branch sequence number to which the fusion input belongs, and r represents the sequence number of the branch to which the fusion belongs. Down(H nr ) t and Upper(H nr ) t Respectively represent the feature map H nr Perform t times of downsampling and upsampling, then the formula of the newly fused branch result is as follows:

[0021] f nr (H (n-1)r )=H (n-1)r b=r

[0022] f nr (H (n-1)r)=Down(H (n-1)r ) r-b b <r

[0023] f nr (H (n-1)r )=Upper(H (n-1)r ) b-r b>r

[0024] After the SE module, each H nr As input X, X = [x 1 ,x 2 ,……,x C′ ], where x C′ Represents the feature map set of the C′th channel; for each mapping input X∈R H′×W′×C′ To feature map U∈R H×W×C′ The weights corresponding to their features will be calculated separately;

[0025] Among them, H nr is the feature map of the rth branch in the nth stage, f nr (*) represents the sampling operation of the rth branch in the nth stage, H', W', C' are the height, width, and number of channels of the original input feature map, and H and W are the height and width of the output feature map of the SE module, respectively;

[0026] F tr is a convolution operator, and uses V = [v1, v2, ..., v c ] represents the learned filter kernel set, where v c represents the parameter set of the cth filter,

[0027] Finally, the feature map output by the SE module is represented by U = [u1, u2, ..., u c ] indicates that:

[0028]

[0029] Among them, u c is the feature map of the cth channel output by the SE module, u c ∈R H×W , is a two-dimensional spatial kernel representing a single channel acting on the corresponding channel of the input X.

[0030] Optionally, the calculation process of the weighted feature map includes:

[0031] After obtaining the cross-channel feature map, the convolution feature map is processed to obtain a one-dimensional vector with the same number of channels as the evaluation score of each channel, and then the score is applied to the corresponding channels to obtain the result;

[0032] First, the global spatial information is compressed into a channel descriptor through global pooling; the feature map of each channel has a local receptive field, so each unit of the output feature map U cannot use contextual information outside the area; the weight value set z of the feature map is obtained by compressing the spatial dimension of the feature map U, where the cth element of z is calculated as:

[0033]

[0034] Among them, F sq () indicates compressing the image, u c In (i, j), i and j are the subscripts for the accumulation of the height and width of the output feature map;

[0035] After inputting the feature map, a one-dimensional vector is obtained by global pooling, which is the weight corresponding to each channel. The one-dimensional vector after global pooling is regarded as having a global receptive field. In the excitation link, the channel-level dependency is obtained by adaptive recalibration.

[0036] Therefore, after the Sigmoid activation function and a simple gating mechanism, it is satisfied:

[0037] Sigmoid=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z))

[0038] Among them, δ is the ReLU function, F ex () indicates activation operation;

[0039] In order to limit the complexity of the model and help generalization, after two fully connected layers, the first is a dimensionality reduction layer with a reduction ratio of l, and then a ReLU function is used to increase the dimension, and then the channel dimension of the converted output feature map U is returned. The final output weighted feature map is:

[0040] G c =F scale (u c ,s c )=s c u c

[0041] Where G = [G1, G2, ..., G C ],S c Represents the weight value after the complexity and generalization operation of the help model, Fscale (u c ,s c ) is c and feature map u c ∈R H×W The channel-level multiplication between them, that is, multiplication with the corresponding position of the original feature map, finally obtains the weighted feature map.

[0042] Optionally, the convolution kernel size of the dilated convolution is:

[0043] d*(k-1)+1

[0044] Among them, d is the expansion rate, and k represents the size of the convolution kernel before expansion.

[0045] Optionally, the training step of optimizing the HRNet network includes:

[0046] Step 1: Initialize the network parameters, including: the type of tensor category represented in training, the target type of training in class.txt, the number of learning batches, the learning rate, the learning rate decay, and the number of learning times;

[0047] Step 2: Initialize the dataset;

[0048] Store different categories of image data in corresponding folders and generate classification serial numbers for different categories;

[0049] Step 3: Initialize the training network;

[0050] Input the image into the channel of the original resolution to obtain the feature map of the first stage;

[0051] Step 4: Multi-resolution parallel training;

[0052] For the feature map obtained in the first stage, a subnet will sample the original resolution to 1 / 2 in each subsequent stage, connecting the high-resolution to low-resolution subnets in series. Each subnet forms a stage, which consists of a series of convolutions. There is a downsampling layer between adjacent subnets to halve the resolution, which can maintain the representation from high to low resolution throughout the process; starting from the high-resolution subnet as the first stage, gradually increase the high-to-low resolution subnets to form more stages, and connect multi-resolution subnets in parallel;

[0053] Step 5: Cross-resolution feature map sampling;

[0054] The feature maps of sub-networks with different resolutions are sampled by dilated convolution, so that the feature maps obtained after sampling have a larger receptive field. The exchange unit is introduced into the parallel sub-network so that each sub-network repeatedly receives information from other parallel sub-networks.

[0055] Step 6: Cross-resolution feature map fusion;

[0056] The feature maps obtained by the dilated convolution are upsampled or downsampled, and fused after reaching the same resolution. The feature map fusion strategy adopts the channel attention model, which introduces the attention mechanism to perform global average pooling on the feature map of each channel and then shrink the spatial dimension to obtain the corresponding weight value;

[0057] Step 7: Learn deep features and generate model files;

[0058] After completing step 6, the feature map continues to be used as input to the next stage, and steps 3 to 6 are repeated. Through multi-layer feature extraction and feature learning of the deep network, the final feature map is output as the result of classification learning, and the output model is saved.

[0059] Optionally, the method for obtaining the video stream includes video reading or camera shooting.

[0060] Optionally, the data set is: KTH data set.

[0061] Optionally, the human body postures include: walking, jogging, running, boxing, waving and clapping.

[0062] The beneficial effects of the present invention are:

[0063] The human posture recognition method of the present invention takes the high-resolution network HRNet as the basic framework, introduces dilated convolution and attention mechanism compression excitation SE modules, and realizes the improvement of the basic modules of the HRNet network model: first, dilated convolution is added in the cross-channel fusion process of feature maps with different resolutions, which can increase the receptive field without changing the size of the low-resolution feature map without generating additional parameters and calculation amount, and ensure that important information is not ignored when making decisions to ensure recognition accuracy; secondly, a new feature fusion strategy is proposed, which is different from HRNet directly fusing after upsampling. The channel attention mechanism is introduced to perform weighted fusion of feature maps of different resolutions, and the other direction of the feature mapping is adaptively recalibrated to enhance meaningful features and suppress weak features, thereby effectively improving the recognition performance of HRNet, converging faster while keeping the parameter amount basically unchanged, saving computing cost, and further improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0065] Figure 1 It is a flow chart of human body posture recognition of the present invention.

[0066] Figure 2 This is the overall network structure diagram of HRNet.

[0067] Figure 3 It is a block diagram of the optimized HRNet network module of the present invention.

[0068] Figure 4 This is the network structure diagram of the second stage of HRNet.

[0069] Figure 5 This is the BasicBlock network structure diagram.

[0070] Figure 6 Figure 2 is the original module (left) and the SE module (right).

[0071] Figure 7 It is a MEI picture of clapping.

[0072] Figure 8 It is a structural diagram of the dense trajectory algorithm for human pose estimation.

[0073] Fig. 9 This is the HRNet network model diagram.

[0074] Fig.10 It is a video sequence diagram of the KTH dataset. DETAILED DESCRIPTION

[0075] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0076] First, the basic theoretical knowledge involved in the present invention is introduced as follows:

[0077] 1. Human behavior recognition

[0078] The key to human behavior recognition is to extract robust behavior features. Different from the features in the image space, the behavior features of people in the video not only need to describe the appearance of people in the image space, but also must extract the changes in appearance and posture, that is, expand from two-dimensional spatial features to three-dimensional spatiotemporal features. In recent years, many human behavior recognition methods based on RGB data have been proposed, including traditional manual feature extraction methods and deep learning-based methods. With the application of some depth sensors, such as Microsoft's Kinect device, this is because the depth data is more robust to the background environment. The research on these human behavior recognition methods only focuses on a specific aspect, such as methods based on depth data, methods based on deep learning, and methods based on 3D convolution. In addition, many new behavior recognition methods have emerged in recent years, such as methods based on graph convolutional neural networks.

[0079] 1.1 Human posture detection based on spatiotemporal volume

[0080] The methods based on spatiotemporal volume are mainly template matching techniques, but unlike object recognition in image processing, they use three-dimensional spatiotemporal templates for human behavior recognition. The core of these methods is to build a reasonable human behavior template and perform effective matching based on this template. The earliest method used contours to describe the motion information of the human body, and proposed to use motion energy images (MEI) and motion history images (MHI) to represent behavioral characteristics. Figure 7 The MEI and MHI of three different behaviors are shown. The central region of the human body is divided in the MHI using polar coordinates, and the motion context (MC) descriptor based on the scale-invariant feature transform (SIFT) is used to represent the behavior. The histogram of gradient (HOG) features of the image are expanded to the spatiotemporal dimension, and the 3D HOG features are used to describe the human behavior in the video.

[0081] 1.2 Based on trajectory features

[0082] Trajectory-based features use the trajectories of key points or joints in the human skeleton to represent behavior, such as the dense trajectory method (DT) and its improved method (IDT). Figure 8 As shown in the figure; in the video frame, a dense point cloud is sampled, and these feature points are tracked by the optical flow method, the motion trajectory is calculated, and more effective boundary information (MBH) of the moving object is extracted along the trajectory to describe human behavior. Methods for improving the IDT algorithm, such as using split clustering to analyze local motion trajectories, and using the clustering results to represent different motion levels to calculate human behavior characteristics.

[0083] 2. HRNet Deep Network

[0084] HRNet connects high-resolution to low-resolution subnetworks in series. Each subnetwork constitutes a stage, which consists of a series of convolution operations. The resolution between adjacent subnetworks is halved through the downsampling layer. At the same time, different branches transmit information in parallel to maintain high-resolution representation throughout the process. Starting from the high-resolution subnetwork as the first stage, high-resolution to low-resolution subnetworks are gradually added to form more stages, and multi-resolution subnetworks are connected in parallel. The HRNet network model is as follows Fig. 9 shown.

[0085] In the multi-resolution subnet, N sr To represent the subnet of stage s with resolution index r. The high-to-low network with stage S can be represented as:

[0086] N 11 →N 22 →N 33 →N44

[0087] The resolution of the parallel subnetwork in the next stage is composed of the resolution of the previous stage and the resolution of the next stage. For example, a network structure containing 4 parallel subnetworks is expressed as:

[0088] N 11 →N 21 →N 31 →N 41

[0089] ↘N 22 →N 32 →N 42

[0090] ↘N 33 →N 43

[0091] ↘N 44

[0092] Through the whole process, information is repeatedly exchanged on parallel multi-resolution subnetworks, thereby performing repeated multi-scale fusion. Introducing exchange units in parallel subnetworks enables each subnetwork to repeatedly receive information from other parallel subnetworks. Below is an example showing the information exchange scheme. Divide the third stage into three exchange blocks, each block consists of 3 parallel convolution units, and one exchange unit spans the parallel units, as shown below:

[0093] C 1 31↘↗ C 2 31↘↗ C 3 31↘

[0094] C 1 32→ ε 1 3→ C 2 32→ ε 2 3→ C 3 32→ ε 3 3.

[0095] C 1 33↗↘ C 2 33↗↘ C 3 33↗

[0096] Among them C b sr represents the convolutional unit in the rth resolution in the bth block of the Sth stage, ε b sis the corresponding switching unit. Response mapping: {X1, X2, ..., X s} is the input. The output response mapping is: {Y1, Y2, ..., Y s}, whose resolution and width are the same as the input. Function a(X i , k) includes upsampling or downsampling from resolution i to resolution k. Sampling uses a 3×3 convolution kernel with a stride of 2 for 2x downsampling, and two consecutive 3×3 convolutions with a stride of 2 for 4x downsampling. For upsampling, a 1x1 convolution after nearest neighbor upsampling is used to align the number of channels. Each output is an aggregation of the input maps, The cross-stage exchange unit has an additional output map Y s+1 : Y s+1 =a(Y s , s+1).

[0097] For cross-channel feature fusion, when Stage n-1 To Stage n When performing cross-resolution interaction, r is used to represent the fusion input of one stage, X is the fusion output, and D(H) t ,U(H) t They represent t times of downsampling and upsampling of H. Then the formula of the newly fused branch result is as follows:

[0098] f nr (H (n-1)r )=H (n-1)r x=r

[0099] f nr (H (n-1)r )=D(H (n-1)r ) r-x x <r

[0100] f nr (H (n-1)r )=U(H (n-1)r ) x-r x>r

[0101] Most convolutional neural network keypoint heatmap estimation is similar to the classification network, consisting of a sub-network that reduces the resolution, the main body produces a representation of the same resolution of the input, followed by a regressor to estimate the keypoint location estimates of the heatmap, and then convert it to the full resolution. The main body mainly adopts a high-to-low, low-to-high framework, which may strengthen multi-scale fusion and medium (deep) supervision. Most existing fusion schemes aggregate low-level and high-level representations. HRNet utilizes low-resolution representations of the same depth and similar level, performing repeated multi-scale fusion to improve high-resolution representations, and vice versa, so that high-resolution representations also have rich pose estimation capabilities. Therefore, the predicted feature map may be more accurate.

[0102] Embodiment 1:

[0103] This embodiment provides a human posture recognition system based on an optimized HRNet network, the recognition system comprising: a video stream acquisition module, an optimized HRNet network module, and a classification result output module; the video stream acquisition module, the optimized HRNet network module, and the classification result output module are connected in sequence;

[0104] The optimized HRNet network module includes: a basic HRNet module, a dilated convolution module, and an attention mechanism module; the dilated convolution module is located between subnets of different resolutions in the basic HRNet module, and is used to increase the receptive field during the upsampling or downsampling process between subnets of different resolutions; the attention mechanism module is used to perform weighted fusion of feature maps of different resolutions.

[0105] Embodiment 2:

[0106] This embodiment provides a human posture recognition method based on an optimized HRNet network. The method is implemented based on the human posture recognition system described in Embodiment 1 and includes:

[0107] Step 1: Get the video stream;

[0108] Step 2: The optimized HRNet network module obtains the human posture image to be recognized in the video stream, and inputs the image into the channel of the original resolution to obtain a high-resolution subnet feature map, which is the output feature map of the first stage;

[0109] Step 3: downsample the high-resolution feature map to obtain a low-resolution subnetwork feature map with a resolution of 1 / 2 times the original one, and use dilated convolution to increase the receptive field during the sampling process;

[0110] Step 4: Introduce the attention mechanism to perform cross-resolution feature fusion on the feature maps of subnetworks with different resolutions, perform global average pooling on the feature map of each channel, shrink the spatial dimension to obtain the corresponding weight value, and obtain the weighted feature map, which is the output feature map of the second stage;

[0111] Step 5: Use the weighted feature map obtained in step 4 as the input of the next stage, and repeat steps 2 to 4 until the output feature map of the fourth stage is output;

[0112] Step 6: Obtain the final human posture recognition result according to the output feature map of the fourth stage.

[0113] Embodiment three:

[0114] This embodiment provides a human posture recognition method based on an optimized HRNet network. The method is implemented based on the human posture recognition system described in Embodiment 1 and includes:

[0115] Step 1: Obtain the video stream. The method for obtaining the video stream in this embodiment is video reading;

[0116] Step 2: The optimized HRNet network module obtains the human posture image to be recognized in the video stream, and inputs the image into the channel of the original resolution to obtain a high-resolution subnet feature map, which is the output feature map of the first stage;

[0117] Step 3: downsample the high-resolution feature map to obtain a low-resolution subnetwork feature map with a resolution of 1 / 2 times the original one, and use dilated convolution to increase the receptive field during the sampling process;

[0118] In this embodiment, the calculation method of the dilated convolution input and output feature maps is:

[0119]

[0120] Among them, W1 is the input feature map, W2 is the output feature map, d is the expansion rate, p is the number of pixels filled in the convolution, k is the size of the channel, and s is the step size.

[0121] Step 4: Introduce the attention mechanism to perform cross-resolution feature fusion on the feature maps of subnetworks with different resolutions, perform global average pooling on the feature map of each channel, shrink the spatial dimension to obtain the corresponding weight value, and obtain the weighted feature map, which is the output feature map of the second stage;

[0122] The feature maps obtained by the dilated convolution are uniformly upsampled or downsampled, and the feature maps with the same resolution are obtained and then fused. The feature map fusion strategy adopts a channel attention model, and introduces a compression excitation SE module to perform global average pooling on the feature map of each channel and then shrink the spatial dimension to obtain the corresponding weight value;

[0123] The SE module performs cross-channel feature fusion in a weighted manner. When cross-resolution interaction is performed from the n-1 stage to the n stage, b represents the branch sequence number to which the fusion input belongs, and r represents the sequence number of the branch to which the fusion belongs. Down(H nr ) t and Upper(H nr ) t Respectively represent the feature map H nr Perform t times of downsampling and upsampling, then the formula of the newly fused branch result is as follows:

[0124] f nr (H (n-1)r )=H (n-1)rb=r

[0125] f nr (H (n-1)r )=Down(H (n-1)r ) r-b b <r

[0126] f nr (H (n-1)r )=Upper(H (n-1)r ) b-r b>r

[0127] After the SE module, each H nr As input X, X = [x 1 ,x 2 ,……,x C′ ], where x C′ Represents the feature map set of the C′th channel; for each mapping input X∈R H′×W′×C′ To feature map U∈R H×W×C′ The weights corresponding to their features will be calculated separately;

[0128] Among them, H nr is the feature map of the rth branch in the nth stage, f nr (*) represents the sampling operation of the rth branch in the nth stage, H', W', C' are the height, width, and number of channels of the original input feature map, and H and W are the height and width of the output feature map of the SE module, respectively;

[0129] F tr is a convolution operator, and uses V = [v1, v2, ..., v c ] represents the learned filter kernel set, where v c represents the parameter set of the cth filter,

[0130] Finally, the feature map output by the SE module is represented by U = [u1, u2, ..., u c ] indicates that:

[0131]

[0132] Among them, u c is the feature map of the cth channel output by the SE module, u c ∈R H×W , is a two-dimensional spatial kernel representing a single channel acting on the corresponding channel of the input X.

[0133] After obtaining the cross-channel feature map, the convolution feature map is processed to obtain a one-dimensional vector with the same number of channels as the evaluation score of each channel, and then the score is applied to the corresponding channels to obtain the result;

[0134] First, the global spatial information is compressed into a channel descriptor through global pooling; the feature map of each channel has a local receptive field, so each unit of the output feature map U cannot use contextual information outside the area; the weight value set z of the feature map is obtained by compressing the spatial dimension of the feature map U, where the cth element of z is calculated as:

[0135]

[0136] Among them, F sq () indicates compressing the image, u c In (i, j), i and j are the subscripts for the accumulation of the height and width of the output feature map;

[0137] After inputting the feature map, a one-dimensional vector is obtained by global pooling, which is the weight corresponding to each channel. The one-dimensional vector after global pooling is regarded as having a global receptive field. In the excitation link, the channel-level dependency is obtained by adaptive recalibration.

[0138] Therefore, after the Sigmoid activation function and a simple gating mechanism, it is satisfied:

[0139] Sigmoid=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z))

[0140] Among them, δ is the ReLU function, F ex () indicates activation operation;

[0141] In order to limit the complexity of the model and help generalization, after two fully connected layers, the first is a dimensionality reduction layer with a reduction ratio of l, and then a ReLU function is used to increase the dimension, and then the channel dimension of the converted output feature map U is returned. The final output weighted feature map is:

[0142] G c =F scale (u c ,s c )=scu c

[0143] Where G = [G1, G2, ..., G C ],S c Represents the weight value after the complexity and generalization operation of the help model, F scale (uc ,s c ) is c and feature map u c ∈R H×W The channel-level multiplication between them, that is, multiplication with the corresponding position of the original feature map, finally obtains the weighted feature map.

[0144] Step 5: Use the weighted feature map obtained in step 4 as the input of the next stage, and repeat steps 2 to 4 until the output feature map of the fourth stage is output;

[0145] Step 6: Obtain the final human posture recognition result according to the output feature map of the fourth stage.

[0146] The human body postures recognized in this embodiment include: walking, jogging, running, boxing, waving and clapping.

[0147] Before performing human posture recognition in this embodiment, the training model needs to be trained first. The method of the training model is as follows:

[0148] Step 1: Initialize training parameters Initialize the project and set training model parameters

[0149] Modify the relevant data in the configuration file. This project uses a single GPU and default parameters. Then modify the categories of the types you recognize. There are two places: first, the types represented by the tensor categories in training, and second, the target types of training in class.txt. Then modify the training hyperparameters: the number of learning batches, learning rate, learning rate decay, and the number of learning times.

[0150] Step 2: Initialize the dataset

[0151] Store image data of different categories in corresponding folders and generate classification numbers of different categories.

[0152] Step 3: Initialize the training network

[0153] The human body posture picture is input into the network, and the picture size is preprocessed to 3x256x256. The feature map of the first stage is obtained in the maximum resolution channel. At this time, the feature map size is 8x128x128. At this time, the feature map is H 11 .

[0154] Step 4: Multi-resolution parallel training

[0155] For the feature map H obtained in the first stage 11 In the second stage, the network will sample the feature map to 1 / 2 times, connect the high-resolution to low-resolution sub-networks in series, and obtain the feature map H 21 , H 22 , at this time the feature map size H22 32x64x64. In each subsequent stage, a subnetwork will cyclically downsample the original resolution. Each subnetwork forms a stage, which consists of a series of convolutions. There is a downsampling layer between adjacent subnetworks, which halves the resolution and can maintain the representation from high to low resolution throughout the process. Starting from the high-resolution subnetwork as the first stage, high to low resolution subnetworks are gradually added to form more stages, and multi-resolution subnetworks are connected in parallel. If you want to get the branches of the fourth stage, you need to use H 31 (8x128x128),H 32 (32x64x64), H 33 (64x32x32) to generate H 41 (8x128x128),H 42 (32x64x64),H 43 (64x32x32),H 44 (128x16x16). Aggregation method: Direct addition after size matching: H 41 =f1(H 31 )+f2(H 32 )+f3(H 33 ), for feature maps of the same resolution, for f1(H 31 )=H 31 , while f2 and f3 restore the size of the posture feature map and channel transformation through upsampling.

[0156] Step 5: Cross-resolution feature map sampling

[0157] To restore the size of the posture feature map and the channel transformation, the feature maps trained at different resolutions must be fused across resolutions at the end of each stage. The whole process repeatedly exchanges information on parallel multi-resolution subnetworks, thereby performing repeated multi-scale fusion.

[0158] The classification result is often predicted by the output of the last layer. Then, the number of pixels of the original image to which a point on the feature map can be mapped determines the upper limit of the size that the network can detect. Downsampling is required to ensure the receptive field. The result of downsampling is that small targets are not easily detected. Dilated convolution is a special module designed for dense prediction to further improve accuracy. In the sampling process, dilated convolution is introduced to replace the traditional 1x1 convolution, so that the feature map has a larger receptive field. Introducing exchange units in parallel subnets enables each subnet to repeatedly receive information from other parallel subnets. The calculation method of the dilated convolution input (W1) and output (W2) feature map is as follows:

[0159]

[0160] For the task of classifying human posture, it is necessary to aggregate multi-scale contextual information after sampling without losing resolution or analyzing rescaled images. By adding 0-valued pixels between each pixel in each channel, the size of the kernel is increased in a directional manner to increase the receptive field. Let k represent the convolution kernel of a normal convolution. After introducing the expansion rate d, the convolution kernel size increases to:

[0161] d*(k-1)+1

[0162] After dilating the convolution kernel, we can get H from f2 and f3. 31 ,H 32 ,H 33 Perform convolution operation to get H 42 ,H 43 ,H 44 .

[0163] Step 6: Cross-resolution feature map fusion

[0164] The feature maps obtained by the dilated convolution are uniformly restored (upsampled / downsampled) to the same resolution and then fused. The feature map fusion strategy adopts the channel attention model and introduces the compression excitation SE module to perform global average pooling on the feature map of each channel and then shrink the spatial dimension to obtain the corresponding weight value.

[0165] The SE module performs cross-channel feature fusion in a weighted manner. When cross-resolution interaction is performed from the n-1 stage to the n stage, b represents the branch sequence number to which the fusion input belongs, and r represents the sequence number of the branch to which the fusion belongs. Down(H nr ) t and Upper(H nr ) t Respectively represent the feature map H nr Perform t times of downsampling and upsampling, then the formula of the newly fused branch result is as follows:

[0166] f nr (H (n-1)r )=H (n-1)r b=r

[0167] f nr (H (n-1)r )=Down(H (n-1)r ) r-b b <r

[0168] f nr (H (n-1)r )=Upper(H (n-1)r ) b-r b>r

[0169] After the SE module, each H nr As input X, X = [x1 ,x 2 ,……,x C′ ], where x C′ Represents the feature map set of the C′th channel; for each mapping input X∈R H′×W′×C′ To feature map U∈R H×W×C′ The weights corresponding to their features will be calculated separately;

[0170] Among them, H nr is the feature map of the rth branch in the nth stage, f nr (*) represents the sampling operation of the rth branch in the nth stage, H', W', C' are the height, width, and number of channels of the original input feature map, and H and W are the height and width of the output feature map of the SE module, respectively;

[0171] F tr is a convolution operator, and uses V = [v1, v2, ..., v c ] represents the learned filter kernel set, where v c represents the parameter set of the cth filter,

[0172] Finally, the feature map output by the SE module is represented by U = [u1, u2, ..., u c ] indicates that:

[0173]

[0174] Among them, u c is the feature map of the cth channel output by the SE module, u c ∈R H×W , is a two-dimensional spatial kernel representing a single channel acting on the corresponding channel of the input X.

[0175] After obtaining the cross-channel feature map, the convolution feature map is processed to obtain a one-dimensional vector with the same number of channels as the evaluation score of each channel, and then the score is applied to the corresponding channels to obtain the result;

[0176] First, the global spatial information is compressed into a channel descriptor through global pooling; the feature map of each channel has a local receptive field, so each unit of the output feature map U cannot use contextual information outside the area; the weight value set z of the feature map is obtained by compressing the spatial dimension of the feature map U, where the cth element of z is calculated as:

[0177]

[0178] Among them, F sq() indicates compressing the image, u c In (i, j), i and j are the subscripts for the accumulation of the height and width of the output feature map;

[0179] After inputting the feature map, a one-dimensional vector is obtained by global pooling, which is the weight corresponding to each channel. The one-dimensional vector after global pooling is regarded as having a global receptive field. In the excitation link, the channel-level dependency is obtained by adaptive recalibration.

[0180] Therefore, after the Sigmoid activation function and a simple gating mechanism, it is satisfied:

[0181] Sigmoid=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z))

[0182] Among them, δ is the ReLU function, F ex () indicates activation operation;

[0183] In order to limit the complexity of the model and help generalization, after two fully connected layers, the first is a dimensionality reduction layer with a reduction ratio of l, and then a ReLU function is used to increase the dimension, and then the channel dimension of the converted output feature map U is returned. The final output weighted feature map is:

[0184] G c =F scale (u c ,s c )=s c u c

[0185] Where G = [G1, G2, ..., G C ],S c Represents the weight value after the complexity and generalization operation of the help model, F scale (u c ,s c ) is c and feature map u c ∈R H×W The channel-level multiplication between them, that is, multiplication with the corresponding position of the original feature map, finally obtains the weighted feature map.

[0186] Step 7: Learn deep features and generate model files

[0187] After the end of step 6, the feature map continues to be used as input to the next stage, and steps 3 to 6 are repeated. Through multi-layer feature extraction and feature learning of the deep network, the final feature map is output as the result of classification learning, and the output model is saved.

[0188] Step 8: Use the weight file to detect posture and achieve posture classification

[0189] Use the trained model to load its weight file to detect human posture.

[0190] In order to further illustrate the beneficial effects that can be achieved by the human body posture recognition method of the present application, the following experiment is conducted:

[0191] 1. Experimental conditions and parameters

[0192] The dataset used in the present invention is the KTH dataset. The video sequences contain six different types of human behaviors: walking, jogging, running, boxing, waving and clapping. There are changes in the environment and objects, which helps to determine the accuracy of our proposed method. Each action corresponds to 25 different objects performing multiple times in a variety of different scenes, with different changes in the proportions of indoor and outdoor and clothing. All action sequences consider static and uniform backgrounds, with a video frame rate of 25 frames per second, a resolution of 160×120 pixels, and an average duration of actions in the video of 4 seconds. There are 25 videos for each action in four different categories. Some snapshots of video sequences from the KTH dataset are shown in Figure 1. Fig.10 shown.

[0193] 2. Experiment and result analysis

[0194] This application method is implemented using PyCharm Community Edition 2021.1.2x64, and runs on a workstation with an Intel Core i7-8700, 3.2GHz, 12 cores, 16GB of memory, and an NVIDIA Geforce GTX 1080Ti graphics card. The performance is compared and analyzed with the HRNet network method proposed in the paper "Deep High-ResolutionRepresentation Learning for Human Pose Estimation" published by Ke Sun Wang et al. in 2020. Since the original HRNet method lacks a global feature attention channel mechanism, SE is also added in this experiment for detection, and the experimental results are as follows.

[0195] (1) Ablation analysis

[0196] In order to verify the influence of the added channel attention module and dilated convolution on the feature extraction capability of HRNet and the accuracy of human posture prediction, the network structures with and without attention mechanism were constructed for ablation experiments. The network model without attention mechanism used 1x1 convolution for cross-channel feature fusion. The experiment was trained on the KTH dataset and verified on the validation set, and no pre-trained model was loaded.

[0197] The ablation experiment observed the results through two comparisons. The first was the performance comparison of 0-50 rounds without loading the pre-trained model. The experimental results are shown in Table 1. The control variable method was used to conduct four groups of ablation experiments A, B, C, and D. The network model without the attention mechanism only used dilated convolution to improve the basic module in the HRNet network model. Compared with the HRNet network model, the number of parameters and computational complexity did not increase. Compared with the network model with the attention mechanism, the network model without the attention mechanism used dilated convolution modules.

[0198] Table 1 Ablation experiment

[0199]

[0200] The experimental results show that when the compression excitation SE module is not added to the HRNet network, the overall performance of the model only decreases by 1.6%; after adding the compression excitation SE module, not only does it not increase the number of parameters and computational complexity of the network model, but it can also ensure the performance of the network model and achieve the purpose of improving accuracy. In addition, the network model with attention mechanism does not have a significant impact on the computational complexity of the model after adding dilated convolution, and the accuracy is improved by 0.5%.

[0201] (2) Performance comparison

[0202] The performance of the network model proposed in this paper on the KTH dataset is shown in Table 2. The KTH dataset based on human posture classification has fewer types of actions, but they have high similarity in actions. After removing the time correlation, the deep features of many actions are difficult to learn. Since the posture estimation algorithms are different, including histograms, visual word bags, and some current mainstream networks and their improved versions, we will compare them only in terms of performance, regardless of the computational complexity and the number of parameters:

[0203] Table 2 Comparative experiment

[0204]

[0205] Tables 3 and 4 show different combinations of feature extraction and classifier techniques for the KTH dataset. The results show that the classification accuracy is improved in all categories compared to the native HRNet network. There are only a few classification ambiguities, which come from actions such as boxing, waving, and clapping. In addition, a small number of actions such as running, walking, and jogging are also misclassified by the classifier.

[0206] Table 3 HRNet classification results

[0207]

[0208] Table 4 Optimized HRNet classification results

[0209]

[0210] It can be seen that the recognition accuracy of different categories has been improved overall, basically by more than 1%. Among them, a small number of the three categories of slow walking, running and walking were misjudged. On the one hand, it is due to the continuity of the video frames in the dataset itself, which leads to the misjudgment of similar actions in the posture preparation process. On the other hand, the three postures themselves are difficult to distinguish by only judging the data stream captured by the video frames.

[0211] In summary, the present invention is a human posture recognition method of an embodiment, which can effectively improve the recognition accuracy without increasing the parameter amount and computational complexity of the network model; and the present invention introduces a channel attention mechanism to perform weighted fusion of feature maps of different resolutions, and adaptively recalibrates another direction of the feature map to enhance meaningful features while suppressing weak features, with a faster convergence speed, which can further reduce the computational cost and speed up the recognition speed.

[0212] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0213] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A human posture recognition method based on an optimized HRNet network, characterized in that: The method comprises: Step 1: Get the video stream; Step 2: The optimized HRNet network module obtains the human posture image to be recognized in the video stream, and inputs the image into the channel of the original resolution to obtain a high-resolution subnet feature map, which is the output feature map of the first stage; Step 3: downsample the high-resolution feature map to obtain a low-resolution subnetwork feature map with a resolution of 1 / 2 times the original one, and use dilated convolution to increase the receptive field during the sampling process; Step 4: Introduce the attention mechanism to perform cross-resolution feature fusion on the sub-network feature maps of different resolutions obtained in step 3, perform global average pooling on the feature map of each channel, shrink the spatial dimension to obtain the corresponding weight value, and obtain a weighted feature map, which is the output feature map of the second stage; Step 5: Use the weighted feature map obtained in step 4 as the input of the next stage, and repeat steps 2 to 4 until the output feature map of the fourth stage is output; Step 6: Obtain the final human posture recognition result according to the output feature map of the fourth stage; The calculation process of step 4 includes: The feature maps obtained by the dilated convolution are uniformly upsampled or downsampled, and the feature maps with the same resolution are obtained and then fused. The feature map fusion strategy adopts the channel attention model, and introduces the compression excitation module SE to perform global average pooling on the feature map of each channel and then shrink the spatial dimension to obtain the corresponding weight value; The SE module performs cross-channel feature fusion in a weighted manner. When cross-resolution interaction is performed from the n-1 stage to the n stage, b represents the branch sequence number to which the fusion input belongs, and r represents the sequence number of the branch to which the fusion belongs. Down(H nr ) t and Upper(H nr ) t Respectively represent the feature map H nr Perform t times of downsampling and upsampling, then the formula of the newly fused branch result is as follows: f nr (H (n-1)r )=H (n-1)r b=r f nr (H (n-1)r )=Down(H (n-1)r ) r-b b<r f nr (H (n-1)r )=Upper(H (n-1)r ) b-r b>r After the SE module, each H nr As input X, X = [x 1 ,x 2 ,……,x C′ ], where x C′ Represents the feature map set of the C′th channel; for each mapping input X∈R H′×W′×C′ To feature map U∈R H×W×C′ The weights corresponding to their features will be calculated separately; Among them, H nr is the feature map of the rth branch in the nth stage, f nr (*) represents the sampling operation of the rth branch in the nth stage, H', W', C' are the height, width, and number of channels of the original input feature map, and H and W are the height and width of the output feature map of the SE module, respectively; F tr is a convolution operator, corresponding to the conv2D operation in the network layer, using V = [v1, v2, ..., v c ] represents the learned filter kernel set, where v c represents the parameter set of the cth filter, Finally, the feature map output by the SE module is represented by U = [u1, u2, ..., u c ] indicates that: Among them, u c is the feature map of the cth channel output by the SE module, u c ∈R H×W , is a two-dimensional spatial kernel representing a single channel acting on the corresponding channel of the input X; The calculation process of the weighted feature map includes: After obtaining the cross-channel feature map, the convolution feature map is processed to obtain a one-dimensional vector with the same number of channels as the evaluation score of each channel, and then the score is applied to the corresponding channels to obtain the result; First, the global spatial information is compressed into a channel descriptor through global pooling; the feature map of each channel has a local receptive field, so each unit of the output feature map U cannot use contextual information outside the area; the weight value set z of the feature map is obtained by compressing the spatial dimension of the feature map U, where the cth element of z is calculated as: Among them, F sq () indicates compressing the image, u c In (i, j), i and j are the subscripts for the accumulation of the height and width of the output feature map; After inputting the feature map, a one-dimensional vector is obtained by global pooling, which is the weight corresponding to each channel. The one-dimensional vector after global pooling is regarded as having a global receptive field. In the excitation link, the channel-level dependency is obtained by adaptive recalibration. Therefore, after the Sigmoid activation function and a simple gating mechanism, the following is satisfied: Sigmoid=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z)) Among them, δ is the ReLU function, F ex () indicates activation operation; In order to limit the complexity of the model and help generalization, after two fully connected layers, the first is a dimensionality reduction layer with a reduction ratio of l, and then a ReLU function is used to increase the dimension, and then the channel dimension of the converted output feature map U is returned. The final output weighted feature map is: G c =F scale (u c ,s c )=s c u c Where G = [G1, G2, ..., G C ],S c Represents the weight value after the complexity and generalization operation of the help model, F scale (u c ,s c ) is c and feature map u c ∈R H×W The channel-level multiplication between them, that is, multiplication with the corresponding position of the original feature map, finally obtains the weighted feature map.

2. The human body posture recognition method according to claim 1, characterized in that: The calculation method of the dilated convolution input and output feature maps is: Among them, W1 is the input feature map, W2 is the output feature map, d is the expansion rate, p is the number of pixels filled in the convolution, k is the channel size, and s is the step size.

3. The human body posture recognition method according to claim 1, characterized in that: The convolution kernel size of the dilated convolution is: d*(k-1)+1 Among them, d is the expansion rate, and k represents the size of the convolution kernel before expansion.

4. The human body posture recognition method according to claim 1, characterized in that: The training steps of optimizing the HRNet network include: Step 1: Initialize the network parameters, including: the type of tensor category represented in training, the target type of training in class.txt, the number of learning batches, the learning rate, the learning rate decay, and the number of learning times; Step 2: Initialize the dataset; Store different categories of image data in corresponding folders and generate classification serial numbers for different categories; Step 3: Initialize the training network; Input the image into the channel of the original resolution to obtain the feature map of the first stage; Step 4: Multi-resolution parallel training; For the feature map obtained in the first stage, a subnet will sample the original resolution to 1 / 2 in each subsequent stage, connecting the high-resolution to low-resolution subnets in series. Each subnet forms a stage, which consists of a series of convolutions. There is a downsampling layer between adjacent subnets to halve the resolution, which can maintain the representation from high to low resolution throughout the process; starting from the high-resolution subnet as the first stage, gradually increase the high-to-low resolution subnets to form more stages, and connect multi-resolution subnets in parallel; Step 5: Cross-resolution feature map sampling; The feature maps of sub-networks with different resolutions are sampled by dilated convolution, so that the feature maps obtained after sampling have a larger receptive field. The exchange unit is introduced into the parallel sub-network so that each sub-network repeatedly receives information from other parallel sub-networks. Step 6: Cross-resolution feature map fusion; The feature maps obtained by the dilated convolution are upsampled or downsampled, and fused after reaching the same resolution. The feature map fusion strategy adopts the channel attention model, which introduces the attention mechanism to perform global average pooling on the feature map of each channel and then shrink the spatial dimension to obtain the corresponding weight value; Step 7: Learn deep features and generate model files; After completing step 6, the feature map continues to be used as input to the next stage, and steps 3 to 6 are repeated. Through multi-layer feature extraction and feature learning of the deep network, the final feature map is output as the result of classification learning, and the output model is saved.

5. The human body posture recognition method according to claim 1, characterized in that: The method for obtaining the video stream includes video reading or camera shooting.

6. The human body posture recognition method according to claim 4, characterized in that: The data set is: KTH data set.

7. The human body posture recognition method according to claim 1, characterized in that: The human body postures include: walking, jogging, running, boxing, waving and clapping.

8. A human posture recognition system based on an optimized HRNet network, characterized in that: The recognition system is used to implement the human posture recognition method according to any one of claims 1 to 7, comprising: a video stream acquisition module, an optimized HRNet network module, and a classification result output module; the video stream acquisition module, the optimized HRNet network module, and the classification result output module are connected in sequence; The optimized HRNet network module includes: a basic HRNet module, a dilated convolution module, and an attention mechanism module; the dilated convolution module is located between subnets of different resolutions in the basic HRNet module, and is used to perform dilated convolution in the upsampling or downsampling process between subnets of different resolutions to increase the receptive field; the attention mechanism module is used to perform weighted fusion of feature maps of different resolutions.

Images