A human posture estimation method suitable for complex scenes in coal mines

Abstract

The application provides a human body posture estimation method suitable for a complex scene in a coal mine, from the perspective of fusion of time characteristics and space characteristics, a top-down posture estimation framework is designed, a human body detector is used to explore the spatial position of a person on each frame of image, the person is cut from the original frame of image, and then is sent into a posture estimation network for detection, a time road module and a space road module are adopted in the posture estimation network, the input of the time road module is image information cut by a human body detection algorithm, a grouping interval frame insertion strategy is designed to encode upper and lower frame information, and the feature information of a joint in the image in multiple historical frames is fused through a weighted manner to supplement the feature of a disturbed frame. The application expands the YH-Pose framework through various ingenious modules and strategies, so that the human body posture estimation can be performed on the video frame of the coal mine under the uneven brightness, the blurred detail texture and the more noise.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, deep learning and pose estimation, and is mainly applied in the fields of human motion analysis, human-computer interaction, security monitoring, sports competition and human pose estimation, and particularly relates to a human pose estimation method suitable for a complex scene in a coal mine. BACKGROUND

[0002] In today's information age, the Internet technology is developing rapidly. The research of artificial intelligence on human data mainly includes human pose estimation, pedestrian re-identification, behavior analysis, face detection, etc. Among them, human pose estimation as a basic task in the recognition task plays an irreplaceable role in the field of computer vision, which can make the computer learn and understand human actions and infer human behaviors. Human pose estimation is defined as predicting the positions of each skeletal joint of the human body in an image or video, such as shoulders and knees.

[0003] The early traditional method of human pose estimation is a model-based method and a feature-based method. Cherian et al. decomposed the limbs based on a graph structure model, thereby generating individual limb sequences, and then recombined the limb sequences to realize the reconstruction of the human shape. With the rapid development of deep learning technology, recent methods are built on deep convolutional neural networks, and there are different methods according to different features: (1) top-down method (from high-level feature abstraction to low-level pixel evidence), Hao-Shu Fang et al. proposed a method of AlphaPose, which first detects the person in the image and then uses a single-person pose estimation network to predict the key points in all images; (2) bottom-up method (from low-level pixel evidence to high-level abstraction), Zhe Cao et al. first proposed an openpose method, which uses the partial affinity between joints to detect each joint in a bottom-up manner, and then associates them with individuals; (3) regression-based method (from input image to body joint position), Carreira et al. first proposed a heat map regression method, which first obtains the heat map of the human joint position on the image, and then uses the heat map as a feature map to realize the prediction of the multi-stage skeletal point position.

[0004] Some researchers also grasp the continuity and geometric consistency between video frames by designing time convolutional neural networks such as long short-term memory network (LSTM), gated recurrent unit (GRU) or 3D CNN to capture the inter-frame information. Wentao Jiang et al. use time convolutional neural network to capture time clues and directly predict the key point heat map sequence in the video. Yue Xu et al. propose a 3DHRNet for extracting spatial and temporal features across video frames to estimate the pose sequence, which shows excellent results. Philipp et al. use optical flow method to calculate the dense optical flow between two frames, and use additional flow-based method to realize frame alignment and prediction.

[0005] Their design idea is based on the design of static images, although different ways are used to extract the feature information of the human body joint points in static images, but the information connection between frames in the video is not considered, which causes the phenomenon that the human body posture cannot be detected in some disturbed frames when detecting the video in the complex scene of coal mine.

[0006] When the human motion changes greatly between frames in the complex scene video, the interval between the predicted joint point positions becomes quite large, and when the interval continues to increase, the network will lose the ability to learn to connect such distant information, which will cause the problem of missing posture data and the human body joint points cannot be detected. At the same time, when there is much noise in the video, the effect of the optical flow method will also be weakened. By extracting the feature information of the blurred image from the resolution angle, but the repeated resolution conversion, the information of the feature map is easy to lose, which will reduce the accuracy of the result. SUMMARY

[0007] In view of the defects and problems existing in the prior art, a new human pose estimation framework is proposed to be applicable to the human pose estimation task in the complex scene of coal mine, to make up for the shortcomings of the current mainstream algorithm, to improve the detection accuracy, and to solve the problem of uneven image brightness.

[0008] The solution adopted by this invention to solve its technical problem is: a human pose estimation method suitable for complex coal mine scenes. This method framework includes a human detector module, an SA-HRNet module, a temporal path module, and a spatial path module, and includes the following steps: First, the human detector module is used to explore the spatial location of the human image in each frame, and the human image is cropped from the original frame image. Then, it is fed into the pose estimation network for detection. In the pose estimation network, the temporal path module and the spatial path module are used to extract the features of the human skeletal joints. The input of the temporal path module is the image information cropped by the human detection algorithm. Here, a grouped interval frame interpolation strategy is designed to... The next frame information is encoded, and the feature information of key points in multiple historical frames is fused in a weighted manner to supplement the features of the interfered frame. The interfered frame refers to certain frames in the complex coal mine scene video that have problems such as uneven brightness, blurred image details and textures, and a lot of noise. The spatial road module is built by convolutional neural networks and deformable convolutional networks, and the convolutional networks of different resolutions are connected in a parallel structure. A spatial attention module is also introduced, which is integrated into the multi-resolution fusion stage of each sub-network. The grouped interval frame interpolation strategy is to cut the continuous video frame sequence into a group of n frames {F1, F2, ... F... n Then, a set of n consecutive low-resolution video frames is randomly selected as the learning frames, where the nth video frame F n As the reference frame, the (n+1)th video frame F n+1 The frame to be detected is defined as the frame to be detected, and the previous frame of the frame to be detected with similar and matching features is defined as the reference frame. The framework learns human pose features from the time and space dimensions through heatmap estimation and predicts the poses of multiple people in complex scenes.

[0009] The implementation method of grouped interval frame interpolation is as follows: First, a YOLOv7 human detection module is introduced into n consecutive video frames. The YOLOv7 module will detect human bodies in the video frames and mark them with bounding boxes, such as... Figure 3 As shown, the bounding boxes are then used to crop out local frames containing only human figures from the original video frames. These cropped local human figures are then bound together into a continuous frame sequence and fed into the SA-HRNet network. The SA-HRNet network outputs a preliminary keypoint heatmap estimate H. t, 1<t<n; in the time road module, n video frames are decoupled into 3 groups according to the degree of adjacency with the reference frame, each group represents a different frame rate, and the intervals are different; the similarity of each frame and the reference frame is calculated in each group, and the corresponding weight key is generated, and the weight value is assigned to selectively extract the posture feature information in the time dimension between different frames, and the posture feature information between groups is further fused by accumulation and multiplication; finally, in the spatial road module, the fused features output by the time road module network are stacked into a 3x3 basic block and then input into a network built by two-dimensional convolution and deformable convolution for feature re-extraction, and finally the position of the key point on the frame to be detected is predicted by the way of heat map regression, and the key points are drawn according to the skeleton sequence to generate the two-dimensional posture of the human body in the frame to be detected.

[0010] The human body detector module adopts a top-down manner, first detects the human in the image, then adds a target frame to the human body, and uses YOLOv7 to learn the features of the human in the complex scene to generate a weight capable of detecting the human body in the complex scene. The detector detects the position information of each human body in the image, i.e. the coordinates of the top point of the boundary box (x, y), the width w and the height H of the boundary box, and the class confidence c; for the boundary box of a human, five elements are predicted, and the overall prediction vector is defined as formula (1) P = {C x ,C y ,W,H,box conf} (1).

[0011] According to the element information predicted and output by YOLOv7, the 2D coordinate values of the four vertices of the boundary box are obtained, in order to obtain more feature information while avoiding redundant features and increasing the amount of calculation, the boundary box is enlarged by β times, as formula (2) is enlarged , and the coordinate values of the four enlarged vertices are obtained

[0012]

[0013] The SA-HRNet network continues to use the parallel mode of the HRNet network to connect convolution networks of different resolutions, so that the input of the entire network maintains high resolution from beginning to end, and the feature information between different resolutions can repeatedly perform multi-scale information fusion, so that the effect of pose estimation is improved. In the process of human pose estimation, the SA-HRNet network first detects the approximate position of the target human body, and then detects the specific position of all key points of the human body. In the process of human key point detection, a regressor is used to estimate the position of each key point of the human body. After estimating all the key points, the key points are regressed to the original image in the form of a heat map, and a convolution layer is added to the last layer of the HRNet-W48 model, so that the obtained key point position is converted into a full-resolution heat map. The heat map calculation is shown in formula (3) and formula (4), and the true heat map value is represented by H gt The estimated heat map value is represented by H pred The mean square error of the two is represented as L H , which represents the overall loss function, and the loss calculation is shown in formula (4). The center point position of the true heat map calculated by the above formula is the position of the key point of the human body to be detected. W ′ and H ’ are the width and height of the heat map respectively, K is the number of channels, and c is the standard deviation.

[0014]

[0015]

[0016] The spatial attention module is integrated into the HRNet at different stages and is used when the resolution is reduced. The receptive field is expanded, and more ignored or missed feature information is effectively extracted.

[0017] The time road module: by comparing the predicted frame with the previous historical frame, dynamically aggregating and learning the similar pose feature information of the historical frame and the predicted frame, the SA-HRNet network extracts the heat map features of each historical frame, and then the time grouping module divides the historical frame into three groups with different frame numbers from the reference frame; the features are fused by weighting within the group and accumulated and multiplied between the groups; different fusion strategies are performed synchronously in the whole encoding propagation process; the time grouping module: based on the time distance from the reference frame, each group is divided into N frames, N∈[1,3], so that the original sequence is reordered as

[0018] group1: {F5, F6, F7}, group2: {F3, F5, F7}, group3: {F1, F4, F7}, each group is a subsequence of the original video frame;The historical frame key point heat map feature outputted after the time road module is fused is marked as H fusion Then the pose feature of each channel is stacked into a 3x3 basic block, which facilitates input into a new network for a new round of pose feature learning.

[0019] The convolutional neural network constructs three parallel 2D convolutional layers to extract the stacked historical frame key point heat map feature H fusion The convolutional feature outputted by the CNN contains rich semantic information and can more comprehensively represent object features;First, the input layer is normalized and resized, the size of the input layer is changed to 128x128x3, the convolution kernel size is 3x3, the stride is 2, and the padding operation is performed;A nonlinear unit Relu activation function is used in each convolution process to activate;Then, an average pooling layer is used to downsample the high-dimensional pose feature information to obtain low-dimensional feature information;The kernel of the average pooling layer is 2x2, and the stride is 2. The feature H fusion outputted by the convolutional neural network has a size of 3x3x256;

[0020]

[0021] The time grouping module grouping method of the present application can explicitly and efficiently integrate historical adjacent frames at different time distances, for two reasons:(1) Different time distance historical frames contain different degrees of pose feature information similar to the frame to be detected, and the contribution to network learning pose feature to predict the frame to be detected is also unequal, especially for frames with large deformation, occlusion and motion blur. When a certain area in a group is blocked (e.g. occluded), the missing information can be recovered by other groups, i.e. the information of different groups complements each other, which is also the key to pose detection in complex situations.(2) The reference frame in each group guides the model to extract useful information from adjacent frames, achieving efficient information extraction and fusion.

[0022] The present application proposes a new human pose estimation framework to be applicable to the human pose estimation task in the complex scene of coal mines, makes up for the shortcomings of the current mainstream algorithm, improves the detection accuracy, and solves the problem that the current mainstream algorithm cannot detect the human pose due to the uneven brightness of the image, the blurred image detail texture, and the more noise caused by the dim light in the coal mine, the strong light interference of the miner's overhead cold light lamp, coal dust, water mist, and transmission equipment. The YH-Pose framework is expanded by various ingenious modules and strategies, so that the human pose estimation can be performed on the video frames with uneven brightness, blurred detail texture, and more noise in the coal mine. Attached Figure Description

[0023] Figure 1 This is the overall framework diagram of the YH-Pose pose estimation technology solution;

[0024] Figure 2 This is an overview of spatiotemporal feature extraction using the YH-Pose pose estimation network;

[0025] Figure 3 It is the original human bounding box and the manually bounding box enlarged by β.

[0026] Figure 4 This is a schematic diagram of the Non-localblock module;

[0027] Figure 5 It is the SA-HRNet network after incorporating the Non-localblock module. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] This invention provides a human pose estimation method applicable to complex coal mine scenarios. From the perspective of fusing temporal and spatial features, a top-down pose estimation framework (YH-Pose, as shown below) is designed. Figure 1 This is used to estimate the poses of multiple people in complex coal mine scene video frames and predict the spatial location information of human skeleton points in each video frame.

[0030] First, a human detector is used to locate the spatial position of people in each frame of the image. The person is then cropped from the original frame and fed into a pose estimation network for detection. The pose estimation network consists of two modules that extract the features of the human skeletal joints: the temporal path module (PRM) and the spatial path module (SRM). The input to the temporal path module is the image information cropped by the human detection algorithm. A grouped interleaving strategy is designed to encode the information of consecutive frames. The feature information of joints from multiple historical frames is fused using a weighted method to supplement the features of the interfered frames. These interfered frames refer to frames in complex coal mine scene videos that exhibit uneven brightness, blurred image details and textures, and excessive noise. The spatial path module is built using convolutional neural networks and deformable convolutional networks, with convolutional networks of different resolutions connected in parallel. To address the low efficiency and insufficient accuracy of the HRNet network in human pose pre-extraction, a spatial attention module is introduced, integrated into the multi-resolution fusion stage of each sub-network. The specific steps for implementing the above scheme are as follows.

[0031] I.The application is a human pose estimation method suitable for complex scenes in coal mines, and the method framework mainly includes four modules, namely a human detector module, a SA-HRNet module, a PRM module and a SRM module.

[0032] II.Overall technical solution:

[0033] The continuous video frame sequence is cropped into a group of 7 frames {F1, F2,... F7}, and then a group of 7 low-resolution continuous video frames is randomly selected as learning frames, wherein the seventh frame video frame F7 is the reference frame and the eighth frame video frame F8 is the frame to be detected. The purpose of this is to enable the network to fully learn the pose feature information of the historical frames, so that the prediction result is more accurate, and the previous frame of the frame to be detected has high similarity and matching degree of feature information with the frame to be detected, so it is set as the reference frame.

[0034] This is a way of learning human pose features from the time dimension and the space dimension through heat map estimation, and a framework for predicting multiple human poses in complex scenes. First, a human detector YOLOv7 is introduced for the continuous 7 frames of video frames, which detects the human bodies in the video frames and labels them with bounding boxes, then further uses the bounding boxes to crop the local frames with only human bodies from the original video frames, and then rebinds the cropped local human body frames into a continuous frame sequence and sends it into the SA-HRNet network. The SA-HRNet network outputs the preliminary key point heat map estimation H t (1<t<7). In the time road module (PRM), the seven video frames are decoupled into three groups according to their proximity to the reference frame, each group representing a different frame rate and being separated by different frames. In each group, the feature similarity between each frame and the reference frame is calculated, and the corresponding weight key is generated. By assigning weights, the pose feature information in the time dimension between different frames is selectively extracted, and the pose feature information between groups is further fused by accumulation and multiplication. Finally, in the spatial road module (SRM), the fused features output by the PRM network are stacked into 3x3 basic blocks and then sent into a network built by two-dimensional convolution and deformable convolution for feature re-extraction. Finally, the position of the key point on the frame to be detected is predicted by heat map regression, and the key points are drawn according to the skeleton sequence to generate the two-dimensional pose of the human body in the frame to be detected.

[0035] III.Human detector module:

[0036] Using a top-down approach, people in the image are detected first, and then a bounding box is added to the human body. YOLOv7 is a state-of-the-art object detector based on the YOLO model family. It learns the features of people in complex scenes and then generates weights that can detect people in complex scenes. The detector detects the position information of each human body in the image, namely the coordinates (x, y) of a vertex of the bounding box, the width w and height H of the bounding box, and the class confidence c. For a person's bounding box, five elements are predicted, and the overall prediction vector is defined as shown in formula (1).

[0037] P={C x C y ,W,H,box conf} (2)

[0038] Based on the element information predicted by YOLOv7, obtain the four vertices of the bounding box. To obtain more feature information while avoiding redundant features that would increase computational load, the bounding box was enlarged by a factor of β, as shown in formula (2). Expand the area to obtain the coordinates of the four vertices after the expansion. Based on the alignment with the original video frames, the corresponding video frames are cropped from the original video and reconstructed into a continuous video frame sequence.

[0039]

[0040] IV.SA-HRNet module:

[0041] The SA-HRNet network continues to use the parallel approach of the HRNet network to connect convolutional networks of different resolutions. This ensures that the input of the entire network remains high-resolution from beginning to end, and the feature information between different resolutions can be repeatedly fused at multiple scales, thereby improving the pose estimation effect. Its general process in human pose estimation is as follows: First, the approximate position of the target human body is detected, and then the specific positions of all key points of the human body are detected. During the detection of human key points, a regressor is used to estimate the position of each key point. After estimating all key points, the key points are regressed to the original image in the form of a heatmap. A convolutional layer is added to the last layer of the HRNet-W48 model to realize the conversion of the obtained key point positions into a full-resolution heatmap. The heatmap calculation is shown in formulas (3) and (4). The actual heatmap value is calculated by H gt This indicates that the heatmap values ​​obtained through prediction are derived from H. pred The mean square error of both can be expressed as L. H , which represents the overall loss function, and the loss calculation is shown in formula (4). The center point of the actual heatmap calculated by the above formula is the location of the key points of the human body to be detected. W′ and H ’ are the width and height of the heat map, respectively, K is the number of channels, and c is the standard deviation.

[0042]

[0043]

[0044] In addition, the spatial attention module (Non-local block) is shown as Figure 4 which is integrated into the middle of different stages of HRNet and down-sampling at low resolution, so as to expand the receptive field and more effectively extract more ignored or missing feature information.

[0045] V. PRM module:

[0046] The core idea of the time road module (PRM) is to compare the frame with the previous historical frame, dynamically aggregate and learn the similar pose feature information of the historical frame and the predicted frame. The SA-HRNet network extracts the heat map features of each historical frame, and then the time grouping module divides the historical frames into three groups with different frame numbers from the reference frame. The features are fused by intra-group feature weighted fusion and inter-group feature accumulation and multiplication. Different fusion strategies are synchronized in the whole encoding propagation process. As shown in Figure 5 .

[0047] 1. Time grouping module: Based on the time distance to the reference frame, each group is divided into N frames, N ∈ [1, 3]. In this way, the original sequence is reordered as group1: {F5, F6, F7}, group2: {F3, F5, F7}, group3: {F1, F4, F7}. Each group is a subsequence of the original video frame. It is worth noting that the method can be easily extended to any frame as input. This grouping method can explicitly and efficiently integrate the historical adjacent frames with different time distances. The reasons are as follows: (1) The historical frames with different time distances contain different degrees of similarity to the pose feature information of the frame to be detected, and their contributions to the network learning pose features to predict the frame to be detected are also unequal, especially for frames with large deformation, occlusion and motion blur. When a certain region in a group is blocked (e.g. occluded), the missing information can be recovered by other groups, that is, the information of different groups complements each other, which is also the key to pose detection in complex situations. (2) The reference frame in each group guides the model to extract useful information from adjacent frames, achieving efficient information extraction and fusion.

[0048] 2. Similarity fusion module: This module is applied to the weighted fusion of posture features in the group. The similarity here refers to the similarity between the posture heat map features extracted from the historical frames in the group and the reference frame after encoding. For example, the historical frames with a long time distance from the reference frame are different people in different time periods, and naturally their posture heat map features have weak correlation. On the contrary, the historical frames with a short time distance have strong correlation. First, the heat map features H m are converted into key space key and value space value, the key space is used for similarity comparison to generate similarity weight, and the value space is used for feature aggregation.

[0049]

[0050] Here, ρ k and ρ v are feature extraction layers in the SA-HRNet network, which are composed of different convolutional layers, and then the cosine similarity is calculated to measure the similarity between the reference frame heat map feature and the heat map feature of other historical frames in the group , as shown in equation (6)

[0051]

[0052] In this way, a series of weight values [μ m , u 7-N , μ7] are calculated, and then the calculated weight values are normalized by softmax, so a series of attention masks are obtained. Since each represents the similarity between the historical frames in the group and the reference frame, the value features of the historical frames in the group and the reference frame can be aggregated to obtain the aggregated features S:

[0053]

[0054] At this time, the posture heat map feature fusion in each group has been completed, and then addition (Add), multiplication (Mul) and other operations are performed to fuse the features between different groups and output VI.SRM module:

[0055] The historical frame key point heat map feature output by the PRM module is denoted as H fusion , and then the posture features of each channel are stacked into a 3x3 basic block, which facilitates input to the new network for a new round of posture feature learning.

[0056] 1. Convolutional neural network: Three parallel 2D convolutional layers are constructed to extract the stacked historical frame key point heat map features H fusion, the CNN output convolutional features contain rich semantic information, which can more comprehensively represent the object features. First, the input layer is normalized and resized to change the size of the input layer to 128x128x3, the convolution kernel size is 3x3, the stride is 2, and the padding operation is performed. A nonlinear unit Relu activation function is used in each convolution process to activate. Then the average pooling layer is used here, which down-samples the high-dimensional pose feature information to obtain low-dimensional feature information. The kernel of the average pooling layer is 2x2, and the stride is 2. The size of the feature H fusion output by the convolutional neural network is 3x3x256.

[0057]

[0058] 2. Deformable convolutional network: DCN-v2

[26] convolutional method as formula (8), here the deformable convolution kernel is 3x3, p k , W k is the k-th position in a convolution kernel and the corresponding convolution parameter, respectively, where p k ∈{(-1, -1),..., (0, 0),..., (1, 1)}, K = 9 represents the number of convolution points in a convolution kernel, p0 is the position on H fusion , then p0 + p k + Δp k represents the new sampling position to be convolved, X, Y are the input pose feature H fusion and the output pose feature H fusion , respectively, Δp is the learned offset, and Δm is a parameter for modulating the network to control the offset amplitude, so that the new position of the pose heat map feature sampling will be better moved to the region of interest, such as the feature of the complex region. The offset, Δm both need an additional CNN convolutional neural network to learn it, so the DCN-v2 network is concatenated after each CNN network. The three groups of offsets of the subsequent deformable convolutional network are learned using the convolutional neural network, and the form of the offset is calculated as formula (9)

[0059]

[0060] Different dilations correspond to different receptive field sizes, here three different dilations (d = 3, d = 6, d = 9) are used to encode local to global feature information, and 3 groups of modulation parameter masks Δm are calculated. Through the deformable convolution v2 network of three dilations d, the two-dimensional pose prediction of the frame to be detected is realized. The input of DCN v2 is H fusion , kernel offset O fusion，d and modulation parameter mask Δm, and the output is a two-dimensional human pose of the frame to be detected.

[0061] VII. Dataset collection:

[0062] A video dataset for human pose estimation in different complex scenes of coal mine, called Colliery-1, is proposed. The original data comes from the surveillance video data of six different working scenes in a coal mine. In the original data, the video segments with complex conditions, such as uneven brightness, blurred image texture details, and more noise, are randomly selected to build the dataset. There are 600 video segments in each working scene, each with a resolution of 640x480, a frame rate of 25 frames / s, and a duration of 7-9 seconds. The video images are real and aligned, and their real human pose labels are also provided. When dividing the dataset, I selected 360 video segments from each scene as the training set, 120 video segments as the test set, and 120 video segments as the validation set. With such data samples, the training set has a total of 2160 video segments, the test set has a total of 720 video segments, and the validation set has a total of 720 video segments.

[0063] VIII. Model training:

[0064] The YH-Pose designed is implemented in PyTorch. The subnetwork YOLOv7 in the framework is trained to detect people in video frames, the SA-HRNet network is trained to extract human skeleton keypoint heat map information, and the PRM and SRM modules are trained to learn features and perform pose estimation on the frames to be detected. The input image is first normalized to float32 format 0.0-1.0, then scale transformation and flipping are used for early data augmentation. To reduce the amount of calculation and improve the inference speed of the model, the Conv layer and BatchNorm layer in the model are fused into a new 2DConv layer. Distributed 4x precision training is performed using PyTorch's AMP, the Adam optimizer is used, and a learning rate scheduler is defined to dynamically adjust the learning rate size in the optimizer. As shown in Figure 2 .

[0065] IX. Loss function:

[0066] The standard pose estimation loss function is used as the cost function. The purpose of training is to minimize the total Euclidean or L2 distance between the predicted and actual heat maps of all joints. The cost function formula (10) is defined as:

[0067]

[0068] F(j),M(j),v jrespectively represent the ground truth heat map, predicted heat map, and visibility of joint j. In training, a total of 17 joints are set according to the order of human skeleton connection. The ground truth heat map information is generated by a two-dimensional Gaussian graph centered on the joint.

[0069] X. Model training parameter settings:

[0070] The device used for training is a CPU with a 12th Gen Intel(R) Core(TM) i9-12900K 3.19 GHz, 64 G of memory, a 64-bit Windows 10 operating system, and a NVIDIA 3090 GPU server. The program uses Anaconda3 Version 5.2.0 as the integrated development environment, the programming language is Python Version 3.6.5, and various networks designed under the Pytorch deep learning framework are built. In the training process, data augmentation is combined, including random rotation, scaling, truncation, and horizontal flipping to increase variation. The input image size is fixed at 384x288. The backbone parameters are fixed as the pre-trained HRNet-W48 model weights. All subsequent weight parameters are initialized as a Gaussian distribution of μ = 0, σ = 0.001, and the bias parameter is initialized to 0. The Adam optimizer is used with an initial learning rate of 0.0001, and a 10% decay every 4 cycles.

[0071] Based on the above scheme, the YH-Pose framework is expanded through various ingenious modules and strategies, making it possible to estimate the human pose of video frames with uneven brightness, blurred details, and more noise in coal mines.

[0072] (1) The spatial path module (SPM) focuses on the propagation of inter-frame feature information, and supplements the features of the disturbed frame by weighted fusion of historical frames, and refines the pose features of the frame to be detected through the motion context temporal information of consecutive frames.

[0073] (2) The weighted way is to selectively fuse historical frame features rather than indiscriminately superimposed fusion, which avoids the introduction of redundant historical feature information.

[0074] The spatial path module is a parallel network built with ordinary convolutional neural networks and deformable convolutional neural networks, which learns the input features and regresses the position information of the skeleton points in the form of heat maps.

Claims

1. A human pose estimation method suitable for complex scenes in coal mines, using a pose estimation framework to estimate the poses of multiple people in video frames of complex scenes in coal mines, characterized in that, Wherein, The pose estimation framework comprises a human detector module, an SA-HRNet module, a temporal path module, and a spatial path module. The steps are as follows: First, the human detector module explores the spatial location of the human image in each frame and crops the human image from the original frame. In the pose estimation network, the temporal path and spatial path modules extract the skeletal joint information features. The input to the temporal path module is obtained by repackaging the cropped local human frames into a continuous frame sequence using the human detection algorithm and feeding it into the SA-HRNet network. The temporal path module uses a grouped interleaving strategy to encode the information of consecutive frames and uses a weighted method... The method integrates feature information from key points in multiple historical frames to supplement the features of the interfered frames. The interfered frames refer to frames in complex coal mine scene videos that exhibit uneven brightness, blurred image details and textures, and excessive noise. The spatial road module is constructed using convolutional neural networks and deformable convolutional networks, with different resolution convolutional networks connected in parallel. The SA-HRNet module also introduces a spatial attention module, which is integrated into the multi-resolution fusion stage of each subnet. The grouped interval interpolation strategy involves cropping a continuous video frame sequence into a group of n frames {F1, F2, ..., F...}. n Then, a set of n consecutive low-resolution video frames is randomly selected as the learning frames, where the nth video frame F n As the reference frame, the (n+1)th video frame F n+1 The frame to be detected is used to learn human pose features through heatmap estimation from the temporal and spatial dimensions, and to predict the poses of multiple people in complex scenes.

2. The human pose estimation method suitable for complex scenes in coal mines according to claim 1, characterized in that, The implementation method of the inter-frame interval frame insertion includes the following steps: first, a human body detector YOLOv7 module is introduced to the continuous n frames of video frames, the YOLOv7 module detects the human bodies in the video frames and labels the boundary boxes for the human bodies, then the boundary boxes are further used to crop the local frames with only the human bodies from the original video frames, and then the cropped local human body frames are re-bound into a continuous frame sequence and sent into an SA-HRNet network, the SA-HRNet network outputs an initial key point heat map estimation H t , 1 < t < n; in the time road module, the n video frames are decoupled into 3 groups according to the degree of adjacency with the reference frame, each group represents a different frame rate and has different frame intervals; the feature similarity of each frame in each group with the reference frame is calculated, and a corresponding weight key is generated, the weight value is used to selectively extract the posture feature information in the time dimension between different frames, and the posture feature information between groups is further fused by accumulation and multiplication; finally, in the spatial road module, the fused features output by the time road module are stacked into 3x3 basic blocks and then sent into a network built by two-dimensional convolution and deformable convolution for feature re-extraction, finally the position of the key point on the frame to be detected is predicted by the heat map regression method, the key points are drawn according to the skeleton sequence, and the two-dimensional posture of the human body in the frame to be detected is generated.

3. The human pose estimation method suitable for complex scenes in coal mines according to claim 2, characterized in that, The human body detector module adopts a top-down manner, first detects a person in an image, then adds a target frame to the human body, uses YOLOv7 to learn the characteristics of the person in a complex scene to generate a weight capable of detecting the human body in the complex scene, and the detector detects the position information of each human body in the image, that is, the coordinates (x, y) of a vertex of a boundary frame, the width W and the height H of the boundary frame, and the category confidence, according to the element information predicted and output by YOLOv7, obtains the 2D coordinate values of the four vertices of the boundary frame In order to obtain more feature information while avoiding redundant features and increasing the amount of calculation, the boundary frame is expanded by β times, the four vertex coordinate values after expansion are obtained The four vertex coordinate values after expansion are obtained On the basis of alignment with the original video frame, the corresponding video frame is cut from the original video and reconstructed into a continuous video frame sequence.

4. The human pose estimation method suitable for complex scenes in coal mines according to claim 1, characterized in that, The SA-HRNet network continues to use the parallel mode of the HRNet network to connect convolution networks of different resolutions, so that the input of the entire network remains high resolution from beginning to end, and the feature information between different resolutions can be repeatedly fused with multi-scale information fusion, so that the effect of pose estimation is improved, wherein the SA-HRNet network in the process of human pose estimation is as follows: first, the approximate position of the target human body is detected, and then the specific position of all key points of the human body is detected; in the process of human key point detection, a regressor is used to estimate the position of each key point of the human body; after estimating all the key points, the key points are regressed to the original image in the form of a heat map, and a convolution layer is added to the last layer of the HRNet-W48 model, so that the obtained key point position is converted into a full-resolution heat map; The heat map calculation is shown in equation (3), and the real heat map value is represented by H gt The estimated heat map value is represented by H pred The mean square error of the two can be represented as L H , which represents the overall loss function, and the loss calculation is shown in equation (4); the center point position of the real heat map calculated by the above equation is the position of the key point of the human body to be detected; W ′ and H ’ are the width and height of the heat map, respectively, K is the channel number, and c is the standard deviation; 5. The human pose estimation method suitable for complex scenes in coal mines according to claim 1, characterized in that, The spatial attention module is integrated into the middle of different stages of the HRNet and is used when the resolution is reduced, so as to expand the receptive field and more effectively extract more ignored or missed feature information.

6. The human pose estimation method suitable for complex scenes in coal mines according to claim 1, characterized in that, The time road module: by comparing the predicted frame with the previous historical frame, dynamically aggregating and learning the similar pose feature information of the historical frame and the predicted frame, the SA-HRNet network extracts the heat map features of each historical frame, and then the time grouping module divides the historical frame into three groups with different frame numbers from the reference frame; the features are fused by intra-group feature weighting and inter-group feature accumulation and multiplication; different fusion strategies are synchronized in the whole encoding propagation process; the time grouping module: based on the time distance to the reference frame, each time interval N frames N∈[1,3] are divided into a group, so that the original sequence is reordered as group1: {F5,F6,F7}, group2: {F3,F5,F7}, group3: {F1,F4,F7}, each group is a sub-sequence of the original video frame; the spatial road module fuses the time road module and outputs the historical frame key point heat map feature H fusion Then the pose features of each channel are stacked into a 3x3 basic block, which facilitates input to the new network for a new round of pose feature learning.

7. The human pose estimation method suitable for complex scenes in coal mines according to claim 1, characterized in that, The convolutional neural network constructs three parallel 2D convolutional layers to extract the stacked historical frame key point heat map features H fusion The convolutional features output by the CNN contain rich semantic information and can more comprehensively represent object features; first, normalization and size adjustment operations are performed on the input layer to change the size of the input layer to 128x128x3, the convolution kernel size is 3x3, the stride is 2, and a padding operation is performed; a nonlinear unit Relu activation function is used in each convolution process to activate; then, an average pooling layer is used here to down-sample high-dimensional pose feature information to obtain low-dimensional feature information; the kernel of the average pooling layer is 2x2, the stride is 2, and the feature H fusion with a size of 3x3x256 is obtained after the convolutional neural network outputs.

8. The human pose estimation method suitable for complex scenes in coal mines according to claim 1, characterized in that, The data set acquisition module: a coal mine video data set with different complex scenes for human pose estimation is used, called Colliery-1; the original data comes from the monitoring video data of six different working scenes in a coal mine, and the video segments with complex conditions: uneven brightness, blurred image texture details, and more noise are randomly selected from the original data to form the data set, with 600 video segments in each working scene, each video segment having a resolution of 640x480 and a frame rate of 25 frames / s and a length of 7-9 seconds; This is a real and aligned video image, which also provides its real human pose label; when dividing the data set, 360 video segments are selected from each scene as the training set, 120 video segments are selected as the test set, and 120 video segments are selected as the validation set; such data samples, the training set has a total of 2160 video segments, the test set has a total of 720 video segments, and the validation set has a total of 720 video segments.

9. The human pose estimation method suitable for complex scenes in coal mines according to claim 8, characterized in that, The SA-HRNet model training steps are as follows: The sub-network YOLOv7 in the framework is trained on the dataset Colliery-1 to detect people in video frames, the SA-HRNet network is trained to extract human body skeleton key point heat map information, and the time road module and the space road module are trained to learn features and perform pose estimation on the frames to be detected; the input image is first normalized to float32 format; then scale transformation and flipping are used for early data enhancement, in order to reduce the amount of calculation and improve the inference speed of the model, the Conv layer and the BatchNorm layer in the model are fused into a new 2DConv layer; distributed 4-fold precision training is performed by using the AMP provided by PyTorch, the Adam optimizer is used and a learning rate scheduler is defined to dynamically adjust the learning rate size in the optimizer; a standard pose estimation loss function is used as a cost function, and the purpose of training is to minimize the total Euclidean distance between the predicted and actual heat maps of all joints; the ground truth heat map information is generated by a two-dimensional Gaussian graph centered on the joint.

Images