Construction site personnel working state identification method based on deep residual network

Abstract

The invention provides a construction site personnel working state identification method based on a deep residual network, and the method comprises the steps: collecting a real-time scene image of a construction site, carrying out the character detection of the collected real-time scene image, and cutting a personnel region image of a preset size for any detected personnel; inputting the personnel area image into a pre-constructed dual-branch sensing local and global mixer network based on a deep residual network; and acquiring human body articulation points and human body actions in the personnel area image output by the mixer network to realize identification of the working state of the personnel. The invention designs a mixer network model suitable for detecting the working state of the construction site personnel, and the working state of the construction site personnel can be effectively identified.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of human action prediction, and particularly relates to a construction site personnel working state recognition method based on a deep residual network. BACKGROUND

[0002] The construction industry is an important pillar industry of the country, and the optimization of its duration, quality, cost and other factors is the goal that construction personnel constantly pursue. However, as construction personnel, they work in a construction site with complex operation procedures and numerous hazards, and their life safety is difficult to be guaranteed. According to statistical data in recent years, the construction industry has become a high-risk industry with high casualty rate and frequent accidents, second only to mines. Although the industry has begun to focus on construction safety training, personnel can be monitored and identified at the first time when they encounter danger and an alarm can be sent, which can help the injured personnel to obtain valuable first aid time and reduce the casualties of construction personnel.

[0003] Although the development of deep learning has greatly improved the prediction results of human behavior, there are few studies on the identification of unsafe behaviors of construction site personnel, and no behavior identification model for construction site groups has been proposed. Due to the special working environment of construction site workers, the environment background is complex and irregular, and the images collected cannot be well recognized by the general human behavior recognition model. The human behavior recognition model has poor recognition effect on construction site personnel, which is usually because the existing model lacks effective combination of local information and global information, resulting in poor effect of the model on complex background and human body occlusion of construction site images. SUMMARY

[0004] The present application aims to overcome the defects of the prior art and provides a construction site personnel working state recognition method based on a deep residual network, which designs a model suitable for construction site personnel working state detection and can effectively recognize the working state of construction site personnel.

[0005] The present application is implemented as follows:

[0006] A construction site personnel working state recognition method based on a deep residual network, the method comprising:

[0007] Collecting real-time scene images of a construction site, performing person detection on the collected real-time scene images, and cropping a preset size of a personnel region image for any detected personnel;

[0008] Inputting the personnel region image into a pre-constructed dual-branch perception local and global mixer network based on a deep residual network;

[0009] Obtaining the human joint points and human actions in the personnel region image output by the mixer network to realize recognition of the working state of the personnel.

[0010] The mixer network comprises: a cascaded STEM section, several combination modules, a high-resolution parallel pyramid, and a joint fine-tuning module. Each combination module includes two parallel branches, one of which includes two cascaded local sensing modules and the other includes two cascaded global sensing modules. The outputs of the two parallel branches are summed to serve as the output of the combination module. The outputs of all combination modules are summed to serve as the input of the high-resolution parallel pyramid. Finally, the output of the high-resolution parallel pyramid serves as the input of the joint fine-tuning module.

[0011] Optionally, the feature map input to the local sensing module is H t ∈R B×C×H×W Where B is the batch size, C is the number of input channels, H is the height of the feature map, and W is the width of the feature map;

[0012] The local perception module includes a series of cascaded 3×3 convolutions, average pooling, two 1×1 convolutions, a sigmoid function, a 3×3 convolution, a 1×1 convolution, a softmax function, and a 1×1 convolution, as shown in the following mathematical expression:

[0013] y 3×3 =f 3×3 (H t )

[0014] y s1 =fc2(fc1(avge(f 3×3 (H t ))))

[0015] y s =s(y s1 )·y 3×3

[0016] Among them, f 3×3 It is a 3×3 convolution, avge is the average pooling function, fc1 is a 1×1 convolution, the number of channels input to fc1 is C, and the number of channels output after fc1 becomes C / r, fc2 is a 1×1 convolution, the number of channels input to fc2 is C / r, and the number of channels output after fc2 is C, where r is the scaling ratio, S is the sigmoid function, and · is the dot product sign;

[0017] y1 = sft(f1(f 3×3 (y s )))

[0018]

[0019] Among them, f 3×3is 3x3 convolution, f1 is 1x1 convolution, the dimension of f1 input is C, the dimension of f1 output is 1, sft is softmax function, f2 is 1x1 convolution, the dimension of f2 input is 1, the dimension of f2 output is C, represents the corresponding element multiplication.

[0020] Optionally, the global perception module comprises a multi-head attention mechanism and a multi-scale feedforward network, and the input feature map is H t ∈R B×C×H×W , wherein B is batchsize, C is the number of input channels, H is the height of the feature map, and W is the width of the feature map.

[0021] For the input feature map, segmentation is performed through 1x1 convolution to obtain a one-dimensional sequence , that is, three vectors, w is the window scale size, and h is the number of heads.

[0022] The weight value of the attention mechanism of each head is obtained by using the following formula:

[0023]

[0024] , wherein softmax is a softmax function, T is a transpose operation, d is the dimension of the vector Q or K, and m is equal to

[0025] The specific implementation is to cut the feature map HxW into parts, and the size of each part is w x w, and the attention mechanism operation is performed on the w x w size; after the above operation, each part of the attention mechanism is spliced.

[0026] The spliced features are dimensionally converted to obtain H t ′∈R B×C×H×W , H t ′∈R B×C×H×W The final output H tout ∈R B×C×H×W is obtained through the multi-scale feedforward network, and the mathematical description about H t ′∈R B×C×H×W The expression of input into the multi-scale feedforward network is as follows:

[0027] y′ 1×1 =f 1×1 (BN(H′ t +H t ))

[0028] H tout =f 3×3 (y′ 1×1 )+f 5×5 (y′1×1 )+f 7×7 (y′ 1×1 )+H′ t +H t

[0029] where BN is a batchnormal function, f 1×1 is a 1x1 convolution, f 3×3 , f 5×5 and f 7×7 are 3x3, 5x5 and 7x7 convolutions respectively.

[0030] Optionally, the high-resolution parallel pyramid comprises a first part from bottom to top and a second part from top to bottom, and the size of the input feature map is set to The size of the output feature map is The specific mathematical expression is as follows:

[0031] First, the mathematical expression of the first part from bottom to top is as follows:

[0032] The first branch feature map size: H rin = H rin

[0033] f 41-3×3 = f 3×3 (f 3×3 (f 3×3 (f 3×3 (H rin ))))

[0034] The second branch feature map size: H rin / 2 = down(H rin )

[0035] f 42-3×3 = f 3×3 (f 3×3 (f 3×3 (f 3×3 (H rin / 2))))

[0036] The third branch feature map size: H rin / 4 = down(H rin / 2)

[0037] f 43-3×3 = f 3×3 (f 3×3 (f 3×3 (f 3×3 (H rin / 4))))

[0038] The fourth branch feature map size: H rin / 8 = down(H rin / 4)

[0039] f 44-3×3 = f 3×3 (f 3×3 (f 3×3 (f 3×3 (H rin / 8))

[0040] Secondly, the mathematical expression of the second part from top to bottom is as follows:

[0041] f4 = up2(f 44-3×3 + H rin / 8)

[0042] f3 = up2(f4 + f 43-3×3 + H rin / 4)

[0043] f2 = up2(f3 + f 42-3×3 + H rin / 2)

[0044] H rout = f2 + f 41-3×3

[0045] Wherein, f 3×3 is a 3x3 convolution, up2 is an up-sampling function with a sampling rate of 2, and down is a down-sampling function with a sampling rate of 2.

[0046] Optionally, the mathematical expression of the joint point fine-tuning module is as follows:

[0047] y3 = f 3×3 (H in )

[0048] y m = sig1(f 1×1 (y3))

[0049]

[0050] H out = y3 + y b

[0051] Wherein, H in is the input feature map of the joint point fine-tuning module, H out is the output feature map of the joint point fine-tuning module, sig1 and sig2 are sigmoid functions, f 3×3 is a 3x3 convolution, f 7×7 is a 7x7 depth separable convolution, and f 1×1 is a 1x1 convolution.

[0052] Optionally, the number of the combination modules is 4.

[0053] Optionally, the stem part comprises a cascaded 7*7 convolution, a 3*3 convolution and a down-sampling function, wherein the step length of the 7*7 convolution is 2, the patch is 3, the step length of the 3*3 convolution is 1, the patch is 1, and the down-sampling rate of the down-sampling function is 2.

[0054] Optionally, the character detection on the collected real-time scene image comprises:

[0055] inputting the real-time scene image into a pre-trained character detection neural network model, performing personnel detection on the real-time scene image by using the character detection neural network model, and outputting a personnel candidate frame.

[0056] Optionally, in the process of training the mixer network, the human pose prediction adopts a mean square error loss L2-norm to calculate a human pose prediction action x′ a and a real human pose action x a , to obtain Loss1, Loss1 = L2(x′ a ,x a ).

[0057] The training process is completed by using the Loss1 for reverse gradient back propagation.

[0058] The present application has the following beneficial effects:

[0059] 1. The local perception module designed in the present application contains rich spatial information, which can make up for the lack of cross-window interaction in the non-overlapping local window design of the global perception module. In the field of building scene, spatial consistency is crucial, therefore, the local perception module designed in the present application not only contains a spatial attention mechanism, but also designs a channel attention mechanism, aiming to strengthen the spatial consistency of the building scene features and improve the accuracy of the building scene joint node positioning.

[0060] 2. The global perception module designed in the present application constructs the spatial relationship between the joints and between the joints, and strengthens the correlation of the human joints that are not directly connected in the human skeleton. The long-distance dependency feature extraction is completed by using the transformer (referring to the multi-head attention mechanism, which can obtain the long-distance association information between the joint nodes), which is very effective for the problem of complex background and a large number of human part occlusions often existing in the construction site personnel image, because the long-distance dependency relationship can well extract the global semantic information in the image, which is beneficial for the alleviation of the above-mentioned problems, and makes up for the lack of global semantic information in the local perception module, and improves the classification accuracy of the joint nodes.

[0061] 3. In the joint fine-tuning module, the remaining connections (i.e. skip connections) of the first branch keep high-resolution features, containing rich low semantic level to achieve accurate positioning of the joint. In the middle path, the previous high-level semantic features are re-learned, and the intermediate semantic information is learned to make the classification information clearer. The bottom path focuses on the features in the spatial dimension of the image, containing high-level semantic information that can identify the joint category. The three branches respectively perform information fusion from high, medium and low semantic levels, prompting the joint fine-tuning module to further improve the recognition and classification of the joint. The joint fine-tuning module has a more dense connection architecture and a wider receptive field range, making the intra-layer feature fusion more effective.

[0062] 4. In order to solve the problem of variable size of construction site personnel, the present application designs a high-resolution parallel pyramid. The high-resolution parallel pyramid extracts features of different scales using a top-down path, and then uses a bottom-up path for feature enhancement while maintaining the existence of high-resolution features. In this way, the fusion between the strong semantic information of low-resolution features and the detailed information of high-resolution features can be achieved. This module promotes the flow of information between low-level features and high-level features, and provides a basis for accurate positioning of the joint by high-resolution feature maps. At the same time, the fusion of shallow features and deep features in this module can further improve the foreground and background recognition, which is beneficial to improve the recognition accuracy of construction site personnel in a blurred background scene. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0064] Figure 1 A flow chart of a construction site personnel working state recognition method based on a deep residual network is provided for the embodiment of the present application.

[0065] Figure 2 A structure diagram of a two-branch perception local and global hybrid mixer network TBALGMN based on a deep residual network is provided for the embodiment of the present application.

[0066] Figure 3 A structure diagram of a local perception module is provided for the embodiment of the present application.

[0067] Figure 4 A structure diagram of a global perception module is provided for the embodiment of the present application.

[0068] Figure 5A structural diagram of a high-resolution parallel pyramid provided for an embodiment of the present application is provided.

[0069] Figure 6 A structural diagram of a joint point fine-tuning module provided for an embodiment of the present application is provided.

[0070] Figure 7 An action prediction result achieved by the method of the present application is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0072] As shown in Figure 1 The present application provides a construction site personnel working state recognition method based on a deep residual network, which comprises the following steps:

[0073] S101, collecting real-time scene images of a construction site, performing person detection on the collected real-time scene images, and cutting out a personnel region image of a preset size for any detected personnel;

[0074] S102, inputting the personnel region image into a pre-constructed double-branch perception local and global mixer network based on a deep residual network;

[0075] S103, obtaining a human joint point and a human action in the personnel region image output by the mixer network, and realizing recognition of the personnel working state.

[0076] Generally, several cameras are installed at a construction site for collecting real-time scene images of the construction site. In an implementation, the cameras are connected to a server, which acquires real-time scene images collected by the cameras and performs subsequent processing to identify the working state of personnel at the construction site. First, to improve detection efficiency, the server can input the real-time scene images into a pre-trained human detection neural network model, perform personnel detection on the real-time scene images using the human detection neural network model, and output a personnel candidate box. The human detection neural network model can use a convolutional neural network. The specific training method of this network model can refer to the prior art, and will not be described here. According to the personnel candidate box, a personnel region image is cropped from the real-time scene image, for example, the size of the personnel region image can be pre-set to 256x192. Then the personnel region image is input into a pre-constructed Two-Branch Aware Local and Global Mixer Networks (TBALGMN) based on a deep residual network, to obtain the human joint points and human actions in the personnel region image output by the mixer network. Based on the human joint points and human action information, the working state of the personnel can be determined. It can be understood that the present application uses a mixer network to estimate human pose, identify the position of the joint points of a person, and then determine the posture action of the personnel according to a pre-set pose rule, and further infer the working state of a person according to continuous actions, for example, long-time lying down indicates that the employee may not be working or an accident has occurred, and the personnel need to be reminded to check.

[0077] The above-mentioned mixer network TBALGMN will be described in detail below.

[0078] As shown in Figure 2 , the mixer network includes a stem part, a plurality of combination modules, a high-resolution parallel pyramid, and a joint point fine-tuning module, which are sequentially cascaded; each combination module includes two parallel branches, one of which includes two local perception modules cascaded, and the other of which includes two global perception modules cascaded, the outputs of the two parallel branches are added to serve as the output of the combination module, the outputs of all combination modules are added to serve as the input of the high-resolution parallel pyramid, and the output of the high-resolution parallel pyramid finally serves as the input of the joint point fine-tuning module. In addition, a 1x1 convolution is arranged between adjacent two combination modules, which can make the features between the modules better transition and improve the robustness of network training. A 1x1 convolution is arranged after the joint point fine-tuning module, which can increase the depth of the network.

[0079] The STEM part includes cascaded 7×7 convolutions, 3×3 convolutions, and a downsampling function. The 7×7 convolutions have a stride of 2 and a patch size of 3, the 3×3 convolutions have a stride of 1 and a patch size of 1, and the downsampling function has a downsampling rate of 2. The STEM part is used to reduce the image to a certain size; in this embodiment, the STEM part reduces the image from 256×192 to 48×64.

[0080] The above combination module has 4 components, but the number can be set according to the situation.

[0081] The local sensing module not only enhances local spatial features but also strengthens channel information, aiding in the localization of key points. Its structure is as follows: Figure 3 As shown, the mathematical description is: First, the input feature map is H t ∈R B×C×H×W Where B is the batch size, C is the number of input channels, H is the height of the feature map, and W is the width of the feature map. This module includes sequentially cascaded 3×3 convolutions, average pooling, two 1×1 convolutions, a sigmoid function, another 3×3 convolution, a 1×1 convolution, a softmax function, and another 1×1 convolution. The upper part, consisting of 3×3 convolutions, average pooling, two 1×1 convolutions, and a sigmoid function, constitutes the channel attention mechanism, while the lower part, consisting of 3×3 convolutions, 1×1 convolutions, a softmax function, and another 1×1 convolution, constitutes the spatial attention mechanism. The specific mathematical expression is as follows:

[0082] y 3×3 =f 3×3 (H t )

[0083] y s1 =fc2(fc1(avge(f 3×3 (H t ))))

[0084] y s =s(y s1 )·y 3×3

[0085] Among them, f 3×3 It is a 3×3 convolution, avge is the average pooling function, fc1 is a 1×1 convolution, the number of channels input to fc1 is C, and the number of channels output after fc1 becomes C / r, fc2 is a 1×1 convolution, the number of channels input to fc2 is C / r, and the number of channels output after fc2 is C, where r is the scaling ratio, S is the sigmoid function, and · is the dot product sign;

[0086] y1 = sft(f1(f 3×3 (y s )))

[0087]

[0088] Among them, f 3×3 f1 is a 3×3 convolution, f2 is a 1×1 convolution with input dimension C and output dimension 1, sft is the softmax function, and f3 is a 1×1 convolution with input dimension 1 and output dimension C. It represents the multiplication sign of the corresponding element.

[0089] The complex background of construction sites and the presence of numerous occluded body joints contribute to reduced accuracy in identifying construction workers. To effectively improve the accuracy of occluded joint points, this invention designs a global perception module. This module comprises a multi-head attention mechanism and a multi-scale feedforward network. The input feature map is H... t ∈R B×C×H×W Where B is the batch size, C is the number of input channels, H is the height of the feature map, and W is the width of the feature map. For example... Figure 4 As shown, for the input feature map H t ∈R B×C×H×W The segmentation is processed by 1×1 convolution to obtain a one-dimensional sequence. Right now Three vectors: w is the window scale, which can be set to 7; h is the number of heads, which can be set to 8. The specific mathematical representation is as follows:

[0090] The weights of the attention mechanism for each head are obtained using the following formula:

[0091]

[0092] Where softmax is the softmax function, T is the transpose operation, d is the dimension of the input information (which can be the dimension of vector Q or K), and m equals... Specifically, the feature map of size H×W is cut into... Each component is w×w, and the operation to complete the attention mechanism is performed on a w×w component. After the above operation is completed, each attention mechanism is spliced ​​together.

[0093] The concatenated features are then subjected to dimensionality transformation to obtain H. t ′∈R B×C×H×W H t ′∈R B×C×H×W The final output H is obtained after passing through a multi-scale feedforward network. tout ∈R B×C×H×W Mathematical description of H t ′∈R B×C×H×W The expression input into the multi-scale feedforward network is as follows:

[0094] y' 1×1 = f 1×1 (BN(H' t + H t ))

[0095] H tout = f 3×3 (y' 1×1 )+ f 5×5 (y' 1×1 )+ f 7×7 (y' 1×1 )+ H' t + H t

[0096] where BN is a batchnormal function, f 1×1 is a 1x1 convolution, f 3×3 , f 5×5 and f 7×7 are 3x3, 5x5 and 7x7 convolutions respectively.

[0097] Although the behavior recognition of deep learning construction site personnel can achieve greater improvement compared with traditional methods, the recognition effect needs to be improved in complex construction environment. One of the main reasons is that the feature extraction of these algorithms for different scale postures needs to be developed. Therefore, the invention designs a high-resolution parallel pyramid, including a first part from bottom to top and a second part from top to bottom, as shown in Figure 5 The size of the input feature map is set to The size of the output feature map is C r , H r , W r are the number of channels and the height and width of the feature map. The specific formula is as follows:

[0098] First, the mathematical expression of the first part from bottom to top is as follows:

[0099] The first branch feature map size: H rin = H rin

[0100] f 41-3×3 = f 3×3 (f 3×3 (f 3×3 (f 3×3 (H rin )))

[0101] The second branch feature map size: H rin / 2 = down(H rin )

[0102] f 42-3×3 = f 3×3 (f 3×3 (f 3×3 (f 3×3 (H rin / 2))

[0103] Third branch feature map size: H rin / 4 = down(H rin / 2)

[0104] f 43-3×3 = f 3×3 (f 3×3 (f 3×3 (f 3×3 (H rin / 4))

[0105] Fourth branch feature map size: H rin / 8 = down(H rin / 4)

[0106] f 44-3×3 = f 3×3 (f 3×3 (f 3×3 (f 3×3 (H rin / 8))

[0107] Second, the mathematical expression of the second part from top to bottom is as follows:

[0108] f4 = up2(f 44-3×3 + H rin / 8)

[0109] f3 = up2(f4 + f 43-3×3 + H rin / 4)

[0110] f2 = up2(f3 + f 42-3×3 + H rin / 2)

[0111] H rout = f2 + f 41-3×3

[0112] Where H rout is the output feature map of the high-resolution parallel pyramid, f 3×3 is a 3x3 convolution, up2 is an up-sampling function with a sampling rate of 2, and down is a down-sampling function with a sampling rate of 2.

[0113] Although the image spatial information helps to locate the joint, the semantic features help to classify the joint, and these features have different effects on the final prediction. In order to further learn the spatial information and high-level semantic features, the present application designs a joint fine-tuning module as shown in Figure 6 Firstly, the input passes through a 3x3 convolution, and then the obtained input is input into three branches. In the first branch at the top, the features are input into a skip connection. In the middle is the second branch, the features are input into a 1x1 convolution and a sigmoid activation function, the purpose is to obtain a series of new feature weights. At the bottom is the third branch, the features pass through a 1x1 convolution and a 7x7 depth separable convolution and a sigmoid activation function, the purpose is to obtain a spatial attention map s, finally multiply the corresponding elements of the second branch and the feature map output by the 3x3 convolution, and then add the multiplied value to the first branch to obtain the mixed features. The specific mathematical expression is as follows:

[0114] y3=f 3×3 (H in )

[0115] y m =sig1(f 1×1 (y3))

[0116]

[0117] H out =y3+y b

[0118] Wherein, H in is the input feature map of the joint fine-tuning module, H out is the output feature map of the joint fine-tuning module, sig1 is the sigmoid function of the middle branch, sig2 is the sigmoid function of the bottom branch, f 3×3 is a 3x3 convolution, f 7×7 is a 7x7 depth separable convolution, and f 1×1 is a 1x1 convolution.

[0119] During the training process of the mixer network, the human pose prediction adopts the mean square error loss L2-norm to calculate the human pose predicted action x′ a and the human pose real action x a , to obtain Loss1, Loss1=L2(x′ a ,x a ); using Loss1 to perform backward gradient backpropagation to complete the training process.

[0120] Figure 7For the final detection result of the present application, in order to reflect the performance of the present application, the human joint point and the human action recognition result are visualized, as shown in Figure 7

[0121] The present application carries out the following experiment: the self-built data sample is classified and processed, and the behavior sample of the construction site personnel is divided into three dimensions of completely safe behavior, safer behavior and unsafe behavior, wherein the completely safe behavior includes walking, sitting and standing (these are changed according to actual needs), the safer behavior includes bending, squatting, lifting things, pushing, pulling, waving hands, clapping hands (these are changed according to actual needs), and the unsafe behavior includes falling down and lying down (these are changed according to actual needs). The test is carried out on the self-built data set and the official action test data set, the overall running speed of the model is 54fps (after stable test, the average value in one minute), the present application obtains an AP value of 89.9%. In addition, in order to reflect the effectiveness of the method of the present application on the official data set, the human pose prediction module of the present application is used on the mscoco data set, the input image is 256x192, the AP value obtained by the method of the present application is 76.6%. The method of the present application has exceeded the result of Human Body-Aware Feature Extractor Using Attachable Feature Corrector for Human Pose Estimation.

[0122] The above description of the disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those of ordinary skill in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Accordingly, the application is not to be restricted based on the embodiments set forth in this description, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.​

Claims

1. A construction site worker working state recognition method based on a deep residual network, characterized by, The method comprises: Collecting a real-time scene image of a construction site, performing person detection on the collected real-time scene image, and cropping a preset size of a personnel region image for any detected personnel; Inputting the personnel region image into a pre-constructed dual-branch perception local and global mixer network based on a deep residual network; Obtaining a human joint node and a human action in the personnel region image output by the mixer network to realize identification of a personnel working state; The mixer network comprises a stem part, a plurality of combination modules, a high-resolution parallel pyramid and a joint node fine-tuning module which are sequentially cascaded; each combination module comprises two parallel branches, one of which comprises two local perception modules which are cascaded, and the other of which comprises two global perception modules which are cascaded, the outputs of the two parallel branches are added to serve as the output of the combination module, the outputs of all combination modules are added to serve as the input of the high-resolution parallel pyramid, and finally the output of the high-resolution parallel pyramid serves as the input of the joint node fine-tuning module.

2. The method of claim 1, wherein, The feature map input by the local perception module is H t ∈R B×C×H×W , wherein B is batchsize, C is the number of input channels, H is the height of the feature map, and W is the width of the feature map. The local perception module comprises a 3x3 convolution, an average pooling, two 1x1 convolutions, a sigmoid function, a 3x3 convolution, a 1x1 convolution, a softmax function and a 1x1 convolution which are sequentially cascaded, and the specific mathematical expression is as follows: y 3×3 = f 3×3 (H t ) y s1 = fc2(fc1(avge(f 3×3 (H t )))) y s = s(y s1 ) · y 3×3 wherein f 3×3 is a 3x3 convolution, avge is an average pooling function, fcl is a 1x1 convolution, the number of channels of the input of fcl is C, the number of channels of the output after fcl is C / r, fc2 is a 1x1 convolution, the number of channels of the input of fc2 is C / r, the number of channels of the output after fc2 is C, wherein r is a scaling ratio, S is a sigmoid function, and • is a dot product sign; y1 = sft(f1(f 3×3 (y s ))) wherein f 3×3 is a 3x3 convolution, f1is a 1x1 convolution, the dimension of f1input is C, the dimension of f1output is 1, sftis a softmax function, f2is a 1x1 convolution, the dimension of f2input is 1, the dimension of f2output is C, represents the multiplication of corresponding elements.

3. The method of claim 1, wherein, The global perception module comprises a multi-head attention mechanism and a multi-scale feedforward network, and an input feature map is H t ∈R B×C×H×W where B is a batch size, C is a number of input channels, H is a height of a feature map, and W is a width of the feature map. For the input feature map, the segmentation is obtained by 1x1 convolution to obtain a one-dimensional sequence That is Three vectors, w is the window scale size, h is the number of heads The weight value of the attention mechanism of each head is obtained by using the following formula: where softmax is the softmax function, T is the transpose operation, d is the dimension of the vector Q or K, and m is equal to The specific implementation is to cut the feature map HxW size into Each part has a size of w x w, and the operation of the attention mechanism is completed on the w x w size; after the above operation is completed, each part of the attention mechanism is spliced; The concatenated features are dimensionally converted to obtain H t ′∈R B×C×H×W , H t ′∈R B×C×H×W The final output H tout ∈R B×C×H×W is obtained through a multi-scale feedforward network t ′∈R B×C×H×W The expression input into the multi-scale feedforward network is as follows: y' 1×1 = f 1×1 (BN(H' t + H t )) H tout = f 3×3 (y′ 1×1 )+ f 5×5 (y′ 1×1 )+ f 7×7 (y′ 1×1 )+ H′ t + H t where BN is a batchnormal function, f 1×1 is a 1 x 1 convolution, f 3×3 , f 5×5 , and f 7×7 are 3 x 3, 5 x 5, and 7 x 7 convolutions, respectively. The high-resolution parallel pyramid comprises a first part from bottom to top and a second part from top to bottom, and the size of the input feature map is set to The size of the output feature map is The specific mathematical expression is as follows: First, the mathematical expression of the first part from bottom to top is as follows: First branch feature map size: H rin = H rin f 41-3×3 = f 3×3 (f 3×3 (f 3×3 (f 3×3 (H rin )))) Second branch feature map size: H rin / 2 = down(H rin ) f 42-3×3 = f 3×3 (f 3×3 (f 3×3 (f 3×3 (H rin / 2)))) Third branch feature map size: H rin / 4 = down(H rin / 2) f 43-3×3 = f 3×3 (f 3×3 (f 3×3 (f 3×3 (H rin / 4)))) Fourth branch feature map size: H rin / 8 = down(H rin / 4) f 44-3×3 = f 3×3 (f 3×3 (f 3×3 (f 3×3 (H rin / 8)))) Second, the mathematical expression of the second part from top to bottom is as follows: f4 = up2(f 44-3×3 + H rin / 8) f3 = up2(f4 + f 43-3×3 + H rin / 4) f2 = up2(f3 + f 42-3×3 + H rin / 2) H rout = f2+ f 41-3×3 where f 3×3 is a 3x3 convolution, up2 is an up-sampling function with a sampling rate of 2, and down is a down-sampling function with a sampling rate of 2.

4. The method of claim 1, wherein, The mathematical expression of the joint node fine-tuning module is as follows: y3 = f 3×3 (H in ) y m = sig1(f 1×1 (y3)) H out = y3 + y b wherein H in is an input feature map of the joint fine-tuning module, H out is an output feature map of the joint fine-tuning module, sig1 and sig2 are sigmoid functions, f 3×3 is a 3x3 convolution, f 7×7 is a 7x7 depthwise separable convolution, f 1×1 is a 1x1 convolution.

5. The method of claim 1, wherein, The number of combination modules is 4.

6. The method of claim 1, wherein, The stem part comprises a 7x7 convolution, a 3x3 convolution and a downsampling function which are cascaded, wherein the step length of the 7x7 convolution is 2 and the patch is 3, the step length of the 3x3 convolution is 1 and the patch is 1, and the downsampling rate of the downsampling function is 2.

7. The method of claim 1, wherein, The person detection on the collected real-time scene image comprises: Inputting the real-time scene image into a pre-trained person detection neural network model, performing personnel detection on the real-time scene image by using the person detection neural network model, and outputting a personnel candidate frame.

8. The method of claim 1, wherein, In the process of training the mixer network, the human pose prediction adopts the mean square error loss L2-norm on the obtained human pose predicted action x′ a and the human pose real action x a to calculate Loss1, Loss1 = L2(x′ a ,x a ). The training process is completed by using Loss1 for reverse gradient backpropagation.

Images