A method for detecting safety helmets based on structure reparameterization applied to edge devices

Abstract

The application relates to a safety helmet detection method based on structure reparameterization applied to an edge device, and comprises the following steps: 1) constructing a dense reparameterization module; 2) constructing a standard YOLOv3-tiny model suitable for safety helmet detection and a training data set; 3) reconfiguring and training the standard YOLOv3-tiny model; 4) equivalently converting the trained reconfigured model into an inference model, and performing safety helmet detection. Compared with the prior art, the application has the advantages of high real-time performance, high accuracy, strong generalization ability, the ability to avoid gradient diffusion and gradient explosion, the ability to reduce feature redundancy, the ability to improve the learning ability of the network and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep neural network structure reparameterization and object detection, in particular to a hard hat detection method based on structure reparameterization applied to edge devices. BACKGROUND

[0002] The engineering, construction and other industries are typical labor-intensive industries, and their working environment is complex and safety accidents occur frequently. Brain injury and death caused by high-altitude falling objects are common and typical accidents in the construction industry. Safety helmets, as an effective safety protection device, can block the impact energy of falling objects from a high place and reduce head shock injuries, and are widely used in construction sites. In the safety management of construction sites, real-time and accurate supervision of safety helmet wearing behavior is an important link. The safety management of construction sites requires small-sized and highly mobile detection equipment, so embedded edge computing devices are often used, resulting in deficiencies in real-time performance and accuracy of safety helmet detection algorithms.

[0003] Many domestic and foreign works have studied the automatic recognition technology of safety helmets. Dalal et al. first proposed to realize automatic detection of safety helmets by extracting gradient histogram features. Feng Jie et al. combined the Adaboost classifier to detect the position of the safety helmet and judged whether the safety helmet was worn according to the position relationship between the person and the safety helmet. Hu Tian et al. proposed a neural network safety helmet recognition model based on the analysis of the application of wavelet transform and deep learning in safety helmet recognition. Liu Xiaohui et al. used the skin color detection method to locate the face, and then used the support vector machine (SVM) to realize the recognition of the safety helmet. Liu Yunbo et al. judged whether the safety helmet was worn by detecting the color value distribution of the pixel points in the upper 1 / 3 part of the moving target. Although the above methods can realize relatively accurate recognition of safety helmets in specific scenarios, they still have problems such as high environmental requirements, poor real-time performance, weak generalization ability, and complex user operation process.

[0004] In recent years, with the in-depth research and application of deep learning technology, the deep network target detection method represented by the YOLOv3 (You Only Look Once V3) model not only has good real-time performance, but also has high accuracy. However, the complex and diverse convolution structure leads to network structure fragmentation, increased network complexity, high feature redundancy, low memory access efficiency, and poor flexibility, which seriously hinders the deployment and application of the safety helmet detection algorithm based on YOLOv3 on edge intelligent devices with weak computing power and low memory. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a hard hat detection method based on structure reparameterization applied to edge devices.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A safety helmet detection method based on structural reparameterization for edge devices, the method includes the following steps:

[0008] 1) Construct a dense, heavily parameterized module;

[0009] 2) Construct a standard YOLOv3-tiny model and training dataset suitable for helmet detection;

[0010] 3) Reconstruct and train the standard YOLOv3-tiny model;

[0011] 4) Convert the trained reconstruction model into an equivalent inference model and perform safety helmet detection.

[0012] Step 1) specifically includes the following steps:

[0013] 11) Define the basic unit;

[0014] 12) Construct the transformation structure;

[0015] 13) Construct a learning structure;

[0016] 14) Constructing a dense reparameterization module DR-Block: This dense reparameterization module DR-Block consists of one transform structure cascaded with one learning structure. The parameter of the transform structure is F. trans (x;C rep_in The parameter of the learning structure is F. learn (x learn ;2×C rep_in C rep_out ,K rep ,S rep ), and x learn =F trans (x;C rep_in If the dense reparameterization module DR-Block is constructed, then it is denoted as F. rep (x;C rep_in C rep_out ,K rep ,S rep ).

[0017] In step 11), the basic unit consists of one convolutional layer cascaded with one batch normalization layer, denoted as F. basic (x;C basic_in C basic_out ,K basic ,S basic ), where x is the input, C basic_in For the number of input channels, Cbasic_out is the number of output channels, K basic is the size of the convolution kernel, S basic is the step size. Specifically, the number of input channels of the convolution layer is C basic_in , the number of output channels is C basic_out , the size of the convolution kernel is K basic , and the number of channels of the batch normalization layer is C basic_out .

[0018] The step 12) is specifically:

[0019] First, four basic units are cascaded to form a deep cascaded network structure to realize network over-parameterization, then a skip connection is added between any two basic units to realize model integration of different levels of features, and finally the output of each basic unit is spliced into one, denoted as F trans (x;C trans_in ).

[0020] In the step 13), the learning structure is composed of one basic unit, denoted as F learn (x;C learn_in ,C learn_out ,K learn ,S learn ).

[0021] The step 2) specifically includes the following steps:

[0022] 21) Collect and label the hard hat detection data set in the construction site and network pictures, and perform standard data preprocessing;

[0023] 22) Build a standard YOLOv3-tiny model, and set the number of detection categories to 2, specifically, a human without a hard hat and a human with a hard hat.

[0024] The step 3) specifically includes the following steps:

[0025] 31) Replace all non-1x1 convolution layers and their cascaded batch normalization layers in the standard YOLOv3-tiny model with the dense reparameterization module DR-Block, and the reconstructed model after replacement is denoted as DR-Net; replace the convolution with an input channel number of C in , an output channel number of C out , a convolution kernel of K (K≠1) and a step size of S in the standard YOLOv3-tiny model with F rep (x;C in ,C out ,K,S).

[0026] 32) Using the safety helmet detection dataset, the restructured model DR-Net is trained by using the YOLOv3-tiny standard training parameters and training strategy.

[0027] The step 4) specifically comprises the following steps:

[0028] 41) Convert the basic unit into a single convolution layer, and the weights of the convolution in the basic unit are denoted as w conv , the mean of the batch normalization layer is denoted as μ, the standard deviation is denoted as σ, the scaling coefficient is denoted as γ, and the offset coefficient is denoted as β, then the weights of the reconstructed single convolution layer F′ basic are the bias is

[0029] 42) Convert the convolution using the jump connection into a single convolution layer, and the weights of the converted single convolution layer F′ skip_connect are w′=concat([w prev ,w]) and the bias is b′=concat([b prev ,b]), wherein concat represents the concatenation operation, w prev is the weights of the equivalent convolution, b prev is the bias of the equivalent convolution, w is the weights of the converted convolution, and b is the bias of the converted convolution;

[0030] 43) Convert two concatenated convolutions into a single convolution layer;

[0031] 44) Convert the transformation structure into a single 1x1 convolution layer, convert all the convolutions in the transformation structure into single convolution layers according to step 41), and then sequentially reconstruct all the jump connections into single convolution layers according to step 42), and finally obtain a single convolution layer F′ trans equivalent to the entire transformation structure;

[0032] 45) Convert the DR-Block into a single convolution layer;

[0033] 46) Convert all the DR-Blocks in the DR-Net into single convolution layers according to step 45), so as to obtain the inference stage structure required for deployment;

[0034] 47) Extract the live image frame, input the model obtained in step 46) to perform safety helmet detection, and output the detection result.

[0035] The step 43) specifically comprises the following steps:

[0036] First, transpose the first and second dimensions of the weights w1 of the first convolution, and the result is denoted as w1′, then the weights of the reconstructed single convolution layer F′ cascadeThe weight of the first convolution is w1' = Conv2d(w2, w1), and the bias is b1' = b2 + (b1 x w2), wherein Conv2d represents a two-dimensional convolution operation, b1 is the bias of the first convolution, w2 is the weight of the second convolution, and b2 is the bias of the second convolution.

[0037] The step 45) is specifically:

[0038] First, the transformation structure is converted into a single convolution layer F' according to step 44) trans Then, the single convolution layer F' trans is converted into a single convolution layer F' according to step 43 rep , F' rep , that is, the single convolution layer structure of DR-Block after reparameterization.

[0039] Compared with the prior art, the present application has the following advantages:

[0040] I. In view of the insufficient real-time performance of the existing safety helmet detection algorithm based on deep learning, the present application adopts the yolov3-tiny network for training and reasoning to realize high real-time performance.

[0041] II. The YOLOv3-tiny network is relatively shallow in network depth due to the pursuit of real-time performance, and the accuracy is not high. The present application adopts a network reparameterization method to decouple the training stage and the reasoning stage of the network. In the training stage, a complex structure is used for model learning to improve the network accuracy. In the reasoning stage, the complex structure is equivalent to the original yolov3-tiny simple structure, and the real-time performance of the network is restored.

[0042] III. Due to the limited training data, the model generalization ability is insufficient. The present application adds a deep cascade structure in the network reparameterization structure to realize network over-parameterization and increase the model capacity, thereby producing an implicit regularization effect on the network and improving the generalization ability of the original model.

[0043] IV. Due to the large number of layers in the deep network, the gradient backpropagation has dispersion and explosion problems, and the training is slow and not easy to converge. The existing method does not substantially change the network depth, nor does it perform additional processing on the backpropagation gradient. The present application adjusts the backpropagation gradient through multiple levels of batch normalization layers, better ensures the effectiveness of the gradient backpropagation, and to some extent, avoids gradient dispersion and gradient explosion.

[0044] Fifth, due to the large number of parameters in deep networks, there is a lot of redundancy, which weakens the learning ability of the network. Existing methods mainly enhance the expressive power of a single convolutional layer by designing multi-branch structures of different scales and complexities, but this introduces a lot of feature redundancy, which affects the ability to improve performance. Our method achieves model integration of features at different levels through dense connections, which greatly reduces feature redundancy, improves the learning ability of the network, and further improves the accuracy of the original model. Attached Figure Description

[0045] Figure 1 A flowchart for the method design of this invention is provided.

[0046] Figure 2 This is a diagram of the DR-Block structure and its reparameterization transformation. Detailed Implementation

[0047] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0048] Example

[0049] like Figure 1 As shown, this invention provides a safety helmet detection method based on structural reparameterization for edge devices, comprising the following steps:

[0050] First, construct the dense reparameterization module according to the following steps.

[0051] S1. Define the "basic unit": The parameters of this "basic unit" are: input x, number of input channels C. basic_in Number of output channels C basic_out Kernel size K basic Step size is S basic This "basic unit" consists of one convolutional layer cascaded with one batch normalization layer, wherein the number of input channels of the convolutional layer is C. basic_in The number of output channels is C basic_out The kernel size is K basic The number of channels in the batch normalization layer is C. basic_out Let F be the "basic unit". basic (x;C basic_in C basic_out ,K basic ,S basic );

[0052] S2. Based on S1, construct a "transformation structure": the parameters of this "transformation structure" are input x and the number of input channels C. trans_in Number of output channels: 2 × C trans_in, which includes: first cascading 4 "basic units" to form a deep cascaded network structure, for realizing network over-parameterization, introducing implicit regularization effect, and increasing model capacity; wherein, the introduction of multi-layer batch normalization layer enables the reparameterization module to effectively adjust the network back propagation gradient, avoiding gradient vanishing and gradient explosion, wherein the parameters of the i-th "basic unit" are Then add a skip connection between any two "basic units" to realize model integration of different layer features, thereby reducing feature redundancy; finally, the output of each "basic unit" is spliced into one, and the "transform structure" is denoted as F trans (x;C trans_in );

[0053] S3, on the basis of S1, define a "learning structure" for extracting important features and expanding the receptive field: the parameters of the "learning structure" are input x, input channel number C learn_in , output channel number C learn_out , convolution kernel size K learn , and step size S learn . The structure includes 1 "basic unit" with parameters F learn (x;C learn_in ,C learn_out ,K learn ,S learn );

[0054] S4, on the basis of S2 and S3, construct a "densely reparameterized block" denoted as DR-Block, with parameters: input x, input channel number C rep_in , output channel number C rep_out , convolution kernel size K rep , and step size S rep . The DR-Block includes 1 "transform structure" cascaded with 1 "learning structure", wherein the parameters of the "transform structure" are F trans (x;C rep_in ), and the parameters of the "learning structure" are F learn (x learn ;2×C rep_in ,C rep_out ,K rep ,S rep ), wherein x learn =F trans (x;C rep_in ), and the DR-Block is denoted as F rep (x;C rep_in ,C rep_out ,K rep ,Srep )。

[0055] Second, build YOLOv3-tiny model suitable for safety helmet detection and training data set

[0056] S5, collect and label safety helmet detection data set in construction site and network pictures, and carry out standard data pretreatment;

[0057] S6, build standard YOLOv3-tiny model, and set the number of detection categories as 2 (two categories: human body without safety helmet, human body with safety helmet);

[0058] Third, the original YOLOv3-tiny model is restructured and trained according to the following steps

[0059] S7, given the network model built in S6, replace all non-1x1 convolutional layers and their cascaded batch normalization layers in the original model with DR-Block, and the replaced model is recorded as DR-Net. For the original parameters, the input channel number is C in , the output channel number is C out , the convolution kernel is K (K≠1), and the step is S. Replace F rep (x;C in ,C out ,K,S).

[0060] S8, using the data set prepared in S5, using YOLOv3-tiny standard training parameters and training strategy, training the model restructured in S7;

[0061] Finally, the trained restructured model is converted into an inference model according to the following steps, and detection is carried out

[0062] S9, convert the "basic unit" into a single convolutional layer. The weight of the convolution in the "basic unit" is w conv ; The mean of the batch normalization layer is μ, the standard deviation is σ, the scaling coefficient is γ, and the offset coefficient is β; The restructured single convolutional layer F′ basic , its weight is , and the bias is

[0063] S10, based on S9, convert the convolution using skip connection into a single convolutional layer. The input of the structure can be represented as the original input x after equivalent convolution F prev , the weight of the equivalent convolution is w prev , and the bias is b prev ; The weight of the convolution in this structure is w, and the bias is b. The converted single convolutional layer F′ skip_connect , its weight is w′=concat([wprev w), and the bias is b' = concat([b prev , b]), where concat represents the concatenation operation;

[0064] S11, based on S10, convert the two concatenated convolutions into a single convolution layer. The weights of the first convolution are denoted as w1, and the bias is denoted as b1; the weights of the second convolution are denoted as w2, and the bias is denoted as b2. First, transpose the first and second dimensions of w1, and the result is denoted as w1', then the reconstructed single convolution layer F' cascade , whose weights are w' = Conv2d(w2, w1'), and the bias is b' = b2 + (b1 x w2), where Conv2d represents the two-dimensional convolution operation.

[0065] S12: based on S11, convert the "transform structure" into a single 1x1 convolution layer. First, according to S9, convert all the convolutions in the "transform structure" into a single convolution layer; then, according to S10, sequentially reconstruct all the jump connections into a single convolution layer; finally, obtain a single convolution layer F' trans that is equivalent to the entire "transform structure".

[0066] S13, based on S12, convert the DR-Block into a single convolution layer. First, according to S12, convert the "transform structure" into a single convolution layer F' trans ; then, according to S11, convert the concatenated structure of F' trans and the "learning structure" into a single convolution layer F' rep . F' rep is the single convolution layer structure after reparameterization of the DR-Block.

[0067] S14, finally convert all the DR-Blocks in the DR-Net according to S13 into single convolution layers, thereby obtaining the inference stage structure required for deployment;

[0068] S15, extract the on-site image frame, input it into the model obtained in S14 for safety hat detection, and output the detection result.

[0069] Table 1 shows the experimental results of the present application on the safety hat detection data set, and the results show that the present application can improve the accuracy of the safety hat detection task on the edge device in a plug-and-play manner without changing the original method network structure and without increasing additional inference overhead.

[0070] Table 1 shows the experimental results of the present application on the safety hat detection data set, and the results show that the present application can improve the accuracy of the safety hat detection task on the edge device in a plug-and-play manner without changing the original method network structure and without increasing additional inference overhead.

[0071] Indicator AP@0.5-0.95 AP@0.5 Original model 78.1 92.1 DR-Block 81.2(↑3.1) 96.4(↑4.3)

[0072] Since the DR-Block is plug-and-play, the application can be applied to various tasks, and promote efficient deployment and real-time calculation of deep networks on edge devices.

[0073] In summary, the application designs a dense linear composite structure for the performance improvement of the safety helmet detection algorithm on the edge device, which is used to replace the convolution of the safety helmet detection algorithm in the training stage, and is reconstructed into a simple convolution of the original model in the inference stage, so that the performance is improved while the inference speed is unchanged.

[0074] The application first constructs a dense network reparameterization structure (DR-Block), which first realizes network over-parameterization through deep cascading structure, increases model capacity, thus produces implicit regularization effect on the network, and improves the generalization ability of the original model; secondly, the backpropagation gradient is adjusted by using the cascaded batch normalization layer, which better guarantees the effectiveness of the gradient backpropagation, and to some extent, avoids gradient diffusion and gradient explosion; finally, through dense connection, the model integration of different levels of features is realized, which greatly reduces the feature redundancy and improves the learning ability of the original model. Secondly, the application constructs a yolov3-tiny network with strong real-time performance, and makes it adapt to the safety helmet detection task. Then, the application structures the training and inference structure of the yolov3-tiny network, replaces the original network convolution with the DR-Block in the training stage, and realizes the performance improvement of the network; finally, the DR-Block is equivalently converted into a simple convolution of the original yolov3-tiny network, so that the performance is improved without increasing additional inference overhead, and the application is deployed for the safety helmet detection task.

Claims

1. A safety helmet detection method based on structural reparameterization for edge devices, characterized in that, The method includes the following steps: 1) Construct a dense, heavily parameterized module; 2) Construct a standard YOLOv3-tiny model and training dataset suitable for helmet detection; 3) Reconstruct and train the standard YOLOv3-tiny model; 4) Convert the trained reconstructed model into an equivalent inference model and perform safety helmet detection; Step 1) specifically includes the following steps: 11) Define the basic unit; 12) Construct the transformation structure; 13) Construct a learning structure; 14) Constructing a dense reparameterization module DR-Block: This dense reparameterization module DR-Block consists of one transform structure cascaded with one learning structure. The parameter of the transform structure is F. trans (x;C rep_in The parameter of the learning structure is F. learn (x learn ;2×C rep_in C rep_out ,K rep ,S rep ), and x learn =F trans (x;C rep_in If the dense reparameterization module DR-Block is constructed, then it is denoted as F. rep (x;C rep_in C rep_out K rep S rep ); where C is the number of dense input channels. rep_in Number of dense output channels C rep_out Dense convolution kernel size K rep Dense step size is S rep , In step 11), the basic unit consists of one convolutional layer cascaded with one batch normalization layer, denoted as F. basic (x;C basic_in C basic_out ,K basic ,S basic ), where x is the input, C basic_in For the basic number of input channels, C basic_out For the basic number of output channels, K basic For the basic convolution kernel size, S basic The basic stride is C; specifically, the number of input channels in the convolutional layer is C. basic_in The number of output channels is C basic_out The kernel size is K basic The number of channels in the batch normalization layer is C. basic_out ; Step 12) specifically refers to: First, four basic units are cascaded to form a deep cascaded network structure, achieving overparameterization of the network. Then, skip connections are added between any two basic units to integrate models with features from different levels. Finally, the outputs of each basic unit are concatenated into a single unit, denoted as F. trans (x1;C trans_in ); Cascaded input is x1; Number of cascaded input channels C trans_in ; In step 13), the learning structure consists of one basic unit, denoted as F. learn (x2;C learn_in C learn_out K learn S learn The learning input is x2, and the number of learning input channels is C. learn_in Learning output channel number C learn_out Learning the kernel size K learn The learning step size is S learn .

2. The safety helmet detection method based on structural reparameterization for edge devices according to claim 1, characterized in that, Step 2) specifically includes the following steps: 21) Collect and label safety helmet inspection datasets from construction sites and online images, and perform standard data preprocessing; 22) Build a standard YOLOv3-tiny model and set the number of detection categories to 2, specifically for people not wearing safety helmets and people wearing safety helmets.

3. The safety helmet detection method based on structural reparameterization for edge devices according to claim 2, characterized in that, Step 3) specifically includes the following steps: 31) Replace all non-1×1 convolutional layers and their cascaded batch normalization layers in the standard YOLOv3-tiny model with a dense reparameterization module DR-Block, and denote the reconstructed model after the replacement as DR-Net; for the standard YOLOv3-tiny model with C input channels... in The model outputs C channels. out A convolution with kernel K (K≠1) and stride S is replaced with F. rep (x;C in C out ,K,S), DR-Block is denoted as F rep ; 32) Using the safety helmet detection dataset, the reconstructed model DR-Net was trained using the standard YOLOv3-tiny training parameters and training strategy.

4. The safety helmet detection method based on structural reparameterization for edge devices according to claim 1, characterized in that, Step 4) specifically includes the following steps: 41) Convert the basic unit into a single convolutional layer, and denote the weights of the convolutions in the basic unit as w. conv Let μ be the mean of the batch normalized layer, σ be the standard deviation, γ be the scaling factor, and β be the offset factor. Then, the reconstructed single convolutional layer F′ basic The weight is Bias is 42) Convert the convolution using skip connections into a single convolutional layer, the converted single convolutional layer F′ skip_connect The weights are w′=concat([w prev ,w]), bias is b′=concat([b prev ,b]), where concat represents a cascading operation, w prev b represents the weights of the equivalent convolution. prev is the bias of the equivalent convolution, w is the weight of the transformed convolution, and b is the bias of the transformed convolution. 43) Convert two cascaded convolutions into a single convolutional layer; 44) Convert the transformed structure into a single 1×1 convolutional layer. Following step 41), convert all convolutions in the transformed structure into a single convolutional layer. Then, following step 42), reconstruct all skip connections into single convolutional layers in sequence, finally obtaining a single convolutional layer F′ that is equivalent to the entire transformed structure. trans ; 45) Convert DR-Block to a single convolutional layer; 46) Convert all DR-Blocks in DR-Net into single convolutional layers as per step 45) to obtain the inference stage structure required for deployment; 47) Extract the on-site image frames, input them into the model obtained in step 46) for safety helmet detection, and output the detection results.

5. The safety helmet detection method based on structural reparameterization for edge devices according to claim 4, characterized in that, Step 43) specifically refers to: First, transpose the first and second dimensions of the weights w1 of the first convolution, denoted as w1′. Then, the reconstructed single convolutional layer F′ is obtained. cascade The weights are w′=Conv2d(w2,w1′) and the biases are b′=b2+(b1×w2), where Conv2d represents a two-dimensional convolution operation, b1 is the bias of the first convolution, w2 is the weight of the second convolution, and b2 is the bias of the second convolution.

6. The safety helmet detection method based on structural reparameterization for edge devices according to claim 4, characterized in that, Step 45) specifically refers to: First, following step 44), the transformed structure is converted into a single convolutional layer F′. trans Then a single convolutional layer F′ trans The cascaded structure with the learning structure is converted into a single convolutional layer F′ according to step 43. rep F′ rep This is the single convolutional layer structure after DR-Block reparameterization.

Images