A method for detecting cyclists not wearing helmets based on multi-task deep learning

By using multi-task deep learning method and sub-network structure to share feature extraction module and feature pyramid network in detection without wearing helmets, the problem of detection performance degradation caused by target size differences is solved, and more accurate No helmet wears a test.

CN113887343BActive Publication Date: 2024-11-22ZHEJIANG GONGSHANG UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111086458.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-16
Publication Date
2024-11-22
Estimated Expiration
2041-09-16

AI Technical Summary

Technical Problem

In the prior art, when detecting that electric bicycle riders are not wearing helmets, the performance of the detection system is degraded due to significant differences in target sizes, making it difficult to achieve accurate detection.

Method used

Using a multi-task deep learning method, two subnets B-Net and H-Net are constructed, and the feature extraction module and feature pyramid network are shared, respectively, used to detect the head areas of non-motor vehicles and cyclists, and through preset position relationships. Determine whether helmet is not worn.

Benefits of technology

It improves the accuracy of behavior detection of cyclists without wearing helmets, effectively avoiding the impact of target size differences on the detection system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113887343B_ABST
    Figure CN113887343B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for detecting electric vehicle riders who are not wearing helmets based on multi-task deep learning, and relates to the field of computer vision. The method uses two sub-networks, which are respectively used to detect objects occupying larger areas such as non-motor vehicles and the head area of ​​riders in images. The two sub-networks share the same feature extraction module and part of the layers of the feature pyramid network. In addition, the two sub-networks also have their own input modules and detection modules, thereby effectively avoiding the impact of target sizes that differ greatly on the detection system. Compared with the method used in the prior art to detect targets with significantly different sizes using the same set of weight parameters in a network, the present invention can greatly improve the accuracy of detecting the behavior of riders not wearing helmets.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for detecting the behavior of an electric bicycle rider not wearing a helmet based on multi-task deep learning. Background Art

[0002] Physical experiments show that if a head collision occurs on an electric bicycle rider traveling at a speed of 25 kilometers per hour, the rider's head will bear the weight equivalent to that of two cars; if the rider wears a helmet, the weight on the head can be reduced to about 170 kilograms. Therefore, requiring riders to wear helmets can effectively reduce the casualties in electric bicycle traffic accidents and protect the lives of riders. The existing traffic management system usually educates and handles electric bicycle riders who do not wear helmets manually through intersection control, which has the disadvantages of being time-consuming, labor-intensive, and inefficient. At the technical level, there is an urgent need for a technology and system that uses images or videos as input, uses algorithms to automatically determine if riders are not wearing helmets as required, and gives warnings.

[0003] Chinese Patent 202010644994.6 discloses a method for detecting the wearing of helmets by non-motor vehicle riders based on video streams, combining bounding box regression with SSD non-motor vehicle detection, highlight removal based on threshold method, non-local balance filtering noise reduction and image enhancement; Chinese Patent 202110160273.2 discloses a method for identifying whether a person is wearing a helmet based on the head and shoulder position of the portrait target, which can improve the application accuracy in complex scenes.

[0004] Considering that the area of ​​the non-motor vehicle region in the image is much larger than the cyclist's head area, using the same set of weight parameters of a network to detect the non-motor vehicle and the cyclist's head area is likely to lead to a decrease in the performance of the detection system. Therefore, how to detect the behavior of non-motor vehicle riders not wearing helmets is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to solve the difficulties existing in the prior art when using the same set of weight parameters of a network to detect targets with significantly different sizes, and to provide a method for detecting the behavior of electric bicycle riders not wearing helmets based on multi-task deep learning.

[0006] The technical solutions specifically adopted in the present invention are as follows:

[0007] A method for detecting a cyclist not wearing a helmet based on multi-task deep learning, the method comprising:

[0008] Construct two sub-networks, B-Net and H-Net. The two sub-networks share a feature extraction module and a feature pyramid network for fusing features of each level extracted by the feature extraction module. The two sub-networks also have their own input module and detection module.

[0009] The images to be detected are input into the input modules of B-Net and H-Net respectively. The shared feature extraction module extracts feature maps of different levels, and the feature pyramid network is used to fuse the feature maps of different levels. The fused feature maps are then input into the detection modules of B-Net and H-Net respectively. The detection module of B-Net detects the bounding box of the non-motor vehicle in the image, and the detection module of H-Net detects the bounding box of the head area of ​​the cyclist in the image that contains two classification categories: wearing or not wearing a helmet. If the bounding box of the non-motor vehicle and the bounding box of the head area of ​​the cyclist detected as not wearing a helmet meet the preset positional relationship, it is determined that the cyclist is not wearing a helmet.

[0010] Preferably, the input modules of the B-Net and H-Net have the same structure. According to the data flow during forward propagation, the input data first passes through a convolution layer containing 64 7×7 convolution kernels and a convolution step of 2, followed by a batch normalization operation and a ReLU activation operation, and then passes through a pooling layer with a sampling kernel size of 3×3, a maximum pooling operation and a step of 2, and then passes through the first-level convolution layer group of ResNet101 after the pooling layer.

[0011] Preferably, the feature extraction module shared by the B-Net and H-Net comprises the 2nd, 3rd and 4th level convolutional layer groups of ResNet101, and a feature pyramid network is used to fuse the feature maps output by each convolutional layer group;

[0012] The output feature map C4 of the 4th level convolution layer group of ResNet101 is used as the input of the top layer of the feature pyramid network. It first passes through a convolution layer containing 256 1×1 convolution kernels and a convolution step size of 1 to generate a feature map Then After a convolution layer containing 256 3×3 convolution kernels and a convolution step size of 1, the feature map P4 is formed;

[0013] The output feature map C3 of the third-level convolutional layer group of ResNet101 first passes through a convolutional layer containing 256 1×1 convolution kernels and a convolution step size of 1 in the feature pyramid network, and the resulting feature map is the same as The feature map formed after the upsampling operation is summed element by element to generate a feature map Then After a convolution layer containing 256 3×3 convolution kernels and a convolution step size of 1, the feature map P3 is formed;

[0014] The output feature map C2 of the second-level convolutional layer group of ResNet101 first passes through a convolutional layer containing 256 1×1 convolution kernels and a convolution step size of 1 in the feature pyramid network, and the resulting feature map is the same as The feature map formed after the upsampling operation is summed element by element to generate a feature map Then After a convolution layer containing 256 3×3 convolution kernels and a convolution step size of 1, the feature map P2 is formed.

[0015] Preferably, the detection modules of the B-Net and H-Net have the same structure, which includes a classification subnetwork and a regression subnetwork;

[0016] The classification subnetwork includes 5 convolutional layers. The first 4 convolutional layers have the same structure, each containing 256 3×3 convolution kernels, with a convolution step of 1, and each convolutional layer is followed by a ReLU activation operation. The fifth convolutional layer contains K×A 3×3 convolution kernels, with a convolution step of 1, where K is the number of classification categories and A is the number of detection boxes for each point. Finally, the Sigmoid function is used to output the predicted classification category for each detection box.

[0017] The regression subnetwork includes 5 convolutional layers, the first 4 convolutional layers have the same structure, all of which contain 256 3×3 convolution kernels, the convolution step size is 1, and each convolution layer is followed by a ReLU activation operation, the 5th convolutional layer contains 4A 3×3 convolution kernels, the convolution step size is 1, where A is the number of preset detection frames for each preset point, if the classification subnetwork predicts that a certain point (m, n) detects a certain type of target in the i-th preset detection frame, where 0≤i<A, and i is an integer, then in the output feature map of the 5th convolutional layer of the regression subnetwork, the values ​​of channel 4i and channel 4i+1 represent the offsets in the width and height directions between the center of the preset detection frame and the center of the actual detection frame, respectively, and the values ​​of channel 4i+2 and channel 4i+3 represent the ratios of the width and height of the preset detection frame to the width and height of the actual detection frame, respectively.

[0018] Preferably, the width and height of the image to be detected are M and N respectively, and the image to be detected is input into the input module of B-Net according to the original size, and the output sizes of each level of the feature pyramid network are respectively and The feature maps P2, P3, and P4 are respectively input into the detection module of B-Net to detect non-motor vehicle objects in the image to be detected.

[0019] Furthermore, the width and height of the image to be detected are M and N respectively, and the image to be detected is magnified by 2 times in both length and width directions and then input into the input module of H-Net. After passing through the input module of H-Net, a The feature map is denoted as F2;

[0020] After the image to be detected is input into the input module of B-Net according to its original size, the size obtained by the feature pyramid network is The feature map P2 is denoted as F1;

[0021] The feature map F1 is upsampled by 2 times in width and height and then concatenated with the feature map F2 into a The feature map is input into the detection module of H-Net to detect the head area of ​​cyclists with and without helmets in the image to be detected.

[0022] Preferably, when determining whether the rider is not wearing a helmet, the upper left corner of the non-motor vehicle area enclosing frame B detected by B-Net is used as the coordinate origin. If the center of the rider head area enclosing frame R detected by H-Net is at a distance Δx from the origin in the X direction and Δy from the origin in the Y direction, and the width and height are w and h respectively, and the width and height of the enclosing frame B detected by B-Net are W and H respectively, then the center coordinates of the enclosing frame R detected by H-Net are calculated as: x = Δx / W, y = Δy / H, and the width and height of R relative to the enclosing frame detected by B-Net are: And will As the feature of the rider's head area detected by H-Net; for the bounding box R and the bounding box B, there are two possibilities: the area surrounded by R is the head area of ​​the rider surrounded by B, recorded as c = 1, or the area surrounded by R is not the head area of ​​the rider surrounded by B, recorded as c = 0; the detection of the rider of the electric bicycle not wearing a helmet is determined by the following probability problem: if the area surrounded by R is detected as the head area without a helmet, the area surrounded by B is detected as the electric vehicle area, and Then the prediction alarm of the electric bicycle rider not wearing a helmet is output; and According to the Bayesian formula:

[0023] Where c=1 or 0.

[0024] Preferably, the Using the equivalent formula Instead, and The calculation formula is: where P(x|c), P(y|c), and Learned through training samples.

[0025] Preferably, in the 1st, 2nd, 3rd and 4th level convolutional layer groups of the ResNet101, the 1st level convolutional layer group includes 3 residual blocks, the 2nd level convolutional layer group includes 4 residual blocks, the 3rd level convolutional layer group includes 23 residual blocks, and the 4th level convolutional layer group includes 3 residual blocks, and the i-th residual block of the 1st, 2nd, 3rd and 4th level convolutional layer groups is respectively recorded as ResBlock1_i, ResBlock2_i, ResBlock3_i and ResBlock4_i, where i is an integer and 1≤i≤n B , n Bis the number of residual blocks in the convolutional layer group; the first residual blocks of the 1st, 2nd, 3rd and 4th level convolutional layer groups are ResBlock1_1, ResBlock2_1, ResBlock3_1 and ResBlock4_1 respectively, and their structures include two branches respectively, where the main branch contains 3 convolutional layers, the first convolutional layer contains C 1×1 convolutional kernels, the second convolutional layer contains C 3×3 convolutional kernels, and the third convolutional layer contains 4C 1×1 convolutional kernels. A batch normalization layer and a ReLU layer are connected in sequence after each convolutional layer; the 3 convolutional layers of ResBlock1_1 The convolution step size of the convolution layer is 1; the convolution step size of the first convolution layer of ResBlock2_1, ResBlock3_1 and ResBlock4_1 is 2, and the convolution step size of other convolution layers is 1; the other branch of ResBlock1_1, ResBlock2_1, ResBlock3_1 and ResBlock4_1 is a shortcut branch, which contains a convolution layer followed by a batch normalization layer; the convolution layer of the shortcut branch of ResBlock1_1 has 4C 1×1 convolution kernels and a convolution step size of 1, Res The convolutional layer of the shortcut branch of ResBlock2_1, ResBlock3_1 and ResBlock4_1 has 4C 1×1 convolution kernels and a convolution step of 2; the output of the last convolutional layer of the main branch and the output of the shortcut branch are summed element by element as the output of the residual block; in the 1st, 2nd, 3rd and 4th level convolutional layer groups of ResNet101, any residual block except ResBlock1_1, ResBlock2_1, ResBlock3_1 and ResBlock4_1 has a structure including two branches, One of the main branches contains 3 convolutional layers, of which the first convolutional layer contains C 1×1 convolutional kernels, the second convolutional layer contains C 3×3 convolutional kernels, and the third convolutional layer contains 4C 1×1 convolutional kernels. A batch normalization layer and a ReLU layer are connected in sequence after each convolutional layer. The convolution step of all convolutional layers is 1. The other branch directly copies the feature map input to the residual block, and performs element-by-element summation with the output of the last convolutional layer of the main branch as the output of the residual block; in the 1st, 2nd, 3rd and 4th level convolutional layer groups, the value of C is 64, 128, 256 and 512 respectively.

[0026] Preferably, the non-motor vehicle is a bicycle or an electric bicycle.

[0027] Compared with the prior art, the present invention provides a method for detecting electric vehicle riders who are not wearing helmets based on multi-task deep learning. The method includes two sub-networks, which are used to detect objects occupying larger areas such as non-motor vehicles and the head area of ​​riders in the image. The two sub-networks share the same feature extraction module and part of the layers of the feature pyramid network. In addition, the two sub-networks also have their own input modules and detection modules, which can effectively avoid the impact of the large difference in target size on the detection system. Compared with the method used in the prior art to detect targets with significant size differences using the same set of weight parameters in a network, the present invention can greatly improve the accuracy of detecting the behavior of riders not wearing helmets. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of a method for detecting the behavior of an electric bicycle rider not wearing a helmet based on multi-task deep learning of the present invention;

[0029] Figure 2 This is a schematic diagram of the parameters of the cyclist's head area;

[0030] Figure 3 This is an example of the result of applying the method provided in an embodiment of the present invention to detect the behavior of an electric bicycle rider not wearing a helmet. DETAILED DESCRIPTION

[0031] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the essence of the present invention.

[0032] like Figure 1 As shown, the present invention provides a method for detecting the behavior of non-motor vehicle riders not wearing helmets based on multi-task deep learning. The specific technical solutions adopted by the present invention are as follows:

[0033] (1) The present invention uses two sub-networks: B-Net and H-Net to implement a method for detecting the behavior of non-motor vehicle riders not wearing helmets, wherein the task of B-Net is to detect non-motor vehicle objects such as electric bicycles and bicycles that occupy a large area in the image, and the task of H-Net is to detect the head area of ​​riders wearing and not wearing helmets. The two sub-networks B-Net and H-Net share a feature extraction module and a feature pyramid network for fusing the features of each level extracted by the feature extraction module, and the two sub-networks also each have an input module and a detection module. The method of detecting the behavior of non-motor vehicle riders not wearing helmets using the two sub-networks B-Net and H-Net is as follows:

[0034] The image to be detected is input into the input modules of B-Net and H-Net respectively, and the feature maps of different levels are extracted by the shared feature extraction module, and the feature maps of different levels are fused using the feature pyramid network, and then the fused feature maps are input into the detection modules of B-Net and H-Net respectively, and the non-motor vehicle encircling frame in the image is detected by the detection module of B-Net, and the cyclist head area encircling frame of the image containing two classification categories of wearing or not wearing a helmet is detected by the detection module of H-Net; if the non-motor vehicle encircling frame and the cyclist head area encircling frame detected as not wearing a helmet meet the preset positional relationship, it is determined that the cyclist is not wearing a helmet. In an embodiment of the present invention, the preset positional relationship here can be obtained according to the intersection and union ratio between the non-motor vehicle encircling frame and the cyclist head area encircling frame detected as not wearing a helmet when the cyclist is driving a non-motor vehicle, and a minimum threshold of the intersection and union ratio for determination can be set according to a large number of samples.

[0035] The specific structures and detection processes of the two sub-networks, B-Net and H-Net, are described in detail below.

[0036] (2) The input modules of B-Net and H-Net have the same structure. According to the data flow during forward propagation, the input data first passes through a convolutional layer containing 64 7×7 convolution kernels and a convolution step of 2. The convolutional layer is followed by a batch normalization operation and a ReLU activation operation. Then, it passes through a pooling layer with a sampling kernel size of 3×3, a maximum pooling operation and a step of 2. After the pooling layer, it passes through the first-level convolutional layer group of ResNet101.

[0037] (3) The feature extraction module shared by B-Net and H-Net contains the 2nd, 3rd and 4th level convolutional layer groups of ResNet101, and the feature pyramid network is used to fuse the feature maps output by each convolutional layer group;

[0038] Assume that the output feature map of the 4th level convolution layer group of ResNet101 is C4. C4 is used as the input of the top layer of the feature pyramid network. It first passes through a convolution layer containing 256 1×1 convolution kernels and a convolution step size of 1 to generate a feature map Then After a convolution layer containing 256 3×3 convolution kernels and a convolution step size of 1, the feature map P4 is formed;

[0039] Assume that the output feature map of the third-level convolutional layer group of ResNet101 is C3. In the feature pyramid network, C3 first passes through a convolutional layer containing 256 1×1 convolution kernels and a convolution step size of 1. The resulting feature map is the same as The feature map formed after the upsampling operation is summed element by element to generate a feature map Then After a convolution layer containing 256 3×3 convolution kernels and a convolution step size of 1, the feature map P3 is formed;

[0040] Assume that the output feature map of the second-level convolutional layer group of ResNet101 is C2. In the feature pyramid network, C2 first passes through a convolutional layer containing 256 1×1 convolution kernels and a convolution step size of 1. The resulting feature map is the same as The feature map formed after the upsampling operation is summed element by element to generate a feature map Then After a convolution layer containing 256 3×3 convolution kernels and a convolution step size of 1, the feature map P2 is formed;

[0041] If the width and height of the original size of the image to be detected are M and N respectively, then the image to be detected is input into B-Net according to its original size, and the output feature maps of each level of the feature pyramid network are P2, P3, and P4, and their sizes are and They will be input into the detection module of B-Net respectively to detect non-motor vehicle objects such as electric bicycles and bicycles in the input image;

[0042] (4) The detection modules of B-Net and H-Net have the same structure, which includes a classification subnetwork and a regression subnetwork, wherein the classification subnetwork includes 5 convolutional layers, the first 4 convolutional layers have the same structure, each including 256 3×3 convolution kernels, the convolution step size is 1, and each convolutional layer is followed by a ReLU activation operation, the 5th convolutional layer includes K×A 3×3 convolution kernels, the convolution step size is 1, wherein K is the number of classification categories, and A is the number of detection boxes preset for each point. In one embodiment of the present invention, A=9 is taken, and finally the predicted classification category is output for each detection box using the Sigmoid function;

[0043] The fifth convolutional layer of the classification subnetwork will generate a feature map containing K×A channels. During the training phase, if a region is marked as belonging to the kth class object, the intersection over union (IoU) of the ath preset detection box of point (m, n) and the region is the maximum of the intersection over union of all preset detection boxes and the region, and is greater than 0.6, then the preset detection box is expected to have a positive sample response. If it is less than 0.4, it is expected to have a negative sample response. If it is between 0.4 and 0.6, it is ignored, where the value of the intersection over union is equal to the ratio of the intersection to the union of the two regions. The k×ath channel of the feature map output by the fifth convolutional layer of the classification subnetwork will output the probability that the ath preset detection box corresponding to the point detects the kth class object. For positive samples, the expected probability value is close to 1, and for negative samples, the expected probability value is close to 0. For each positive or negative sample, if its expected output probability is c and the actual output is p, the classification error loss is calculated as follows:

[0044] L class = -α(1-q) γ log(q) (1)

[0045] Where q is determined by the following formula

[0046]

[0047] α and γ are two parameters that adjust the contribution of samples of different categories;

[0048] The regression subnetwork includes 5 convolutional layers. The first 4 convolutional layers have the same structure, all of which contain 256 3×3 convolution kernels with a convolution step of 1, and each convolution layer is followed by a ReLU activation operation. The 5th convolutional layer contains 4A 3×3 convolution kernels with a convolution step of 1, where A is the number of preset detection frames for each point. If the classification subnetwork predicts that a certain type of target is detected in the i-th preset detection frame, where 0≤i<A, and i is an integer, then in the output feature map of the 5th convolutional layer of the regression subnetwork, the values ​​of channel 4i and channel 4i+1 represent the offsets in the width and height directions between the center of the preset detection frame and the center of the actual detection frame, respectively, and the values ​​of channel 4i+2 and channel 4i+3 represent the ratios of the width and height of the preset detection frame to the width and height of the actual detection frame, respectively.

[0049] If a preset detection box detects a certain type of object, the position and width and height differences between the preset detection box output by the regression subnetwork and the annotated target box are Δ x , Δ y , Δ w and Δ h , then the regression loss is calculated as follows:

[0050]

[0051] Among them, smooth L1 The function is calculated as follows:

[0052]

[0053] In the training phase, the parameters of B-Net and H-Net are updated by back-propagation with the goal of minimizing the sum of the above classification loss function and regression loss function.

[0054] (5) See Figure 1 When using H-Net to detect the head area of ​​a cyclist, considering that the area of ​​the head area of ​​a cyclist in an image is much smaller than the area occupied by non-motorized vehicles such as electric bicycles and bicycles, the embodiment of the present invention enlarges the image to be detected to twice the original image in both length and width directions as the input of the H-Net network;

[0055] Assume that the original width and height of the original image to be detected input to B-Net are M and N respectively. After passing through the input module and the 2nd, 3rd and 4th level convolutional layers of ResNet101 described in (3), and the feature pyramid network, a The feature map is denoted as F1;

[0056] The width and height of the image to be detected after being enlarged and input to H-Net are 2M and 2N respectively. After passing through the input module, a The feature map is denoted as F2;

[0057] Feature map F1 is upsampled by 2 times in width and height and concatenated with F2 into one The feature map is input into the detection module of H-Net to detect the head areas of cyclists wearing and not wearing helmets. Each cyclist's head area is labeled with a classification of wearing a helmet or not wearing a helmet.

[0058] (6) See Figure 2 , Figure 2The figure shows a non-motor vehicle and its rider detected by B-Net, and the rider's head region detected by H-Net. Intuitively, there is a certain geometric constraint relationship between the non-motor vehicle region enclosing frame B detected by B-Net and the rider's head region enclosing frame R detected by H-Net. The present invention uses the upper left corner point of the enclosing frame detected by B-Net as the coordinate origin. If the center of the enclosing frame detected by H-Net is Δx from the origin in the X direction and Δy from the origin in the Y direction, and the width and height are w and h respectively, and the width and height of the enclosing frame detected by B-Net are W and H respectively, then the center coordinates of the enclosing frame R detected by H-Net are calculated as: x = Δx / W, y = Δy / H, and the width and height of R relative to the enclosing frame detected by B-Net are: And will As the feature of the rider's head area detected by H-Net; for the bounding box R and the bounding box B, there are two possibilities: the area surrounded by R is the head area of ​​the rider surrounded by B, recorded as c=1, or the area surrounded by R is not the head area of ​​the rider surrounded by B, recorded as c=0; Therefore, the problem of detecting electric bicycle riders without helmets can be reduced to the following probability problem: if the area surrounded by R is detected as the head area without helmets, the area surrounded by B is detected as the electric vehicle area, and Then the prediction alarm of the electric bicycle rider not wearing a helmet is output, where and It can be calculated according to the Bayesian formula, and according to the Bayesian theorem:

[0059] Where c = 1 or 0

[0060] Since the denominator on the right side of the above equation is independent of the category, further, if The output is the prediction that the e-bike rider is not wearing a helmet.

[0061] Since x,y, The properties are independent of each other, so the numerator on the right side of the equation can be calculated equivalently as follows:

[0062]

[0063] Among them, P(x|c), P(y|c), and It can be learned through training samples.

[0064] In addition, the first level of ResNet101 described in (2) of the present invention, the second, third and fourth level convolutional layer groups of ResNet101 described in (3), wherein the first level convolutional layer group includes 3 residual blocks, the second level convolutional layer group includes 4 residual blocks, the third level convolutional layer group includes 23 residual blocks, and the fourth level convolutional layer group includes 3 residual blocks, the i-th residual block of the first, second, third and fourth level convolutional layer groups is respectively denoted as ResBlock1_i, ResBlock2_i, ResBlock3_i and ResBlock4_i, where i is an integer and 1≤i≤n B , n Bis the number of residual blocks in the convolutional layer group; the first residual blocks of the 1st, 2nd, 3rd and 4th level convolutional layer groups are ResBlock1_1, ResBlock2_1, ResBlock3_1 and ResBlock4_1 respectively, and their structures include two branches respectively, where the main branch contains 3 convolutional layers, the first convolutional layer contains C 1×1 convolution kernels, the second convolutional layer contains C 3×3 convolution kernels, and the third convolutional layer contains 4C 1×1 convolution kernels. A batch normalization layer and a ReLU layer are connected in sequence after each convolutional layer; the 3 convolutional layers of ResBlock1_1 The convolution step size of the convolution layer is 1. The convolution step size of the first convolution layer of ResBlock2_1, ResBlock3_1 and ResBlock4_1 is 2, and the convolution step size of other convolution layers is 1. The other branch of ResBlock1_1, ResBlock2_1, ResBlock3_1 and ResBlock4_1 is a shortcut branch, which contains a convolution layer followed by a batch normalization layer. The convolution layer of the shortcut branch of ResBlock1_1 has 4C 1×1 convolution kernels and a convolution step size of 1. The convolutional layer of the shortcut branch of esBlock2_1, ResBlock3_1 and ResBlock4_1 has 4C 1×1 convolution kernels and a convolution step of 2; the output of the last convolutional layer of the main branch and the output of the shortcut branch are summed element by element as the output of the residual block; in the 1st, 2nd, 3rd and 4th level convolutional layer groups of ResNet101, any residual block except ResBlock1_1, ResBlock2_1, ResBlock3_1 and ResBlock4_1, its structure includes two-way split One of the main branches contains 3 convolutional layers, the first convolutional layer contains C 1×1 convolutional kernels, the second convolutional layer contains C 3×3 convolutional kernels, and the third convolutional layer contains 4C 1×1 convolutional kernels. A batch normalization layer and a ReLU layer are connected in sequence after each convolutional layer, and the convolution step size of all convolutional layers is 1; the other branch directly copies the feature map input to the residual block, and performs element-by-element summation with the output of the last convolutional layer of the main branch as the output of the residual block; in the 1st, 2nd, 3rd and 4th convolutional layer groups, the values ​​of C are 64, 128, 256 and 512 respectively.

[0065] like Figure 3 The figure shows the result of non-motor vehicle and rider head area detection on video images by the method provided in an embodiment of the present invention. The non-motor vehicle area detected by B-Net and the rider head area detected by H-Net are marked with rectangular boxes. The results show that the method provided in an embodiment of the present invention can well detect the head areas of bicycles and electric bicycles, with and without helmets, and can accurately give an alarm for electric bicycle riders not wearing helmets.

[0066] The above description is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any modification or replacement made within the spirit and principle of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for detecting cyclists not wearing helmets based on multi-task deep learning. It is characterized in that The method includes: Construct two sub-networks, B-Net and H-Net. The two sub-networks share a feature extraction module and a feature pyramid network for fusing features of each level extracted by the feature extraction module. The two sub-networks also have their own input module and detection module. The image to be detected is input into the input modules of B-Net and H-Net respectively, and the shared feature extraction module extracts feature maps of different levels, and the feature pyramid network is used to fuse the feature maps of different levels, and then the fused feature maps are input into the detection modules of B-Net and H-Net respectively, and the detection module of B-Net detects the non-motor vehicle encircling frame in the image, and the detection module of H-Net detects the cyclist head region encircling frame in the image containing two classification categories of wearing or not wearing a helmet; if the non-motor vehicle encircling frame and the cyclist head region encircling frame detected as not wearing a helmet meet the preset positional relationship, it is determined that the cyclist is not wearing a helmet; The input modules of the B-Net and H-Net have the same structure. According to the data flow direction during forward propagation, the input data first passes through a convolutional layer containing 64 7×7 convolution kernels and a convolution step of 2, followed by a batch normalization operation and a ReLU activation operation, and then passes through a pooling layer with a sampling kernel size of 3×3, a maximum pooling operation and a step of 2, and then passes through the first-level convolutional layer group of ResNet101. The feature extraction module shared by the B-Net and H-Net includes the 2nd, 3rd and 4th level convolutional layer groups of ResNet101, and a feature pyramid network is used to fuse the feature maps output by each convolutional layer group; The output feature map C of the 4th convolutional layer group of ResNet101 is 4 As the input of the top layer of the feature pyramid network, it first passes through a convolution layer containing 256 1×1 convolution kernels and a convolution step size of 1 to generate a feature map Then After a convolution layer containing 256 3×3 convolution kernels and a convolution step size of 1, the feature map P is formed. 4 ; Output feature map C of the 3rd convolutional layer group of ResNet101 3 In the feature pyramid network, the first step is to pass through a convolution layer containing 256 1×1 convolution kernels and a convolution step size of 1, and the resulting feature map is the same as The feature map formed after the upsampling operation is summed element by element to generate a feature map Then After a convolution layer containing 256 3×3 convolution kernels and a convolution step size of 1, the feature map P is formed. 3 ; Output feature map C of the second-level convolutional layer group of ResNet101 2 In the feature pyramid network, the first step is to pass through a convolution layer containing 256 1×1 convolution kernels and a convolution step size of 1, and the resulting feature map is the same as The feature map formed after the upsampling operation is summed element by element to generate a feature map Then After a convolution layer containing 256 3×3 convolution kernels and a convolution step size of 1, the feature map P is formed. 2 ; The detection modules of the B-Net and H-Net have the same structure, which includes a classification subnetwork and a regression subnetwork; The classification subnetwork includes 5 convolutional layers. The first 4 convolutional layers have the same structure, each containing 256 3×3 convolution kernels, with a convolution step of 1, and each convolutional layer is followed by a ReLU activation operation. The fifth convolutional layer contains K×A 3×3 convolution kernels, with a convolution step of 1, where K is the number of classification categories and A is the number of detection boxes for each point. Finally, the Sigmoid function is used to output the predicted classification category for each detection box. The regression subnetwork includes 5 convolutional layers, the first 4 convolutional layers have the same structure, all of which contain 256 3×3 convolution kernels, the convolution step size is 1, and each convolution layer is followed by a ReLU activation operation, the 5th convolutional layer contains 4A 3×3 convolution kernels, the convolution step size is 1, where A is the number of preset detection frames for each preset point, if the classification subnetwork predicts that a certain point (m, n) detects a certain type of target in the i-th preset detection frame, where 0≤i<A, and i is an integer, then in the output feature map of the 5th convolutional layer of the regression subnetwork, the values ​​of channel 4i and channel 4i+1 represent the offsets in the width and height directions between the center of the preset detection frame and the center of the actual detection frame, respectively, and the values ​​of channel 4i+2 and channel 4i+3 represent the ratios of the width and height of the preset detection frame to the width and height of the actual detection frame, respectively.

2. The method for detecting a rider not wearing a helmet based on multi-task deep learning as claimed in claim 1, It is characterized in that The width and height of the image to be detected are M and N respectively. The image to be detected is input into the input module of B-Net according to its original size. The output sizes of each level of the feature pyramid network are and The feature map P 2 , P 3 , P 4 , the three feature maps are then input into the detection module of B-Net to detect non-motor vehicle objects in the image to be detected.

3. The method for detecting a rider not wearing a helmet based on multi-task deep learning as claimed in claim 2, It is characterized in that The width and height of the image to be detected are M and N respectively. The image to be detected is magnified by 2 times in both length and width directions and then input into the input module of H-Net. After passing through the input module of H-Net, a The feature map is denoted as F 2 ; After the image to be detected is input into the input module of B-Net according to its original size, the size obtained by the feature pyramid network is The feature map P 2 , denoted as F 1 ; The feature map F 1 After upsampling by 2 times in width and height, the feature map F 2 Splice into one The feature map is input into the detection module of H-Net to detect the head area of ​​cyclists with and without helmets in the image to be detected.

4. The method for detecting a rider not wearing a helmet based on multi-task deep learning as claimed in claim 1, It is characterized in that When determining whether the rider is not wearing a helmet, the upper left corner of the non-motor vehicle area bounding box B detected by B-Net is used as the coordinate origin. If the distance between the center of the rider head area bounding box R detected by H-Net and the origin in the X direction is Δx, the distance between the center of the rider head area bounding box R detected by H-Net and the origin in the Y direction is Δy, the width and height are w and h respectively, and the width and height of the bounding box B detected by B-Net are W and H respectively, then the center coordinates of the bounding box R detected by H-Net are calculated as: x = Δx / W, y = Δy / H, and the width and height of R relative to the bounding box detected by B-Net are: And will As the feature of the rider's head area detected by H-Net; for the bounding box R and the bounding box B, there are two possibilities: the area surrounded by R is the head area of ​​the rider surrounded by B, recorded as c = 1, or the area surrounded by R is not the head area of ​​the rider surrounded by B, recorded as c = 0; the detection of the rider of the electric bicycle not wearing a helmet is determined by the following probability problem: if the area surrounded by R is detected as the head area without a helmet, the area surrounded by B is detected as the electric vehicle area, and Then the prediction alarm of the electric bicycle rider not wearing a helmet is output; and According to the Bayesian formula: Where c=1 or 0.

5. The method for detecting a rider not wearing a helmet based on multi-task deep learning as claimed in claim 1, It is characterized in that Said Using the equivalent formula Instead, and The calculation formula is: where P(x|c), P(y|c), and Learned through training samples.

6. According to claim 1, the method for detecting a rider not wearing a helmet based on multi-task deep learning, It is characterized in that In the 1st, 2nd, 3rd and 4th level convolutional layer groups of the ResNet101, the 1st level convolutional layer group includes 3 residual blocks, the 2nd level convolutional layer group includes 4 residual blocks, the 3rd level convolutional layer group includes 23 residual blocks, and the 4th level convolutional layer group includes 3 residual blocks. The i-th residual block of the 1st, 2nd, 3rd and 4th level convolutional layer groups is respectively denoted as ResBlock1_i, ResBlock2_i, ResBlock3_i and ResBlock4_i, where i is an integer and 1≤i≤n B , n B is the number of residual blocks in the convolutional layer group; the first residual blocks of the 1st, 2nd, 3rd and 4th level convolutional layer groups are ResBlock1_1, ResBlock2_1, ResBlock3_1 and ResBlock4_1 respectively, and their structures include two branches respectively, where the main branch contains 3 convolutional layers, the first convolutional layer contains C 1×1 convolutional kernels, the second convolutional layer contains C 3×3 convolutional kernels, and the third convolutional layer contains 4C 1×1 convolutional kernels. A batch normalization layer and a ReLU layer are connected in sequence after each convolutional layer; the 3 convolutional layers of ResBlock1_1 The convolution step size of the convolution layer is 1; the convolution step size of the first convolution layer of ResBlock2_1, ResBlock3_1 and ResBlock4_1 is 2, and the convolution step size of other convolution layers is 1; the other branch of ResBlock1_1, ResBlock2_1, ResBlock3_1 and ResBlock4_1 is a shortcut branch, which contains a convolution layer followed by a batch normalization layer; the convolution layer of the shortcut branch of ResBlock1_1 has 4C 1×1 convolution kernels and a convolution step size of 1, Res The convolutional layer of the shortcut branch of ResBlock2_1, ResBlock3_1 and ResBlock4_1 has 4C 1×1 convolution kernels and a convolution step of 2; the output of the last convolutional layer of the main branch and the output of the shortcut branch are summed element by element as the output of the residual block; in the 1st, 2nd, 3rd and 4th level convolutional layer groups of ResNet101, any residual block except ResBlock1_1, ResBlock2_1, ResBlock3_1 and ResBlock4_1 has a structure including two branches, One of the main branches contains 3 convolutional layers, of which the first convolutional layer contains C 1×1 convolutional kernels, the second convolutional layer contains C 3×3 convolutional kernels, and the third convolutional layer contains 4C 1×1 convolutional kernels. A batch normalization layer and a ReLU layer are connected in sequence after each convolutional layer. The convolution step of all convolutional layers is 1. The other branch directly copies the feature map input to the residual block, and performs element-by-element summation with the output of the last convolutional layer of the main branch as the output of the residual block; in the 1st, 2nd, 3rd and 4th level convolutional layer groups, the value of C is 64, 128, 256 and 512 respectively.

7. According to claim 1, the method for detecting a rider not wearing a helmet based on multi-task deep learning, It is characterized in that The non-motor vehicle is a bicycle or an electric bicycle.