A mask wearing manner detection method

Abstract

The application discloses a mask wearing mode detection method, which comprises the following steps: firstly, using a YOLO series basic network to perform feature extraction and fusion on an input image to generate three layers of feature maps; then, using five different convolution layers and pooling layer operations to splice to generate three layers of enhanced feature maps; subsequently, using an attention mechanism module, a convolution operation and a Softmax operation to jointly process to obtain three layers of noise reduction feature maps and calculate attention loss; finally, using a YOLO series prediction module to classify and coordinate regress the three layers of noise reduction feature maps and the attention loss, and outputting a marked mask position and wearing mode after non-maximum suppression processing. The application makes the feature information of the target in the image more clear, the useless noise of the background information is effectively suppressed, the detection precision is obviously improved, and the method has good application prospect and economic benefits.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of natural image target detection, and relates to a mask wearing method, in particular to a mask wearing method based on an improved single-stage target detection network. BACKGROUND

[0002] Nowadays, detecting and supervising people to wear masks in public places is an important part of effectively preventing the spread of the epidemic. At present, most areas still use manual detection when carrying out this work, which has the problems of small monitoring range, low detection efficiency, increased risk of epidemic spread and other constraints. With the rise of deep learning, the target detection algorithm through camera data collection and computer data processing provides a new idea for replacing manual detection of mask wearing.

[0003] Target detection algorithm has always been a widely concerned research hotspot in the field of computer vision. Its wide application, accurate detection and wide coverage fully show the important research value of target detection. Based on the process of target prediction box from nothing to something and the change of prediction box in target detection, it can be roughly divided into two directions: single-stage algorithm and two-stage algorithm. Among them, the more classic single-stage algorithm is YOLO (You Only Look Once) series algorithm and SSD (Single Shot MultiBox Detector) series algorithm, and the Faster R-CNN series algorithm in two-stage algorithm is widely used. Single-stage algorithm simultaneously processes the positioning of target position and the classification of target category, that is, it does not need to produce a prediction box in advance, but directly produces the probability and position information of the target category, and only through successive detection can the final detection result be obtained. Therefore, the advantage of single-stage algorithm is fast processing speed, but the precision is lost. Two-stage algorithm first finds the position of the target to get the prediction box, and then classifies and regresses the prediction box. Compared with single-stage algorithm, two-stage algorithm is slow in processing speed but high in precision.

[0004] The target detection of mask wearing method needs to be close to real-time detection speed, and the single-stage algorithm is more suitable for the application background of this detection technology. However, when collecting image information, the mask is small in size, the non-standard wearing of the mask is easy to confuse and there is a possibility of similar background information, which leads to the fact that the existing detection algorithm cannot meet the detection requirements. To improve the detection accuracy, the core is to effectively extract and enhance the features of the target information in a complex background, make up for the shortcomings of target detection in processing feature information, and solve the problems of background noise drowning object information and object boundary blur. How to process the extracted target feature information is the key technology. SUMMARY

[0005] The technical problem to be solved by the present application in the prior art is to provide a mask wearing method based on an improved single-stage target detection network, which can effectively improve the detection accuracy of natural image targets.

[0006] To solve the above technical problems, the mask wearing method of the present application comprises the following steps:

[0007] Step 1: using the YOLO series basic network to extract and fuse the features of the input image, generating three layers of feature maps with different resolutions and different semantic strengths;

[0008] Step 2: using five different convolutional layers and pooling layer operations to process the three layers of feature maps obtained in step 1, and splicing the generated results in the channel dimension to produce three layers of enhanced feature maps with multi-level enhanced target feature information;

[0009] Step 3: using the attention mechanism module, convolution operation and Softmax operation to process each layer of enhanced feature map obtained in step 2, multiplying the output result with the input enhanced feature map to obtain three layers of denoising feature maps, and calculating the attention loss;

[0010] Step 4: using the YOLO series prediction module to classify and coordinate regression on the three layers of denoising feature maps and attention loss obtained in step 3, and using the output after non-maximum suppression processing to mark the mask position and wearing method.

[0011] Further, the step two of using five different convolutional layers and pooling layer operations to process the three layers of feature maps obtained in step 1, and splicing the generated results in the channel dimension to produce three layers of enhanced feature maps with multi-level enhanced target feature information comprises:

[0012] Step 2.1: extracting the features of the input feature map through a convolutional layer with a size of 1x1;

[0013] Step 2.2: extracting the features of the input feature map through three layers of convolutional layers with sizes of 1x1, 1x3 and 3x1;

[0014] Step 2.3: extracting the features of the input feature map through three layers of convolutional layers with sizes of 1x1, 1x5 and 5x1;

[0015] Step 2.4: extracting the features of the input feature map through a convolutional layer with a size of 1x1 and an average pooling layer with a size of 3x3;

[0016] Step 2.5: extracting the features of the input feature map through a convolutional layer with a size of 1x1 and a maximum pooling layer with a size of 3x3;

[0017] Step 2.6: The output feature maps of the above five steps are spliced in the channel dimension to obtain three layers of enhanced feature maps with multi-level enhanced target feature information.

[0018] Further, in step 3, each layer of enhanced feature maps obtained in step 2 is processed using an attention mechanism module, convolution operation and Softmax operation respectively, the output result is multiplied by the input enhanced feature map to obtain three layers of noise reduction feature maps, and the attention loss is calculated including:

[0019] Step 3.1: Each layer of fused enhanced feature maps obtained in step 2 is input into the attention module for channel and spatial feature weight calculation. First, average pooling and maximum pooling operations are used to sequentially infer attention feature descriptors along the channel and spatial two independent dimensions, and then the attention feature descriptors are multiplied by the input feature map for adaptive feature refinement.

[0020] Step 3.2: Each layer of fused enhanced feature maps obtained in step 2 is processed by convolution operation to generate a two-channel saliency feature map, and then Softmax operation is performed on it. One of the channels is multiplied by the output feature map of step 3.1 and the output feature map of step 2 to obtain three layers of feature noise reduction feature maps.

[0021] Step 3.3: The two-channel saliency feature map obtained in step 3.2 is pixel Softmax cross-entropy loss calculated with the binary feature map obtained from the real box as the attention loss of this process.

[0022] Further, in step 4, the three layers of noise reduction feature maps and attention loss obtained in step 3 are classified and coordinate regression using the YOLO series prediction module, and the output after non-maximum suppression processing marks the mask position and wearing method including:

[0023] Step 4.1: Each layer of noise reduction feature maps is divided into grids, and each grid predicts position information and confidence information of three prediction boxes.

[0024] Step 4.2: Set a confidence threshold to filter out prediction boxes with low scores, and perform non-maximum suppression processing on the remaining prediction boxes to obtain the final detection result.

[0025] The present application provides a mask wearing method based on an improved single-stage target detection network, which first uses a YOLO series basic network to extract and fuse features of an input image, generates three layers of feature maps, then splices three layers of enhanced feature maps after using five different convolution layers and pooling layer operations, uses an attention mechanism module, convolution operation and Softmax operation to process respectively, obtains three layers of noise reduction feature maps, and calculates attention loss, finally uses a YOLO series prediction module to classify and coordinate regress the three layers of noise reduction feature maps and the attention loss, and outputs a marked mask position and wearing method after non-maximum suppression processing. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a whole schematic diagram of the mask wearing method based on the improved single-stage target detection network provided by the embodiment of the present application;

[0027] Figure 2 is a feature enhancement module network structure schematic diagram provided by the embodiment of the present application;

[0028] Figure 3 is a feature noise reduction module network structure schematic diagram provided by the embodiment of the present application;

[0029] Figure 4 is a four kinds of mask wearing method detection category schematic diagram provided by the embodiment of the present application;

[0030] Figure 5 is a detection result schematic diagram of the mask wearing method based on the improved single-stage target detection network provided by the embodiment of the present application. DETAILED DESCRIPTION

[0031] The present application will be further described below in combination with the drawings of the specification and the embodiments.

[0032] The present application first extracts and fuses features through a YOLO series basic network, then uses a feature enhancement and feature noise reduction module to enhance target feature information and suppress useless information in the background, and improves the low accuracy of the single-stage algorithm, in order to achieve the above-mentioned target, the method scheme of the present application is as follows:

[0033] (1) using a YOLO series basic network to extract and fuse features of an input image, generating three layers of feature maps with different resolutions and different semantic strengths;

[0034] (2) The three-layer feature maps obtained in (1) are processed using five different convolutional layers and pooling layer operations, and the generated results are spliced in the channel dimension to produce three-layer enhanced feature maps with multi-level enhanced target feature information.

[0035] (3) Each layer of the enhanced feature maps obtained in (2) is processed using an attention mechanism module, a convolution operation, and a Softmax operation, respectively, the output results are multiplied with the input enhanced feature maps, three layers of noise reduction feature maps are obtained, and attention loss is calculated.

[0036] (4) The three layers of noise reduction feature maps and attention loss obtained in (3) are classified and coordinate regression using a YOLO series prediction module, and the output after non-maximum suppression processing marks the mask position and wearing method.

[0037] Processing the three-layer feature maps using the feature enhancement module enhances the features containing target information. The specific training and detection process includes:

[0038] (2-a): The input feature map is processed through a 1x1 convolutional layer for feature extraction;

[0039] (2-b): The input feature map is processed through three layers of convolutional layers with sizes of 1x1, 1x3, and 3x1 for feature extraction;

[0040] (2-c): The input feature map is processed through three layers of convolutional layers with sizes of 1x1, 1x5, and 5x1 for feature extraction;

[0041] (2-d): The input feature map is processed through a 1x1 convolutional layer and a 3x3 average pooling layer for feature extraction;

[0042] (2-e): The input feature map is processed through a 1x1 convolutional layer and a 3x3 maximum pooling layer for feature extraction;

[0043] (2-f): The output feature maps of the above five steps are spliced in the channel dimension to obtain three-layer enhanced feature maps with multi-level enhanced target feature information.

[0044] Processing the three-layer enhanced feature maps of the feature enhancement module using the feature noise reduction module suppresses the useless noise information mixed in the feature maps. The specific training and detection process includes:

[0045] (3-a): The fusion enhanced feature map of each layer obtained in step 2 is input into an attention module for channel and spatial feature weight calculation. First, average pooling and maximum pooling operations are used to sequentially infer attention feature descriptors along the channel and spatial two independent dimensions, and then the attention feature descriptors are multiplied by the input feature map to perform adaptive feature refinement.

[0046] (3-b): The fusion enhanced feature map of each layer obtained in step 2 is subjected to convolution operation to generate a two-channel salient feature map, and then a Softmax operation is performed thereon, and one of the channels is selected and multiplied by the output feature map of step 3.1 and the output feature map of step 2 to obtain a three-layer denoised feature map after feature denoising.

[0047] (3-c): The two-channel salient feature map obtained in step 3.2 is subjected to pixel Softmax cross-entropy loss calculation with the binary feature map obtained from the real frame as the attention loss of the process.

[0048] The three-layer feature map after feature denoising and the attention loss are subjected to classification and coordinate regression using a YOLO series prediction module, and after non-maximum suppression processing, the detected mask-wearing mode is output. The specific training and detection process includes:

[0049] (4-a): Each layer of denoised feature map is divided into grids, and each grid predicts position information and confidence information of three prediction boxes.

[0050] (4-b): A confidence threshold is set to filter out prediction boxes with low scores, and non-maximum suppression processing is performed on the remaining prediction boxes to obtain the final detection result.

[0051] In combination Figure 1 with specific parameters, an embodiment is given, and the specific implementation steps of the present application are as follows:

[0052] S1, the screened and supplemented four types of mask wearing mode data set is randomly divided into a training set, a validation set and a test set in a ratio of 5:3:2. Then the pictures in the training set are input into a YOLO series basic network for feature extraction and fusion to generate three-layer feature maps with different resolutions and different semantic strengths.

[0053] The detection object of the network is four types of targets, and the four types of mask wearing modes are: standard mask wearing (face_mask), mask covering the mouth and not covering the nose (mouth_chin), mask not covering the nose and the mouth (chin), and not wearing a mask (face).

[0054] The pre-processing process of the input image of the YOLO series basic network mainly includes feature extraction and feature fusion. The backbone network in the basic network extracts the target feature information beneficial to target detection in the image through multiple convolution layers of different sizes, and in this process, it can automatically learn how to effectively process and extract information from a large amount of data, providing a basis for subsequent feature map processing steps. The feature pyramid structure in the basic network generates a feature map of the same size as the next layer from a feature map of different sizes and fuses it, while combining the detailed information of the bottom layer and the semantic information of the high layer, increasing the receptive field of the bottom layer, so that the bottom layer can obtain more context information when detecting small targets.

[0055] S2, processing the three-layer feature map using a feature enhancement module to enhance the features containing target information, as shown in Figure 2 The specific implementation steps are as follows:

[0056] S21: performing feature extraction on the input feature map through a convolution layer with a size of 1x1;

[0057] Using a 1x1 convolution can reduce the accumulation of parameters, so the 1x1 convolution layer here increases the depth of the network while increasing the width of the network and reducing the model parameters, thereby improving the network performance.

[0058] S22: performing feature extraction on the input feature map through three convolution layers with sizes of 1x1, 1x3, and 3x1;

[0059] S23: performing feature extraction on the input feature map through three convolution layers with sizes of 1x1, 1x5, and 5x1;

[0060] In order to control the network computation and network parameters, first, a 1x1 convolution layer is used for dimension reduction, so that the correlation of different dimensional information is reduced during dimension reduction, and then different size separable convolution layers are used to obtain different size visual field ranges and the correlation between spatial information. Compared with the ordinary convolution layer, the parameter quantity and the computational complexity are greatly reduced under the condition of the same input and output.

[0061] S24: performing feature extraction on the input feature map through a convolution layer with a size of 1x1 and an average pooling layer with a size of 3x3;

[0062] S25: performing feature extraction on the input feature map through a convolution layer with a size of 1x1 and a maximum pooling layer with a size of 3x3;

[0063] In order to avoid the bottleneck of signal representation, that is, the loss of feature information may be caused when a large proportion of compression is performed on the spatial dimension of the feature, the convolution operation in S21 and the pooling operation in S24 and S25 can be used to reduce the sampling, and then the results are connected in parallel, so that the dimension is increased while the sampling is reduced, and the calculation amount is ensured while the loss of feature is avoided. For the pooling operation, the maximum pooling operation and the average pooling operation are selected for parallel processing, which can compensate for each other and capture the clues of different object features, so as to enhance the feature information.

[0064] S26: The output feature maps of the above five steps are spliced in the channel dimension to obtain three layers of enhanced feature maps with multi-level enhanced target feature information.

[0065] S3, using the feature denoising module to process the three layers of enhanced feature maps of the feature enhancement module so that the useless noise information mixed in the feature map is suppressed, as shown in Figure 3 The specific training and detection process includes:

[0066] S31: The three layers of enhanced feature maps after feature enhancement are used to calculate the channel and spatial feature weights in turn by using the attention module. First, the average pooling operation and the maximum pooling operation are used in parallel to compress the spatial dimension of the input feature map, and then the results of the two pooling operations are input into the shared network to extract features, and then the activation function is used and multiplied with the input feature map to obtain the channel attention feature map.

[0067] The attention mechanism aims to enhance the representation ability of the network by focusing on important feature information and ignoring unnecessary feature information. The channel attention module generates a channel attention feature map by using the channel relationship between features, and effectively calculates the channel attention by compressing the spatial dimension of the input feature map. The main concern is "what" is useful in the input feature map. The expression of the channel attention module is as follows:

[0068]

[0069] Where F represents the input feature map; M c (F) represents the output feature map of the channel attention module; MLP() represents the shared network; AvgPool(), MaxPool() represent the average pooling and maximum pooling operations, respectively; represent the feature maps after the average pooling and maximum pooling operations, respectively; W0 and W1 represent the weights of the shared network; and sigma() represents the activation function.

[0070] S32: The obtained channel attention feature map is compressed in the channel dimension using a maximum pooling operation and an average pooling operation, respectively, and the results of the two pooling operations are connected into a feature map with a channel number of 2, which is input into a convolution layer for feature aggregation. Finally, the spatial attention feature map is obtained by multiplying the result by an activation function and the channel attention feature map.

[0071] The spatial attention module differs from the channel attention module in that it focuses on "where" the information is, and is a supplement to the channel attention module. Its expression is as follows:

[0072]

[0073] where F represents the input feature map; AvgPool() and MaxPool() represent average pooling and maximum pooling operations, respectively; f 7×7 represents a convolution layer with a size of 7x7; represent the feature maps after the average pooling and maximum pooling operations, respectively; and σ() represents an activation function.

[0074] S33: Each layer of the fusion enhanced feature map obtained in S2 is subjected to convolution operation to generate a two-channel saliency feature map, which is then subjected to Softmax operation, and one channel is selected to multiply the output feature map of S32 and the output feature map of S2 to obtain three layers of denoised feature maps after feature denoising.

[0075] After the convolution layer, each layer of the fusion enhanced feature map generates a saliency feature map with a channel number of 2, which represents the scores of the foreground and background, respectively. Subsequently, the saliency feature map is subjected to Softmax processing to enhance the target feature and suppress the background noise. The reason is that the Softmax operation can map the input to the interval [0, 1], that is, the target information can be enhanced after the processing, and on the contrary, the noise can be suppressed or even close to 0. However, since the saliency map is continuous in this process, the Softmax processing does not completely eliminate the non-target information, which is beneficial to preserve some context information and improve the robustness of the network.

[0076] S34: The two-channel saliency feature map obtained in step 3.2 is subjected to pixel Softmax cross-entropy loss calculation with the binary feature map obtained from the real box, which is used as the attention loss of this process.

[0077] The attention loss can be trained together with the classification loss, target confidence loss and positioning loss in the forward propagation process and the backward propagation process, and the loss function can reflect the difference between the network and the actual data during the training process, which is beneficial to the network to better optimize the subsequent training to reduce the difference.

[0078] S4, the three-layer denoised feature map after feature denoising and attention loss are classified and coordinate regression using a YOLO series prediction module, and the detected mask-wearing mode is output after non-maximum suppression processing. The specific training and detection process includes:

[0079] S41: divide each layer of denoised feature map into a grid, and predict the position information and confidence information of three prediction boxes for each grid.

[0080] S42: set a confidence threshold, filter out prediction boxes with low scores, and perform non-maximum suppression processing on the remaining prediction boxes to obtain the final detection result.

[0081] Figure 4 The schematic diagram of four mask wearing modes is shown in FIG. 1, which are: standard mask wearing (face_mask), mask covering mouth but not covering nose (mouth_chin), mask not covering nose and mouth (chin), and not wearing mask (face). As shown in FIG. 2, the result of network detection of four mask wearing modes is shown. From the figure, it can be observed that when the target appears in the image, the network can accurately identify the category to which the target belongs. When the mask is not worn in a standard manner, the network's identification result is accurate. Figure 5

[0082] Table 1 lists the detection results of the four mask wearing modes in this embodiment. The image data set is based on the public data sets MAFA dataset, WIDERFACE dataset and MaskedFace-net dataset, and suitable scene data is selected and further expanded. Subsequently, the LabelImg tool is used to identify and make xml type files storing target positions and categories, and all data types are converted into PASCAL VOC data set format, and then into the format required by the YOLO algorithm. The data set is divided into training set, validation set and test set in the ratio of 5:3:2 for model training, validation and testing. From the results in the table, it can be seen that the mask wearing mode detection method based on the improved single-stage target detection network used in this example has obvious improvement in detection accuracy, and the detection effect for similar categories that are difficult to distinguish is good.

[0083] Table 1 Comparison table of detection results of four mask wearing modes in this embodiment

[0084]

[0085] ​In summary, the mask wearing method detection method based on the improved single-stage target detection network effectively improves the accuracy and detection effect of natural images, and has strong practicality. The mask wearing method detection method based on the improved single-stage target detection network can effectively enhance the target feature information and reduce the useless noise information in the background, so as to more accurately detect the small target difficult to classify in the complex background. And this method is not limited to the above four categories, and is also applicable to other categories in natural images after changing the data set and adjusting the parameters.

[0086] The basic principle, main features and practical advantages of the mask wearing method detection method based on the improved single-stage target detection network are shown and described above. Those skilled in the art should understand that the above description of the embodiments is only used to help understand the method technology and core idea of the present application, and does not limit the present application. Meanwhile, according to the idea of the present application, there will be changes in specific implementation and application range, and these changes all fall within the protection scope of the present application.

Claims

1. A method for detecting mask wearing method, characterized in that, Includes the following steps: Step 1: Use the YOLO series of basic networks to extract and fuse features from the input image to generate three-layer feature maps with different resolutions and semantic intensities; Step 2: Process the three-layer feature map obtained in Step 1 using five different convolutional and pooling layer operations, and then concatenate the generated results along the channel dimension to produce a three-layer enhanced feature map with multi-level enhanced target feature information. Step 2.1: Extract features from the input feature map using a 1×1 convolutional layer; Step 2.2: Extract features from the input feature map using three convolutional layers of size 1×1, 1×3, and 3×1; Step 2.3: Extract features from the input feature map using three convolutional layers with sizes of 1×1, 1×5, and 5×1; Step 2.4: Extract features from the input feature map using a 1×1 convolutional layer and a 3×3 average pooling layer; Step 2.5: Extract features from the input feature map using a 1×1 convolutional layer and a 3×3 max pooling layer; Step 2.6: Concatenate the output feature maps from the above five steps along the channel dimension to obtain a three-layer enhanced feature map with multi-level enhanced target feature information; Step 3: For each layer of enhanced feature map obtained in Step 2, use the attention mechanism module, convolution operation and softmax operation to process them together. Multiply the output result with the input enhanced feature map to obtain three layers of denoised feature map, and calculate the attention loss. Step 4: Use the YOLO series prediction module to classify and regress the three-layer noise reduction feature map and attention loss obtained in Step 3, and use the output after non-maximum suppression to mark the mask position and wearing method.

2. The method for detecting mask wearing method according to claim 1, characterized in that: Step 3 describes processing each enhanced feature map obtained in Step 2 using an attention mechanism module, convolution operations, and softmax operations. The output is then multiplied by the input enhanced feature map to obtain a three-layer denoised feature map. The attention loss is then calculated, including: Step 3.1: Input each layer of fused and enhanced feature map obtained in Step 2 into the attention module to calculate the channel and spatial feature weights. First, use average pooling and max pooling operations to infer the attention feature descriptor along the two independent dimensions of channel and space. Then, multiply the attention feature descriptor by the input feature map to perform adaptive feature refinement. Step 3.2: Generate a two-channel salient feature map by performing convolution operation on each layer of the fused and enhanced feature map obtained in Step 2. Then, perform a Softmax operation on it and select one channel to multiply with the output feature map of Step 3.1 and the output feature map of Step 2 to obtain a three-layer denoised feature map after feature denoising. Step 3.3: Calculate the pixel Softmax cross-entropy loss by combining the two-channel salient feature map obtained in Step 3.2 with the binary feature map obtained from the ground truth bounding box, and use it as the attention loss.

3. The method for detecting mask wearing method according to claim 1, characterized in that: Step 4 describes using the YOLO series prediction module to perform classification and coordinate regression on the three-layer denoised feature map and attention loss obtained in Step 3. The output after non-maximum suppression processing marks the mask position and wearing method, including: Step 4.1: Divide the denoised feature map of each layer into a grid, and predict the location information and confidence information of three prediction boxes in each grid; Step 4.2: Set the confidence threshold, filter out the prediction boxes with low scores, and perform non-maximum suppression on the remaining prediction boxes to obtain the final detection results.

Images