Method for automatically monitoring face mask wearing in complex scene based on improved YOLOv5s

Abstract

The invention relates to a face mask wearing automatic monitoring method suitable for a complex scene, and belongs to the technical field of artificial intelligence. Comprising functions of detecting whether a pedestrian wears a mask and automatically alarming after detecting that the pedestrian does not wear the mask. And the content of the detection part is realized by using a crawler and a camera to obtain video streams to collect picture data, clean the picture data, label a data set, train the data and improve a model. The model is improved as follows. Replacing a Focus module in the trunk with a convolution with a convolution kernel size of 6 * 6, and replacing a C3 module with a Ghostv2-C3 module provided by the invention; an original PAN feature fusion network is replaced by combining a BiFPN feature fusion structure, and standard convolution is replaced by GSConv; and finally, replacing the bounding box CIoU loss function with a Wise-IoU loss function. Therefore, the problems of false detection and missing detection of an existing model under a complex background are solved, and the memory occupied by hardware is reduced. The method reduces the loss of manpower and material resources, and promotes the high-quality development of enterprises.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to the monitoring of face mask wearing in some complex places such as pharmaceutical factories, food factories, and hospitals. Background Art

[0002] Respiratory infectious disease viruses, toxic and harmful gases, dust and other harmful substances to the human body can enter the body through the human respiratory circulatory system, thereby causing damage to the human body, and even endangering life in severe cases. Standardized wearing of masks can effectively prevent viruses, dust, toxic and harmful gases from entering the lungs, nerves and other vital tissues, and reduce the risk of people being infected, serving as a source of transmission, and poisoning injuries. In response to the special requirements of epidemic prevention and control and some positions, some public places require people in crowded places and some special workplaces to wear masks. For the supervision of wearing masks in crowded places and other workplaces, most places still use traditional methods such as manual inspections, which wastes human resources and is inefficient. Therefore, artificial intelligence methods can be used for supervision to improve work efficiency.

[0003] In summary, the problems existing in the prior art are:

[0004] 1. Monitoring with traditional manual means is not only inefficient, but also wastes a lot of manpower and material resources.

[0005] 2. Through the means of artificial intelligence, based on current research, the face mask wearing detection method based on target detection still has problems of false detection and missed detection, and the hardware occupies a large amount of memory.

[0006] 3. Due to the limited open source data on face mask datasets, the relevant datasets are still insufficient to realize face mask wearing detection in complex scenarios, resulting in insufficient category, environment, and scale information, which in turn easily leads to problems such as poor generalization ability of the model, information redundancy, and low precision. Summary of the invention

[0007] The first purpose of the present invention is to provide a highly efficient method for automatically monitoring whether pedestrians are wearing masks in complex scenes, which can automatically monitor whether pedestrians are wearing masks, replace traditional manual monitoring, improve work efficiency, and reduce the problems of false detection and missed detection in manual supervision. A method for automatically monitoring whether pedestrians are wearing masks in complex scenes based on improved YOLOv5s, the method comprising:

[0008] 1. Crawling and acquiring datasets, collecting video stream data of complex scenes, and combining and producing datasets using AIZOO public datasets;

[0009] 2. Use data enhancement to increase the complexity of the data set;

[0010] 3. Input the enhanced image into the improved YOLOv5s model for face mask detection training, obtain the trained model weights, and then test the trained model. Repeat the above process and select the model with the best test results;

[0011] 4. Load the trained model for detection, locate and classify the image to be detected according to the detection results, so as to indicate that the target is detected;

[0012] 5. The synthesis of alarm sounds and the combination with the previous detection will broadcast when the system detects that a pedestrian is not wearing a mask: "Attention, please wear a mask" and the detection platform will display a prompt message that the pedestrian is not wearing a mask, thereby achieving the function of automatic monitoring.

[0013] The process of training the detection model in the above method 3 includes:

[0014] (1) Acquisition and preparation of data sets;

[0015] (2) enhancing the data in the multi-label multi-classification dataset to obtain a training dataset and a test dataset for face mask wearing;

[0016] (3) Before model training, data enhancement is performed on the data, such as mosaic, copy-paste, mixup, etc., to enhance the robustness of the model in complex backgrounds;

[0017] (4) Inputting the training data set for multi-target detection into the backbone network and the target classification and positioning branches to obtain the classification and positioning results of face mask wearing;

[0018] (5) Input the data in the face mask test dataset into the trained model, conduct testing, obtain the test results, and finally verify the effectiveness of the improved model.

[0019] The method of introducing data enhancement described in the above process (3), wherein the mosaic data enhancement method is characterized in that first, the coordinates of the reference points (xc, yc) are spliced ​​for the randomly selected pictures, and four pictures are randomly selected. Secondly, the four pictures are resized and scaled according to the reference points, and then placed at the upper left, upper right, lower left, and lower right positions of the specified size picture. Then, according to the size transformation method of each picture, the mapping relationship is mapped to the picture label. Finally, according to the specified horizontal and vertical coordinates, the pictures are spliced. The detection box coordinates that exceed the boundary are processed. This method effectively enhances the robustness of the model and improves the detection of small targets. The CopyPaste and Mixup technologies can be used to increase the number of samples while maintaining the authenticity of the data. The above data enhancement methods are used to enrich the data. Then, an adaptive image scaling operation is performed to obtain a 640×640 standard image for training, which is mainly used to solve the problem that the reference of the category with too few samples during model training is small, which may make the model pay more attention to the category with more samples. The model parameters are also mainly adjusted according to the loss of the category with more samples, which ultimately leads to a decrease in the accuracy of the category with fewer samples. In addition, when the anchor point and the target size are quite different, the K-means algorithm is used to cluster the bounding boxes of the training set. The k clusters together form a set in which the distance from the sample point of each cluster to its centroid is the smallest. Therefore, the following formula (1) can be obtained. i is the value of each sample point, z j It is the result of averaging all sample points in the cluster.

[0020]

[0021] Among them, the K-means anchor box clustering process is as follows:

[0022] a. First, choose to divide the dataset into k clusters, and then randomly select k data points as the initial cluster centers.

[0023] b. Secondly, assign each data point to the cluster center closest to it. Each data point can only belong to one cluster.

[0024] c. Next, the cluster centers are updated based on the assigned data points. This is done by calculating the average of the data points belonging to each cluster.

[0025] d. After that, repeat steps b and c until the cluster center point no longer changes or the predetermined number of iterations is reached.

[0026] e. Finally, we get k clusters and the center point of each cluster.

[0027] Through this method, the automatically generated prior anchor points are more applicable and efficient, and more effective prior knowledge can be learned, which accelerates the convergence of the model.

[0028] The specific steps in the image input network of the above process (4) are as follows:

[0029] ① Input each frame of image into the feature extraction network, and obtain three feature maps of different sizes after downsampling by 1 / 8, 1 / 16 and 1 / 32;

[0030] ② Then, the image is first fused from bottom to top through feature fusion, and then the target is fused at multiple scales through multi-scale feature fusion from top to bottom;

[0031] ③ The fused feature maps are respectively input into the classification branch of the predicted large and medium-sized targets and the detection embedding feature extraction branch to obtain the prediction information of large, medium and small targets, including the category and position of the targets.

[0032] In step ③, the detector is operated through the convolution layer to obtain the prediction result, and finally the NMS method is used to remove the redundant prediction boxes and detect our target. The loss functions used in the model training process include the BCELoss (Binary CrossEntropy Loss) loss function and the CIoU Loss loss function. Among them, the BCE Loss loss function, as the loss function for calculating the target and classification, can effectively reduce the time cost of model training. The CIoU Loss loss function, as the loss function for bounding box regression, accelerates model convergence. The BCE Loss and CIoU Loss formulas are shown in formulas (2) and (3). Among them, p in formula (2) represents the predicted value, t represents the true value, and b and b in formula (3) are gt They represent the center points of the predicted box and the real box respectively, ρ represents the Euclidean distance between the two center points, and c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the real box. αv is an influencing factor.

[0033] Loss(p,t)=-t*log(p)-(1-t)log(1-p) (2)

[0034]

[0035] The synthesis process of the alarm sound in the above method 5 includes: pyttsx3 realizes text-to-speech conversion and saves the voice in mp3 format.

[0036] The second purpose of the present invention is to design a face mask wearing detector with low missed detection rate and false detection rate and fast detection speed, which mainly improves the YOLOv5s detection model. The main improvement steps for improving the YOLOv5s model algorithm are as follows:

[0037] 1. In order to design a network structure with better detection effect and less hardware resource occupation, in order to improve the detection effect, this paper first replaces the Focus module in the backbone with a standard convolution with a convolution kernel size of 6*6 to increase the receptive field of the model, and replaces the C3 module in the backbone with the Gh-ostv2C3 module proposed in this paper, so as to reduce the number of parameters while strengthening the extraction of network features;

[0038] 2. Secondly, the BiFPN feature fusion structure is used to replace the original PAN feature fusion network, which improves the model's detection of large, medium and small targets without significantly increasing the number of parameters. And the standard convolution is replaced by GSConv, which can reduce the complexity of the model while maintaining a good detection effect;

[0039] 3. Finally, the bounding box CIoU loss function is replaced by the Wise-IoU loss function, which reduces the competitiveness of high-quality anchor boxes while also reducing the harmful gradients generated by low-quality examples. This allows WIoU to focus on anchor boxes of average quality and improve the overall performance of the detector. In addition, an adaptive image sampling strategy is used to alleviate the problem of data imbalance. At the same time, mosaic data enhancement is used to make full use of the target information in the feature map and improve the robustness of the model in complex backgrounds.

[0040] Among them, Ghostv2-C3 in the above improvement step 1 is a module proposed in this paper, and its structure is as follows Figure 3 As shown in the figure. This module replaces Bottleneck in GhostNetv2 with Bottleneck in C3, in order to improve the model's feature extraction capability while reducing the number of parameters. Bottleneck in GhostNetv2 sends the input feature map X to two branches, one is the original Ghost Module, generating feature map Y, and the other is the DFC Module generating Attention map A. Then, the output A and Y are multiplied, and the result after the product operation is input to GhostModule, and then the result is added to the original input feature map to obtain the final result.

[0041] The Ghost Module operation process is divided into three steps. Since the amount of calculation for ordinary convolution operations is related to the dimensions of the input and output feature maps, and many features in the output features are similar "phantom" features, there is no need to use a lot of calculations and parameters to generate these redundant feature maps. Therefore, m valid information X' in the input X is selected, and first a conventional convolution operation is performed on some features of the input X' (there are n feature maps), and the formula is shown in Formula 4:

[0042] Y'=X*f' (4)

[0043] Where f'∈R c*k*k*m is the convolution kernel used, m≤n. At this time, we obtain m feature maps after the convolution operation. In order to complete the n feature maps, a cheap linear transformation is performed on each feature in Y' to obtain s "phantom" feature maps. The formula is shown in Formula 5.

[0044]

[0045] In the formula, y' i Represents the i-th original feature map, which undergoes linear transformation Φ i,j , get the jth "phantom" feature map y ij , that is, each original feature map y' i Can generate s "phantom" feature maps The last Φ i,s is an identity mapping used to preserve the original feature map.

[0046] By using cheap operations, we can obtain n = m * s feature maps Y = [y 11 ,y 12 ,...,y ms ] as the output data of the Ghost Module module. DFC fully decouples the fully connected attention mechanism, which is aimed at the original Self-attention. It can simulate long-range correlation well, but it does not meet the requirements of high-efficiency deployment. Therefore, a simpler and easier to implement fully connected layer (FC) with fixed weights is used to generate an attention map with a global receptive field. Among them, given the input feature Z∈R H*W*C , the formula for implementing the attention map using the FC layer is shown in Formula 6:

[0047] a hw =∑F hw,h',w' ⊙z h',w' (6)

[0048] Among them, ⊙ represents element-wise multiplication, and DFC decomposes the above into two FC layers and aggregates features in the horizontal and vertical directions respectively. The formulas are shown in the following formulas 7 and 8:

[0049]

[0050]

[0051] Among them, F H and F W is the weight.

[0052] The improvement step 2 mentioned above is mainly to solve the problem of obtaining more information of features of three different scales from the backbone feature extraction network for fusion. When the features are processed by multiple layers of networks, some key information may be lost. To solve this problem, we use the BiFPN feature fusion method to introduce shallow information into the deep layer to effectively solve the problem of some deep network information loss; secondly, since the feed image in CNN must almost undergo a similar conversion process in Backbone: spatial information is gradually transmitted to the channel. And each spatial compression and channel expansion of the feature map will lead to partial loss of semantic information. Dense convolution calculation maximizes the retention of hidden connections between each channel, while sparse convolution completely cuts off these connections. GSConv will retain these connections as much as possible. However, if it is used in all stages of the model, the network layer of the model will be deeper, and the deep network will increase the resistance to data flow and significantly increase the inference time. When these feature maps reach the Neck, they have become slender (the channel dimension reaches the maximum and the width and height dimensions reach the minimum), and no transformation is required. Therefore, a better choice is to use GSConv only in the Neck. At this stage, it is just right to use GSConv to process Concatenated feature maps, and its redundant and repeated information will be reduced. Therefore, in this paper, the standard convolution is replaced by GSConv in feature fusion to reduce the amount of calculation and improve the inference speed of the model.

[0053] The loss function is used to calculate the gap between the predicted value of the model output and the actual value of the input, which is crucial in the model training and optimization process. Target detection predicts the target in the positioning image through Bounding box regression. Early target detection used IoU as the loss function. When the predicted box does not overlap with the real box, the loss will have a gradient vanishing problem, resulting in slow convergence and inaccurate detectors. With the introduction of CIoU, CIoU not only introduces the problem of gradient vanishing in the penalty term link in the IoU loss, but also considers the distance and aspect ratio between the center points of the predicted box and the real box to obtain a better Bounding box regression. The CIoU loss function is selected in YOLOv5s, as shown in the following formula 9:

[0054]

[0055] Among them, ρ(b,b gt ) represents the Euclidean distance between the center point of the predicted box and the real box, c represents the diagonal length of the minimum enclosing rectangle of the predicted box and the real box, represents the weight coefficient, It measures the consistency of the length ratio of the predicted box and the real box.

[0056] However, CIoU can speed up the regression of the prediction box to a certain extent, but the problem is that during the prediction box regression process, once the aspect ratio of the prediction box to the true box is in linear proportion, the prediction box w and h cannot be increased or decreased at the same time, and the regression optimization cannot be continued. And it is noted that the target detection training set contains low-quality examples. If the bounding box is blindly strengthened for the regression of low-quality examples, it will obviously harm the improvement of the model detection performance. Focal-EIoU was proposed to solve this problem, but because its focusing mechanism is static, it does not fully explore the potential of the non-monotonic focusing mechanism. Based on this point of view, this paper chooses the Wise-IoU loss function with a dynamic non-monotonic focusing mechanism. The dynamic non-monotonic focusing mechanism uses "outliers" instead of IoU to evaluate the quality of the anchor box and provides a wise gradient gain allocation strategy. This strategy reduces the competitiveness of high-quality anchor boxes while reducing the harmful gradients generated by low-quality examples. This allows Wise-IoU to focus on anchor boxes of ordinary quality and improve the overall performance of the detector. The formula of its Wise-IoU loss function is shown in the following formulas 10, 11, and 12:

[0057] L WIoUv1 =R WIoU L IoU (10)

[0058]

[0059]

[0060] Among them, R WIoU ∈[1,e), which will significantly enlarge the L of the normal quality anchor box IoU , L IoU ∈[0,1], which will significantly reduce the R of the high-quality anchor box WIoU , and significantly reduce its attention to the distance of the center point when the anchor box and the target box overlap well. WIoU Produce gradients that hinder convergence, W g ,H g Detach from the computation graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of face mask wearing monitoring in complex scenes based on improved YOLOv5s in the present invention;

[0062] Figure 2 This is a structural diagram of the GhostNetV2-Bottleneck module used in the present invention;

[0063] Figure 3 This is the structure diagram of the Ghostv2-C3 module proposed in the present invention;

[0064] Figure 4 Schematic diagram of the improved YOLOv5s network structure proposed in the present invention. DETAILED DESCRIPTION

[0065] Specific implementation 1:

[0066] Among them, implement 1 structure diagram, such as Figure 1 An automatic monitoring method for face mask wearing in complex scenes based on improved YOLOv5s, the method comprising:

[0067] 1. Acquisition of data sets and data collection and production of complex scenes. This paper combines the AIZOO data set to produce a total of 12,000 data sets;

[0068] 2. Use data enhancement to expand the data set;

[0069] 3. Input the enhanced image into the improved YOLOv5s model for face mask detection training, obtain the trained model weights, and then test the trained model. Repeat the above process and select the model with the best test results;

[0070] 4. Load the trained model for detection, locate and classify the image to be detected according to the detection results, so as to indicate that the target is detected;

[0071] 5. The synthesis of alarm sounds and the combination with the previous detection. When the system detects that a pedestrian is not wearing a mask, it will broadcast a reminder sound: "Attention, please wear a mask." The detection platform will display a reminder message that the pedestrian is not wearing a mask, thereby achieving the function of automatic monitoring.

[0072] In the above method 1, we use the Requset library, which can use the http1.1 protocol to quickly request and access the corresponding URL in memory, and quickly parse the corresponding return result after completing the URL request. The crawling process of image data is as follows:

[0073] (1) Call Baidu image search portal through network API;

[0074] (2) Use the search entry results to quickly obtain keyword-related image links;

[0075] (3) Download the corresponding pictures in batches locally according to the picture links;

[0076] (4) Determine whether the system has created a directory for storing images locally. If not, the system creates one.

[0077] (5) Convert the image file to a local file image, name and mark the image, and save it.

[0078] In the above method 1, the process of making the data set is as follows:

[0079] ① Make sure the Python environment has been installed;

[0080] ② Install labelimg using pip install labelimg command;

[0081] ③Enter labelimg in the command line to start labelimg;

[0082] ④ Store the pictures that need to be annotated in a fixed folder;

[0083] ⑤ Enter the command window and go to Labelimg, click Open Dir to open the folder;

[0084] ⑥Click the button under Save on the left until the label changes to YOLO;

[0085] ⑦Double-click to open the picture on the lower right side and start annotating a picture;

[0086] ⑧Click Create RectBox, drag on the image to draw a rectangular box, click CreateRectBox in the pop-up dialog box, drag on the image to draw a rectangular box, enter the label name in the pop-up dialog box and click OK;

[0087] ⑨ Use the same steps to label the target object. The used labels can be directly selected next time without re-entering. The marked boxes can be seen on the right and can be dragged and adjusted in the image;

[0088] ⑩ Click Save on the right to save the annotated data to the same folder. Two new txt files will appear locally; then click Next Image to annotate the next image.

[0089] In the above method 3, the implementation process of this technical solution is as follows:

[0090] 1) Input each frame of image into the feature extraction network, and obtain three feature maps of different sizes after downsampling by 1 / 8, 1 / 16 and 1 / 32;

[0091] 2) Then, the image is first fused from bottom to top through feature fusion, and then the target is fused at multiple scales through multi-scale feature fusion from top to bottom;

[0092] 3) The fused feature maps are respectively input into the classification branch and the detection / embedding feature extraction branch of the predicted large and medium-sized targets to obtain the prediction information of large, medium and small targets, including the category and position of the targets.

[0093] In step 3), the detector is operated through the convolution layer to obtain the prediction result. Finally, the NMS method is used to remove redundant prediction boxes and detect our target. The loss functions used in the model training process include BCELoss loss function and CIoU Loss loss function. Among them, BCE Loss function is used as the loss function for calculating the target and classification, and CIoU Loss function is used as the loss function for bounding box regression.

[0094] In the above method 4, the process of obtaining the RTSP video stream is as follows:

[0095] 1) Get the IP address of the camera, the configured username and password, and write out the RTSP protocol address code in the format;

[0096] 2) Use VLC tool to connect and test whether the video stream can be retrieved;

[0097] 3) Read the video stream through OpenCV in Python;

[0098] The synthesis steps of the prompt tone in the above method 5 are as follows:

[0099] (1) Use the Pyttsx library. Pyttsx is a cross-platform third-party library that converts text into speech. It wraps the built-in speech engine of the operating system.

[0100] (2) Then use gTTS, which can convert text into speech and save it in MP3 format, and then read the speech to achieve pronunciation.

[0101] Specific implementation 2: Improve the YOLOv5s model algorithm. The main improvement steps are as follows. Its structure diagram is shown in Figure 2 .

[0102] ① This paper first replaces the Focus module in the backbone with a standard convolution with a convolution kernel size of 6*6 to increase the receptive field of the model, and replaces the C3 module in the backbone with the Ghostv2-C3 module proposed in this paper to reduce the number of parameters while strengthening the extraction of network features;

[0103] ② Secondly, the BiFPN feature fusion structure is used to replace the original PAN feature fusion network, which improves the model's detection of large, medium and small targets without significantly increasing the number of parameters. And replacing the standard convolution with GSConv can reduce the complexity of the model while maintaining a good detection effect.

[0104] ③Finally, the bounding box CIoU loss function is replaced by the Wise-IoU loss function, which reduces the competitiveness of high-quality anchor boxes while also reducing the harmful gradients generated by low-quality examples. This allows Wise-IoU to focus on anchor boxes of ordinary quality and improve the overall performance of the detector. In addition, an adaptive image sampling strategy is used to alleviate the problem of data imbalance. At the same time, mosaic data enhancement is used to make full use of the target information in the feature map and improve the robustness of the model in complex backgrounds.

Claims

1. An automatic monitoring method for face mask wearing in complex scenes based on improved YOLOv5s, characterized in that: include: (1) A method for automatically monitoring whether pedestrians are wearing masks in complex scenes is provided. This method uses deep learning methods instead of traditional manual supervision methods. This method is easy to manage. The system can automatically monitor whether pedestrians are wearing masks. When pedestrians who are not wearing masks are detected, the voice alarm function can be completed. No manual supervision is required, which greatly reduces manpower and material resources, improves work efficiency, and promotes high-quality development of enterprises. (2) The present invention improves the YOLOv5s detector. The improvements are as follows: ① This paper first replaces the Focus module in the YOLOv5s backbone with a standard convolution with a convolution kernel size of 6*6 to increase the receptive field of the model, and replaces the C3 module in the backbone with the Ghostv2-C3 module proposed in this paper to achieve the goal of reducing the number of parameters while strengthening the extraction of network features; ② Secondly, the BiFPN feature fusion structure is used to replace the original PAN feature fusion network, which improves the model's detection of large, medium and small targets without significantly increasing the number of parameters. And replacing the standard convolution with GSConv can reduce the complexity of the model while maintaining a good detection effect; ③Finally, the bounding box CIoU loss function is replaced by the Wise-IoU loss function, which reduces the competitiveness of high-quality anchor boxes while also reducing the harmful gradients generated by low-quality examples. Because Wise-IoU can focus on anchor boxes of ordinary quality and improve the overall performance of the detector. In addition, an adaptive image sampling strategy is used to alleviate the problem of data imbalance. At the same time, mosaic data enhancement is used to make full use of the target information in the feature map and improve the robustness of the model in complex backgrounds.

Images