A smoking behavior detection model construction method based on a double backbone network

Abstract

The application discloses a smoking behavior detection model construction method based on a double-main network. The model is composed of five parts: a ROI main network, a space constraint network, a cigarette main network, a cigarette neck network and a cigarette detection head. Through migration learning, the ROI main network is used to obtain smoking posture features of various scales; the space constraint network is used to face the smoking posture feature maps of various scales and extract smoking posture masks; the smoking posture masks of different scales are used to perform spatial filtering on the cigarette features of various scales obtained by the cigarette main network; and the filtered cigarette features are output to the neck network and the detection head to output the coordinates and confidence of the cigarette target. By adopting the method, posture detection, image segmentation and target detection are fused; through the spatial semantic constraint of the context, the false detection rate of similar cigarette objects in the image is effectively reduced, and the detection accuracy is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of deep learning target detection, and particularly relates to a smoking behavior detection model construction method based on a double-main-network. TECHNICAL BACKGROUND

[0002] The cigarette target detection belongs to small target detection, and there are a large number of objects similar to cigarettes in a complex scene, so that a large number of false detection and missed detection situations occur when the cigarettes are directly detected. The traditional segmented detection method of "first face and then cigarette" can effectively improve the detection accuracy, however, the segmented detection needs to call the YOLO model twice, and the calculation time is long, and the real-time performance is poor. SUMMARY

[0003] The application aims at the above problems, and provides a smoking behavior detection model construction method based on a double-main-network.

[0004] The application effectively utilizes the machine learning and deep learning methods, optimizes the main network structure of the existing YOLO model, and makes the cigarette main network consider the context relationship between the smoking posture and the cigarette target before outputting the features, so that the accuracy of the smoking detection is enhanced.

[0005] The technical scheme is as follows:

[0006] Step 1) taking YOLOv5 as the model, using the smoking posture data set, and obtaining the parameters of the ROI main network through transfer learning.

[0007] Step 2) taking YOLOv5 as the model, using the cigarette data set, and obtaining the parameters of the cigarette detection model including the cigarette main network, the cigarette neck network and the cigarette detection head through transfer learning.

[0008] Step 3) taking the full convolution network FCN as the space constraint network, connecting the ROI main network and the cigarette main network in parallel, and constructing the double-main-network smoking behavior detection model.

[0009] Step 4) using the constructed double-main-network smoking behavior detection model to detect the smoking behavior of the to-be-detected image.

[0010] Furthermore, in step 1), images of people smoking are collected, labeled, and a "smoking posture dataset" is created; the labeled boxes must contain both a face and a cigarette; then, a YOLOv5 model is trained using the "smoking posture dataset"; the backbone network CSPDarknet in the trained smoking posture YOLOv5 model is saved as the ROI backbone network.

[0011] Furthermore, in step 2), cigarette images are collected, labeled, and a "cigarette dataset" is created; then, the YOLOv5 model is trained using the "cigarette dataset"; the trained cigarette detection YOLOv5 model includes three parts: the cigarette backbone network, the cigarette neck network, and the cigarette detection head.

[0012] Furthermore, in step 3), an FCN spatial constraint network is constructed using AlexNet; the FCN spatial constraint network is then used to connect the ROI backbone network with the cigarette backbone network, the cigarette neck network, and the cigarette detection head to construct a dual-backbone network smoking behavior detection model. The specific steps are as follows:

[0013] Step 3-1: Using AlexNet as the encoder, construct the FCN spatial constraint network: keep the five convolutional layers and three pooling layers of AlexNet unchanged, remove the three fully connected layers and place equivalent convolutional layers; use deconvolution to upsample the feature map output by the last convolutional layer of the encoder so that the feature map is restored to the resolution of the input feature map;

[0014] Step 3-2: Using the three scale output feature maps of the ROI backbone network as input, generate smoking pose masks of different scales through the FCN spatial constraint network; multiply the generated pose masks with the cigarette feature maps of the same scale from the cigarette backbone network to obtain optimized cigarette features. Spatial filtering is then used to reduce the probability of false cigarette detection. The specific calculation formula is as follows:

[0015]

[0016] In the formula This represents the i-th output feature of the ROI backbone network. This represents the i-th output feature of the cigarette backbone network. Mask is the i-th output feature of the optimized cigarette backbone network after spatial filtering; i For the reason The resulting smoking posture mask;

[0017] Step 3-3: Optimize the spatially filtered cigarette features It connects with the cigarette neck network and the cigarette detection head to form a dual-trunk smoking detection network model.

[0018] Further, in the step 4), according to the structure and operation defined in the step 1), the step 2) and the step 3), the double-main-stem smoking behavior detection model is used to detect smoking in the image to be detected.

[0019] Beneficial effects: by using the method, the ROI main-stem network and the FCN spatial constraint network are used to generate a smoking posture mask; spatial filtering is performed in the feature extraction process of the cigarette main-stem network to reduce false detection of similar cigarette objects; meanwhile, the smoking posture detection, image segmentation and cigarette detection are integrated to design an end-to-end smoking detection model to enhance real-time performance. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 FIG. 1 is a step schematic diagram of a double-main-stem network-based smoking behavior detection model construction method according to the present application;

[0021] Figure 2 FIG. 5 is a migration learning data set labeling method schematic diagram according to the present application;

[0022] Figure 3 FIG. 6 is a smoking posture detection ROI main-stem network structure schematic diagram according to the present application;

[0023] Figure 4 FIG. 7 is an AlexNet-FCN spatial constraint network structure schematic diagram according to the present application;

[0024] Figure 5 FIG. 8 is a double-main-stem network-based smoking behavior detection network structure schematic diagram according to the present application;

[0025] Figure 6 FIG. 9 is a comparison diagram of detection accuracy of a double-main-stem network-based smoking behavior detection network and YOLOv5 on the same data set according to the present application;

[0026] Figure 7 FIG. 10 is a comparison diagram of detection results of a double-main-stem network-based smoking behavior detection network and YOLOv5 according to the present application. DETAILED EMBODIMENT

[0027] The technical method of the present application will be further described in detail below in combination with the drawings of the specification.

[0028] As shown in FIG. 1, a double-main-stem network-based smoking behavior detection model construction method includes the following steps: Figure 1

[0029] Step 1): using YOLOv5 as a model, using a smoking posture data set, and obtaining parameters of a ROI main-stem network through migration learning. In the step 1), images of a mouth smoking are collected, and a “smoking posture data set” is labeled and made, as shown in FIG. 2. Figure 2 ​(b) as shown: the annotation box requires the face and cigarette to be contained; then, the YOLOv5 model is trained using the "smoking posture data set"; the CSPDarknet in the trained smoking posture YOLOv5 model is saved as the ROI backbone network, as shown in Figure 3

[0030] Step 2) Use YOLOv5 as the model, use the cigarette data set, and obtain the cigarette detection model through transfer learning, including the cigarette backbone network, the cigarette neck network, and the cigarette detection head parameters. In step 2), cigarette images are collected, and the annotation method is as shown in Figure 2 (a) as shown: the annotation box requires only the cigarette to be contained; the "cigarette data set" is made, and then the YOLOv5 model is trained using the "cigarette data set"; the trained cigarette detection YOLOv5 model includes: cigarette backbone network, cigarette neck network and cigarette detection head three parts.

[0031] Step 3) Take the full convolutional network FCN as a spatial constraint network, connect the ROI backbone network and the cigarette backbone network in parallel to build a double backbone network smoking behavior detection model. In step 3), the FCN spatial constraint network is constructed using AlexNet; the ROI backbone network and the cigarette backbone network, the cigarette neck network and the cigarette detection head are connected using the FCN spatial constraint network to build a double backbone network smoking behavior detection model, and the specific steps are as follows:

[0032] Step 3-1: Use AlexNet as the encoder to build the FCN spatial constraint network: keep the five convolutional layers and three pooling layers of AlexNet unchanged, remove the three fully connected layers, and place equivalent convolutional layers; use deconvolution to upsample the feature map output by the last convolutional layer of the encoder, so that the feature map is restored to the resolution of the input feature map; as shown in Figure 4

[0033] Step 3-2: Use the three scale output feature maps of the ROI backbone network as input to generate different scale segmented smoking posture masks through the FCN spatial constraint network; and multiply the generated posture mask with the cigarette feature map of the same scale of the cigarette backbone network to obtain the optimized cigarette feature, and through spatial filtering to reduce the probability of cigarette false detection, the specific calculation formula is as follows:

[0034]

[0035] wherein is the i-th output feature of the ROI backbone network; is the i-th output feature of the cigarette backbone network; is the i-th output feature of the spatially filtered and optimized cigarette backbone network; Mask i is the​​ The generated smoking posture mask.

[0036] Step 3-3: cigarette feature optimized by spatial filtering Connect with the cigarette neck network and the cigarette detection head to form a double-main-stem cigarette detection network model, as shown in Figure 5 .

[0037] Step 4) Use the built double-main-stem network smoking behavior detection model to detect the smoking behavior of the image to be detected. In step 4), according to the structure and operation defined in steps 1), 2) and 3), the double-main-stem smoking behavior detection model is used to detect the smoking behavior of the image to be detected.

[0038] As Figure 6 , the detection accuracy comparison chart of the method of the present application and the YOLOv5 two cigarette detection methods on the 5630 smoking data set, the experimental results show that the method proposed in the present application has the highest detection accuracy and the lowest missed detection and false detection rate.

[0039] As Figure 7 , the detection result comparison chart of the method of the present application and the YOLOv5 detection result. It can be seen from it that the method proposed in this paper can effectively enhance the accuracy of cigarette detection in complex environment.

[0040] The technical content and technical features of the present application have been disclosed as above, and the protection scope of the present application is not limited by the above-mentioned embodiments, but any equivalent modifications and other modifications and changes made by those skilled in the art according to the disclosed content of the present application shall be included in the protection scope recited in the claims.

Claims

1. A method for constructing a smoking behavior detection model based on a double backbone network, characterized in that: It comprises the following steps: Step 1) using YOLOv5 as the model, using the smoking posture dataset, obtaining the parameters of the ROI backbone network through transfer learning; Step 2) using YOLOv5 as the model, using the cigarette dataset, obtaining the cigarette detection model including the cigarette backbone network, the cigarette neck network and the cigarette detection head parameters through transfer learning; Step 3) using full convolution network FCN as a spatial constraint network, connecting the ROI backbone network and the cigarette backbone network in parallel to build a double backbone network smoking behavior detection model; Step 4) using the double backbone network smoking behavior detection model built to detect the smoking behavior of the image to be detected. In step 3), the full convolution network FCN is used as a spatial constraint network, the ROI backbone network and the cigarette backbone network are connected in parallel, and a double backbone network smoking behavior detection model is built, and the specific steps are as follows: Step 3-1: using AlexNet as an encoder to build an FCN spatial constraint network: keeping the five convolution layers and three pooling layers of AlexNet unchanged, removing the three fully connected layers, and placing equivalent convolution layers; using deconvolution to upsample the feature map output by the last convolution layer of the encoder, so that the feature map is restored to the resolution of the input feature map; Step 3-2: using the three scale output feature maps of the ROI backbone network as input, generating different scale segmented smoking posture masks through the FCN spatial constraint network; and multiplying the generated posture mask with the cigarette feature map of the same scale of the cigarette backbone network to obtain the optimized cigarette feature, and reducing the probability of cigarette false detection through spatial filtering, the specific calculation formula is as follows: wherein is the i-th output feature of the ROI backbone network; is the i-th output feature of the cigarette backbone network; is the i-th output feature of the optimized cigarette backbone network after spatial filtering; Mask i is the smoking pose mask generated by the smoking pose mask generated by Step 3-3: cigarette features after spatial filtering optimization Connect with the cigarette neck network and the cigarette detection head to form a double-main-stem network smoking behavior detection model. 2.The method of claim 1, wherein the method further comprises: In step 1), YOLOv5 is used as the model, the smoking posture dataset is used, and the parameters of the ROI backbone network are obtained through transfer learning, and the specific steps are as follows: First, collect images of people smoking with their mouths, label and make a smoking posture dataset; the label box requires to contain the face and the cigarette; then, use the smoking posture dataset to train the YOLOv5 model; save the backbone network CSPDarknet in the trained smoking posture YOLOv5 model as the ROI backbone network.

3. The method for constructing a smoking behavior detection model based on a dual-backbone network according to claim 1, characterized in that: In step 2), YOLOv5 is used as the model, the cigarette dataset is used, and the parameters of the cigarette detection model including the cigarette backbone network, the cigarette neck network and the cigarette detection head are obtained through transfer learning, and the specific steps are as follows: First, collect cigarette images, label and make a cigarette dataset; then, use the cigarette dataset to train the YOLOv5 model; the trained cigarette detection YOLOv5 model includes three parts: the cigarette backbone network, the cigarette neck network and the cigarette detection head. 4.The method of claim 1, wherein the method further comprises: In step 4), according to the structure and operation defined in steps 1), 2) and 3), the double backbone smoking behavior detection model is used to detect the smoking behavior of the image to be detected.

Images