Smoking behavior detection method based on improved YOLOv5s model

Abstract

The invention relates to a smoking behavior detection method based on an improved YOLOv5s model, and belongs to the field of target detection. Comprising the following steps: collecting a smoking behavior image, and carrying out preprocessing, image enhancement and annotation on the smoking behavior image to construct a data set; constructing a smoking behavior detection model, training the smoking behavior detection model based on the data set, performing positive sample expansion in the training process, and obtaining a trained smoking behavior detection model after the training is completed; and inputting a to-be-detected image into the trained smoking behavior detection model to realize smoking behavior detection. According to the method, the calculation efficiency is improved by improving the activation function, the model convergence speed and the target detection precision are improved by improving the loss function, and the problems that an existing smoking behavior detection method is low in real-time performance and low in accuracy are solved.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a smoking behavior detection method based on an improved YOLOv5s model. Background Art

[0002] Smoking not only seriously harms people's health but also easily causes accidents such as fires. Therefore, how to automatically detect smoking behavior has always been a hot research issue. With the rapid development of artificial intelligence, object detection technology based on deep learning has been widely used in the field of automatic detection of smoking behavior.

[0003] The existing deep learning-based smoking behavior detection methods usually adopt the following methods:

[0004] (1) Using deep neural networks to extract features of smoking gestures, thereby achieving automatic detection of smoking behavior;

[0005] (2) Identify cigarette smoke by analyzing its HOG features and texture features, thereby achieving automatic detection of smoking behavior;

[0006] (3) By establishing a target detection model based on a deep learning algorithm, cigarettes can be identified, thereby achieving automatic detection of smoking behavior.

[0007] The above method has the following defects:

[0008] For method (1), due to the factors such as the variability of smoking gestures, the diversity of human skin color, different camera shooting angles, and the easy confusion with gestures similar to smoking behavior, the recognition error rate of smoking gestures is relatively high, and the accuracy of smoking behavior detection is relatively low;

[0009] For method (2), due to factors such as low concentration, easy diffusion, and unclear edges of cigarette smoke, cigarette smoke is difficult to distinguish from the background, and the accuracy of smoking behavior detection is low;

[0010] For method (3), due to factors such as the cigarette target being small and easily confused with tobacco-like objects, the cigarette target is difficult to identify and the accuracy of smoking behavior detection is low. In addition, due to the high spatiotemporal complexity of the established target detection model, the real-time performance of smoking behavior detection is low.

[0011] It can be seen that providing a smoking behavior detection method with high detection accuracy and high real-time performance is an urgent problem to be solved. Summary of the invention

[0012] In view of the above analysis, the present invention aims to provide a smoking behavior detection method based on an improved YOLOv5s model to solve the problems of low real-time performance and low accuracy of existing smoking behavior detection methods.

[0013] The present invention provides a smoking behavior detection method based on an improved YOLOv5s model, the method comprising the following steps:

[0014] Collecting smoking behavior images, and performing preprocessing, image enhancement, and annotation on the smoking behavior images to construct a data set;

[0015] Constructing a smoking behavior detection model, training the smoking behavior detection model based on the data set, performing positive sample expansion during the training process, and obtaining a trained smoking behavior detection model after the training is completed;

[0016] The image to be detected is input into the trained smoking behavior detection model to realize smoking behavior detection.

[0017] Furthermore, the smoking behavior detection model is trained based on the data set, positive samples are expanded during the training process, and a trained smoking behavior detection model is obtained after the training is completed, including:

[0018] Input each sample in the training set of the data set into the smoking behavior detection model for training, and output a corresponding reasoning graph; the reasoning graph includes prediction boxes, each prediction box includes the coordinates of the center point corresponding to the prediction box, the height and width of the prediction box, the detection target category number, and the confidence score;

[0019] If the prediction box in the inference graph is a positive sample, the corresponding prediction box is added to expand the positive sample;

[0020] The loss function is calculated by the error between the labeled box of the sample and the predicted box of the corresponding reasoning graph. When the loss function reaches a threshold, a trained smoking behavior detection model is obtained.

[0021] Furthermore, if the prediction box in the reasoning graph is a positive sample, adding a corresponding prediction box to expand the positive sample includes: if the prediction box in the reasoning graph is a positive sample, establishing four adjacent prediction boxes in the four directions of the prediction box above, below, left and right according to the size of the prediction box, selecting two prediction boxes whose center points are closest to the center point of the annotation box corresponding to the prediction box from the four adjacent prediction boxes as newly added prediction boxes; setting the category of the newly added prediction box to be consistent with the prediction box.

[0022] Furthermore, the loss function includes detection target confidence loss, detection target category loss, and prediction box loss.

[0023] Furthermore, the expression of the loss function is as follows:

[0024] Loss = λ obj BCE obj +λcls BCE cls +λ box FIoU,

[0025] Among them, BCE obj To detect the target confidence loss, BCE cls is the target category loss, FIoU is the prediction box loss, and λ obj , cls , box is the weight coefficient.

[0026] Furthermore, the detection target confidence loss represents the error in prediction accuracy, that is, whether there is a target in the prediction box. The loss function expression of the detection target confidence loss is as follows:

[0027]

[0028]

[0029] Among them, N is the total number of samples, m is the number of positive samples, Nm is the number of negative samples, and y i =1 means there is a target in the prediction box, y i =0 means there is no target in the prediction box, p 1 is the probability of the existence of an object in the prediction box, and α is the coefficient.

[0030] Furthermore, the detection target category loss represents the error between the category of the prediction box and the category of the annotation box. The loss function expression of the detection target category loss is as follows:

[0031]

[0032] Among them, N p is the number of positive samples, n is the number of categories, C i =1 means the predicted category belongs to C i Category, C i =0 means the predicted category does not belong to C i Category, β i C is the positive sample i The proportion of the category, γ is the control coefficient, p i Indicates that the prediction is C i The probability of the class.

[0033] Furthermore, the prediction box loss represents the position error between the prediction box and the sample annotation box, and the loss function expression of the prediction box loss is as follows:

[0034]

[0035] Among them, Bpre Indicates the area occupied by the prediction box, B gt represents the area occupied by the annotation box, IoU is the intersection-over-union ratio of the predicted box and the annotation box, ρ 2 (B pre , B gt ) represents the distance between the center point of the prediction box and the center point of the annotation box, w pre is the width of the prediction box, w gt is the width of the annotation box, ρ 2 (w pre , w gt ) represents the square of the difference between the width of the prediction box and the width of the annotation box, h pre is the height of the prediction box, h gt is the height of the annotation box, ρ 2 (h pre ,h gt ) represents the square of the difference between the height of the predicted box and the height of the labeled box, cw 2 Represents the square of the minimum enclosing rectangle width of the prediction box and the annotation box, c h 2 Represents the square of the minimum enclosing rectangle height of the prediction box and the annotation box, c 2 Represents the square of the diagonal length of the minimum circumscribed rectangle of the prediction box and the annotation box. Represents the empty set.

[0036] Furthermore, the preprocessing, image enhancement and annotation of the smoking behavior image to construct a data set includes:

[0037] Eliminate smoking behavior images with unclear cigarette feature points, and adjust the pixel size and format of the remaining smoking behavior images to be consistent;

[0038] Performing image enhancement processing on the smoking behavior images after pixel size adjustment;

[0039] The smoking behavior images after pixel size adjustment and image enhancement are annotated to construct a dataset, and the dataset is divided into a training set and a test set.

[0040] Furthermore, the image enhancement processing of the smoking behavior image after pixel size adjustment includes one or more of the following methods:

[0041] Randomly selecting part of the content in the smoking behavior image and copying it to other images to generate a new smoking behavior image;

[0042] Performing a two-dimensional linear transformation on the smoking behavior image to generate a new smoking behavior image; the two-dimensional linear transformation includes rotation, translation, scaling, shearing, and flipping;

[0043] Randomly selecting two images from the smoking behavior image, and mixing the two images in a certain ratio to generate a new smoking behavior image;

[0044] Randomly adjusting the hue, saturation, and brightness of the smoking behavior image to generate a new smoking behavior image;

[0045] Randomly block a part of the smoking behavior image to generate a new smoking behavior image.

[0046] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0047] 1. The NHSwish activation function proposed in the present invention uses a piecewise function to simplify the expression, which improves the calculation efficiency and is suitable for embedded devices with limited resources.

[0048] 2. The C5_1 module and C5_2 module proposed in the present invention are composed of CBH modules through multiple channel superposition, which not only reduces the number of parameters and calculations, but also increases the information fusion between different channels; the SPPFS module has a deeper information fusion effect by increasing the information fusion between different channels, which is conducive to more comprehensive extraction of global feature information and provides a basis for improving the accuracy of model prediction.

[0049] 3. The present invention inputs the output of the p2 unit in the backbone network to the neck network FPN and inputs the output of the second feature fusion layer in the FPN to the PAN, which can reduce the attenuation of the feature information of small targets in the multi-layer convolution calculation process, and enables the feature information to be fused in the FPN and PAN, thereby enhancing the propagation of shallow semantic information of the image, balancing the contradiction between the deep network for target category recognition and the shallow network for small target positioning and extracting image features of different scales; the output of the SPPFS module in the backbone network is directly input into the neck network PAN as an information flow, which improves the accuracy of the prediction box position and provides a basis for improving the accuracy of model prediction.

[0050] 4. BCE proposed by the present invention obj The loss function can dynamically adjust the loss values ​​of positive and negative samples, thereby improving the speed of model convergence and the accuracy of target detection; BCE cls The loss function can dynamically adjust the loss values ​​of different categories to balance the imbalance in the number of samples between different categories. The FIoU loss function improves the accuracy of the aspect ratio by calculating the length ratio and width ratio of the predicted box and the labeled box respectively instead of considering the consistency of the aspect ratio. It can balance the loss values ​​when the predicted box and the labeled box have no intersection, few intersections, and high overlap, avoiding the prediction box loss value being too large or too small, and accelerating the convergence of the model.

[0051] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0053] Figure 1 This is a flow chart of a smoking behavior detection method based on an improved YOLOv5s model according to an embodiment of the present invention;

[0054] Figure 2 is a schematic diagram of a smoking behavior image according to an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the network structure of the initial smoking behavior detection model according to an embodiment of the present invention;

[0056] Figure 4 It is a schematic diagram of the structure of the CBH, C5_1 and C5_2 modules according to an embodiment of the present invention;

[0057] Figure 5 It is the NHSwish activation function image and its derivative function image according to the embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the structure of the SPPFS module according to an embodiment of the present invention;

[0059] Figure 7 Schematic diagram of the network structure of the smoking behavior detection model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0061] A specific embodiment of the present invention discloses a smoking behavior detection method based on an improved YOLOv5s model. Figure 1 As shown, the method comprises the following steps:

[0062] Step S1, collecting smoking behavior images, preprocessing the smoking behavior images, image enhancement and annotation to construct a data set;

[0063] Step S2, constructing a smoking behavior detection model, training the smoking behavior detection model based on the data set, performing positive sample expansion during the training process, and obtaining a trained smoking behavior detection model after the training is completed;

[0064] Step S3: input the image to be detected into the trained smoking behavior detection model to implement smoking behavior detection.

[0065] Specifically, in step S1, smoking behavior images are collected through monitoring cameras and the Internet; the smoking behavior images include images of smoking and images of holding a cigarette.

[0066] Furthermore, the preprocessing, image enhancement and annotation of the smoking behavior image to construct a data set includes:

[0067] Eliminate smoking behavior images with unclear cigarette feature points, and adjust the pixel size and format of the remaining smoking behavior images to be consistent;

[0068] Performing image enhancement processing on the smoking behavior images after pixel size adjustment;

[0069] The smoking behavior images after pixel size adjustment and image enhancement are annotated to construct a dataset, and the dataset is divided into a training set and a test set.

[0070] It should be noted that among the collected smoking behavior images, the cigarette feature points of some images may not be obvious, for example, the image itself is blurry, the area occupied by the cigarette in the image is very small, the cigarette itself is not clear enough, etc., and such images are eliminated by manual screening. Since the pixel size and format of the smoking behavior images collected by the monitoring camera may be different from those collected through the Internet, their pixel size and format are adjusted to be consistent; the minimum pixel size of the remaining images is obtained, and the pixel sizes of all the remaining images are adjusted to the minimum pixel size, and the format of all the remaining images is adjusted to jpeg.

[0071] For example, Figure 2 As shown in the figure, 8400 smoking behavior images with good lighting conditions were collected by two surveillance cameras, the image pixel size was 1920×1080, and the image format was jpeg; 4524 smoking behavior images were collected through the Internet, the image pixel size was 1280×720, and the image format was bmp; 100 images with unclear cigarette feature points were eliminated from the above 12924 images; the minimum pixel size of the remaining 12824 images was 1280×720, and the pixel sizes of the above 12824 images were adjusted to 1280×720 and the formats were adjusted to jpeg through image processing software.

[0072] Furthermore, the image enhancement processing of the smoking behavior image after pixel size adjustment includes one or more of the following methods:

[0073] Randomly selecting part of the content in the smoking behavior image and copying it to other images to generate a new smoking behavior image;

[0074] Performing a two-dimensional linear transformation on the smoking behavior image to generate a new smoking behavior image; the two-dimensional linear transformation includes rotation, translation, scaling, shearing, and flipping;

[0075] Randomly selecting two images from the smoking behavior image, and mixing the two images in a certain ratio to generate a new smoking behavior image;

[0076] Randomly adjusting the hue, saturation, and brightness of the smoking behavior image to generate a new smoking behavior image;

[0077] Randomly block a part of the smoking behavior image to generate a new smoking behavior image.

[0078] It should be noted that the shearing includes horizontal shearing and vertical shearing. Horizontal shearing achieves a distortion effect by adding a horizontal displacement on each row of the image, and vertical shearing achieves a distortion effect by adding a vertical displacement on each column of the image.

[0079] It can be understood that through image enhancement, not only the number of samples is increased, but also the background of the image is enriched, so that the smoking behavior detection model can learn more diverse knowledge during training, providing a basis for improving the robustness and generalization ability of the model.

[0080] Furthermore, a dataset is constructed by annotating each target to be detected in the smoking behavior image after pixel size adjustment and image enhancement with a rectangular box through the LabelImg annotation tool; the targets to be detected include smoking actions, people, and cigarettes; and the annotation content includes the category number, the coordinates of the center point of the target to be detected, and the height and width of the annotation box.

[0081] Specifically, the data set includes smoking behavior images after pixel size adjustment and image enhancement and their corresponding annotation content; the data set is divided into a training set and a test set in a ratio of 8:2.

[0082] It should be noted that the annotation content of each image is generated as a file in txt format with the same name as the image. Therefore, the data set includes a set of smoking behavior images and a set of corresponding annotation content files.

[0083] Specifically, in step S2, the construction of the smoking behavior detection model includes:

[0084] An initial smoking behavior detection model is constructed based on the YOLOv5s model, wherein the initial smoking behavior detection model includes an input terminal (Input), a backbone network (Backbone), a neck network (Neck) and a detection terminal (Detect);

[0085] The CBS module of the backbone network is improved to obtain a CBH module, the C3_1 module of the backbone network is improved to obtain a C5_1 module, and the SPPF module of the backbone network is improved to obtain a SPPFS module;

[0086] The C3_2 module of the neck network is improved to obtain the C5_2 module;

[0087] The CBS module, C3_1 module and SPPF module in the backbone network are replaced by the CBH module, C5_1 module and SPPFS module respectively, and the CBS module and C3_2 module in the neck network are replaced by the CBH module and C5_2 module respectively, so as to construct a smoking behavior detection model.

[0088] Specifically, the initial smoking behavior detection model is constructed based on the YOLOv5s model, and its network structure diagram is as follows: Figure 3As shown. The input end is used to obtain the image to be detected; the backbone network is used to extract feature information of multiple scales of the image to obtain feature maps of multiple corresponding scales; the neck network includes a feature pyramid network (FPN) and a path aggregation network (PAN), which are used to fuse the feature maps of multiple corresponding scales, fuse the shallow feature information of the image with the deep feature information, and input them to the detection end; the detection end is used to output target detection results of multiple corresponding scales. Among them, the CBS module includes a convolutional layer (Conv2d), a batch normalization layer (BN), and a SiLu activation function connected in sequence, which are used to extract feature information of multiple scales of the image. The C3_1 module includes a first convolution module, a first feature fusion layer (Concat), and a first CBS module connected in sequence, which are used to fuse feature information of multiple different scales; wherein, the first convolution module includes a second CBS module and a first bottleneck module connected in parallel, the first bottleneck module includes a third CBS module and a first bottleneck layer (BottleNeck1) connected in sequence, the first bottleneck layer includes a residual module and a second feature fusion layer connected in sequence, the residual module includes a jump connection and a dual CBS module connected in parallel, and the dual CBS module includes a fourth and a fifth CBS module connected in sequence. The C3_2 module includes a second convolution module, a first feature fusion layer, and a first CBS module connected in sequence, which are used to fuse feature information of multiple different scales; the second convolution module includes a second CBS module and a second bottleneck module connected in parallel, the second bottleneck module includes a third CBS module and a second bottleneck layer (BottleNeck2) connected in sequence, and the second bottleneck layer includes a fourth and a fifth CBS module connected in sequence. The SPPF module includes a first CBS module, three maximum pooling layers (MaxPool2d), a feature fusion layer, and a second CBS module connected in sequence, and is used to extract global feature information of the image; wherein the output of the first CBS module and the output of the three maximum pooling layers are also jump-connected to the feature fusion layer.

[0089] Furthermore, if Figure 4 As shown, the CBH module includes a convolutional layer, a batch normalization layer, and an NHSwish activation function connected in sequence.

[0090] Specifically, the expression of the NHSwish activation function is:

[0091]

[0092] Among them, x represents the output result of the batch normalization layer;

[0093] Its image and its derivative image are as follows Figure 5 shown.

[0094] It can be understood that the NHSwish activation function uses piecewise functions to simplify the expression, improves the computational efficiency, and is suitable for embedded devices with limited resources.

[0095] Furthermore, if Figure 4 As shown, the C5_1 module includes the first and second segmentation layers (Split), the first to fifth CBH modules, and a feature fusion layer; wherein the input feature map is divided into two paths through the first segmentation layer and then respectively input into the feature fusion layer through the first CBH module and the second CBH module, the output of the second CBH module is also divided into two paths through the second segmentation layer and then respectively input into the feature fusion layer through the third CBH module and the fourth CBH module, and the output of the feature fusion layer is input into the fifth CBH module.

[0096] Furthermore, if Figure 4 As shown, the C5_2 module includes a segmentation layer, first to fifth CBH modules, and a feature fusion layer; wherein the input feature map is divided into two paths through the segmentation layer and then respectively input into the feature fusion layer through the first CBH module and the second CBH module, the output of the second CBH module is also input into the feature fusion layer through the third CBH module, the output of the third CBH module is also input into the feature fusion layer through the fourth CBH module, and the output of the feature fusion layer is input into the fifth CBH module.

[0097] It can be understood that the C5_1 module and the C5_2 module are composed of CBH modules through multiple channel superposition, which not only reduces the number of parameters and calculations, but also increases the information fusion between different channels.

[0098] Furthermore, if Figure 6 As shown, the SPPFS module includes a segmentation layer, a first and a second CBH module, a first feature fusion layer, and a maximum pooling module; wherein the input feature map is divided into two paths by the segmentation layer and then respectively input into the first feature fusion layer through the maximum pooling module and the first CBH module, and the output of the first feature fusion layer is input into the second CBH module; the maximum pooling module includes a third CBH module, three maximum pooling layers, a second feature fusion layer, and a fourth CBH module connected in sequence, and the output of the third CBH module and the output of the three maximum pooling layers are also jump-connected to the second feature fusion layer.

[0099] It can be understood that the SPPFS module has a deeper information fusion effect by increasing the information fusion between different channels, which is conducive to extracting global feature information more comprehensively.

[0100] Further, the backbone network of the initial smoking behavior detection model and all CBS modules in the neck network are replaced with the CBH module; all C3_1 modules in the backbone network of the initial smoking behavior detection model are replaced with the C5_1 module; the SPPF module in the backbone network of the initial smoking behavior detection model is replaced with the SPPFS module; and all C3_2 modules in the neck network of the initial smoking behavior detection model are replaced with the C5_2 module, thereby constructing a smoking behavior detection model. The network structure diagram of the smoking behavior detection model is shown in FIG. Figure 7 shown.

[0101] Furthermore, in the smoking behavior detection model:

[0102] The output of the third layer C5_1 module in the backbone network, i.e., the p2 unit, is input to the first layer CBH module in the neck network FPN;

[0103] The output of the SPPFS module in the backbone network is input to the sixth feature fusion layer in the neck network PAN;

[0104] The output of the second feature fusion layer in the neck network FPN is input to the first CBH module in the neck network PAN.

[0105] It can be understood that inputting the output of the p2 unit in the backbone network to the neck network FPN and inputting the output of the second feature fusion layer in FPN to PAN can reduce the attenuation of small target feature information in the multi-layer convolution calculation process, enable feature information to be fused in FPN and PAN, enhance the propagation of shallow semantic information of the image, and balance the contradiction between the deep network for target category recognition and the shallow network for small target positioning and image feature extraction of different scales. The output of the SPPFS module in the backbone network is directly input into the neck network PAN as an information flow, which improves the accuracy of the predicted box position.

[0106] Furthermore, the smoking behavior detection model is trained based on the data set, positive samples are expanded during the training process, and a trained smoking behavior detection model is obtained after the training is completed, including:

[0107] Input each sample in the training set of the data set into the smoking behavior detection model for training, and output a corresponding reasoning graph; the reasoning graph includes prediction boxes, each prediction box includes the coordinates of the center point corresponding to the prediction box, the height and width of the prediction box, the detection target category number, and the confidence score;

[0108] If the prediction box in the inference graph is a positive sample, the corresponding prediction box is added to expand the positive sample;

[0109] The loss function is calculated by the error between the labeled box of the sample and the predicted box of the corresponding reasoning graph. When the loss function reaches a threshold, a trained smoking behavior detection model is obtained.

[0110] Furthermore, if the prediction box in the reasoning graph is a positive sample, adding a corresponding prediction box to expand the positive sample includes: if the prediction box in the reasoning graph is a positive sample, establishing four adjacent prediction boxes in the four directions of the prediction box above, below, left and right according to the size of the prediction box, selecting two prediction boxes whose center points are closest to the center point of the annotation box corresponding to the prediction box from the four adjacent prediction boxes as newly added prediction boxes; setting the category of the newly added prediction box to be consistent with the prediction box.

[0111] Specifically, a positive sample refers to a sample in which the error between the predicted box in the inference graph and the actual labeled box of the sample is less than the threshold, indicating that the area intersection ratio of the predicted box and the labeled box is greater than the threshold and the category of the predicted box is consistent with the category of the labeled box, otherwise it is a negative sample.

[0112] It can be understood that by increasing the number of matching prediction boxes, a real annotation box can be predicted by multiple prediction boxes, thereby increasing the number of positive samples and reducing the impact caused by the imbalance of positive and negative samples.

[0113] Furthermore, the loss function includes detection target confidence loss, detection target category loss, and prediction box loss; the expression of the loss function is as follows:

[0114] Loss = λ obj BCE obj +λ cls BCE cls +λ box FIoU,

[0115] Among them, BCE obj To detect the target confidence loss, BCE cls is the target category loss, FIoU is the prediction box loss, and λ obj , cls , box is the weight coefficient.

[0116] Preferably, λ obj Set to 1.0, λ cls Set to 0.5, λ box Set to 0.05.

[0117] Specifically, the detection target confidence loss represents the error in prediction accuracy, that is, whether there is a target in the prediction box. The loss function expression of the detection target confidence loss is as follows:

[0118]

[0119]

[0120] Among them, N is the total number of samples, m is the number of positive samples, Nm is the number of negative samples, and y i =1 means there is a target in the prediction box, y i =0 means there is no target in the prediction box, p 1 is the probability of the existence of an object in the prediction box, and α is the coefficient.

[0121] It can be understood that negative samples correspond to the image background, and positive samples correspond to the actual target area in the image. Usually, there are more negative samples and fewer positive samples, which easily leads to the loss value of negative samples overwhelming the loss value of positive samples, thus reducing the convergence speed of the model and the accuracy of target detection. obj The loss function can dynamically adjust the loss values ​​of positive and negative samples, thereby improving the speed of model convergence and the accuracy of target detection.

[0122] Specifically, the detection target category loss represents the error between the category of the prediction box and the category of the annotation box. The loss function expression of the detection target category loss is as follows:

[0123]

[0124] Among them, N p is the number of positive samples, n is the number of categories, C i =1 means the predicted category belongs to C i Category, C i =0 means the predicted category does not belong to C i Category, β i C is the positive sample i The proportion of the category, γ is the control coefficient, p i Indicates that the prediction is C i The probability of the class.

[0125] Understandably, through β i The loss values ​​of different categories can be adjusted dynamically to balance the imbalance in the number of samples between different categories.

[0126] Specifically, the prediction box loss represents the position error between the prediction box and the sample annotation box. The loss function expression of the prediction box loss is as follows:

[0127]

[0128] Among them, B pre Indicates the area occupied by the prediction box, B gtrepresents the area occupied by the annotation box, IoU is the intersection-over-union ratio of the predicted box and the annotation box, ρ 2 (B pre , B gt ) represents the distance between the center point of the prediction box and the center point of the annotation box, w pre is the width of the prediction box, w gt is the width of the annotation box, ρ 2 (w pre , w gt ) represents the square of the difference between the width of the prediction box and the width of the annotation box, h pre is the height of the prediction box, h gt is the height of the annotation box, ρ 2 (h pre ,h gt ) represents the square of the difference between the height of the predicted box and the height of the labeled box, cw 2 Represents the square of the minimum enclosing rectangle width of the prediction box and the annotation box, c h 2 Represents the square of the minimum enclosing rectangle height of the prediction box and the annotation box, c 2 Represents the square of the diagonal length of the minimum circumscribed rectangle of the prediction box and the annotation box. Represents the empty set.

[0129] It can be understood that if the IoU value between the prediction box and the annotation box is small or even 0, the prediction box loss value will fluctuate violently, which is not conducive to the convergence of the model. The FIoU loss function proposed in the present invention improves the accuracy of the aspect ratio by respectively calculating the length ratio and width ratio of the prediction box and the annotation box instead of considering the consistency of the aspect ratio. It can balance the loss value when the prediction box and the annotation box have no intersection, less intersection, and high overlap, avoids the prediction box loss value being too large or too small, and accelerates the convergence of the model.

[0130] Specifically, the number of training rounds is set to 200. In each round of training, the batch size is set to 8, the optimizer is Adam, the initial learning rate is 0.01, the weight decay rate is 0.001, the momentum factor is 0.937, the weight decay factor is 0.0005, the detection target confidence loss threshold is 0.5, the detection target category loss threshold is 0.75, and the prediction box loss threshold is 0.5.

[0131] Specifically, in step S3, the image to be detected is input into the trained smoking behavior detection model to obtain an inference graph, and redundant prediction frames are removed from the inference graph to obtain a smoking behavior detection result, thereby realizing smoking behavior detection.

[0132] Specifically, the following method is used to remove redundant detection frames:

[0133] A score is calculated for each prediction box in the inference graph, and the prediction boxes are sorted from high to low according to the score; starting from the prediction box with the highest score, all prediction boxes whose overlap degree is greater than a threshold are deleted.

[0134] It can be understood that each prediction box in the reasoning graph has coordinates, categories, and confidence scores. For each prediction box in the reasoning graph, its score is obtained in the following way:

[0135] F=r 1 *F 1 +r 2 *F 2 ,

[0136] Among them, F is the score of the prediction box, F 1 is the confidence score of the predicted box, F 2 is the score corresponding to the category to which the prediction box belongs, r 1 、r 2 is the weight.

[0137] Specifically, the target in the smoking behavior detection result is judged, and when a smoking action or a cigarette target appears, it is judged that the smoking behavior is detected.

[0138] Performance evaluation:

[0139] Specifically, the performance of the smoking behavior detection model proposed in the present invention and YOLOv5s was evaluated from three aspects: mean average precision (mAP), precision (P), and recall (R) through ablation experiments. Precision P is used to evaluate the accuracy of smoking behavior detection, and recall R is used to evaluate the comprehensiveness of smoking behavior detection. mAP refers to the average area under the PR curve of all categories, which comprehensively considers the precision and recall of the model.

[0140] Specifically, the calculation formulas for average accuracy, precision, and recall are as follows:

[0141]

[0142]

[0143]

[0144]

[0145] Among them, TP represents the number of correct smoking behavior detection results, FP represents the number of incorrect smoking behavior detection results, FN represents the number of missed detections where smoking behavior was not detected, AP represents the area under the PR curve of this category, and n cis the total number of categories.

[0146] Specifically, the YOLOv5s model parameters trained based on the COCO dataset are used as the pre-training model; as shown in Table 1, ablation experiments are conducted on the five models using the above datasets, each model is trained for 200 rounds, and then the results of the above indicators are obtained:

[0147] Table 1. Ablation comparison experimental results

[0148]

[0149] It can be seen from the above table that compared with YOLOv5s, the smoking behavior detection model proposed in the present invention has obvious improvements in precision, recall and average precision, which effectively solves the problems of low accuracy and low real-time performance of existing smoking behavior detection.

[0150] Compared with the prior art, the smoking behavior detection method based on the improved YOLOv5s model provided by the present invention has the following beneficial effects:

[0151] 1. The NHSwish activation function proposed in the present invention uses a piecewise function to simplify the expression, which improves the calculation efficiency and is suitable for embedded devices with limited resources.

[0152] 2. The C5_1 module and C5_2 module proposed in the present invention are composed of CBH modules through multiple channel superposition, which not only reduces the number of parameters and calculations, but also increases the information fusion between different channels; the SPPFS module has a deeper information fusion effect by increasing the information fusion between different channels, which is conducive to more comprehensive extraction of global feature information and provides a basis for improving the accuracy of model prediction.

[0153] 3. The present invention inputs the output of the p2 unit in the backbone network to the neck network FPN and inputs the output of the second feature fusion layer in the FPN to the PAN, which can reduce the attenuation of the feature information of small targets in the multi-layer convolution calculation process, and enables the feature information to be fused in the FPN and PAN, thereby enhancing the propagation of shallow semantic information of the image, balancing the contradiction between the deep network for target category recognition and the shallow network for small target positioning and extracting image features of different scales; the output of the SPPFS module in the backbone network is directly input into the neck network PAN as an information flow, which improves the accuracy of the prediction box position and provides a basis for improving the accuracy of model prediction.

[0154] 4. BCE proposed by the present invention obj The loss function can dynamically adjust the loss values ​​of positive and negative samples, thereby improving the speed of model convergence and the accuracy of target detection; BCE clsThe loss function can dynamically adjust the loss values ​​of different categories to balance the imbalance in the number of samples between different categories. The FIoU loss function improves the accuracy of the aspect ratio by calculating the length ratio and width ratio of the predicted box and the labeled box respectively instead of considering the consistency of the aspect ratio. It can balance the loss values ​​when the predicted box and the labeled box have no intersection, few intersections, and high overlap, avoiding the prediction box loss value being too large or too small, and accelerating the convergence of the model.

[0155] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0156] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A smoking behavior detection method based on an improved YOLOv5s model, characterized in that: The method comprises the following steps: Collecting smoking behavior images, and performing preprocessing, image enhancement, and annotation on the smoking behavior images to construct a data set; Constructing a smoking behavior detection model, training the smoking behavior detection model based on the data set, performing positive sample expansion during the training process, and obtaining a trained smoking behavior detection model after the training is completed; The image to be detected is input into the trained smoking behavior detection model to realize smoking behavior detection.

2. The smoking behavior detection method based on the improved YOLOv5s model according to claim 1, characterized in that: The smoking behavior detection model is trained based on the data set, positive samples are expanded during the training process, and a trained smoking behavior detection model is obtained after the training is completed. The training includes: Input each sample in the training set of the data set into the smoking behavior detection model for training, and output a corresponding reasoning graph; the reasoning graph includes prediction boxes, each prediction box includes the coordinates of the center point corresponding to the prediction box, the height and width of the prediction box, the detection target category number, and the confidence score; If the prediction box in the inference graph is a positive sample, the corresponding prediction box is added to expand the positive sample; The loss function is calculated by the error between the labeled box of the sample and the predicted box of the corresponding reasoning graph. When the loss function reaches a threshold, a trained smoking behavior detection model is obtained.

3. The smoking behavior detection method based on the improved YOLOv5s model according to claim 2, characterized in that: If the prediction box in the reasoning graph is a positive sample, adding a corresponding prediction box to expand the positive sample includes: if the prediction box in the reasoning graph is a positive sample, establishing four adjacent prediction boxes in the upper, lower, left and right directions of the prediction box according to the size of the prediction box, selecting two prediction boxes whose center points are closest to the center point of the annotation box corresponding to the prediction box from the four adjacent prediction boxes as newly added prediction boxes; and setting the category of the newly added prediction box to be consistent with the prediction box.

4. The smoking behavior detection method based on the improved YOLOv5s model according to claim 2, characterized in that: The loss function includes detection target confidence loss, detection target category loss, and prediction box loss.

5. The smoking behavior detection method based on the improved YOLOv5s model according to claim 2, characterized in that: The expression of the loss function is as follows: Loss=λ obj BCE obj +λ cls BCE cls +λ box FIoU, Among them, BCE obj To detect the target confidence loss, BCE cls is the target category loss, FIoU is the prediction box loss, and λ obj , cls , box is the weight coefficient.

6. The smoking behavior detection method based on the improved YOLOv5s model according to claim 5, characterized in that: The detection target confidence loss represents the error in prediction accuracy, that is, whether there is a target in the prediction box. The loss function expression of the detection target confidence loss is as follows: Among them, N is the total number of samples, m is the number of positive samples, Nm is the number of negative samples, and y i =1 means there is a target in the prediction box, y i =0 means that there is no target in the prediction box, p1 is the probability that there is a target in the prediction box, and α is the coefficient.

7. The smoking behavior detection method based on the improved YOLOv5s model according to claim 5, characterized in that: The detection target category loss represents the error between the category of the prediction box and the category of the annotation box. The loss function expression of the detection target category loss is as follows: Among them, N p is the number of positive samples, n is the number of categories, C i =1 means the predicted category belongs to C i Category, C i =0 means the predicted category does not belong to C i Category, β i C is the positive sample i The proportion of the category, γ is the control coefficient, p i Indicates that the prediction is C i The probability of the class.

8. The smoking behavior detection method based on the improved YOLOv5s model according to claim 5, characterized in that: The prediction box loss represents the position error between the prediction box and the sample annotation box. The loss function expression of the prediction box loss is as follows: Among them, B pre Indicates the area occupied by the prediction box, B gt represents the area occupied by the annotation box, IoU is the intersection-over-union ratio of the predicted box and the annotation box, ρ 2 (B pre , B gt ) represents the distance between the center point of the prediction box and the center point of the annotation box, w pre is the width of the prediction box, w gt is the width of the annotation box, ρ 2 (B pre , B gt ) represents the square of the difference between the width of the prediction box and the width of the annotation box, h pre is the height of the prediction box, h gt is the height of the annotation box, ρ 2 (h pre ,h gt ) represents the square of the height difference between the predicted box and the labeled box, c w 2 Represents the square of the minimum enclosing rectangle width of the prediction box and the annotation box, c h 2 Represents the square of the minimum enclosing rectangle height of the prediction box and the annotation box, c 2 Represents the square of the diagonal length of the minimum circumscribed rectangle of the prediction box and the annotation box. Represents the empty set.

9. The smoking behavior detection method based on the improved YOLOv5s model according to claim 1, characterized in that: The preprocessing, image enhancement and annotation of the smoking behavior image to construct a data set includes: Eliminate smoking behavior images with unclear cigarette feature points, and adjust the pixel size and format of the remaining smoking behavior images to be consistent; Performing image enhancement processing on the smoking behavior images after pixel size adjustment; The smoking behavior images after pixel size adjustment and image enhancement are annotated to construct a dataset, and the dataset is divided into a training set and a test set.

10. The smoking behavior detection method based on the improved YOLOv5s model according to claim 9, characterized in that: The image enhancement processing of the smoking behavior image after pixel size adjustment includes one or more of the following methods: Randomly selecting part of the content in the smoking behavior image and copying it to other images to generate a new smoking behavior image; Performing a two-dimensional linear transformation on the smoking behavior image to generate a new smoking behavior image; the two-dimensional linear transformation includes rotation, translation, scaling, shearing, and flipping; Randomly selecting two images from the smoking behavior image, and mixing the two images in a certain ratio to generate a new smoking behavior image; Randomly adjusting the hue, saturation, and brightness of the smoking behavior image to generate a new smoking behavior image; Randomly block a part of the smoking behavior image to generate a new smoking behavior image.

Images