A method for detecting smoking behavior under video surveillance

Abstract

This invention discloses a method for detecting smoking behavior under video surveillance. Addressing the issues of false detection and missed detection of smoke and cigarettes in complex environments, this invention achieves smoking behavior detection by combining cigarette and smoke features. To address the low accuracy of small smoke detection in traditional smoke detection methods, this invention completes a smoke detection technique based on feature fusion. For real-time detection in surveillance, this invention designs a lightweight cigarette detection model based on depthwise separable convolution and improved YOLOv5. Experimental results show that the smoking behavior detection method provided by this invention has high accuracy and robustness, can effectively distinguish between smoking and non-smoking behaviors, and maintains good detection performance even under complex environments with varying lighting, shadows, and occlusion.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition technology, and specifically relates to a method for detecting smoking behavior. Background Technology

[0002] Smoking not only harms the health of oneself and others, but the loss of life and property caused by fires resulting from smoking is incalculable. my country has a large smoking population, and smoking in places such as transportation vehicles, gas stations, airports, offices, chemical plants, and forests can easily cause fires, resulting in significant economic losses and casualties. Therefore, controlling smoking in public places and in safe production is an essential safety measure.

[0003] Initially, smoking detection relied primarily on manual review of past surveillance videos, which was prone to errors due to user fatigue. Detecting smoking through sensors significantly improves the intelligence of the detection process, reducing wasted manpower and time, and increasing efficiency and quality. Current detection methods include smoke sensors and image processing. With the popularization and development of artificial intelligence, video surveillance for smoking detection is gradually being applied. However, because most current smoking detection algorithms focus on detecting individual smoking actions or smoke characteristics, these methods are prone to misjudgments and missed detections when the environment changes. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a smoking behavior detection method under video surveillance. Addressing the issues of false detection and missed detection of smoke and cigarettes in complex environments, this invention achieves smoking behavior detection by combining cigarette and smoke features. To address the low accuracy of small smoke detection in traditional smoke detection methods, this invention completes a smoke detection technique based on feature fusion. To achieve real-time detection in surveillance, this invention designs a lightweight cigarette detection model based on depthwise separable convolution and improved YOLOv5. Experimental results show that the smoking behavior detection method provided by this invention has high accuracy and robustness, can effectively distinguish between smoking and non-smoking behaviors, and maintains good detection performance even under complex environments with varying lighting, shadows, and occlusion.

[0005] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0006] Step 1: Read the surveillance video, segment the video sequence, and obtain the image dataset;

[0007] Step 2: Detect the smoke region by motion, that is, use the frame difference method to extract the moving foreground image and determine the region in the foreground that may be smoke; filter the foreground image to exclude non-smoke regions; perform cross-block processing on the final smoke region to extract multiple smoke images of the same size.

[0008] Step 3: Extract local texture features and grayscale histogram features from the smoke image obtained in Step 2. Then, normalize these two features and add them together to obtain a fused feature map. Use the feature map to train a support vector machine classifier. Pass the fused feature map to be detected through the trained support vector machine classifier to achieve smoke detection.

[0009] Step 4: After detecting smoke, label the cigarettes in all images of the image dataset; unify the resolution of all images through adaptive scaling and padding, and perform adaptive anchor box calculation and image feature enhancement;

[0010] Step 5: Construct a cigarette detection model. The backbone network of the cigarette detection model retains the YOLOv5 Focus structure, and replaces the original YOLOv5 CSPBlock structure with a residual structure that integrates a compressed attention mechanism. The residual structure that integrates the compressed attention mechanism consists of two branches. The first branch is the backbone of this structure, which integrates an attention module on top of a convolutional neural network. The second branch connects the input to the output of the backbone through a skip connection and adds it directly to the result of the first branch, thus obtaining the output of the residual structure that integrates the compressed attention mechanism. Depthwise separable convolutions are used instead of convolutions in the cigarette detection model to achieve model lightweighting.

[0011] Step 6: Adjust the learning rate, batch size, number of iterations, convolution kernel size, and network optimization function parameters of the cigarette detection model to improve the model's learning performance and effectiveness;

[0012] Step 7: Through training in Step 6, a lightweight cigarette detection model is obtained. When an image to be detected is input, the model is used to detect whether a person is holding a cigarette.

[0013] Step 8: Combine the results of cigarette detection and smoke detection to determine smoking behavior. If both cigarette and smoke are detected at the same time, it is determined to be smoking behavior; otherwise, it is not smoking behavior.

[0014] Preferably, step 2 specifically comprises:

[0015] The frame difference method is used to detect moving targets in an image. Specifically, the possible range of smoke is determined by comparing the current frame with past frames. The difference between the current frame and the past 10 frames is used to extract a background image that may contain smoke outlines.

[0016] The formula for calculating the frame difference is as follows:

[0017]

[0018] Among them I t This is the current image, I t-10 It is the image of the past 10 frames, δ is the grayscale threshold, D is the newly generated image, and m and n represent the position coordinates of each pixel in the image;

[0019] After obtaining the foreground image that may contain smoke outlines, the foreground image is filtered to exclude non-smoke areas; then, multiple rectangles of the same size are used to cross-divide the smoke area of ​​the image, and the size of the smoke image after segmentation is adjusted to 64×64 by scaling.

[0020] Preferably, step 3 specifically comprises:

[0021] Step 3-1: Extract local texture features:

[0022] Step 3-1-1: Divide the smoke image into 64 8×8 image sub-blocks and calculate the LBP value of each pixel in each sub-block;

[0023] Step 3-1-2: Perform histogram statistics for each sub-block;

[0024] Step 3-1-3: Normalize the histograms of all image sub-blocks;

[0025] Step 3-1-4: Connect the normalized histograms of all sub-blocks to obtain the texture features of the smoke image;

[0026] Step 3-2: Calculate the gradient histogram features of the smoke image:

[0027] Step 3-2-1: Perform Gamma normalization on the smoke image;

[0028] Step 3-2-2: Calculate the gradient of the smoke image in the horizontal and vertical directions, and then obtain the gradient of each pixel;

[0029] Step 3-2-3: Divide the smoke image into several 8×8 image sub-blocks, divide each sub-block into 9 directions according to the average pixel gradient, and calculate the gradient histogram of each sub-block;

[0030] Step 3-2-3: Concatenate the gradient histogram features of adjacent 2×2 sub-blocks to obtain the gradient histogram features;

[0031] Step 3-3: Normalize and add the local texture features and gradient histogram features to obtain the final fused feature map; then expand the feature map column by column to form a 64-dimensional feature vector.

[0032] Steps 3-4: Construct a support vector machine classifier, using a multinomial kernel function, and train the SVM classifier using 64-dimensional feature vectors;

[0033] Steps 3-5: After training, the fused feature map to be detected is passed through the trained support vector machine classifier to achieve smoke detection.

[0034] Preferably, the support vector machine classifier has a penalty coefficient c = 1.0 for incorrect terms and a kernel function coefficient g = 0.5 during training, with an error precision of 10 at which training stops. -3 .

[0035] Preferably, step 5 specifically comprises:

[0036] After inputting an image, the backbone network of the cigarette detection model first performs intermittent downsampling on the image and then stitches and reassembles the sampled images, transforming the original RGB three-channel image into 12 channels. It then passes through a 32-channel convolutional layer, outputting a 32-feature layer. Next, it uses a residual structure with a fused compression-enhanced attention mechanism. The backbone of this structure passes through a 1×1 convolutional layer and a K×K depthwise separable convolutional layer, with each convolutional layer followed by a batch normalization layer to accelerate model convergence. Both activation functions use Swish. Following this is an attention module, which... The input is subjected to global max pooling on the feature map, followed by two fully connected layers. The first fully connected layer uses the ReLU activation function, and the second fully connected layer uses the Sigmoid activation function. The output of this part is multiplied by the corresponding element of the feature map to obtain the output of the attention module. After that, there is a 1×1 convolutional layer, followed by a regularization layer to improve the generalization ability of the model. Finally, the input is connected to the output of the backbone part through a skip connection to obtain the output of the backbone network. The neck and prediction output parts adopt the Yolov5 structure, and depthwise separable convolutions are used in all parts of the network instead of ordinary convolutions to achieve model lightweighting.

[0037] The beneficial effects of this invention are as follows:

[0038] The feature fusion-based smoke detection technology proposed in this invention achieves significant improvements in small smoke detection, enhancing both accuracy and reliability. Furthermore, the lightweight cigarette detection model designed in this invention, based on depthwise separable convolution and improved YOLOv5, maintains high accuracy while exhibiting low computational complexity and fast detection speed, enabling real-time detection. Therefore, the smoking behavior detection method under video surveillance provided by this invention has high practical value and promising application prospects. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the residual structure of the fusion compression-induced attention mechanism of the present invention.

[0040] Figure 2 This is a schematic diagram of the attention mechanism used in this invention.

[0041] Figure 3 This is a flowchart of the smoke detection method of the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] To address the low efficiency of smoking behavior detection in manual spot checks of video surveillance, this invention proposes a smoking behavior detection technology under surveillance video. To address the issues of false positives and false negatives in complex environments, this invention combines cigarette characteristics and smoke characteristics to achieve smoking behavior detection. To address the low accuracy of small smoke detection in traditional smoke detection methods, this invention proposes a smoke detection technology based on feature fusion. For real-time detection in surveillance, this invention proposes a lightweight cigarette detection model based on depthwise separable convolution and improved YOLOv5.

[0044] To address the aforementioned problems and technical requirements, this invention proposes a smoking behavior detection technology under video surveillance, combining cigarette characteristics and smoke characteristics. The technical solution of this invention is as follows:

[0045] like Figure 3 As shown, for smoke detection, this invention constructs a smoke detection model that fuses local texture information with grayscale histograms, including the following steps:

[0046] 1. Read surveillance video, segment video sequences, and collect smoke dataset.

[0047] 2. Detect smoke regions using motion analysis, specifically by extracting the moving foreground image using frame differencing to identify potential smoke areas within the foreground. The foreground image is then filtered to exclude non-smoke regions. Finally, the final smoke region is cross-segmented to extract multiple smaller, identically sized smoke image blocks.

[0048] 3. Extract local texture features and grayscale histogram features from the smoke image in step 2. Then, normalize these two features and add them together to obtain a fused feature map. Finally, use the feature map to train a support vector machine classifier. Pass the fused feature map to be detected through the classifier to achieve smoke detection.

[0049] For cigarette detection, this invention constructs a cigarette detection model based on a depthwise separable convolutional fusion attention mechanism, the specific steps of which are as follows:

[0050] 4. Label the cigarettes in all images of the dataset. At the input end, the images are processed through adaptive scaling, padding, and other methods to unify the resolution of all images, and adaptive anchor box calculation and image feature enhancement are performed.

[0051] 5. The model retains the original Focus structure from YOLOv5 in the backbone network part, using... Figure 1 The residual structure shown replaces the original CSPBlock structure with a fusion compression-enhanced attention mechanism. This structure consists of two branches: the first branch is the backbone, incorporating an attention module on top of the convolutional neural network; the second branch is directly added to the first branch to obtain the output of the structure. Furthermore, depthwise separable convolutions are used instead of ordinary convolutions in the improved model, achieving a lightweight design.

[0052] 6. Adjust parameters such as the learning rate, batch size, number of iterations, convolutional kernel size, and network optimization function of the model to improve the learning performance and effectiveness of the model.

[0053] 7. Through the training in step 6, a lightweight cigarette detection model is obtained. When an image to be detected is input, the model is used to detect whether the person is holding a cigarette.

[0054] Step 5 also includes:

[0055] like Figure 2 As shown, the residual structure backbone used to improve the fusion compression-induced attention mechanism of the backbone network consists of two 1×1 convolutional layers, one k×k depthwise separable convolutional layer, and an attention module. The k×k depthwise separable convolutional layer separates the input feature channels, and then stacks the convolution results of all channels layer by layer to obtain the output, where k is the size of the convolutional kernel. After each convolutional layer, batch normalization is performed, and finally, an activation function is applied. Finally, the input is added to the backbone through skip connections to obtain the output of the structure. Specific implementation examples:

[0057] To more accurately detect smoking behavior, this invention combines smoke characteristics and cigarette characteristics to improve detection accuracy. The specific implementation scheme for smoke detection is as follows:

[0058] Since the algorithm of this invention is applied to a camera, and the captured video footage is relatively static, this invention utilizes the frame difference method to detect moving targets within a location. Specifically, it determines the approximate extent of smoke by comparing the current frame with past frames. In this invention, the difference between the current frame and the past 10 frames is used to extract the outline that may be smoke.

[0059] The formula for calculating the frame difference method is as follows:

[0060]

[0061] Among them I t This is the current image, I t-10 It represents the images from the past 10 frames, δ is the grayscale threshold which can be adjusted according to the actual scene, and D is the newly generated image.

[0062] After obtaining the background image containing the possible smoke outlines, the background image is filtered to obtain a more accurate smoke outline image. However, smoke outlines are often small and irregular. In order to enhance the perception of small targets, this invention uses multiple rectangular boxes of the same size and small area to cross-divide the smoke area of ​​the image, and the size of the divided image is adjusted to 64×64 by scaling.

[0063] Next, local texture features are extracted from the smoke image of small targets. The specific process is as follows:

[0064] (1) Divide the image into 64 8×8 image sub-blocks and calculate the LBP value of each pixel in each sub-block;

[0065] (2) Perform histogram statistics for each sub-block;

[0066] (3) Normalize the histograms of all image sub-blocks;

[0067] (4) Connecting the normalized histograms of all sub-blocks yields the texture features of the entire image.

[0068] The gradient histogram features of the smoke image of small targets are calculated, and the specific process is as follows:

[0069] (1) Perform Gamma normalization on the image;

[0070] (2) Calculate the gradient of the image in the horizontal and vertical directions, and then obtain the gradient of each pixel;

[0071] (3) Divide the image into several 8×8 image sub-blocks, and divide each sub-block into equal parts according to the pixel gradient.

[0072] The gradient histogram of each sub-block is calculated in 9 directions;

[0073] (4) Connect the gradient histogram features of adjacent 2×2 sub-blocks to obtain gradient histogram features.

[0074] Finally, the feature maps from the two feature extractions are normalized and summed to obtain the final fused feature map. The feature map is then expanded column-wise to form a 64-dimensional feature vector. An SVM classifier is constructed using a multinomial kernel function, with a penalty coefficient c = 1.0 for error terms, a kernel coefficient g = 0.5, and a stop-training error precision of 10.-3 By collecting a large dataset of smoke features, an SVM classifier was trained using 64-dimensional feature vectors.

[0075] When the surveillance video sequence is read, the detection output of the smoke area is obtained according to the above method.

[0076] To more accurately detect smoking behavior, this invention further detects the cigarette itself, thereby improving the detection accuracy. The specific steps are as follows:

[0077] Cigarettes are labeled in all images of the dataset. At the input end, the images are processed through adaptive scaling and padding to unify the resolution of all images, and adaptive anchor box calculation and image feature enhancement are performed.

[0078] After obtaining the input image, the backbone network of the cigarette detection model first performs interval downsampling on the image and then stitches and reassembles the sampled images, changing the original RGB three channels into 12 channels. It then passes the image through a 32-channel convolutional layer, outputting a 32-feature layer. Following this, it passes through... Figure 1 The residual structure shown employs a fusion compression-driven attention mechanism. The backbone of this structure sequentially passes through a 1×1 convolutional layer and a K×K depthwise separable convolutional layer, followed by a batch normalization layer after each convolutional layer to accelerate model convergence. Both activation functions use Swish. Next is an attention module that performs global max pooling on the input feature map, passing it through two fully connected layers: layer 1 uses ReLU activation, and layer 2 uses Sigmoid activation. The output of this module is multiplied by the corresponding element of the feature map to obtain the output of the attention module. Following this is another 1×1 convolutional layer, followed by a regularization layer to improve the model's generalization ability. Finally, the input is skipped to the output of the backbone, resulting in the backbone network output. The neck and prediction output sections adopt the YOLOv5 architecture, and depthwise separable convolutions are used instead of ordinary convolutions in all parts of the network to achieve model lightweighting.

[0079] Adjusting parameters such as the learning rate, batch size, number of iterations, kernel size, and network optimization function improves the model's learning performance and effectiveness. During training, 200 epochs are selected, with a batch size of 32, and training is performed on a single GPU. The Adam optimizer is used for optimization during training, with a learning rate of 0.01 for the first 10 epochs, 0.001 for 10-50 epochs, and 0.001 for 50-200 epochs. After training, the detected video is input into the network to obtain the cigarette detection results.

[0080] Ultimately, the smoking behavior is determined by combining the results of cigarette detection and smoke detection. If both cigarette and smoke are detected, it is determined to be smoking behavior; otherwise, it is not smoking behavior.

Claims

1. A method for detecting smoking behavior under video surveillance, characterized in that, Includes the following steps: Step 1: Read the surveillance video, segment the video sequence, and obtain the image dataset; Step 2: Detect the smoke region by motion, that is, use the frame difference method to extract the moving foreground image and determine the region in the foreground that may be smoke; filter the foreground image to exclude non-smoke regions; perform cross-block processing on the final smoke region to extract multiple smoke images of the same size. Step 3: Extract local texture features and grayscale histogram features from the smoke image obtained in Step 2. Then, normalize these two features and add them together to obtain a fused feature map. Use the feature map to train a support vector machine classifier. Pass the fused feature map to be detected through the trained support vector machine classifier to achieve smoke detection. Step 4: After detecting smoke, label the cigarettes in all images of the image dataset; unify the resolution of all images through adaptive scaling and padding, and perform adaptive anchor box calculation and image feature enhancement; Step 5: Construct a cigarette detection model. The backbone network of the cigarette detection model retains the Focus structure of YOLOv5, and replaces the original CSPBlock structure of YOLOv5 with a residual structure that integrates the compressed attention mechanism. The residual structure that integrates the compressed attention mechanism consists of two branches. The first branch is the backbone of the structure, which integrates an attention module on the basis of a convolutional neural network. The second branch connects the input to the output of the backbone through a jump connection and adds it directly to the result of the first branch, thus obtaining the output of the residual structure that integrates the compressed attention mechanism. In the cigarette detection model, depthwise separable convolution is used instead of convolution to achieve a lightweight model; Step 6: Adjust the learning rate, batch size, number of iterations, convolution kernel size, and network optimization function parameters of the cigarette detection model to improve the model's learning performance and effectiveness; Step 7: Through training in Step 6, a lightweight cigarette detection model is obtained. When an image to be detected is input, the model is used to detect whether a person is holding a cigarette. Step 8: Combine the results of cigarette detection and smoke detection to determine smoking behavior. If both cigarette and smoke are detected at the same time, it is determined to be smoking behavior; otherwise, it is not smoking behavior.

2. The method for detecting smoking behavior under video surveillance according to claim 1, characterized in that, Step 2 specifically involves: The frame difference method is used to detect moving targets in an image. Specifically, the possible range of smoke is determined by comparing the current frame with past frames. The difference between the current frame and the past 10 frames is used to extract a background image that may contain smoke outlines. The formula for calculating the frame difference is as follows: Among them I t This is the current image, I t-10 It is the image of the past 10 frames, δ is the grayscale threshold, D is the newly generated image, and m and n represent the position coordinates of each pixel in the image; After obtaining the foreground image that may contain smoke outlines, the foreground image is filtered to exclude non-smoke areas; then, multiple rectangles of the same size are used to cross-divide the smoke area of ​​the image, and the size of the smoke image after segmentation is adjusted to 64×64 by scaling.

3. The method for detecting smoking behavior under video surveillance according to claim 2, characterized in that, Step 3 specifically involves: Step 3-1: Extract local texture features: Step 3-1-1: Divide the smoke image into 64 8×8 image sub-blocks and calculate the LBP value of each pixel in each sub-block; Step 3-1-2: Perform histogram statistics for each sub-block; Step 3-1-3: Normalize the histograms of all image sub-blocks; Step 3-1-4: Connect the normalized histograms of all sub-blocks to obtain the texture features of the smoke image; Step 3-2: Calculate the gradient histogram features of the smoke image: Step 3-2-1: Perform Gamma normalization on the smoke image; Step 3-2-2: Calculate the gradient of the smoke image in the horizontal and vertical directions, and then obtain the gradient of each pixel; Step 3-2-3: Divide the smoke image into several 8×8 image sub-blocks, divide each sub-block into 9 directions according to the average pixel gradient, and calculate the gradient histogram of each sub-block; Step 3-2-3: Concatenate the gradient histogram features of adjacent 2×2 sub-blocks to obtain the gradient histogram features; Step 3-3: Normalize and add the local texture features and gradient histogram features to obtain the final fused feature map; then expand the feature map column by column to form a 64-dimensional feature vector. Steps 3-4: Construct a support vector machine classifier, using a multinomial kernel function, and train the SVM classifier using 64-dimensional feature vectors; Steps 3-5: After training, the fused feature map to be detected is passed through the trained support vector machine classifier to achieve smoke detection.

4. The method for detecting smoking behavior under video surveillance according to claim 3, characterized in that, The support vector machine classifier has a penalty coefficient c = 1.0 and a kernel function coefficient g = 0.5 for incorrect terms during training, and the error precision at which training stops is 10. -3 .

5. The method for detecting smoking behavior under video surveillance according to claim 3, characterized in that, Step 5 specifically involves: After inputting the image, the backbone network of the cigarette detection model first performs downsampling of the image at intervals, and then stitches and reassembles the sampled images, so that the image changes from the original RGB three channels to 12 channels. Then, it passes through a convolutional layer with 32 channels to output a 32-feature layer. Next, a residual structure that compresses and stimulates attention is fused. The backbone of this structure passes through a 1×1 convolutional layer and a K×K depth-separable convolutional layer. Each convolutional layer is followed by a batch normalization layer to accelerate the convergence speed of the model. Both activation functions are Swish. Next is an attention module, which performs global max pooling on the input feature map, passing it through two fully connected layers. The first fully connected layer uses the ReLU activation function, and the second fully connected layer uses the Sigmoid activation function. The output of this part is multiplied by the corresponding element of the feature map to obtain the output of the attention module. This is followed by a 1×1 convolutional layer, and then a regularization layer to improve the model's generalization ability. Finally, the input is connected to the output of the backbone part through a skip connection to obtain the output of the backbone network. The neck and prediction output parts adopt the Yolov5 structure, and depthwise separable convolutions are used in all parts of the network instead of ordinary convolutions to achieve lightweight model.

Images