A method and system for detecting smoking behavior in an industrial scene monitoring video

Abstract

This invention discloses a method and system for detecting smoking behavior in industrial scene surveillance videos. It employs a combination of metric learning and temperature detection, using metric learning to extract features and calculate distances for target classification. A TransReID neural network is used as the feature extraction network to extract features combining smoking actions and cigarettes. A self-attention mechanism is employed for better feature extraction and accurate identification of smoking behavior, overcoming the shortcomings of traditional smoking detection methods. By establishing a template library, feature maps of the worker image frames to be detected are matched with feature maps in the library, improving the accuracy of smoking action judgment.

Description

Technical Field

[0001] This invention belongs to the field of target detection, and more specifically, relates to a method and system for detecting smoking behavior in industrial scene surveillance videos. Background Technology

[0002] Smoking is harmful to health, pollutes the air, and causes passive smoking for others. In industrial settings, smoking or carelessly discarding cigarette butts poses significant safety hazards, easily leading to accidents such as fires. However, current production practices primarily rely on manual inspections for smoking detection, which is resource-intensive, inefficient, and unable to promptly identify safety hazards. With the rapid development of artificial intelligence, more and more deep learning algorithms are being applied to smoking detection; however, due to low detection accuracy, their application is not yet widespread.

[0003] Machine vision inspection technology, as an important technology in modern industry, is widely used in various fields. With technological advancements, existing smoke detection algorithms include image-based smoke detection using various target detection algorithms such as SSD, Faster R-CNN, and YOLO, which issue smoking alerts upon detecting lit cigarettes. However, common detection schemes cannot meet real-time requirements and consume significant resources. Furthermore, in complex industrial scenarios, due to interference from complex backgrounds and the fact that cigarettes are typically small targets in surveillance videos, traditional detection algorithms are prone to false detections or missed detections of cigarettes. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method and system for detecting smoking behavior in industrial scene monitoring videos, thereby solving the technical problem that the detection accuracy of the existing detection methods needs to be improved.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a method for detecting smoking behavior in industrial scene surveillance videos is provided, comprising:

[0006] S1, a TransReID network is trained using a worker image dataset; wherein, the worker image dataset includes images of workers' heads, hands and whole bodies, which are obtained by performing target detection on each image frame in an industrial scene monitoring video; after labeling whether the worker image dataset is being smoked or not, it is input into the trained TransReID network to obtain the corresponding feature maps, which are used as the feature library;

[0007] S2, target detection is performed on the worker image frame to be detected to obtain the head, hand and whole body images of the target worker, and these images are input into the trained TransReID network to obtain the head, hand and whole body feature maps of the worker to be classified.

[0008] S3, determine the probability that a lit cigarette exists in the worker image frame region to be detected. ;in, The first image of the target worker's head, hands, and whole body. i The probability that the temperature value of each pixel corresponds to the temperature at which a cigarette is lit can be calculated using Bayes' theorem. = , The temperature measurement result for lighting a cigarette was as follows: The probability is obtained from the temperature distribution function of igniting a cigarette. The probability of a cigarette being lit. Temperature during temperature measurement The probability of its occurrence;

[0009] The top M head, hand, and body feature images of the worker to be classified are determined from the feature library that are closest to the head, hand, and body feature images of the worker to be classified. The number of head, hand, and body feature images with smoking marks is divided by M to obtain the probabilities p2, p3, and p4 of smoking behavior in the head, hand, and body regions of the worker image frame to be detected.

[0010] S4, if a*p1+b*p2+c*p3+d*p4 is greater than the threshold, then the worker image frame to be detected is considered to have smoking behavior; otherwise, it is not; where a,b,c,d are weighting coefficients.

[0011] According to a second aspect of the present invention, a smoking behavior detection system in industrial scene surveillance video is provided, comprising: a computer-readable storage medium and a processor;

[0012] The computer-readable storage medium is used to store executable instructions;

[0013] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect.

[0014] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0015] 1. The smoking behavior detection method in industrial scene monitoring videos provided by this invention adopts a combination of metric learning and temperature detection. It uses metric learning to extract features and calculate distance to obtain target classification. It uses a TransReID neural network as the feature extraction network. Compared with the traditional detection methods that focus on extracting cigarette features and thus have difficulty in detecting small targets, this invention focuses on extracting the relative relationship between cigarettes and people, people's hands, people's heads and people's posture and movements, etc., thus overcoming the problem of difficulty in detecting small targets and enriching the detection range. At the same time, it uses a self-attention mechanism, which can better extract features and accurately identify smoking behavior, thus overcoming the defects of traditional smoking detection methods.

[0016] 2. The smoking behavior detection method in industrial scene monitoring videos provided by this invention uses a template library (i.e., feature library) to perform template matching between the feature maps of the worker image frames to be detected and the feature maps in the template library. The advantages of this approach are: First, using a template library improves the accuracy of judging smoking actions; second, it enhances the interpretability of the algorithm, avoiding the shortcomings of traditional neural network methods that cannot be interpreted, while increasing reliability; third, using a template library makes the detection method more flexible, facilitating algorithm updates, iterations, and optimizations. Users can update the template library themselves according to specific operating conditions, thereby quickly and cost-effectively updating the algorithm without the involvement of developers.

[0017] 3. The smoking behavior detection method in industrial scene monitoring video provided by the present invention can overcome the shortcomings of traditional methods that use temperature thresholds and smoke to detect smoking behavior, which are easily confused with other objects in the background. Especially in industrial scenes, traditional methods will produce a large number of false alarms and false alarms due to interference from high-temperature equipment, normal smoke emissions, etc., while the present invention can effectively eliminate such interference.

[0018] 4. The method for detecting smoking behavior in industrial scene monitoring videos provided by this invention has a fast detection speed and can achieve real-time detection and real-time alarm effects, enabling timely detection of smoking behavior and maximizing the safety of the factory area.

[0019] 5. The smoking behavior detection method in industrial scene monitoring videos provided by this invention does not require the addition of additional hardware equipment and can be implemented directly using a large number of existing monitoring devices in the factory area, which is cost-effective; it is convenient and flexible to deploy and can be flexibly deployed on cameras or computing servers with computing capabilities; it is robust, and even if a certain error occurs in some part of the detection during actual use, it will not affect the accuracy of the final result. Attached Figure Description

[0020] Figure 1 This is one of the flowcharts of the method for detecting smoking behavior in industrial scene monitoring videos provided by the present invention;

[0021] Figure 2 This is a schematic diagram of the smoking behavior detection and alarm process in industrial scene monitoring video provided by the present invention;

[0022] Figure 3 The second flowchart of the method for detecting smoking behavior in industrial scene monitoring videos provided by the present invention;

[0023] Figure 4 This is a diagram of the vit_jpm structure in TransReID used in this invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] Existing methods for detecting smoking have the following four drawbacks:

[0026] 1. Since cigarettes are usually very small targets in surveillance videos, using traditional target detection methods to detect cigarettes will result in a high false detection rate and a high false negative rate.

[0027] 2. Smoking detection methods based on smoking posture and gestures are easily confused with other postures, which can lead to false detections.

[0028] 3. For the method of using a dual-spectrum temperature measurement camera to determine the temperature of a cigarette butt, in industrial scenarios, there may be devices in the background with temperatures similar to those of the cigarette butt, which may cause interference and false alarms.

[0029] 4. When traditional machine learning methods for smoke detection are applied to different scenarios, the model usually needs to be retrained to ensure the effect, which increases the cost.

[0030] Based on this, the present invention employs a method combining metric learning and temperature detection to detect smoking behavior. An embodiment of the present invention provides a method for detecting smoking behavior in industrial scene surveillance videos, such as... Figure 1 As shown, it includes:

[0031] S1, a TransReID network is trained using a worker image dataset; wherein, the worker image dataset includes images of workers' heads, hands and bodies, which are obtained by detecting head, hand and body targets in each image frame of an industrial scene monitoring video; after labeling whether the worker image dataset is being smoked or not, it is input into the trained TransReID network to obtain the corresponding feature maps, which are used as the feature library;

[0032] S2, target detection is performed on the worker image frame to be detected to obtain the head, hand and whole body images of the target worker, and these images are input into the trained TransReID network to obtain the head, hand and whole body feature maps of the worker to be classified.

[0033] S3, determine the probability that a lit cigarette exists in the worker image frame region to be detected. ;in, The first image of the target worker's head, hands, and whole body. i The probability that the temperature value of each pixel corresponds to the temperature at which a cigarette is lit can be calculated using Bayes' theorem. = , The temperature measurement result for lighting a cigarette was as follows: The probability is obtained from the temperature distribution function of igniting a cigarette. The probability of a cigarette being lit. Temperature during temperature measurement The probability of its occurrence;

[0034] The top M head, hand, and body feature images of the worker to be classified are determined from the feature library that are closest to the head, hand, and body feature images of the worker to be classified. The number of head, hand, and body feature images with smoking marks is divided by M to obtain the probabilities p2, p3, and p4 of smoking behavior in the head, hand, and body regions of the worker image frame to be detected.

[0035] S4, if a*p1+b*p2+c*p3+d*p4 is greater than the threshold, then the worker image frame to be detected is considered to have smoking behavior; otherwise, it is not; where a,b,c,d are weighting coefficients.

[0036] Preferably, the temperature data of the lit cigarette is collected by a temperature-measuring camera and statistically analyzed. and , The probability of a cigarette being lit (i.e., the ratio of the number of temperature points in the image frame captured by the temperature measuring camera that correspond to the number of lit cigarettes to the total number of temperature points in the image frame). Temperature during temperature measurement The probability of occurrence (i.e., the temperature value) The temperature distribution function of a lit cigarette is obtained by fitting the temperature data (the ratio of the number of points to the total number of temperature points in the image frame).

[0037] For example: 1,000,000 temperature points are collected during data acquisition, of which 500 temperature points are from lit cigarettes, and there are 800 temperature points at 100 degrees Celsius. Therefore, p(c) = 500 / 1,000,000, p( =100) = 800 / 1000000.

[0038] Preferably, the temperature distribution function of the lit cigarette is a normal distribution function.

[0039] Preferably, the distance is a Euclidean distance or a cosine distance.

[0040] Preferably, the target detection is implemented based on a pre-trained YOLOv5 neural network; the training samples are image frames containing workers, and the labels are the coordinates of the worker's head, hands, and whole body in the image frames.

[0041] Preferably, the method further includes: correcting the annotations of the image frames with detection errors, and adding the worker's head, hand, and whole-body feature maps obtained by inputting them into the trained TransReID model after target detection to the feature library.

[0042] The following is combined with Figure 2-3 The method provided by this invention will be described.

[0043] like Figure 2 As shown in S101, the first step is to collect and label the worker dataset from the surveillance video of the industrial scene, labeling the worker's head, hands, and whole body. When collecting data, it is necessary to collect camera data from different angles and locations.

[0044] The preset neural network is trained using a worker dataset until the model converges, resulting in a target detection neural network for detecting workers' heads, hands, and whole bodies. In the specific implementation, the YOLOv5m neural network is used, with 100 training generations.

[0045] Data sets of industrial scene monitoring videos, including both smoking and non-smoking workers, are collected. The aforementioned neural network is used to automatically detect and crop images of the worker's head, hands, and whole body. Based on whether it can be determined from the image whether the worker is smoking, the image is labeled with the corresponding category. Specifically, if the image shows a worker with a cigarette in their mouth, holding a cigarette and smoking, or blowing smoke rings, it can be determined as smoking behavior and the image is labeled as smoking; otherwise, it is labeled as not smoking. Through the above steps, labeled data of smoking and non-smoking areas of the worker's head, hands, and whole body can be obtained.

[0046] The TransReID network for deep feature extraction was trained using images cropped from the aforementioned dataset, which included both smoking and non-smoking workers. Figure 4 As shown, the trained neural network is used to extract the depth feature map of the image, and the above feature map is classified according to the category label to obtain the template library (i.e., feature library) of smoking and non-smoking areas of the worker's head, worker's hands and worker's whole body.

[0047] Temperature datasets of cigarette butts are collected from industrial scene monitoring videos using temperature-measuring cameras. The temperature of the lit cigarette butts is labeled and recorded. The temperature data of lit cigarettes from the temperature-measuring cameras are statistically analyzed, and a temperature distribution function of lit cigarettes is fitted based on the data and statistical knowledge. The most suitable distribution should be selected for fitting based on the specific characteristics of the statistically obtained data. Specifically, a normal distribution function is used for fitting, and the mean and variance of the normal distribution function are obtained using the least squares method.

[0048] In the application phase, as shown in S102, video frames (i.e., worker image frames to be detected) are pulled from the on-site monitoring video stream. The aforementioned target detection neural network is used to obtain the worker's head, worker's hands, and worker's whole body targets, and the images of the corresponding areas (i.e., worker's head, worker's hands, and worker's whole body areas) are cropped.

[0049] Temperature data for all areas of the worker's head, hands, and body ( Predict using the distribution function fitted in step S101. That is, the first image of the target worker's head, hands, and whole body. i Each temperature value corresponds to the probability of a cigarette being lit; the probability of a cigarette being lit exists in this region. This method of judgment can remove interference from background objects with a temperature similar to that of cigarettes.

[0050] For example, in the image frames acquired by the temperature measuring camera, the worker's head image has 1,000 pixels, the worker's hand image has 2,000 pixels, and the worker's full-body image has 5,000 pixels. Each pixel corresponds to one temperature value and one probability. The value with the highest probability is taken as p1.

[0051] S103, a deep feature extraction network is used to extract depth features from the worker's head, hands, and whole body, and these features are matched with data in the template library. Specifically, the Euclidean distance between the current region's depth features and the corresponding features in the template library is calculated, and then the distances are sorted in ascending order. The N closest templates are selected, and the number of templates labeled "smoking" is counted. The probability that smoking occurs in this area is . Based on the matching results, the probabilities of smoking behavior in these three areas are p2, p3, and p4, respectively. Template matching can improve the accuracy of judging smoking behavior.

[0052] S104, set the threshold for judging smoking behavior, in the specific implementation the threshold value is 0.7; set the counter for smoking behavior.

[0053] For the current video frame, if a*p1+b*p2+c*p3+d*p4>threshold, then smoking behavior is considered to exist, and the smoking behavior counter is incremented by 1. The values ​​of a, b, c, and d can be flexibly set according to different scenarios.

[0054] S105: Based on the smoking behavior counter, a smoking alarm is issued when the number of smoking behaviors detected within a certain period exceeds a certain number. This step helps reduce false alarms. In the specific implementation, an alarm is issued when the number of smoking behaviors exceeds 60 within 5 seconds in a 25-frame camera.

[0055] S106 During operation, the detection results are checked manually at irregular intervals. When false alarms or missed alarms occur, the video frames with detection errors are used to form new templates and added to the template library according to the above method (regional cropping, network training, and extraction of deep features). This method allows the algorithm to iterate quickly and adapt to different environments at minimal cost.

[0056] This invention provides a system for detecting smoking behavior in industrial scene surveillance videos, comprising: a computer-readable storage medium and a processor;

[0057] The computer-readable storage medium is used to store executable instructions;

[0058] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0059] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting smoking behavior in industrial scene surveillance videos, characterized in that, include: S1, a TransReID network is trained using a worker image dataset; wherein, the worker image dataset includes images of workers' heads, hands and whole bodies, which are obtained by performing target detection on each image frame in an industrial scene monitoring video; after labeling whether the worker image dataset is being smoked or not, it is input into the trained TransReID network to obtain the corresponding feature maps, which are used as the feature library; S2, target detection is performed on the worker image frame to be detected to obtain the head, hand and whole body images of the target worker, and these images are input into the trained TransReID network to obtain the head, hand and whole body feature maps of the worker to be classified. S3, determine the probability that a lit cigarette exists in the worker image frame region to be detected. ;in, The first image of the target worker's head, hands, and whole body. i The temperature value of each pixel corresponds to the probability of the cigarette being lit. = , The temperature measurement result for lighting a cigarette was as follows: The probability, The probability of a cigarette being lit. Temperature during temperature measurement The probability of its occurrence; The top M head, hand, and body feature images of the worker to be classified are determined from the feature library that are closest to the head, hand, and body feature images of the worker to be classified. The number of head, hand, and body feature images with smoking marks is divided by M to obtain the probabilities p2, p3, and p4 of smoking behavior in the head, hand, and body regions of the worker image frame to be detected. S4, if a*p1+b*p2+c*p3+d*p4 is greater than the threshold, then the worker image frame to be detected is considered to have smoking behavior; otherwise, it is not; where a,b,c,d are weighting coefficients.

2. The method as described in claim 1, characterized in that, Temperature data of a lit cigarette is collected using a temperature-measuring camera, and the resulting data is statistically analyzed. and The temperature distribution function of a lit cigarette is obtained by fitting the temperature data.

3. The method as described in claim 2, characterized in that, The temperature distribution function of the lit cigarette is a normal distribution function.

4. The method as described in claim 1, characterized in that, The distance is either Euclidean or cosine distance.

5. The method as described in claim 1, characterized in that, The target detection is based on a pre-trained YOLOv5 neural network; the training samples are image frames containing workers, and the labels are the coordinates of the worker's head, hands and whole body in the image frame.

6. The method as described in claim 1, characterized in that, The method further includes: correcting the annotations of the image frames with detection errors, and adding the worker's head, hand and whole body feature maps obtained by inputting them into the trained TransReID model after target detection to the feature library.

7. A smoking behavior detection system in industrial scene surveillance video, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any one of claims 1-6.

Images