Real-time video smoking detection method, device, equipment and medium

Abstract

The invention discloses a real-time video smoking detection method, device and equipment and a medium, and relates to the technical field of image processing, and the method comprises the steps: through the cascade design of a YOLOv8 personnel detection model and a smoking detection model, firstly detecting personnel in a scene, and then carrying out cigarette and smoking action detection on a cut personnel image, thereby reducing background interference, and improving the detection efficiency. And the smoking detection precision is improved. Meanwhile, statistical decision making is performed on smoking detection results of video stream continuous frames, video stream continuous frame information is comprehensively utilized, and false detection caused by accidental hand raising and other actions of personnel and suspected cigarette objects is effectively reduced. Besides, when target labeling is carried out, two kinds of labels of the cigarette target and smoking actions (holding the cigarette by hand and holding the cigarette in the mouth) are labeled, missing detection and false detection caused by the fact that the size of the cigarette is small are avoided, and the accuracy of smoking detection is further improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a real-time video smoking detection method, device, equipment and medium based on YOLOv8. Background Art

[0002] In modern smart factories, monitoring workers' behavior is becoming increasingly important. Detecting smoking behavior is crucial for preventing fires and ensuring production safety. However, the application of existing smoking detection technology in complex environments still faces many challenges, such as low detection accuracy and high false detection rate.

[0003] In smart factories, surveillance cameras are usually installed at high positions with a wide shooting range, but the size of the objects (such as workers and cigarettes) in the image is small, which puts higher requirements on the accuracy of smoking detection. In addition, the light and background in the factory environment are complex and changeable, which further increases the difficulty of detection.

[0004] In recent years, deep learning-based target detection technology has often been used for smoking detection, but the existing YOLO series algorithms have difficulty in dealing with the problem of small target detection in complex scenarios.

[0005] For example, patent document CN114495258B describes a safety monitoring method for helmet wearing and smoking detection. This method extracts a high-definition picture of the upper body of the human body from the original image through a cascade design of a human key point detection network and a target detection network, and then performs helmet wearing and cigarette detection. This method uses the human key point information to eliminate misdetected cigarettes, thereby improving the detection accuracy. However, in complex environments, the detection accuracy of the human key points is limited, and there are interactive movements between the hands and the head of the person, such as when processing movements such as raising the hand, false detections will still occur.

[0006] For another example, patent document CN210983433U describes a driver smoking detection system based on computer vision technology. The system acquires image information through an infrared camera, performs real-time face detection, and performs smoking detection on the mouth area of ​​the face. The system has shown good results in detecting driver smoking, but it is mainly suitable for close-range, single-viewing scenes. For factory environments with long distances and complex environments, people and cigarettes are small in size, and it is difficult to achieve good detection results. Summary of the invention

[0007] In view of the above problems, the present invention provides a real-time video smoking detection method, device, equipment and medium for overcoming the above problems or at least partially solving the above problems. The present invention solves the problems of low smoking detection accuracy and high false detection rate in complex environments such as factories due to complex movements of personnel, long distances, small sizes of people and cigarettes, etc. in the prior art. It can achieve high-precision and low false detection smoking detection in complex environments such as smart factories.

[0008] The present invention provides the following scheme:

[0009] A real-time video smoking detection method, comprising:

[0010] Acquire a surveillance video stream of a target area in a factory, wherein the surveillance video stream includes continuous frame images;

[0011] Input each frame of image into the person detection model in sequence to perform person detection and obtain several person detection results;

[0012] Cutting out the person boundary box in each of the person detection results from the original image to obtain a plurality of person images;

[0013] Input each of the person images into a smoking detection model to detect and identify cigarettes and smoking actions, and obtain cigarette detection results and smoking action detection results around several persons;

[0014] The cigarette detection results around the person and the smoking action detection results in the images of the target frame number of the surveillance video stream are counted; if the number of images containing cigarette targets accounts for more than 80% of the target frame number and the number of images containing smoking actions accounts for more than 95% of the target frame number, it is determined that smoking behavior exists.

[0015] Preferably: the personnel detection model includes a YOLOv8 network structure.

[0016] Preferably, the continuous frame images include at least 30 frames.

[0017] Preferably, the person image is enlarged by at least 20% based on the person boundary box.

[0018] Preferably, the smoking detection model training data set includes images with two types of labels, namely, cigarettes and smoking actions, which are manually annotated.

[0019] Preferably, the smoking action includes holding the cigarette in hand and holding the cigarette in the mouth.

[0020] Preferably: the label adopts the YOLO format [cls, x, y, w, h] required for target detection, wherein cls represents the target category index, x, y represent the normalized rectangular box center point coordinates, and w, h represent the normalized rectangular box width and height.

[0021] A real-time video smoking detection device is used to perform the above-mentioned real-time video smoking detection method, the device comprising:

[0022] A video stream acquisition unit, used to acquire a monitoring video stream of a target area in the factory, wherein the monitoring video stream includes continuous frame images;

[0023] A person detection unit, used for sequentially inputting each frame of image into a person detection model to perform person detection and obtain a number of person detection results;

[0024] A cropping unit, used for cropping a person boundary box in each of the person detection results from the original image to obtain a plurality of person images;

[0025] A detection result acquisition unit, used for inputting each of the person images into a smoking detection model to detect and identify cigarettes and smoking actions, and obtaining cigarette detection results and smoking action detection results around a number of people;

[0026] A statistical output unit is used to count the cigarette detection results around the person and the smoking action detection results in the images of the target frame number of the monitoring video stream. If the number of images containing cigarette targets accounts for more than 80% of the target frame number and the number of images containing smoking actions accounts for more than 95% of the target frame number, it is determined that smoking behavior exists.

[0027] A real-time video smoking detection device, the device comprising a processor and a memory:

[0028] The memory is used to store program code and transmit the program code to the processor;

[0029] The processor is used to execute the above-mentioned real-time video smoking detection method according to the instructions in the program code.

[0030] A computer-readable storage medium is used to store program codes, and the program codes are used to execute the above-mentioned real-time video smoking detection method.

[0031] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0032] The embodiments of the present application provide a real-time video smoking detection method, device, equipment and medium. Through the cascade design of the YOLOv8 personnel detection model and the smoking detection model, the personnel in the scene are first detected, and then the cropped personnel image is detected for cigarette and smoking action detection, thereby reducing background interference and improving the accuracy of smoking detection. At the same time, by making statistical decisions on the smoking detection results of continuous frames of the video stream, the continuous frame information of the video stream is comprehensively utilized, effectively reducing the false detection caused by accidental hand-raising and other actions of personnel and suspected cigarettes. In addition, when performing target labeling, two types of labels, cigarette targets and smoking actions (holding cigarettes, holding cigarettes in mouths), are marked to avoid missed detections and false detections caused by the small size of cigarettes, further improving the accuracy of smoking detection.

[0033] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 is a flow chart of a real-time video smoking detection method provided by an embodiment of the present invention;

[0036] Figure 2 is a system framework diagram of a real-time video smoking detection method provided by an embodiment of the present invention;

[0037] Figure 3 is a flow chart of a smoking detection algorithm provided by an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of a smoking detection result provided by an embodiment of the present invention;

[0039] Figure 5 is a schematic diagram of a real-time video smoking detection device provided by an embodiment of the present invention;

[0040] Figure 6 It is a schematic diagram of a real-time video smoking detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0042] See also Figure 1 , is a real-time video smoking detection method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0043] S101: Acquire a surveillance video stream of a target area in a factory, wherein the surveillance video stream includes continuous frame images. To ensure detection accuracy, an embodiment of the present application may provide that the continuous frame images include at least 30 frames.

[0044] S102: Input each frame of image into the personnel detection model in sequence to perform personnel detection and obtain a number of personnel detection results; in specific implementation, the embodiment of the present application can provide that the personnel detection model includes a YOLOv8 network structure.

[0045] S103: Cut out the person boundary box in each of the person detection results from the original image to obtain a plurality of person images; in a specific implementation, the person image is enlarged by at least 20% based on the person boundary box.

[0046] S104: inputting each of the person images into a smoking detection model to detect and identify cigarettes and smoking actions, and obtaining cigarette detection results and smoking action detection results around several persons;

[0047] S105: Count the cigarette detection results around the person and the smoking action detection results in the images of the target frame number of the monitoring video stream. If the number of images containing cigarette targets accounts for more than 80% of the target frame number and the number of images containing smoking actions accounts for more than 95% of the target frame number, it is determined that smoking behavior exists.

[0048] In order to improve the training effect of the model, the embodiment of the present application can provide that the dataset for training the smoking detection model includes two types of labels, namely, cigarettes and smoking actions, which are manually annotated in the images.

[0049] Furthermore, the smoking action includes holding a cigarette in hand and holding a cigarette in the mouth. The label adopts the YOLO format [cls, x, y, w, h] required for target detection, where cls represents the target category index, x, y represent the normalized coordinates of the center point of the rectangular frame, and w, h represent the normalized width and height of the rectangular frame.

[0050] The method provided by this application is described in detail below using 30 frames of video statistics as an example.

[0051] The system framework of the real-time video smoking detection method based on YOLOv8 is as follows Figure 2 As shown, it mainly includes the following steps:

[0052] Constructing a dataset of cigarettes and smoking actions: First, we collected 2,000 images of cigarettes and people smoking from the Internet, and actually collected videos of people smoking at different angles and distances in the factory. We manually annotated the images with two types of labels, cigarettes and smoking actions, and finally constructed a dataset of 5,000 cigarettes and smoking actions.

[0053] In order to avoid missed detection and false detection caused by the small size of cigarettes, this method annotates two types of labels: cigarette targets and smoking actions (holding cigarettes in hand, holding cigarettes in mouth) when annotating targets. The labels use the YOLO format [cls, x, y, w, h] required for target detection, where cls is the target category index, x, y are the normalized coordinates of the center point of the rectangular box, and w, h are the normalized width and height of the rectangular box. Finally, the dataset is divided into training and test sets in a ratio of 8:2.

[0054] Smoke detection algorithm: This method performs smoke detection on the video stream, such as Figure 3 As shown, it mainly includes the following processes:

[0055] Video stream acquisition,Video streams can be acquired through surveillance cameras, which are installed in various key locations in the factory, covering the main working areas.,The video stream is transmitted to the central processing unit through the network.

[0056] Person detection is performed by inputting each frame of the video stream into the YOLOv8 person detection model to obtain the person detection result. The YOLOv8 person detection model can quickly and accurately detect people in the image and output the bounding box of each detected person, including the position and confidence of the bounding box. The bounding box with a confidence greater than a threshold (e.g. 0.7) is selected as a valid person detection result.

[0057] Image cropping: crop the person bounding box in the person detection result from the original image to form a person image. The cropped image includes the person and the area within a certain range around him to ensure the accuracy of cigarette detection. In order to ensure the detection accuracy, the size of the cropped image should be appropriately enlarged, for example, 20% larger than the bounding box.

[0058] Cigarette and smoking action detection, the cropped person image is input into the smoking detection model for cigarette and smoking action detection. The smoking detection model can detect cigarettes and smoking actions in the image, and output the bounding box of each detected cigarette and smoking action, including the position and confidence of the bounding box. The bounding box with a confidence greater than a threshold (e.g. 0.5) is selected as a valid cigarette and smoking action detection result.

[0059] Results statistics: Count the detection results of cigarettes and smoking actions around people in 30 frames (target frame number) of the video. If the cigarette target accounts for more than 80% and the smoking action accounts for more than 95%, it is determined that the person is smoking. The specific steps are as follows:

[0060] Calculation of cigarette target ratio: Assume that the number of consecutive frames counted is N = 30, and the number of frames where cigarettes are detected is N cigarette , then the cigarette target proportion

[0061] Calculation of smoking action ratio: Assume that the number of consecutive frames counted is N = 30, and the number of frames where smoking action is detected is N smoke , then the proportion of smoking actions

[0062] Smoking behavior judgment: If R cigarette >80%and R smoke If the rate is >95%, smoking behavior is judged to exist.

[0063] By counting the detection results of consecutive frames, false detections caused by accidental hand-raising and suspected cigarettes are effectively reduced.

[0064] Model training: The smoking detection model is trained using a cigarette and smoking action dataset, which contains cigarettes and images of people smoking obtained by cropping the original images.

[0065] Model testing: First, use the YOLOv8 person detection model to detect people in each frame of the video stream and crop the person image. Then use the trained smoking detection model to detect cigarettes and smoking actions in the person image, and perform statistical analysis to obtain the final smoking detection and recognition results.

[0066] Figure 4 The results of detecting smoking behavior of people at a long distance and in a factory environment are demonstrated. The results show that this method can achieve high-precision and low-false-detection smoking detection in complex environments such as smart factories.

[0067] In summary, the real-time video smoking detection method provided by this application, through the cascade design of the YOLOv8 personnel detection model and the smoking detection model, first detects the personnel in the scene, and then detects the cigarette and smoking action of the cropped personnel image, thereby reducing background interference and improving the accuracy of smoking detection. At the same time, by making statistical decisions on the smoking detection results of continuous frames of the video stream, the continuous frame information of the video stream is comprehensively utilized, effectively reducing the false detection caused by accidental hand-raising and other actions of personnel and suspected cigarettes. In addition, when performing target labeling, two types of labels, cigarette targets and smoking actions (holding cigarettes in hand, holding cigarettes in mouth), are marked to avoid missed detections and false detections caused by the small size of cigarettes, further improving the accuracy of smoking detection.

[0068] See also Figure 5 , the embodiment of the present application can also provide a real-time video smoking detection device, such as Figure 5 As shown, the device for executing the above-mentioned real-time video smoking detection method may include:

[0069] The video stream acquisition unit 501 is used to acquire a monitoring video stream of a target area in the factory, wherein the monitoring video stream includes continuous frame images;

[0070] A person detection unit 502, used to sequentially input each frame of image into a person detection model to perform person detection and obtain a number of person detection results;

[0071] A cropping unit 503, used for cropping a person boundary box in each of the person detection results from the original image to obtain a plurality of person images;

[0072] The detection result acquisition unit 504 is used to input each of the person images into the smoking detection model to detect and identify cigarettes and smoking actions, and obtain cigarette detection results and smoking action detection results around several people;

[0073] The statistical output unit 505 is used to count the cigarette detection results around the person and the smoking action detection results in the images of the target frame number of the monitoring video stream. If the number of images containing cigarette targets accounts for more than 80% of the target frame number and the number of images containing smoking actions accounts for more than 95% of the target frame number, it is determined that smoking behavior exists.

[0074] The embodiment of the present application may also provide a real-time video smoking detection device, the device comprising a processor and a memory:

[0075] The memory is used to store program code and transmit the program code to the processor;

[0076] The processor is used to execute the steps of the above-mentioned real-time video smoking detection method according to the instructions in the program code.

[0077] like Figure 6 As shown, a real-time video smoking detection device provided in an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 communicate with each other through the communication bus 13.

[0078] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices.

[0079] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in an embodiment of the real-time video smoking detection method.

[0080] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0081] Acquire a surveillance video stream of a target area in a factory, wherein the surveillance video stream includes continuous frame images;

[0082] Input each frame of image into the person detection model in sequence to perform person detection and obtain several person detection results;

[0083] Cutting out the person boundary box in each of the person detection results from the original image to obtain a plurality of person images;

[0084] Input each of the person images into a smoking detection model to detect and identify cigarettes and smoking actions, and obtain cigarette detection results and smoking action detection results around several persons;

[0085] The cigarette detection results around the person and the smoking action detection results in the images of the target frame number of the surveillance video stream are counted; if the number of images containing cigarette targets accounts for more than 80% of the target frame number and the number of images containing smoking actions accounts for more than 95% of the target frame number, it is determined that smoking behavior exists.

[0086] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0087] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0088] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0089] Of course, it should be noted that Figure 6 The structure shown does not constitute a limitation on the real-time video smoking detection device in the embodiment of the present application. In actual applications, the real-time video smoking detection device may include Figure 6 More or fewer components than shown, or combinations of certain components.

[0090] The embodiment of the present application may also provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program codes, and the program codes are used to execute the steps of the above-mentioned real-time video smoking detection method.

[0091] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0092] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0093] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A real-time video smoking detection method, characterized in that: include: Acquire a surveillance video stream of a target area in a factory, wherein the surveillance video stream includes continuous frame images; Input each frame of image into the person detection model in sequence to perform person detection and obtain several person detection results; Cutting out the person boundary box in each of the person detection results from the original image to obtain a plurality of person images; Input each of the person images into a smoking detection model to detect and identify cigarettes and smoking actions, and obtain cigarette detection results and smoking action detection results around several persons; The cigarette detection results around the person and the smoking action detection results in the images of the target frame number of the surveillance video stream are counted; if the number of images containing cigarette targets accounts for more than 80% of the target frame number and the number of images containing smoking actions accounts for more than 95% of the target frame number, it is determined that smoking behavior exists.

2. The real-time video smoking detection method according to claim 1, characterized in that: The personnel detection model includes a YOLOv8 network structure.

3. The real-time video smoking detection method according to claim 1, characterized in that: The continuous frame images include at least 30 frames.

4. The real-time video smoking detection method according to claim 1, characterized in that: The person image is enlarged by at least 20% based on the person bounding box.

5. The real-time video smoking detection method according to claim 1, characterized in that: The smoking detection model training data set includes images with two types of labels, namely, cigarettes and smoking actions, which are manually annotated.

6. The real-time video smoking detection method according to claim 5, characterized in that: The smoking action includes holding the cigarette in hand and holding the cigarette in the mouth.

7. The real-time video smoking detection method according to claim 5, characterized in that: The label adopts the YOLO format [cls, x, y, w, h] required for target detection, where cls represents the target category index, x and y represent the normalized coordinates of the center point of the rectangular box, and w and h represent the normalized width and height of the rectangular box.

8. A real-time video smoking detection device, characterized in that: The device is used to execute the real-time video smoking detection method according to any one of claims 1 to 7, comprising: A video stream acquisition unit, used to acquire a monitoring video stream of a target area in the factory, wherein the monitoring video stream includes continuous frame images; A person detection unit, used for sequentially inputting each frame of image into a person detection model to perform person detection and obtain a number of person detection results; A cropping unit, used for cropping a person boundary box in each of the person detection results from the original image to obtain a plurality of person images; A detection result acquisition unit, used for inputting each of the person images into a smoking detection model to detect and identify cigarettes and smoking actions, and obtaining cigarette detection results and smoking action detection results around a number of people; A statistical output unit is used to count the cigarette detection results around the person and the smoking action detection results in the images of the target frame number of the monitoring video stream. If the number of images containing cigarette targets accounts for more than 80% of the target frame number and the number of images containing smoking actions accounts for more than 95% of the target frame number, it is determined that smoking behavior exists.

9. A real-time video smoking detection device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the real-time video smoking detection method according to any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the real-time video smoking detection method according to any one of claims 1 to 7.

Images