Infrared-visible light smoking detection method and system

Abstract

The invention discloses an infrared-visible light smoking detection method in one aspect. On the other hand, the invention discloses an infrared-visible light smoking detection system which is characterized by being used for realizing the infrared-visible light smoking detection method and comprising an infrared camera; a visible light camera; a detection analysis module; an output module; and an alarm output module. By analyzing the infrared image and the visible light image respectively, the smoking behavior can be effectively detected in various environments, especially in low-light or sheltered scenes, and the public safety monitoring capability is improved. In addition, the method has real-time performance and high accuracy, and is suitable for public places and sensitive areas where smoking behaviors need to be strictly monitored.

Description

Technical Field

[0001] The present invention relates to a smoking detection method and system based on the fusion of infrared and visible light image data, focusing on detecting smoking behavior by analyzing registered image data, and is suitable for monitoring safety-sensitive areas such as construction sites, chemical parks, gas stations, shopping malls and other places, belonging to the field of computer vision detection technology. Background Art

[0002] In the field of public safety monitoring, especially in flammable and explosive environments, the detection of smoking behavior is crucial. Traditional video surveillance systems often perform poorly in low-light or occluded conditions, and often lack the ability to accurately distinguish between smoking and non-smoking behaviors. In recent years, image fusion technology, especially infrared and visible light image fusion, has provided new possibilities for detecting specific behaviors in complex environments. By combining the thermal imaging data captured by infrared cameras and the detailed images provided by visible light cameras, smoking behavior can be more accurately monitored and located, even in low-light or occluded conditions.

[0003] The challenges facing current smoking detection technology include how to effectively integrate information from different spectra and accurately identify temperature anomalies and related human behavior characteristics caused by smoking behavior. In addition, a system is required that can process large amounts of data and make accurate judgments without affecting real-time performance. Therefore, a reliable solution is urgently needed to improve the accuracy and efficiency of smoking detection, especially in situations where environmental conditions are poor and traditional methods are difficult to apply. Summary of the invention

[0004] The purpose of the present invention is to solve the problem of accurately detecting smoking behavior in a complex environment.

[0005] In order to achieve the above object, one aspect of the present invention discloses an infrared-visible light smoking detection method, which is characterized by comprising the following steps:

[0006] Step 1: Acquire an infrared image and a visible light image of the target area through an infrared camera and a visible light camera;

[0007] Step 2: After preprocessing the acquired infrared image, use the adaptive threshold technology to detect temperature anomaly and identify the center coordinates of the temperature anomaly area;

[0008] The YOLOv8 deep learning model is used to analyze the visible light image to obtain the human body feature area, wherein the YOLOv8 deep learning model processes the visible light image including the following steps:

[0009] Step 201: The YOLOv8 deep learning model extracts high-level features from the input visible light image through multi-layer convolution operations. These features include edges, textures, shapes, and colors in the image, especially features of human body parts that can represent smoking behavior.

[0010] Step 202: After feature extraction is completed, the YOLOv8 deep learning model generates candidate regions through a region proposal network, and extracts candidate boxes that may contain target objects from the feature map obtained in the previous step. These candidate boxes cover regions of different scales and proportions in the image;

[0011] Step 203: After generating the candidate regions, the YOLOv8 deep learning model classifies and regresses the bounding box for each candidate box. In the classification stage, the probability of each candidate box containing a specific target is calculated to select the most likely target region. In the bounding box regression stage, the position and size of the candidate box are finely adjusted to ensure more accurate positioning of the target. The bounding box outputted in the end is the position of the detected target object.

[0012] Step 204: The YOLOv8 deep learning model integrates the features of different levels obtained in the previous step through multi-scale feature fusion technology;

[0013] Step 205: The final output result of the YOLOv8 deep learning model is the bounding box of the human features detected in the image and its corresponding coordinate range, that is, the human feature area;

[0014] Step 3: Determine whether the center coordinate of the temperature abnormality area is located within the human body feature area: if so, the temperature abnormality is considered to be caused by smoking behavior and is marked as a suspected smoking event; otherwise, the temperature abnormality is considered to be caused by other factors and is ignored;

[0015] Step 4: after confirming the possible existence of smoking behavior, generate a warning message, which includes the specific location and time of the smoking behavior, issue a warning through an audible and visual alarm device, and display the warning message graphically on the monitoring interface.

[0016] Preferably, in step 1, the infrared camera and the visible light camera work synchronously.

[0017] Preferably, in step 2, preprocessing of the infrared image includes image enhancement and noise elimination to improve the signal-to-noise ratio and contrast of the infrared image.

[0018] Preferably, in step 2, training the YOLOv8 deep learning model includes the following steps:

[0019] Build a dataset for training the model:

[0020] Visible light image data is collected from open source datasets of public smoking scenes. At the same time, visible light image data is obtained from actual monitoring scenes. All the collected visible light image data covers smoking behaviors in various complex scenes, different lighting conditions, and multiple camera angles.

[0021] Perform data enhancement and fine annotation on the collected visible light image data, annotate the bounding boxes of key features of the human body, and store the annotation information in standard XML format;

[0022] Divide the dataset into training, validation and test sets;

[0023] In the training process of the YOLOv8 deep learning model, the YOLOv8 deep learning model pre-trained on the COCO dataset was selected as the initial weight. In the specific training, the hyperparameters suitable for the smoking detection task were set. The initial learning rate was set to 0.001, and the cosine annealing scheduling mechanism was used to gradually reduce the learning rate.

[0024] The model training was conducted in a high-performance computing environment for 200 epochs. During the training process, the loss function adopted the form of multi-task loss, combined with the binary cross entropy loss function for classification tasks, and the IoU loss function was used to optimize bounding box regression. After each epoch, the YOLOv8 deep learning model was evaluated on the validation set. By monitoring the loss value, precision, and recall on the validation set, the model training strategy was dynamically adjusted.

[0025] After training, the YOLOv8 deep learning model has been quantized and pruned for optimization.

[0026] Preferably, in step 4, the warning information is remotely transmitted to the monitoring center or the mobile device of the relevant person in charge through the built-in network module to ensure timely response.

[0027] Preferably, in step 4, detailed information of all warning events is stored for future query and analysis.

[0028] Another aspect of the present invention discloses an infrared-visible light smoking detection system, which is used to implement the above-mentioned infrared-visible light smoking detection method, and comprises:

[0029] an infrared camera for capturing temperature-related infrared image data;

[0030] a visible light camera for capturing visible light image data related to human features;

[0031] Detection and analysis module, used for:

[0032] Analyze the infrared image data to obtain the center coordinates of the temperature anomaly area;

[0033] The YOLOv8 deep learning model is used to analyze visible light image data to obtain human body feature areas;

[0034] Calculate the spatial overlap between the center coordinates of the temperature abnormality area and the human body feature area, and obtain the smoking detection result based on the spatial overlap;

[0035] Output module, used to display the results of smoking detection;

[0036] The alarm output module is used to generate a warning message after confirming that there may be smoking behavior.

[0037] By analyzing infrared images and visible light images separately, the present invention can effectively detect smoking behavior in various environments, especially in low-light or obstructed scenes, and improve the ability of public safety monitoring. In addition, the present invention also has real-time performance and high accuracy, and is suitable for public places and sensitive areas where smoking behavior needs to be strictly monitored. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0040] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in the present invention refers to and includes any or all possible combinations of one or more associated listed items.

[0041] One aspect of an embodiment of the present invention discloses an infrared-visible light smoking detection method, which is a smoking detection method based on independent processing of infrared images and visible light images, and can accurately detect smoking behavior in complex environments, especially suitable for low-light or occluded scenes, and specifically includes the following steps:

[0042] Step 1: Image acquisition

[0043] The infrared camera and visible light camera are used to obtain infrared images and visible light images of the target area. The infrared camera is used to capture the thermal radiation information in the scene and generate an infrared image, which can reflect the temperature distribution in the target area. At the same time, the visible light camera is used to obtain a visible light image of the same area to extract the visible features of the human body. The infrared camera and the visible light camera work synchronously to ensure that the infrared image and the visible light image obtained are highly consistent in time and space.

[0044] Step 2: Infrared image analysis

[0045] After acquiring the infrared image, the infrared image is first preprocessed. Preprocessing includes steps such as image enhancement and noise elimination to improve the signal-to-noise ratio and contrast of the infrared image and ensure the accuracy of subsequent temperature anomaly detection. On the preprocessed infrared image, the adaptive threshold technology is used to detect temperature anomalies. This technology analyzes the temperature distribution characteristics of different areas in the infrared image and dynamically adjusts the detection threshold to accurately identify possible temperature anomaly areas. The center coordinates of the identified temperature anomaly area (for example, (x0, y0)) will be recorded as possible smoking points.

[0046] Visible light image analysis

[0047] In parallel with infrared image analysis, human features are detected and identified in visible light images. Specifically, the embodiment of the present invention adopts the YOLOv8 deep learning model, which has been trained with a large amount of smoking scene data and can accurately identify key human body parts such as heads and hands in complex backgrounds. The core of the design of the YOLOv8 deep learning model lies in its fast and efficient target detection capabilities. The model is based on an improved multi-layer convolutional neural network (CNN) and combines a series of optimization algorithms to improve detection accuracy and speed.

[0048] In the embodiment of the present invention, in order to ensure that the YOLOv8 deep learning model can accurately detect smoking behavior, a high-quality data set is first constructed. The process of constructing the data set includes:

[0049] Collect visible light image data from multiple sources, some of which come from open source datasets of public smoking scenes. These visible light images cover a variety of complex scenes, different lighting conditions, and smoking behaviors from multiple camera angles.

[0050] In addition, in order to enhance the diversity of the data set, infrared images and visible light images from actual monitoring scenes were collected. These infrared images and visible light images were obtained through cameras installed in public places, which truly reflect the smoking behavior in actual application scenarios. After being collected, all visible light images were finely annotated using professional annotation tools, including bounding boxes of key human features such as heads, hands, and smoke. These annotation information were stored in a standard XML format to provide a basis for subsequent training.

[0051] In addition, in order to increase the size of the dataset and the generalization ability of the model, a variety of data enhancement processes were performed on the collected visible light images, such as rotation, flipping, scaling, cropping, brightness adjustment, and adding noise;

[0052] Finally, the dataset was divided into training set, validation set, and test set in a ratio of 7:2:1 to ensure that the training, validation, and testing processes of the model can be fully supported by data.

[0053] During the training of the YOLOv8 deep learning model, the YOLOv8 deep learning model pre-trained on the COCO dataset can be selected as the initial weight. This pre-trained model has rich feature extraction capabilities, which helps to accelerate the convergence of the model. In the specific training, hyperparameters suitable for the smoking detection task are set, where the initial learning rate is set to 0.001, and the cosine annealing scheduling mechanism is used to gradually reduce the learning rate to maintain the stability and accuracy of the model in the later stage of training. The batch size is set to 32, and the model is trained in a high-performance computing environment for a total of 200 epochs to ensure that the model can be fully trained in a complex environment. During the training process, the loss function adopts the form of multi-task loss, combined with the binary cross entropy loss function for classification tasks, and the IoU loss function is used to optimize the bounding box regression. After each epoch, the YOLOv8 deep learning model will be evaluated on the validation set. By monitoring the loss value, precision, and recall on the validation set, the training strategy of the model is dynamically adjusted to prevent overfitting. After training, the YOLOv8 deep learning model has been quantized and pruned to reduce computational complexity and memory usage, so as to adapt to the real-time deployment requirements on edge devices. After these optimization steps, the YOLOv8 deep learning model can maintain efficient operation performance and accurate detection capabilities in practical applications.

[0054] Specifically, the YOLOv8 deep learning model includes the following main processing stages:

[0055] Feature extraction layer: In this stage, the YOLOv8 deep learning model extracts high-level features from the input image through multi-layer convolution operations. These features include important information such as edges, textures, shapes, and colors in the image, especially the features of human body parts related to smoking behavior (such as heads and hands). The output of the feature extraction layer is a multi-dimensional feature map that retains the spatial information of the input image that may contain the target object.

[0056] Region Proposal Network (RPN): After feature extraction, the YOLOv8 deep learning model generates candidate regions through RPN. The main function of RPN is to extract candidate boxes (anchor boxes) that may contain target objects from the feature map. These candidate boxes cover areas of different scales and proportions in the image, ensuring that the target object can be fully captured even in complex backgrounds. In this embodiment, RPN generates a large number of candidate regions, some of which may contain features of human heads and hands.

[0057] Target classification and bounding box regression: After generating candidate regions, the YOLOv8 deep learning model classifies and regresses each candidate box. The classification stage selects the most likely target region by calculating the probability that each candidate box contains a specific target (such as a head or a hand). At the same time, the bounding box regression stage fine-tunes the position and size of the candidate box to ensure more accurate positioning of the target. The final output bounding box is the position of the detected target object.

[0058] Multi-scale feature fusion: The YOLOv8 deep learning model integrates features at different levels through multi-scale feature fusion technology. This process ensures that the model can maintain consistent detection accuracy when dealing with target objects of different scales. For the task of smoking detection, the application of this technology enables the model to maintain high accuracy and high robustness when processing heads and hands of different sizes in the image, avoiding missed detection or false detection due to changes in target scale.

[0059] Output stage: The final output of the YOLOv8 deep learning model is the bounding box of the human features detected in the image and its corresponding coordinate range C. The coordinates of the bounding box (x1, y1, x2, y2) identify the location of the target area in the image, and these coordinates will be used in the subsequent comprehensive analysis stage.

[0060] The advantage of the YOLOv8 deep learning model lies in its highly parallel design, which can significantly improve the processing speed while ensuring detection accuracy. Its parallel design enables the model to process multiple candidate areas at the same time and quickly obtain the final detection results. This is especially important for real-time smoking detection tasks, because the system needs to process a large amount of image data in a short period of time and respond quickly. In addition, the YOLOv8 deep learning model further improves the detection accuracy and reduces the false detection rate and missed detection rate by adopting advanced optimization algorithms such as non-maximum suppression (NMS). Combined with the above characteristics, the YOLOv8 deep learning model can accurately identify key human features related to smoking behavior under complex backgrounds and changing lighting conditions, enabling the system to effectively detect and locate potential smoking behavior in a variety of environments.

[0061] Step 4: Comprehensive analysis of results

[0062] After analyzing the infrared image and the visible light image independently, the analysis results of the two are processed comprehensively. Specifically, the coordinates (x0, y0) of the temperature anomaly area in the infrared image are first obtained, and it is determined whether the coordinates are located in the human feature area C detected by the YOLOv8 deep learning model in the visible light image. If (x0, y0) is within the range of the human feature area C, it is considered that the temperature anomaly is very likely caused by smoking behavior and is marked as a suspected smoking event. If (x0, y0) is not within the range of the human feature area C, it is considered that the temperature anomaly may be caused by other factors and the anomaly is ignored.

[0063] Step 5: Warning message output

[0064] After confirming the possible existence of smoking behavior, a warning message is generated. The warning message includes the specific location of the smoking behavior (i.e. the coordinate location of the temperature anomaly) and the time of occurrence. The warning is issued through the sound and light alarm device, and the warning message is displayed graphically on the monitoring interface. In addition, the warning information can be remotely transmitted to the monitoring center or the mobile device of the relevant person in charge through the built-in network module to ensure timely response. The detailed information of all warning events can also be stored for future query and analysis.

[0065] Another aspect of the embodiments of the present invention is to provide a system for implementing the above infrared-visible light smoking detection method, comprising:

[0066] an infrared camera for capturing temperature-related infrared image data;

[0067] a visible light camera for capturing visible light image data related to human features;

[0068] In the embodiment of the present invention, both the infrared camera and the visible light camera are equipped with high-resolution sensors to improve image quality and detection accuracy;

[0069] Detection and analysis module, used for:

[0070] Analyze the infrared image data to obtain the center coordinates of the temperature anomaly area;

[0071] The YOLOv8 deep learning model is used to analyze visible light image data to obtain human body feature areas;

[0072] Calculate the spatial overlap between the center coordinates of the temperature abnormality area and the human body feature area, and obtain the smoking detection result based on the spatial overlap;

[0073] Output module, used to display the results of smoking detection;

[0074] In an embodiment of the present invention, the output module includes at least one display device for displaying the detection results in real time and providing an alarm function to trigger a response when smoking behavior is detected;

[0075] The alarm output module is used to generate warning information after confirming the possible existence of smoking behavior, and issue a warning through the sound and light alarm device. At the same time, the warning information is displayed in a graphical way on the output module. In addition, the warning information is remotely transmitted to the monitoring center or the mobile device of the relevant person in charge through the built-in network module to ensure timely response; it also has a real-time recording function, which can store detailed information of all warning events for future query and analysis.

Claims

1. An infrared-visible light smoking detection method, characterized in that: The following steps are involved: Step 1: Acquire an infrared image and a visible light image of the target area through an infrared camera and a visible light camera; Step 2: After preprocessing the acquired infrared image, the temperature anomaly detection is performed using the adaptive threshold technology to identify the center coordinates of the temperature anomaly area; The YOLOv8 deep learning model is used to analyze the visible light image to obtain the human body feature area, wherein the YOLOv8 deep learning model processes the visible light image including the following steps: Step 201: The YOLOv8 deep learning model extracts high-level features from the input visible light image through multi-layer convolution operations. These features include edges, textures, shapes, and colors in the image, especially features of human body parts that can represent smoking behavior. Step 202: After feature extraction is completed, the YOLOv8 deep learning model generates candidate regions through a region proposal network, and extracts candidate boxes that may contain target objects from the feature map obtained in the previous step. These candidate boxes cover regions of different scales and proportions in the image; Step 203: After generating the candidate regions, the YOLOv8 deep learning model classifies and regresses the bounding box for each candidate box. In the classification stage, the probability of each candidate box containing a specific target is calculated to select the most likely target region. In the bounding box regression stage, the position and size of the candidate box are finely adjusted to ensure more accurate positioning of the target. The bounding box outputted in the end is the position of the detected target object. Step 204: The YOLOv8 deep learning model integrates the features of different levels obtained in the previous step through multi-scale feature fusion technology; Step 205: The final output result of the YOLOv8 deep learning model is the bounding box of the human features detected in the image and its corresponding coordinate range, that is, the human feature area; Step 3: Determine whether the center coordinate of the temperature abnormality area is located within the human body feature area: if so, the temperature abnormality is considered to be caused by smoking behavior and is marked as a suspected smoking event; otherwise, the temperature abnormality is considered to be caused by other factors and is ignored; Step 4: after confirming the possible existence of smoking behavior, generate a warning message, which includes the specific location and time of the smoking behavior, issue a warning through an audible and visual alarm device, and display the warning message graphically on the monitoring interface.

2. An infrared-visible light smoking detection method as claimed in claim 1, characterized in that: In step 1, the infrared camera and the visible light camera work synchronously.

3. The infrared-visible light smoking detection method according to claim 1, characterized in that: In step 2, the preprocessing of the infrared image includes image enhancement and noise elimination to improve the signal-to-noise ratio and contrast of the infrared image.

4. The infrared-visible light smoking detection method according to claim 1, characterized in that: In step 2, training the YOLOv8 deep learning model includes the following steps: Build a dataset for training the model: Visible light image data is collected from open source datasets of public smoking scenes. At the same time, visible light image data is obtained from actual monitoring scenes. All the collected visible light image data covers smoking behaviors in various complex scenes, different lighting conditions, and multiple camera angles. Perform data enhancement and fine annotation on the collected visible light image data, annotate the bounding boxes of key features of the human body, and store the annotation information in standard XML format; Divide the dataset into training, validation and test sets; In the training process of the YOLOv8 deep learning model, the YOLOv8 deep learning model pre-trained on the COCO dataset was selected as the initial weight. In the specific training, the hyperparameters suitable for the smoking detection task were set. The initial learning rate was set to 0.001, and the cosine annealing scheduling mechanism was used to gradually reduce the learning rate. The model training was conducted in a high-performance computing environment for 200 epochs. During the training process, the loss function adopted the form of multi-task loss, combined with the binary cross entropy loss function for classification tasks, and the IoU loss function was used to optimize bounding box regression. After each epoch, the YOLOv8 deep learning model was evaluated on the validation set. By monitoring the loss value, precision, and recall on the validation set, the model training strategy was dynamically adjusted. After training, the YOLOv8 deep learning model has been quantized and pruned for optimization.

5. The infrared-visible light smoking detection method according to claim 1, characterized in that: In step 4, the warning information is remotely transmitted to the monitoring center or the mobile device of the relevant person in charge through the built-in network module to ensure timely response.

6. The infrared-visible light smoking detection method according to claim 1, characterized in that: In step 4, the detailed information of all warning events is stored for future query and analysis.

7. An infrared-visible light smoking detection system, characterized in that: The infrared-visible light smoking detection method according to claim 1 comprises: an infrared camera for capturing temperature-related infrared image data; a visible light camera for capturing visible light image data related to human features; Detection and analysis module, used for: Analyze the infrared image data to obtain the center coordinates of the temperature anomaly area; The YOLOv8 deep learning model is used to analyze visible light image data to obtain human body feature areas; Calculate the spatial overlap between the center coordinates of the temperature abnormality area and the human body feature area, and obtain the smoking detection result based on the spatial overlap; Output module, used to display the results of smoking detection; The alarm output module is used to generate a warning message after confirming that there may be smoking behavior.

Images