Smoking detection method and system based on deep learning
Through deep learning-based smoking detection method, the YOLOv9 neural network and ByteTrack target tracking algorithm are used to solve the accuracy and response time of smoking detection in complex environments, and the rapid and accurate identification and tracking of smoking behavior is achieved, reducing false alarms rate, suitable for resource-constrained devices.
Patent Information
- Application Number
- CN202410993275.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-12
AI Technical Summary
The existing smoking detection methods have problems such as low detection accuracy, slow response time, high false alarm rate and blind spots in complex environments, especially in terms of occlusion, dynamic behavior, distance and angle changes, similar behavior distinction, and resource limitation. It is difficult to effectively monitor smoking behavior.
Using deep learning-based smoking detection method, the smoking area detection model is constructed using the YOLOv9 neural network, and combined with the ByteTrack target tracking algorithm, the smoking action trajectory is generated, and the smoking behavior is accurately identified and tracked through image feature extraction and similarity measurement, and the smoking behavior is achieved quickly. and low false alarm rate.
In complex dynamic scenarios, areas where smoking behaviors may be found can be quickly and accurately screened out, smoking actions can be tracked stably, false alarm rates, improved detection accuracy and real-time performance, and reduced labor costs. It is suitable for equipment with resource-constrained.
Smart Images

Figure CN118941862A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of smoking detection, and in particular to a smoking detection method and system based on deep learning. Background Art
[0002] At present, under the current technological background, the development of smoking detection technology has become an important issue in the field of public health and safety. With the improvement of social health awareness and the restriction of smoking in public places, how to effectively monitor and prevent smoking behavior, especially in non-smoking areas, has become an urgent problem to be solved.
[0003] Existing smoking detection methods mainly include traditional smoking detection methods that rely on smoke detectors and manual monitoring and vision-based smoking detection methods. Traditional smoking detection methods often have disadvantages such as slow response time, high false alarm rate and monitoring blind spots. Vision-based smoking detection methods mainly have the following problems: (1) Occlusion problem: In complex environments, smokers or cigarettes in their hands may be partially or completely obscured, which poses a challenge to accurate detection. (2) Dynamic behavior and posture changes: Smoking behavior is accompanied by various dynamic behaviors and posture changes of hands and faces, which make accurate identification difficult. (3) Distance and angle problems: The distance between the detection system and the smoking individual and the angle of the camera may affect the detection accuracy. Long distances or extreme angles may cause the loss of key features. (4) Distinguishing between similar behaviors: It is necessary to distinguish smoking from other similar hand movements or behaviors, such as eating, drinking water or wiping the mouth, which requires highly accurate pattern recognition technology. (5) Real-time processing and resource constraints: Real-time smoking detection requires efficient algorithms to ensure fast response. At the same time, running these algorithms on resource-constrained devices (such as embedded systems or smart cameras) may face challenges in computing resources and power consumption. Summary of the invention
[0004] The present invention provides a smoking detection method based on deep learning, which is used to solve the problems that the existing vision-based smoking detection methods exist in complex environments, resulting in low detection accuracy. However, these methods often have slow response time, high false alarm rate and monitoring blind spots, including:
[0005] Acquire image acquisition data of a no-smoking area to be identified;
[0006] Based on the image acquisition data, using a pre-built smoking area detection model, determining whether there is a suspected smoking area in the image acquisition data;
[0007] When there is a suspected smoking area in the image acquisition data, a ByteTrack target tracking algorithm is used to generate a smoking action trajectory;
[0008] Determining whether there is any smoking behavior in the no-smoking area to be identified according to the smoking action trajectory;
[0009] The smoking area detection model uses historical image acquisition data as input of training data and actual annotation data of whether there is a smoking area in the historical image acquisition data as output of training data, and is obtained by training the YOLOv9 neural network.
[0010] Optionally, the smoking area detection model includes the following construction process:
[0011] Initialize network parameters;
[0012] Using historical image acquisition data as input for training data;
[0013] Whether there is actual annotated data of a smoking area in the historical image acquisition data is used as output of training data;
[0014] Based on the input and output of the training data, the YOLOv9 neural network is trained, and the network parameters in the YOLOv9 neural network are updated by a gradient optimization method of an adaptive learning rate momentum factor;
[0015] Based on the updated network parameters, the YOLOv9 neural network is updated, and by calculating the cross entropy loss function of the updated YOLOv9 neural network, the network parameters in the updated YOLOv9 neural network are adjusted using a back propagation algorithm to obtain a smoking area detection model.
[0016] Optionally, when there is a suspected smoking area in the image acquisition data, using the ByteTrack target tracking algorithm to generate a smoking action trajectory includes:
[0017] When there is a suspected smoking area in the image acquisition data, the suspected smoking area is subjected to lossless enlargement of the local area by using an image segmentation and enhancement method to obtain a smoking enlargement area;
[0018] Based on the smoking expansion area, the smoking action trajectory is generated using the ByteTrack target tracking algorithm.
[0019] Optionally, the step of generating a smoking action trajectory based on the smoking expansion area using a ByteTrack target tracking algorithm includes:
[0020] Performing feature extraction on each frame of image data corresponding to the smoking enlarged area to obtain image feature data;
[0021] The similarity between the image feature data of adjacent frames is calculated using the intersection-over-union ratio as a similarity measure;
[0022] Generate a smoking action trajectory using a ByteTrack target tracking algorithm according to the similarity between the image feature data of adjacent frames;
[0023] Wherein, the image feature data includes: appearance features and motion features;
[0024] The appearance features include: color features, texture features and contour features;
[0025] The motion characteristics include: coordinate change characteristics and speed change characteristics.
[0026] Optionally, generating a smoking action trajectory using a ByteTrack target tracking algorithm according to the similarity between the image feature data of adjacent frames includes:
[0027] According to the similarity between the image feature data of adjacent frames, selecting image feature data whose similarity is greater than a set threshold;
[0028] Using the ByteTrack algorithm, coordinate point matching is performed on the image feature data whose similarity is greater than a set threshold;
[0029] Connect the center points or key points of the successfully matched image feature data to form a continuous trajectory line;
[0030] The continuous trajectory lines are superimposed and displayed to generate a smoking action trajectory.
[0031] Optionally, determining whether there is smoking behavior in the to-be-identified no-smoking area according to the smoking action trajectory includes:
[0032] Generate an action sequence and a time sequence according to the smoking action trajectory;
[0033] Determine whether smoke appears in the image acquisition data corresponding to the no-smoking area in the time series, and obtain a determination result;
[0034] Determining whether there is smoking behavior in the no-smoking area to be identified according to the action time sequence, the time sequence and the judgment result;
[0035] The action sequence includes one or more of the following: raising the hand to the mouth, moving an object with the fingertips, and mouth movements.
[0036] Optionally, after acquiring the image acquisition data of the no-smoking area to be identified, before determining whether there is a suspected smoking area in the image acquisition data by using a pre-built smoking area detection model based on the image acquisition data, the method further includes:
[0037] Performing data preprocessing on the image acquisition data of the no-smoking area to be identified to obtain preprocessed image acquisition data;
[0038] The data preprocessing includes one or more of the following: image enhancement, noise removal and image standardization.
[0039] Optionally, after determining whether there is any smoking behavior in the no-smoking area to be identified according to the smoking action trajectory, the method further includes:
[0040] When it is determined that smoking behavior exists in the no-smoking area to be identified, an alarm is automatically triggered, and the alarm information is pushed to the background system through the http network.
[0041] Based on the same inventive concept, the present invention also provides a smoking detection system based on deep learning, comprising:
[0042] Data acquisition module: used to obtain image acquisition data of the no-smoking area to be identified;
[0043] Behavior judgment module: used to determine whether there is a suspected smoking area in the image acquisition data based on the image acquisition data and using a pre-built smoking area detection model;
[0044] Trajectory generation module: used to generate a smoking action trajectory using the ByteTrack target tracking algorithm when there is a suspected smoking area in the image acquisition data;
[0045] Behavior determination module: used for determining whether there is smoking behavior in the no-smoking area to be identified according to the smoking action trajectory;
[0046] Among them, the smoking area detection model in the behavior judgment module uses historical image acquisition data as input of training data, and uses actual annotation data of whether there is a smoking area in the historical image acquisition data as output of training data, and is obtained by training the YOLOv9 neural network.
[0047] Optionally, the smoking area detection model in the behavior judgment module includes the following construction process:
[0048] Initializing network parameters of the EEG signal recognition model;
[0049] Using historical image acquisition data as input for training data;
[0050] Whether there is actual annotated data of a smoking area in the historical image acquisition data is used as output of training data;
[0051] Based on the input and output of the training data, the YOLOv9 neural network is trained, and the network parameters in the YOLOv9 neural network are updated by a gradient optimization method of an adaptive learning rate momentum factor;
[0052] Based on the updated network parameters, the YOLOv9 neural network is updated, and by calculating the cross entropy loss function of the updated YOLOv9 neural network, the network parameters in the updated YOLOv9 neural network are adjusted using a back propagation algorithm to obtain a smoking area detection model.
[0053] Acquire image acquisition data of a no-smoking area to be identified;
[0054] Based on the image acquisition data, using a pre-built smoking area detection model, determining whether there is a suspected smoking area in the image acquisition data;
[0055] When there is a suspected smoking area in the image acquisition data, a ByteTrack target tracking algorithm is used to generate a smoking action trajectory;
[0056] Determining whether there is any smoking behavior in the no-smoking area to be identified according to the smoking action trajectory;
[0057] The smoking area detection model uses historical image acquisition data as input of training data and actual annotation data of whether there is a smoking area in the historical image acquisition data as output of training data, and is obtained by training the YOLOv9 neural network.
[0058] Optionally, the trajectory generation module is specifically used to:
[0059] When there is a suspected smoking area in the image acquisition data, the suspected smoking area is subjected to lossless enlargement of the local area by using an image segmentation and enhancement method to obtain a smoking enlargement area;
[0060] Based on the smoking expansion area, the smoking action trajectory is generated using the ByteTrack target tracking algorithm.
[0061] Optionally, the trajectory generation module is specifically used to:
[0062] Performing feature extraction on each frame of image data corresponding to the smoking enlarged area to obtain image feature data;
[0063] The similarity between the image feature data of adjacent frames is calculated using the intersection-over-union ratio as a similarity measure;
[0064] Generate a smoking action trajectory using a ByteTrack target tracking algorithm according to the similarity between the image feature data of adjacent frames;
[0065] Wherein, the image feature data includes: appearance features and motion features;
[0066] The appearance features include: color features, texture features and contour features;
[0067] The motion characteristics include: coordinate change characteristics and speed change characteristics.
[0068] Optionally, the trajectory generation module generates a smoking action trajectory using a ByteTrack target tracking algorithm according to the similarity between the image feature data of adjacent frames, including:
[0069] According to the similarity between the image feature data of adjacent frames, selecting image feature data whose similarity is greater than a set threshold;
[0070] Using the ByteTrack algorithm, coordinate point matching is performed on the image feature data whose similarity is greater than a set threshold;
[0071] Connect the center points or key points of the successfully matched image feature data to form a continuous trajectory line;
[0072] The continuous trajectory lines are superimposed and displayed to generate a smoking action trajectory.
[0073] Optionally, the behavior determination module is specifically used to:
[0074] Generate an action sequence and a time sequence according to the smoking action trajectory;
[0075] Determine whether smoke appears in the image acquisition data corresponding to the no-smoking area in the time series, and obtain a determination result;
[0076] Determining whether there is smoking behavior in the no-smoking area to be identified according to the action time sequence, the time sequence and the judgment result;
[0077] The action sequence includes one or more of the following: raising the hand to the mouth, moving an object with the fingertips, and mouth movements.
[0078] Optionally, after the data acquisition module acquires the image acquisition data of the no-smoking area to be identified, the method further includes:
[0079] Performing data preprocessing on the image acquisition data of the no-smoking area to be identified to obtain preprocessed image acquisition data;
[0080] The data preprocessing includes one or more of the following: image enhancement, noise removal and image standardization.
[0081] Optionally, after the behavior determination module determines whether there is smoking behavior in the to-be-identified no-smoking area according to the smoking action trajectory, it further includes:
[0082] When it is determined that smoking behavior exists in the no-smoking area to be identified, an alarm is automatically triggered, and the alarm information is pushed to the background system through the http network.
[0083] Compared with the closest prior art, the present invention has the following beneficial effects:
[0084] The present invention provides a smoking detection method and system based on deep learning, including: acquiring image acquisition data of a no-smoking area to be identified; based on the image acquisition data, using a pre-constructed smoking area detection model to determine whether there is a suspected smoking area in the image acquisition data; when there is a suspected smoking area in the image acquisition data, using a ByteTrack target tracking algorithm to generate a smoking action trajectory; according to the smoking action trajectory, determining whether there is smoking behavior in the no-smoking area to be identified; wherein the smoking area detection model uses historical image acquisition data as input of training data, and uses actual annotation data of whether there is a smoking area in the historical image acquisition data as output of training data, and is obtained by training a YOLOv9 neural network; the present invention can quickly and accurately screen out areas where smoking behavior may exist from current image acquisition data through a smoking area detection model constructed based on a YOLOv9 neural network, and can continuously track suspected smoking behavior specifically through a ByteTrack target tracking algorithm, and can stably track smoking actions in complex dynamic scenes, even if the target moves or temporarily moves in the picture.
[0085] Other features of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.
[0086] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0087] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0088] Figure 1 A schematic diagram of a process flow of a smoking detection method based on deep learning provided by the present invention;
[0089] Figure 2A schematic diagram of the training process of a smoking area detection model in a smoking detection method based on deep learning provided by the present invention;
[0090] Figure 3 A schematic diagram of the overall framework of a smoking detection method based on deep learning provided by the present invention;
[0091] Figure 4 A schematic diagram of the structural composition of a deep learning-based smoking detection system provided by the present invention. DETAILED DESCRIPTION
[0092] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0093] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "Multiple" means two or more, unless otherwise clearly and specifically defined. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0094] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
[0095] Embodiment 1:
[0096] The present invention provides a smoking detection method based on deep learning, such as Figure 1 As shown, including:
[0097] Step 1: Obtain image acquisition data of the no-smoking area to be identified;
[0098] Step 2: Based on the image acquisition data, using a pre-built smoking area detection model, determine whether there is a suspected smoking area in the image acquisition data;
[0099] Step 3: When there is a suspected smoking area in the image acquisition data, the ByteTrack target tracking algorithm is used to generate a smoking action trajectory;
[0100] Step 4: Determine whether there is any smoking behavior in the no-smoking area to be identified according to the smoking action trajectory;
[0101] The smoking area detection model uses historical image acquisition data as input of training data and actual annotation data of whether there is a smoking area in the historical image acquisition data as output of training data, and is obtained by training the YOLOv9 neural network.
[0102] The image acquisition data in the above step 1 is collected in real time by a video acquisition and processing system installed in the no-smoking area to be identified, and each installation location has a fixed position code to ensure that the corresponding location can be found according to the video acquisition system. The resolution can be 1920*1080, and the collected data is transmitted back to the server.
[0103] After obtaining the image acquisition data of the no-smoking area to be identified in the above step 1, the following steps are also included:
[0104] Performing data preprocessing on the image acquisition data of the no-smoking area to be identified to obtain preprocessed image acquisition data;
[0105] The data preprocessing includes one or more of the following: image enhancement, noise removal and image standardization.
[0106] In one implementation, the smoking area detection model in step 2 above may include the following construction process: Figure 2 As shown:
[0107] Initialize network parameters;
[0108] Using historical image acquisition data as input for training data;
[0109] Whether there is actual annotated data of a smoking area in the historical image acquisition data is used as output of training data;
[0110] Based on the input and output of the training data, the YOLOv9 neural network is trained, and the network parameters in the YOLOv9 neural network are updated by a gradient optimization method of an adaptive learning rate momentum factor;
[0111] Based on the updated network parameters, the YOLOv9 neural network is updated, and by calculating the cross entropy loss function of the updated YOLOv9 neural network, the network parameters in the updated YOLOv9 neural network are adjusted using a back propagation algorithm to obtain a smoking area detection model.
[0112] The above-mentioned historical image collection data may include: specific data on the presence and absence of smoking behavior in public places, and data annotation of the frame containing cigarettes near the hands, mouth and ears of a person. If there is cigarette in these three parts of the person, it is marked as smoking, and if there is no cigarette, it is marked as non-smoking. The collected annotated data containing cigarettes near the hands, mouth and ears are used to train a neural network detection model, such as yolov5, yolov9 and other models. The yolov5 network has fast inference speed, small memory usage, and is suitable for terminal device deployment. Yolov9 is the latest and most accurate target detection network, suitable for server-side deployment. According to the application scenario, the images of the parts that are difficult to detect and identify are supplemented and annotated to retrain and optimize the model to improve the detection accuracy so that the smoking detection model can detect smoking behavior more accurately. According to the model trained by the smoking behavior detection system, the presence of smoke is detected, and the pictures with smoking behavior are put into the folder '1', and the pictures without smoking behavior are put into the folder '0'. Then, the neural network classification model resnet50 is trained in a binary classification manner to complete the training of the model so that it has classification function. In a large number of 1920*1080 video images, smoking areas are of various sizes, some are very large, some are very small, and there may be 10*20 similar ones. The detection network may make false detections when detecting smoking areas, but the classification network input is the smoking area, eliminating the interference of various backgrounds, and can focus more on the characteristics of smoking, making up for the shortcomings of the detection network recognition and improving the accuracy of smoking recognition.
[0113] In one implementation, in the above step 3, when there is a suspected smoking area in the image acquisition data, the process of generating a smoking action trajectory using the ByteTrack target tracking algorithm may include:
[0114] When there is a suspected smoking area in the image acquisition data, the suspected smoking area is subjected to lossless enlargement of the local area by using an image segmentation and enhancement method to obtain a smoking enlargement area;
[0115] Based on the smoking expansion area, the smoking action trajectory is generated using the ByteTrack target tracking algorithm.
[0116] In this implementation, the process of generating a smoking action trajectory based on the smoking expansion area using the ByteTrack target tracking algorithm may include:
[0117] Performing feature extraction on each frame of image data corresponding to the smoking enlarged area to obtain image feature data;
[0118] The similarity between the image feature data of adjacent frames is calculated using the intersection-over-union ratio as a similarity measure;
[0119] Generate a smoking action trajectory using a ByteTrack target tracking algorithm according to the similarity between the image feature data of adjacent frames;
[0120] Wherein, the image feature data includes: appearance features and motion features;
[0121] The appearance features include: color features, texture features and contour features;
[0122] The motion features include: coordinate change features and speed change features. For example, by doubling the suspected cigarette areas near the hands, mouth, and ears in the surveillance video, and then using tracking algorithms such as ByteTrack, we can get several pictures of smoking trajectories. Then, each picture of the smoking trajectory is further confirmed by a classification network to determine whether it is smoking, eating, wiping the mouth, etc. Finally, each trajectory votes to determine whether it is smoking, thereby further improving the accuracy of smoking recognition.
[0123] In this implementation, the above-mentioned generation of the smoking action trajectory based on the similarity between the image feature data of adjacent frames using the ByteTrack target tracking algorithm may include:
[0124] According to the similarity between the image feature data of adjacent frames, selecting image feature data whose similarity is greater than a set threshold;
[0125] Using the ByteTrack algorithm, coordinate point matching is performed on the image feature data whose similarity is greater than a set threshold;
[0126] Connect the center points or key points of the successfully matched image feature data to form a continuous trajectory line;
[0127] The continuous trajectory lines are superimposed and displayed to generate a smoking action trajectory. In this implementation, the smoking area is formed into several action trajectories using a tracking algorithm such as ByteTrack. ByteTrack divides the smoking area into two parts, high score and low score, according to the image detection result of 0.5 threshold. The high score part usually corresponds to a more accurate detection result, while the low score part may contain some false detection or inaccurate detection targets. First, the initial matching is performed: the detection results in the high score are matched and tracked with the existing tracking trajectories. This step is usually performed using algorithms such as SORT. In this process, some trajectories will successfully match the detection results, but some trajectories may not be matched successfully. Then match again: For the trajectories that failed to match successfully in the initial matching, the ByteTrack algorithm will match them with the detection results in the low score again. The purpose of this is to make the best use of all detection results and improve the accuracy and robustness of tracking. Create and retain trajectories: For tracking trajectories that fail to match any detection results, the ByteTrack algorithm will retain these trajectories for a period of time (such as m frames) and wait for these trajectories to appear again before matching. For detection boxes that do not match the tracking track but have a high enough score, the ByteTrack algorithm will create a new tracking track. Through the ByteTrack tracking algorithm, the smoking area in the image is synthesized into an action track.
[0128] In one implementation, the process of determining whether there is smoking behavior in the to-be-identified no-smoking area according to the smoking action trajectory in the above step 4 may include:
[0129] Generate an action sequence and a time sequence according to the smoking action trajectory;
[0130] Determine whether smoke appears in the image acquisition data corresponding to the no-smoking area in the time series, and obtain a determination result;
[0131] Determining whether there is smoking behavior in the no-smoking area to be identified according to the action time sequence, the time sequence and the judgment result;
[0132] The action sequence includes one or more of the following: raising the hand to the mouth, moving an object with the fingertips, and mouth movements.
[0133] In this implementation, the multiple smoking area images in the trajectory are predicted by the smoking classification system to obtain the scores of whether there is smoking behavior, and the average of these scores is calculated to see whether it is greater than 0.5, and finally determine whether there is smoking behavior. Because the smoking area is usually very small, sometimes the resolution is less than 30*30. When detecting, it is difficult to distinguish whether there is smoking behavior in an area smaller than 30*30 from a large 1920*1080 image. However, if the smoking area is cropped out and the classification network is used to determine whether there is smoking, the accuracy will be greatly improved. The model is deployed in the smoking behavior recognition system, and each frame of the image collected by the surveillance camera is classified to determine whether there is smoking behavior in the area.
[0134] After determining whether there is smoking behavior in the no-smoking area to be identified according to the smoking action trajectory, the method further includes:
[0135] When it is determined that there is smoking in the no-smoking area to be identified, an alarm is automatically triggered, and the alarm information is pushed to the backend system through the http network. Once smoking is detected in the current area, the system immediately issues an alarm, which is generally a sound alarm, playing a pre-recorded alarm voice, and pushing relevant information to the current area manager. The information push system transmits data (image information and specific location captured by monitoring) to the backend system through the http network, which is convenient for the area manager to intervene quickly.
[0136] The present invention aims at the existing vision-based smoking detection methods that have low detection accuracy in complex environments. However, these methods often have problems such as slow response time, high false alarm rate and monitoring blind spots. A smoking detection method based on deep learning is proposed. The overall framework diagram is shown in the figure. Figure 3 As shown, by using the smoking area detection model built based on the YOLOv9 neural network, it is possible to quickly and accurately screen out areas where smoking behavior may exist from the current image acquisition data, and by using the ByteTrack target tracking algorithm, it is possible to continuously track suspected smoking behavior, and it is possible to stably track smoking actions in complex dynamic scenes. Even if the target moves or is temporarily blocked in the picture, it can maintain tracking continuity, which helps to build a complete smoking action trajectory. The method of the present invention can identify smoking behavior by analyzing the visual features in the surveillance video, which not only improves the accuracy and real-time performance of the detection, but also greatly reduces the labor cost. Therefore, the use of the present invention can overcome the limitations of the prior art and provide a more accurate and reliable solution, which is of great significance for protecting public health and maintaining public environmental hygiene.
[0137] Embodiment 2:
[0138] The present invention provides a smoking detection system based on deep learning, such as Figure 4 As shown, including:
[0139] Data acquisition module: used to obtain image acquisition data of the no-smoking area to be identified;
[0140] Behavior judgment module: used to determine whether there is a suspected smoking area in the image acquisition data based on the image acquisition data and using a pre-built smoking area detection model;
[0141] Trajectory generation module: used to generate a smoking action trajectory using the ByteTrack target tracking algorithm when there is a suspected smoking area in the image acquisition data;
[0142] Behavior determination module: used for determining whether there is smoking behavior in the no-smoking area to be identified according to the smoking action trajectory;
[0143] Among them, the smoking area detection model in the behavior judgment module uses historical image acquisition data as input of training data, and uses actual annotation data of whether there is a smoking area in the historical image acquisition data as output of training data, and is obtained by training the YOLOv9 neural network.
[0144] The smoking area detection model in the above-mentioned behavior judgment module includes the following construction process:
[0145] Initializing network parameters of the EEG signal recognition model;
[0146] Using historical image acquisition data as input for training data;
[0147] Whether there is actual annotated data of a smoking area in the historical image acquisition data is used as output of training data;
[0148] Based on the input and output of the training data, the YOLOv9 neural network is trained, and the network parameters in the YOLOv9 neural network are updated by a gradient optimization method of an adaptive learning rate momentum factor;
[0149] Based on the updated network parameters, the YOLOv9 neural network is updated, and by calculating the cross entropy loss function of the updated YOLOv9 neural network, the network parameters in the updated YOLOv9 neural network are adjusted using a back propagation algorithm to obtain a smoking area detection model.
[0150] Acquire image acquisition data of a no-smoking area to be identified;
[0151] Based on the image acquisition data, using a pre-built smoking area detection model, determining whether there is a suspected smoking area in the image acquisition data;
[0152] When there is a suspected smoking area in the image acquisition data, a ByteTrack target tracking algorithm is used to generate a smoking action trajectory;
[0153] Determining whether there is any smoking behavior in the no-smoking area to be identified according to the smoking action trajectory;
[0154] The smoking area detection model uses historical image acquisition data as input of training data and actual annotation data of whether there is a smoking area in the historical image acquisition data as output of training data, and is obtained by training the YOLOv9 neural network.
[0155] The above-mentioned trajectory generation module is specifically used for:
[0156] When there is a suspected smoking area in the image acquisition data, the suspected smoking area is subjected to lossless enlargement of the local area by using an image segmentation and enhancement method to obtain a smoking enlargement area;
[0157] Based on the smoking expansion area, the smoking action trajectory is generated using the ByteTrack target tracking algorithm.
[0158] The above-mentioned trajectory generation module is specifically used for:
[0159] Performing feature extraction on each frame of image data corresponding to the smoking enlarged area to obtain image feature data;
[0160] The similarity between the image feature data of adjacent frames is calculated using the intersection-over-union ratio as a similarity measure;
[0161] Generate a smoking action trajectory using a ByteTrack target tracking algorithm according to the similarity between the image feature data of adjacent frames;
[0162] Wherein, the image feature data includes: appearance features and motion features;
[0163] The appearance features include: color features, texture features and contour features;
[0164] The motion characteristics include: coordinate change characteristics and speed change characteristics.
[0165] The above-mentioned trajectory generation module generates a smoking action trajectory according to the similarity between the image feature data of adjacent frames using the ByteTrack target tracking algorithm, including:
[0166] According to the similarity between the image feature data of adjacent frames, selecting image feature data whose similarity is greater than a set threshold;
[0167] Using the ByteTrack algorithm, coordinate point matching is performed on the image feature data whose similarity is greater than a set threshold;
[0168] Connect the center points or key points of the successfully matched image feature data to form a continuous trajectory line;
[0169] The continuous trajectory lines are superimposed and displayed to generate a smoking action trajectory.
[0170] The above-mentioned behavior determination module is specifically used for:
[0171] Generate an action sequence and a time sequence according to the smoking action trajectory;
[0172] Determine whether smoke appears in the image acquisition data corresponding to the no-smoking area in the time series, and obtain a determination result;
[0173] Determining whether there is smoking behavior in the no-smoking area to be identified according to the action time sequence, the time sequence and the judgment result;
[0174] The action sequence includes one or more of the following: raising the hand to the mouth, moving an object with the fingertips, and mouth movements.
[0175] After the image acquisition data of the no-smoking area to be identified is acquired in the above-mentioned data acquisition module, the following is further included:
[0176] Performing data preprocessing on the image acquisition data of the no-smoking area to be identified to obtain preprocessed image acquisition data;
[0177] The data preprocessing includes one or more of the following: image enhancement, noise removal and image standardization.
[0178] After the above-mentioned behavior determination module determines whether there is a smoking behavior in the no-smoking area to be identified according to the smoking action trajectory, it also includes:
[0179] When it is determined that smoking behavior exists in the no-smoking area to be identified, an alarm is automatically triggered, and the alarm information is pushed to the background system through the http network.
[0180] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.
[0181] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0182] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0184] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A smoking detection method based on deep learning, It is characterized in that include: Acquire image acquisition data of a no-smoking area to be identified; Based on the image acquisition data, using a pre-built smoking area detection model, determining whether there is a suspected smoking area in the image acquisition data; When there is a suspected smoking area in the image acquisition data, a ByteTrack target tracking algorithm is used to generate a smoking action trajectory; Determining whether there is any smoking behavior in the no-smoking area to be identified according to the smoking action trajectory; The smoking area detection model uses historical image acquisition data as input of training data and actual annotation data of whether there is a smoking area in the historical image acquisition data as output of training data, and is obtained by training the YOLOv9 neural network.
2. The method according to claim 1, It is characterized in that The smoking area detection model includes the following construction process: Initialize network parameters; Using historical image acquisition data as input for training data; Whether there is actual annotated data of a smoking area in the historical image acquisition data is used as output of training data; Based on the input and output of the training data, the YOLOv9 neural network is trained, and the network parameters in the YOLOv9 neural network are updated by a gradient optimization method of an adaptive learning rate momentum factor; Based on the updated network parameters, the YOLOv9 neural network is updated, and by calculating the cross entropy loss function of the updated YOLOv9 neural network, the network parameters in the updated YOLOv9 neural network are adjusted using a back propagation algorithm to obtain a smoking area detection model.
3. The method according to claim 1, It is characterized in that When there is a suspected smoking area in the image acquisition data, the ByteTrack target tracking algorithm is used to generate a smoking action trajectory, including: When there is a suspected smoking area in the image acquisition data, the suspected smoking area is subjected to lossless enlargement of the local area by using an image segmentation and enhancement method to obtain a smoking enlargement area; Based on the smoking expansion area, the smoking action trajectory is generated using the ByteTrack target tracking algorithm.
4. The method according to claim 3, It is characterized in that The method of generating a smoking action trajectory based on the smoking expansion area and using the ByteTrack target tracking algorithm includes: Performing feature extraction on each frame of image data corresponding to the smoking enlarged area to obtain image feature data; The similarity between the image feature data of adjacent frames is calculated using the intersection-over-union ratio as a similarity measure; Generate a smoking action trajectory using a ByteTrack target tracking algorithm according to the similarity between the image feature data of adjacent frames; Wherein, the image feature data includes: appearance features and motion features; The appearance features include: color features, texture features and contour features; The motion characteristics include: coordinate change characteristics and speed change characteristics.
5. The method according to claim 4, It is characterized in that The method of generating a smoking action trajectory by using a ByteTrack target tracking algorithm according to the similarity between the image feature data of adjacent frames includes: According to the similarity between the image feature data of adjacent frames, selecting image feature data whose similarity is greater than a set threshold; Using the ByteTrack algorithm, coordinate point matching is performed on the image feature data whose similarity is greater than a set threshold; Connect the center points or key points of the successfully matched image feature data to form a continuous trajectory line; The continuous trajectory lines are superimposed and displayed to generate a smoking action trajectory.
6. The method according to claim 1, It is characterized in that The determining, based on the smoking action trajectory, whether there is smoking behavior in the no-smoking area to be identified includes: Generate an action sequence and a time sequence according to the smoking action trajectory; Determine whether smoke appears in the image acquisition data corresponding to the no-smoking area in the time series, and obtain a determination result; Determining whether there is smoking behavior in the no-smoking area to be identified according to the action time sequence, the time sequence and the judgment result; The action sequence includes one or more of the following: raising the hand to the mouth, moving an object with the fingertips, and mouth movements.
7. The method according to claim 1, It is characterized in that After acquiring the image acquisition data of the no-smoking area to be identified, and before obtaining whether there is a suspected smoking area in the image acquisition data by using a pre-built smoking area detection model based on the image acquisition data, the method further includes: Performing data preprocessing on the image acquisition data of the no-smoking area to be identified to obtain preprocessed image acquisition data; The data preprocessing includes one or more of the following: image enhancement, noise removal and image standardization.
8. The method according to claim 1, It is characterized in that After determining whether there is smoking behavior in the no-smoking area to be identified according to the smoking action trajectory, the method further includes: When it is determined that smoking behavior exists in the no-smoking area to be identified, an alarm is automatically triggered, and the alarm information is pushed to the background system through the http network.
9. A smoking detection system based on deep learning, It is characterized in that include: Data acquisition module: used to obtain image acquisition data of the no-smoking area to be identified; Behavior judgment module: used to determine whether there is a suspected smoking area in the image acquisition data based on the image acquisition data and using a pre-built smoking area detection model; Trajectory generation module: used to generate a smoking action trajectory using the ByteTrack target tracking algorithm when there is a suspected smoking area in the image acquisition data; Behavior determination module: used for determining whether there is smoking behavior in the no-smoking area to be identified according to the smoking action trajectory; Among them, the smoking area detection model in the behavior judgment module uses historical image acquisition data as the input of training data, and uses the actual annotation data of whether there is a smoking area in the historical image acquisition data as the output of training data, and is obtained by training the YOLOv9 neural network.
10. The system of claim 9, It is characterized in that The smoking area detection model in the behavior judgment module includes the following construction process: Initializing network parameters of the EEG signal recognition model; Using historical image acquisition data as input for training data; Whether there is actual annotated data of a smoking area in the historical image acquisition data is used as output of training data; Based on the input and output of the training data, the YOLOv9 neural network is trained, and the network parameters in the YOLOv9 neural network are updated by a gradient optimization method of an adaptive learning rate momentum factor; Based on the updated network parameters, the YOLOv9 neural network is updated, and by calculating the cross entropy loss function of the updated YOLOv9 neural network, the network parameters in the updated YOLOv9 neural network are adjusted using a back propagation algorithm to obtain a smoking area detection model.
Citation Information
Patent Citations
Smoking detection method and system and computer medium
CN112766149A
Smoking behavior detection method, electronic equipment and readable storage medium
CN113505707A
Cited By
Urban carbon emission prediction and control method and system based on deep learning
CN119272945A
A method and system for urban carbon emission prediction and control based on deep learning
CN119272945B
Target detection and image segmentation-based intelligent detection method for smoking behavior in smoking-forbidden place
CN122290207A
