Intelligent detection method for smoking behavior in no-smoking place based on target detection and image segmentation

Abstract

This invention discloses an intelligent detection method for smoking behavior in no-smoking areas based on target detection and image segmentation. The method involves scaling up the image / video frames of the no-smoking area to be detected, and then preprocessing the scaled images using a standardized approach. A YOLO11 target detection model is then constructed to detect smoking-related targets in the preprocessed no-smoking area images, generating detection images. Rectangular regions containing suspected cigarettes and related human hands are cropped from the detection images and input into a SAM image segmentation model for optimization. Simultaneously, the center coordinates of the suspected cigarettes are also input into the model, achieving pixel-level precise segmentation of the cigarettes, hands, and smoke, extracting accurate target contour features. Finally, by combining spatial relationships and feature judgment rules, the final identification and determination of smoking behavior is completed.

Description

Technical Field

[0001] This invention relates to computer vision technology, specifically to an intelligent detection method for smoking behavior in no-smoking areas based on target detection and image segmentation. Background Technology

[0002] Secondhand smoke exposure significantly increases the risk of developing ischemic heart disease, lung cancer, diabetes, and other diseases. Currently, the supervision of smoking behavior in smoke-free areas still relies mainly on manual patrols and public reports. This model has many practical problems, such as many blind spots in supervision coverage, low real-time response efficiency, difficulty in obtaining evidence of smoking behavior, which can easily lead to on-site conflicts, and high long-term labor costs. It can no longer meet the needs of large-scale and refined public management.

[0003] With the development of deep learning and computer vision technologies, behavior recognition technologies based on object detection and image segmentation have been gradually applied to the field of smoking detection. However, existing technical solutions still have many core defects: single object detection models have low accuracy in recognizing small objects such as cigarettes, and are prone to misidentifying similar objects such as pens, lollipops, and fingers as cigarettes, resulting in a high false alarm rate; traditional image segmentation models have large inference delays and lack effective coordination mechanisms with detection models, which cannot meet the real-time requirements of monitoring scenarios; smoking behavior determination relies only on single visual features and does not combine spatial relationships and temporal behavioral features between targets, resulting in poor robustness in complex environments such as occlusion, low light, and dense crowds; at the same time, existing methods lack standardized target region optimization processes and fail to extract sufficient correlation features between cigarettes and human hands and mouths, further affecting detection accuracy.

[0004] Existing patents for smoking behavior detection all have significant technical limitations and are difficult to meet the actual needs of smoking ban supervision. Specific shortcomings are as follows: For example, Chinese patent applications (CN112115775A, CN114758303A) only adopt a two-stage architecture of "detection + classification" without introducing image segmentation technology and lacking pixel-level feature extraction capabilities, resulting in a high risk of false detection of similar objects; Chinese patent applications (such as CN114445849A) can only achieve single-target segmentation of cigarettes and cannot combine multi-target spatial relationships to complete behavior-level judgment, resulting in insufficient scenario adaptability; other existing solutions (such as Chinese patent application CN116189288B) use segmentation only as an auxiliary tool for detection, failing to form a complete technical closed loop, and none of them fully consider the practical adaptation needs of smoking ban supervision. Summary of the Invention

[0005] Purpose of the Invention: The purpose of this invention is to address the shortcomings of existing technologies and provide an intelligent detection method for smoking behavior in smoke-free areas based on target detection and image segmentation. This invention integrates the high-speed small target detection advantages of the YOLO11 model with the pixel-level fine segmentation capabilities of the SAM module to construct a full-process intelligent detection method for smoking behavior that includes preprocessing, detection, target optimization, segmentation, and joint judgment. Through standardized image processing procedures, targeted feature selection rules, and multi-dimensional joint judgment mechanisms, it achieves accurate and real-time identification of smoking behavior, effectively reducing false alarm and false negative rates. At the same time, it completes automatic early warning of violations and standardized evidence retention, providing an efficient and practical technical solution for the supervision of smoke-free areas.

[0006] Technical solution: The present invention provides an intelligent detection method for smoking behavior in no-smoking areas based on target detection and image segmentation, comprising the following steps:

[0007] Step 1: Standardize and preprocess the images / video frames of the no-smoking areas to be detected, unify the image size and complete pixel normalization to obtain the preprocessed images to be detected;

[0008] Step 2: Construct and train a target detection model based on the YOLO11 model. Input the preprocessed image to be detected into the target detection model to detect smoking-related targets, and obtain detection images labeled with human body, suspected cigarette, and smoke regions. Output the rectangular bounding box and center coordinates of each target. The target detection model is a dedicated detection model to enhance the detection capability of small targets, including an improved CSPDarknet backbone network, an adaptive spatial feature pyramid neck network, and a lightweight detection head network. Among them, the improved CSPDarknet backbone network extracts visual features of different levels from the input image. Shallow features are used to represent target edge and texture information, and deep features are used to represent target semantic information. The adaptive spatial feature pyramid neck network introduces an attention mechanism to fuse the small target detail information in shallow features and the large target semantic information in deep features, generating feature maps of three scales to adapt to the detection needs of small cigarette targets, medium targets in the hand, and large targets in the smoke, respectively, realizing multi-scale feature fusion and enhancing the model's feature extraction capability for small cigarette targets. The lightweight detection head network predicts 3 at each grid point of the three scale feature maps based on a preset anchor box. The system takes class information, namely bounding box offset, target confidence, and class probability. Then, it uses non-maximum suppression (NMS) to filter out duplicate predicted boxes, and finally outputs the target detection results with a confidence of not less than a preset threshold of 0.7.

[0009] Step 3: Use the ROI optimization module to crop the target area of ​​suspected cigarettes and related human hands in the detection image obtained in Step 2, and use image enhancement and noise reduction to obtain the optimized target area image;

[0010] Step 4: Construct the SAM image segmentation module. Input the optimized target area image and the center coordinates of the suspected cigarette as prompts into the SAM image segmentation module to complete pixel-level instance segmentation and accurately delineate the contour features of the cigarette, hand, and smoke. The SAM image segmentation module includes an image encoder, a prompt encoder, and a mask decoder. The image encoder converts the target area image into image embedding features. The prompt encoder uses the center coordinates of the suspected cigarette as a point prompt and the target bounding box as a box prompt, performs position encoding, and generates prompt embedding features. The mask decoder fuses the image embedding and prompt embedding features to generate multiple segmentation masks and labels the confidence level, finally outputting the pixel-level contours of the cigarette, hand, and smoke.

[0011] Step 5: Using the behavior judgment module, combined with spatial relationship analysis and behavior judgment rules, determine whether the segmented target features constitute smoking behavior. If it is determined to be smoking behavior, an alert is triggered and evidence is retained; otherwise, a result indicating no smoking behavior is output. Here, a three-dimensional joint judgment system based on pixel-level spatial relationships, visual features, and temporal behavior is constructed. Based on the precise contour output by the SAM image segmentation module, the relationship between the hand and the cigarette, and the spatial distance between the cigarette and the mouth are calculated to verify the core smoking action. This is different from the rough bounding box coordinate judgment of existing technologies, and the accuracy is greatly improved. Finally, a multi-dimensional comprehensive judgment scoring formula is used, which integrates detection confidence, segmentation effectiveness, and continuous frame temporal trajectory features to quantitatively judge smoking behavior. The weights can be flexibly adjusted to adapt to different regulatory scenarios, and the false alarm rate is controlled within 5% in actual tests.

[0012] This invention can preserve the edge features of core targets such as cigarettes and hands, making the target outline clearer and the details more accurate. The target detection and segmentation collaborative model adopted greatly reduces the dependence on high-quality smoking behavior images. At the same time, it overcomes the technical difficulties of locating small targets such as cigarettes and determining behavior in complex environments. It has extremely high recognition accuracy and strong scene robustness, and has low single-frame processing latency, which can be adapted to the deployment needs of mobile terminals, edge devices and other scenarios.

[0013] Furthermore, the detailed process of step 1 is as follows:

[0014] First, the image to be detected is scaled to a uniform size of 640×640 pixels (to precisely match the 32x downsampling rate of YOLO11, balancing the detection accuracy of small cigarette targets with the model's inference speed); then, the RGB three-channel pixel values ​​are standardized so that the pixel values ​​fall within the [0,1] range (to adapt to the uneven lighting and large differences in equipment parameters in no-smoking monitoring scenarios, while unifying the input specifications of the YOLO11 network model and the SAM image segmentation module to avoid feature bias). The standardization formula is:

[0015] ;

[0016] in, These are the original pixel values. The minimum value of the image pixels. The maximum value of the image pixels. These are the standardized pixel values;

[0017] Finally, the images obtained from the preprocessing were used to construct a smoking behavior dataset, which was then divided into a training set, a validation set, and a test set in a 7:2:1 ratio. The batch size was set to 16, and the maximum number of iterations was set to 10,000. The model was then trained and fine-tuned.

[0018] Furthermore, the specific method for step 2, which uses the YOLO11 model to process the target detection model and obtain the bounding box offset, target confidence, and class probability, is as follows:

[0019] Step 2.1: Calculate the bounding box coordinates (based on anchor box offset regression). The YOLO11 network model uses an anchor box combined with an offset regression mechanism. Instead of directly predicting the coordinates, it predicts the adjustment amount relative to the preset anchor box. The calculation process is as follows:

[0020] First, anchor frames are preset. For feature maps of three scales, nine sets of anchor frames with different widths and heights are predefined using K-Means clustering (small-scale feature maps are paired with large anchor frames, and large-scale feature maps are paired with small anchor frames) to adapt to different sizes of cigarettes, human bodies, and smoke. The offset is predicted, that is, the detection head predicts four offsets tx, ty, tw, th at each grid point. tx, ty, tw, th represent the center offset relative to the upper left corner of the grid and the scaling logarithm of the anchor frame width and height, respectively.

[0021] Step 2.2: Calculate the target confidence score. The confidence score represents the probability that the target is contained within the predicted bounding box. The raw value output by the detector head is mapped to the [0,1] interval using the Sigmoid function. The formula is as follows: ;

[0022] in, The raw values ​​for the target prediction by the detection head, and the raw values ​​in the target confidence calculation formula. The target confidence and the four offsets (tx, ty, tw, th) of bounding box regression are both parallel and independent prediction branch outputs of the YOLO11 lightweight detection head. tx, ty, tw, and th are only used to calculate the predicted bounding box coordinates to complete the target spatial localization.

[0023] During training, supervision is provided by the IoU between the predicted bounding boxes and the ground truth bounding boxes. This ensures that the higher the confidence level, the more likely a real target is to exist within the box;

[0024] Step 2.3: Calculate the category probabilities. For the three categories of human body, cigarette, and smoke, the detector head predicts the raw score for each category. After normalization using the Softmax function, the probabilities of each category are obtained (ensuring that the sum of the probabilities is 1). The formula is as follows: ;

[0025] in, The original score for the i-th class predicted by the detection head. The probability of the i-th type in the final output (e.g., the probability of a human body is 0.95, the probability of a cigarette is 0.03, and the probability of smoke is 0.02).

[0026] The improved CSPDarknet backbone network of this invention extracts visual features at different levels from images (shallow layers capture edges and textures, deep layers capture semantics), reducing the loss of detailed features of small targets; the neck network introduces an attention mechanism to enhance the effective features of cigarettes, hands, and smoke, and suppress background interference; the lightweight detection head simultaneously adapts to the multi-scale detection needs of cigarettes (small targets), hands (medium targets), and smoke (large targets), and simultaneously detects three related targets: human body, suspected cigarettes, and smoke, rather than a single cigarette, thus identifying the core subject of smoking behavior in advance; a screening threshold with a confidence level ≥ 0.7 is set to filter out low-confidence interference targets in advance, reducing invalid computation.

[0027] Furthermore, for the constructed object detection model based on the YOLO11 model, the AdamW optimizer combined with a cosine annealing learning rate strategy is used for training and loss optimization. The multi-task loss function includes:

[0028] The bounding box regression loss uses CIoU loss, which considers the overlap area, center distance, and aspect ratio of the bounding boxes to make the predicted boxes fit the ground truth as closely as possible. The confidence loss uses binary cross-entropy loss to distinguish between positive sample boxes containing targets and negative sample boxes without targets, thus improving the reliability of the confidence score. The class classification loss uses cross-entropy loss to supervise the model to correctly distinguish between three types of targets: human body, cigarette, and smoke. Through training and iterative optimization, the model eventually learns to extract target features from images and accurately predict bounding boxes, confidence scores, and class probabilities.

[0029] The bounding box regression loss formula is as follows:

[0030] =1−IoU+ +αv;

[0031] The confidence loss formula is as follows:

[0032] = − ;

[0033] The category classification loss expression is as follows:

[0034] =− .

[0035] This invention employs the AdamW optimizer combined with a cosine annealing learning rate strategy to improve model convergence stability; it integrates CIoU bounding box regression loss and class confidence loss to significantly improve the localization accuracy of small targets such as cigarettes; it constructs a dedicated dataset for no-smoking scenarios to complete scenario-based training and fine-tuning; the trained model detects preprocessed images and outputs the bounding box coordinates, class probabilities, and confidence scores of human bodies, cigarettes, and smoke; and it filters suspected targets with confidence scores ≥0.7 for subsequent processing. Joint supervision is provided during training to ensure output accuracy.

[0036] The detailed method for further utilizing the ROI optimization module in step 3 to obtain the target region image is as follows:

[0037] Step 3.1: Using the bounding box of the suspected cigarette as the center, crop the associated region containing the human hand by extending it outward by 20% of the area to obtain the target region image. Setting the outward extension of 20% of the area is a comprehensive trade-off between the visual characteristics of smoking behavior, model input requirements, and computational efficiency. The core reasons are as follows:

[0038] a) Ensure complete coverage of the areas where hands and cigarettes are in contact;

[0039] The core visual feature of smoking behavior is the "hand holding the cigarette," but the "suspected cigarette bounding box" detected by YOLO11 usually only tightly frames the cigarette itself (a very small target, only a few to tens of pixels). If it is not expanded or the expansion range is too small, the crucial hand area will be directly cropped, causing subsequent SAM segmentation to fail to obtain the "hand-cigarette" interaction features, and spatial relationship determination will also fail due to the lack of hand contours. A 20% expansion range can include the hand holding the cigarette and some fingers in the cropped area in most scenarios, fully preserving the core interaction features of smoking behavior and providing necessary information for subsequent segmentation and determination.

[0040] b), Balance feature integrity and minimize background interference; If the expansion range is too large (e.g., > 30%): it will introduce too much irrelevant background (such as desktops, clothing, crowds, etc.), which will not only increase the computational load of SAM segmentation, reduce the inference speed, but also may cause segmentation misjudgment due to background interference (mis-segmenting background clutter as cigarette sticks / smoke); If the expansion range is too small (e.g., < 10%): it cannot stably cover the hand area, and the situation of "the hand being truncated" is likely to occur, destroying the integrity of the spatial relationship between the hand and the cigarette.

[0041] c), Adapt to the segmentation focus requirements of the SAM model. The SAM image segmentation module has higher segmentation accuracy for small targets + local areas. An expansion range of 20% can strictly confine the segmentation input within the core ROI centered on the cigarette stick and assisted by the hand, enabling the attention of SAM to focus preferentially on the areas related to the smoking behavior, avoiding redundant calculations for whole-image segmentation, and at the same time improving the pixel-level segmentation accuracy of the cigarette stick, hand, and smoke.

[0042] d), Based on the statistical empirical values of the smoking scenario, the ratio is based on the statistical results of a large number of surveillance images in no-smoking areas: In a normal smoking posture, the hand area holding the cigarette stick usually concentrates within the range of 15% - 25% outward expansion from the bounding box of the cigarette stick. As the median value, 20% can adapt to the vast majority of scenarios with different holding postures (such as pinching the cigarette, clamping the cigarette), different shooting angles (such as top view, front view), and different human distances, and has strong generalization ability.

[0043] Step 3.2, Feature screening and false detection elimination, that is, realizing the quantitative evaluation of target features through the confidence adjustment formula. The formula is:

[0044] ;

[0045] Among them, is the adjusted confidence, is the original detection confidence of the corresponding suspected cigarette stick target output by the YOLO11 object detection model, is the adjustment coefficient of various features (0 < ≤1), and n is the number of feature types participating in the evaluation (the number of shape feature types, the number of size feature types, the number of color feature types, the number of skin feature types);

[0046] For the shape feature coefficient: when the aspect ratio r of the cigarette stick satisfies 3 < r < 8, k = 1, otherwise k = 0.6 or 0.7;

[0047] For the size feature coefficient: when the target area S satisfies 20 < S < 2000 pixels, k = 1, otherwise k = 0.5 or 0.8;

[0048] For the color feature coefficient: k=1 when the proportion of white / yellow in the cigarette meets the threshold, otherwise k=0.8;

[0049] For the false detection characteristic coefficient: when a suspected interference object such as a finger / earring is detected, k = 0.3 or 0.4;

[0050] Step 3.3: Perform a light enhancement on the denoised image by random rotation (±30°) and brightness adjustment (±20%) to improve the model's adaptability to complex lighting and poses.

[0051] This invention abandons the commonly used bounding box (bbox) tooltip in existing technologies and uses the cigarette center coordinates output by the YOLO11 network model as the core point tooltip, combined with auxiliary box tooltips, to accurately anchor the small cigarette target, fundamentally solving the industry pain point of missegmentation and missed segmentation of small targets; multi-target synchronous segmentation: it achieves synchronous pixel-level instance segmentation of three types of targets with greatly different features: cigarette, hand, and smoke, while covering rigid small targets and semi-transparent weak feature targets, extracting complete smoking behavior-related contour features, quantitatively evaluating the matching degree between the mask and the detection box, eliminating invalid segmentation results, and forming a two-way verification closed loop of detection and segmentation, further reducing the false detection rate.

[0052] The specific details of obtaining the corresponding contour features using the SAM image segmentation module in step 4 are as follows:

[0053] Step 4.1: The image encoder uses a pre-trained ViT-H model to convert the target region image into image embedding features. Here, the ViT-H model structure is kept unchanged. The cropped "hand-cigarette" associated ROI map is used as input to strictly limit the processing range to the core area of ​​smoking behavior. This reduces background computation redundancy and allows the attention mechanism to focus on the feature extraction of cigarettes, hands, and smoke.

[0054] Step 4.2: The prompt encoder uses the center coordinates of the suspected cigarette as a point prompt and the target bounding box as a bounding box prompt, and generates a prompt embedding feature after positional encoding. Instead of using the existing method of only using bounding box prompts, it innovatively adopts a dual-prompt combination of the suspected cigarette's center coordinates as a point prompt and the target bounding box as an auxiliary prompt. Specifically, the cigarette's center coordinates output by YOLO11 are used as the core point prompt to accurately anchor the small target; the bounding box of the associated region is used as an auxiliary bounding box prompt to constrain the segmentation range. The prompt encoder performs positional encoding on both types of prompts separately and then fuses them into a unified prompt embedding feature, driving the mask decoder to focus on the "hand-cigarette" core region.

[0055] Step 4.3: The mask decoder fuses image embedding and cue embedding features to generate multiple segmentation masks and labels them with confidence levels. Masks with confidence levels ≥ 0.8 are selected as the final segmentation results, outputting pixel-level contours of cigarettes, hands, and smoke. The original feature fusion and multi-mask generation structure is retained, and the output logic is optimized for smoking detection scenarios to enable the simultaneous generation of segmentation masks for three types of targets with significantly different features: cigarettes, hands, and smoke. A screening threshold of confidence level ≥ 0.8 is set. A new segmentation coverage quantification evaluation mechanism (formula Csam = Smask / Sbbox × 100%) is added to quantify the matching degree between the mask and the detection box. When Csam < 30%, the segmentation is deemed invalid, forming a two-way verification closed loop of "detection-segmentation".

[0056] Step 4.4: The SAM image segmentation module uses a segmentation coverage formula to evaluate the matching degree between the mask and the detection box. The formula is: ;

[0057] in, SAM segmentation coverage, The effective mask pixel area generated for SAM (area with mask value > 0.5). The pixel area of ​​the bounding box in YOLO11, and the segmentation coverage, are used as the core features for determining smoking behavior. If the segmentation is invalid, the confidence level of the corresponding target is reduced.

[0058] Furthermore, the detailed method for determining smoking behavior using the behavior determination module in step 5 is as follows:

[0059] Step 5.1, Spatial Relationship Determination: The intersection-union ratio (IoU) between the cigarette outline and the human hand outline is ≥0.3, and the pixel distance between the cigarette outline and the human mouth area is ≤50 pixels;

[0060] The formula for calculating the intersection-union ratio is: ;

[0061] In the above formula, This represents the intersection pixel area of ​​the cigarette mask and the hand mask, which is the total number of pixels that are simultaneously 1 in both masks. The coordinates of the pixels in the mask are calculated using the following formula:

[0062] ;

[0063] The area of ​​the union of the cigarette mask and the hand mask is the total number of pixels where at least one of the two masks is 1. The formula for this is:

[0064] ;

[0065] in, These are the pixel coordinates in the mask. and Represent the coordinates of the cigarette and the hand, respectively. The mask value at the location (which can be 0 or 1).

[0066] Step 5.2, Feature Determination: The segmentation result contains a clear cigarette outline, or a smoke outline that is connected to the cigarette outline;

[0067] Step 5.3, the method according to claim 1, characterized in that, the joint decision-making in step (5) uses a comprehensive judgment score formula to accurately identify smoking behavior, the formula being:

[0068] ;

[0069] in, The overall score is determined by a comprehensive assessment (out of 100). , and These are the weighting coefficients. The default values ​​are 0.5, 0.3, and 0.2 respectively, where T is the temporal behavior feature score (whether the target has a motion trajectory of raising a cigarette to its mouth in consecutive frames, with a full score of 100). When the comprehensive judgment score S≥60, it is determined that smoking behavior has been detected, triggering an alert and reporting; otherwise, it is determined that there is no smoking behavior.

[0070] Step 5.4: If the above two judgment conditions are met, it is determined to be smoking behavior. The system will automatically capture the image / video clip containing the behavior, add a timestamp and GPS location information to complete the evidence preservation, and send a warning message to the administrator through APP push, SMS and other channels.

[0071] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0072] The intelligent detection method for smoking behavior in no-smoking areas based on target detection and image segmentation described in this invention has the following significant advantages compared to existing technologies:

[0073] 1. High detection accuracy and low false alarm rate: By improving the YOLO11 model, the detection capability of small cigarette targets is enhanced. Combined with the dual filtering mechanism of feature selection and segmentation effect verification, interference objects such as fingers and pens are effectively eliminated. At the same time, a multi-dimensional joint judgment mechanism is constructed, which integrates spatial relationship, visual features and temporal behavior features to achieve accurate judgment of smoking behavior. The target recognition accuracy is no less than 90%, and the false alarm rate is controlled within 5%.

[0074] 2. High real-time performance and good adaptability: It achieves efficient collaboration between YOLO11 and SAM models, uses detection results as prompts to drive the segmentation model, greatly improves segmentation efficiency, and controls the single-frame processing latency within 200ms to meet the real-time requirements of monitoring scenarios. At the same time, the model supports lightweight processing and can be deployed on mobile devices and edge computing devices, adapting to the monitoring needs of various no-smoking areas.

[0075] 3. Robust and adaptable to various environments: Through standardized image preprocessing and targeted image enhancement strategies, the model can maintain stable detection performance in complex environments such as occlusion, low light, and dense crowds. It also supports smoking behavior detection under multiple scenes and lighting conditions, and is suitable for various non-smoking public places such as schools, shopping malls, and office buildings.

[0076] 4. High regulatory efficiency and standardized evidence collection: The entire process of "detection-judgment-early warning-evidence collection" is automated, which can replace more than 60% of the workload of manual inspections. It enables 24-hour uninterrupted intelligent monitoring of smoke-free areas, and automatically completes the retention of violation evidence with timestamps and location information, providing objective and effective evidence support for tobacco control law enforcement, avoiding on-site conflicts caused by manual evidence collection, and improving the standardization and efficiency of tobacco control supervision.

[0077] 5. Highly scalable and easy to implement: The method adopts a modular design, with each link independent and linked through standardized interfaces. The model parameters and judgment rules can be flexibly adjusted according to actual regulatory needs. At the same time, no complex hardware deployment is required, and it can be seamlessly integrated with existing monitoring systems, making it highly valuable for implementation and promotion. Attached Figure Description

[0078] Figure 1 This is an overall flowchart of the method of the present invention;

[0079] Figure 2 This is a diagram showing the positive example detection results of smoking behavior using the enhanced segmented version of YOLO11+SAM in the implementation example.

[0080] Figure 3 This is a diagram showing the SAM segmentation coverage effect of a positive example of smoking behavior using YOLO11+SAM in the embodiment.

[0081] Figure 4 This is a flowchart of the SAM image segmentation workflow of the present invention;

[0082] Figure 5 This is a magnified segmentation effect of the target region in the YOLO11+SAM smoking behavior positive example.

[0083] Figure 6 This is a graph showing the negative example detection results of smoking behavior using YOLO11+SAM in the implementation example;

[0084] Figure 7 Example Figure 2 Structured log output corresponding to positive example detection scenarios;

[0085] Figure 8 for Figure 6 Structured log output for negative example detection scenarios. Detailed Implementation

[0086] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0087] like Figure 1 and Figure 2 As shown, the intelligent detection method for smoking behavior in smoke-free areas based on target detection and image segmentation of the present invention includes the following steps:

[0088] Step 1: Standardize and preprocess the images / video frames of the no-smoking areas to be detected, unify the image size and complete pixel normalization to obtain the preprocessed images to be detected;

[0089] Step 2: Construct and train a target detection model based on the YOLO11 model. Input the preprocessed image to be detected into the target detection model to detect smoking-related targets. Obtain the detection images labeled with human body, suspected cigarette, and smoke area, and output the rectangular bounding box and center coordinate value of each target.

[0090] The target detection model comprises an improved CSPDarknet backbone network, an adaptive spatial feature pyramid neck network, and a lightweight detection head network. The improved CSPDarknet backbone network extracts visual features at different levels from the image. The pyramid neck network introduces an attention mechanism to fuse shallow small target details with deep large target semantics, generating feature maps at three scales to achieve multi-scale feature fusion. The lightweight detection head network simultaneously predicts the bounding box offset, target confidence, and class probability at each grid point of the feature maps at the three scales based on preset anchor boxes. Then, non-maximum suppression (NMS) is used to output the desired target result.

[0091] Step 3: Use the ROI optimization module to crop the target area of ​​suspected cigarettes and related human hands in the detection image obtained in Step 2, and use image enhancement and noise reduction to obtain the optimized target area image;

[0092] Step 4: Construct a SAM image segmentation module. Input the optimized target region image and the center coordinates of the suspected cigarette as prompts into the SAM image segmentation module to complete pixel-level instance segmentation and output the contour features of the cigarette, hand, and smoke. The SAM image segmentation module includes an image encoder, a prompt encoder, and a mask decoder. The image encoder converts the target region image into image embedding features. The prompt encoder uses the center coordinates of the suspected cigarette as a point prompt and the target bounding box as a box prompt, performs position encoding, and generates prompt embedding features. The mask decoder fuses the image embedding and prompt embedding features to generate multiple segmentation masks and labels the confidence level, finally outputting the pixel-level contours of the cigarette, hand, and smoke.

[0093] Step 5: Use the behavior judgment module to combine spatial relationship analysis, visual features and temporal behavior judgment rules, and calculate the relationship between the hand and the cigarette and the spatial distance between the cigarette and the mouth. In this way, determine whether the target contour features segmented in Step 4 constitute smoking behavior. If it is determined to be smoking behavior, trigger an alert and complete the evidence retention. Otherwise, output the result of no smoking behavior.

[0094] The detailed process of step 1 in this embodiment is as follows:

[0095] First, the image to be detected is scaled to a uniform size of 640×640 pixels;

[0096] Then, the RGB three-channel pixel values ​​are standardized so that the pixel values ​​fall within the [0,1] interval. The standardization formula is:

[0097] ;

[0098] in, The original pixel values ​​of the image. The minimum value of the image pixels. The maximum value of the image pixels. These are the standardized pixel values;

[0099] Finally, the images obtained from the preprocessing were used to construct a smoking behavior dataset, which was then divided into a training set, a validation set, and a test set in a 7:2:1 ratio. The batch size was set to 16, and the maximum number of iterations was set to 10,000. The model was then trained and fine-tuned.

[0100] In this embodiment, step 2, based on the YOLO11 model, processes the image to be detected using a target detection model to obtain the bounding box offset, target confidence score, and class probability. The specific method for this is as follows:

[0101] Step 2.1: Calculate the bounding box coordinates based on the anchor frame offset regression. First, preset the anchor frames. For three scale feature maps, use K-Means clustering to predefine 9 sets of anchor frames with different widths and heights to adapt to different sizes of cigarettes, human bodies, and smoke. Then, predict the offset, that is, use the detection head to predict 4 offsets tx, ty, tw, th at each grid point. tx, ty, tw, th represent the center offset relative to the upper left corner of the grid and the scaling logarithm of the anchor frame width and height, respectively.

[0102] Step 2.2: Calculate the target confidence score, which involves mapping the raw values ​​output by the detector head to the [0,1] interval using the Sigmoid function. The formula is as follows: ;

[0103] in, The target raw value predicted by the detection head;

[0104] Step 2.3: Calculate the category probabilities. For the three categories of human body, cigarette, and smoke, the detector head predicts the raw score for each category. After normalization using the Softmax function, the probability of each category is obtained, as shown in the following formula:

[0105] ;

[0106] in, The original score for the i-th class predicted by the detection head. This represents the final output probability of the i-th class.

[0107] In this embodiment, the AdamW optimizer combined with a cosine annealing learning rate strategy is used to train and optimize the object detection model based on the YOLO11 model. The multi-task loss function includes bounding box regression loss, confidence loss and category classification loss.

[0108] 1. Bounding box regression loss: CIoU loss:

[0109] =1−IoU+ +αv

[0110] 2. Confidence Loss: Binary Cross-Entropy Loss (BCE)

[0111] = −

[0112] 3. Classification loss: Cross-entropy loss (CE)

[0113] =− ;

[0114] The bounding box regression loss uses CIoU loss, and also considers the overlapping area, center distance, and aspect ratio of the bounding boxes to make the predicted boxes fit the real annotations as closely as possible.

[0115] The confidence loss uses binary cross-entropy loss to distinguish between positive bounding boxes containing targets and negative bounding boxes without targets. The category classification loss uses cross-entropy loss to supervise the model to correctly distinguish between three types of targets: human body, cigarette, and smoke.

[0116] Through training and iterative optimization using a multi-task loss function, the model eventually learns to extract target features from images and accurately predict bounding boxes, confidence levels, and class probabilities.

[0117] The detailed method for obtaining the target region image using the 3ROI optimization module in this embodiment is as follows:

[0118] Step 3.1: Using the suspected cigarette outline as the center, expand outward by 20% of the area to crop the associated region containing the human hand, thus obtaining the target region image;

[0119] Step 3.2, Feature Selection and False Detection Removal, involves quantitatively evaluating the target features using a confidence adjustment formula. The formula is as follows:

[0120] ;

[0121] ;

[0122] in, To adjust the confidence level, for This represents the original detection confidence score of the corresponding suspected cigarette target output by the YOLO11 target detection model. Here, n represents the adjustment coefficients for various features, and n is the number of feature types involved in the evaluation, including shape feature coefficients, size feature systems, color feature sparsity, and false detection feature coefficients. is the adjustment coefficient for various features, and n is the number of feature types involved in the evaluation, including shape feature coefficient, size feature system, color feature sparsity, and false detection feature coefficient;

[0123] Step 3.3: Perform a light enhancement on the denoised image by random rotation (±30°) and brightness adjustment (±20%).

[0124] In step 4 of this embodiment, the specific content of obtaining the corresponding contour features using the SAM image segmentation module is as follows:

[0125] Step 4.1: The image encoder uses a pre-trained ViT-H model to convert the target region image into image embedding features;

[0126] Step 4.2: The prompt encoder uses the center coordinates of the suspected cigarette as a point prompt and the target bounding box as a box prompt, and generates prompt embedding features after position encoding;

[0127] Step 4.3: The mask decoder fuses image embedding and cue embedding features to generate multiple segmentation masks and labels them with confidence levels. The mask with a confidence level ≥ 0.8 is selected as the final segmentation result, and the pixel-level contours of the cigarette, hand, and smoke are output.

[0128] Step 4.4: Quantify the SAM segmentation effect. The matching degree between the mask and the detection box is evaluated using the segmentation coverage formula: [Formula omitted for brevity]

[0129] ;

[0130] in, SAM segmentation coverage, The effective mask pixel area generated for SAM. The pixel area of ​​the YOLO11 detection bounding box;

[0131] Segmentation coverage is a core feature for determining smoking behavior. If the segmentation is invalid, the confidence level of the corresponding target is reduced.

[0132] The detailed method for determining smoking behavior using the behavior determination module in step 5 of this embodiment is as follows:

[0133] Step 5.1: Spatial relationship determination: The intersection-union ratio (IoU) between the cigarette outline and the human hand outline is ≥0.3, and the pixel distance between the cigarette outline and the human mouth area is ≤50 pixels.

[0134] The formula for calculating the intersection-union ratio is: IoU = ;

[0135] In the above formula, This represents the intersection pixel area of ​​the cigarette mask and the hand mask, which is the total number of pixels that are simultaneously 1 in both masks. The coordinates of the pixels in the mask are calculated using the following formula:

[0136] ;

[0137] The area of ​​the union of the cigarette mask and the hand mask is the total number of pixels where at least one of the two masks is 1. The formula for this is:

[0138] ;

[0139] in, These are the pixel coordinates in the mask. and Represent the coordinates of the cigarette and the hand, respectively. The mask value at that location;

[0140] Step 5.2, Feature determination: The segmentation result contains a clear cigarette outline or a smoke outline that is connected to the cigarette outline.

[0141] Step 5.3: Use a comprehensive judgment scoring formula to accurately identify smoking behavior. The formula is as follows:

[0142] ;

[0143] in, To determine the overall score, , , These are the weighting coefficients. , Scoring of temporal behavioral features;

[0144] When the comprehensive judgment score S≥60, smoking behavior is detected, triggering an alert and reporting; otherwise, no smoking behavior is detected.

[0145] Step 5.4: If the above two judgment conditions are met, it is determined to be smoking behavior. The system will automatically capture the image / video clip containing the behavior, add a timestamp and GPS location information to complete the evidence preservation, and send a warning message to the administrator through APP push, SMS and other channels.

[0146] The present invention has the following advantages:

[0147] 1. More accurate feature extraction: Based on the pixel-level accurate contour features of cigarette, hand and smoke output by SAM, the precise spatial position relationship, relative distance and pixel-level adjacency / overlap relationship of the three are extracted. It abandons the judgment method that only relies on the coarse bounding box (bbox) coordinates in the conventional solution, which can effectively solve the feature extraction shortcomings of the existing solution.

[0148] 2. More rigorous judgment logic: Combining customized spatial relationships and feature judgment rules to complete the final recognition of smoking behavior, the system achieves more accurate behavior judgment and significantly reduces the false judgment rate by verifying the pixel-level adjacency between the cigarette and the hand, the spatial distance constraint between the cigarette and the mouth, and the co-occurrence relationship between the smoke and the cigarette.

[0149] 3. More innovative technology implementation: Construct a closed-loop technology system covering the entire process of "target detection → precise segmentation → behavior determination", upgrade and adapt the general object perception capabilities of YOLO+SAM to behavior understanding capabilities specifically for smoking ban scenarios, while taking into account small target detection, anti-interference performance and engineering deployment adaptability. It has completed the creative transformation from a general technical tool to a scenario-based application, which can be directly adapted to the actual implementation needs of smoking ban supervision.

[0150] Figure 1 As shown, the present invention involves the following steps: image / video frame to be detected → image preprocessing (uniform size + pixel normalization) → YOLO11 target detection (output bounding box + center coordinates) → target region cropping + noise reduction and enhancement → SAM pixel-level segmentation (output contour mask) → smoking behavior determination (spatial relationship + features) → yes / smoking behavior → warning + evidence retention; no / no smoking behavior → continue detection.

[0151] Figure 2 The image shown is a positive example detection result of smoking behavior using the enhanced segmentation version YOLO11+SAM of this invention. First, there is the original video frame of an outdoor natural light scene (clearly showing the human smoking action, the cigarette itself, and the characteristics of the smoke, without obvious occlusion or lighting interference); and then the detection output image, with the identified cigarette target marked in green, and the target category simultaneously labeled as "cigarette", confidence level 0.69, and SAM segmentation coverage 23.9%. The system name "YOLO11+SAMSmokingDetection (Enhanced Segmentation)" is marked at the top of the screen, and the final detection conclusion "SMOKINGDETECTED: 1 target(s)" is marked in green at the bottom, completing the accurate identification of smoking behavior and visualization of the results.

[0152] Figure 3 The image shown is a diagram illustrating the SAM segmentation coverage effect of a positive example of smoking behavior using YOLO11+SAM according to the present invention. Figure 2 The original outdoor natural light video frames of the same scene fully preserve the human smoking action and the cigarette target; and for the segmentation and overlay output image, based on the target bounding box output by YOLO11, the SAM model is used to complete the pixel-level segmentation of the cigarette target, generate the cigarette contour mask and complete the overlay annotation, and simultaneously annotate SAMCoverage:23.9%, to achieve accurate segmentation of the cigarette target and quantification of the overlay effect.

[0153] like Figure 4 As shown, the SAM image segmentation workflow of this embodiment is as follows: Target region image → Image encoder (ViT-H) → Image embedding features; Cigarette center coordinates / boundary box → Cue encoder → Cue embedding features; Dual feature fusion → Mask decoder → Multi-mask generation (with confidence) → Optimal mask selection → Pixel-level contour output.

[0154] Figure 5 This is a magnified segmentation diagram of the target region of a positive example of smoking behavior using YOLO11+SAM according to the present invention.

[0155] yes Figure 2The image of the cigarette target area identified in the image is magnified, clearly showing the morphological features of the cigarette itself. After magnification and segmentation, the green box locks the cigarette target area, and the target category is labeled as "cigarette". The image fully presents the pixel-level segmentation outline of the cigarette output by the SAM model, realizing the detailed segmentation effect of small target cigarettes.

[0156] Figure 6 This image shows the negative example detection result of smoking behavior using YOLO11+SAM in this invention. It is an original video frame of an outdoor street scene under natural light (without real smoking behavior, no effective cigarette targets, only a normally standing human body and urban street background, and no smoking-related actions or features). The output image is labeled with the system name "YOLO11+SAMSmokingDetection (Enhanced Segmentation)" at the top and the final detection conclusion "NOSMOKINGDETECTED" in green text at the bottom, thus completing the accurate determination and visualization of the no-smoking behavior scene.

[0157] Figure 7 for Figure 2 The structured log output corresponding to the positive example detection scenario fully records the core information of the entire detection process: including image file path, detection time, detection result (smoking detected), and detection quantity 1 (original: 1, removed: 0); at the same time, for the identified single target, the target sequence number, category is "cigarette", confidence level is 0.688, target bounding box coordinates, segmentation area is 418 pixels, and SAM coverage is 23.9%, realizing the full-element structured retention and traceability of the detection results.

[0158] Figure 8 for Figure 6 The structured log output for the corresponding negative example detection scenario fully records the core information of the entire detection process: including image file path, detection time, detection result of no smoking detected, detection quantity of 0 (original: 0, removed: 0), and output of relevant information on no valid cigarette targets, realizing the structured recording and compliance verification of detection results in non-smoking behavior scenarios.

[0159] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. An intelligent detection method for smoking behavior in no-smoking areas based on target detection and image segmentation, characterized in that: Includes the following steps: Step 1: Standardize and preprocess the image frames / video frames of the no-smoking areas to be detected, unify the image size and complete pixel normalization to obtain the preprocessed image to be detected; Step 2: Construct and train a target detection model based on the YOLO11 model. Input the preprocessed image to be detected into the target detection model to detect smoking-related targets. Obtain the detection images labeled with human body, suspected cigarette, and smoke area, and output the rectangular bounding box and center coordinate value of each target. The target detection model comprises an improved CSPDarknet backbone network, an adaptive spatial feature pyramid neck network, and a lightweight detection head network. The improved CSPDarknet backbone network extracts visual features at different levels from the image. The pyramid neck network introduces an attention mechanism to fuse shallow small target details with deep large target semantics, generating feature maps at three scales to achieve multi-scale feature fusion. The lightweight detection head network simultaneously predicts the bounding box offset, target confidence, and class probability at each grid point of the feature maps at the three scales based on preset anchor boxes. Then, non-maximum suppression (NMS) is used to output the desired target result. Step 3: Use the ROI optimization module to crop the target area of ​​suspected cigarettes and related human hands in the detection image obtained in Step 2, and use image enhancement and noise reduction to obtain the optimized target area image; Step 4: Construct the SAM image segmentation module. Input the optimized target area image and the center coordinates of the suspected cigarette as prompts into the SAM image segmentation module to complete pixel-level instance segmentation and output the contour features of the cigarette, hand, and smoke. The SAM image segmentation module includes an image encoder, a prompt encoder, and a mask decoder. The image encoder converts the target area image into image embedding features. The prompt encoder uses the center coordinates of the suspected cigarette as a point prompt and the target bounding box as a box prompt, performs position encoding, and generates prompt embedding features. The mask decoder fuses the image embedding and prompt embedding features to generate multiple segmentation masks and labels the confidence level, finally outputting the pixel-level contours of the cigarette, hand, and smoke. Step 5: Use the behavior judgment module to combine spatial relationship analysis, visual features and temporal behavior judgment rules, and calculate the relationship between the hand and the cigarette and the spatial distance between the cigarette and the mouth. In this way, determine whether the target contour features segmented in Step 4 constitute smoking behavior. If it is determined to be smoking behavior, trigger an alert and complete the evidence retention. Otherwise, output the result of no smoking behavior.

2. The intelligent detection method for smoking behavior in no-smoking areas based on target detection and image segmentation according to claim 1, characterized in that: The detailed process of step 1 is as follows: First, the image to be detected is scaled to a uniform size of 640×640 pixels; Then, the RGB three-channel pixel values ​​are standardized so that the pixel values ​​fall within the [0,1] interval. The standardization formula is: ； in, The original pixel values ​​of the image. The minimum value of the image pixels. The maximum value of the image pixels. These are the standardized pixel values; Finally, the images obtained from the preprocessing were used to construct a smoking behavior dataset, which was then divided into a training set, a validation set, and a test set in a 7:2:1 ratio. The batch size was set to 16, and the maximum number of iterations was set to 10,000. The model was then trained and fine-tuned.

3. The empirical location smoking behavior detection method based on target detection and image segmentation according to claim 1, characterized in that, The specific method for step 2, which uses the YOLO11 model to process the image to be detected and obtain the bounding box offset, target confidence, and class probability, is as follows: Step 2.1: Calculate the bounding box coordinates based on the anchor frame offset regression. First, preset the anchor frames. For three scale feature maps, use K-Means clustering to predefine 9 sets of anchor frames with different widths and heights to adapt to different sizes of cigarettes, human bodies, and smoke. Then, predict the offset, that is, use the detection head to predict 4 offsets tx, ty, tw, th at each grid point. tx, ty, tw, th represent the center offset relative to the upper left corner of the grid and the scaling logarithm of the anchor frame width and height, respectively. Step 2.2: Calculate the target confidence score, which involves mapping the raw values ​​output by the detector head to the [0,1] interval using the Sigmoid function. The formula is as follows: ; in, The raw values ​​for the target prediction by the detection head; the raw values ​​in the target confidence calculation formula. Step 2.3: Calculate the category probabilities. For the three categories of human body, cigarette, and smoke, the detector head predicts the raw score of each category. After normalization using the Softmax function, the probability of each category is obtained, as shown in the following formula: ； in, The original score for the i-th class predicted by the detection head. This represents the final output probability of the i-th class.

4. The intelligent detection method for smoking behavior in no-smoking areas based on target detection and image segmentation according to claim 1 or 3, characterized in that, For the constructed object detection model based on YOLO11, the AdamW optimizer combined with the cosine annealing learning rate strategy is used for training and loss optimization. The multi-task loss function includes bounding box regression loss, confidence loss and category classification loss. The bounding box regression loss uses CIoU loss, and also considers the overlapping area, center distance, and aspect ratio of the bounding boxes to make the predicted boxes fit the real annotations as closely as possible. The confidence loss uses binary cross-entropy loss to distinguish between positive bounding boxes containing targets and negative bounding boxes without targets. The category classification loss uses cross-entropy loss to supervise the model to correctly distinguish between three types of targets: human body, cigarette, and smoke. Through training and iterative optimization using a multi-task loss function, the model eventually learns to extract target features from images and accurately predict bounding boxes, confidence levels, and class probabilities.

5. The intelligent detection method for smoking behavior in no-smoking areas based on target detection and image segmentation according to claim 1, characterized in that, The detailed method for obtaining the target region image using the ROI optimization module in step 3 is as follows: Step 3.1: Using the suspected cigarette outline as the center, expand outward by 20% of the area to crop the associated region containing the human hand, thus obtaining the target region image; Step 3.2, Feature Selection and False Detection Removal, involves quantitatively evaluating the target features using a confidence adjustment formula. The formula is as follows: ； ； in, To adjust the confidence level, The original detection confidence score of the corresponding suspected cigarette target output by the YOLO11 target detection model; is the adjustment coefficient for various features, and n is the number of feature types involved in the evaluation, including shape feature coefficient, size feature system, color feature sparsity, and false detection feature coefficient; Step 3.3: Perform a light enhancement on the denoised image by random rotation (±30°) and brightness adjustment (±20%).

6. The intelligent detection method for smoking behavior in no-smoking areas based on target detection and image segmentation according to claim 1, characterized in that, The specific details of obtaining the corresponding contour features using the SAM image segmentation module method in step 4 are as follows: Step 4.1: The image encoder uses a pre-trained ViT-H model to convert the target region image into image embedding features; Step 4.2: The prompt encoder uses the center coordinates of the suspected cigarette as a point prompt and the target bounding box as a box prompt, and generates prompt embedding features after position encoding; Step 4.3: The mask decoder fuses image embedding and cue embedding features to generate multiple segmentation masks and labels them with confidence levels. The mask with a confidence level ≥ 0.8 is selected as the final segmentation result, and the pixel-level contours of the cigarette, hand, and smoke are output. Step 4.4: Quantify the SAM segmentation effect. The matching degree between the mask and the detection box is evaluated using the segmentation coverage formula: [Formula omitted for brevity] ； in, SAM segmentation coverage, The effective mask pixel area generated for SAM. The pixel area of ​​the YOLO11 detection bounding box; Segmentation coverage is a core feature for determining smoking behavior. If the segmentation is invalid, the confidence level of the corresponding target is reduced.

7. The intelligent detection method for smoking behavior in no-smoking areas based on target detection and image segmentation according to claim 1, characterized in that, The detailed method for determining smoking behavior using the behavior determination module in step 5 is as follows: Step 5.1: Spatial relationship determination: The intersection-union ratio (IoU) between the cigarette outline and the human hand outline is ≥0.3, and the pixel distance between the cigarette outline and the human mouth area is ≤50 pixels. The formula for calculating the intersection-union ratio is: IoU = ; In the above formula, This represents the intersection pixel area of ​​the cigarette mask and the hand mask, which is the total number of pixels that are simultaneously 1 in both masks. The coordinates of the pixels in the mask are calculated using the following formula: ； The area of ​​the union of the cigarette mask and the hand mask is the total number of pixels where at least one of the two masks is 1. The formula for this is: ； in, These are the pixel coordinates in the mask. and Represent the coordinates of the cigarette and the hand, respectively. The mask value at that location; Step 5.2, Feature determination: The segmentation result contains a clear cigarette outline or a smoke outline that is connected to the cigarette outline. Step 5.3: Use a comprehensive judgment scoring formula to accurately identify smoking behavior. The formula is as follows: ； in, To determine the overall score, , , These are the weighting coefficients. , Scoring of temporal behavioral features; When the comprehensive judgment score S≥60, smoking behavior is detected, triggering an alert and reporting; otherwise, no smoking behavior is detected. Step 5.4: If the above two judgment conditions are met, it is determined to be smoking behavior. The system will automatically capture the image / video clip containing the behavior, add a timestamp and GPS location information to complete the evidence preservation, and send a warning message to the administrator through APP push, SMS and other channels.

Images