Smoking behavior detection system based on edge device
By using a lightweight spatiotemporal action detection model and TensorRT optimization, the problems of high computational load and complex deployment on edge devices are solved, achieving low latency and high accuracy in real-time smoking behavior detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-23
- Publication Date
- 2026-03-13
AI Technical Summary
Existing video Transformer models are computationally intensive on edge devices and have difficulty eliminating temporal redundancy, leading to issues with recognition accuracy and deployment complexity, making it difficult to achieve real-time smoking behavior detection.
A lightweight spatiotemporal motion detection model is adopted, which combines a multi-scale visual Transformer and a differentiable token selection mechanism. Redundant input data is identified and pruned by frame difference, and efficient deployment is achieved by combining TensorRT optimization.
Achieve real-time smoking behavior detection with low computational cost and low latency on edge devices, ensuring recognition accuracy and providing a complete system deployment solution.
Smart Images
Figure CN121661713A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision, specifically relating to a video smoking behavior detection method and system based on edge devices, and more particularly to a behavior recognition scheme that combines lightweight video spatiotemporal feature extraction with a differentiable temporal token selection mechanism, which is suitable for real-time smoking behavior detection in computing-constrained environments. Background Technology
[0002] In intelligent monitoring and behavior recognition technologies, the automatic identification of smoking behavior has wide-ranging applications, such as smoking control in public places and laboratory safety management. Deep learning-based video behavior recognition models have high accuracy in cloud environments, but they typically suffer from high computational demands and high GPU memory consumption, making them difficult to run in real time on low-power edge devices such as Jetson Nano.
[0003] While existing video Transformer models boast high recognition accuracy, the number of input tokens increases significantly with temporal and spatial resolution, leading to massive computational overhead for inference. Recent methods, primarily focused on pruning spatiotemporal tokens in the input data, aim to achieve high-efficiency video Transformer models. This involves calculating attention scores between non-keyframe tokens and keyframe tokens, selecting the top k tokens with the highest attention scores to reduce the number of tokens in the Transformer input and thus lower computational complexity. However, this approach is not suitable for running more efficient multi-scale visual Transformer models. Furthermore, the hard Top-K method of selecting the top k tokens based on size is non-differentiable during backpropagation, potentially affecting model stability during training. Additionally, this method of calculating token attention scores is not only computationally intensive but also has limited effectiveness in eliminating redundant tokens.
[0004] Furthermore, the actual deployment of existing smoking behavior detection models at the edge often involves model compression, format conversion, and inference acceleration, and there is a lack of a complete real-time monitoring solution at the edge.
[0005] Therefore, there is an urgent need for a lightweight video smoking behavior detection method that can run in real time on edge devices, which can effectively reduce the amount of inference computation while ensuring recognition accuracy and provide a deployable system architecture. Summary of the Invention
[0006] This disclosure provides a smoking behavior detection system and a lightweight spatiotemporal action detection model based on edge devices, aiming to solve the problems of high computational overhead, difficulty in effectively eliminating temporal redundancy, and complex deployment at the edge of existing video Transformers, so as to improve the inference efficiency of Transformer-based action detection models on edge devices, and proposes a complete method for deploying the smoking behavior detection system on edge devices.
[0007] In embodiments according to this disclosure, the lightweight spatiotemporal action detection model includes: a video action detection model framework with selectable differentiable tokens. The video action detection model framework is primarily based on a multi-scale visual Transformer, employing a spatiotemporal feature extraction method that prioritizes spatial features over temporal ones. The input data after the patchbed layer is pooled and downsampled in the spatial dimensions h and w based on the attention matrix of a single-frame token, and then pooled and downsampled in the temporal dimension t based on the spatiotemporal attention matrix of a set of sequence frame tokens. Finally, the data is fed into an MLP layer in the classification head to calculate the action category array for the current sequence frame based on the global information cls of each frame. Differentiable token selection first calculates the difference between the current frame and the next frame, then incorporates the difference value into the importance score in the time dimension according to weights. The larger the difference value, the more obvious the action change, the less redundancy, and the higher the importance. The score is then normalized using min-max. After adding noise to the score, multiple hard Top-K tests are performed, and the expectation of the results is calculated to obtain a soft Top-K mask array to select important frames. This method is differentiable during model backpropagation, making training more stable and enabling end-to-end training. At the same time, noise is gradually reduced during training to make the results approach hard Top-K. Hard Top-K is then used directly in the model inference stage to improve speed.
[0008] In embodiments according to this disclosure, the smoking behavior detection system includes: video frame acquisition and preprocessing, efficient inference deployment, and alarm logic. Video frame acquisition and preprocessing involves acquiring continuous video frames via a CSI camera on an edge device, performing preprocessing operations such as normalization, size adjustment, and RGB reordering transformation on the video frames to obtain a frame sequence conforming to the input format of a lightweight video recognition model. Efficient inference deployment first requires exporting the trained model to ONNX format, then converting it to the engine file required by TensorRT. Simultaneously, a dual-threaded structure is used for deployment on the Jetson Nano edge device. The main thread is used for video frame acquisition and preprocessing and smoking alarms, while the sub-thread loads the engine model file through the TensorRT engine for asynchronous real-time inference. A queue is used to decouple video preprocessing from model inference, improving overall processing efficiency. The alarm logic displays the currently captured video frames in real time, and when the inference result indicates smoking behavior, it executes the actions of displaying a smoking warning message and playing a voice warning through the speaker.
[0009] Based on the above, the smoking behavior detection system and lightweight spatiotemporal motion detection model described in the embodiments of this disclosure have the following technical effects. Significantly reduced computational load, adaptable to real-time operation on edge devices: This invention is based on a two-stage multi-scale visual Transformer model framework and utilizes frame difference to determine redundant input data tokens and prune them, significantly reducing the overall computational load of the model. This enables models that were previously difficult to run in real-time on Jetson Nano to achieve real-time inference. A two-stage token selection mechanism for differentiable training and efficient inference: This invention employs a soft differentiable Top-K mechanism to ensure gradient propagation during model training and uses a hard Top-K mechanism during inference to maximize efficiency, balancing accuracy and speed, achieving optimal performance at different stages of model training and inference. Combined with highly engineered TensorRT optimizations, lower latency: Through ONNX and TensorRT conversion, this invention's model can achieve multiple underlying optimizations such as operator fusion, half-precision acceleration, and memory optimization, further reducing inference latency and improving overall system stability and throughput. Complete system deployment, capable of independent operation, and engineering implementation: This invention not only includes the model algorithm but also covers a complete system solution including edge device thread architecture, queue communication mechanism, and alarm logic, enabling the model algorithm to run directly in actual monitoring scenarios. Attached Figure Description
[0010] Figure 1 This is a schematic diagram of the overall architecture of a smoking behavior detection system based on edge devices according to the present invention; Figure 2 This is a schematic diagram of the real-time inference workflow at the edge in this invention; Figure 3 This is a schematic diagram of the overall structure of the lightweight spatiotemporal motion detection model proposed in this invention; Figure 4 For the present invention Figure 3 A schematic diagram of the specific structure of the time-series token pruning module. Detailed Implementation
[0011] The technical solution of the present invention will now be further described in detail with reference to the accompanying drawings. This embodiment is a preferred embodiment of the present invention, but equivalent modifications made by those skilled in the art without departing from the spirit of the present invention should also be considered within the scope of protection of the present invention.
[0012] like Figure 1As shown, this invention provides a video behavior detection system based on edge devices. Its overall system architecture includes a video frame acquisition and preprocessing module, a high-efficiency inference module, and an alarm module. The high-efficiency inference module loads a lightweight spatiotemporal action detection model that includes a spatiotemporal feature extraction network and a temporal token pruning module. The system is deployed on a Jetson Nano edge device, acquiring video streams in real time via a camera. After data preprocessing, the data is input into the improved spatiotemporal behavior recognition model to achieve real-time recognition of smoking behavior and trigger a voice alarm when the target behavior is detected.
[0013] Please see Figure 2 This paper illustrates the real-time inference process of this invention at the edge. In the main thread, the video frame acquisition and preprocessing module calls the camera to acquire continuous image frames at a fixed frame rate of 24FPS and preprocess them. The preprocessing steps mainly include pixel value normalization, size adjustment to 224×224, channel arrangement transformation to adjust to the shape that conforms to the model input, and constructing a continuous frame sequence of length T using a sliding window method. The processed data is stored in the input queue and transmitted asynchronously to the inference thread to ensure the stability and real-time performance of frame processing. In the inference thread, the efficient inference module uses the TensorRT engine to load a lightweight spatiotemporal motion detection model file and reads the first frame sequence from the queue. Forward inference is performed through the optimized engine model; the inference result is stored in the output queue and obtained by the alarm module of the main thread. After obtaining the inference result, the alarm module first performs SoftMax calculation processing on it, and then determines whether the result contains information labeled as smoking behavior. If smoking behavior information is found, alarm information is superimposed on the currently captured video frame, and an alarm voice is triggered. This dual-threaded architecture ensures that data acquisition and inference do not block each other, effectively improving system throughput.
[0014] like Figure 3 As shown, the lightweight spatiotemporal action detection model of this invention is constructed based on a multi-scale visual Transformer structure. The model first uses Patch Embedding to divide the input T-frame video sequence into spatiotemporal patches and encode them as token sequences. Then, in the Spatial Transformer, the model performs pooling downsampling in the spatial dimension based on the single-frame attention matrix through 4 layers of Spatial Attention. Next, in the Temporal Transformer, the model further pools the token sequences in the spatiotemporal dimension using a cross-frame attention matrix through 12 layers of Temporal Attention to extract key action change features and further downsample the space, thereby significantly reducing the computational complexity of the Transformer.
[0015] Combination Figure 4 The lightweight spatiotemporal motion detection model of this invention uses a temporal token pruning module to evaluate and select the importance of the input temporal tokens. This module first divides the token features of a single frame into local and global representations through a fully connected layer. The two representations are then concatenated and passed through another fully connected layer to obtain the importance score for each frame. A higher score indicates that the frame contains more key motion information. Then, the difference between adjacent frames is calculated. Let the input frame be I. t The next frame is I t+1 The difference is divided into: D t =||I t+1 -I t ||1 The difference value characterizes the magnitude of action changes between frames; a larger difference indicates that the frame contains less redundant information. Subsequently, the module weighted and fused the difference score with the basic importance score to obtain the temporal importance score S. t And perform min-max normalization: To ensure a stable range of values.
[0016] like Figure 4 As shown, during the training phase, a perturbation maximization method is employed to achieve differentiable Top-K selection. This is done to obtain K one-hot vectors of dimension T as a mask for our token selection, i.e., Y∈{0,1}. T×K For the originally non-differentiable hard Top-k algorithm: Y * =argmax(Y,S) By adding a random noise vector Z to the scores, and controlling its magnitude with the noise variance σ, the discrete hard Top-K algorithm can be softened into taking the expectation of the random noise: Based on the above formula, noise is sampled 500 times from a standard normal distribution, and the mean of the one-hot result for each Top-K sample is calculated to obtain Y. σ ∈[0,1] T×K The real-valued matrix approximates a hard Top-K matrix. It is used to obtain the K most important frames from our T sequence frames, maintaining gradient backpropagation for more stable training and enabling end-to-end training. As training progresses, σ is gradually decreased until it approaches 0, thus obtaining selection results that more closely approximate the hard Top-K selection results. During the inference phase, the module directly adopts a hard Top-K matrix to maximize inference speed.
[0017] like Figure 3 and Figure 4As shown, the selected keyframe tokens are fed into a multi-scale Transformer backbone network for joint spatial-temporal modeling. In the spatial encoding stage, a self-attention matrix is used to weight and converge each frame's tokens to capture key region features; in the temporal encoding stage, a cross-frame attention mechanism is used to extract action change information. Finally, the global information (CLS) corresponding to each frame is input into the classification head, and a multilayer perceptron outputs the probability distribution of smoking behavior.
[0018] In this invention, the trained model is first exported to ONNX format during inference module deployment, and then the engine file is constructed using TensorRT. TensorRT implements operator fusion, FP16 acceleration, and memory optimization during the construction process to reduce inference latency and power consumption on edge devices. The inference thread loads the engine model during initialization and, during runtime, reads frame sequences from the input queue in a loop, performs fast inference, and then returns the results to the output queue for alarm display.
[0019] Combination Figure 1 and Figure 2 Upon detecting smoking, the system overlays a slowly fading red warning text onto the currently displayed screen and plays an alarm audio through the speaker connected to the Jetson Nano. This alarm mechanism allows for setting trigger thresholds, text display duration, or warning content based on the application scenario, enabling the system to operate independently in monitored environments.
[0020] In summary, this invention, by introducing a temporal token pruning mechanism, a multi-scale Transformer structure, and TensorRT deployment optimization, enables the video smoking behavior detection system to achieve low-latency, high-precision, and high-stability real-time smoking action recognition on edge devices such as the Jetson Nano. Modifications made by those skilled in the art to the inference model framework or deployment method without departing from the principles of this invention are within the scope of protection of this invention.
Claims
1. A smoking behavior detection system based on edge devices, characterized in that, The smoking behavior detection system in question includes: The video frame acquisition and preprocessing module is used to acquire video frames from the monitored area, perform image normalization and size adjustment, and then integrate them into a single inference model input sequence. The temporal token pruning module, configured in the feature modeling front end, is used to filter out redundant frames from the input sequence based on inter-frame difference information and feature importance, so as to generate a simplified video frame sequence. The spatiotemporal feature modeling module is used to extract spatial features and model temporal features based on the multi-scale visual Transformer attention mechanism, and extract spatiotemporal joint features from the simplified video frame sequence. The high-efficiency inference module runs on edge computing devices. It loads an optimized spatiotemporal action recognition model through an inference acceleration engine, asynchronously performs inference and judgment on the processed input data, and generates behavior recognition results. The alarm module is used to generate alarm information and display it on the currently extracted video frame image when the behavior recognition result indicates that smoking behavior exists, and to issue an alarm voice.
2. The smoking behavior detection system according to claim 1, characterized in that, The video frame acquisition and preprocessing module includes: an image acquisition device for real-time acquisition of video streams; and a preprocessing unit for normalizing, resizing, and serializing the acquired video frames.
3. The smoking behavior detection system according to claim 1, characterized in that, The temporal token pruning module includes: an inter-frame difference evaluation unit for calculating the difference in pixel domain or feature domain between adjacent frames; a scoring unit for generating a temporal-based frame importance score by combining the difference with the frame feature importance score; and a filtering unit for retaining a target number of video frames based on the score to form the simplified video frame sequence.
4. The smoking behavior detection system according to claim 1, characterized in that, The spatiotemporal feature modeling module includes: a spatial coding unit for extracting spatial features of a single frame; and a temporal coding unit for modeling cross-frame correlation of multi-frame sequences. The temporal coding unit extracts features in the temporal dimension based on a multi-scale multi-head attention mechanism.
5. The smoking behavior detection system according to claim 1, characterized in that, The high-efficiency inference module uses a pre-compiled inference acceleration engine to load the spatiotemporal action recognition model and generates the behavior recognition results in an asynchronous inference manner.
6. The smoking behavior detection system according to claim 1, characterized in that, The alarm module includes: a result processing unit for generating target behavior prompt information based on the reasoning result; and an alarm output unit for outputting visual or sound alarms when smoking behavior is detected.