Driver smoking behavior detection method based on improved yolov7

Abstract

This invention discloses a driver smoking behavior detection method based on an improved YOLOv7, relating to the field of visual positioning technology. The method first constructs and trains a driver smoking behavior model based on an improved YOLOv7; then, it acquires monitoring video from the car's cockpit and extracts image data for each frame; it loads the driver smoking behavior model based on the improved YOLOv7 to detect whether any person appears in each frame; if a person appears, it loads a DeepSORT detection model to detect the person in the current frame and assigns them an ID; based on the person's ID and coordinates, it extracts the person's image from the current frame and inputs it into a pre-trained cigarette butt detection model for cigarette butt detection; when a cigarette butt is detected, it retains the positional and temporal information of the detected cigarette butt in the current frame; it compares the retained temporal information of the frames with pre-matching information to calculate the goodness of fit, and uses the goodness of fit to determine whether smoking behavior exists, thus improving the accuracy and practicality of smoking behavior detection.

Description

[0001] This application is a divisional application of a patent application entitled "Detection Method of Smoking Behavior Based on Improved YOLOv7", the original application was filed on August 29, 2023, application number 202311092548.9. Technical Field

[0002] This invention relates to the field of visual positioning technology, and more specifically to a method for detecting driver smoking behavior based on an improved YOLOv7. Background Technology

[0003] Existing deep learning-based smoking behavior detection methods are mainly divided into single-stage and two-stage detection, and these methods share some common drawbacks. On the one hand, these methods oversimplify the judgment of smoking behavior, focusing primarily on whether a person is smoking while ignoring the influence of other factors. For example, they do not consider the background environment of smoking, subtle differences in smoking gestures, and the duration of smoking. Therefore, in complex scenarios, these methods are prone to false alarms or missed detections, resulting in low detection accuracy. On the other hand, these methods have limitations in algorithm design. For example, single-stage detection methods require large datasets and annotation work, as well as high computational resources, limiting their feasibility in practical applications. Furthermore, the target detection and tracking modules in two-stage detection methods have low accuracy in detecting small targets (such as cigarette butts), easily leading to false detections or false alarms. In addition, the introduction of an additional skeleton detection model in two-stage detection methods increases the system load, which is not conducive to practical deployment and application. Due to the lack of comprehensive consideration and accurate judgment of smoking behavior in existing smoking behavior detection methods, as well as limitations in algorithm design and practical application, problems such as false alarms, missed detections, and high computational resource requirements exist. Summary of the Invention

[0004] The purpose of this invention is to propose a driver smoking behavior detection method based on an improved YOLOv7, so as to improve the accuracy and practicality of smoking behavior detection.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] A method for detecting driver smoking behavior based on an improved YOLOv7 system includes the following steps: Construct and train a driver smoking behavior model based on an improved YOLOv7; Acquire surveillance video from the car's cockpit and extract image data from each frame; Load a driver smoking behavior model based on improved YOLOv7 and detect whether there are people in each frame of the image; If a person is detected in the current frame image, the DeepSORT detection model is loaded to detect the person in the current frame image and assign an ID to them; Based on the person's ID and coordinates, the person's image is extracted from the current frame and then input into a pre-trained cigarette butt detection model for cigarette butt detection; When a cigarette butt is detected, retain the location and timing information of the detected cigarette butt in the current frame image; The temporal information of the retained frames is compared with the pre-matching information to calculate the goodness of fit, and the goodness of fit is used to determine whether smoking behavior exists.

[0007] Optionally, the construction and training of the driver smoking behavior model based on the improved YOLOv7 specifically includes: The YOLOv7 backbone network was replaced with MobileNet v3-small, and the SimAM attention mechanism was embedded in the YOLOv7 neck network to build an improved YOLOv7 algorithm model. Input data and train the algorithm to obtain the model weights and a driver smoking behavior model based on the improved YOLOv7.

[0008] Optionally, the improved YOLOv7 algorithm model includes the following four parts: input, backbone network, neck network, and head network; The input portion uses the MixUp data augmentation method on the input dataset; The backbone network includes BConv convolutional layers, E-ELAN convolutional layers, and MPCConv convolutional layers. The BConv convolutional layers consist of convolutional layers, BN layers, and the LeakyReLU activation function, used to extract image features at different scales. The E-ELAN convolutional layers adopt the original ELAN design architecture, guiding computational blocks of different feature groups to learn more diverse features. The MPCConv convolutional layers, based on the BConv layers, add a Maxpool layer, forming two branches. The upper branch halves the image width and height using Maxpool and halves the image channels using the BConv layer; the lower branch halves the image channels using the first BConv layer. The features extracted by the two branches are fused using the Cat operation. The neck network adopts the Path Aggregation Feature Pyramid Network (PAFPN) structure, which introduces a bottom-up path to achieve efficient fusion of features at different levels. The head network adjusts the number of image channels for the P3, P4, and P5 features output by PAFPN at three different scales through a REP structure, and finally uses a 1×1 convolution for the prediction of confidence, category, and anchor box; the categories include cigarette butts and people.

[0009] Optionally, a lightweight attention SE module is introduced into the Bneck structure of the MobileNet v3 Small backbone network; the h-swish activation function is used instead of swish in the backbone network block, as shown in the formula: ; In the SE module, the h-sigmoid activation function is used instead of the sigmoid function, with the following formula: ; The formula for the ReLU6 activation function is: .

[0010] Optionally, the neck network includes an SPPCSPC module and a Concatenate layer; the SPP layer of the SPPCSPC module is used to capture object information of different scales and proportions, the CSP connection is used to enhance the expressive power and stability of features, and the Conv layer further extracts features; the SimAM attention mechanism is embedded in the SPPCSPC structure to improve the model's adaptability to changes in scale, proportion and orientation; SimAM is introduced in the Concatenate layer to enable the network to better capture the correlation between input and context features, emphasizing more critical features.

[0011] Optionally, acquiring the monitoring video from the vehicle's cockpit and extracting image data from each frame specifically includes: The acquired surveillance video is processed by frame extraction to obtain image data for each frame, and then enhanced using data augmentation techniques, including rotation, cropping, and increasing or decreasing the brightness of the image.

[0012] Optionally, loading the DeepSORT detection model to detect people appearing in the current frame image and assigning them IDs specifically includes: The Kalman filter is used to predict the trajectory. The Kalman filter algorithm consists of two processes: prediction and update. The algorithm defines the motion state of the target as eight normally distributed vectors. Prediction: When the target moves, the position and velocity parameters of the target box in the current frame are predicted based on the target box and velocity parameters of the previous frame. Update: The predicted value and the observed value are linearly weighted to obtain the current predicted state of the system. The predicted trajectory is matched with the detection results in the current frame using the Hungarian algorithm; the matching includes cascade matching and IOU matching. The trajectory is updated using a Kalman filter, and the tracking result is determined to identify the personnel ID, thus completing the personnel ID tracking.

[0013] Optionally, the step of extracting a person's image from the current frame based on the person's ID and coordinates and then inputting it into a pre-trained cigarette butt detection model for cigarette butt detection specifically includes: After obtaining the tracked person IDs through the DeepSORT detection model, the person images are extracted from the corresponding frames of the input image data using the coordinates of the person detection bounding boxes. Based on a pre-trained cigarette butt detection model, cigarette butts are detected in frame-level images of people, preserving temporal information and the coordinates of the detected cigarette butts.

[0014] Optionally, the step of comparing the temporal information of the retained frames with the pre-matching information to calculate the goodness of fit, and determining whether smoking behavior exists based on the goodness of fit, specifically includes: Let P be the time series of the retained frame time information, where P = {(t1, y1), ..., (t p ,y p ),…,(t M ,y M )}; The time series in the time series information of the pre-matching information is denoted as Q, Q={(t1,y1),…,(t q ,y q ),…,(t N ,y N Thus, the distance matrix W between points P and Q is obtained. M×N The formula is: ; Matrix W M×N The element w(p,q) in the equation is the point (t) p ,y p ) and (t q ,y q The Euclidean distance d(p,q) is obtained by summing the distances of path segments v. The formula for summing the distances of path segments v is: ; In the formula l c =(p,q) c Let the cumulative distance matrix be denoted as D, then we obtain the elements of the first row: ; And the first row and column elements are: ; The other elements in the cumulative distance matrix D are obtained recursively, using the following formula: ; Where 2≤m≤M, 2≤n≤N; the minimum planning path γ between time series P and time series Q. minThe cumulative distance is the DTW distance between time series P and time series Q, i.e., DTW(P,Q). The smaller DTW(P,Q), the smaller the difference between P and Q, which indicates that the person in the current frame is smoking.

[0015] Optionally, after determining whether smoking behavior exists based on the goodness of fit, the method further includes: An alarm will be triggered if someone is smoking.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: By replacing the YOLOv7 backbone network with the lightweight MobileNetv3 network structure, the SimAM attention mechanism is introduced to enhance the features of small cigarette butt targets and reduce background interference. This improves the accuracy of small cigarette butt target detection while maintaining high efficiency, making it easier to deploy. Embedding the SimAM attention mechanism in the SPPCSPC structure improves the model's adaptability to changes in scale, proportion, and orientation, thereby increasing detection accuracy. Introducing SimAM into the concatenate layer allows the network to better capture the correlation between input and context features, emphasizing more critical features and making the feature maps more discriminative for the current task, reducing interference from irrelevant features and thus improving model performance. The algorithm of secondary detection plus DTW effectively reduces the false negative and false positive rates in smoking behavior detection. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the driver smoking behavior detection method based on the improved YOLOv7 of this invention. Figure 2 This is a comparison chart of the Mosaic+RE and Mosaic-REL experiments; Figure 3 This is the network structure diagram of the improved YOLOv7 algorithm model; Figure 4 This is a diagram of the Bneck structure in MobileNetv3; Figure 5 This is a SimAM structure diagram; Figure 6 This is a schematic diagram of the detection process for drivers smoking. Detailed Implementation

[0018] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings.

[0019] like Figure 1 As shown, the present invention provides a driver smoking behavior detection method based on the improved YOLOv7, which specifically includes the following steps S1 to S7.

[0020] S1: Training a driver smoking behavior model based on an improved YOLOv7.

[0021] Step S1 specifically includes steps S11 and S12.

[0022] S11: Collect and split the data, then use Mosaic-REL data augmentation to divide it into training and test sets.

[0023] In real-world smoking detection, cigarette butts are frequently obscured when multiple people are smoking, leading to both missed and false detections. For targets centrally obscured in the dataset, MixUp data augmentation alone is not ideal. Therefore, inspired by random erasure RE data augmentation, this paper proposes Mosaic-REL data augmentation, combining it with Mosaic data augmentation.

[0024] RE data augmentation involves randomly selecting a rectangular region on the input image and filling it with noise blocks of grayscale values ​​between 0 and 255 to cover the object within that region, simulating occlusion without altering the overall features of the input image. First, a wipe rate threshold P is set. Images from the input dataset are randomly assigned a wipe rate P1. If P1 < P, the original image is output; otherwise, a rectangular region I is randomly selected. e Perform erasure. The random erasure process is as follows: 1) Assume the image size is (w, h), and the image area s = w × h; 2) Initialize the rectangle I. e The relevant parameters are set, and the rectangle I is randomly set. e The aspect ratio is re (the value of re can be manually assigned or randomly set in the range of (0.02, 0.2)), and the erasure area is se (the value of se is between the erasure area thresholds s1 and s2); 3) Generate a size of (w e h e rectangular region I e First, randomly select point P in its upper left corner. e =(x e y e The restrictions are as follows: .

[0025] Because the detection targets in this invention are cigarette butts and people, with cigarette butts being small target samples, the original algorithm locates the rectangular area globally, which cannot guarantee that each erasure will cover the target area, thus failing to simulate occlusion and wasting computational resources. Therefore, in step S11, the Mosaic-REL local random erasure algorithm is used for data augmentation. Specifically, the true coordinates (T) of the detection type C in the tag file are first read. x T y Tw T h ), determine if it is a small target; if it is a small target, select point P at the top left corner of the rectangular area. e =(x e y e This process generates a rectangular region and assigns values ​​to the pixels within that region to simulate the effect of armor being occluded, thus outputting the processed image. The corrected top-left corner point P... e =(x e y e The restrictions are: If the target is large, there is no significant difference in detection performance between local and global random erasure for large target samples, so no erasure is performed.

[0026] like Figure 2 As shown, this illustrates a comparison between the Mosaic+RE and Mosaic-REL experiments. From Figure 2 As can be seen, the improved data augmentation algorithm Mosaic-REL can effectively erase small target samples within the bounding box, avoiding the data redundancy problem introduced by the original algorithm. With the same amount of dataset, this improved algorithm can enable the network input to obtain more occluded small target samples.

[0027] S12: Replace the YOLOv7 backbone network with MobileNet v3-small, and embed the SimAM attention mechanism into the YOLOv7 Neck network to build an improved YOLOv7 algorithm model; input data and train to obtain the algorithm model weights and obtain a driver smoking behavior model based on the improved YOLOv7.

[0028] Figure 3 The diagram shows the network structure of the improved YOLOv7 algorithm model. The improved YOLOv7 algorithm model mainly consists of four parts: Input, Backbone, Neck, and Head networks, which are described in detail below.

[0029] Input: The input dataset was augmented using the MixUp data augmentation method, an improvement upon the Mosaic data augmentation technique. MixUp utilizes a linear blending of two images, while also linearly blending the labels. This augmentation method merges two images into one, which not only enriches the dataset and improves the network's generalization ability but also reduces the risk of overfitting.

[0030] The backbone network consists of multiple convolutional layers, including BConv, E-ELAN, and MPCConv layers. The BConv layer comprises convolutional layers, BatchNormalization (BN) layers, and a LeakyReLU activation function, used to extract image features at different scales. The E-ELAN layer adopts the original ELAN architecture, guiding computational blocks of different feature groups to learn more diverse features, improving the network's learning ability without disrupting the original gradient path. The MPCConv layer, based on the BConv layer, adds a Maxpool layer, forming two branches. The upper branch halves the image width and height using Maxpool and halves the image channels using the BConv layer. The lower branch halves the image channels using the first BConv layer. The features extracted by the two branches are fused using a Cat operation, aiming to further optimize the effect of the convolutional layers and improve the network's feature extraction capability.

[0031] Neck Network: The Path Aggregation Feature Pyramid Network (PAFPN) structure is adopted, which introduces a bottom-up path, making it easier for information from the lower levels to be transmitted to the higher levels, thereby achieving efficient fusion of features from different levels.

[0032] The Head Network adjusts the image channel count of the P3, P4, and P5 features output by PAFPN using a REP (RepVGG Block) structure. Finally, it performs 1×1 convolutions for predicting confidence, category, and anchor box. The Head Network predicts categories including cigarette butts and people, and outputs the confidence score for a cigarette butt or a person (person) and the corresponding anchor box position.

[0033] In the improved YOLOv7 algorithm model, the MobileNet v3 Small network is selected as the backbone network. For example... Figure 3 and Figure 4 As shown, in the Bneck structure of the MobileNet V3 Small network, a lightweight attention SE module is introduced. NAS and NetAdapt are used to optimize the network configuration and parameters, reduce the number of channels in the convolutional kernels, modify the tail structure to reduce computation, and redesign the activation function. Specifically, the h-swish activation function is replaced with the swish function in the backbone module, with the formula: In the SE module, the h-sigmoid activation function is used instead of the sigmoid function, and the formula is: The ReLU6 activation function formula is as follows: .

[0034] This invention replaces the YOLOv7 backbone network with a lightweight MobileNet network structure and introduces the SimAM attention mechanism to enhance the features of small cigarette butt targets and reduce background interference. This improves the accuracy of small cigarette butt target detection while maintaining high efficiency, making it easier for practical deployment. Figure 5 The diagram shows the SimAM architecture. The SimAM attention mechanism is a simple, parameter-free attention module designed for convolutional neural networks. Unlike existing channel and spatial attention modules, it directly infers the 3D weights of each neuron in the feature map without requiring additional subnetworks or parameters. It is inspired by the spatial inhibition theory in neuroscience, which states that neurons exhibiting different firing patterns from surrounding neurons are more important in visual processing. It calculates the weights by defining an energy function to measure the linear separability of each neuron and deriving a closed-form solution. It then uses a scaling operation to refine the feature map. The SimAM attention mechanism can be added as a plug-in module after any convolutional layer without increasing the network's complexity.

[0035] like Figure 3 As shown, the SPPCSPC module in the Neck network improves the efficiency and accuracy of feature extraction. Its SPP layer can capture object information at different scales and proportions, CSP connections enhance the expressive power and stability of features, and the Conv layer further extracts features. This invention embeds the SimAM attention mechanism into the SPPCSPC structure to improve the model's adaptability to changes in scale, proportion, and orientation, thereby improving detection accuracy. Furthermore, SimAM is introduced in the Concatenate layer, enabling the network to better capture the correlation between input and context features, emphasizing more critical features, making the feature map more discriminative for the current task, reducing interference from irrelevant features, and thus improving model performance.

[0036] This invention combines Mosaic data augmentation with an improved version called Mosaic-REL data augmentation. The improved Mosaic-REL algorithm can effectively erase small target samples within their bounding boxes, avoiding the data redundancy drawbacks of the original algorithm. With the same dataset size, it allows the network input to obtain more occluded small target samples. Furthermore, this invention introduces the SimAM attention mechanism into the improved YOLOv7 algorithm model to enhance the features of small cigarette butt targets and reduce background interference, thereby improving the accuracy of cigarette butt target detection to a certain extent while maintaining high detection efficiency.

[0037] S2: Acquire surveillance video from the car's cockpit and extract image data. Obtain each frame of image data through video or RTSP stream and perform enhancement processing.

[0038] In step S2, the acquired surveillance video is processed by frame extraction to obtain image data for each frame, generating a personnel detection dataset. Data augmentation techniques are then used to enhance the personnel detection dataset. To improve the accuracy of the model's detection, a large number of personnel datasets are collected, and different smokers in open scenes are simulated and filmed. Frame extraction is then used to process the video, generating the personnel detection dataset. Simultaneously, common data augmentation techniques, such as rotation, cropping, and adjusting image brightness, are employed to enhance the dataset and improve the model's generalization ability.

[0039] S3: Load the driver smoking behavior model based on the improved YOLOv7 and detect whether there are people in each frame of the image; if people are detected in the current frame of the image, save the detection result and proceed to step S4.

[0040] In the improved YOLOv7 algorithm model, its head network can output the confidence score and corresponding anchor box position for targets such as cigarette butts or people. Therefore, after training with different datasets (e.g., people datasets and cigarette butt datasets), it can be used as a people detection model and a cigarette butt detection model. The detection process is as follows: Figure 6 As shown.

[0041] S4: Load the DeepSORT detection model, detect people appearing in each frame of the image and assign them IDs, thus completing the person ID tracking.

[0042] Deep SORT, the precursor to the people tracking algorithm, stands for Simple Online and Realtime Tracking. SORT's most significant feature is its object detection method based on Faster R-CNN, utilizing Kalman filtering and the Hungarian algorithm to greatly improve the speed of multi-object tracking while achieving state-of-the-art (SOTA) accuracy. The core of the SORT algorithm consists of two algorithms: Kalman filtering and the Hungarian algorithm. This algorithm defines the motion state of a target as eight normally distributed vectors. The Kalman filtering algorithm comprises two processes: prediction and update. Prediction: When the target moves, the position and velocity of the target bounding box in the current frame are predicted using parameters such as the target bounding box and velocity from the previous frame. Update: The predicted and observed values ​​are linearly weighted to obtain the current predicted state. The Hungarian algorithm solves a bipartite graph assignment problem. In the main steps of MOT, it calculates the similarity cost matrix (IOU) to obtain the similarity matrix between consecutive frames. The Hungarian algorithm solves this similarity matrix to identify truly matched targets between consecutive frames. The main features of DeepSORT are that it adds appearance information to the SORT algorithm, borrows the ReID domain model to extract appearance features, and reduces the number of ID switches; the matching mechanism has changed from the original matching based on the IOU cost matrix to cascade matching and IOU matching.

[0043] In step S4, cascaded matching and new trajectory confirmation are added to the DeepSORT detection model. Specifically: S41: Use a Kalman filter to predict the trajectory; S42: Use the Hungarian algorithm to match the predicted trajectory with the detection results in the current frame; the matching in step S42 includes cascaded matching and IOU matching; S43: Use a Kalman filter to update the trajectory and determine the tracking result, thereby determining the ID and completing the personnel ID tracking.

[0044] S5: Extract the person's image from the source input image data based on the person's ID and coordinates, and then input it into the pre-trained cigarette butt detection model for cigarette butt detection.

[0045] After obtaining the tracked person IDs using the DeepSORT model, each person is cropped from the corresponding frame of the input image data using the coordinates of the person detection bounding box (anchor box). The cropped person images are then input into a pre-trained cigarette butt detection model, and the output targets are the confidence score of the cigarette butt and the corresponding anchor box position.

[0046] S6: If a cigarette butt is detected, retain the location and timing information of the cigarette butt detected in the current frame.

[0047] The specific steps in step S6 for retaining the location and timing information of the cigarette butts detected in the current frame are as follows: S61: After obtaining the tracked personnel IDs through the DeepSORT model, each person is captured in the corresponding frame of the input image data using the coordinates of the personnel detection box; S62: Cigarette butts are detected in the personnel images at the frame level, and the valid timing information and the coordinate information of the detected cigarette butts are retained; that is, when a cigarette butt is detected, the coordinates of the currently detected cigarette butt are retained for comparison with the coordinates of the next frame; when the coordinates of the cigarette butts change in the next frame, an array is generated to record the coordinate information and timing information for a specified total number of frames; if the coordinates do not change, they are not saved.

[0048] S7: Compare the timing information of the retained frames with the pre-matching information to calculate the degree of fit. Determine whether there is smoking behavior by the degree of fit. If the degree of fit is higher than the preset threshold, issue an alarm; otherwise, return to step S6 to clear the information and re-record until the cigarette butt disappears from the screen.

[0049] In step S7, the DTW algorithm is used to compare the temporal information of the retained frame number with the pre-matching information to calculate the fitting degree. The specific process is as follows.

[0050] Let P be the time series of the retained frame time information, where P = {(t1, y1), ..., (t... p y p ), …, (t M y M )}; The time series in the time series information of the pre-matching information is denoted as Q, Q={(t1,y1),…,(t q y q ), …, (t N y N Thus, the distance matrix W between points P and Q is obtained. M×N The formula is: ; Matrix W M×N The element w(p, q) in the equation is the point (t) p y p ) and (t q y q The Euclidean distance d(p,q) is given by the Euclidean distance d(p,q).

[0051] The planned path γ is obtained by summing the distances of path segments v. The formula for summing the distances of path segments v is: ; In the formula l c =(p, q) c Let the cumulative distance matrix be denoted as D, then we obtain the elements of the first row: ; And the first row and column elements are: .

[0052] The other elements in the cumulative distance matrix D can be obtained recursively, using the following formula: ; Where 2≤m≤M, 2≤n≤N.

[0053] Minimum planning path γ between time series P and time series Q min The cumulative distance is the DTW distance between time series P and time series Q, i.e., DTW(P,Q). The smaller the DTW(P,Q), the smaller the difference between P and Q, indicating that the person in the current frame is smoking. If smoking is detected, an alarm is generated, completing the smoking behavior detection; otherwise, the information is saved again.

[0054] Using the above technical solution, a method for determining smoking behavior based on trajectory is proposed. The coordinates of cigarette butts detected frame by frame in the secondary detection are preserved, and their curvature is obtained by plotting coordinate curves. Then, using a predefined curve, the algorithm compares the goodness of fit between the two curves to determine whether smoking behavior exists. An alarm is generated when the goodness of fit is high. By using cigarette butt trajectories, the false positives and false negatives of the model can be greatly reduced.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting driver smoking behavior based on an improved YOLOv7, characterized in that, Includes the following steps: Construct and train a driver smoking behavior model based on an improved YOLOv7; Acquire surveillance video from the car's cockpit and extract image data from each frame; Load a driver smoking behavior model based on improved YOLOv7 and detect whether there are people in each frame of the image; If a person is detected in the current frame image, the DeepSORT detection model is loaded to detect the person in the current frame image and assign an ID to them; Based on the person's ID and coordinates, the person's image is extracted from the current frame and then input into a pre-trained cigarette butt detection model for cigarette butt detection; When a cigarette butt is detected, retain the location and timing information of the detected cigarette butt in the current frame image; The temporal information of the retained frames is compared with the pre-matching information to calculate the goodness of fit, and the goodness of fit is used to determine whether smoking behavior exists.

2. The driver smoking behavior detection method based on improved YOLOv7 according to claim 1, characterized in that, The construction and training of the driver smoking behavior model based on the improved YOLOv7 specifically includes: The YOLOv7 backbone network was replaced with MobileNet v3-small, and the SimAM attention mechanism was embedded in the YOLOv7 neck network to build an improved YOLOv7 algorithm model. Input data and train the algorithm to obtain the model weights and a driver smoking behavior model based on the improved YOLOv7.

3. The driver smoking behavior detection method based on improved YOLOv7 according to claim 2, characterized in that, The improved YOLOv7 algorithm model consists of the following four parts: input, backbone network, neck network, and head network; The input portion uses the MixUp data augmentation method on the input dataset; The backbone network includes BConv convolutional layers, E-ELAN convolutional layers, and MPCConv convolutional layers. The BConv convolutional layers consist of convolutional layers, BN layers, and the LeakyReLU activation function, used to extract image features at different scales. The E-ELAN convolutional layers adopt the original ELAN design architecture, guiding computational blocks of different feature groups to learn more diverse features. The MPCConv convolutional layers, based on the BConv layers, add a Maxpool layer, forming two branches. The upper branch halves the image width and height using Maxpool and halves the image channels using the BConv layer; the lower branch halves the image channels using the first BConv layer. The features extracted by the two branches are fused using the Cat operation. The neck network adopts the Path Aggregation Feature Pyramid Network (PAFPN) structure, which introduces a bottom-up path to achieve efficient fusion of features at different levels. The head network adjusts the number of image channels for the P3, P4 and P5 features output by PAFPN at three different scales through a REP structure, and finally uses 1×1 convolution for the prediction of confidence, class and anchor box.

4. The driver smoking behavior detection method based on improved YOLOv7 according to claim 3, characterized in that, A lightweight attention SE module is introduced into the Bneck structure of the MobileNet v3 Small backbone network; the h-swish activation function is replaced with the swish function in the backbone network block, as shown in the formula: ； In the SE module, the h-sigmoid activation function is used instead of the sigmoid function, with the following formula: ； The formula for the ReLU6 activation function is: 。 5. The driver smoking behavior detection method based on improved YOLOv7 according to claim 3, characterized in that, The neck network includes an SPPCSPC module and a Concatenate layer; the SPP layer of the SPPCSPC module is used to capture object information of different scales and proportions, the CSP connection is used to enhance the expressive power and stability of features, and the Conv layer further extracts features. Embedding the SimAM attention mechanism in the SPPCSPC structure improves the model's adaptability to changes in scale, proportion, and orientation; introducing SimAM in the Concatenate layer enables the network to better capture the correlation between input and contextual features, emphasizing more critical features.

6. The driver smoking behavior detection method based on improved YOLOv7 according to claim 1, characterized in that, The process of acquiring the monitoring video from the car's cockpit and extracting each frame of image data specifically includes: The acquired surveillance video is processed by frame extraction to obtain image data for each frame, and then enhanced using data augmentation techniques, including rotation, cropping, and increasing or decreasing the brightness of the image.

7. The driver smoking behavior detection method based on improved YOLOv7 according to claim 1, characterized in that, The loading of the DeepSORT detection model to detect people appearing in the current frame image and assign them IDs specifically includes: The Kalman filter is used to predict the trajectory. The Kalman filter algorithm consists of two processes: prediction and update. The algorithm defines the motion state of the target as eight normally distributed vectors. Prediction: When the target moves, the position and velocity parameters of the target box in the current frame are predicted based on the target box and velocity parameters of the previous frame. Update: The predicted value and the observed value are linearly weighted to obtain the current predicted state of the system. The predicted trajectory is matched with the detection results in the current frame using the Hungarian algorithm; the matching includes cascade matching and IOU matching. The trajectory is updated using a Kalman filter, and the tracking result is determined to identify the personnel ID, thus completing the personnel ID tracking.

8. The driver smoking behavior detection method based on improved YOLOv7 according to claim 1, characterized in that, The process of extracting a person's image from the current frame based on their ID and coordinates and then inputting it into a pre-trained cigarette butt detection model for cigarette butt detection specifically includes: After obtaining the tracked person IDs through the DeepSORT detection model, the person images are extracted from the corresponding frames of the input image data using the coordinates of the person detection bounding boxes. Based on a pre-trained cigarette butt detection model, cigarette butts are detected in frame-level images of people, preserving temporal information and the coordinates of the detected cigarette butts.

9. The driver smoking behavior detection method based on improved YOLOv7 according to claim 1, characterized in that, The step of comparing the temporal information of the retained frames with the pre-matching information to calculate the goodness of fit, and determining whether smoking behavior exists based on the goodness of fit, specifically includes: Let P be the time series of the retained frame time information, where P = {(t1, y1), ..., (t p ,y p ),…,(t M ,y M )}; The time series in the time series information of the pre-matching information is denoted as Q, Q={(t1,y1),…,(t q ,y q ),…,(t N ,y N Thus, the distance matrix W between points P and Q is obtained. M×N The formula is: ； Matrix W M×N The element w(p,q) in the equation is the point (t) p ,y p ) and (t q ,y q The Euclidean distance d(p,q) is obtained by summing the distances of path segments v. The formula for summing the distances of path segments v is: ； In the formula l c =(p,q) c Let the cumulative distance matrix be denoted as D, then we obtain the elements of the first row: ； And the first row and column elements are: ； The other elements in the cumulative distance matrix D are obtained recursively, using the following formula: ； Where 2≤m≤M, 2≤n≤N; the minimum planning path γ between time series P and time series Q. min The cumulative distance is the DTW distance between time series P and time series Q, i.e., DTW(P,Q). The smaller DTW(P,Q) is, the smaller the difference between P and Q, which indicates that the person in the current frame is smoking.

10. The driver smoking behavior detection method based on improved YOLOv7 according to claim 9, characterized in that, After determining whether smoking behavior exists based on the goodness of fit, the method further includes: An alarm will be triggered if someone is smoking.

Images