Smoking and phone calling detection method and system combining action recognition and target detection

Abstract

The application discloses a smoking and phone calling detection method and system combining action recognition and target detection, video data of a monitoring scene is collected through a monitoring camera; the video data is cut into short-time video clips, and personnel in the video are marked with actions; the video clips are converted into video frames, human key point coordinates, cigarette coordinates and mobile phone coordinates are marked on each video frame, and a model training data set is obtained; a YOLOv5-S target detection network model is established, then the 2D human key point detection network, the 3D human posture estimation model and the YOLOv5-S target detection network model are trained by using the constructed data set, and the weight of each trained network model is obtained; finally, the monitoring video of a monitoring place is input into the trained network model, and whether the behavior of smoking and calling in the video clip occurs is determined. The application effectively improves the accuracy of the smoking and calling event detection, and reduces the missed detection rate of the event.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent detection, and particularly relates to a smoking and phone call detection scheme combining human posture estimation and target detection, which can accurately detect the behavior of people smoking and making phone calls in an indoor scene. BACKGROUND

[0002] The smoking phenomenon in public places has always been the focus of social attention. The harm of smoking in public places mainly lies in the following aspects: 1. Easy to cause fire: Smoking is a very important cause of disaster in a large number of fires. Under certain conditions, the unextinguished cigarette butt can cause the combustion of various loose fiber materials such as rags, cotton, sawdust, etc. The cigarette butt dropped on the fiber board pressed by wood chips may also be ignited according to the pressing condition, temperature and ventilation condition, etc. 2. Serious pollution of public place air: Tobacco smoke is the main source of indoor inhalable particulate matter, accounting for more than 90% of indoor inhalable particulate matter. These inhalable particulate matters are the carriers of most pollutants and microorganisms, and the fine particulate matters below PM2.5 are more easily inhaled into the lungs, which are the main particulate matters that harm health in tobacco smoke. 3. Serious harm to the health of passive smokers: Second-hand smoke has been identified as a class A carcinogen by the U.S. Environmental Protection Agency and the International Cancer Research Agency. The smoke inhaled by passive smokers contains a variety of toxic substances and carcinogens.

[0003] In addition, in some special places, such as mine, flour, lime manufacturing site, chemical plant and other closed places prone to fire and explosion, preventing the occurrence of fire is the top priority of safety work. The occurrence of smoking and phone call events may cause fire, even huge explosion in these places, causing huge casualties and economic losses. Staff with a lucky mentality may evade safety inspection, so omnidirectional, dead angle-free monitoring cameras are usually installed in these places to monitor the behavior of personnel in all aspects to avoid possible safety problems.

[0004] In smaller enclosed spaces, smoke alarms are one of the choices for detecting whether a person is smoking. This device can also perform fire alarm, which has strong practicality. However, in open and larger places, the smoke produced by smoking dissipates too quickly, and the smoke alarm is difficult to detect the smoking behavior and alarm. At present, with the widespread use of cameras and the application of computer vision in the engineering field, using target detection and action recognition methods to detect smoking and phone call events has become the mainstream research direction in this field.

[0005] Object detection is one of the core problems in computer vision, and the task of object detection is to find all the objects of interest in the image and determine their categories and locations. Due to the different sizes, categories and appearances of various objects, as well as various factors such as image background lighting and occlusion, object detection has always been the most challenging problem in the field of computer vision. The most mainstream object detection model is the R-CNN series and the YOLO series, among which Faster-RCNN and YOLOv5 are the two most commonly used network models in the industrial object detection task. They can achieve good detection accuracy and fast detection speed in the conventional object detection task. However, the problem of poor small target detection performance has not been completely solved in the field of small target detection.

[0006] There are usually two definitions of small targets in object detection: in the COCO

[50] dataset, small targets are defined as absolute scales smaller than 32x32 pixels; the other is defined according to the relative size of the target. The International Society of Optical Engineering defines small targets as targets with an imaging area of less than 80 pixels in a 256x256 pixel image, i.e. the size of small targets is less than 0.12% of the original image. Compared with regular-sized targets, small targets have fewer pixels and lower resolution in the image, and the expression ability of features is weaker. At present, small target detection mainly increases the accuracy of small target detection through feature enhancement, multi-scale feature fusion, learning high-resolution features with large receptive fields, introducing attention mechanisms, introducing context information, and designing appropriate anchor boxes.

[0007] Human keypoint detection is crucial for describing human poses and predicting human behaviors. Therefore, human keypoint detection is the basis for many computer vision tasks. Three-dimensional human pose estimation from monocular observations is a basic visual task that reconstructs three-dimensional body joint positions from input images or videos. Since this task can obtain meaningful body geometry and motion expressions, it has a wide range of applications. MixSTE is a seq2seq method based on Transformer for three-dimensional human pose estimation from monocular videos. This model can better capture the overall sequence coherence and temporal motion trajectory of different body joints, while greatly improving the efficiency of three-dimensional human pose estimation, and achieving the best results on various 3D human pose estimation datasets. The model first extracts the 2D keypoint sequence of the person from the monocular video sequence, then predicts the 3D human pose sequence from the 2D keypoint sequence, and finally performs human action recognition based on the 3D human pose sequence. However, since current human pose estimation methods only rely on the motion changes of the human body to recognize the actions performed, they do not fully utilize environmental information, which leads to poor performance of many human pose estimation methods in recognizing "human interaction" behaviors.

[0008] It can be seen that the prior art still lacks a solution that meets the demand for implementation effect, and the present application has important practical significance. SUMMARY

[0009] When using the target detection method to detect smoking and calling, the model needs to detect the cigarette and the mobile phone. Obviously, the cigarette and the mobile phone can be considered as a small target under the monitoring angle. In the case that the target is completely exposed to the monitoring angle at a close distance, the model can have good detection effect on the smoking and calling event. However, in the case that the cigarette and the mobile phone are blocked, or the cigarette and the mobile phone are too far away from the monitoring camera, the missed detection is very serious. In addition, if similar objects to the cigarette and the mobile phone appear in the monitoring angle, or the cigarette and the mobile phone are not on the hand or the mouth, false positives of smoking and calling will occur.

[0010] When using the human pose estimation and action recognition method to detect the smoking and calling behavior, the human body needs to completely appear in the monitoring angle range. Whether the human body is far or close to the monitoring camera, the human pose can be well detected and the action can be recognized. However, when the personnel make similar actions to smoking and calling, this method has serious false positives, such as normal head touching action, which is very close to the smoking and calling behavior.

[0011] In order to solve the above problems, the present application combines human pose estimation and target detection to design a smoking and calling detection method, which includes video acquisition, video slicing and frame processing, small target detection model construction, small target detection model and human pose estimation model training and detection, and time sequence analysis and result output. This method can effectively improve the accuracy of smoking and calling event detection, while reducing the missed detection rate of the event, and has great application value.

[0012] The technical scheme of the present application provides a smoking and calling detection method combining action recognition and target detection. Video data of a monitoring scene is collected by a monitoring camera. The video data is cut into short-time video segments, and personnel in the video are labeled for action. The video segments are converted into video frames, and each video frame is labeled with human key point coordinates, cigarette coordinates and mobile phone coordinates to obtain a model training data set. A YOLOv5-S target detection network model is established. The YOLOv5-S target detection network model is based on the YOLOv5 model and adds variability convolution and attention mechanism, so that it is more suitable for detecting small targets. Then, the constructed data set is used to train a 2D human key point detection network, a 3D human pose estimation model and a YOLOv5-S target detection network model to obtain the weights of the trained network models. Finally, the monitoring video of the monitoring place is input into the trained network model to determine whether the smoking and calling behavior occurs in the video segment.

[0013] Moreover, the implementation process includes the following steps,

[0014] (1) Install a camera in the monitoring place to transmit the video images obtained by the camera in real time;

[0015] (2) Preprocess the video data, crop the high-pixel long video where smoking and calling behaviors exist into short segments, and convert them into video frames, label the human behaviors in the video and the targets in the video frames, and build a data set;

[0016] (3) Establish a YOLOv5-S target detection network model, including improving the YOLOv5 model to make it more suitable for small target detection. The improvement method is to change the convolution operation in the Backbone module of the model to deformable convolution, and change the pooling operation to deformable pooling; add a spatial attention mechanism; set the corresponding anchor box according to the size of the smoke and mobile phone in the training data;

[0017] (4) Train the 2D human key point detection network, 3D human pose estimation model and YOLOv5-S small target detection network model using video training data and image training data, obtain the model parameters of the three models respectively, and finally obtain the trained detection model;

[0018] (5) Use the monitoring camera to obtain the monitoring video, crop the video segment as the input of the smoking / calling detection model, input the trained detection model, and obtain the coordinate sequence of the 2D human key points, the human pose estimation result and the confidence, and the corresponding coordinates and confidence of the existence of smoke or mobile phone;

[0019] (6) Analyze the output results of the detection model, combine the time sequence information, human key points and the relative position of the smoke or mobile phone for comprehensive analysis, and finally determine whether the smoking or calling behavior occurs in the video segment.

[0020] Moreover, the annotation of the data set and the implementation method of the model training are as follows,

[0021] After obtaining the smoking and calling monitoring videos in various scenes, the videos are cropped into video segments with shorter time, and the video segments are converted into video frames;

[0022] According to the needs of the model, the obtained video and image data are annotated in various forms, including human key point annotation, finally obtaining a 2D human key point sequence data set; annotating the behaviors of personnel appearing in the video segment, finally obtaining a human pose estimation data set; annotating the smoke and mobile phone appearing in the image, finally obtaining a target detection data set;

[0023] The model is trained using the constructed data set, including taking pictures as input, training the 2D human key point detection network CPN using the 2D human key point sequence data set, obtaining the CPN network model parameters; using the video segment and the 2D human key point sequence data set as input, the human pose estimation data set as the GT value, training the MixSTE 3D human pose estimation model, obtaining the MixSTE network model parameters; taking pictures as input, training the YOLOv5-S network model using the target detection data set, obtaining the YOLOv5-S model parameters.

[0024] Moreover, when the YOLOv5 model is converted into the YOLOv5-S model focusing on small target detection, the Backbone module of the YOLOv5 model is adjusted, and the input picture in the adjusted Backbone module passes through DBL, CSP1_1, a spatial attention module, DBL, CSP1_3, a spatial attention module in sequence to obtain feature Figure 1 , feature Figure 1 passes through DBL, CSP1_3, a spatial attention module in sequence to obtain feature Figure 2 , feature Figure 2 passes through DBL, SPP, CSP2_1, a spatial attention module, DBL in sequence to obtain feature Figure 3 , feature Figure 1 , 2 , 3 input FPN+PAN feature pyramid structure, and then is input to a prediction module, and finally, according to the size of the cigarette and the mobile phone in the training data, corresponding anchor boxes are set to predict the target position and category.

[0025] Moreover, when comprehensively analyzing the output results of the detection model, if the following conditions are met at the same time, it is considered that there is a smoking behavior in the video:

[0026] The human action category output by the 3D human pose estimation model is “smoking”;

[0027] The picture frames contained in the video segment are detected by the YOLOv5-S model, and 1 / 3 or more of the picture frames detect the cigarette;

[0028] In the same picture, the relative distance between the coordinates of the cigarette and the coordinates of the “wrist joint” in the human key point sequence obtained by the 2D human key point detection model is less than a preset threshold.

[0029] Moreover, when comprehensively analyzing the output results of the detection model, if the following conditions are met at the same time, it is considered that there is a phone call behavior in the video:

[0030] The human action category output by the 3D human pose estimation model is “making a phone call”;

[0031] The picture frames contained in the video clip are detected by the YOLOv5-S model, and more than 1 / 3 of the picture frames detect the mobile phone;

[0032] In the same picture, the relative distance between the coordinates of the mobile phone and the coordinates of the "nose tip" in the human key point sequence obtained by the 2D human key point detection model is less than a preset threshold.

[0033] In another aspect, the application also provides a smoking and phone call detection system combining action recognition and target detection, which is used to implement the smoking and phone call detection method combining action recognition and target detection as described above.

[0034] Moreover, it includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the smoking and phone call detection method combining action recognition and target detection as described above.

[0035] Alternatively, it includes a readable storage medium, and the readable storage medium stores a computer program, and the computer program is executed to implement the smoking and phone call detection method combining action recognition and target detection as described above.

[0036] Overall, compared with the existing smoking and phone call detection technology, the above technical solutions conceived by the application can achieve the following beneficial effects:

[0037] (1) High accuracy and low false detection rate. The target detection model YOLOv5-S in the application focuses more on the detection of small targets, improving the detection accuracy of cigarettes and mobile phones. In addition, the application fully combines the advantages of human pose estimation and target detection model, that is, it analyzes human action behavior, detects two small targets of cigarettes and mobile phones, and combines the positional relationship between the two and the time sequence information of the event, making up for the defects caused by only using the target detection model and only using the human pose analysis model, and can obtain higher accuracy and lower false detection rate.

[0038] (2) Lower cost and stronger robustness. The training and debugging of the deep learning method only need to use GPU, the energy consumption of the testing process is low, and the images used for training are relatively easy to obtain, so the method cost is low. The application can detect in various scenes, such as:

[0039] Indoor public places, mine tunnels, chemical plants, etc. No additional hardware devices need to be installed, as long as the camera can work normally, the application can analyze the monitoring video to detect smoking and phone call behavior.

[0040] Therefore, the application combines human posture estimation and target detection to design a smoking and phone call detection scheme, the model training speed of which is fast and converges fast. The model has high detection accuracy, low missed detection rate, strong generalizability and robustness by combining time sequence information, human key points and the relative position of the cigarette or the mobile phone for comprehensive analysis. The problems of high missed detection rate and low accuracy of the current single model for smoking and phone call event detection can be effectively solved. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a data set annotation and model training flowchart of the embodiment of the application.

[0042] Figure 2 is a YOLOv5-S network model diagram improved by the embodiment of the application.

[0043] Figure 3 is a smoking and phone call method flowchart of the embodiment of the application. DETAILED DESCRIPTION

[0044] The technical scheme of the application will be described in detail below in combination with the drawings and embodiments.

[0045] Compared with general target detection methods, the method combines real-time human key point detection, human posture estimation, the position of the cigarette and the mobile phone relative to the human body and the time sequence information of the smoking and phone call behavior, effectively solving the problems of low detection accuracy and high missed detection rate of the smoking and phone call behavior caused by the shielding of the cigarette and the mobile phone and the long distance between the personnel and the monitoring camera in the detection process.

[0046] The embodiment of the application proposes a smoking and phone call detection method based on the combination of human posture estimation and target detection, including the following contents:

[0047] (1) installing a camera in a monitoring place, and transmitting the video images obtained by the camera in real time through a network and an rtsp protocol;

[0048] (2) pre-processing the video data, cutting the high-pixel long video with smoking and phone call behavior into short segments with a certain time length (considering the action duration and model processing capacity, the time length can be set to 4-6 seconds, and the embodiment preferably uses 4 seconds), and labeling the action behavior as the video training data of the MixSTE 3D human posture estimation model; at the same time, extracting video frames every 100 milliseconds from the short videos, labeling the cigarette and the phone, and taking them as the picture training data of the target detection model;

[0049] (3) For the task of smoking and calling detection, the YOLOv5 model is improved to be more suitable for small target detection, and the specific improvement scheme is: the convolution operation in the Backbone module of the YOLOv5 model is changed to deformable convolution, and the pooling operation (Maxpool) in the SPP module in the Backbone is changed to deformable pooling; at the same time, a spatial attention module (corresponding to the attention module of Figure 2 ) is added between each CSP structure; according to the size of the smoke and the mobile phone in the training data, an anchor box of a suitable size is designed (deformable convolution, deformable pooling, and spatial attention module are prior art schemes, which will not be described herein). In the present application, the improved YOLOv5 model is named YOLOv5-S.

[0050] (4) The 2D human key point detection network, the human pose estimation model and the YOLOv5-S small target detection network model are trained respectively using video training data and picture training data, and the model parameters of the three models are obtained, and finally the trained detection model is obtained; the 2D human key point detection network is preferably CPN, and the 3D human pose estimation model is preferably MixSTE, and CPN and MixSTE are prior network models, which will not be described herein.

[0051] (5) The monitoring video is obtained by using a monitoring camera, and 4 seconds is taken as a video segment as the input of the detection model. The complete smoking and calling behavior detection model is divided into two routes, the first route: the obtained video segment is input into the MixSTE 3D human pose estimation model to obtain the coordinate sequence of the 2D human key points (intermediate result) and the human pose estimation result and the result confidence (final result); the second route: the obtained video segment is converted into a picture frame, which is input into the YOLOv5-S target detection model to obtain the coordinates and confidence of the smoke or the mobile phone (if existing).

[0052] (6) According to the output result of the detection model, the relative positions of the human key points and the smoke or the mobile phone are comprehensively analyzed in combination with the time sequence information, and finally it is determined whether the smoking and calling behavior occurs in the video segment.

[0053] The present application further proposes that the labeling of the data set of steps (2) and (4) and the model training are implemented as follows:

[0054] After obtaining the smoking and calling monitoring videos in various scenes, the videos are cut into video segments with a time of 4 seconds, the video segments are converted into video frames;

[0055] The obtained video and image data are labeled in various forms according to the needs of the model:

[0056] a. human key points are labeled, and finally a 2D human key point sequence dataset is obtained;

[0057] b. the behavior of the personnel appearing in the video segment is labeled, and finally a human pose estimation dataset is obtained;

[0058] c. smoke and mobile phones appearing in the picture are labeled, and finally a target detection dataset is obtained;

[0059] The above-mentioned dataset is used to train the model involved in the application:

[0060] a. The 2D human key point detection network CPN is trained using the 2D human key point sequence dataset as input, and the CPN network model parameters are obtained;

[0061] b. The MixSTE 3D human pose estimation model is trained using the video segment and the 2D human key point sequence dataset as input, and the MixSTE network model parameters are obtained;

[0062] c. The YOLOv5-S network model is trained using the target detection dataset as input, and the YOLOv5-S model parameters are obtained.

[0063] Figure 1 The labeling and model training process of the dataset in the embodiment is shown in the following flowchart, and the specific steps are described as follows:

[0064] (1-1) After obtaining the monitoring videos of smoking and making phone calls in various scenes, the videos are cut into video segments with a time of 4 seconds.

[0065] (1-2) The video segments are converted into video frames to obtain image data. The collected images are cropped, mirror flipped, and adjusted for color contrast, light brightness, image angle, and other data enhancement operations. Line stripe noise points are added to enhance the robustness of the dataset.

[0066] (1-3) The obtained video and image data are labeled in various forms according to the needs of the model.

[0067] a. Human key points are labeled, and the specific labeling requirements are as follows: for each human body in the image, 17 feature points are labeled in order: 0 nose tip, 1 left eye, 2 right eye, 3 left ear, 4 right ear, 5 left shoulder joint, 6 right shoulder joint, 7 left elbow joint, 8 right elbow joint, 9 left wrist joint, 10 right wrist joint, 11 left hip joint, 12 right hip joint, 13 left knee joint, 14 right knee joint, 15 left ankle joint, 16 right ankle joint, and finally a 2D human key point sequence dataset is obtained;

[0068] b. Label the behavior of the personnel appearing in the video clip, and for the detection task of smoking and making a phone call, label the personnel action as smoking, making a phone call or normal behavior, and finally obtain the human pose estimation dataset;

[0069] c. Label the smoke and mobile phone appearing in the picture: 1 smoke, 2 mobile phone, and finally obtain the target detection dataset;

[0070] (S1-4) Using the dataset constructed in (S1-3) to train the model involved in the application.

[0071] a. Using the picture as input, training the 2D human key point detection network CPN using the 2D human key point sequence dataset to obtain the CPN network model parameter;

[0072] b. Using the video clip and the 2D human key point sequence dataset as input, and using the human pose estimation dataset as GT value, training the MixSTE 3D human pose estimation model to obtain the MixSTE network model parameter;

[0073] c. Using the picture as input, training the YOLOv5-S network model using the target detection dataset to obtain the YOLOv5-S model parameter.

[0074] The basic theoretical support for improving the YOLOv5 model is:

[0075] In the field of small target detection, the problem of poor small target detection performance has not been completely solved. Compared with the regular size target, the small target occupies fewer pixels in the image, has lower resolution and weaker feature expression ability. At present, small target detection mainly increases the precision of small target detection through feature enhancement, multi-scale feature fusion, learning high-resolution features with large receptive field, introducing attention mechanism, introducing context information and designing appropriate anchor box.

[0076] On the basis of this theory, the following improvements are made:

[0077] The YOLOv5-S network model is improved on the basis of the YOLOv5 model, so as to focus more on the detection of small targets. The network model mainly has four modules: input end, Backbone, Neck and prediction module, the input end, Neck and prediction module use the YOLOv5 network module, the Neck adopts the feature pyramid structure of FPN+PAN, and the improvement of the model is mainly concentrated in the Backbone module;

[0078] The backbone module mainly comprises the following network sub-modules: DBL, Res unit, CSP1_X, CSP2_X and SPP. The size of the convolution kernel in all convolution layers is aligned with the size of the convolution kernel in the original YOLOv5 network model. The structure implementation of each network sub-module is introduced as follows:

[0079] DBL: After the input feature map is subjected to deformable convolution, it is subjected to a BN layer, and finally subjected to a Leaky relu activation function to obtain an output feature map;

[0080] Res unit: The input feature map is subjected to two DBL modules to obtain a new feature map, and the new feature map is added to the input feature map to obtain an output feature map;

[0081] CSP1_X: The input feature map is subjected to a DBL module, X Res unit modules and deformable convolution to obtain a new feature Figure 1 , the input feature map is subjected to deformable convolution to obtain a new feature Figure 2 , the new feature Figure 1 and the new feature Figure 2 are spliced, and then subjected to a BN layer and a Leaky relu activation function, and finally subjected to a DBL module to obtain an output feature map;

[0082] CSP2_X: The input feature map is subjected to (2X+1) DBL modules and deformable convolution to obtain a new feature Figure 1 , the input feature map is subjected to deformable convolution to obtain a new feature Figure 2 , the new feature Figure 1 and the new feature Figure 2 are spliced, and then subjected to a BN layer and a Leaky relu activation function, and finally subjected to a DBL module to obtain an output feature map;

[0083] SPP: The input feature map is subjected to three Maxpool layers to obtain new features Figure 1 , new feature Figure 2 and new feature Figure 3 , the obtained new features Figure 1 , 2 , 3 and the input feature map are spliced, and then subjected to a DBL module to obtain an output feature map;

[0084] The input picture is sequentially subjected to DBL, CSP1_1, a spatial attention module, DBL, CSP1_3, a spatial attention module to obtain feature Figure 1 , feature Figure 1 is sequentially subjected to DBL, CSP1_3, a spatial attention module to obtain feature Figure 2 , feature Figure 2 is sequentially subjected to DBL, SPP, CSP2_1, a spatial attention module, DBL to obtain feature Figure 3characteristics Figure 1 , 2 , 3 input FPN+PAN feature pyramid structure, input to the prediction module. Finally, according to the size of the smoke and mobile phone in the training data, set the appropriate size of the anchor box, and predict the target position and category.

[0085] In summary, the improved YOLOv5-S small target detection model can make the model more focused on the feature representation of small targets when detecting small targets (smoke and mobile phones in the task), enrich the expression of small target feature information, and ultimately improve the detection accuracy of small targets.

[0086] Figure 2 The improved YOLOv5-S network model diagram for the embodiment of the application is as follows:

[0087] The YOLOv5-S network model is improved on the YOLOv5 model to focus more on small target detection. The network model mainly has four modules: input end, Backbone, Neck and prediction module, the input end, Neck and prediction module use the YOLOv5 network module, the Neck adopts the FPN+PAN feature pyramid structure, and the improvement of the model is mainly concentrated in the Backbone module;

[0088] The Backbone module mainly includes the following network sub-modules: DBL, Res unit, CSP1_X, CSP2_X and SPP. The size of the convolution kernel in all convolution layers is aligned with the size of the convolution kernel in the original YOLOv5 network model, and the introduction of each part is as follows:

[0089] DBL: the input feature map is obtained after the deformable convolution DCN, through the BN layer, and finally through the Leaky relu activation function to obtain the output feature map;

[0090] Res unit: the input feature map is obtained through 2 DBL modules to obtain a new feature map, and the new feature map is added to the input feature map to obtain an output feature map;

[0091] CSP1_X: the input feature map is obtained through a DBL module, X Res unit modules and deformable convolution to obtain a new feature Figure 1 , the input feature map is obtained through deformable convolution to obtain a new feature Figure 2 , the new feature Figure 1 and the new feature Figure 2 are spliced, and then a BN layer and a Leaky relu activation function are used, and finally a DBL module is used to obtain an output feature map;

[0092] CSP2_X: the input feature map is obtained through (2X+1) DBL modules and deformable convolution to obtain a new featureFigure 1 , the input feature map is obtained by deformable convolution to obtain a new feature Figure 2 , the new feature Figure 1 and the new feature Figure 2 After splicing, an output feature map is obtained through a BN layer and a Leakyrelu activation function, and finally through a DBL module.

[0093] SPP: the input feature map is respectively obtained by three Maxpool layers to obtain new features Figure 3 , the new feature Figure 1 and the new feature Figure 1 , the obtained new features Figure 1 , 2 , 3 and the input feature map are spliced, and then an output feature map is obtained through a DBL module.

[0094] The input picture is sequentially passed through DBL, CSP1_1, a spatial attention module, DBL, CSP1_3, a spatial attention module to obtain feature Figure 2 , the feature Figure 2 is sequentially passed through DBL, CSP1_3, a spatial attention module to obtain feature Figure 3 , the feature Figure 1 is sequentially passed through DBL, SPP, CSP2_1, a spatial attention module, DBL to obtain feature Figure 3 , the feature Figure 1 , 2 , 3 input FPN+PAN feature pyramid structure, and then to the prediction module for target position and category prediction.

[0095] Figure 1 The flow chart of the detection method of the embodiment of the application is shown in the following table (the data enhancement, data set construction and model training of the flow chart are subject to the following table): ​

[0096] (3-1) Obtain the monitoring video of the monitoring place by the network and the rtsp protocol real-time transmission camera;

[0097] (3-2) Divide the high-resolution long-time video into short video segments of 4 seconds each;

[0098] (3-3) Construct three types of data sets according to the ​ flow and train three network models;

[0099] (3-4) Input the video segment into the CPN 2D human key point detection model to obtain the 2D human key point coordinate sequence output by the model;

[0100] ​(3-5) input the 2D human key point coordinate sequence obtained in (3-4) into the MixSTE 3D human pose estimation model to obtain a human action category in a video output by the model;

[0101] (3-6) convert the video segment obtained in (3-2) into picture frames, and input the picture frames into a YOLOv5-S target detection model to obtain coordinates and confidence of a cigarette or a mobile phone detected by the YOLOv5-S model

[0102] (3-7) combine the 2D human key point coordinate sequence obtained in (3-4), the human action category in the video obtained in (3-5), and the coordinates and confidence of the cigarette or the mobile phone obtained in (3-6) to perform comprehensive analysis in time sequence, and specifically:

[0103] when the following conditions are simultaneously satisfied, it is considered that there is a smoking behavior in the video: the human action category obtained in (3-5) is "smoking"; the picture frames included in the video segment are detected by the YOLOv5-S model in (3-6), and 1 / 3 or more of the picture frames detect a cigarette (a confidence of 0.5 or more is determined as "detecting a cigarette"); in the same picture, the relative distance between the coordinates of the cigarette and the coordinates of the "wrist joint" in the human key point sequence obtained in (3-4) is less than 100 pixels;

[0104] when the following conditions are simultaneously satisfied, it is considered that there is a phone call behavior in the video: the human action category obtained in (3-5) is "making a phone call"; the picture frames included in the video segment are detected by the YOLOv5-S model in (3-6), and 1 / 3 or more of the picture frames detect a mobile phone (a confidence of 0.5 or more is determined as "detecting a mobile phone"); in the same picture, the relative distance between the coordinates of the mobile phone and the coordinates of the "nose tip" in the human key point sequence obtained in (3-4) is less than 100 pixels;

[0105] (3-8) obtain the analysis result in (3-7), and if the video segment has a smoking and phone call behavior, issue a smoking and phone call alarm.

[0106] The above method fully combines the advantages of human pose estimation and target detection model, and makes up for the shortcomings of single model detection. Meanwhile, the relative positions of the time sequence information, the human key points, and the cigarette or the mobile phone are combined to effectively improve the accuracy of the smoking and phone call event detection, reduce the event missing rate, and have strong generalizability and robustness.

[0107] In specific implementation, the method proposed in the technical scheme of the present application can be automatically run by a computer software technology, and the system device for running the method, such as a computer readable storage medium storing the corresponding computer program of the technical scheme of the present application and a computer device including the running corresponding computer program, should also be within the protection scope of the present application.

[0108] In some possible embodiments, a smoking and phone call detection system combining action recognition and target detection is provided, comprising a processor and a memory, the memory being configured to store program instructions, and the processor being configured to invoke the stored instructions in the memory to perform a smoking and phone call detection method combining action recognition and target detection as described above.

[0109] In some possible embodiments, a smoking and phone call detection system combining action recognition and target detection is provided, comprising a readable storage medium, and a computer program is stored on the readable storage medium, and the computer program is configured to implement a smoking and phone call detection method combining action recognition and target detection as described above when executed.

[0110] The above merely provides preferred embodiments of the present application but not for limiting the present application. For those skilled in the art, the present application can have various modifications and changes. Any modified, equivalent replaced, improved, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting smoking and phone calls by combining action recognition and target detection, characterized in that: Video data of the monitored scene is collected through surveillance cameras; the video data is segmented into short video clips, and the actions of the people in the video are annotated; the video clips are converted into video frames, and the coordinates of human key points, smoke, and mobile phones are annotated for each video frame to obtain the model training dataset; a YOLOv5-S object detection network model is established, which adds variable convolution and attention mechanisms to the YOLOv5 model, making it more suitable for detecting small targets; then, the 2D human key point detection network, the 3D human pose estimation model, and the YOLOv5-S object detection network model are trained using the constructed dataset to obtain the weights of each trained network model; finally, the surveillance video of the monitored location is input into the trained network model to determine whether smoking or making a phone call occurs in the video clip; The dataset annotation and model training are implemented as follows: After acquiring surveillance videos of smoking and making phone calls in various scenarios, the videos are trimmed into shorter video clips and then converted into video frames. According to the model requirements, the obtained video and image data are annotated in various forms, including annotating human key points to obtain a 2D human key point sequence dataset; annotating human behaviors in video clips to obtain a human pose estimation dataset; and annotating smoke and mobile phones in images to obtain an object detection dataset. The models were trained using constructed datasets, including: using images as input to train the 2D human keypoint detection network CPN using a 2D human keypoint sequence dataset to obtain CPN network model parameters; using video clips and a 2D human keypoint sequence dataset as input, and a human pose estimation dataset as ground truth (GT) values ​​to train the MixSTE 3D human pose estimation model to obtain MixSTE network model parameters; and using images as input to train the YOLOv5-S network model using an object detection dataset to obtain YOLOv5-S model parameters. When improving the YOLOv5 model to a YOLOv5-S model focused on small object detection, the Backbone module of the YOLOv5 model is adjusted. In the adjusted Backbone module, the input image is sequentially passed through DBL, CSP1_1, spatial attention module, DBL, CSP1_3, and spatial attention module to obtain feature map 1. Feature map 1 is sequentially passed through DBL, CSP1_3, and spatial attention module to obtain feature map 2. Feature map 2 is sequentially passed through DBL, SPP, CSP2_1, spatial attention module, and DBL to obtain feature map 3. Feature maps 1, 2, and 3 are input into the FPN+PAN feature pyramid structure and then into the prediction module. Finally, based on the size of the smoke and mobile phone in the training data, corresponding anchor boxes are set to predict the target location and category.

2. The smoking and phone call detection method combining action recognition and target detection according to claim 1, characterized in that: The implementation process includes the following steps: (1) Install cameras in the monitored area and transmit the video images acquired by the cameras in real time; (2) Preprocess the video data, cut the high-pixel long videos with smoking and phone calls into short segments and convert them into video frames, label the human behavior in the video and the targets in the video frames, and construct a dataset. (3) Establish a YOLOv5-S object detection network model, including improvements to the YOLOv5 model to make it more suitable for small object detection. The improvement method is to change the convolution operation in the Backbone module of the model to deformable convolution and the pooling operation to deformable pooling. Incorporate a spatial attention mechanism; set corresponding anchor boxes based on the sizes of the cigarette and mobile phone in the training data; (4) The 2D human key point detection network, the 3D human pose estimation model and the YOLOv5-S small target detection network model were trained using video training data and image training data. The model parameters of the three models were obtained respectively, and the trained detection model was finally obtained. (5) Use surveillance cameras to acquire surveillance videos, crop video clips, and use them as input to the smoking / phone call detection model. Input the trained detection model to obtain the coordinate sequence of 2D human key points, human pose estimation results and confidence scores, as well as the corresponding coordinates and confidence scores when there is smoke or a mobile phone. (6) Analyze the output of the detection model, and combine it with the time sequence information, key points of the human body and the relative position of the cigarette or mobile phone to conduct a comprehensive analysis, and finally determine whether smoking or making a phone call occurred in the video clip.

3. The smoking and phone call detection method combining action recognition and target detection according to claim 1 or 2, characterized in that: When comprehensively analyzing the output of the detection model, smoking behavior is considered to exist in the video if the following conditions are met simultaneously: The 3D human pose estimation model outputs a human action category of "smoking"; The YOLOv5-S model detected smoke in one-third or more of the image frames contained in the video clip; In the same image, the relative distance between the coordinates of the smoke and the coordinates of the "wrist joint" in the human keypoint sequence obtained by the 2D human keypoint detection model is less than a preset threshold.

4. The smoking and phone call detection method combining action recognition and target detection according to claim 1 or 2, characterized in that: When comprehensively analyzing the output of the detection model, the video is considered to contain phone call behavior if the following conditions are met simultaneously: The 3D human pose estimation model outputs a human action category of "making a phone call"; The YOLOv5-S model detected that 1 / 3 or more of the image frames in the video clips were from a mobile phone. In the same image, the relative distance between the coordinates of the mobile phone and the coordinates of the "nose tip" in the human keypoint sequence obtained by the 2D human keypoint detection model is less than a preset threshold.

5. A smoking and phone call detection system combining action recognition and target detection, characterized in that: This method is used to implement a smoking and phone call detection method that combines action recognition and target detection as described in any one of claims 1-4.

6. An electronic device, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a smoking and phone call detection method combining action recognition and target detection as described in any one of claims 1-4.

7. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed, implements a smoking and phone call detection method combining action recognition and target detection as described in any one of claims 1-4.

Images