A smoking recognition method based on graph neural network

Abstract

This paper discloses a method for detecting smoking based on a graph neural network. This method constructs a deep learning framework that can automatically identify people who are smoking. The method uses a surveillance camera to capture a human frame diagram and a convolutional neural network to capture key features of human movements and features of cigarette accessories. These two features are then modeled and analyzed using a graph neural network. Compared to conventional object detection algorithms, this method reduces false positives due to detecting only cigarettes and false negatives due to cigarette obstructions. Furthermore, compared to attribute classification algorithms, it exhibits greater robustness and generalization in new scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of graph neural network deep learning technology and relates to a smoking recognition method based on graph neural network. Background Art

[0002] Smoking is not only harmful to health but also a major cause of indoor fires. With the development of deep learning, automatic smoke detection algorithms based on deep learning have emerged. Representative works in recent years include:

[0003] 1. CN201910082457.4: A deep learning-based smoking detection and analysis system. This invention uses a video stream as input and uses a multi-scale convolutional fusion mechanism to detect objects such as cigarettes, license plates, and car windows. However, its application scenario is limited, applicable only to monitoring scenarios such as checkpoints to detect driver smoking. It does not consider motion characteristics or the presence of occluded objects.

[0004] 2.CN202111511659.X A safety monitoring method for helmet wearing and smoking detection. This invention uses pictures as input, and in the power plant scene, obtains the key point features of the human body through the OpenPose algorithm, and obtains the accessory features through yolov5. Although the key point information of the human body is used, the feature processing of the key points is too simple and only stays at the prior processing of spatial relationships. For example, if there is a cigarette near the hand, it is judged as smoking, but the overall action of smoking is not taken into account. In actual use, it is also necessary to reasonably set hyperparameters such as thresholds, and the application is cumbersome and the generalization is not strong. In addition, two feature extraction algorithms are used, which require separate training each time, which also results in a waste of hardware resources.

[0005] 3.CN202110795174.1 A driver smoking detection method based on multiple models. This invention inputs the driver smoking detection area image into the smoking gesture recognition model and the cigarette area recognition model. Obtain the probability map of smoking gesture recognition and the probability map of cigarette recognition; analyze the results of the probability map of smoking gesture recognition and the probability map of cigarette recognition, and when the smoking gesture and cigarette are recognized at the same time, it is detected that the driver is smoking. Cigarette features are obtained through the instance segmentation model, but the application scenario of this method is too ideal. For small and delicate targets such as gestures and cigarettes, the effect will be very unsatisfactory in the actual scene when there is occlusion, blur, and insufficient lighting. The use of instance segmentation also makes the sample data annotation cost of this invention too high. If new scenarios and new requirements are added, a lot of annotation time and labor costs will be consumed. In addition, the multi-model solution is adopted, which cannot achieve end-to-end training and testing.

[0006] To address these issues, the present invention employs a graph neural network architecture, performing deeper modeling and learning of key point features and cigarette accessory features. This allows the algorithm to learn more essential smoking characteristics, significantly reducing false positives and false negatives. This allows for rapid implementation in new scenarios while maintaining algorithm accuracy. Furthermore, the feature extraction models employed in this invention share a common algorithmic framework, reducing the number of model parameters, increasing algorithm speed, and enabling end-to-end training and testing. Summary of the Invention

[0007] To achieve more effective smoking recognition, the present invention provides a smoking recognition method based on graph neural network. This method adopts a unified feature extraction network and uses the graph neural network structure to learn more essential smoking features.

[0008] The present invention adopts the following technical solutions:

[0009] A smoking recognition method based on graph neural network, comprising:

[0010] S1. Create a dataset: Set up a surveillance camera at the location where the algorithm is to be used. Use the camera feed as input. Extract an image every few frames and annotate the key points of the human body and the cigarette outline. If a person is smoking in the image, label the image as "smoking" (hereinafter referred to as a "positive sample"). If no one is smoking in the image, label the image as "not smoking" (hereinafter referred to as a "negative sample").

[0011] S2. Use the above dataset to train a smoking recognition model based on graph neural network

[0012] The smoking recognition model includes a skeleton network, a target detection sub-network, a key point feature extraction network, and a graph convolutional neural network.

[0013] The skeleton network uses ResNet50, which accepts the input image I and the feature maps C3, C4, and C5 of the 3rd, 4th, and 5th convolutional layers as the output of the skeleton network;

[0014] The target detection sub-network uses the RetinaNet network, which includes the FPN module, the detection block diagram sub-network and the detection classification sub-network. The FPN module accepts C3, C4, and C5 as inputs, and the detection block diagram sub-network and the detection classification sub-network take the output of FPN as input and output the block diagram B of the i-th cigarette accessory in the image. i And the visible value v; if a cigarette is detected, set v to 1, otherwise it is 0.

[0015] The key point feature extraction network uses KeyPointRCNN, which shares a skeleton network with the target detection subnetwork. The output is an N*3 matrix, where N is preferably 17, which contains information about the 17 key points of the human body (the key points are set with reference to the coco dataset: 0: nose, 1: left eye, 2: right eye, 3: left ear, 4: right ear, 5: left shoulder, 6: right shoulder, 7: left elbow, 8: right elbow, 9: left wrist, 10: right wrist, 11: left hip, 12: right hip, 13: left knee, 14: right knee, 15: left ankle, 16: right ankle), as well as the x, y, and v values ​​of each key point. x and y represent coordinates, and v represents visibility. If visible, v is set to 1, otherwise it is set to 0.

[0016] The obtained cigarette attachment block diagram B i , calculate its center point and merge it with the output of the key point feature extraction network to form an (N+1)*3 matrix A. At the same time, add a 1-dimensional vector marker (token) to indicate whether the person in the image is smoking, and finally form an (N+2)*3 matrix B.

[0017] The graph convolutional neural network takes B as input and outputs a marker token. If it is a 0 vector, it indicates that there is no smoking, and if it is a 1 vector, it indicates that someone is smoking.

[0018] Furthermore, the input I is unified to the same size, and data enhancement and normalization are performed.

[0019] Furthermore, the ImageNet pre-trained model was selected as the initial model, and a public dataset (coco key point dataset) was also used for pre-training before the above training.

[0020] Furthermore, the annotations for this dataset are also in the (N+2)*3 format of the matrix B, which is used to train graph convolutional neural networks. After pre-training, the network can also be used to automatically annotate negative samples, saving annotation costs and time.

[0021] This invention utilizes a graph neural network architecture to perform deeper modeling and learning of key point features and cigarette accessory features. Compared to conventional object detection algorithms, this invention reduces false positives due to detecting only cigarettes and false negatives due to cigarette obstructions. It can also be quickly deployed in new scenarios while maintaining algorithm accuracy, offering enhanced robustness and generalization capabilities. Furthermore, the feature extraction model employed in this invention shares a common algorithmic framework, reducing the number of model parameters, improving algorithm execution speed, and enabling end-to-end training and detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Feature extraction network training flow chart;

[0023] Figure 2 End-to-end training flowchart of the smoking recognition algorithm based on graph neural network; DETAILED DESCRIPTION

[0024] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments. The technical features of each embodiment of the present invention may be combined accordingly, provided that there is no conflict between them.

[0025] Example

[0026] During the production process, a company needs to monitor employees' smoking behavior in multiple locations due to production safety requirements.

[0027] 1. Dataset Construction

[0028] In this embodiment, a data collection system was established. 150 hours of video data were collected through surveillance cameras installed in various locations. One frame was extracted every 24 seconds, and an open source human detection algorithm based on yolov5 was used to obtain 40,000 valid human samples, of which 3,500 were positive samples of smoking. The 17 key points of the human body and the location of the cigarette were marked for all samples, as well as whether they were visible. The data was merged into an 18*3 matrix. At the same time, a 1-dimensional vector marker (token) was added to indicate whether the person in the image was smoking. Finally, it became a 19*3 matrix.

[0029] 2. Smoking recognition algorithm training

[0030] like Figure 1 As shown in the figure, during end-to-end training, the feature extraction module network is pre-trained first. This means that the ImageNet pre-trained model is used as the initial model and pre-trained using a public dataset (the Coco keypoint dataset). The feature extraction module includes a skeleton network, an object detection sub-network, and a keypoint feature extraction network.

[0031] The skeleton network uses ResNet50, which accepts the input image I and the feature maps C3, C4, and C5 of the 3rd, 4th, and 5th convolutional layers as the output of the skeleton network;

[0032] The target detection sub-network uses the RetinaNet network, which includes the FPN module, the detection block diagram sub-network and the detection classification sub-network. The FPN module accepts C3, C4, and C5 as inputs, and the detection block diagram sub-network and the detection classification sub-network take the output of FPN as input and output the block diagram B of the i-th cigarette accessory in the image. i And the visible value v; if a cigarette is detected, set v to 1, otherwise it is 0.

[0033] The keypoint feature extraction network uses KeyPointRCNN, which shares a skeleton network with the object detection subnetwork. The output is a 17x3 matrix containing information about the 17 keypoints of the human body, including the x, y, and v values ​​of each keypoint. x and y represent coordinates, and v represents visibility. If visible, v is set to 1, otherwise it is set to 0.

[0034] During the pre-training process, the weights of the entire model are updated through back-propagation.

[0035] After completing the pre-training, end-to-end training is performed. i Calculate the center point x and y and combine them with the output of the key point feature extraction network to form an 18*3 matrix A. Also add a 1-dimensional vector token to indicate whether the person in the image is smoking. The final matrix becomes a 19*3 matrix B.

[0036] The graph convolutional neural network takes B as input and outputs a vector token. A 0 vector indicates no smoking, while a 1 vector indicates smoking.

[0037] To verify the effectiveness of this method, a comparative experiment was conducted based on the above dataset. The baseline models used for comparison were a ResNet50 classification model using the same backbone (referred to as the classification model) and an accessory recognition model that only includes the object detection subnetwork (referred to as the object detection model).

[0038]

[0039] To verify its generalization, smoking samples collected from another environment were used as the test set for cross-domain generalization testing.

[0040]

[0041] This invention utilizes a graph neural network architecture to perform deeper modeling and learning of key point features and cigarette accessory features. This allows the algorithm to learn more essential smoking characteristics, significantly reducing false positives and false negatives. It can be quickly deployed in new scenarios while ensuring algorithm accuracy. Furthermore, the feature extraction model employed in this invention shares a common algorithmic framework, which reduces the number of model parameters, improves algorithm execution speed, and enables end-to-end training and testing.

Claims

1. A smoking recognition method based on graph neural network, characterized in that: These include: S1. Create a dataset: Annotate surveillance images with key points of human bodies and cigarette frames. If a person is smoking in the image, label the image as "smoking" (positive sample); if no one is smoking in the image, label the image as "not smoking" (negative sample). S2. Use the above dataset to train a graph neural network-based smoking recognition model. The smoking recognition model includes a skeleton network, a target detection sub-network, a key point feature extraction network, and a graph convolutional neural network; The skeleton network uses ResNet50 and accepts input images I , the feature maps of the 3rd, 4th, and 5th convolutional layers C 3 ,C 4 ,C 5 As the output of the skeleton network; The target detection sub-network uses the RetinaNet network, which includes the FPN module, the detection block diagram sub-network and the detection classification sub-network; the FPN module accepts C 3 ,C 4 ,C 5 As input, the detection block diagram sub-network and the detection classification sub-network take the output of FPN as input and output the image first i Block diagram of cigarette accessories B i and visible values v ; If a cigarette is detected, v is 1, otherwise it is 0; The key point feature extraction network uses KeyPointRCNN, which shares the skeleton network with the target detection sub-network and outputs an N*3 matrix containing the information of N key points of the human body and the information of each key point. x,y,v ; x,y represents the coordinates, v Indicates whether it is visible or not. If visible, set v is 1, otherwise it is 0; Block diagram of the cigarette accessories obtained B i , calculate its center point, and merge it with the output of the key point feature extraction network into a (N+1)*3 matrix A At the same time, a 1-dimensional vector marker token is added to indicate whether the person in the picture is smoking, and finally it becomes a (N+2)*3 matrix B ; The graph convolutional neural network is based on B The input is token, and the output is a 0 vector indicating no smoking, and a 1 vector indicating someone is smoking.

2. The smoking recognition method based on graph neural network according to claim 1, characterized in that: The key points of the human body are set as 0: nose, 1: left eye, 2: right eye, 3: left ear, 4: right ear, 5: left shoulder, 6: right shoulder, 7: left elbow, 8: right elbow, 9: left wrist, 10: right wrist, 11: left hip, 12: right hip, 13: left knee, 14: right knee, 15: left ankle, 16: right ankle.

3. The smoking recognition method based on graph neural network according to claim 1, characterized in that: The input of the smoking recognition model I All are unified to the same size, and data augmentation and normalization are performed.

4. The smoking recognition method based on graph neural network according to claim 1, characterized in that: The ImageNet pre-trained model is selected as the initial model, and the public dataset coco key point dataset is used for pre-training.

5. The smoking recognition method based on graph neural network according to claim 1, characterized in that: The annotations when constructing the dataset are also in the (N+2)*3 format of matrix B, which is used to train the graph convolutional neural network. After pre-training, the network is used to automatically annotate negative samples.

Images