A method, apparatus, device and storage medium for detecting smoking behavior

Abstract

This invention discloses a method, apparatus, device, and storage medium for detecting smoking behavior. It employs a two-stage judgment process to detect smoking behavior, solving the problem of low accuracy associated with single-target detection schemes. Furthermore, this method utilizes a camera for indirect detection, offering broad coverage and 24 / 7 real-time monitoring capabilities. This reduces labor costs while ensuring accurate detection without missed or false alarms, providing a foundation for safe production in factories. The method also features real-time, non-contact, and multi-dimensional detection, representing a component of three-dimensional image detection technology in factory safety assurance applications. It possesses strong generalization and compatibility, adapting to different environments through training, and can be subsequently applied to office areas, school areas, hospitals, and other similar locations.

Description

Technical Field

[0001] This invention relates to the field of behavior detection technology, and in particular to a method, apparatus, equipment, and storage medium for detecting smoking behavior in a factory setting using image detection technology. Background Technology

[0002] In recent years, factory safety accidents have occurred frequently, with fires causing the greatest harm and losses. Countless fires are caused by carelessly discarded cigarette butts in factory areas. Therefore, fire prevention and fire detection are particularly important. However, direct methods of detecting smoking by workers are limited due to restrictions. In modern factories, cameras are ubiquitous, offering wide coverage, numerous deployments, and flexible customization options for additions and reductions. Furthermore, with the increasing maturity and widespread application of deep learning technologies, such as convolutional neural networks, in factories, many examples of using neural network image detection technology for safety monitoring have emerged.

[0003] There are several existing solutions for smoke detection in factories. The most traditional solution is to use a combination of administrator patrols and entry inspections, which is carried out by manual inspection. This method is generally suitable for areas with fewer personnel and smaller factory areas. It is not suitable for factories with larger factory areas and more employees. At the same time, this solution consumes a lot of manpower and cannot achieve 24-hour real-time monitoring.

[0004] In addition, some factories use smoke detectors for smoke detection. This solution is only suitable for smaller indoor environments and is not applicable to open factory areas or outdoor settings. Also, this method requires additional equipment installation, which is not conducive to cost control.

[0005] Similarly, with the advancement of image detection technology, there are also solutions that use factory cameras to collect images and perform target detection to detect smoking. However, since this solution only uses a single target detection method to detect smoking, its detection accuracy does not reach the level of practical use, and the occurrence of more false alarms and false negatives will cause additional screening work. Summary of the Invention

[0006] In view of the above problems, the present invention provides a method, apparatus, device, and storage medium for detecting smoking behavior to overcome or at least partially solve the above problems. The method employs a two-stage judgment process to detect smoking behavior, solving the problem of low accuracy when using a single target detection scheme. Furthermore, this method uses a camera for indirect detection, providing wide coverage and 24 / 7 real-time continuous detection capability. While reducing labor costs, it achieves zero missed detections and zero false detections, providing a foundation for safe production in factories.

[0007] This invention provides the following solution:

[0008] A method for detecting smoking behavior includes:

[0009] The trained first object detection network is used to detect pedestrians in multiple consecutive frames of target images, obtaining multiple pedestrian detection boxes and key point information of various parts of the pedestrian's body; the target image is an image of the target region obtained by an image capturing device; the key point information of various parts of the pedestrian's body is used to determine the skeletal information of the pedestrian.

[0010] Multiple pedestrian detection boxes are used to match pedestrians with the same ID, and the ST_GCN behavior recognition algorithm is used to perform action behavior recognition to generate the first judgment result;

[0011] If the first judgment result indicates that smoking behavior exists, the trained second target detection network is used to detect the position of the cigarette within the pedestrian detection box to determine the cigarette position information.

[0012] Using the cigarette location information and the pedestrian's skeletal information, a second judgment result is generated by judging the smoking behavior based on prior knowledge.

[0013] If the second judgment result confirms the presence of smoking behavior, the audible and visual alarm in the target area will issue a warning, and the detection result will be uploaded to the intelligent monitoring system.

[0014] Preferably, both the first and second object detection networks include an improved YOLOv8 object detection network; the improved YOLOv8 object detection network includes a C2f module and a fusion module; the C2f module has 3-6-6-3 blocks and the first convolutional layer has a 3*3 kenel; the fusion module includes a feature fusion network FPN and a PAN.

[0015] Preferably, the improved YOLOV8 object detection network uses BCEloss as the classification loss and the detection box CIOU_Loss as the location information loss, as shown in Equation 1; the classification CLS_loss uses BCE_loss as shown in Equation 2 and the key point KP_loss as shown in Equation 4. Multimodal loss regression determines the category, location and key point information of the detection box through multiple dimensions.

[0016] CIOU_Loss=1―IOU(t ′ ,t) (Equation 1)

[0017] Among them, t ′ For GTbbox information;

[0018] CLs_Loss = BCE(c ′ n ,c n (Equation 2)

[0019] BCE=―((c n *logc ′ n )+(1―c n )*log(1―c ′ n (Formula 3)

[0020] Among them, c ′ n c represents the probability that the nth sample is a positive sample. n This represents the label value of the nth sample;

[0021]

[0022] Among them, v ′ k This refers to key information.

[0023] Preferably, the first target detection network further includes a KAPAO keypoint detection network; the KAPAO keypoint detection network integrates keypoint information into the anchor information generated by the improved YOLOV8 target detection network through a multi-task loss function, and uses the keypoint loss function to complete keypoint detection.

[0024] Preferably: the ST_GCN behavior recognition algorithm is used to perform action behavior recognition and generate the first judgment result; including:

[0025] Based on the characteristics of the human skeleton, the nodes of pedestrians with the same ID in each frame are connected to form a spatial boundary E. s ={v ti v tj |(i,j)∈H};

[0026] Among them, v ti v represents the position of the i-th node in frame t. tj This represents the position of the j-th node in the t-th frame;

[0027] Connect nodes with the same ID in two consecutive frames to form an edge, creating a frame subtraction graph structure E. f ={v ti v (t+1)i Establish graph relationships between pedestrians with the same ID across single frames, and graph relationships between pedestrians with the same ID across frames;

[0028] The graph relationships of the pedestrians and the graph relationships of the joints are used as input matrices and normalized. Spatial transformation is performed by alternating GCN and TCN using 9 sets of ST_GCN units. The GCN is used to learn the feature information of adjacent joints in space using graph convolutional networks, and the TCN is used to learn the features of joint changes in time series.

[0029] By utilizing the temporal and spatial features of the Graph overlay, the ATT module is used to weight different joints, pooling is used to obtain a 256-dimensional feature vector for each sequence, and the SoftMax function is used for classification to obtain the first judgment result.

[0030] Preferably: A second judgment result is generated by using the cigarette location information and the pedestrian's skeletal information to judge the smoking behavior based on prior knowledge; including:

[0031] The skeletal information of the pedestrian is used to determine key wrist points and body height information H.

[0032] The Euclidean distance S between the acquired wrist key point information and the cigarette position information is calculated, wherein the cigarette position information includes the centroid of the cigarette detection frame;

[0033] A second judgment result is generated based on the proportional relationship between the Euclidean distance S and the human body height information H.

[0034] Preferably, when the ratio of the Euclidean distance S to the human body height information H is 0.12, it is determined that the second judgment result indicates smoking behavior.

[0035] A smoking behavior detection device, the device comprising:

[0036] The pedestrian detection box acquisition unit is used to detect pedestrians in multiple consecutive frames of target images using a trained first target detection network, and to obtain multiple pedestrian detection boxes and key point information of various parts of the pedestrian's body; the target image is an image of the target region obtained by an image capturing device; the key point information of various parts of the pedestrian's body is used to determine the skeletal information of the pedestrian.

[0037] The first judgment result generation unit is used to match pedestrians with the same ID using multiple pedestrian detection boxes, and to generate the first judgment result by using the ST_GCN behavior recognition algorithm to perform action behavior recognition.

[0038] The cigarette location information determination unit is used to determine that the first judgment result indicates the presence of smoking behavior, and uses the trained second target detection network to detect and determine the location information of the cigarette contained in the pedestrian detection box;

[0039] The second judgment result generation unit is used to generate a second judgment result by using the cigarette location information and the pedestrian skeleton information to judge the smoking behavior through prior knowledge.

[0040] The alarm unit is used to determine that the second judgment result indicates smoking behavior, trigger the audible and visual alarm in the target area to issue a warning, and upload the detection result to the intelligent monitoring system.

[0041] A smoking behavior detection device, the device comprising a processor and a memory:

[0042] The memory is used to store program code and transmit the program code to the processor;

[0043] The processor is used to execute the smoking behavior detection method described above according to the instructions in the program code.

[0044] A computer-readable storage medium for storing program code for executing the above-described smoking behavior detection method.

[0045] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] This application provides a smoking behavior detection method, apparatus, device, and storage medium. It employs a two-stage judgment process to detect smoking behavior, solving the problem of low accuracy associated with single-target detection schemes. Furthermore, this method uses a camera for indirect detection, offering wide coverage and 24 / 7 real-time continuous detection capabilities. This reduces labor costs while ensuring no missed or false detections, providing a foundation for safe production in factories. The method also features real-time, non-contact, and multi-dimensional detection, representing a component of three-dimensional image detection technology in factory safety assurance applications. It possesses strong generalization and compatibility, adapting to different environments through training, and can be subsequently applied to office areas, school areas, hospital areas, and other similar locations.

[0047] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0049] Figure 1This is a flowchart of a smoking behavior detection method provided in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the factory smoke detection method provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of target detection and key point detection marking provided in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the KAPAO embedded yolov8 detection framework network provided in an embodiment of the present invention;

[0053] Figure 5 This is a diagram showing the detection results of pedestrians and key points in the laboratory environment provided in an embodiment of the present invention;

[0054] Figure 6 This is a diagram of the ST_GCN network structure provided in an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of smoking behavior recognition provided in an embodiment of the present invention;

[0056] Figure 8 This is a diagram showing the cigarette target detection results provided in an embodiment of the present invention;

[0057] Figure 9 This is a schematic diagram showing the distance between the cigarette and the elbow provided in an embodiment of the present invention;

[0058] Figure 10 This is a schematic diagram of a smoking behavior detection device provided in an embodiment of the present invention;

[0059] Figure 11 This is a schematic diagram of a smoking behavior detection device provided in an embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0061] See Figure 1 This invention provides a method for detecting smoking behavior, such as... Figure 1 As shown, the method may include:

[0062] S101: The trained first object detection network is used to detect pedestrians in multiple consecutive frames of target images, obtaining multiple pedestrian detection boxes and key point information of various parts of the pedestrian's body; the target image is an image of the target region obtained by an image capturing device; the key point information of various parts of the pedestrian's body is used to determine the skeletal information of the pedestrian;

[0063] S102: Match pedestrians with the same ID using multiple pedestrian detection boxes, and use the ST_GCN behavior recognition algorithm to perform action behavior recognition and generate the first judgment result; specifically, connect the nodes of pedestrians with the same ID in each frame according to the characteristics of the human skeleton to form a spatial boundary E. s ={v ti v tj |(i,j)∈H};

[0064] Among them, v ti v represents the position of the i-th node in frame t. tj This represents the position of the j-th node in the t-th frame;

[0065] Connect nodes with the same ID in two consecutive frames to form an edge, creating a frame subtraction graph structure E. f ={v ti v (t+1)i Establish graph relationships between pedestrians with the same ID across single frames, and graph relationships between pedestrians with the same ID across frames;

[0066] The graph relationships of the pedestrians and the graph relationships of the joints are used as input matrices and normalized. Spatial transformation is performed by alternating GCN and TCN using 9 sets of ST_GCN units. The GCN is used to learn the feature information of adjacent joints in space using graph convolutional networks, and the TCN is used to learn the features of joint changes in time series.

[0067] By utilizing the temporal and spatial features of the Graph overlay, the ATT module is used to weight different joints, pooling is used to obtain a 256-dimensional feature vector for each sequence, and the SoftMax function is used for classification to obtain the first judgment result.

[0068] S103: If the first judgment result indicates that smoking behavior exists, the trained second target detection network is used to detect the position of the cigarette contained in the pedestrian detection box to determine the cigarette position information;

[0069] S104: Using the cigarette location information and the pedestrian's skeletal information, a second judgment result is generated by judging the smoking behavior based on prior knowledge; specifically, the wrist key point information and the human body height information H are determined using the pedestrian's skeletal information.

[0070] The Euclidean distance S between the acquired wrist key point information and the cigarette position information is calculated, wherein the cigarette position information includes the centroid of the cigarette detection frame;

[0071] A second judgment result is generated based on the proportional relationship between the Euclidean distance S and the human body height information H.

[0072] Furthermore, when the ratio of the Euclidean distance S to the human body height information H is 0.12, it is determined that the second judgment result indicates the presence of smoking behavior.

[0073] S105: If the second judgment result indicates that smoking behavior exists, trigger the audible and visual alarm in the target area to issue a warning, and upload the detection result to the intelligent monitoring system.

[0074] Furthermore, both the first and second object detection networks include an improved YOLOv8 object detection network; the improved YOLOv8 object detection network includes a C2f module and a fusion module; the C2f module has 3-6-6-3 blocks and the first convolutional layer has a 3*3 kenel; the fusion module includes a feature fusion network FPN and a PAN.

[0075] The improved YOLOV8 object detection network uses BCEloss as the classification loss and the detection box CIOU_Loss as the location information loss, as shown in Equation 1. The classification CLS_loss uses BCE_loss as shown in Equation 2 and the key point KP_loss as shown in Equation 4. Multimodal loss regression determines the category, location and key point information of the detection box through multiple dimensions.

[0076] CIOU_Loss=1-IOU(t′,t) (Formula 1)

[0077] Where t′ represents GTbbox information;

[0078] CLs_Loss=BCE(c′ n ,c n (Equation 2)

[0079] BCE=―((c n *logc′ n )+(1―c n )*log(1―c′ n (Formula 3)

[0080] Where, c′ n c represents the probability that the nth sample is a positive sample. n This represents the label value of the nth sample;

[0081]

[0082] Where, v′ k This refers to key information.

[0083] The first target detection network also includes a KAPAO keypoint detection network; the KAPAO keypoint detection network integrates keypoint information into the anchor information generated by the improved YOLOV8 target detection network through a multi-task loss function, and uses the keypoint loss function to complete keypoint detection.

[0084] The smoking behavior detection method provided in this application, in a factory no-smoking scenario, utilizes a combination of YOLOV8 target detection network based on convolutional neural network, KAPAO key point recognition framework, and ST_GCN behavior recognition technology to detect smoking behavior of personnel in the factory area in real time. If smoking behavior is detected in the factory area, an alarm is triggered by sound and light, and the detection results are uploaded to the intelligent monitoring system.

[0085] The specific implementation scheme is as follows: image processing is performed on the video stream information collected by the factory cameras to obtain the video stream and convert it into RGB images. Then, YOLOv8 is used to detect cigarette butts in the personnel and personnel areas in the camera. Next, the KAPAO key point recognition scheme is used to detect key points of the target pedestrians. Finally, the ST_GCN algorithm is used to analyze the pedestrian behavior in continuous frames. Based on the above results, it is judged whether the target personnel are smoking. If smoking is detected, the sound and light alarms installed in the factory area are activated according to the results, and the detection results are uploaded to the monitoring system. This forms a complete smoking detection and recognition scheme for the factory area.

[0086] The method proposed in this application adopts a two-level detection scheme to detect smoking behavior in factory areas. First, it uses the improved YOLOV8 and KAPAO key point technology to detect pedestrian targets and key points. Then, it uses the ST_GCN algorithm to judge smoking behavior as the first level judgment. Next, it uses target detection to detect cigarettes and combines the position of the cigarettes and their relationship with the human body to make a second level judgment of smoking behavior.

[0087] This method uses a two-stage judgment to detect smoking behavior, which solves the problem of low accuracy when using a single target detection scheme. At the same time, this method uses a camera for indirect detection, which has a wide coverage and the ability to detect in real time 24 hours a day. While reducing labor costs, it can achieve the effect of no missed detection and no false detection, thus providing a foundation for safe production in factories.

[0088] By applying this method to different factories, it can be seen that it possesses real-time efficiency and detection accuracy that other existing solutions lack. Furthermore, this method is not highly restricted by the environment, and can be applied to both indoor and outdoor scenarios, demonstrating strong generalization ability. The algorithm's independent deployment scheme allows for integration into the system in both factories with existing cameras and those without, enabling detection. Therefore, this method is characterized by low deployment difficulty, low cost, high detection accuracy, strong real-time performance, and comprehensive factory safety monitoring solution, while also considering its generalization ability across different scenarios.

[0089] This application provides a method for detecting smoking behavior in a factory area by combining three technologies in deep learning image algorithms: target detection, key point skeleton recognition, and behavior recognition. The method consists of four steps: First, video streams are acquired through factory cameras and converted into single-frame RGB images, which are then used as the test images. Pedestrian target detection and key point detection are performed on the employees and pedestrians within the images. Behavior recognition and detection algorithms are then used to determine smoking behavior as the first-level judgment criterion. Next, target detection is used to detect cigarettes, and the cigarette's location and its relationship to the person are used as the second-level judgment criterion for smoking behavior. Figure 2 As shown, the scheme consists of the following steps:

[0090] Step 1: Establish a network framework for object detection and keypoint detection, train the object detection and keypoint detection models, and perform object detection and keypoint detection on pedestrians in each frame of the image.

[0091] Samples of pedestrians were collected from the factory area. Annotation tools were used to perform object detection and keypoint detection to label the images. Then, an improved YOLOv8 framework was adopted for object detection, combined with the KAPAO keypoint detection method to form the training framework for both object detection and keypoint detection. The YOLOv8 framework backbone uses the CSP construction idea, achieving lightweighting by replacing the C3 module with a C2f module. The fusion module uses the Feature Fusion Network (FPN) and PAN. Next, a combined anchor structure of keypoints and detection boxes was generated. Redundant anchors were filtered using NMS, VFLLosse was used as the classification loss, and CIOULoss was used as the location information loss. Finally, detection boxes containing keypoint information were obtained. The network was trained using a monitoring dataset to generate the model needed for prediction. Video image information parsed from cameras in the factory was used as input to the model to extract personnel detection boxes and generate keypoints in the factory area.

[0092] In step one of the above solution, images captured by cameras in the factory area are used to obtain pedestrian target bounding boxes and corresponding key point information of various body parts of employees through target detection and key point detection technologies. The specific implementation steps are as follows:

[0093] Step 101: First, a dataset for object detection and keypoint detection in a factory area needs to be established. The basic dataset format used in this method is the COCO dataset format. Images are extracted from video streams at different times and locations within the factory area, and then labeled. The bounding box (Bbox) information and keypoint information are labeled separately. There are a total of 14 keypoints, and their specific labels are as follows:

[0094] 1 = face, 2 = neck, 3 = left shoulder, 4 = right shoulder, 5 = left elbow, 6 = right elbow, 7 = left wrist, 8 = right wrist, 9 = left hip, 10 = right hip, 11 = left knee, 12 = right knee, 13 = left ankle, 14 = right ankle. Due to the confidentiality requirements of factory samples, the illustrations used in this method are all laboratory samples. The marking diagram is as follows: Figure 3 As shown, the points are keypoint information (KP), and the green boxes are bounding boxes (Bbox). A total of 2000 images were labeled as the training dataset, and the pedestrian part of the COCO dataset was added to enhance robustness.

[0095] Step 102: Establish an improved YOLOV8 object detection and KAPAO keypoint detection network. The KAPAO method integrates keypoint information into the anchor information generated by YOLOV8 through a multi-task loss function and uses the keypoint loss function to complete keypoint detection.

[0096] Step 103: Based on step 102, establish the YOLOv8 structure. The framework structure diagram is as follows: Figure 4As shown, the YOLOv8 framework Backbone adopts the CSP construction idea, and achieves lightweight processing by replacing the C3 module with the C2f module. The C2f module utilizes the ElAN idea to improve the design, changing the number of blocks from 3-6-9-3 to 3-6-6-3, and adopting more skip connections and more split operations to achieve lightweight processing. At the same time, the kenel of the first convolutional layer is changed from 6*6 to 3*3. After this operation, the optimal feature extraction module is achieved. The fusion module adopts the feature fusion network FPN and PAN, and then generates a combined anchor structure of keypoints and detection boxes. Then, NMS is used to filter redundant anchors, BCELosse is used as the classification loss, CIOU_Loss is used as the location information loss, and then the detection box CIOU_loss is used as Equation 1, the classification CLS_loss is used as BCE_loss as Equation 2, and the keypoint KP_loss is used as Equation 4. Multimodal loss regression determines the category, location, and keypoint information of the detection box from multiple dimensions. The training parameters are adjusted by jointly updating the three, and finally the training of the entire network is completed.

[0097] CIOU_Loss=1-IOU(t′,t) (Formula 1)

[0098] Where t′ represents GTbbox information.

[0099] CLs_Loss=BCE(c′ n ,c n (Equation 2)

[0100] BCE=―((c n *logc′ n )+(1―c n )*log(1―c′ n (Equation 3)

[0101] Where, c′ n c represents the probability that the nth sample is a positive sample. n This represents the label value of the nth sample.

[0102]

[0103] Where, v′ k This refers to key information.

[0104] Step 104: The object detection and keypoint detection datasets from Step 101 are used as the training set for the KAPAO embedded YOLOV8 detection framework network in Step 103. The network is trained on a H3C R4900 server using two Tesla-t4 deep learning training cards for a total of 500 epochs. Finally, the KAPAO-L-YOLOV8 model achieves an Ap value of 72.2 for keypoints, a bounding box accuracy of 98.6%, and a recall of 98.7%.

[0105] Step 105: Using the network from step 103 in conjunction with the model obtained in step 104, target detection and keypoint prediction are performed. First, the factory camera images need to be encoded and decoded. FFMPEG is used to decode the camera's video stream information into RGB images as a prediction buffer. Then, the buffer information is inspected for still frames, black levels, and sharpness. After confirming there are no such issues, forward prediction is performed using the trained model file. Based on the prediction results, pedestrian detection boxes and keypoint information for employees are obtained. Then, the corresponding keypoint information is concatenated to obtain skeleton information, preparing for subsequent behavior recognition. The detection results are as follows: Figure 5 As shown.

[0106] Step 2: Extract consecutive frames from the detection boxes in Step 1, match pedestrians with the same ID, and then use the ST_GCN algorithm for behavior recognition to perform behavior recognition as the first level of smoking detection judgment.

[0107] Using the pedestrian detection bounding boxes and pedestrian skeleton information obtained in step one, multiple consecutive frames of information are used, and pedestrians with the same ID are matched. The pedestrian detection bounding boxes and skeleton information are used as input to the ST_GCN behavior recognition framework for smoking behavior recognition. Based on the behavior recognition results, the first-level smoking detection is predicted.

[0108] In step two of the above scheme, the detection boxes from step one are extracted in consecutive frames, and pedestrians with the same ID are matched. Then, the ST_GCN algorithm is used for behavior recognition, which serves as the first-level smoking detection judgment. The specific implementation steps of step two are as follows:

[0109] Step 201: The images sampled in Step 1 are used as input for action recognition in groups of 30 consecutive frames. The Hungarian algorithm is used to perform similarity matching to match the results of Step 105. Pedestrians with the same identity are set as a unified ID as a label for the action behavior, thus completing the matching.

[0110] Step 202: After establishing the initial 30-frame video stream, when a human skeleton is detected, the nodes of pedestrians with the same ID in each frame are connected according to the characteristics of the human skeleton to form a spatial boundary E. s ={v ti v tj |(i,j)∈H}, where v ti v represents the position of the i-th node in frame t. tj This represents the position of the j-th node in frame t. Additionally, nodes with the same ID in two consecutive frames are connected as edges, forming a frame-subtracting graph structure E. f ={v ti v (t+1)i At this point, a graph relationship is established between pedestrians with the same ID in a single frame, as well as a graph relationship between pedestrians with the same ID and the same joint between frames.

[0111] Step 203, proceed to ST_GCN action behavior recognition, whose network structure is as follows: Figure 6 As shown, firstly, the result from step 302 is used as the input matrix and normalized, that is, the positional features of a joint in different frames are normalized. Then, 9 sets of ST_GCN units are used to alternately perform spatial transformation using GCN and TCN. GCN uses a graph convolutional network to learn the feature information of adjacent joints in space, and TCN uses the feature of joint changes in the time series. Then, the temporal and spatial features are superimposed using GRAPH. ATT, short for Attention module, is used before GCN and TCN. Since the importance of different trunk joints is different during the movement, for the smoking and raising of the hand in this method, the importance of the leg and the hand is different. Therefore, the ATT module weights different joints, so that each joint has its own corresponding weight parameters. Then, pooling is used to obtain a 256-dimensional feature vector for each sequence. Finally, the SoftMax function is used for classification to obtain the final classification result, as shown. Figure 7 As shown.

[0112] Step 204: When smoking behavior is detected in step 203, the smoking behavior is judged solely by the behavior and is used as the first level of alarm.

[0113] Step 3: Use the target detection network framework established in Step 1 to detect the location of the cigarette.

[0114] Using the improved YOlOV8 framework established in step one, the pedestrian detection category in the object detection model is changed to cigarettes. A cigarette detection dataset is established through network and on-site data collection, and the network established in step one is used to detect cigarettes.

[0115] In step three of the above scheme, the target detection network framework established in step one is used to detect the position of the cigarette. The specific implementation steps of step three are as follows:

[0116] Step 301: Establish a cigarette detection dataset. The dataset mainly consists of network images and factory area images. In order to adapt to the detection requirements of different environments inside and outside the factory area, the copy-paste sample generation method and GAN generation method are used to expand the dataset size. The final dataset is a cigarette detection dataset with 1200 images for training, 100 images for validation, and 100 images for test.

[0117] Step 302: Establish a cigarette detection network. Similar to the detection framework in Step 103, use an improved YOLOV8 network to remove the KAPAO module. Remove key point information during input and anchor generation, and also remove key point KP_loss in the loss function. Only retain the complete object detection process. Train the network using the same hardware configuration as in Step 303 to obtain the optimal model.

[0118] Step 303: When the Level 1 alarm is triggered in Step 204, the pedestrian detection bounding box is used as input. Step 302: Using the established cigarette detection network as the framework, the model trained in Step 303 is used for forward propagation to obtain the cigarette detection results within the pedestrian bounding box, as shown below. Figure 8 As shown.

[0119] Step 4: Based on the results of Step 3, the cigarette location information and the pedestrian skeleton information obtained in Step 1 are used to judge the smoking behavior through prior knowledge, which is used as the second level of smoking detection judgment, and this result is uploaded to the intelligent monitoring system.

[0120] The second level of judgment on smoking behavior is made by using the cigarette detection location information and its relationship with the human wrist in step three. When both levels of judgment detect smoking, the results are uploaded to the intelligent monitoring system. The system will then trigger an alarm by linking the on-site sound and light alarms, and at the same time, it will alert the manager in the monitoring room.

[0121] In step four of the above scheme, the cigarette location information is combined with the pedestrian skeleton information obtained in step one to judge the smoking behavior based on prior knowledge, which serves as the second level of smoking detection judgment. This result is then uploaded to the intelligent monitoring system. The specific implementation steps of step four are as follows:

[0122] Step 401: When a cigarette is detected in step 304, based on prior knowledge, smoking requires holding the cigarette while smoking. Therefore, it can be calculated whether the cigarette is within reach. The distance S between the center of gravity of the cigarette detection frame and the wrist, and the height H of the human body, are shown in the diagram. Figure 9As shown, the center of gravity of the cigarette is calculated from the cigarette detection frame result in step 304, and the wrist position is obtained from the wrist key point information in the detection information in step 105. The distance S between the two is calculated from the Euclidean distance between the two points. Figure 9 The midline segment represents the segment.

[0123] Step 402: Based on prior knowledge, it can be seen that when the ratio of S to H is less than 0.12, it can be determined that the cigarette is in the hand. When the human body is not obscured, the height H of the human body is estimated as the height of the detection frame. When the human body is obscured, the height value of the human body H can be estimated by the distance between the shoulder and elbow in the key point information in step 105. When the ratio of S to H is less than 0.12, it can be determined that the employee has smoked, which is the second level of alarm.

[0124] Step 403: To ensure the accuracy of smoking detection, when a second-level alarm is triggered, the alarm result is transmitted back to the intelligent monitoring system, and the audible and visual alarm is activated.

[0125] By combining the above four steps, this method can detect smoking behavior among factory employees. Since it employs a two-level alarm system, each factory can adjust the weights of the two alarm levels to suit different scenarios. This method primarily targets paper cigarettes; however, many different types of electronic cigarettes exist, and the detection range can be expanded by adding more electronic cigarette categories. This concludes the detailed implementation process of this method. It features real-time, non-contact, and multi-dimensional detection, representing a component of three-dimensional image detection technology in factory safety assurance applications. Furthermore, this method has strong generalization and compatibility; through training, it can adapt to detection in various environments, making it suitable for future applications in office areas, school areas, hospitals, and other similar locations.

[0126] See Figure 10 This application embodiment can also provide a smoking behavior detection device, such as... Figure 10 As shown, the device may include:

[0127] The pedestrian detection box acquisition unit 1001 is used to detect pedestrians in multiple consecutive frame target images using a trained first target detection network, and obtain multiple pedestrian detection boxes and key point information of various parts of the pedestrian's body; the target image is an image of the target region obtained by an image capturing device; the key point information of various parts of the pedestrian's body is used to determine the skeletal information of the pedestrian.

[0128] The first judgment result generation unit 1002 is used to match pedestrians with the same ID using multiple pedestrian detection boxes, and to generate the first judgment result by using the ST_GCN behavior recognition algorithm to perform action behavior recognition.

[0129] The cigarette location information determination unit 1003 is used to determine that the first judgment result indicates the presence of smoking behavior, and to use the trained second target detection network to detect and determine the location information of the cigarette contained in the pedestrian detection box.

[0130] The second judgment result generation unit 1004 is used to generate a second judgment result by using the cigarette location information and the pedestrian skeleton information to judge the smoking behavior through prior knowledge.

[0131] Alarm unit 1005 is used to determine that the second judgment result indicates smoking behavior, trigger the sound and light alarm in the target area to issue a warning, and upload the detection result to the intelligent monitoring system.

[0132] This application embodiment can also provide a smoking behavior detection device, the device including a processor and a memory:

[0133] The memory is used to store program code and transmit the program code to the processor;

[0134] The processor is used to execute the steps of the smoking behavior detection method described above according to the instructions in the program code.

[0135] like Figure 11 As shown in the figure, a smoking behavior detection device provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0136] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0137] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiments of the smoking behavior detection method.

[0138] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0139] The trained first object detection network is used to detect pedestrians in multiple consecutive frames of target images, obtaining multiple pedestrian detection boxes and key point information of various parts of the pedestrian's body; the target image is an image of the target region obtained by an image capturing device; the key point information of various parts of the pedestrian's body is used to determine the skeletal information of the pedestrian.

[0140] Multiple pedestrian detection boxes are used to match pedestrians with the same ID, and the ST_GCN behavior recognition algorithm is used to perform action behavior recognition to generate the first judgment result;

[0141] If the first judgment result indicates that smoking behavior exists, the trained second target detection network is used to detect the position of the cigarette within the pedestrian detection box to determine the cigarette position information.

[0142] Using the cigarette location information and the pedestrian's skeletal information, a second judgment result is generated by judging the smoking behavior based on prior knowledge.

[0143] If the second judgment result confirms the presence of smoking behavior, the audible and visual alarm in the target area will issue a warning, and the detection result will be uploaded to the intelligent monitoring system.

[0144] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0145] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0146] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0147] Of course, it should be noted that, Figure 11 The structure shown does not constitute a limitation on the smoking behavior detection device in the embodiments of this application. In practical applications, the smoking behavior detection device may include more than Figure 11 More or fewer components as shown, or combinations of certain components.

[0148] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the smoking behavior detection method described above.

[0149] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0150] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0151] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0152] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for detecting smoking behavior, characterized in that, include: The trained first object detection network is used to detect pedestrians in multiple consecutive frames of target images, and multiple pedestrian detection boxes and key point information of various parts of the pedestrian's body are obtained. The target image is an image of the target area acquired by an image capturing device; the key point information of each part of the pedestrian's body is used to determine the pedestrian's skeletal information; Multiple pedestrian detection boxes are used to match pedestrians with the same ID, and the ST_GCN behavior recognition algorithm is used to perform action behavior recognition to generate the first judgment result; The methods for generating the first judgment result include: Based on the characteristics of the human skeleton, the nodes of pedestrians with the same ID in each frame are connected to form a spatial boundary. ; in, This represents the position of the i-th node in frame t. This represents the position of the j-th node in the t-th frame; Connect nodes with the same ID in two consecutive frames to form an edge, thus creating a frame subtraction graph structure. Establish graph relationships between pedestrians with the same ID in single frames, and graph relationships between pedestrians with the same ID and the same joint between frames; The graph relationships of the pedestrians and the graph relationships of the joints are used as input matrices and normalized. Spatial transformation is performed by alternating GCN and TCN using 9 sets of ST_GCN units. The GCN is used to learn the feature information of adjacent joints in space using graph convolutional networks, and the TCN is used to learn the features of joint changes in time series. By utilizing the temporal and spatial features of the GRAPH stack, the ATT module is used to weight different joints, pooling is used to obtain a 256-dimensional feature vector for each sequence, and the SoftMax function is used for classification to obtain the first judgment result. If the first judgment result indicates that smoking behavior exists, the trained second target detection network is used to detect the position of the cigarette within the pedestrian detection box to determine the cigarette position information. Using the cigarette location information and the pedestrian's skeletal information, a second judgment result is generated based on prior knowledge of the smoking behavior; the method for generating the second judgment result includes: The skeletal information of the pedestrian is used to determine key wrist points and body height information H. The Euclidean distance S between the acquired wrist key point information and the cigarette position information is calculated, wherein the cigarette position information includes the centroid of the cigarette detection frame; A second judgment result is generated based on the proportional relationship between the Euclidean distance S and the human body height information H; If the second judgment result confirms the presence of smoking behavior, the audible and visual alarm in the target area will issue a warning, and the detection result will be uploaded to the intelligent monitoring system.

2. The smoking behavior detection method according to claim 1, characterized in that, Both the first and second object detection networks include an improved YOLOv8 object detection network; the improved YOLOv8 object detection network includes a C2f module and a fusion module; the C2f module has 3-6-6-3 blocks and the first convolutional layer has a 3*3 kenel; the fusion module includes a feature fusion network FPN and a PAN.

3. The smoking behavior detection method according to claim 2, characterized in that, The improved YOLOV8 object detection network uses BCEloss as the classification loss and the detection box CIOU_Loss as the location information loss, as shown in Equation 1. The classification CLS_loss uses BCE_loss as shown in Equation 2 and the key point KP_loss as shown in Equation 4. Multimodal loss regression determines the category, location and key point information of the detection box through multiple dimensions. (Equation 1) in, For GTbbox information; (Equation 2) (Equation 3) in, This represents the probability that the nth sample is a positive sample. This represents the label value of the nth sample; (Equation 4) in, This refers to key information.

4. The smoking behavior detection method according to claim 2, characterized in that, The first target detection network also includes a KAPAO keypoint detection network; the KAPAO keypoint detection network integrates keypoint information into the anchor information generated by the improved YOLOV8 target detection network through a multi-task loss function, and uses the keypoint loss function to complete keypoint detection.

5. The smoking behavior detection method according to claim 1, characterized in that, When the ratio of the Euclidean distance S to the human body height information H is 0.12, it is determined that the second judgment result indicates the presence of smoking behavior.

6. A smoking behavior detection device, characterized in that, The apparatus for performing the smoking behavior detection method according to any one of claims 1-5, the apparatus comprising: The pedestrian detection box acquisition unit is used to detect pedestrians in multiple consecutive frames of target images using a trained first target detection network, and to obtain multiple pedestrian detection boxes and key point information of various parts of the pedestrian's body; the target image is an image of the target region obtained by an image capturing device; the key point information of various parts of the pedestrian's body is used to determine the skeletal information of the pedestrian. The first judgment result generation unit is used to match pedestrians with the same ID using multiple pedestrian detection boxes, and to generate the first judgment result by using the ST_GCN behavior recognition algorithm to perform action behavior recognition. The cigarette location information determination unit is used to determine that the first judgment result indicates the presence of smoking behavior, and uses the trained second target detection network to detect and determine the location information of the cigarette contained in the pedestrian detection box; The second judgment result generation unit is used to generate a second judgment result by using the cigarette location information and the pedestrian skeleton information to judge the smoking behavior through prior knowledge. The alarm unit is used to determine that the second judgment result indicates smoking behavior, triggering the audible and visual alarm in the target area to issue a warning, and uploading the detection result to the intelligent monitoring system.

7. A smoking behavior detection device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the smoking behavior detection method according to any one of claims 1-5 according to the instructions in the program code.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the smoking behavior detection method according to any one of claims 1-5.

Images