A substation on-site personnel smoking behavior detection method

Abstract

The present application belongs to the technical field of computer vision, and particularly relates to a substation on-site personnel smoking behavior detection method. The substation on-site personnel smoking behavior image is collected to form a data set; the people and cigarette butts in the image in the data set are subjected to target detection class box labeling, and the human body joint points in the image are subjected to key point labeling; the labeled data set is input into a preset smoking behavior detection model for training to obtain a trained smoking behavior detection model; the trained smoking behavior detection model is used to detect the cigarette butt detection box and human body joint point position information in the substation on-site, and whether the smoking behavior occurs is judged according to the relative position of the cigarette butt detection box and the human body joint point. In the smoking behavior detection model, the detection head is decoupled, two tasks of target detection and human body joint point detection can be completed at the same time, the false detection can be reduced, and the problems of slow detection speed and high resource occupancy rate caused by using two models in the traditional method are solved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method for detecting smoking behavior of personnel at a substation. Background Technology

[0002] With the rapid development of computer technology, artificial intelligence has become the core engine and key strategic technology of the current global technological revolution. Applying intelligent technology to industrial development is an ever-growing trend. Computer vision is a simulation of biological vision, and its main task is to enable computers to understand the content of images or videos like humans, essentially giving computers "eyes" and a "brain," allowing them to perceive their environment autonomously. Computer vision includes many branches, including object detection and human pose estimation. Currently, these technologies are widely used in industry, but due to technological limitations, the results are often less than ideal.

[0003] Object detection can identify various objects in a substation, such as people and cigarette butts, while human pose estimation can obtain the coordinates of human joints. Combining these two technologies can effectively identify smoking behavior by personnel in the substation. However, running both models simultaneously leads to slow detection speed and high resource consumption. Therefore, a multi-task learning model is designed to perform both tasks simultaneously, improving detection speed while saving hardware costs. Summary of the Invention

[0004] To overcome the problems in the prior art, this invention proposes a method for detecting smoking behavior of personnel at substation sites.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0006] This invention provides a method for detecting smoking behavior of personnel at a substation site, comprising the following steps:

[0007] Step S1: Collect images of smoking behavior of personnel at the substation site to form a dataset; perform target detection category box annotation on people and cigarette butts in the images in the dataset, and perform key point annotation on human body joints in the images;

[0008] Step S2: Input the labeled dataset into the preset smoking behavior detection model for training to obtain the trained smoking behavior detection model;

[0009] The preset smoking behavior detection model is built based on the YOLO-Pose model, which includes a backbone network, a neck network, and a decoupled detection head connected in sequence.

[0010] The neck network is a path aggregation network PANet. By setting different fusion factors, the feature ratio from different layers is adjusted. The multi-scale fusion factor is added during the process of PANet transferring features from shallow to deep layers.

[0011] Step S3: Use the trained smoking behavior detection model to detect the cigarette butt detection frame and human joint point position information in the substation site, and determine whether smoking behavior has occurred based on the relative position of the cigarette butt detection frame and human joint point.

[0012] Furthermore, in step S2, the multi-scale fusion factor is added during the process of PANet transferring features from shallow to deep layers. The specific steps are as follows:

[0013] Step S201: The backbone network performs multi-scale feature extraction on the image, passing it from the shallow layer to the deep layer, and obtains four different scales of first feature maps {F3, F4, F5, F6}.

[0014] Step S202: The feature maps are progressively enlarged through upsampling and then added to the features extracted by the backbone network through lateral connections to obtain second feature maps {C3, C4, C5, C6} at four different scales. f inner For horizontal connection, f downsample For a double downsampling convolution operation, C i For the second feature map of layer i, C i+1 For the second feature map of layer i+1, F i This is the first feature map of layer i;

[0015] Step S203: Transfer the second feature map C i Perform a double upsampling operation and add a fusion factor. By controlling the transfer of features from shallow to deep layers, we obtain the fused third feature maps {P3, P4, P5, P6} at four scales. f layer For convolution operations, f inner For horizontal connection, f downsample For a double downsampling convolution operation, P i P is the third feature map of layer i; i-1 This is the third feature map of layer i-1. The fusion factor is the feature transfer factor from layer (i-1) to layer (i).

[0016] Furthermore, the formula for calculating the fusion factor α is as follows:

[0017]

[0018] in, This represents the number of ground truths at the i-th scale. This represents the number of ground truths at the (i-1)th level scale.

[0019] Furthermore, in step S2, the decoupled detection head includes three parallel branches. The first branch obtains category information through convolution operations; the second branch obtains regression and confidence information; the third branch first enhances the features of the joints through 6 deep convolution units, then obtains the correlation between the joints through mixed multi-head attention, and finally outputs the joint coordinate information through 1×1 convolution.

[0020] Furthermore, in step S2, five loss functions are used to train the smoking behavior detection model; the target detection loss function includes classification loss, localization loss, and target confidence loss; the key point detection loss function includes key point loss and key point confidence loss.

[0021] Furthermore, step S3 specifically includes:

[0022] Obtain the cigarette butt detection bounding box and the coordinates of the wrist and nose joints output by the trained smoking behavior detection model;

[0023] Expand the cigarette butt detection frame outwards by a preset distance;

[0024] Whether smoking has occurred is determined by the relative distance between the cigarette butt and the wrist or nose. If the wrist or nose is within the cigarette butt detection frame, smoking is determined to have occurred; otherwise, the person is determined not to have smoked.

[0025] Compared with the prior art, the present invention has the following technical effects:

[0026] To detect smoking behavior among personnel at substation sites, this invention improves the YOLO-Pose model by proposing a multi-task learning method that can simultaneously perform object detection and keypoint detection. On one hand, this invention incorporates a multi-scale fusion factor into the neck network to balance the number of samples between large and small targets, improving the accuracy of cigarette butt detection. On the other hand, the traditional detection head is decoupled into three branches, with information from each branch being independent, reducing coupling and avoiding mutual interference between the two tasks. This invention combines object detection and keypoint detection techniques, significantly reducing false positives. Furthermore, by compressing the two models into a single model, this not only improves inference speed and saves time but also reduces resource consumption and hardware costs. Attached Figure Description

[0027] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of the method of the present invention;

[0029] Figure 2 It is a multi-task learning dataset format;

[0030] Figure 3 This is a schematic diagram of the transmission process of the fusion factor;

[0031] Figure 4 This is the network structure diagram of YOLO-Pose;

[0032] Figure 5 This is a schematic diagram of the decoupled detection head structure;

[0033] Figure 6 This is a flowchart illustrating the logic for judging smoking behavior.

[0034] Figure 7 This is a diagram showing the results of on-site testing at a substation. Detailed Implementation

[0035] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the specific implementation methods, structures, features, and effects of the technical solutions proposed according to the present invention are described in detail below with reference to the accompanying drawings and preferred embodiments. Specific features, structures, or characteristics in one or more embodiments may be combined in any suitable form. Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0036] In one embodiment of the present invention, reference is made to... Figure 1 A method for detecting smoking behavior of personnel at substation sites is provided, comprising the following steps:

[0037] Step S1: Collect images of smoking behavior of personnel at the substation site to form a dataset; perform target detection category box annotation on people and cigarette butts in the images in the dataset, and perform key point annotation on human body joints in the images;

[0038] Step S2: Input the labeled dataset into the preset smoking behavior detection model for training to obtain the trained smoking behavior detection model;

[0039] Step S3: Use the trained smoking behavior detection model to detect the cigarette butt detection frame and human joint point position information in the substation site, and determine whether smoking behavior has occurred based on the relative position of the cigarette butt detection frame and human joint point.

[0040] The following is a detailed explanation of each of the above steps:

[0041] Step S1: Collect images of smoking behavior of personnel at the substation site to form a dataset; label the people and cigarette butts in the images in the dataset with category boxes, and label the human body joints in the images with key points.

[0042] In some embodiments, step S1 may include:

[0043] There are two methods for collecting images of smoking behavior by personnel at substation sites: one is direct shooting with a DSLR camera, and the other is shooting with a camera at the substation site. To make the smoking behavior detection model more robust, it is necessary to capture smoking postures at three distances (far, medium, and near) and different angles, while also considering lighting conditions, and collecting data at different time periods.

[0044] The collected images of smoking behavior by personnel at the substation site were preprocessed to form a dataset; the preprocessing included operations such as cropping and deduplication. The dataset may contain 8,000 images.

[0045] Object detection bounding boxes were used to annotate the people and cigarette butts in the images of the dataset, and keypoints were annotated for the human body joints in the images. Labelme image annotation tool was used to annotate 17 human body joints, including: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0046] The labeled dataset was divided into training and validation sets in a 9:1 ratio. Then, the JSON files generated by the Labelme annotation tool were converted to YOLO dataset file format using a program. The YOLO dataset file format is a text file (.txt) and includes class numbers, object detection bounding boxes, and human joint information.

[0047] Specifically, refer to Figure 2 In the txt text type, the vector length of each image is 56. The first dimension is the class number, which is the category number of the target object. The second to fifth dimensions are the target detection category box information; these four dimensions typically represent the coordinates of the four corners of the target detection category box, or the coordinates of the center point, width, and height, etc. The sixth to fifty-sixth dimensions are the human keypoint information; these dimensions represent the coordinate information of the human keypoints. For human keypoint detection tasks, each keypoint typically has three-dimensional information; therefore, if there are N keypoints, there will be 3N dimensions of keypoint information. In this embodiment, keypoint annotation is performed on 17 human keypoints, using 51 dimensions to represent all human keypoint information.

[0048] Step S2: Input the labeled dataset into the preset smoking behavior detection model for training to obtain the trained smoking behavior detection model.

[0049] The smoking behavior detection model is built based on the YOLO-Pose model, referring to... Figure 4 The YOLO-Pose model comprises a backbone network, a neck network, and a decoupled detection head, connected sequentially. The backbone network, which can be Darknet, is used for multi-scale feature extraction from the image, obtaining feature maps at multiple scales. In shallower network layers, the feature maps have higher resolution and contain more detailed information; while in deeper network layers, the feature maps have lower resolution but contain richer semantic information. The neck network is a path aggregation network (PANet), which adjusts the feature ratio from different network layers by setting different fusion factors. The multi-scale fusion factor is added during the transfer of features from shallow to deep layers in PANet. The decoupled detection head consists of three parallel branches. The first branch obtains object detection category information through convolution operations; the second branch obtains regression and confidence information; the third branch first strengthens the features of the joints through six DW (Depthwise Convolution) convolutional units, then obtains the correlation between joints through multi-head attention, and finally outputs the human joint coordinate information through a 1×1 convolution.

[0050] In this embodiment, Figure 3 This diagram illustrates the transfer process of the multi-scale fusion factor in PANet. The multi-scale fusion factor is added during the transfer of features from shallow to deep layers in PANet. The specific steps are as follows:

[0051] Step S201: The backbone network performs multi-scale feature extraction on the image, passing it from the shallow layer to the deep layer, and obtains four different scales of first feature maps {F3, F4, F5, F6}.

[0052] Step S202: The feature maps are progressively enlarged through upsampling and then added to the features extracted by the backbone network through lateral connections to obtain second feature maps {C3, C4, C5, C6} at four different scales. f inner For horizontal connection, f downsample For a double downsampling convolution operation, C i For the second feature map of layer i, C i+1 For the second feature map of layer i+1, F i This is the first feature map of layer i;

[0053] Step S203: Transfer the second feature map C i Perform a double upsampling operation and add a fusion factor. By controlling the transfer of features from shallow to deep layers, we obtain the fused third feature maps {P3, P4, P5, P6} at four scales. f layer For convolution operations, f inner For horizontal connection, f downsample For a double downsampling convolution operation, P i P is the third feature map of layer i; i-1 This is the third feature map of layer i-1. The fusion factor is the feature transfer factor from layer (i-1) to layer (i).

[0054] In this embodiment, the multi-scale fusion factor α is calculated as follows:

[0055] Step S2031: Select the positive anchor frame;

[0056] For all images: iterate through every image in the dataset;

[0057] Select the anchor box with the largest relationship to the ground truth as the positive anchor box: For each image, calculate the Intersection over Union (IoU) between all anchor boxes and the ground truth. Select the anchor box with the largest IoU value as the positive anchor box, that is, the anchor box that best matches a certain ground truth.

[0058] Step S2032: Calculate the number of ground truths at each scale;

[0059] For each scale of positive anchor frames: After the positive anchor frames are determined, they are grouped according to the scale to which they belong.

[0060] Compare with predefined anchor frames at the current scale: For each scale, there is typically a set of predefined anchor frame sizes and scales. Compare these predefined anchor frames with the positive anchor frame to determine which predefined anchor frames are closest to the positive anchor frame.

[0061] Calculate the number of ground truths at each scale: Count the number of ground truths that match the positive anchor box at each scale. This can be achieved by calculating the number of ground truths covered by the positive anchor box at each scale.

[0062] Step S2033: Obtain the final statistical results and calculate the fusion factor;

[0063] Repeat steps S2031 and S2032: Repeat the above process for all images to obtain statistical results of the number of ground truths at each scale;

[0064] The formula for calculating the fusion factor α is:

[0065]

[0066] in, This represents the number of ground truths at the i-th scale. This represents the number of ground truths at the (i-1)th level scale. This indicates that there is ground truth within the anchor frame; otherwise, it equals 0.

[0067] In this embodiment, refer to Figure 5 The outputs of PANet's P3, P4, P5, and P6 are used as inputs to the decoupled detection head at different scales. After a 1×1 convolution, the number of channels is unified to 256. Then, three parallel branches are derived. The first branch obtains category information through convolution; the second branch obtains regression and confidence information; the third branch first enhances the features of the joints through 6 DW (depthwise convolution) units, then obtains the correlation between joints through multi-head attention, and finally outputs the joint coordinate information through a 1×1 convolution. After passing through the decoupled detection head, the vectors are finally merged into a one-dimensional vector P for output.

[0068]

[0069] In the above formula, C x C y Here are the coordinates of the center point of the detection box, W and H are the width and height of the detection box, and box...conf The confidence level of the detection box; class conf For category confidence, For the first category confidence level, K represents the confidence level for category c. x K y represents the x and y coordinates of the joint, and Kconf represents the confidence level of the joint. Here are the coordinates of the first joint. The confidence level for the first key point. Here are the coordinates of the joint. Let be the confidence level of the nth joint.

[0070] In this embodiment, five loss functions are used for training the smoking behavior detection model. There are three target detection loss functions: classification loss, localization loss, and target confidence loss; and two keypoint detection loss functions: keypoint loss and keypoint confidence loss.

[0071] The classification loss function is changed from the cross-entropy loss function to Focal loss (a local loss function):

[0072] L cls (C p C ge = Focal Loss(C p C gt );

[0073] In the above formula, L cls Represents classification loss, Focal loss is a local loss function, C p To predict class confidence, C gt The true class confidence score; this formula represents the confidence score C for the predicted class. p confidence level C of the true category gt Focalloss is calculated between them.

[0074] Localization loss is implemented based on CIoU:

[0075] L box (box p box gt ) = 1 - CIoU(box) p box gt );

[0076] In the above formula, L box Indicates positioning loss, box p This represents the center coordinates of the predicted bounding box. gtThe coordinates of the center of the ground truth bounding box are represented by the CIoU (Complete Intersection over Union) loss function, which is a loss function used in object detection to measure the similarity between the predicted bounding box and the ground truth bounding box.

[0077] The target confidence loss uses the cross-entropy loss function:

[0078] L obj (p o p IoU )=BCE(p o p IoU );

[0079] In the above formula, L obj p represents the confidence loss. o p represents the target confidence score in the prediction box. IOU This represents the IoU value between the predicted bounding box and the corresponding ground truth bounding box; BCE represents the cross-entropy loss function.

[0080] Joint loss was evaluated using the OKS (OKS) metric.

[0081]

[0082] Among them, L kpts Indicates the joint loss, d n k represents the Eulerian distance between the predicted and actual values ​​of the nth keypoint. n Let N represent the penalty factor for the nth keypoint, and s represent the target scale. If the ground truth bounding box matches the kth anchor box, then the loss of the anchor box at position (i, j) with scale s will be effective; kpts v represents the number of key points to be predicted. n A flag indicating whether the joint is visible, δ indicating whether the joint is valid, v n =0 indicates that the joint is not visible and is not labeled, v n =1 indicates that the key point is occluded but has been labeled, v n =2 indicates that the joints are not occluded and are labeled. This parameter ensures that only labeled joints are calculated.

[0083] The key confidence loss uses cross-entropy loss for each key:

[0084]

[0085] Among them, L kpts_conf BCE represents the cross-entropy loss function, where BCE represents the cross-entropy loss function. This represents the confidence level of the prediction for the nth key point.

[0086] Step S3: Use the trained smoking behavior detection model to detect the cigarette butt detection frame and human joint point position information in the substation site, and determine whether smoking behavior has occurred based on the relative position of the cigarette butt detection frame and human joint point.

[0087] Figure 6 A flowchart of the decision logic is provided. Figure 7 The test results at the substation site are shown in the diagram. (Refer to...) Figure 6 Step S3 specifically includes:

[0088] Obtain the cigarette butt detection bounding box and the coordinates of the wrist and nose joints output by the trained smoking behavior detection model;

[0089] Expand the cigarette butt detection frame outwards by a preset distance;

[0090] Whether smoking has occurred is determined by the relative distance between the cigarette butt and the wrist or nose. If the wrist or nose is within the cigarette butt detection frame, smoking is determined to have occurred; otherwise, the person is determined not to have smoked.

[0091] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting smoking behavior of personnel at a substation site, characterized in that, Includes the following steps: Step S1: Collect images of smoking behavior of personnel at the substation site to form a dataset; perform target detection category box annotation on people and cigarette butts in the images in the dataset, and perform key point annotation on human joints in the images; Step S2: Input the labeled dataset into the preset smoking behavior detection model for training to obtain the trained smoking behavior detection model; The preset smoking behavior detection model is built based on the YOLO-Pose model, which includes a backbone network, a neck network, and a decoupled detection head connected in sequence. The neck network is a path aggregation network PANet. By setting different fusion factors, the feature ratio from different layers is adjusted. The multi-scale fusion factor is added during the process of PANet transferring features from shallow to deep layers. Step S3: Use the trained smoking behavior detection model to detect the cigarette butt detection frame and human joint point position information in the substation site, and determine whether smoking behavior has occurred based on the relative position of the cigarette butt detection frame and human joint point. In step S2, the multi-scale fusion factor is added during the process of PANet transferring features from shallow to deep layers. The specific steps are as follows: Step S201: The backbone network performs multi-scale feature extraction on the image, passing it from the shallow layer to the deep layer, and obtains four different scales of first feature maps {F3, F4, F5, F6}. Step S202: The feature maps are progressively enlarged through upsampling and then added to the features extracted by the backbone network through lateral connections to obtain second feature maps {C3, C4, C5, C6} at four different scales. , f inner For horizontal connection, f downsample For a double downsampling convolution operation, for i Second feature map of layer, for i+ Layer 1, second feature map for i First feature map of layer; Step S203: Transfer the second feature map C i Perform a double upsampling operation and add a fusion factor. By controlling the transfer of features from shallow to deep layers, the third feature map {P3, P4, P5, P6} at four scales is obtained after fusion. , f layer For convolution operations, f inner For horizontal connection, f downsample For a double downsampling convolution operation, for i Third feature map of layer; for i- Layer 1, third feature map For the first i -1 floor towards the first i The fusion factor for layer feature transfer.

2. The method for detecting smoking behavior of personnel at a substation site according to claim 1, characterized in that, The fusion factor The calculation formula is: ； in, Indicates the first i The number of ground truths at the layer scale Indicates the first i- Number of ground truths at a 1-layer scale.

3. The method for detecting smoking behavior of personnel at a substation site according to claim 2, characterized in that, In step S2, the decoupled detection head includes three parallel branches. The first branch obtains target detection category information through convolution operation; the second branch obtains regression and confidence information; the third branch first enhances the features of the joints through 6 deep convolution units, then obtains the correlation between the joints through mixed multi-head attention, and finally outputs the joint coordinate information through 1×1 convolution.

4. The method for detecting smoking behavior of personnel at a substation site according to claim 1, characterized in that, In step S2, five loss functions are used to train the smoking behavior detection model; the target detection loss function includes classification loss, localization loss, and target confidence loss; the key point detection loss function includes key point loss and key point confidence loss.

5. The method for detecting smoking behavior of personnel at a substation site according to claim 1, characterized in that, Step S3 specifically includes: Obtain the cigarette butt detection bounding box and the coordinates of the wrist and nose joints output by the trained smoking behavior detection model; Expand the cigarette butt detection frame outwards by a preset distance; Whether smoking has occurred is determined by the relative distance between the cigarette butt and the wrist or nose. If the wrist or nose is within the cigarette butt detection frame, smoking is determined to have occurred; otherwise, the person is determined not to have smoked.

Images