A safety monitoring method for helmet wearing and smoking detection

Through the cascading design of the human body key point identification network and the target detection network, the upper body image of the human body is extracted for safety helmet and cigarette detection, combined with spatial information to eliminate false detection, solving the existing technology's high hardware requirements, slow detection speed and false detection rate Higher problem achieves higher detection accuracy and lower error detection rate.

CN114495258BActive Publication Date: 2024-11-01SHANGHAI UNIV
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202111511659.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-06
Publication Date
2024-11-01
Estimated Expiration
2041-12-06

AI Technical Summary

Technical Problem

The existing safety helmet wear and smoking detection methods have problems such as high hardware requirements, slow detection speed, and prone to frequent false detection, especially the poor detection effect for small targets.

Method used

The cascading method of the human body key point recognition network and the target detection network is adopted. The upper body image of the human body is extracted through the human body key point recognition network and input it into the target detection network for safety helmet wear and smoking detection, combined with spatial information to eliminate false detection, improve detection Accuracy.

Benefits of technology

It improves the recognition accuracy of hard helmet wear and smoking detection, reduces the error detection rate, and provides more possibilities for the expansion of system functions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114495258B_ABST
    Figure CN114495258B_ABST
Patent Text Reader

Abstract

The present invention discloses a safety monitoring method for helmet wearing and smoking detection, comprising the following steps: collecting human key point information and helmet wearing and cigarette labeling information, constructing and training a human key point recognition network and a target detection network; acquiring real-time video stream information, decoding the real-time video stream information to obtain a single-frame image; transmitting the obtained single-frame image to a human key point recognition network to detect human key points; through the detection of human key points, the upper body of the human body is cut out to obtain an upper body image of the human body; the upper body image of the human body is transmitted to a target detection network to perform helmet wearing and smoking detection, and obtain a detection result; performing spatial position analysis on the detection result to remove false alarm information; then performing single-frame image processing in the next frame of video stream, and repeating the above steps until all real-time video stream information is detected. The present invention improves detection accuracy, reduces false detection, and provides more possibilities for expansion.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of security monitoring, and in particular to a safety monitoring method for helmet wearing and smoking detection. Background Art

[0002] In recent years, with the popularization of surveillance cameras and the improvement of chip computing power, computer vision-based security monitoring methods have developed rapidly, and the traditional method of relying on manual monitoring of sites has gradually been replaced. For security monitoring of power machine rooms, the compliance of helmets is a personal protection for inspectors, and the detection of smoking behavior can effectively prevent fires.

[0003] In terms of structure, the existing detection methods are generally single-stage networks, that is, one-stage target detection networks for detection. Using this method to detect small targets such as cigarettes or helmets generally cannot achieve good results. This is because the image input to the target detection network needs to be compressed into a size of 416x416 first, and this process will cause the loss of small target information. The current solution is generally to increase the input size of the target detection network. This method not only has high hardware requirements for training, but also greatly reduces the inference speed, affecting the real-time performance of the model.

[0004] Single target detection networks will also produce many false detections when detecting small targets such as cigarettes or helmets, which will have a great impact on detection accuracy. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is that the existing detection of helmet wearing and smoking behavior is difficult, has high hardware requirements, slow detection speed, and is prone to many false detections. The present invention provides a safety monitoring method for helmet wearing and smoking detection, and provides a safety monitoring method for helmet wearing and smoking detection based on human key point recognition and target detection network, which improves detection accuracy, reduces false detection, and provides more possibilities for expansion.

[0006] To achieve the above object, the present invention provides a safety monitoring method for helmet wearing and smoking detection, comprising the following steps:

[0007] Collect human key point information and helmet wearing and cigarette labeling information, construct and train human key point recognition network and target detection network;

[0008] Acquire real-time video stream information, and decode the real-time video stream information to obtain a single frame image;

[0009] The obtained single frame image is passed into the human key point recognition network to detect the human key points;

[0010] By detecting key points of the human body, the upper body of the human body is cut out to obtain an image of the upper body of the human body;

[0011] The upper body image of the human body is transmitted to the target detection network for helmet wearing and smoking detection to obtain the detection results;

[0012] Perform spatial position analysis on the detection results to remove false positives;

[0013] Then, a single frame image processing is performed in the next frame of the video stream, and the above steps are repeated until all the real-time video stream information is detected.

[0014] Further, the key point information of the human body and the labeling information of the helmet and cigarette are collected, wherein the key point information of the human body includes one or more of the hands, shoulders, torso and head;

[0015] The information on helmet wearing and cigarette marking includes the marking of a person's head wearing a helmet correctly, a person's head not wearing a helmet, and cigarettes.

[0016] Furthermore, a human key point recognition network and a target detection network are constructed and trained, wherein the human key point recognition network uses OpenPose and the target detection network uses Yolov5.

[0017] Furthermore, the obtained single-frame image is passed into a human key point recognition network to detect the human key points, which specifically includes the following steps:

[0018] Preprocess single-frame images, including resizing, pixel value normalization, and color channel order adjustment;

[0019] The preprocessed single-frame image is sent to the human key point recognition network for detection to obtain the feature map output by the network;

[0020] Generate a confidence map of the key point position according to the feature map, perform non-maximum suppression, and obtain a set of candidate positions of the body key points;

[0021] According to the feature map, establishing associations between feature points to generate a vector field for each limb;

[0022] The PAF (Part Affinity Fields) algorithm is used to fuse the candidate key point positions and the limb vector field, and to associate the valid key points with individuals, so as to obtain a one-to-one correspondence between each key point and each person, and finally connect and assemble them into a skeleton of multiple people.

[0023] Furthermore, through the detection of human key points, the upper body of the human body is cut out to obtain the upper body image of the human body, which specifically includes calculating the position of the minimum rectangle containing the human body key point information, and then expanding the minimum rectangle to fully include the edge part of the human body, and finally cutting out the upper body image of the human body.

[0024] Furthermore, the upper body image of the human body is transmitted to the target detection network for helmet wearing and smoking detection to obtain the detection results, which specifically includes the following steps:

[0025] Preprocess the upper body images, including resizing, pixel value normalization, and color channel order adjustment;

[0026] The preprocessed image is divided into grids and output, and each grid is responsible for predicting several rectangular boxes and their corresponding confidence levels;

[0027] The prediction box is parsed from the output information. The confidence of all prediction boxes needs to be judged first, and a threshold is set. When the confidence of the prediction box is greater than the threshold, the prediction box is retained.

[0028] According to the predicted valid frame, the x, y coordinates and w, h length and width are analyzed to obtain the real frame.

[0029] Furthermore, the hand and head key points obtained by the human key point recognition network and the cigarette prediction frame obtained by the target detection network are combined to judge the smoking behavior, calculate the spatial confidence of the smoking behavior, and set a second threshold. When the spatial confidence is less than the second threshold, it is judged as smoking behavior, and when the spatial confidence is greater than the second threshold, it is judged as a false detection.

[0030] Furthermore, the object detection network will also output a head wearing a helmet or a head not wearing a helmet to determine whether a helmet is worn.

[0031] Furthermore, the spatial confidence of smoking behavior is calculated using the following formula:

[0032]

[0033] x s and s are the coordinates of the center point of the cigarette prediction box on the image, x k and k are the coordinates of the hand key points and the head key points on the image, w h is the width of the prediction box of the head wearing a helmet or the prediction box of the head not wearing a helmet output by the target detection network, and C is the spatial confidence of the smoking behavior.

[0034] Technical Effects

[0035] The present invention proposes a safety monitoring method for helmet wearing and smoking detection, which adopts a human body key point recognition network as the first stage to extract human body key point information, and then extracts a high-definition image of the upper body of the human body, and sends it to the target detection network for helmet wearing and cigarette object detection. At the same time, it also makes full use of spatial information to eliminate misdetected cigarette objects, thereby improving the recognition accuracy of helmet and smoking detection, and also provides more possibilities for the expansion of the system function.

[0036] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of a safety monitoring method for helmet wearing and smoking detection according to a preferred embodiment of the present invention;

[0038] Figure 2 It is a schematic diagram of a safety monitoring method for helmet wearing and smoking detection according to a preferred embodiment of the present invention, in which the upper body of a human body is cut out based on a human skeleton;

[0039] Figure 3 This is a rendering of a preferred embodiment of the present invention, a safety monitoring method for helmet wearing and smoking detection used in an actual power room scenario. DETAILED DESCRIPTION

[0040] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] In the following description, specific details such as specific internal procedures and techniques are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0042] like Figure 1 As shown, an embodiment of the present invention provides a safety monitoring method for helmet wearing and smoking detection, comprising the following steps:

[0043] Step 100, collect human key point information and helmet wearing and cigarette labeling information, construct and train a human key point recognition network and a target detection network; wherein, collect human key point information and helmet wearing and cigarette labeling information, wherein human key point information includes one or more of hands, shoulders, torso and head; helmet wearing and cigarette labeling information includes the labeling of a head wearing a helmet correctly, a head not wearing a helmet and cigarettes. The human key point recognition network uses OpenPose, and the target detection network uses Yolov5.

[0044] Step 200, obtaining real-time video stream information, and decoding it to obtain each frame of video image, specifically includes the following steps:

[0045] Unpack the video stream to get a complete frame of data;

[0046] Decode the data to obtain a frame of image in YUV format;

[0047] Convert a YUV image to a BGR image.

[0048] Step 300, passing the obtained single-frame image into a human key point recognition network to detect the human key points, specifically including preprocessing the single-frame image, including size scaling, pixel value normalization, and color channel order adjustment;

[0049] The preprocessed single-frame image is sent to the human key point recognition network for detection to obtain the feature map output by the network;

[0050] Generate a confidence map of the key point position according to the feature map, perform non-maximum suppression, and obtain a set of candidate positions of the body key points;

[0051] According to the feature map, establishing associations between feature points to generate a vector field for each limb;

[0052] The PAF (Part Affinity Fields) algorithm is used to fuse the candidate key point positions and the limb vector field, and to associate the valid key points with individuals, so as to obtain a one-to-one correspondence between each key point and each person, and finally connect and assemble them into a skeleton of multiple people.

[0053] The PAF algorithm is as follows:

[0054] Suppose the set of all discrete key points is Where J represents the type of body key points, N j represents the number of discrete points detected by the jth key point. Assume that the connection relationship between all key points is Z = {z j1mj2n |j1,j2∈{1...J},m∈{1...Nj1},n∈{1...N j2}}, where z j1mj2n ∈{0,1} when z j1mj2n =1 means the two points belong to the same part of the same person and can be connected. j1mj2n = 0 means that it cannot be connected. The problem of determining the connection relationship of key points is transformed into finding a suitable Z.

[0055] Since the correlation vector field implies global information, for each limb c, the correlation vector field E between the limbs is used mn , calculate the weighted sum E of it and Z c , the problem is transformed into finding a solution that makes E c The set of Z that is maximized. That is:

[0056]

[0057]

[0058]

[0059] The constraint on Z limits any discrete point to at most one limb. Applying the above formula to all limb regions yields the optimized formula:

[0060]

[0061] After obtaining Z that maximizes E through the above formula, the optimal matching pair of all key points can be determined, and all limbs belonging to the same target are connected to obtain a complete multi-person human posture graph.

[0062] Step 400, by detecting the key points of the human body, the upper body of the human body is cut out to obtain the upper body image of the human body; specifically, the position of the smallest rectangle containing the key point information of the human body is calculated, and then the smallest rectangle is expanded to fully contain the edge part of the human body, and finally the upper body image of the human body is cut out, such as Figure 2 As shown, the overall figure is the human skeleton obtained by the output of the human key point recognition network. First, the minimum enclosing rectangle containing the key points of the upper body is obtained, which is the dotted part in the figure. Further, this rectangle needs to be expanded to include the complete upper body image of the human body. Let the width of the original rectangle be w, then the expanded rectangle needs to expand outward by 0.3w on both ends of the rectangle in the x-axis direction. Let the height of the original rectangle be h, then the expanded rectangle needs to expand upward by 0.4h in the y-axis direction. The final expanded rectangular frame is the solid line frame in the figure.

[0063] Step 500, transmitting the upper body image of the human body to the target detection network for helmet wearing and smoking detection to obtain the detection results;

[0064] The specific steps include:

[0065] Preprocess the upper body images, including resizing, pixel value normalization, and color channel order adjustment;

[0066] The preprocessed image is divided into grids and output, and each grid is responsible for predicting several rectangular boxes and their corresponding confidence levels;

[0067] To parse the output information to obtain the prediction box, it is necessary to first judge the confidence of all prediction boxes and set a threshold. When the confidence of the prediction box is greater than the threshold, the prediction box is retained.

[0068] According to the predicted valid frame, the x, y coordinates and w, h length and width are analyzed to obtain the real frame.

[0069] Specifically, the image is preprocessed, including resizing, pixel value normalization, and color channel order adjustment;

[0070] Divide the image to be detected into SxS grids, each grid is responsible for predicting B rectangular boxes and their corresponding confidence levels;

[0071] Among them, S represents the number of grids to be divided, and B represents the number of borders that each grid is responsible for predicting;

[0072] The predicted valid box is obtained by parsing the output information of the network. First, the confidence of all predicted boxes needs to be judged. The predicted boxes with confidence greater than the threshold are retained. The output related to confidence in the model prediction is (objconf, classconf*n), where objconf is the possibility of an object in the anchor box, and classconf*n is the confidence of the n objects to be predicted. The final confidence of the anchor box is σ(objconf)*max(σ(classconf)). If this confidence is greater than the threshold, the predicted box is retained.

[0073] To get the predicted true box, it is also necessary to parse the box. The output related to box parsing in the model prediction is (tx, ty, tw, th), and the formula for parsing the true box is:

[0074] bx=σ(tx)+cx,

[0075] by=σ(ty)+cy,

[0076] bw=pw*e tw ,

[0077] bh=ph*e th ,

[0078] Where cx and cy represent the grid cell coordinates, pw and ph represent the width and height of the anchor box before prediction, and bx, by, bw and bh are the center coordinates of the predicted box and the width and height of the box.

[0079] Step 600, perform spatial position analysis on the detection results to remove false positive information; combine the key points of the hand and head obtained by the human key point recognition network and the cigarette prediction frame obtained by the target detection network to judge the smoking behavior, calculate the spatial confidence of the smoking behavior, set the second threshold T, and when the spatial confidence is less than the second threshold T, it is determined to be a smoking behavior, and when the spatial confidence is greater than the second threshold T, it is determined to be a false positive. The following formula is used to calculate the spatial confidence of the smoking behavior:

[0080]

[0081] x s and s are the coordinates of the center point of the cigarette prediction box on the image, x k and k are the coordinates of the hand key points and the head key points on the image, w h is the width of the prediction box of the head wearing a helmet or the head not wearing a helmet output by the target detection network, and C is the spatial confidence of the smoking behavior. The target detection network also outputs the head wearing a helmet or the head not wearing a helmet to determine whether the helmet is worn.

[0082] Step 700, then process a single frame image in the next frame of the video stream, and repeat the above steps until all the real-time video stream information is detected.

[0083] In order to verify the effect of this method, a test was carried out in an actual power room scenario, and the results are as follows: Figure 3 As shown, it can be seen that this method can detect the helmet wearing and smoking status of people with high accuracy.

[0084] A safety monitoring method for helmet wearing and smoking detection in an embodiment of the present invention adopts a design method of cascading a human key point recognition network and a target detection network, extracts a high-definition image of the upper body of the human body from the original image for helmet and cigarette target detection, and also uses the human key point information to remove misdetected cigarettes. It overcomes the disadvantages of the traditional single network monitoring method that it is difficult to detect small targets and has many misdetections, improves the detection accuracy of helmet wearing and smoking detection, and the method can also output human key point information, providing more possibilities for function expansion.

[0085] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A safety monitoring method for helmet wearing and smoking detection, characterized in that: The following steps are involved: Collect human key point information and helmet wearing and cigarette labeling information, construct and train human key point recognition network and target detection network; Acquire real-time video stream information, and decode the real-time video stream information to obtain a single frame image; The obtained single frame image is transmitted to the human body key point recognition network to detect the human body key points; By detecting key points of the human body, the upper body of the human body is cut out to obtain an image of the upper body of the human body; Transmitting the upper body image of the human body to the target detection network for helmet wearing and smoking detection to obtain detection results; Performing spatial position analysis on the detection results to remove false positive information; Then, a single frame image processing is performed in the next frame of the video stream, and the above steps are repeated until all the real-time video stream information is detected; The hand and head key points obtained by the human key point recognition network and the cigarette prediction frame obtained by the target detection network are combined to judge the smoking behavior, calculate the spatial confidence of the smoking behavior, set a second threshold, and when the spatial confidence is less than the second threshold, it is determined to be a smoking behavior, and when the spatial confidence is greater than the second threshold, it is determined to be a false detection; the spatial confidence of the smoking behavior is calculated using the following formula: x s and s are the coordinates of the center point of the cigarette prediction box on the image, x k and k are the coordinates of the hand key points and the head key points on the image, w h is the width of the prediction box of the head wearing a helmet or the prediction box of the head not wearing a helmet output by the target detection network, and C is the spatial confidence of the smoking behavior.

2. A safety monitoring method for helmet wearing and smoking detection as claimed in claim 1, characterized in that: Collecting key point information of the human body and the labeling information of helmet wearing and cigarettes, wherein the key point information of the human body includes one or more of the hands, shoulders, torso and head; The information on helmet wearing and cigarette marking includes the marking of a person's head wearing a helmet correctly, a person's head not wearing a helmet, and cigarettes.

3. A safety monitoring method for helmet wearing and smoking detection as claimed in claim 2, characterized in that: Construct and train a human key point recognition network and a target detection network, wherein the human key point recognition network uses OpenPose and the target detection network uses Yolov5.

4. A safety monitoring method for helmet wearing and smoking detection as claimed in claim 1, characterized in that: The obtained single frame image is transmitted to the human body key point recognition network to detect the human body key points, which specifically includes the following steps: Preprocessing the single frame image, including resizing, pixel value normalization, and color channel order adjustment; The preprocessed single-frame image is sent to the human key point recognition network for detection to obtain a feature map output by the network; Generate a confidence map of the key point position according to the feature map, perform non-maximum suppression, and obtain a set of candidate positions of the body key points; According to the feature map, establishing associations between feature points to generate a vector field for each limb; The PAF (Part Affinity Fields) algorithm is used to fuse the candidate key point positions and the limb vector field, and to associate the valid key points with individuals, so as to obtain a one-to-one correspondence between each key point and each person, and finally connect and assemble them into a skeleton of multiple people.

5. A safety monitoring method for helmet wearing and smoking detection as claimed in claim 4, characterized in that: By detecting the key points of the human body, the upper body of the human body is cut out to obtain the upper body image of the human body, which specifically includes calculating the position of the minimum rectangle containing the key point information of the human body, and then expanding the minimum rectangle to fully include the edge part of the human body, and finally cutting out the upper body image of the human body.

6. A safety monitoring method for helmet wearing and smoking detection as claimed in claim 1, characterized in that: The upper body image of the human body is transmitted to the target detection network for helmet wearing and smoking detection to obtain the detection results, which specifically includes the following steps: Preprocessing the upper body image, including resizing, normalizing pixel values, and adjusting the color channel order; The preprocessed image is divided into grids and output, and each grid is responsible for predicting several rectangular boxes and their corresponding confidence levels; The prediction box is parsed from the output information. The confidence of all prediction boxes needs to be judged first, and a threshold is set. When the confidence of the prediction box is greater than the threshold, the prediction box is retained. According to the predicted valid frame, the x, y coordinates and w, h length and width are analyzed to obtain the real frame.

7. A safety monitoring method for helmet wearing and smoking detection as claimed in claim 1, characterized in that: The target detection network also outputs a head wearing a helmet or a head not wearing a helmet to determine whether a helmet is worn.

Citation Information

Cited By

  • Smoking detection method and device based on low-power-consumption GPU equipment

    CN117253288A

  • A smoking detection method and device based on a low-power GPU device

    CN117253288B

  • Real-time video smoking detection method, device, equipment and medium

    CN119888559A