Intelligent monitoring and identification method and system for smoking behavior in industrial environment based on multi-algorithm fusion

Abstract

The present invention discloses a method and system for intelligent monitoring and identification of smoking behavior in an industrial environment with multi-algorithm fusion, which relates to the fields of industrial safety detection, computer vision and image recognition technology, including real-time acquisition of monitoring video frames; performing human body detection to determine whether there is a human body in the video frame; intercepting human body images in video frames with human bodies to perform human body joint point detection to determine whether it is a smoking action; intercepting head images of human body images judged to be smoking actions, and performing cigarette butt detection on the head area to determine whether it is smoking. The present invention does not need to collect materials, mark, and train separately according to actual scenarios, has a certain degree of versatility, and can be used after the model is deployed. By analyzing the characteristics of smoking behavior, the recognition steps are split, and the error conditions can be judged in the first time, thereby improving the recognition efficiency. Different targeted professional algorithms are used in the corresponding steps, and certain algorithm tuning is added according to the smoking behavior to improve the accuracy of recognition.

Description

Technical Field

[0001] The present invention relates to the fields of industrial safety detection, computer vision and image recognition technology, and in particular to a method and system for intelligently monitoring and identifying smoking behavior in an industrial environment by integrating multiple algorithms. Background Art

[0002] Smoking is a huge safety hazard for industrial production. With the rapid development of science and technology, human supervision can no longer meet the daily safety management needs of enterprises. With the continuous development of AI vision, surveillance video services with integrated smoking recognition models have begun to be used to detect smoking behaviors in industrial production processes. The videos collected by the cameras are analyzed in real time to detect smoking behaviors.

[0003] Traditional smoking recognition models are usually trained based on a single or fewer algorithms. In order to improve the efficiency and accuracy of recognition, higher requirements are placed on the materials used to train the models, which are mainly reflected in the following two aspects:

[0004] Winning by quantity, by marking and training a large amount of real materials, the efficiency and accuracy of recognition can be improved. However, a large amount of materials means the arduous work of material collection and labeling, which also affects the size and actual performance of the model to a certain extent.

[0005] Through technical means, the collected facial images are synthesized into smoking images, thereby marking the key labels of cigarettes and then training the recognition model. This method requires the collection of facial images in advance and has certain requirements for the recognition scene. Recognition must be performed according to the direction of the collected faces.

[0006] Therefore, it has certain practical significance to significantly improve recognition efficiency and accuracy without making special requirements on the materials. Summary of the invention

[0007] In view of the above-mentioned problems, the present invention is proposed.

[0008] Therefore, the problem to be solved by the present invention is: how to solve the problem that the existing smoking recognition model is usually based on a single or fewer algorithms to train the recognition model.

[0009] To solve the above technical problems, the present invention provides the following technical solutions: a multi-algorithm fusion intelligent monitoring and identification method for smoking behavior in an industrial environment, comprising: real-time acquisition of monitoring video frames; performing human body detection to determine whether there is a human body in the video frames; capturing human body images in video frames where there is a human body to perform human joint point detection to determine whether it is a smoking action; capturing head images of human body images that are determined to be smoking actions, and performing cigarette butt detection on the head area to determine whether there is smoking.

[0010] As a preferred solution of the multi-algorithm fusion industrial environment smoking behavior intelligent monitoring and identification method of the present invention, the human body detection includes collecting human body picture materials, marking the materials, using YOLOv5 to train the marked material set, and outputting a human body detection model. The training loss function is expressed as:

[0011] ,

[0012] in, is the loss function of the human body detection model, is the weight of coordinate loss, is the confidence loss weight when there is no human body, is the size of the grid, The number of bounding boxes predicted for each grid cell, For grid cells The Whether the bounding box contains a human body, For grid cells Does it contain human body? For grid cells It does not include the human body, , are the center coordinates of the bounding box, is the width of the bounding box, is the height of the bounding box, is the confidence level, is the class probability, , are the center coordinates of the predicted bounding box, is the width of the predicted bounding box, is the height of the predicted bounding box, is the confidence level of the prediction, is the predicted category probability; the trained human body detection model is used to judge the video frame to determine whether there is a human body in the video frame. If there is no human body, it is determined that the smoking behavior is not recognized. If there is a human body, the image of the human body area is captured.

[0013] As a preferred solution of the multi-algorithm fusion method for intelligent monitoring and identification of smoking behavior in industrial environments described in the present invention, the human joint point detection includes collecting an image data set containing human joint points, marking the data set, training the marked data set using a human posture estimation network HRNet, extracting multi-scale high-resolution feature maps, fusing the feature maps, converting the fused feature maps into joint point heat maps through a convolution layer, generating a human joint point detection model, and training the human joint point detection model using a loss function, which is expressed as:

[0014] ,

[0015] in, is the loss function of the human joint point detection model, is the batch size, is the number of joint points, is the predicted heat map, is a real heat map; the trained human joint point detection model is used to extract the human joint points from the intercepted human region image. The coordinates of each joint point are represented by the position of the maximum value in the heat map of each joint point, which is expressed as:

[0016] ,

[0017] in, are the coordinates of the human joint points, The coordinate position of the joint point in the heat map The heat map value of Returns the parameter that maximizes the function value.

[0018] As a preferred solution of the multi-algorithm fusion industrial environment smoking behavior intelligent monitoring and identification method of the present invention, the judgment of whether it is a smoking action includes calculating the extracted human joint points to determine whether the human body image shows a smoking action; obtaining the two-dimensional coordinates of the joint points , where the wrist coordinates are , the shoulder coordinates are , the elbow coordinates are , the nose coordinates are , the left eye coordinates are , the right eye coordinates are ; The calculation of the extracted human joint points includes calculating the arm bending angle and calculating the intersection of the arm extension line and the midpoint of the eyes and the nose extension line; the calculation of the arm bending angle includes calculating the angle of the wrist, elbow, shoulder and calculating the angle between the elbow, shoulder and the vertical; the calculation of the wrist, elbow, shoulder angle is expressed as:

[0019] ,

[0020] in, The angles of the wrist, elbow, and shoulder. The logic for calculating the angle formed by three points is expressed as follows:

[0021] ,

[0022] in, It is the angle between the elbow and shoulder and vertical.

[0023] As a preferred solution of the multi-algorithm fusion industrial environment smoking behavior intelligent monitoring and identification method of the present invention, wherein: the calculation of the intersection of the arm extension line and the midpoint of the eyes and the extension line of the nose includes obtaining the coordinates of the midpoint of the eyes, which is expressed as:

[0024] ,

[0025] in, is the coordinate of the midpoint of both eyes; determine whether the extension line of the arm and the extension line of the midpoint of both eyes and the nose intersect. If so, calculate the intersection point, expressed as:

[0026] ,

[0027] in, If it is equal to 1, then it is parallel. If it is equal to 0, then they intersect. are the intersection coordinates, are the coordinates of the midpoints of both eyes, To calculate whether the lines intersect, if they do, calculate the logic of the intersection point; if they are not parallel, that is, they intersect, then calculate the distance from the wrist to the elbow as:

[0028] ,

[0029] in, is the distance from wrist to elbow, The logic for calculating the distance between two points; the distance from the intersection to the nose is expressed as:

[0030] ,

[0031] in, is the distance from the intersection point to the nose, is the intersection coordinate; the determination of whether it is a smoking action is expressed as:

[0032] ,

[0033] in, For smoking action, is the preset angle threshold, for Equal to 0 intersection, is the preset space size threshold, is logical AND; the arm bending angle is less than , and the extended line of the arm intersects the midpoint of the eyes and the extended line of the nose, and the distance If the action is not smoking, it is considered as not smoking, and the smoking behavior is not recognized.

[0034] As a preferred solution of the multi-algorithm fusion industrial environment smoking behavior intelligent monitoring and identification method of the present invention, the head image capture includes identifying the human head area using a trained head detection model after judging that there is a smoking action. or , then discard the current video frame and capture the head image of the video frame that is not discarded, which is expressed as:

[0035] ,

[0036] ,

[0037] ,

[0038] ,

[0039] ,

[0040] in, The width and height of the recognized head area. is the preset size filtering threshold, is the standard value of size, is the multiple threshold for enlarging the head area, is the coordinate of the center point of the head area, is the video frame width and height, and They are the coordinates of the upper left corner and lower right corner of the head area actually captured, is the boundary threshold; when the width and height of the recognized head area are less than the preset size filtering threshold, the current video frame is discarded, indicating that the detection standard of the cigarette butt recognition model is not met; if it is greater than or equal to the preset size filtering threshold, the larger value of the width and height is multiplied by the preset multiple threshold, indicating that it is enlarged to the specified size and compared with the preset size standard value, the larger value is taken as the boundary threshold for actually intercepting the head area, and then the coordinates of the actual intercepted head area are calculated according to the boundary threshold.

[0041] As a preferred solution of the multi-algorithm fusion industrial environment smoking behavior intelligent monitoring and identification method of the present invention, the cigarette butt detection in the head area includes collecting cigarette butt image materials, marking the materials, using the lightweight target detection network PicoDet to train the marked material set, and outputting a cigarette butt detection model, which is expressed as:

[0042] ,

[0043] ,

[0044] ,

[0045] ,

[0046] ,

[0047] in, The offset of the bounding box predicted for each anchor point on the feature map extracted by the convolutional neural network, are the center coordinates of the bounding box, is the width and height of the bounding box, is the Sigmoid function, is the center coordinate of the anchor point, is the width and height of the anchor point, For the Anchor points, is the total number of anchor points, is the boundary loss, is the confidence loss, is the classification loss; the trained cigarette butt detection model is used to judge whether there is a cigarette butt in the captured head image. If there is a cigarette butt, it is determined that there is a person smoking. If there is no cigarette butt, it is determined that there is no smoking behavior.

[0048] Another object of the present invention is to provide a system for intelligent monitoring and identification of smoking behavior in industrial environments with the fusion of multiple algorithms, which can solve the problem of intelligent monitoring and identification of smoking behavior in industrial environments with the fusion of multiple algorithms by constructing an intelligent monitoring and identification system for smoking behavior in industrial environments.

[0049] To solve the above technical problems, the present invention provides the following technical solutions: an intelligent monitoring and identification system for smoking behavior in an industrial environment with multi-algorithm fusion, comprising a data acquisition module, a human body detection module, a human body joint point detection module and a head area detection module; the data acquisition module is used to acquire monitoring video frames in real time; the human body detection module is used to perform human body detection to determine whether there is a human body in the video frame; the human body joint point detection module is used to capture human body images in video frames where there is a human body, perform human body joint point detection, and determine whether it is a smoking action; the head area detection module is used to capture head images of human body images judged to be smoking actions, and perform cigarette butt detection on the head area to determine whether there is smoking.

[0050] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for intelligent monitoring and identification of smoking behavior in an industrial environment by integrating multiple algorithms as described above are implemented.

[0051] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for intelligent monitoring and identification of smoking behavior in an industrial environment by integrating multiple algorithms as described above.

[0052] The beneficial effects of the present invention are as follows: the multi-algorithm fusion intelligent monitoring and identification method for smoking behavior in industrial environments provided by the present invention does not require separate collection of materials, labeling, and training according to actual scenarios, and has a certain degree of versatility, and the model can be deployed and used. By analyzing the characteristics of smoking behavior, the identification steps are split, and the error conditions can be judged at the first time, thereby improving the recognition efficiency. Different targeted professional algorithms are used in the corresponding steps, and certain algorithm tuning is added according to the smoking behavior, thereby improving the accuracy of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0054] Figure 1 A flowchart of a multi-algorithm fusion method for intelligent monitoring and identification of smoking behavior in an industrial environment provided in the first embodiment of the present invention.

[0055] Figure 2 This is a structural diagram of the multi-algorithm fusion intelligent monitoring and identification system for smoking behavior in industrial environments provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0057] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0058] Example 1, reference Figure 1 , which is the first embodiment of the present invention, provides a multi-algorithm fusion method for intelligent monitoring and identification of smoking behavior in an industrial environment, including: acquiring monitoring video frames in real time; performing human body detection to determine whether there is a human body in the video frames; capturing human body images in video frames where there is a human body to perform human body joint point detection to determine whether it is a smoking action; capturing head images of human body images that are determined to be smoking actions, and performing cigarette butt detection on the head area to determine whether there is smoking.

[0059] Figure 1It is a recognition logic flow chart of an embodiment of the present invention. The figure shows the process from taking frames from the video stream to producing recognition results. At the beginning of the recognition stage, the video frame is taken, and then the personnel detection stage is entered to perform personnel detection to identify whether there is a human body in the video frame. If there is no human body, that is, the human body is not recognized, then no one is recognized to be smoking. If there is a human body, the human body image is captured, and then the action detection stage is entered to perform human joint point detection, extract human joint points, perform action detection, and determine whether it is a smoking action. If the smoking action is not recognized, no one is recognized to be smoking. If it is recognized as a smoking action, the cigarette butt detection stage is entered, and the head image is captured from the human body image. The cigarette butt detection is performed on the head area to determine whether there is a cigarette butt in the head area. If the cigarette butt is not recognized, no one is recognized to be smoking. If the cigarette butt is recognized, then someone is smoking.

[0060] S1. Acquire surveillance video frames in real time.

[0061] S2. Perform human body detection to determine whether there is a human body in the video frame.

[0062] Collect human body image materials, mark the materials, use YOLOv5 to train the marked material set, and produce a human body detection model. The loss function used in the training is expressed as:

[0063] ,

[0064] in, is the loss function of the human body detection model, is the weight of coordinate loss, is the confidence loss weight when there is no human body, is the size of the grid, The number of bounding boxes predicted for each grid cell, For grid cells The Whether the bounding box contains a human body, For grid cells Does it contain human body? For grid cells It does not include the human body, , are the center coordinates of the bounding box, is the width of the bounding box, is the height of the bounding box, is the confidence level, is the class probability, , are the center coordinates of the predicted bounding box, is the width of the predicted bounding box, is the height of the predicted bounding box, is the confidence level of the prediction, is the predicted class probability.

[0065] The training process of the human detection model is shown in Table 1.

[0066] Table 1 Human body detection model construction steps

[0067] ,

[0068] The trained human body detection model is used to judge whether there is a human body in the video frame. If there is no human body, it is determined that the smoking behavior is not recognized. If there is a human body, the image of the human body area is captured.

[0069] S3, capturing a human body image from a video frame in which a human body is present, performing human body joint point detection, and determining whether the action is a smoking action.

[0070] S3.1. Collect image datasets containing human joints, label the datasets, use the human posture estimation network HRNet to train the labeled datasets, extract multi-scale high-resolution feature maps, fuse the feature maps, and convert the fused feature maps into joint heat maps through the convolution layer to produce a human joint detection model. The training process of the human joint detection model is the same as that of the human detection model. The loss function is used to train the human joint detection model, which is expressed as:

[0071] ,

[0072] in, is the loss function of the human joint point detection model, is the batch size, is the number of joint points, is the predicted heat map, This is the real heat map.

[0073] The trained human joint point detection model is used to extract the human joint points from the captured human region image. The coordinates of each joint point are represented by the position of the maximum value in the heat map of each joint point, which is expressed as:

[0074] ,

[0075] in, are the coordinates of the human joint points, The coordinate position of the joint point in the heat map The heat map value indicates the probability or confidence value of this position belonging to a human joint point. To return the parameter that maximizes the function value, the coordinate position that maximizes the heat map value will be returned here .

[0076] The extracted human joint points are calculated to determine whether the human body image shows a smoking action.

[0077] S3.2. Get the two-dimensional coordinates of the joint points , where the wrist coordinates are , the shoulder coordinates are , the elbow coordinates are , the nose coordinates are , the left eye coordinates are , the right eye coordinates are .

[0078] S3.3. Calculate the arm bending angle.

[0079] S3.3.1. Calculate the wrist, elbow, and shoulder angles as follows:

[0080] ,

[0081] in, The angles of the wrist, elbow, and shoulder. Logic for calculating the angle formed by three points.

[0082] S3.3.2. Calculate the angle between the elbow and shoulder and the vertical as:

[0083] ,

[0084] in, is the angle between elbow and shoulder and vertical. To construct a point in the vertical direction, make it on the same vertical line as the shoulder (same horizontal coordinate), but the same vertical coordinate as the elbow.

[0085] calAngle is used to calculate three points , , The logic of the angle formed is expressed as:

[0086] S3.3.2.1. Calculate vectors BA and BC:

[0087] ,

[0088] ,

[0089] S3.3.2.2. Calculate the dot product of vectors BA and BC:

[0090] ,

[0091] S3.3.2.3. Calculate the modulus of vectors BA and BC:

[0092] ,

[0093] ,

[0094] S3.3.2.4. Calculate the angle:

[0095] ,

[0096] S3.4. Calculate the intersection of the extended line of the arm, the midpoint of the eyes and the extended line of the nose.

[0097] S3.4.1. Obtain the coordinates of the midpoints of both eyes, expressed as:

[0098] ,

[0099] in, are the midpoint coordinates of both eyes.

[0100] Determine whether the extended line of the arm, the midpoint of the eyes, and the extended line of the nose intersect. If so, calculate the intersection point, expressed as:

[0101] ,

[0102] in, If it is equal to 1, then it is parallel. If it is equal to 0, then they intersect. are the intersection coordinates, are the coordinates of the midpoints of both eyes, The logic to calculate whether the lines intersect and if so, calculate the intersection point.

[0103] calLine is calculated , Connection , Whether the lines intersect or not, if they do, the logic of calculating the intersection point is expressed as:

[0104] S3.4.1.1. Calculate vectors AB and CD:

[0105] ,

[0106] ,

[0107] S3.4.1.2. Calculate the cross product of vectors AB and CD:

[0108] ,

[0109] S3.4.1.3. Determine whether vectors AB and CD are parallel. If the cross product is zero, then vectors AB and CD are parallel. If they are not parallel (intersecting), calculate the intersection point:

[0110] ,

[0111] S3.4.2. If they are not parallel, that is, they intersect, then the distance from the wrist to the elbow is expressed as:

[0112] ,

[0113] in, is the distance from wrist to elbow, The logic for calculating the distance between two points.

[0114] The distance from the intersection point to the nose is calculated as:

[0115] ,

[0116] in, is the distance from the intersection point to the nose, It is the two-dimensional coordinate of the intersection of the extended line of the arm, the midpoint of the eyes and the extended line of the nose.

[0117] calDis is calculated , The logical distance between two points is expressed as:

[0118] ,

[0119] Judging whether it is a smoking action is expressed as:

[0120] ,

[0121] in, For smoking action, is the preset angle threshold, for Equal to 0 intersection, is the preset space size threshold, is the logical AND.

[0122] That is, the arm bending angle is less than , and the extended line of the arm intersects the midpoint of the eyes and the extended line of the nose, and the distance is close , it is determined as a smoking action, otherwise it is judged as not a smoking action, and it is determined that the smoking behavior is not recognized.

[0123] S5. Capturing a head image of the human body image judged to be in a smoking action, and performing cigarette butt detection on the head area to determine whether the person is smoking.

[0124] When it is determined that there is a smoking action, the trained head detection model is used to identify the human head area. or , then discard the current video frame and capture the head image of the video frame that is not discarded, which is expressed as:

[0125] ,

[0126] ,

[0127] ,

[0128] ,

[0129] ,

[0130] in, The width and height of the recognized head area. is the preset size filtering threshold, is the standard value of size, is the multiple threshold for enlarging the head area, is the coordinate of the center point of the head area, is the video frame width and height, and They are the coordinates of the upper left corner and lower right corner of the head area actually captured, is the boundary threshold.

[0131] That is, when the width and height of the recognized head area are smaller than the preset size filtering threshold, the current video frame is discarded, indicating that it does not meet the detection criteria of the cigarette butt recognition model. If it is greater than or equal to the preset size filtering threshold, the larger value of the width and height is multiplied by the preset multiple threshold, indicating that it is enlarged to the specified size and compared with the preset size standard value. The larger value is taken as the boundary threshold for actually intercepting the head area, and then the coordinates of the actual intercepted head area are calculated according to the boundary threshold.

[0132] Collect cigarette butt image materials, mark the materials, use the lightweight object detection network PicoDet to train the marked material set, and produce a cigarette butt detection model. The training process of the cigarette butt detection model is the same as that of the human body detection model, which can be expressed as:

[0133] ,

[0134] ,

[0135] ,

[0136] ,

[0137] ,

[0138] in, The offset of the bounding box predicted for each anchor point on the feature map extracted by the convolutional neural network, are the center coordinates of the bounding box, is the width and height of the bounding box, is the Sigmoid function, is the center coordinate of the anchor point, is the width and height of the anchor point, For the Anchor points, is the total number of anchor points, is the boundary loss, is the confidence loss, is the classification loss.

[0139] Anchors are a set of predefined bounding boxes that are laid out at each location in the feature map at different scales and ratios. These anchors are fixed at the beginning of training and are used to detect objects on the feature map.

[0140] The trained cigarette butt detection model is used to judge whether there is a cigarette butt in the captured head image. If there is a cigarette butt, it is determined that a person is smoking. If there is no cigarette butt, it is determined that there is no smoking behavior.

[0141] Example 2, reference Figure 2 , which is the second embodiment of the present invention, is different from the previous embodiment in that it provides an intelligent monitoring and identification system for smoking behavior in an industrial environment with multi-algorithm fusion, including: a data acquisition module, a human body detection module, a human joint point detection module and a head area detection module.

[0142] The data acquisition module is used to acquire surveillance video frames in real time.

[0143] The human body detection module is used to perform human body detection and determine whether there is a human body in the video frame.

[0144] The human joint point detection module is used to capture human body images from video frames containing human bodies and perform human joint point detection to determine whether it is a smoking action.

[0145] The head area detection module is used to capture the head image of a human body image that is judged to be smoking, and perform cigarette butt detection on the head area to determine whether the person is smoking.

[0146] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0148] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0149] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0150] Embodiment 3 is the third embodiment of the present invention. It is different from the first two embodiments in that it is used to verify the technical effects adopted in the present invention so as to verify the real effects of the method.

[0151] The smoking recognition method of the present invention significantly improves the accuracy and efficiency of smoking recognition by integrating multiple advanced algorithm models, including human body detection, joint point detection, action judgment, head detection and cigarette butt detection. The following is the data collected from relevant smoking recognition methods on the market. By comparison, it can be proved that the method of the present invention has more advantages in accuracy and efficiency, as shown in Table 2.

[0152] Table 2 Comparison of existing smoking recognition methods

[0153] ,

[0154] From the above data, we can see that:

[0155] Accuracy: The accuracy of the recognition method of the present invention reaches 96.5%, which is significantly higher than the existing smoking recognition method.

[0156] False alarm rate: The false alarm rate of the present invention is only 3.5%, which is significantly lower than that of the existing method.

[0157] Processing time: The present invention also has an advantage in processing time, requiring only 28ms per frame, which greatly improves processing efficiency.

[0158] The present invention achieves higher recognition accuracy and lower false alarm rate through the following improvements:

[0159] Multi-algorithm combination: Combining the advantages of different algorithm models, using human body detection, joint point detection, head detection and cigarette butt detection models to comprehensively improve recognition accuracy.

[0160] Smoking action determination: By calculating the joints of the human body, the positional relationship between the arm bending angle, the arm extension line, and the intersection of the midpoint of the eyes and the extension line of the nose is determined to accurately identify the smoking action.

[0161] Head area capture: Use the head detection model to accurately locate the head area, and use a reasonable image capture method to ensure the accuracy and efficiency of cigarette butt detection.

[0162] In the actual experiment, different detection models were configured for the surveillance video of an industrial unit, and the comparison results of false alarm rate and processing efficiency were shown in Table 3.

[0163] Table 3 Smoking recognition method optimization data table

[0164] ,

[0165] By comparison, it can be found that after multiple optimizations of the method of the present invention, the false alarm rates of all models have been significantly reduced, and the final average false alarm rate has dropped to 2.0%.

[0166] In summary, the method of the present invention shows significant superiority in the accuracy and efficiency of smoking recognition in industrial scenarios, can effectively reduce the false alarm rate, improve the recognition accuracy, is suitable for various complex application scenarios, and has broad application prospects.

[0167] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An intelligent monitoring and identification method for smoking behavior in industrial environments based on multi-algorithm fusion, characterized by: include, Acquire surveillance video frames in real time; Perform human body detection to determine whether there is a human body in the video frame; For video frames with human bodies, human body images are captured to detect human joints and determine whether it is a smoking action; For human images that are judged to be smoking, the head image is captured, and cigarette butt detection is performed on the head area to determine whether the person is smoking; The human body detection includes collecting human body image materials, marking the materials, using YOLOv5 to train the marked material set, and outputting a human body detection model. The training loss function is expressed as: Among them, Loss1 is the loss function of the human detection model, λ coord is the weight of coordinate loss, λ noobj is the confidence loss weight when there is no human body, S is the size of the grid, B is the number of bounding boxes predicted for each grid cell, is whether the jth bounding box in grid cell i contains a human body, is whether grid cell i contains a human body, If grid cell i does not contain a human body, x i ,y i is the center coordinate of the bounding box, w i is the width of the bounding box, h i is the height of the bounding box, C i is the confidence level, p i (c) is the class probability, are the center coordinates of the predicted bounding box, is the width of the predicted bounding box, is the height of the predicted bounding box, is the confidence of the prediction, is the predicted category probability; Use the trained human body detection model to judge whether there is a human body in the video frame. If there is no human body, it is considered that the smoking behavior is not recognized. If there is a human body, the human body area image is captured. The human joint point detection includes collecting an image data set containing human joint points, marking the data set, training the marked data set using a human posture estimation network HRNet, extracting multi-scale high-resolution feature maps, fusing the feature maps, converting the fused feature maps into joint point heat maps through a convolutional layer, generating a human joint point detection model, and training the human joint point detection model using a loss function, which is expressed as: Among them, Loss2 is the loss function of the human joint point detection model, N is the batch size, K is the number of joint points, is the predicted heat map, H n，k is the real heat map; The trained human joint point detection model is used to extract the human joint points from the captured human region image. The coordinates of each joint point are represented by the position of the maximum value in the heat map of each joint point, expressed as, Keypoint(x,y)=argmax(Heatmap(x,y)) Among them, Keypoint(x, y) is the coordinate of the human joint point, Heatmap(x, y) is the heat map value of the coordinate position (x, y) in the joint point heat map, and argmax is the parameter that returns the maximum function value; The determining whether it is a smoking action includes calculating the extracted human joint points to determine whether the human body image shows a smoking action; Get the two-dimensional coordinates of the joint point (pi x , pi y ), where the wrist coordinate is (p1 x , p1 y ), the shoulder coordinates are (p2 x , p2 y ), the elbow coordinate is (p3 x , p3 y ), the nose coordinates are (p4 x , p4 y ), the left eye coordinate is (p5 x , p5 y ), the right eye coordinate is (p6 x , p6 y ); The calculation of the extracted human joint points includes calculating the arm bending angle and calculating the intersection of the arm extension line and the midpoint of the eyes and the extension line of the nose; Calculating the arm bending angle includes calculating the angles of the wrist, elbow, and shoulder and calculating the angles between the elbow and shoulder and the vertical; The angles of the wrist, elbow and shoulder are calculated as follows: degree=calAngle(p1 x ,p1 y ,p3 x ,p3 y ,p2 x ,p2 y ) Among them, degree is the angle of wrist, elbow and shoulder, and calAngle is the logic for calculating the angle formed by the three points; The angle between the elbow, shoulder and vertical is expressed as: degree2=calAngle(p3 x ,p3 y ,p2 x ,p2 y ,p2 x ,p3 y ) Among them, degree2 is the angle between the elbow, shoulder and vertical; The calculation of the intersection of the arm extension line and the midpoint of both eyes and the extension line of the nose includes obtaining the coordinates of the midpoint of both eyes, expressed as: Among them, pt is the coordinate of the midpoint of both eyes; Determine whether the extension line of the arm, the midpoint of the eyes, and the extension line of the nose intersect. If so, calculate the intersection point, expressed as, (pp,ipt)=calLine(pt x ,pt y ,p4 x ,p4 y ,p3 x ,p3 y ,p1 x ,p1 y ) Among them, pP is equal to 1 if they are parallel, pP is equal to 0 if they intersect, ipt is the coordinate of the intersection point, (pt x , pt y ) is the coordinate of the midpoint of both eyes, calLine is the logic for calculating whether the connecting lines intersect, and if so, calculating the intersection point; If they are not parallel, that is, they intersect, then the distance from the wrist to the elbow is expressed as, ds1=calDis(p1 x ,p1 y ,p3 x ,p3 y ) Among them, ds1 is the distance from wrist to elbow, and calDis is the logic for calculating the distance between two points; The distance from the intersection to the nose is expressed as, ds2=calDis(ipt x ,ipt y ,p4 x ,p4 y ) Among them, ds2 is the distance from the intersection to the nose, (ipt x ,ipt y ) are the intersection coordinates; The determination of whether it is a smoking action is expressed as: Among them, flag is the smoking action, θ threshold is the preset angle threshold, The intersection is when pp is equal to 0, λ is the preset space size threshold, and ∧ is the logical AND; The arm bending angle is less than θ threshold , and the extended line of the arm intersects the midpoint of the eyes and the extended line of the nose, and the distance ds2<λ×ds1, it is determined to be a smoking action, otherwise it is judged not to be a smoking action, and it is determined that the smoking behavior is not recognized.

2. The multi-algorithm fusion intelligent monitoring and identification method for smoking behavior in industrial environments as claimed in claim 1 is characterized by: The head image capture includes, when it is determined that there is a smoking action, using the trained head detection model to identify the human head area, if w0<δ or h0<δ, then discarding the current video frame, and capturing the head image of the video frame that is not discarded, expressed as: p=max(stan,max(w0,h0)×θ) x tl =min(max(0,c x -p / 2),width-p) y tl =min(max(0,c y -p / 2),height-p) x br =x tl +p and br =and tl +p Among them, (w0, h0) is the width and height of the recognized head area, δ is the preset size filtering threshold, stan is the size standard value, θ is the multiple threshold of the enlarged head area, (c x , c y ) is the coordinate of the center point of the head area, (width, height) is the width and height of the video frame, (x tl ,y tl ) and (x br ,y br ) are the coordinates of the upper left corner and the lower right corner of the head area actually intercepted, and p is the boundary threshold; When the width and height of the recognized head area are less than the preset size filtering threshold, the current video frame is discarded, indicating that it does not meet the detection criteria of the cigarette butt recognition model. If it is greater than or equal to the preset size filtering threshold, the larger value of the width and height is multiplied by the preset multiple threshold, indicating that it is enlarged to the specified size and compared with the preset size standard value. The larger value is taken as the boundary threshold for actually intercepting the head area, and then the coordinates of the actual intercepted head area are calculated according to the boundary threshold.

3. The multi-algorithm fusion intelligent monitoring and identification method for smoking behavior in industrial environments as claimed in claim 2 is characterized by: The cigarette butt detection in the head area includes collecting cigarette butt image materials, marking the materials, using the lightweight object detection network PicoDet to train the marked material set, and outputting a cigarette butt detection model, which is expressed as: in, The offset of the bounding box predicted for each anchor point on the feature map extracted by the convolutional neural network, are the center coordinates of the bounding box, is the width and height of the bounding box, σ is the Sigmoid function, (c x , c y ) is the center coordinate of the anchor point, (p w , p h ) is the width and height of the anchor point, l is the lth anchor point, Q is the total number of anchor points, L bbox is the boundary loss, L conf is the confidence loss, L cls is the classification loss; The trained cigarette butt detection model is used to judge whether there is a cigarette butt in the captured head image. If there is a cigarette butt, it is determined that a person is smoking. If there is no cigarette butt, it is determined that there is no smoking behavior.

4. A system using the multi-algorithm fusion method for intelligent monitoring and identification of smoking behavior in industrial environments as claimed in any one of claims 1 to 3, characterized in that: Including data acquisition module, human body detection module, human joint point detection module and head area detection module; The data acquisition module is used to acquire monitoring video frames in real time; The human body detection module is used to perform human body detection and determine whether there is a human body in the video frame; The human joint point detection module is used to detect human joint points in human body images captured from video frames containing human bodies, and determine whether it is a smoking action; The head area detection module is used to capture the head image of the human body image judged to be smoking, and perform cigarette butt detection on the head area to determine whether smoking occurs.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for intelligent monitoring and identification of smoking behavior in an industrial environment by integrating multiple algorithms as described in any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for intelligent monitoring and identification of smoking behavior in an industrial environment by integrating multiple algorithms as described in any one of claims 1 to 3 are implemented.

Images