Method and device for detecting smoking based on machine vision and storage medium

Abstract

The present application relates to a kind of based on machine vision's smoking detection method, device and storage medium, wherein method includes: step S1: obtaining live image, respectively input to the first segmentation detection model and the second segmentation detection model of trained good;Step S2: obtain the target detection result and the position descriptor of each target output by segmentation detection model;Step S3: for each all type is continuous cigarette target region detection whether there is cigarette end, if no, then as smoking detection area, otherwise, input to the fourth target detection model, detect whether there is fire, if yes, then judge whether there is overlap region, if yes, then as smoking detection area;Step S4: judge whether all type is not continuous cigarette target region and each type is hand target region and there is overlap, and whether there is smoking detection area according to the size of overlap region is judged.Compared with prior art, the present application has the advantages such as accuracy.

Description

Technical Field

[0001] This invention relates to the field of technical supervision of communication engineering projects, and in particular to a method, device and storage medium for detecting smoking based on machine vision. Background Technology

[0002] During the construction of communication projects, smoking may cause multiple systemic hazards. On the one hand, in enclosed spaces (such as base station equipment rooms and pipe shafts), the concentration of carcinogens in secondhand smoke (benzo[a]pyrene, nitrosamines) exceeds the national standard by 5-10 times (GB 37488-2019), and long-term exposure increases the risk of chronic bronchitis by 35%. On the other hand, when working at heights (such as climbing iron towers), nicotine-induced hand tremors (tremor amplitude ≥2mm) increase the risk of falls and produce a synergistic effect of electromagnetic radiation: base station maintenance personnel are simultaneously exposed to microwaves (SAR≥2W / kg) and smoke, and experiments have confirmed that nicotine exacerbates the rate of DNA single-strand breaks.

[0003] In addition to harming people, it can also cause damage to materials, such as circuit board corrosion, heat dissipation system failure, and signal quality degradation. Tar deposits can increase solder joint impedance by 15-20%. Statistics from a certain operator show that the annual failure rate of equipment in smoking areas is 2.27 times that in non-smoking areas. Ash can clog server fans, raising CPU temperature by 8-12°C and accelerating the aging of electronic components. Ash adsorption can reduce the gain of 5G base station antennas by 0.5-1dB (3GPP standard allows fluctuations of ±0.3dB), and reduce edge user speeds by 15-20%.

[0004] Therefore, smoking is generally strictly controlled in communication projects. To address this, some existing technologies provide smoking detection solutions. For example, Chinese patent CN115424355A discloses a smoking detection method and device, electronic device, and computer-readable storage medium. The method includes: acquiring the original image to be detected; obtaining a user head image based on the original image, the user head image being an image obtained from the pixels of the user's head region in the original image; obtaining first region information and second region information based on the user head image, the first region information indicating the position of the user's mouth region in the user head image, and the second region information indicating the position of the cigarette region in the user head image; and obtaining a target detection result based on the first region information and the second region information, the target detection result indicating whether the user in the original image is smoking.

[0005] However, in the above-mentioned existing technologies, the target smoking detection model is trained based on head images (steps S601-S603), and its training samples only contain cigarettes in the head region. Therefore, it cannot actually solve the problem of holding a cigarette. In response, some improved existing technologies, such as Chinese patent CN116843947A, disclose a smoking behavior detection method. In this method, the image to be detected is input into a pre-trained target detection model, and the position information of the target box containing at least one of the cigarettes, heads, and hands in the image output by the target detection model is obtained. If it is determined that the target boxes containing at least two of the cigarettes, heads, and hands in the image overlap, the box containing the overlapping target box is cropped. This method, which involves taking a sub-image containing the bounding boxes of at least two overlapping targets and then inputting the sub-image containing the bounding boxes of both targets into the classification image, solves the detection of smoking behavior by holding a cigarette to some extent. However, it requires a large number of training photos of people holding cigarettes for the final smoking behavior detection, and there are no readily available tools available, resulting in a large number of training sample annotations and high implementation costs. In addition, it uses the same model for cigarette and hand segmentation. On the one hand, the existing model needs to be improved, otherwise the same region will be cropped into multiple parts, resulting in information loss. On the other hand, the model needs to be retrained, and it involves multiple different targets, so the training cost is high. Summary of the Invention

[0006] The purpose of this invention is to provide a method, device, and storage medium for detecting smoking based on machine vision.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] A machine vision-based method for detecting smoking includes:

[0009] Step S1: Acquire the scene image and input it into the trained first segmentation and detection model and the second segmentation and detection model respectively;

[0010] Step S2: Obtain the target detection results and position descriptors of each target output by the segmentation detection model, wherein the target types output by the first segmentation detection model include continuous cigarettes and discontinuous cigarettes, and the target types output by the second segmentation detection model include hands;

[0011] Step S3: For each target area of ​​type continuous cigarette, detect whether there is a cigarette butt. If not, the target area of ​​type continuous cigarette is used as the smoking detection area. Otherwise, the target area of ​​type continuous cigarette is expanded and input into the fourth target detection model to detect whether there is a fire. If yes, determine whether there is an overlap between the fire area and the cigarette area. If yes, the target area of ​​type continuous cigarette is used as the smoking detection area.

[0012] Step S4: Determine whether the target area of ​​all types of discontinuous cigarettes overlaps with the target area of ​​each type of hand, and determine whether there is a smoking detection area based on the size of the overlapping area.

[0013] The on-site images are extracted from the on-site video through a pre-configured frame-sampling interval.

[0014] Step S4 includes:

[0015] Step S4-1: Determine whether there is any overlap between the target area of ​​all types of discontinuous cigarettes and the target area of ​​each type of hand, and form a recognition pair for each pair of overlapping target areas of type discontinuous cigarettes and target areas of type hand;

[0016] Step S4-2: For each identification pair, calculate the length ratio of the projection line segment of the overlapping area on any straight line to the projection line segment of the target area of ​​the discontinuous cigarette on any straight line. Determine whether the calculated length ratio is greater than the pre-configured first threshold. If it is, determine that the target area of ​​the discontinuous cigarette in the identification pair is the smoking detection area, and delete other untraversed identification pairs that contain the target area of ​​the discontinuous cigarette in the identification pair.

[0017] The second segmentation detection model uses a pre-trained open-source model and model weights, while the first segmentation detection model is self-trained.

[0018] Step S3, which involves detecting the presence of cigarette butts in the target area of ​​a single type of continuous cigarette, specifically includes:

[0019] Convolution calculations are performed to achieve contour detection;

[0020] The ratio of the length of each contour line to the length of the diagonal detected in the current target area is determined as the first ratio of the contour line, and all contour lines with a first ratio exceeding 60% are taken as the first contour lines.

[0021] Determine whether there are two parallel first contour lines, and form a first line pair by any two parallel first contour lines.

[0022] Determine whether any first contour line in each first line pair has a first ratio exceeding 75%. If so, then the first line pair is regarded as the second line pair.

[0023] Determine whether there exists a contour line between the two first contour lines of all second line pairs that is perpendicular to the two first contour lines and whose endpoints are located on the two first contour lines respectively. If so, take the contour line that is perpendicular to the two first contour lines and whose endpoints are located on the two first contour lines respectively as the dividing contour line, and calculate the ratio of the lengths of the two first contour lines on both sides of the dividing contour line. If the ratio is less than 1, take the ratio as the second ratio of the first contour line. Otherwise, take the reciprocal of the ratio as the second ratio of the first contour line.

[0024] Determine whether there exists a second ratio or its reciprocal between the two first contour lines in the second line pair. If yes, determine whether there exists a contour line perpendicular to the shorter end of the first contour line located between the two first contour lines in the second line pair, with the endpoints located on the two first contour lines respectively. If yes, determine that there is a cigarette butt.

[0025] Step S3, which involves detecting the presence of cigarette butts in the target area of ​​a single type of continuous cigarette, specifically includes:

[0026] The target region of type continuous cigarette is input into the trained third segmentation detection model to obtain the target detection results and position descriptors of each target output by the third segmentation detection model, and to determine whether there is a cigarette butt.

[0027] In step S3, during the process of expanding the target area of ​​the type of continuous cigarette, the specific expansion is 200%.

[0028] In the process of performing convolution calculation to achieve contour detection, the convolution kernel size used is 3×3.

[0029] A machine vision-based smoking detection device includes a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method described above.

[0030] A storage medium having a program stored thereon, wherein the program, when executed, implements the method described above.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. Unlike existing technologies that require labeling and training on various smoking actions in different postures and rely on a large number of samples to improve detection accuracy, the sample labeling of this invention only includes two categories: continuous cigarettes and discontinuous cigarettes. This can greatly reduce the interference of various types of smoking actions on the model training process and greatly reduce the sample requirements during model training. Subsequently, the presence of smoking can be determined by the positional relationship of different areas. This is beneficial for directly using edge computing devices such as drones for computational inference, without relying on the network. Therefore, it can effectively solve the problem of smoking detection in construction sites with unstable communication.

[0033] 2. For discontinuous cigarettes, the presence of smoking can be determined by the proportion of the projection line length of the hand intersecting with the cigarette area. On the one hand, this eliminates the need for image annotation, which can greatly reduce costs. On the other hand, it can solve the problem of missed detection caused by the hand not being completely cut off from the cigarette stem due to shooting angle issues.

[0034] 3. The second segmentation detection model uses a pre-trained open-source model and model weights, thus leveraging the existing achievements of the open-source community and reducing training costs.

[0035] 4. By performing convolution calculations and combining the found boundary lines of cigarettes, the presence of cigarette butts is determined by segmentation lines. For parts with cigarette butts, smoking is further determined by smoke and fire recognition. For parts without cigarette butts, the distance to any part of the human body must be relatively close, thus determining that smoking has occurred. This method does not rely on image annotation, which can greatly reduce costs and is suitable for deployment on low-computing-power edge computing platforms. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the main steps of the method of the present invention;

[0037] Figure 2 This is a schematic diagram of step S4 in a certain embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of the process for detecting the presence of cigarette butts in a certain embodiment of the present invention. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0040] A machine vision-based method for detecting smoking, such as Figure 1 As shown, it includes:

[0041] Step S1: Acquire the scene image and input it into the trained first segmentation and detection model and the second segmentation and detection model respectively;

[0042] The on-site images are obtained from the on-site video through a pre-configured frame extraction interval. In this embodiment, the on-site video is captured by a mounted camera. Of course, in some other embodiments, it can also be captured by drone inspection. In addition, the frame extraction interval in this embodiment is a fixed interval, that is, one frame is extracted every 5 seconds. However, in some other embodiments, other fixed intervals can be used, such as 10 seconds, 3 seconds, etc. In addition, in some other embodiments, a dynamic interval can also be used. The specific dynamic interval setting can be determined according to the network transmission quality.

[0043] Furthermore, in all embodiments of this application, the second segmentation detection model uses a pre-trained open-source model and model weights, while only the first segmentation detection model is self-trained. The target types output by the first segmentation detection model include continuous cigarettes and non-continuous cigarettes, while the target types output by the second segmentation detection model include hands. With this design, the detection of hands does not require labeled samples. Unlike existing technologies, which require labeling and training for various smoking actions in different postures and rely on a large number of samples to improve detection accuracy, the sample labeling of this invention only includes two categories: continuous cigarettes and non-continuous cigarettes. This can greatly reduce the interference of various types of smoking actions on the model training process and greatly reduce the sample requirements during model training. Subsequently, the presence of smoking can be determined by the positional relationship of different areas, which is beneficial for directly using edge computing devices such as drones for computational inference without relying on the network. This can effectively solve the problem of smoking detection at construction sites with unstable communication.

[0044] The second segmentation detection model can use MediaPipe Hands or YOLO series, etc., and adopt open source weights. For example, when using MediaPipe Hands, since MediaPipe Hands can output the coordinates of 21 hand key points, the area involving these 21 hand key points is taken as the hand region.

[0045] The first segmentation detection model only detects two types of cigarettes: continuous cigarettes and non-continuous cigarettes.

[0046] Step S2: Obtain the target detection results and position descriptors of each target output by the segmentation detection model. Since the target types output by the first segmentation detection model include continuous cigarettes and discontinuous cigarettes, for its training samples, some cigarette images can be randomly selected, and a portion can be randomly cut from both sides as samples of continuous cigarettes, a portion can be cut from the middle as samples of discontinuous cigarettes, and a portion can be randomly cut from the middle and both sides as samples of discontinuous cigarettes.

[0047] Since the first segmentation detection model and the second segmentation detection model are based on the same image as input and the segmentation detection process does not interfere with each other, the two can calculate the position of the identified results and do not need to be jointly trained, thus reducing the training cost.

[0048] Step S3: For each target area of ​​type continuous cigarette, detect whether there is a cigarette butt. If not, the target area of ​​type continuous cigarette is used as the smoking detection area. Otherwise, the target area of ​​type continuous cigarette is expanded and input into the fourth target detection model to detect whether there is a fire. If yes, determine whether there is an overlap between the fire area and the cigarette area. If yes, the target area of ​​type continuous cigarette is used as the smoking detection area.

[0049] In some embodiments, step S3, which detects whether a cigarette butt exists in the area where a target of type continuous cigarette is located, specifically includes: inputting the area where a target of type continuous cigarette is located into a trained third segmentation detection model, obtaining the target detection result and the position descriptor of each target output by the third segmentation detection model, and determining whether a cigarette butt exists.

[0050] However, in this embodiment, in order to reduce the performance requirements of the end-side devices, such as Figure 3 As shown, step S3, which detects the presence of cigarette butts in the target area of ​​a single type of continuous cigarette, specifically includes:

[0051] Convolutional calculations are performed to achieve contour detection. Generally, the convolutional kernel size used in contour detection is 3×3.

[0052] The ratio of the length of each contour line to the length of the diagonal detected in the current target area is determined as the first ratio of the contour line, and all contour lines with a first ratio exceeding 60% are taken as the first contour lines.

[0053] Determine whether there are two parallel first contour lines, and form a first line pair by any two parallel first contour lines.

[0054] Determine whether any first contour line in each first line pair has a first ratio exceeding 75%. If so, then the first line pair is regarded as the second line pair.

[0055] Determine whether there exists a contour line between the two first contour lines of all second line pairs that is perpendicular to the two first contour lines and whose endpoints are located on the two first contour lines respectively. If so, take the contour line that is perpendicular to the two first contour lines and whose endpoints are located on the two first contour lines respectively as the dividing contour line, and calculate the ratio of the lengths of the two first contour lines on both sides of the dividing contour line. If the ratio is less than 1, take the ratio as the second ratio of the first contour line. Otherwise, take the reciprocal of the ratio as the second ratio of the first contour line.

[0056] Determine whether there exists a second ratio or its reciprocal between the two first contour lines in the second line pair. If yes, determine whether there exists a contour line perpendicular to the shorter end of the first contour line located between the two first contour lines in the second line pair, with the endpoints located on the two first contour lines respectively. If yes, determine that there is a cigarette butt.

[0057] By using convolutional computation, combined with the boundary line of the cigarette, and using the segmentation line to determine whether there is a cigarette butt, for the part with a cigarette butt, further determination of whether smoking has occurred is made through smoke recognition. For the part without a cigarette butt, it must be close to any part of the human body, thus determining that smoking has occurred. This method does not rely on image annotation, which can greatly reduce costs and is suitable for deployment on low-computing-power edge computing platforms.

[0058] Furthermore, in this embodiment, during the process of expanding the target area of ​​the type of continuous cigarette in step S3, the expansion is specifically 200%.

[0059] Step S4: Determine whether the target areas of all types of discontinuous cigarettes overlap with the target areas of all types of hands, and determine whether there is a smoking detection area based on the size of the overlapping area. In this embodiment, specifically, as shown... Figure 2 As shown, it includes:

[0060] Step S4-1: Determine whether there is any overlap between the target area of ​​all types of discontinuous cigarettes and the target area of ​​each type of hand, and form a recognition pair for each pair of overlapping target areas of type discontinuous cigarettes and target areas of type hand;

[0061] Step S4-2: For each identification pair, calculate the length ratio of the projection line segment of the overlapping area on any straight line to the projection line segment of the target area of ​​the discontinuous cigarette on any straight line. Determine whether the calculated length ratio is greater than the pre-configured first threshold. If it is, determine that the target area of ​​the discontinuous cigarette in the identification pair is the smoking detection area, and delete other untraversed identification pairs that contain the target area of ​​the discontinuous cigarette in the identification pair.

[0062] For discontinuous cigarettes, the presence of smoking can be determined by the proportion of the projection line length of the hand intersecting with the cigarette area. On the one hand, this eliminates the need for image annotation, which can greatly reduce costs. On the other hand, it can solve the problem of missed detection caused by the hand not being completely cut off from the cigarette stem due to shooting angle issues.

[0063] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A machine vision-based method for detecting smoking, characterized in that, include: Step S1: Acquire the scene image and input it into the trained first segmentation and detection model and the second segmentation and detection model respectively; Step S2: Obtain the target detection results and position descriptors of each target output by the segmentation detection model, wherein the target types output by the first segmentation detection model include continuous cigarettes and discontinuous cigarettes, and the target types output by the second segmentation detection model include hands; Step S3: For each target area of ​​type continuous cigarette, detect whether there is a cigarette butt. If not, the target area of ​​type continuous cigarette is used as the smoking detection area. Otherwise, the target area of ​​type continuous cigarette is expanded and input into the fourth target detection model to detect whether there is a fire. If yes, determine whether there is an overlap between the fire area and the cigarette area. If yes, the target area of ​​type continuous cigarette is used as the smoking detection area. Step S4: Determine whether the target area of ​​all types of discontinuous cigarettes overlaps with the target area of ​​each type of hand, and determine whether there is a smoking detection area based on the size of the overlapping area.

2. The machine vision-based smoking detection method according to claim 1, characterized in that, The on-site images are extracted from the on-site video through a pre-configured frame-sampling interval.

3. The machine vision-based smoking detection method according to claim 1, characterized in that, Step S4 includes: Step S4-1: Determine whether there is any overlap between the target area of ​​all types of discontinuous cigarettes and the target area of ​​each type of hand, and form a recognition pair for each pair of overlapping target areas of type discontinuous cigarettes and target areas of type hand; Step S4-2: For each identification pair, calculate the length ratio of the projection line segment of the overlapping area on any straight line to the projection line segment of the target area of ​​the discontinuous cigarette on any straight line. Determine whether the calculated length ratio is greater than the pre-configured first threshold. If it is, determine that the target area of ​​the discontinuous cigarette in the identification pair is the smoking detection area, and delete other untraversed identification pairs that contain the target area of ​​the discontinuous cigarette in the identification pair.

4. The machine vision-based smoking detection method according to claim 1, characterized in that, The second segmentation detection model uses a pre-trained open-source model and model weights, while the first segmentation detection model is self-trained.

5. The machine vision-based smoking detection method according to claim 1, characterized in that, Step S3, which involves detecting the presence of cigarette butts in the target area of ​​a single type of continuous cigarette, specifically includes: Convolution calculations are performed to achieve contour detection; The ratio of the length of each contour line to the length of the diagonal detected in the current target area is determined as the first ratio of the contour line, and all contour lines with a first ratio exceeding 60% are taken as the first contour lines. Determine whether there are two parallel first contour lines, and form a first line pair by any two parallel first contour lines. Determine whether any first contour line in each first line pair has a first ratio exceeding 75%. If so, then the first line pair is regarded as the second line pair. Determine whether there exists a contour line between the two first contour lines of all second line pairs that is perpendicular to the two first contour lines and whose endpoints are located on the two first contour lines respectively. If so, take the contour line that is perpendicular to the two first contour lines and whose endpoints are located on the two first contour lines respectively as the dividing contour line, and calculate the ratio of the lengths of the two first contour lines on both sides of the dividing contour line. If the ratio is less than 1, take the ratio as the second ratio of the first contour line; otherwise, take the reciprocal of the ratio as the second ratio of the first contour line. Determine whether there exists a second ratio or its reciprocal between the two first contour lines in the second line pair. If yes, determine whether there exists a contour line perpendicular to the shorter end of the first contour line located between the two first contour lines in the second line pair, with the endpoints located on the two first contour lines respectively. If yes, determine that there is a cigarette butt.

6. The machine vision-based smoking detection method according to claim 1, characterized in that, Step S3, which involves detecting the presence of cigarette butts in the target area of ​​a single type of continuous cigarette, specifically includes: The target region of type continuous cigarette is input into the trained third segmentation detection model to obtain the target detection results and position descriptors of each target output by the third segmentation detection model, and to determine whether there is a cigarette butt.

7. The machine vision-based smoking detection method according to claim 1, characterized in that, In step S3, during the process of expanding the target area of ​​the type of continuous cigarette, the specific expansion is 200%.

8. The machine vision-based smoking detection method according to claim 5, characterized in that, In the process of performing convolution calculation to achieve contour detection, the convolution kernel size used is 3×3.

9. A machine vision-based smoking detection device, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-8.

10. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-8.

Images