Smoking detection method and device based on machine vision and storage medium

Abstract

The invention relates to a smoking detection method and device based on machine vision and a storage medium, and the method comprises the steps: S1, obtaining a field image, and inputting the field image into a trained first segmentation detection model and a trained second segmentation detection model; s2, obtaining a target detection result output by the segmentation detection model and a position descriptor of each target; s3, detecting whether a cigarette butt exists in each region where all types of targets are continuous cigarettes, if not, taking the region as a smoking detection region, otherwise, inputting the region into a fourth target detection model, detecting whether smoke and fire exist, if yes, judging whether an overlapping region exists, and if yes, taking the region as the smoking detection region; and S4, judging whether the areas where all the targets of which the types are discontinuous cigarettes are located are overlapped with the areas where all the targets of which the types are hands are located or not, and judging whether a smoking detection area exists or not according to the size of the overlapped areas. Compared with the prior art, the method has the advantages of high accuracy and the like.

Description

Technical Field

[0001] The present invention relates to the field of technical supervision of communication engineering projects, and in particular to a machine vision-based smoking detection method, device and storage medium. Background Art

[0002] During the construction of communication projects, smoking may cause multiple systemic hazards. On the one hand, in closed spaces (such as base station rooms and pipe wells), the concentration of secondhand smoke carcinogens (benzopyrene and nitrosamines) exceeds the national standard by 5-10 times (GB 37488-2019). Long-term exposure increases the risk of chronic bronchitis by 35%. On the other hand, when working at heights (such as climbing towers), nicotine-induced hand tremors (tremor amplitude ≥ 2mm) increase the risk of falling and produce a synergistic effect of electromagnetic radiation: base station maintenance personnel are exposed to microwaves (SAR ≥ 2W / kg) and smoke at the same time. Experiments have confirmed that nicotine increases the rate of single-strand breaks in DNA.

[0003] In addition to the harm to people, it also causes physical harm, such as circuit board corrosion, cooling system failure, and signal quality degradation. Tar deposition increases solder joint impedance by 15-20%. Statistics from one operator show that the average annual equipment failure rate in smoking areas is 2.27 times that of non-smoking areas. Ash clogs server fans, raising CPU temperatures by 8-12°C and accelerating the aging of electronic components. Ash adsorption reduces the gain of 5G base station antennas by 0.5-1dB (the 3GPP standard allows a fluctuation of ±0.3dB), resulting in a 15-20% drop in edge user data rates.

[0004] Therefore, smoking generally needs to be strictly controlled in communication projects. In this regard, some existing technologies provide smoking detection solutions. For example, Chinese patent CN115424355A discloses a smoking detection method and device, electronic device, and computer-readable storage medium. The method includes: obtaining an original image to be detected: obtaining a user head image based on the original image, and the user head image is an image obtained based on pixels in the user head area in the original image: obtaining first area information and second area information based on the user head image, the first area information indicating the position of the user's mouth area in the user head image, and the second area information indicating the position of the cigarette area in the user head image; obtaining a target detection result based on the first area information and the second area information, and the target detection result indicating whether the user in the original image is in a smoking state.

[0005] However, in the above prior art, the target smoking detection model is trained based on head images (steps S601-S603), and its training samples only include cigarettes in the head area. Therefore, it is actually unable to solve the state of holding cigarettes. In this regard, some improved prior art, such as Chinese patent CN116843947A discloses a smoking behavior detection method, in which the image to be detected is input into a pre-trained target detection model, and the position information of the target frame where at least one target among cigarettes, human heads and hands contained in the image output by the target detection model is obtained; if it is determined that the target frames where at least two targets among cigarettes, human heads and hands contained in the image overlap, the target frame containing the overlapped target is intercepted. In the sub-image of the target frame where at least two overlapping targets are located, the sub-image of the target frame where the two targets are located is then input into the classification image. This method solves the detection of the smoking behavior of holding a cigarette to a certain extent. However, it requires a large number of training photos of holding cigarettes in the final detection of smoking behavior, and there are no ready-made tools to use, resulting in the need for a large number of training sample annotations, and the implementation cost is high. In addition, it uses the same model to segment cigarettes and hands. On the one hand, the existing model needs to be improved, otherwise it will not be possible to cause the same area to be cropped into multiple parts, resulting in information loss. On the other hand, the model needs to be retrained and involves multiple different targets, so the training cost is high. Summary of the Invention

[0006] The purpose of the present invention is to provide a smoking detection method, device and storage medium based on machine vision.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A smoking detection method based on machine vision, comprising:

[0009] Step S1: Acquire a scene image and input it into the trained first segmentation detection model and the second segmentation detection model respectively;

[0010] Step S2: Obtaining target detection results and location descriptors of each target output by the segmentation detection model, wherein the target types output by the first segmentation detection model include continuous cigarettes and discontinuous cigarettes, and the target types output by the second segmentation detection model include hands;

[0011] Step S3: Detecting whether cigarette butts are present in the target area of ​​each continuous cigarette type. If not, the target area of ​​each continuous cigarette type is used as a smoking detection area. Otherwise, the target area of ​​each continuous cigarette type is expanded and input into the fourth target detection model to detect whether there are fireworks. If so, determining whether there is overlap between the fireworks area and the cigarette area. If so, the target area of ​​each continuous cigarette type is used as a smoking detection area.

[0012] Step S4: Determine whether the target areas of all types of discontinuous cigarettes overlap with the target areas of all types of hands, and determine whether there is a smoking detection area based on the size of the overlapping area.

[0013] The live image is obtained by extracting a preconfigured frame interval based on the live video.

[0014] The step S4 comprises:

[0015] Step S4-1: Determine whether all discrete cigarette target regions overlap with all hand target regions, and form an identification pair for each pair of discrete cigarette target regions and hand target regions that have overlapping regions.

[0016] Step S4-2: For each recognition pair, calculate the length ratio of the projected line segment of the overlapping area on any straight line to the projected line segment of the target area of ​​the discontinuous cigarette on any straight line, and determine whether the calculated length ratio is greater than a preconfigured first threshold. If so, determine that the target area of ​​the type of discontinuous cigarette in the recognition pair is the smoking detection area, and delete other untraversed recognition pairs that contain the target area of ​​the type of discontinuous cigarette in the recognition pair.

[0017] The second segmentation detection model uses a trained open source model and model weights, and the first segmentation detection model uses self-training.

[0018] The process of step S3 detecting whether there is a cigarette butt in the area where the target of a single continuous cigarette is located specifically includes:

[0019] Perform convolution calculation to achieve contour detection;

[0020] Determine the ratio of the length of each contour line to the length of the diagonal line detected in the area where the current target is located one by one as the first ratio of the contour line, and take all contour lines whose first ratio exceeds 60% as the first contour line;

[0021] Determine whether there are two first contour lines that are parallel to each other, and form any two first contour lines that are parallel to each other into a first line pair;

[0022] Determine whether any first contour line in each first line pair has a first ratio exceeding 75%, and if so, use the first line pair as the second line pair;

[0023] Determine whether there is a contour line between the two first contour lines of all second line pairs, which is perpendicular to the two first contour lines and has endpoints located on the two first contour lines respectively. If so, use the contour line that is perpendicular to the two first contour lines and has endpoints located on the two first contour lines respectively as the separating contour line, and calculate the ratio of the lengths of the two first contour lines on both sides of the separating contour line. If the ratio is less than 1, use the ratio as the second ratio of the first contour line; otherwise, use the reciprocal of the ratio as the second ratio of the first contour line.

[0024] Determine whether the second ratio of the two first contour lines in the second line pair or the reciprocal of any of the first contour lines is within the first set interval. If so, determine whether a contour line perpendicular to the first contour line exists at the end of the shorter end of the first contour line and is located between the two first contour lines in the second line pair, and the endpoints are respectively located on the two first contour lines. If so, determine that a cigarette butt exists.

[0025] The process of step S3 detecting whether there is a cigarette butt in the area where the target of a single continuous cigarette is located specifically includes:

[0026] The area where the target type of continuous cigarettes is located is input into the trained third segmentation detection model, and the target detection results and position descriptors of each target output by the third segmentation detection model are obtained to determine whether there are cigarette butts.

[0027] In the process of expanding the area where the target of the type of continuous cigarettes is located in step S3, the area is expanded by 200%.

[0028] In the process of performing convolution calculation to realize contour detection, the convolution kernel size used is 3×3.

[0029] A smoking detection device based on machine vision includes a memory, a processor, and a program stored in the memory. When the processor executes the program, the above method is implemented.

[0030] A storage medium stores a program, which implements the above method when the program is executed.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. Unlike the existing technology, which requires labeling and training of smoking actions in various postures and relies on a large number of samples to improve the detection accuracy, the sample labeling of the present invention only includes two categories: continuous cigarettes and discontinuous cigarettes, which can greatly reduce the interference of various types of smoking actions on the model training process, and can greatly reduce the sample requirements in the model training process. Subsequently, the presence of smoking can be judged by the positional relationship of different areas, which is conducive to the direct use of edge computing devices such as drones for computational reasoning, without relying on the network, thereby effectively solving the problem of smoking detection at on-site construction sites with unstable communications.

[0033] 2. For discontinuous cigarettes, the presence of smoking is determined by the ratio of the hand's projection line length that intersects the cigarette area. This eliminates the need for image annotation, significantly reducing costs. It also addresses missed detections caused by the hand not completely cutting off the cigarette stem due to shooting angle issues.

[0034] 3. The second segmentation detection model uses trained open source models and model weights, thereby leveraging the achievements of the existing open source community and reducing training costs.

[0035] 4. Through convolution calculation, the boundary line of the cigarette is found, and the presence of cigarette butts is determined by segmentation. For the part with cigarette butts, the presence of smoking is further determined by smoke recognition. For the part without cigarette butts, it must be close to any part of the human body, and thus it is determined that smoking is present. This eliminates the need for image annotation, greatly reduces costs, and is suitable for deployment on low-computing-power edge computing platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the main steps of the method of the present invention;

[0037] Figure 2 is a schematic diagram of step S4 in an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of a process for detecting whether a cigarette butt is present in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0040] A smoking detection method based on machine vision, such as Figure 1 Shown, including:

[0041] Step S1: Acquire a scene image and input it into the trained first segmentation detection model and the second segmentation detection model respectively;

[0042] The on-site image is obtained based on the on-site video through a pre-configured frame extraction interval. In this embodiment, the on-site video is obtained by shooting with a mounted camera. Of course, in some other embodiments, it can also be obtained by shooting with a drone inspection. In addition, the frame extraction interval in this embodiment adopts a fixed interval. In this embodiment, one frame is extracted every 5 seconds, but in some other embodiments, other fixed intervals can be used, such as 10 seconds, 3 seconds, etc. In addition, in some other embodiments, a dynamic interval can also be used. The specific setting of the dynamic interval can be determined according to the network transmission quality.

[0043] In addition, in all embodiments of the present application, the second segmentation detection model adopts a trained open source model and model weights, and only the first segmentation detection model adopts self-training. The types of targets output by the first segmentation detection model include continuous cigarettes and discontinuous cigarettes, and the types of targets output by the second segmentation detection model include hands. With such a design, there is no need to label samples for the detection of human hands. Unlike the prior art, which requires labeling and training of smoking actions in various postures, and relies on a large number of samples to improve the accuracy of detection, the sample labeling of the present invention only includes two categories: continuous cigarettes and discontinuous cigarettes, which can greatly reduce the interference of various types of smoking actions on the model training process, and can greatly reduce the sample requirements in the model training process. Subsequently, the presence of smoking is judged by the positional relationship of different areas, which is conducive to the direct use of edge computing devices such as drones for computational reasoning, without relying on the network, thereby effectively solving the problem of smoking detection in on-site construction sites with unstable communications.

[0044] The second segmentation detection model may adopt MediaPipe Hands or the YOLO series, etc., and adopt open source weights. For example, when using MediaPipe Hands, since MediaPipe Hands can output the coordinates of 21 hand key points, the area involving these 21 hand key points will be used as the hand area.

[0045] As for the first segmentation detection model, the objects of segmentation detection only include two categories: continuous cigarettes and discontinuous cigarettes.

[0046] Step S2: Obtain the target detection results and the position descriptors of each target output by the segmentation detection model. Since the types of targets output by the first segmentation detection model include continuous cigarettes and discontinuous cigarettes, for its training samples, it is possible to randomly find some cigarette images, randomly cut a part from both sides as a sample of continuous cigarettes, cut a part from the middle as a sample of discontinuous cigarettes, and randomly cut a part from the middle and both sides as a sample of discontinuous cigarettes.

[0047] Since the first segmentation detection model and the second segmentation detection model are based on the same image as input and the segmentation detection processes do not interfere with each other, the two can calculate the position of the recognized results and do not need to be jointly trained, which reduces the training cost.

[0048] Step S3: Detecting whether cigarette butts are present in the target area of ​​each continuous cigarette type. If not, the target area of ​​each continuous cigarette type is used as a smoking detection area. Otherwise, the target area of ​​each continuous cigarette type is expanded and input into the fourth target detection model to detect whether there are fireworks. If so, determining whether there is overlap between the fireworks area and the cigarette area. If so, the target area of ​​each continuous cigarette type is used as a smoking detection area.

[0049] In some embodiments, step S3 is a process of detecting whether there are cigarette butts in the area where a single target of the continuous cigarette type is located, specifically including: inputting the area where the target of the continuous cigarette type is located into a trained third segmentation detection model, obtaining the target detection results and position descriptors of each target output by the third segmentation detection model, and judging whether there are cigarette butts.

[0050] However, in this embodiment, in order to reduce the performance requirements for the terminal side device, such as Figure 3 As shown, step S3 is a process of detecting whether there are cigarette butts in the area where the target of a single type of continuous cigarette is located, specifically including:

[0051] Convolution calculation is performed to realize contour detection. Generally, in the process of performing convolution calculation to realize contour detection, the convolution kernel size used is 3×3;

[0052] Determine the ratio of the length of each contour line to the length of the diagonal line detected in the area where the current target is located one by one as the first ratio of the contour line, and take all contour lines whose first ratio exceeds 60% as the first contour line;

[0053] Determine whether there are two first contour lines that are parallel to each other, and form any two first contour lines that are parallel to each other into a first line pair;

[0054] Determine whether any first contour line in each first line pair has a first ratio exceeding 75%, and if so, use the first line pair as the second line pair;

[0055] Determine whether there is a contour line between the two first contour lines of all second line pairs, which is perpendicular to the two first contour lines and has endpoints located on the two first contour lines respectively. If so, use the contour line that is perpendicular to the two first contour lines and has endpoints located on the two first contour lines respectively as the separating contour line, and calculate the ratio of the lengths of the two first contour lines on both sides of the separating contour line. If the ratio is less than 1, use the ratio as the second ratio of the first contour line; otherwise, use the reciprocal of the ratio as the second ratio of the first contour line.

[0056] Determine whether the second ratio of the two first contour lines in the second line pair or the reciprocal of any of the first contour lines is within the first set interval. If so, determine whether a contour line perpendicular to the first contour line exists at the end of the shorter end of the first contour line and is located between the two first contour lines in the second line pair, and the endpoints are respectively located on the two first contour lines. If so, determine that a cigarette butt exists.

[0057] Through convolution calculation, the boundary line of the cigarette is found, and the presence of cigarette butts is determined by segmentation lines. For the part with cigarette butts, the presence of smoking is further determined by fireworks recognition. For the part without cigarette butts, it must be close to any part of the human body, and thus it is determined that smoking is present. There is no need to rely on image annotation, which can greatly reduce costs and is suitable for deployment on low-computing power edge computing platforms.

[0058] Furthermore, in this embodiment, in the process of expanding the area where the target of the type of continuous cigarettes is located in step S3, the area is expanded by 200%.

[0059] Step S4: Determine whether the target areas of all types of discontinuous cigarettes overlap with the target areas of all types of hands, and determine whether there is a smoking detection area based on the size of the overlapping area. In this embodiment, specifically, Figure 2 Shown, including:

[0060] Step S4-1: Determine whether all discrete cigarette target regions overlap with all hand target regions, and form an identification pair for each pair of discrete cigarette target regions and hand target regions that have overlapping regions.

[0061] Step S4-2: For each recognition pair, calculate the length ratio of the projected line segment of the overlapping area on any straight line to the projected line segment of the target area of ​​the discontinuous cigarette on any straight line, and determine whether the calculated length ratio is greater than a preconfigured first threshold. If so, determine that the target area of ​​the type of discontinuous cigarette in the recognition pair is the smoking detection area, and delete other untraversed recognition pairs that contain the target area of ​​the type of discontinuous cigarette in the recognition pair.

[0062] For discontinuous cigarettes, the presence of smoking is determined by the ratio of the hand's projection line length that intersects the cigarette area. On the one hand, this eliminates the need to rely on image annotation, which can greatly reduce costs. On the other hand, it can solve the problem of missed detection caused by the hand not completely cutting off the cigarette rod due to shooting angle problems.

[0063] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. A smoking detection method based on machine vision, characterized in that: include: Step S1: Acquire a scene image and input it into the trained first segmentation detection model and the second segmentation detection model respectively; Step S2: Obtaining target detection results and location descriptors of each target output by the segmentation detection model, wherein the target types output by the first segmentation detection model include continuous cigarettes and discontinuous cigarettes, and the target types output by the second segmentation detection model include hands; Step S3: Detecting whether cigarette butts are present in the target area of ​​each continuous cigarette type. If not, the target area of ​​each continuous cigarette type is used as a smoking detection area. Otherwise, the target area of ​​each continuous cigarette type is expanded and input into the fourth target detection model to detect whether there are fireworks. If so, determining whether there is overlap between the fireworks area and the cigarette area. If so, the target area of ​​each continuous cigarette type is used as a smoking detection area. Step S4: Determine whether the target areas of all types of discontinuous cigarettes overlap with the target areas of all types of hands, and determine whether there is a smoking detection area based on the size of the overlapping area.

2. The method for detecting smoking based on machine vision according to claim 1, characterized in that: The live image is obtained by extracting a preconfigured frame interval based on the live video.

3. The method for smoking detection based on machine vision according to claim 1, characterized in that: The step S4 comprises: Step S4-1: Determine whether all discrete cigarette target regions overlap with all hand target regions, and form an identification pair for each pair of discrete cigarette target regions and hand target regions that have overlapping regions. Step S4-2: For each recognition pair, calculate the length ratio of the projected line segment of the overlapping area on any straight line to the projected line segment of the target area of ​​the discontinuous cigarette on any straight line, and determine whether the calculated length ratio is greater than a preconfigured first threshold. If so, determine that the target area of ​​the type of discontinuous cigarette in the recognition pair is the smoking detection area, and delete other untraversed recognition pairs that contain the target area of ​​the type of discontinuous cigarette in the recognition pair.

4. The method for detecting smoking based on machine vision according to claim 1, wherein: The second segmentation detection model uses a trained open source model and model weights, and the first segmentation detection model uses self-training.

5. The method for smoking detection based on machine vision according to claim 1, characterized in that: The process of step S3 detecting whether there is a cigarette butt in the area where the target of a single continuous cigarette is located specifically includes: Perform convolution calculation to achieve contour detection; Determine the ratio of the length of each contour line to the length of the diagonal line detected in the area where the current target is located one by one as the first ratio of the contour line, and take all contour lines whose first ratio exceeds 60% as the first contour line; Determine whether there are two first contour lines that are parallel to each other, and form any two first contour lines that are parallel to each other into a first line pair; Determine whether any first contour line in each first line pair has a first ratio exceeding 75%, and if so, use the first line pair as the second line pair; Determine whether there is a contour line between the two first contour lines of all second line pairs, which is perpendicular to the two first contour lines and has endpoints located on the two first contour lines respectively. If so, use the contour line that is perpendicular to the two first contour lines and has endpoints located on the two first contour lines respectively as the separating contour line, and calculate the ratio of the lengths of the two first contour lines on both sides of the separating contour line. If the ratio is less than 1, use the ratio as the second ratio of the first contour line; otherwise, use the reciprocal of the ratio as the second ratio of the first contour line. Determine whether the second ratio of the two first contour lines in the second line pair or the reciprocal of any of the first contour lines is within the first set interval. If so, determine whether a contour line perpendicular to the first contour line exists at the end of the shorter end of the first contour line and is located between the two first contour lines in the second line pair, and the endpoints are respectively located on the two first contour lines. If so, determine that a cigarette butt exists.

6. The method for smoking detection based on machine vision according to claim 1, characterized in that: The process of step S3 detecting whether there is a cigarette butt in the area where the target of a single continuous cigarette is located specifically includes: The area where the target type of continuous cigarettes is located is input into the trained third segmentation detection model, and the target detection results and position descriptors of each target output by the third segmentation detection model are obtained to determine whether there are cigarette butts.

7. The method for smoking detection based on machine vision according to claim 1, characterized in that: In the process of expanding the area where the target of the type of continuous cigarettes is located in step S3, the area is expanded by 200%.

8. The method for detecting smoking based on machine vision according to claim 5, characterized in that: In the process of performing convolution calculation to realize contour detection, the convolution kernel size used is 3×3.

9. A smoking detection device based on machine vision, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images