Smoke and fire detection method, device, equipment, medium and product

Abstract

The invention relates to the technical field of safety supervision, and discloses a smoke and fire detection method, device and equipment, a medium and a product, and a smoke and fire detection model is utilized to analyze each frame of picture so as to determine a smoke and fire detection frame contained in each frame of picture. When the overlapping degree or the similarity of the smoke and fire detection frames contained in the multiple frames of adjacent pictures is greater than or equal to a set first threshold value, outputting a detection result that no smoke and fire exists; and when the overlapping degree and the similarity of the smoke and fire detection frames contained in the multiple frames of adjacent pictures are both smaller than a set first threshold value, outputting a detection result of existence of smoke and fire. And when the detection result is that the smoke exists, an alarm mechanism is triggered. After the smoke and fire detection frame is identified by using the smoke and fire detection model, by combining the change condition of the overlapping degree or the similarity of the smoke and fire detection frames contained in the multiple frames of adjacent pictures, the situation that the smoke and fire detection frame actually does not have the smoke and fire misjudgment situation can be effectively identified, the smoke and fire false alarm situation is reduced, and the smoke and fire detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of safety supervision technology, and in particular to a smoke and fire detection method, device, equipment, medium and product. Background Art

[0002] Teladian's artificial intelligence (AI) device AIBOX can achieve 360° full-scene coverage of charging stations, perform big data simulation on all fireworks, smoking postures, people falling to the ground, etc., and realize safety supervision in unattended scenarios.

[0003] After the fireworks detection algorithm in AIBOX was online for a period of time, it was found that it was affected by weather, lighting, shadows, etc., resulting in some false alarms, which would interfere with the judgment results.

[0004] It can be seen that how to reduce the occurrence of false alarms of fireworks is a problem that those skilled in the art need to solve. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method, device, equipment, medium and product for detecting fireworks, which can reduce the occurrence of false alarms of fireworks.

[0006] In order to solve the above technical problems, an embodiment of the present invention provides a smoke and fire detection method, comprising:

[0007] The fireworks detection model is used to analyze each frame of the image to determine the fireworks detection frame contained in each frame of the image; wherein each frame of the image is an image corresponding to the station to be monitored;

[0008] Determine whether the overlap or similarity of the fireworks detection frames contained in the multiple adjacent frames is greater than or equal to a set first threshold; wherein the value of the first threshold is set based on the change of the fireworks combustion process;

[0009] When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames of the image is greater than or equal to a set first threshold, outputting a detection result that no fireworks exist;

[0010] When the overlap and similarity of the fireworks detection frames included in the multiple adjacent frames of the image are both less than a set first threshold, outputting a detection result indicating the presence of fireworks;

[0011] When the detection result is that fireworks exist, an alarm mechanism is triggered.

[0012] On the one hand, it also includes:

[0013] When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames is greater than or equal to the set first threshold, determining whether the number of times the fireworks detection frames exist in the multiple consecutive frames within a set time is greater than or equal to the set second threshold;

[0014] When the number of times that the fireworks detection frame exists in a plurality of consecutive frames of images within a set time is less than a set second threshold, outputting a detection result that no fireworks exist;

[0015] When the number of times that the fireworks detection frame exists in a plurality of consecutive frames of images within a set time is greater than or equal to a set second threshold, a detection result of the existence of fireworks is output.

[0016] On the one hand, it also includes:

[0017] When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames of the image is greater than or equal to a set first threshold, determining whether the fireworks detection frames included in each frame of the image have an intersection with the station area of ​​the station to be monitored;

[0018] When there is no intersection between the fireworks detection frame included in each frame of the image and the station area of ​​the station to be monitored, outputting a detection result that no fireworks exist;

[0019] When the fireworks detection frame included in each frame of the image intersects with the station area of ​​the station to be monitored, a detection result indicating the presence of fireworks is output.

[0020] On the one hand, it also includes:

[0021] When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames is greater than or equal to a set first threshold, determining whether the proportion of the fireworks detection frames in the corresponding frames falls within a set proportion range;

[0022] When the proportion of the fireworks detection frame in its corresponding picture does not fall within the set proportion range, outputting a detection result that no fireworks exist;

[0023] When the proportion of the fireworks detection frame in its corresponding image falls within a set proportion range, a detection result indicating the presence of fireworks is output.

[0024] On the one hand, it also includes:

[0025] When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames is greater than or equal to a set first threshold, the image corresponding to the fireworks detection frame is classified and identified using a binary classification model to determine the probability value of the fireworks detection frame belonging to smoke or fire; wherein the binary classification model is trained based on images with labeled fireworks detection frames, and the trained convolutional neural network model is obtained by optimizing the model weights through back propagation;

[0026] When the probability values ​​of the fireworks detection frame belonging to smoke or fire are both less than a set third threshold, outputting a detection result that no fireworks exist;

[0027] When the probability value of the smoke and fire detection frame belonging to smoke or fire is greater than or equal to a set third threshold, a detection result indicating the presence of smoke and fire is output.

[0028] On the one hand, it also includes:

[0029] When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames of the image is greater than or equal to a set first threshold, determining whether the fireworks detection frame is in the rectangular area where the vehicle is located;

[0030] When the fireworks detection frame is in the rectangular area where the vehicle is located, outputting a detection result that no fireworks exist;

[0031] When the fireworks detection frame is not located in the rectangular area where the vehicle is located, a detection result indicating the presence of fireworks is output.

[0032] On the one hand, it also includes:

[0033] When the detection result is that fireworks are confirmed to exist, the detection result of the presence of fireworks in the station to be monitored is reported, and the fire extinguishing equipment is triggered to extinguish the fire at the position corresponding to the fireworks detection frame.

[0034] The embodiment of the present invention further provides a smoke and fire detection device, comprising an analysis unit, a judgment unit, a first output unit, a second output unit and an alarm unit;

[0035] The analysis unit is used to analyze each frame of the picture using a fireworks detection model to determine a fireworks detection frame contained in each frame of the picture; wherein each frame of the picture is a picture corresponding to the station to be monitored;

[0036] The judging unit is used to judge whether the overlap or similarity of the fireworks detection frames contained in the multiple adjacent frames is greater than or equal to a set first threshold; wherein the value of the first threshold is set based on the change of the fireworks combustion process;

[0037] The first output unit is configured to output a detection result indicating that no fireworks exist when the overlap or similarity of fireworks detection frames included in the plurality of adjacent frames of images is greater than or equal to a set first threshold;

[0038] The second output unit is used to output a detection result indicating the presence of fireworks when the overlap and similarity of the fireworks detection frames included in the multiple adjacent frames of the image are both less than a set first threshold;

[0039] The alarm unit is used to trigger an alarm mechanism when the detection result shows that fireworks exist.

[0040] On the one hand, it also includes a number judgment unit;

[0041] The number determination unit is used to determine whether the number of times that the fireworks detection frame exists in multiple consecutive frames of pictures within a set time is greater than or equal to a set second threshold when the overlap or similarity of the fireworks detection frames included in multiple adjacent frames of pictures is greater than or equal to a set first threshold;

[0042] The first output unit is further configured to output a detection result that no fireworks exist when the number of times that fireworks detection frames exist in a plurality of consecutive frames of images within a set time is less than a set second threshold;

[0043] The second output unit is further configured to output a detection result of the presence of fireworks when the number of times that fireworks detection frames exist in a plurality of consecutive frames of images within a set time is greater than or equal to a set second threshold.

[0044] On the one hand, it also includes an intersection judgment unit;

[0045] The intersection judgment unit is used to judge whether there is an intersection between the smoke and fire detection frame contained in each frame of the picture and the station area of ​​the station to be monitored when the overlap or similarity of the smoke and fire detection frames contained in multiple adjacent frames of the picture is greater than or equal to a set first threshold;

[0046] The first output unit is further configured to output a detection result indicating that no fireworks exist when no fireworks detection frame included in each frame of the image has any intersection with the station area of ​​the station to be monitored;

[0047] The second output unit is further configured to output a detection result indicating the presence of fireworks when a fireworks detection frame included in each frame of the image intersects with a station area of ​​the station to be monitored.

[0048] On the one hand, it also includes a proportion judgment unit;

[0049] The proportion judgment unit is used to judge whether the proportion of the fireworks detection frame in its corresponding picture falls within a set proportion range when the overlap or similarity of the fireworks detection frames included in the multiple adjacent pictures is greater than or equal to a set first threshold;

[0050] The first output unit is further configured to output a detection result indicating that there is no fireworks when the proportion of the fireworks detection frame in its corresponding picture does not fall within a set proportion range;

[0051] The second output unit is further configured to output a detection result indicating the presence of fireworks when the proportion of the fireworks detection frame in its corresponding image falls within a set proportion range.

[0052] On the one hand, it also includes an identification unit;

[0053] The recognition unit is used to classify and recognize the image corresponding to the fireworks detection frame using a binary classification model when the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames is greater than or equal to a set first threshold, so as to determine the probability value of the fireworks detection frame belonging to smoke or fire; wherein the binary classification model is trained based on the image with the fireworks detection frame labeled, and the trained convolutional neural network model is obtained by optimizing the model weights through back propagation;

[0054] The first output unit is further configured to output a detection result indicating that there is no smoke or fire when the probability values ​​of the smoke and fire detection frame belonging to smoke or fire are both less than a set third threshold;

[0055] The second output unit is further configured to output a detection result indicating the presence of fireworks when the probability value of the fireworks detection frame belonging to smoke or fire is greater than or equal to a set third threshold.

[0056] On the one hand, it also includes a region judgment unit;

[0057] The area determination unit is used to determine whether the fireworks detection frame is in the rectangular area where the vehicle is located when the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames of the image is greater than or equal to a set first threshold;

[0058] The first output unit is further configured to output a detection result indicating that there is no smoke or fire when the smoke or fire detection frame is in the rectangular area where the vehicle is located;

[0059] The second output unit is further configured to output a detection result indicating the presence of fireworks when the fireworks detection frame is not located in the rectangular area where the vehicle is located.

[0060] On the one hand, it also includes reporting units and fire extinguishing units;

[0061] The reporting unit is used to report the detection result of the presence of fireworks at the station to be monitored when the detection result is that fireworks are determined to exist;

[0062] The fire extinguishing unit is used to trigger the fire extinguishing equipment to extinguish the fire at the position corresponding to the smoke and fire detection frame.

[0063] The embodiment of the present invention further provides a smoke and fire detection device, comprising:

[0064] Memory for storing computer programs;

[0065] A processor is used to execute the computer program to implement the steps of the above-mentioned smoke and fire detection method.

[0066] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned smoke and fire detection method are implemented.

[0067] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned smoke and fire detection method when executed by a processor.

[0068] It can be seen from the above technical solution that the fireworks detection model is used to analyze each frame of the picture to determine the fireworks detection frame contained in each frame of the picture; wherein each frame of the picture is a picture corresponding to the station to be monitored. The accuracy of the fireworks detection model in detecting fireworks is limited. Considering that fireworks will change dynamically, in order to reduce the occurrence of fireworks misjudgment, a first threshold can be set based on the change of the fireworks combustion process to determine whether the overlap or similarity of the fireworks detection frames contained in multiple adjacent frames of pictures is greater than or equal to the set first threshold; when the overlap or similarity of the fireworks detection frames contained in multiple adjacent frames of pictures is greater than or equal to the set first threshold, the detection result of no fireworks is output. When the overlap and similarity of the fireworks detection frames contained in multiple adjacent frames of pictures are both less than the set first threshold, the detection result of the presence of fireworks is output. When the detection result is that fireworks exist, an alarm mechanism can be triggered so that fire extinguishing measures can be taken in time. In this technical solution, after the fireworks detection frame is identified using the fireworks detection model, it is further determined whether the overlap or similarity of the fireworks detection frames contained in multiple adjacent frames of images is greater than or equal to a set first threshold. This can effectively identify the misjudgment that there is actually no fireworks in the fireworks detection frame, reduce the occurrence of false alarms of fireworks, and improve the accuracy of fireworks detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0070] FIG1 is a flow chart of a method for detecting fireworks provided by an embodiment of the present invention;

[0071] FIG2 is a flow chart of a method for detecting a fireworks detection frame contained in each frame of an image provided by an embodiment of the present invention;

[0072] FIG3 is a schematic structural diagram of a smoke and fire detection device provided by an embodiment of the present invention;

[0073] FIG. 4 is a structural diagram of a smoke and fire detection device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0076] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0077] Next, a method for detecting fireworks provided by an embodiment of the present invention is described in detail. FIG1 is a flow chart of a method for detecting fireworks provided by an embodiment of the present invention, the method comprising:

[0078] S101: Analyze each frame of the image using a fireworks detection model to determine a fireworks detection frame included in each frame of the image.

[0079] Among them, each frame of the picture is a picture corresponding to the station to be monitored.

[0080] The smoke and fire detection method provided by the embodiment of the present invention can be applied to a variety of scenarios, such as unmanned scenarios, scenarios with high safety management requirements such as airports, energy sources, and charging stations with dense populations, scenarios where site management is often out of control and there are many safety hazards, and scenarios that require the deployment of full-time management personnel.

[0081] Cameras can be deployed at the stations to be monitored, and the cameras can be used to take photos and monitor the stations to be monitored.

[0082] In practical applications, a fireworks detection model can be built based on the Yolov5 algorithm to analyze images, identify whether there are fireworks in the image, and mark the rectangular box where the fireworks are located. The rectangular box where the fireworks are located can be called a fireworks detection box.

[0083] S102: Determine whether the overlap or similarity of the fireworks detection frames included in multiple adjacent frames of images is greater than or equal to a set first threshold.

[0084] Fireworks can include two types: fire and smoke. The accuracy of the firework detection model in detecting firework is limited. Considering that fireworks often undergo dynamic changes during the burning process, in order to evaluate the change of fireworks, in the embodiment of the present invention, the overlap or similarity of the firework detection frames contained in multiple adjacent frames of images can be analyzed.

[0085] Overlap refers to the intersection over union (IOU) of the fire detection frames of consecutive frames. Taking any two frames as an example, the overlap can be the ratio of the intersection area of ​​the fire detection frames of the two frames to their union.

[0086] Similarity refers to the similarity of local features of two fireworks detection frames, such as the similarity of local features such as edge, texture, color, etc. The similarity can be represented by the Structural Similarity Index (SSIM).

[0087] The basic calculation steps of SSIM are as follows: Brightness comparison: Calculate the similarity of local brightness of two images. Contrast comparison: Calculate the similarity of local contrast of two images. Structure comparison: Calculate the similarity of local structure of two images, usually by calculating covariance.

[0088] The value of the first threshold can be set based on the change of the fireworks combustion process, for example, the value of the first threshold can be set to 90%. When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames is greater than or equal to the set first threshold, it means that the fireworks have not changed significantly and do not conform to the change of fireworks during actual combustion. At this time, a detection result that no fireworks exist can be output, that is, the operation of S103 is executed.

[0089] When the overlap and similarity of the fireworks detection frames included in multiple adjacent frames of images are both less than the set first threshold, it indicates that the fireworks change in accordance with actual combustion. In this case, a fireworks detection result may be output, i.e., the operation of S104 is executed.

[0090] S103: Outputting the detection result that there is no fireworks.

[0091] When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames of images is greater than or equal to a set first threshold, a detection result indicating that fireworks do not exist may be output.

[0092] S104: Outputting the detection result of the presence of fireworks.

[0093] When the overlap and similarity of the fireworks detection frames included in multiple adjacent frames of images are both less than a set first threshold, a detection result indicating the presence of fireworks may be output.

[0094] S105: When the detection result shows that fireworks exist, an alarm mechanism is triggered.

[0095] If the detection result shows the presence of fireworks, the alarm mechanism can be triggered so that the station management personnel can take fire-fighting measures in time.

[0096] In addition to issuing an alarm, when the detection result shows that there is fireworks, the detection result of the presence of fireworks at the monitored site can be reported, and the fire extinguishing equipment can be triggered to extinguish the fire at the position corresponding to the fireworks detection frame.

[0097] By reporting the detection results, the occurrence of fireworks in the monitored stations can be recorded in time, which is convenient for the back-end management personnel to understand the situation of the monitored stations. By triggering the fire extinguishing equipment to extinguish the fire at the position corresponding to the fireworks detection frame, the impact caused by fireworks can be effectively avoided and the safety of the monitored stations can be ensured.

[0098] It can be seen from the above technical solution that the fireworks detection model is used to analyze each frame of the picture to determine the fireworks detection frame contained in each frame of the picture; wherein each frame of the picture is a picture corresponding to the station to be monitored. The accuracy of the fireworks detection model in detecting fireworks is limited. Considering that fireworks will change dynamically, in order to reduce the occurrence of fireworks misjudgment, a first threshold can be set based on the change of the fireworks combustion process to determine whether the overlap or similarity of the fireworks detection frames contained in multiple adjacent frames of pictures is greater than or equal to the set first threshold; when the overlap or similarity of the fireworks detection frames contained in multiple adjacent frames of pictures is greater than or equal to the set first threshold, the detection result of no fireworks is output. When the overlap and similarity of the fireworks detection frames contained in multiple adjacent frames of pictures are both less than the set first threshold, the detection result of the presence of fireworks is output. When the detection result is that fireworks exist, an alarm mechanism can be triggered so that fire extinguishing measures can be taken in time. In this technical solution, after the fireworks detection frame is identified using the fireworks detection model, it is further determined whether the overlap or similarity of the fireworks detection frames contained in multiple adjacent frames of images is greater than or equal to a set first threshold. This can effectively identify the misjudgment that there is actually no fireworks in the fireworks detection frame, reduce the occurrence of false alarms of fireworks, and improve the accuracy of fireworks detection.

[0099] Considering the actual scenarios, there are often false alarms of fireworks. For example, station lights, car lights, reflected light, etc. are mistakenly identified as flames, and the presence of smoke in the station is mistakenly identified due to camera obstruction or weather reasons.

[0100] In order to reduce the occurrence of these false alarms, in an embodiment of the present invention, the image corresponding to the fireworks detection frame may be further analyzed to eliminate possible false alarms, thereby obtaining a more accurate detection result.

[0101] Considering that the location distribution of fireworks has certain characteristics and fireworks will also change dynamically, in order to reduce the occurrence of misjudgment of fireworks, the location distribution of the fireworks detection frame and the change of fireworks can be detected. Among them, the location distribution can include the location of the fireworks in the station, the proportion in the picture, the relationship with the location of the vehicle, etc. The change of fireworks can be analyzed based on multiple frames of pictures of the station to be monitored to see whether there are dynamic changes in the fireworks.

[0102] FIG1 shows an analysis of the overlap or similarity of fireworks detection frames included in multiple frames of adjacent images. In an embodiment of the present invention, in addition to determining whether the overlap or similarity of fireworks detection frames included in multiple frames of adjacent images is greater than or equal to a set first threshold as mentioned in FIG1 , a plurality of methods for identifying fireworks false alarms may also be set, and the fireworks detection frames may be detected in turn based on the plurality of methods for fireworks false alarms. When any method identifies the presence of fireworks false alarms, a detection result indicating the absence of fireworks may be output.

[0103] FIG2 is a flow chart of a method for detecting a fireworks detection frame contained in each frame of an image provided by an embodiment of the present invention, the method comprising:

[0104] S201: Determine whether the fireworks detection frame included in each frame image has an intersection with the station area of ​​the station to be monitored.

[0105] In practical applications, only fireworks detection can be performed inside the station to be monitored. There is no need to analyze the situation outside the station to be monitored, so it can be determined whether the fireworks detection frame contained in each frame image has an intersection with the station area of ​​the station to be monitored.

[0106] When there is no intersection between the fireworks detection frame contained in each frame image and the station area of ​​the station to be monitored, it means that the fireworks detection frame does not appear in the station to be monitored. At this time, a detection result that no fireworks exist can be output, that is, the operation of S209 is executed.

[0107] When the fireworks detection frame included in each frame image intersects with the station area of ​​the station to be monitored, it means that there may be fireworks in the station to be monitored. At this time, whether there are actually fireworks in the fireworks detection frame can be further analyzed, and the operation of S202 can be executed.

[0108] S202: Determine whether the proportion of the fireworks detection frame in its corresponding image falls within a set proportion range.

[0109] In actual scenarios, the area of ​​fireworks in the picture often occupies a certain proportion of the picture. If the fireworks detection frame occupies the entire picture, it is very likely that the camera is blocked and caused the fireworks to be misjudged. If the fireworks detection frame occupies too small a proportion of the picture, it is very likely that the plastic bags in the monitored station caused the misjudgment.

[0110] Therefore, in the embodiment of the present invention, a proportion range may be set for the proportion of the fireworks detection frame in the image, for example, it may be set to 1% to 97%.

[0111] When the proportion of the fireworks detection frame in its corresponding image does not fall within the set proportion range, it means that a misjudgment is very likely to occur. In this case, a detection result indicating that no fireworks exist may be output, that is, the operation of S209 is executed.

[0112] When the proportion of the fireworks detection frame in its corresponding image is within the set proportion range, it indicates that there may be fireworks in the monitored site. At this time, whether there are actually fireworks in the fireworks detection frame can be further analyzed, and the operation of S203 can be executed.

[0113] S203: Using a binary classification model to classify and identify the image corresponding to the smoke and fire detection frame to determine a probability value of whether the smoke and fire detection frame belongs to smoke or fire.

[0114] Among them, the binary classification model is trained based on images with labeled fireworks detection frames, and the trained convolutional neural network model is obtained by optimizing the model weights through back propagation.

[0115] In an embodiment of the present invention, a binary classification model can be constructed based on EfficientNetV2. EfficientNetV2 is a convolutional neural network (CNN) architecture that extracts local features such as edges, textures, and location information from the image through the convolution layer to understand the layout of the target in the image. The weights are then optimized through the back-propagation algorithm, and the output confidence is actually the confidence output by the final softmax function, which converts the original score into a probability distribution. For example, the output of a binary classification is: [0.8, 0.2].

[0116] Among them, the back propagation algorithm is done when training the model. The model needs to be trained in advance, and back propagation is not involved in actual reasoning. When training the model, the labeled data will be divided, generally 80% is divided into a training set and 10% is a validation set. 10% is divided into a test set. The labeled data can contain specific categories. For example, the picture contains two fireworks detection boxes, which are manually identified and labeled. For example, label 1 indicates the presence of flames and label 2 indicates the presence of smoke.

[0117] The implementation process of the binary classification model includes: inputting the image corresponding to the fireworks detection frame into the convolution layer to extract features; using the pooling layer to compress the features extracted by the convolution layer to obtain more important features. Then, the highly abstract features after multiple convolutions are fused through the fully connected layer, and then normalized, and each classification situation corresponds to an output.

[0118] The output of the fully connected layer is usually fed into the softmax function, which converts the raw scores of each category into probability values, where the sum of the probabilities of all categories is 1. These probability values ​​represent the model's prediction confidence for each category. The confidence threshold is generally greater than 0.5, which means that the probability must be at least greater than 50%. To balance false positives and false negatives, the empirical value is generally between 0.6 and 0.7; confidence greater than or equal to the threshold is retained, and confidence less than the threshold is considered a misjudgment.

[0119] S204: Determine whether the probability value of the smoke and fire detection frame belonging to smoke or fire is greater than or equal to a set third threshold.

[0120] Fireworks can include two types: smoke and flame. When the probability value of the firework detection frame belonging to smoke or fire is less than the set third threshold, it means that there is a high probability of firework in the monitored station. At this time, the detection result of no firework can be output, that is, the operation of S209 is executed.

[0121] When the probability value that the smoke and fire detection frame belongs to smoke or fire is greater than or equal to the set third threshold, whether there is smoke and fire in the smoke and fire detection frame can be further analyzed, and the operation of S205 can be executed.

[0122] In the embodiment of the present invention, there is no limitation on the value of the third threshold. For example, a value between 0.6 and 0.7 can be selected as the third threshold.

[0123] S205: Determine whether the overlap or similarity of the fireworks detection frames included in multiple adjacent frames of images is greater than or equal to a set first threshold.

[0124] When the overlap or similarity of the fireworks detection frames contained in multiple adjacent frames of images is greater than or equal to the set first threshold, it means that the fireworks have not changed significantly and do not conform to the changes of fireworks during actual burning. At this time, a detection result indicating that no fireworks exist can be output, that is, operation S209 is executed.

[0125] When the overlap and similarity of the fireworks detection frames included in the multiple adjacent frames of images are both less than the set first threshold, it is possible to further analyze whether fireworks actually exist in the fireworks detection frames, and perform the operation of S206.

[0126] S206: Determine whether the fireworks detection frame is within the rectangular area where the vehicle is located.

[0127] Considering the actual application, if the fireworks detection frame is completely within the rectangular area of ​​a car in the picture, it means that the vehicle lights or front windshield may have caused misidentification.

[0128] Therefore, when the smoke and fire detection frame is in the rectangular area where the vehicle is located, a detection result indicating that no smoke and fire exist may be output, that is, the operation of S209 is executed.

[0129] When the smoke and fire detection frame is not in the rectangular area where the vehicle is located, further analysis may be performed on whether smoke and fire actually exist in the smoke and fire detection frame, and the operation of S207 may be performed.

[0130] S207: Determine whether the number of times that a fireworks detection frame exists in a plurality of consecutive frames of images within a set time is greater than or equal to a set second threshold.

[0131] The value of the set time can be set flexibly, for example, it can be set to 30 seconds. In practical applications, continuous multiple frames of images within 30 seconds can be obtained.

[0132] If the number of times that a fireworks detection frame exists in multiple consecutive frames of images within a set time is less than the set second threshold, it means that the fireworks warning only appeared briefly. However, real smoke and flames are generally continuous events and will not disappear after a short appearance. Therefore, if the number of times that a fireworks detection frame exists in multiple consecutive frames of images within a set time is less than the set second threshold, a detection result that no fireworks exist can be output, that is, the operation of S209 is executed. If the number of times that a fireworks detection frame exists in multiple consecutive frames of images within a set time is greater than or equal to the set second threshold, a detection result that fireworks exist can be output, that is, the operation of S208 is executed.

[0133] S208: Outputting the detection result of the presence of fireworks.

[0134] S209: Outputting the detection result that there is no fireworks.

[0135] It should be noted that the above S201 to S207 introduce six different ways of identifying false alarms of fireworks, and FIG. 2 is only a combination of these six ways. In the embodiment of the present invention, the order of the judgment of these six ways is not limited. In addition to the combination of these six ways, different ways can be selected from these six ways to be combined to detect whether there are real fireworks in the fireworks detection frame.

[0136] 3 is a schematic diagram of the structure of a smoke and fire detection device provided by an embodiment of the present invention, comprising an analysis unit 31, a judgment unit 32, a first output unit 33, a second output unit 34 and an alarm unit 35;

[0137] The analysis unit 31 is used to analyze each frame of the picture using a smoke and fire detection model to determine the smoke and fire detection frame contained in each frame of the picture; wherein each frame of the picture is a picture corresponding to the station to be monitored;

[0138] The judging unit 32 is used to judge whether the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames is greater than or equal to a set first threshold; wherein the value of the first threshold is set based on the change of the fireworks combustion process;

[0139] A first output unit 33 is used to output a detection result that no fireworks exist when the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames of the image is greater than or equal to a set first threshold;

[0140] The second output unit 34 is used to output a detection result indicating the presence of fireworks when the overlap and similarity of the fireworks detection frames included in the multiple adjacent frames of the image are both less than a set first threshold;

[0141] The alarm unit 35 is used to trigger an alarm mechanism when the detection result shows that there are fireworks.

[0142] In some embodiments, it further includes a number determination unit;

[0143] A number determination unit, configured to determine whether the number of times that a fireworks detection frame exists in a plurality of consecutive frames of images within a set time is greater than or equal to a set second threshold value when the overlap or similarity of the fireworks detection frames included in the plurality of adjacent frames of images is greater than or equal to a set first threshold value;

[0144] The first output unit is further configured to output a detection result that no fireworks exist when the number of times that fireworks detection frames exist in a plurality of consecutive frames of images within a set time is less than a set second threshold;

[0145] The second output unit is further configured to output a detection result of the presence of fireworks when the number of times that fireworks detection frames exist in a plurality of consecutive frames of images within a set time is greater than or equal to a set second threshold.

[0146] In some embodiments, an intersection determination unit is also included;

[0147] An intersection judgment unit, used to judge whether there is an intersection between the smoke and fire detection frame contained in each frame of the picture and the station area of ​​the station to be monitored when the overlap or similarity of the smoke and fire detection frames contained in multiple adjacent frames of the picture is greater than or equal to a set first threshold;

[0148] The first output unit is further configured to output a detection result indicating that no fireworks exist when no fireworks detection frame included in each frame of the image has any intersection with the station area of ​​the station to be monitored;

[0149] The second output unit is further configured to output a detection result indicating the presence of fireworks when the fireworks detection frame included in each frame of the image intersects with the station area of ​​the station to be monitored.

[0150] In some embodiments, a proportion determination unit is further included;

[0151] A proportion judgment unit, used to judge whether the proportion of the fireworks detection frame in its corresponding picture falls within a set proportion range when the overlap or similarity of the fireworks detection frames included in the multiple adjacent pictures is greater than or equal to a set first threshold;

[0152] The first output unit is further configured to output a detection result indicating that no fireworks exist when the proportion of the fireworks detection frame in its corresponding image does not fall within a set proportion range;

[0153] The second output unit is further configured to output a detection result indicating the presence of fireworks when the proportion of the fireworks detection frame in its corresponding image falls within a set proportion range.

[0154] In some embodiments, further comprising an identification unit;

[0155] The recognition unit is used to classify and recognize the image corresponding to the fireworks detection frame using a binary classification model when the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames is greater than or equal to a set first threshold, so as to determine the probability value of the fireworks detection frame belonging to smoke or fire; wherein the binary classification model is trained based on the image with the fireworks detection frame labeled, and the trained convolutional neural network model is obtained by optimizing the model weights through back propagation;

[0156] The first output unit is further configured to output a detection result indicating that no fireworks exist when the probability values ​​of the fireworks detection frame belonging to smoke or fire are both less than a set third threshold value;

[0157] The second output unit is further configured to output a detection result indicating the presence of fireworks when the probability value of the fireworks detection frame belonging to smoke or fire is greater than or equal to a set third threshold.

[0158] In some embodiments, a region determination unit is also included;

[0159] An area judgment unit, configured to judge whether the smoke and fire detection frame is in the rectangular area where the vehicle is located when the overlap or similarity of the smoke and fire detection frames included in the plurality of adjacent frames of images is greater than or equal to a set first threshold;

[0160] The first output unit is also used to output a detection result that no fireworks exist when the fireworks detection frame is in the rectangular area where the vehicle is located;

[0161] The second output unit is further configured to output a detection result indicating the presence of fireworks when the fireworks detection frame is not located in the rectangular area where the vehicle is located.

[0162] In some embodiments, a reporting unit and a fire extinguishing unit are also included;

[0163] A reporting unit, used for reporting the detection result of the presence of fireworks to the monitored station when the detection result is that fireworks are confirmed to exist;

[0164] The fire extinguishing unit is used to trigger the fire extinguishing equipment to extinguish the fire at the position corresponding to the smoke and fire detection frame.

[0165] For the description of the features in the embodiment corresponding to FIG. 3 , reference can be made to the relevant description of the embodiment corresponding to FIG. 1 , and they will not be described one by one here.

[0166] It can be seen from the above technical solution that the fireworks detection model is used to analyze each frame of the picture to determine the fireworks detection frame contained in each frame of the picture; wherein each frame of the picture is a picture corresponding to the station to be monitored. The accuracy of the fireworks detection model in detecting fireworks is limited. Considering that fireworks will change dynamically, in order to reduce the occurrence of fireworks misjudgment, a first threshold can be set based on the change of the fireworks combustion process to determine whether the overlap or similarity of the fireworks detection frames contained in multiple adjacent frames of pictures is greater than or equal to the set first threshold; when the overlap or similarity of the fireworks detection frames contained in multiple adjacent frames of pictures is greater than or equal to the set first threshold, the detection result of no fireworks is output. When the overlap and similarity of the fireworks detection frames contained in multiple adjacent frames of pictures are both less than the set first threshold, the detection result of the presence of fireworks is output. When the detection result is that fireworks exist, an alarm mechanism can be triggered so that fire extinguishing measures can be taken in time. In this technical solution, after the fireworks detection frame is identified using the fireworks detection model, it is further determined whether the overlap or similarity of the fireworks detection frames contained in multiple adjacent frames of images is greater than or equal to a set first threshold. This can effectively identify the misjudgment that there is actually no fireworks in the fireworks detection frame, reduce the occurrence of false alarms of fireworks, and improve the accuracy of fireworks detection.

[0167] FIG4 is a structural diagram of a smoke and fire detection device provided by an embodiment of the present invention. As shown in FIG4 , the smoke and fire detection device includes: a memory 40 for storing a computer program;

[0168] The processor 41 is used to implement the steps of the smoke and fire detection method in the above embodiment when executing the computer program.

[0169] The smoke and fire detection device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0170] Among them, the processor 41 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 41 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 41 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 41 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 41 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0171] The memory 40 may include one or more computer-readable storage media, which may be non-transitory. The memory 40 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 40 is at least used to store the following computer program 401, wherein, after the computer program is loaded and executed by the processor 41, it can implement the relevant steps of the fireworks detection method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 40 may also include an operating system 402 and data 403, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 402 may include Windows, Unix, Linux, etc. The data 403 may include, but is not limited to, the location distribution of the fireworks detection frame and the changes in the fireworks.

[0172] In some embodiments, the smoke and fire detection device may further include a display screen 42 , an input and output interface 43 , a communication interface 44 , a power supply 45 , and a communication bus 46 .

[0173] Those skilled in the art will appreciate that the structure shown in FIG. 4 does not limit the smoke and fire detection device, and may include more or fewer components than those shown in the figure.

[0174] It is understandable that if the smoke and fire detection method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the current technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0175] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned smoke and fire detection method are implemented.

[0176] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned smoke and fire detection method when executed by a processor.

[0177] The above is a detailed introduction to a method, device, equipment, computer-readable storage medium and computer program product for detecting fireworks provided by the embodiments of the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0178] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0179] The above is a detailed introduction to a method, device, equipment, computer-readable storage medium and computer program product for detecting fireworks provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for detecting fireworks, characterized in that: include: The fireworks detection model is used to analyze each frame of the image to determine the fireworks detection frame contained in each frame of the image; wherein each frame of the image is an image corresponding to the station to be monitored; Determine whether the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames of the image is greater than or equal to a set first threshold; When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames is greater than or equal to a set first threshold, output a detection result that no fireworks exist; wherein the value of the first threshold is set based on the change of the fireworks combustion process; When the overlap and similarity of the fireworks detection frames included in the multiple adjacent frames of the image are both less than a set first threshold, outputting a detection result indicating the presence of fireworks; When the detection result is that fireworks exist, an alarm mechanism is triggered.

2. The method for detecting smoke and fire according to claim 1, characterized in that: Also includes: When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames is greater than or equal to the set first threshold, determining whether the number of times the fireworks detection frames exist in the multiple consecutive frames within a set time is greater than or equal to the set second threshold; When the number of times that the fireworks detection frame exists in a plurality of consecutive frames of images within a set time is less than a set second threshold, outputting a detection result that no fireworks exist; When the number of times that the fireworks detection frame exists in a plurality of consecutive frames of images within a set time is greater than or equal to a set second threshold, a detection result of the existence of fireworks is output.

3. The method for detecting smoke and fire according to claim 1, characterized in that: Also includes: When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames of the image is greater than or equal to a set first threshold, determining whether the fireworks detection frames included in each frame of the image have an intersection with the station area of ​​the station to be monitored; When there is no intersection between the fireworks detection frame included in each frame of the image and the station area of ​​the station to be monitored, outputting a detection result that no fireworks exist; When the fireworks detection frame included in each frame of the image intersects with the station area of ​​the station to be monitored, a detection result indicating the presence of fireworks is output.

4. The method for detecting smoke and fire according to claim 1, characterized in that: Also includes: When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames is greater than or equal to a set first threshold, determining whether the proportion of the fireworks detection frames in the corresponding frames falls within a set proportion range; When the proportion of the fireworks detection frame in its corresponding picture does not fall within the set proportion range, outputting a detection result that no fireworks exist; When the proportion of the fireworks detection frame in its corresponding image falls within a set proportion range, a detection result indicating the presence of fireworks is output.

5. The method for detecting smoke and fire according to claim 1, characterized in that: Also includes: When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames is greater than or equal to a set first threshold, the image corresponding to the fireworks detection frame is classified and identified using a binary classification model to determine the probability value of the fireworks detection frame belonging to smoke or fire; wherein the binary classification model is trained based on images with labeled fireworks detection frames, and the trained convolutional neural network model is obtained by optimizing the model weights through back propagation; When the probability values ​​of the fireworks detection frame belonging to smoke or fire are both less than a set third threshold, outputting a detection result that no fireworks exist; When the probability value of the smoke and fire detection frame belonging to smoke or fire is greater than or equal to a set third threshold, a detection result indicating the presence of smoke and fire is output.

6. The method for detecting smoke and fire according to claim 1, characterized in that: Also includes: When the overlap or similarity of the fireworks detection frames included in the multiple adjacent frames of the image is greater than or equal to a set first threshold, determining whether the fireworks detection frame is in the rectangular area where the vehicle is located; When the fireworks detection frame is in the rectangular area where the vehicle is located, outputting a detection result that no fireworks exist; When the fireworks detection frame is not located in the rectangular area where the vehicle is located, a detection result indicating the presence of fireworks is output.

7. The method for detecting smoke and fire according to any one of claims 1 to 6, characterized in that: Also includes: When the detection result is that fireworks are confirmed to exist, the detection result that fireworks exist in the station to be monitored is reported, and the fire extinguishing equipment is triggered to extinguish the fire at the position corresponding to the fireworks detection frame.

8. A smoke and fire detection device, characterized in that: It includes an analysis unit, a judgment unit, a first output unit, a second output unit and an alarm unit; The analysis unit is used to analyze each frame of the picture using a fireworks detection model to determine a fireworks detection frame contained in each frame of the picture; wherein each frame of the picture is a picture corresponding to the station to be monitored; The judging unit is used to judge whether the overlap or similarity of the fireworks detection frames contained in the multiple adjacent frames is greater than or equal to a set first threshold; wherein the value of the first threshold is set based on the change of the fireworks combustion process; The first output unit is configured to output a detection result indicating that no fireworks exist when the overlap or similarity of fireworks detection frames included in the plurality of adjacent frames of images is greater than or equal to a set first threshold; The second output unit is used to output a detection result indicating the presence of fireworks when the overlap and similarity of the fireworks detection frames included in the multiple adjacent frames of the image are both less than a set first threshold; The alarm unit is used to trigger an alarm mechanism when the detection result shows that fireworks exist.

9. A fire detection device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the smoke and fire detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the smoke and fire detection method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the smoke and fire detection method according to any one of claims 1 to 7 are implemented.

Images