A method for detecting fire smoke, terminal equipment and storage medium

Abstract

This invention relates to a fire smoke detection method, terminal device, and storage medium. The method includes: detecting local features of flame or smoke targets in an image; determining whether each detected feature is stationary in adjacent frames of the image, and filtering out features that are stationary in adjacent frames; fusing features in the image with corresponding features in adjacent frames to obtain fused features in the image; determining whether the bounding boxes of the fused features are in motion, and filtering out fused features corresponding to bounding boxes that are in motion in adjacent frames; and outputting the fused features corresponding to bounding boxes with a size greater than a size threshold in the image. This invention has high detection recall, is sufficiently stable for targets with changing combustion characteristics, and can fully utilize the continuous features of the flame.

Description

Technical Field

[0001] This invention relates to the field of environmental monitoring, and in particular to a method for detecting fire smoke, a terminal device, and a storage medium. Background Technology

[0002] Fire and smoke are among the leading causes of personal injury and property damage. Traditional fire and smoke detection methods typically rely on sensors, smoke detectors, and temperature monitoring devices. However, these methods often require expensive equipment and complex installations, and their effectiveness is limited in detecting fires and smoke over large areas and complex environments, as well as at long distances.

[0003] In recent years, with the rapid development of deep learning technology, significant breakthroughs have been achieved in image recognition and detection methods based on neural networks. Deep learning models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), possess powerful image feature extraction and pattern recognition capabilities, effectively processing information in complex images. Deep learning-based fireworks detection has emerged as a result, but it faces several challenges:

[0004] Smoke and flames have no fixed feature outlines; the shape of smoke and flames changes in real time; smoke and clouds, flames and lights are highly similar and difficult to distinguish; these similar problems are difficult to detect and are unstable, and the false recognition and recall rates are difficult to control. It is very easy to identify lights, especially flowing lights, as flames, and clouds as thick smoke. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a fire smoke detection method, terminal device, and storage medium.

[0006] The specific plan is as follows:

[0007] A fire smoke detection method includes: receiving a video stream to be detected, and processing each frame of the video stream as follows:

[0008] S1: Detect local features of flame or smoke targets in the image;

[0009] S2: Combine the adjacent frame images of the image to determine whether each detected feature is in a static state in the adjacent frame, and filter out the features that are in a static state in the adjacent frame.

[0010] S3: Combine the adjacent frame images of the image, perform feature fusion on each feature in the image and the corresponding features in the adjacent frame images to obtain each fused feature in the image;

[0011] S4: Combining the adjacent frames of the image, determine whether the bounding boxes of each fusion feature of the image are in motion, and filter out the fusion features corresponding to the bounding boxes that are in motion in the adjacent frames.

[0012] S5: Output the fusion features corresponding to the bounding boxes in the image whose size is greater than the size threshold.

[0013] Furthermore, the local features of a flame target are a flame or a flame body; the local features of a smoke target are a smoke cloud.

[0014] Furthermore, before step S2, the method includes: filtering out features whose confidence scores are less than the confidence score threshold for each detected feature.

[0015] Furthermore, before step S2, the method includes: filtering out features whose bounding box size is smaller than the bounding box size threshold for each detected feature.

[0016] Furthermore, the feature fusion process includes the following steps:

[0017] S301: Select the feature with the highest confidence score among all features to be fused as the fusion feature;

[0018] S302: Traverse all other features to be fused, calculate the distance between each feature to be fused and the fused feature, determine whether the distance is less than the distance threshold, if so, fuse the feature to be fused with the fused feature, update the fused feature, until there are no features to be fused with a distance less than the distance threshold, then proceed to S303.

[0019] S303: Select the feature with the highest confidence score among all features to be fused that have a distance greater than or equal to the distance threshold as the new fusion feature, and return to S302.

[0020] Furthermore, during feature fusion, the gain ratio of each fused feature is calculated, and after feature fusion is completed, fused features with a gain ratio less than the gain ratio threshold are filtered out.

[0021] Furthermore, the formula for calculating the gain ratio A of the fused features is:

[0022]

[0023] in, This represents the confidence score of the fusion features before fusion. This represents the confidence score of the features to be fused before fusion. This represents the intersection-union ratio (IURR) of the bounding boxes of two features when they are enlarged by a factor of 1.5, and max(.) represents finding the maximum value.

[0024] A fire smoke detection terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in the embodiments of the present invention.

[0025] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above in the embodiments of the present invention.

[0026] This invention employs the above-mentioned technical solution, utilizing local flame features and a reasonable single-frame / multi-frame fusion method to achieve high detection recall. For targets undergoing combustion changes, detection is sufficiently stable, and the continuous features of the flame are fully utilized. This invention also rationally utilizes two motion filtering schemes to filter out false identifications of static targets without motion information and moving targets, thereby improving the accuracy of the detection results. Attached Figure Description

[0027] Figure 1 The diagram shown is a flowchart of Embodiment 1 of the present invention.

[0028] Figure 2 The diagram shown is a schematic representation of the detection results of local features in this embodiment.

[0029] Figure 3 The diagram shown is a schematic representation of the result after feature fusion in this embodiment. Detailed Implementation

[0030] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments, and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention.

[0031] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0032] Example 1:

[0033] This invention provides a method for detecting fire smoke, such as... Figure 1 As shown, the method includes: receiving a video stream to be detected, and performing the following processing on each frame of the video stream:

[0034] S1: Detect local features of flame or smoke targets in the image.

[0035] S2: Combine the adjacent frames of the image to determine whether each detected feature is stationary in the adjacent frames, and filter out the features that are stationary in the adjacent frames.

[0036] S3: Combine the adjacent frame images of the image, perform feature fusion on each feature in the image and its corresponding features in the adjacent frame images to obtain the fused features in the image.

[0037] S4: Combining the adjacent frames of the image, determine whether the bounding boxes of each fusion feature of the image are in motion, and filter out the fusion features corresponding to the bounding boxes that are in motion in the adjacent frames.

[0038] S5: Output the fusion features corresponding to the bounding boxes in the image whose size is greater than the size threshold.

[0039] The video stream to be detected can be a video stream captured by a camera device installed in the monitoring area.

[0040] In this application, the local features of a flame target can be a flame or a flame body, and the local feature of a smoke target can be a smoke cloud. For example... Figure 2 As shown, target recognition is performed on local features of the flame using deep learning, rather than detecting the entire smoke and flame. This leverages the more uniform characteristics of these local features, making them more suitable for deep learning detection and improving detection accuracy.

[0041] Furthermore, based on preset confidence score thresholds and bounding box size thresholds (too small a bounding box size may indicate a flame created by humans, such as a lighter), the detected features are filtered to make the remaining features more consistent with the characteristics of a real fire.

[0042] In addition, in this embodiment, considering that the flame should not be in a static state, but should be in a state of small-range movement, motion detection is performed on the feature in step S2. This is pixel-based motion detection, which requires that the pixel corresponding to the feature should change in real time. That is, the same feature should be in motion within adjacent frames, which can better match the pixel change characteristics of flame or smoke.

[0043] In step S3, during multi-frame feature fusion, each frame is fused based on the images of the preceding two frames. Since each local feature area is small and difficult to represent the fire situation individually, fusing multiple features from adjacent frames into a single overall feature helps in subsequent fire situation analysis. Figure 3 As shown.

[0044] The fusion process of this application adopts the following steps:

[0045] S301: Select the feature with the highest confidence score among all features to be fused as the fusion feature;

[0046] S302: Traverse all other features to be fused, calculate the distance between each feature to be fused and the fused feature, determine whether the distance is less than the distance threshold, if so, fuse the feature to be fused with the fused feature, update the fused feature, until there are no features to be fused with a distance less than the distance threshold, then proceed to S303.

[0047] S303: Select the feature with the highest confidence score among all features to be fused that have a distance greater than or equal to the distance threshold as the new fusion feature, and return to S302.

[0048] In this embodiment, the distance is calculated using the intersection-union ratio (IOU).

[0049] In the above fusion steps, the confidence score of each feature is used as the selection criterion for fusion features. The distance between two features is the sole criterion for determining whether to further fuse them with the existing fusion feature. If the distance is less than a preset distance threshold, it is fused with the original fusion feature to obtain a new fusion feature. After all features with a distance less than the distance threshold have been fused with the fusion feature (e.g., 6 features have been fused), the remaining feature distances are processed separately. From the remaining features, the next fusion feature is selected based on the confidence score, and the next fusion is performed until all features to be fused are incorporated into the fusion feature. At this point, the image may contain multiple fusion features.

[0050] Furthermore, this embodiment also considers that features such as vehicle lights and flowing light, similar to flames, may be misidentified. After research and analysis, it was found that most of these potentially misidentified features are in a moving state. Therefore, the flame feature required in this embodiment should be in a fixed range, meaning it only moves slightly within a certain range, but its range remains unchanged. Correspondingly, the bounding box of the feature (which needs to be the bounding box of the fused feature) will not change. To achieve this technical objective, this embodiment detects whether the bounding box of each fused feature is in motion in step S4, retaining only the fused features corresponding to bounding boxes in a stationary state.

[0051] Furthermore, this embodiment also includes analyzing and filtering the fused features. That is, during feature fusion, the gain rate of each fused feature is calculated, and after feature fusion is completed, fused features with a gain rate less than the gain rate threshold are filtered out.

[0052] The formula for calculating the gain ratio A of the fused feature is:

[0053]

[0054] in, This represents the confidence score of the fusion features before fusion. This represents the confidence score of the features to be fused before fusion. This represents the intersection-union ratio (IURR) of the bounding boxes of two features when they are enlarged by a factor of 1.5, and max(.) represents finding the maximum value.

[0055]

[0056] in, This represents the confidence score of the fused features.

[0057] The size threshold in step S5 can be set by the user according to their business needs, and the output fusion features may be one or more. Finally, the user determines whether a fire has occurred based on the output fusion features.

[0058] This embodiment uses multi-frame fusion to take the continuity of flame combustion as a necessary condition, which can alleviate the instability of a single frame caused by the morphological instability during combustion to a certain extent.

[0059] Example 2:

[0060] The present invention also provides a fire smoke detection terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the method embodiment described above in Embodiment 1 of the present invention.

[0061] Furthermore, as an executable solution, the fire smoke detection terminal device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The fire smoke detection terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described composition of the fire smoke detection terminal device is merely an example and does not constitute a limitation on the fire smoke detection terminal device. It may include more or fewer components than described above, or combine certain components, or different components. For example, the fire smoke detection terminal device may also include input / output devices, network access devices, buses, etc., and this embodiment of the invention does not limit this.

[0062] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the fire smoke detection terminal equipment, connecting all parts of the equipment via various interfaces and lines.

[0063] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the fire smoke detection terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0064] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the embodiments of the present invention.

[0065] If the modules / units integrated into the fire smoke detection terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc.

[0066] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A method for detecting fire smoke, characterized in that, This includes: receiving the video stream to be detected, and performing the following processing on each frame of the video stream: S1: Detect local features of flame or smoke targets in the image; the local features of flame targets are flames or flame bodies; the local features of smoke targets are smoke plumes. S2: Combine the adjacent frame images of the image to determine whether each detected feature is in a static state in the adjacent frame, and filter out the features that are in a static state in the adjacent frame. S3: Combine the adjacent frame images of the image, perform feature fusion on each feature in the image and the corresponding features in the adjacent frame images to obtain each fused feature in the image; S4: Combining the adjacent frames of the image, determine whether the bounding boxes of each fusion feature of the image are in motion, and filter out the fusion features corresponding to the bounding boxes that are in motion in the adjacent frames. S5: Output the fusion features corresponding to the bounding boxes in the image whose size is greater than the size threshold. The output fusion features are used to determine whether a fire has occurred.

2. The fire smoke detection method according to claim 1, characterized in that: Before step S2, the process also includes: filtering out features whose confidence scores are less than the confidence score threshold for each detected feature.

3. The fire smoke detection method according to claim 1, characterized in that: Before step S2, the method also includes: filtering out features whose bounding box size is smaller than the bounding box size threshold for each detected feature.

4. The fire smoke detection method according to claim 1, characterized in that: The feature fusion process includes the following steps: S301: Select the feature with the highest confidence score among all features to be fused as the fusion feature; S302: Traverse all other features to be fused, calculate the distance between each feature to be fused and the fused feature, determine whether the distance is less than the distance threshold, if so, fuse the feature to be fused with the fused feature, update the fused feature, until there are no features to be fused with a distance less than the distance threshold, then proceed to S303. S303: Select the feature with the highest confidence score among all features to be fused that have a distance greater than or equal to the distance threshold as the new fusion feature, and return to S302.

5. The fire smoke detection method according to claim 1, characterized in that: During feature fusion, the process also includes calculating the gain ratio of each fused feature and filtering out fused features with a gain ratio less than the gain ratio threshold after feature fusion is completed.

6. The fire smoke detection method according to claim 1, characterized in that: The formula for calculating the gain ratio A of the fused feature is: in, This represents the confidence score of the fusion features before fusion. This represents the confidence score of the features to be fused before fusion. This represents the intersection-union ratio (IURR) of the bounding boxes of two features when they are enlarged by a factor of 1.5, and max(.) represents finding the maximum value.

7. A fire smoke detection terminal device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images