Multi-mode indoor fire identification method, device and equipment and storage medium

Abstract

The invention relates to the technical field of image processing, solves the problem of low recognition accuracy of an existing fire recognition technology in a complex scene, and provides a multi-mode indoor fire recognition method, device and equipment and a storage medium. The method comprises the following steps: respectively acquiring a first visible light video and a first thermal infrared video in the same scene; judging the scene type of the scene according to the first visible light video; if the scene type is a non-kitchen scene, judging whether a fire occurs in the non-kitchen scene according to a first visible light video and a first thermal infrared video; if the scene type is a kitchen scene, judging whether a cooking bench area of the kitchen scene is in an abnormal fire state or not according to the first visible light video; and if yes, judging whether a fire occurs in the kitchen scene according to the first visible light video and the first thermal infrared video. According to the invention, fire identification in a complex scene is realized through multi-modal data, and the identification accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-modal indoor fire recognition method, device, equipment and storage medium. Background Art

[0002] In today's society, indoor fires have become a serious safety threat, often resulting in casualties and property losses. Therefore, it is crucial to develop efficient fire identification technology to detect and respond to fire incidents in a timely manner and reduce potential injuries and losses.

[0003] Currently, there are many fire identification schemes, including methods based on traditional devices such as smoke detectors, temperature sensors, and flame detectors. These methods usually rely on data from a single sensor, such as the detection of smoke, temperature, or flame, to determine whether a fire has occurred. In addition, some advanced methods also use image processing and machine learning techniques, such as convolutional neural networks (CNNs), to analyze flames or smoke in video images.

[0004] However, existing fire identification solutions often have some limitations. For example, sensor-based methods may be affected by environmental factors such as dust or humidity, resulting in false alarms or missed alarms. Image processing and machine learning-based methods, while able to improve accuracy, may not perform well for complex scenes and lighting conditions. In addition, the response time of fire identification in existing methods may not be fast enough to detect fires in time and take countermeasures. Summary of the invention

[0005] In view of this, embodiments of the present invention provide a multi-modal indoor fire identification method, device, equipment and storage medium to solve the problem of low identification accuracy of existing fire identification technology in complex scenarios.

[0006] In a first aspect, an embodiment of the present invention provides a multi-modal indoor fire identification method, the method comprising:

[0007] Controlling the visible light camera and the thermal infrared camera to respectively obtain a first visible light video and a first thermal infrared video of the same scene;

[0008] determining a scene type of the scene according to the first visible light video;

[0009] If the scene type is a non-kitchen scene, determining whether a fire occurs in the non-kitchen scene according to the first visible light video and the first thermal infrared video;

[0010] If the scene type is a kitchen scene, determining whether a stove area of ​​the kitchen scene is in an abnormal fire state according to the first visible light video;

[0011] If so, it is determined whether a fire occurs in the kitchen scene based on the first visible light video and the first thermal infrared video.

[0012] Preferably, if the scene type is a non-kitchen scene, judging whether a fire occurs in the non-kitchen scene according to the visible light video and the thermal infrared video includes:

[0013] Segment each frame of the thermal infrared video according to an adaptive threshold segmentation algorithm to obtain a first segmented image sequence, wherein the first segmented image sequence includes a plurality of segmented images;

[0014] Acquire the contour area of ​​each segmented image according to the segmented image sequence to obtain an area sequence;

[0015] According to the area sequence, the contour areas corresponding to two adjacent frames of images in the thermal infrared video are obtained to obtain an area difference sequence;

[0016] If the area difference sequence shows an increasing trend, target detection is performed on the second visible light video of the non-kitchen scene to determine whether a fire occurs in the non-kitchen scene.

[0017] Preferably, the visible light camera and the thermal infrared camera are arranged on a pan / tilt platform, and if the area difference sequence shows an increasing trend, target detection is performed according to the visible light video to determine whether a fire occurs in the non-kitchen scene, including:

[0018] Controlling the pan / tilt head to rotate, and controlling the visible light camera to shoot the non-kitchen scene, to obtain a second visible light video;

[0019] Performing frame-by-frame detection on the second visible light video according to a pre-constructed first target detection model to obtain a target image sequence, wherein the target image sequence includes a plurality of target images, and the target images include smoke and / or flames, and the first target detection model is trained by a first training set, and the first training set includes a plurality of indoor flame images and indoor smoke images;

[0020] If the number of consecutive target images in the target image sequence is greater than or equal to a preset number, a fire occurs in the non-kitchen scene.

[0021] Preferably, if the scene type is a kitchen scene, judging whether the stove area of ​​the kitchen scene is in an abnormal fire state according to the first visible light video includes:

[0022] Acquire a stove area of ​​the kitchen scene according to a pre-built second object detection model and the first visible light video, wherein the second object detection model is trained by a second training set, and the second training set includes a plurality of stove images;

[0023] Detecting the flame in the stove area according to the first target detection model to obtain a flame area;

[0024] According to the flame area and the stove area, it is determined whether the stove area of ​​the kitchen scene is in an abnormal fire state.

[0025] Preferably, judging whether the stove area of ​​the kitchen scene is in a normal fire state according to the flame area and the stove area includes:

[0026] Obtaining the width of the flame area to obtain the flame width;

[0027] Obtain the width of the stove area, and obtain the stove width;

[0028] If the ratio of the flame width to the stove width is greater than or equal to a preset ratio, the kitchen scene is in an abnormal fire state;

[0029] Otherwise, the kitchen scene is in a normal fire-making state.

[0030] Preferably, if the fire is in the kitchen scene, judging whether a fire occurs in the kitchen scene according to the first visible light video and the first thermal infrared video includes:

[0031] Obtaining a high temperature threshold according to the first visible light video and the first thermal infrared video;

[0032] According to the high temperature threshold, performing high temperature threshold segmentation on the first thermal infrared video;

[0033] If the high temperature segmentation area corresponding to the high temperature threshold does not exist in the first thermal infrared video, then no fire occurs in the kitchen scene;

[0034] If there is a high temperature segmentation area corresponding to the high temperature threshold in the first thermal infrared video, the first visible light video is detected frame by frame according to the first target detection model to obtain a flame area in each frame image;

[0035] If the total area of ​​the flame region in the first visible light video increases with the increase in the number of frames, a fire occurs in the kitchen scene;

[0036] Otherwise, no fire occurs in the kitchen scene.

[0037] Preferably, judging the scene type of the scene according to the first visible light video includes:

[0038] Inputting the first visible light video into a pre-built scene recognition model to obtain a scene recognition result, wherein the scene recognition model is a binary classification model built based on a ResNet algorithm;

[0039] The scene type is obtained according to the recognition result.

[0040] In a second aspect, an embodiment of the present invention provides a multi-modal indoor fire identification device, the device comprising:

[0041] A video acquisition module, used to control the visible light camera and the thermal infrared camera to respectively acquire a first visible light video and a first thermal infrared video of the same scene;

[0042] A scene type determination module, configured to determine the scene type of the scene according to the first visible light video;

[0043] A first fire judgment module, configured to judge whether a fire occurs in the non-kitchen scene according to the first visible light video and the first thermal infrared video if the scene type is a non-kitchen scene;

[0044] a kitchen fire risk judgment module, configured to judge whether the stove area of ​​the kitchen scene is in an abnormal fire state according to the first visible light video if the scene type is a kitchen scene;

[0045] The second fire judgment module is used to judge whether a fire occurs in the kitchen scene according to the first visible light video and the first thermal infrared video.

[0046] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of the first aspect in the above-mentioned embodiment.

[0047] In a fourth aspect, an embodiment of the present invention provides a storage medium having computer program instructions stored thereon, which implements the method of the first aspect in the above-mentioned embodiment when the computer program instructions are executed by a processor.

[0048] In summary, the beneficial effects of the present invention are as follows:

[0049] The multimodal indoor fire identification method, device, equipment and storage medium provided by the embodiment of the present invention respectively obtains the first visible light video and the first thermal infrared video in the same scene by controlling the visible light camera and the thermal infrared camera, thereby ensuring that both visible light and thermal infrared data are obtained at the same time, and providing the necessary information source for subsequent scene identification and fire detection. By analyzing the first visible light video, it can be determined whether the current scene is a kitchen scene or a non-kitchen scene, which helps to determine the subsequent detection strategy. In a non-kitchen scene, by combining visible light and thermal infrared video data, it can be more accurately determined whether a fire has occurred, avoiding the false alarm or missed alarm that may occur in a single sensor. Detecting the stove area in the kitchen scene can timely detect whether there is an abnormal flame, such as an unextinguished open flame or abnormal smoke, which helps to prevent the occurrence of fire and distinguish whether the stove is in a normal fire state, reduce false alarms and missed alarms, and improve the accuracy of fire detection. In an abnormal fire state, combined with visible light and thermal infrared video data, the fire in the kitchen scene is detected, which can improve the accuracy and reliability of fire identification, timely discover and respond to fire incidents, and ensure the safety of personnel and property. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0051] Figure 1 It is a flowchart of a multi-modal indoor fire identification method according to an embodiment of the present invention.

[0052] Figure 2 It is a flowchart of a multi-modal indoor fire identification method according to an embodiment of the present invention.

[0053] Figure 3 It is a flowchart of a multi-modal indoor fire identification method according to an embodiment of the present invention.

[0054] Figure 4 4 is a schematic diagram of the structure of a multi-modal indoor fire identification device according to an embodiment of the present invention.

[0055] Figure 5 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention.

[0057] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0058] Example 1

[0059] See also Figure 1 , an embodiment of the present invention provides a multi-modal indoor fire identification method, the method comprising:

[0060] S1, controlling a visible light camera and a thermal infrared camera to respectively obtain a first visible light video and a first thermal infrared video of the same scene;

[0061] Specifically, in order to obtain comprehensive environmental information, multimodal data is used to provide a more accurate scene analysis and fire detection basis. The visible light camera and thermal infrared camera are controlled at the same time to obtain two video streams in the same scene, ensuring the synchronization of the video streams for subsequent data fusion and analysis. The visible light video captures the conventional visual information of the scene, such as the color and shape of objects and the environment, while the thermal infrared video captures the thermal distribution information of the scene, that is, the temperature distribution of different objects. These two video streams provide the necessary data basis for subsequent scene recognition and fire detection.

[0062] In a specific embodiment, the first visible light video stream refers to a video stream with a frame rate of 15 frames per second captured by a color camera, with a resolution of 2k and a field of view of 120 degrees; the thermal infrared video stream refers to a grayscale video stream that uses a thermal infrared module to obtain thermal infrared radiation data and converts the data into a grayscale image, with a frame rate of 15 frames per second, a resolution of 80*62, and a field of view of 160 degrees;

[0063] Specifically, this step distinguishes whether the current scene is a kitchen through scene recognition, thereby determining different fire detection strategies. The judgment of the scene type enables the system to select different detection strategies in a targeted manner, thereby improving the accuracy and efficiency of fire detection.

[0064] In a preferred embodiment, the step S2 specifically includes:

[0065] S21, inputting the first visible light video into a pre-built scene recognition model to obtain a scene recognition result, wherein the scene recognition model is a binary classification model built based on a ResNet algorithm;

[0066] Specifically, the pre-built scene recognition model refers to a deep learning model that has been built in the training phase and is used to classify scenes. In this embodiment, a binary classification model based on the ResNet algorithm is used. ResNet is a deep residual network structure that can effectively train deep neural networks and has achieved good results in image classification tasks. After the first visible light video is input into the scene recognition model, the model outputs the scene classification result. The model will give a probability value or category label to indicate whether the current scene is a kitchen scene or a non-kitchen scene.

[0067] S22. Obtain the scene type according to the recognition result.

[0068] In this embodiment, the scene types mainly include kitchen scenes and non-kitchen scenes. According to the probability value or category label in the scene recognition result, for example, a probability value greater than 0.5 indicates that the current scene is a kitchen scene, and a probability value less than 0.5 indicates that the current scene is a non-kitchen scene. According to this judgment condition, the type of the current scene can be obtained.

[0069] S3. If the scene type is a non-kitchen scene, determine whether a fire occurs in the non-kitchen scene according to the first visible light video and the first thermal infrared video;

[0070] In this step, in non-kitchen scenarios, by comprehensively analyzing visible light and thermal infrared videos, it is determined whether there is a fire. Specifically, the visible light and thermal infrared videos can be analyzed by pre-set rules and statistical models to determine whether a fire has occurred. For example, a series of fire detection rules are set, such as: the temperature of a certain area in the thermal infrared video continues to exceed a certain threshold. Obvious flame shapes or fast-moving smoke appear in the visible light video. Statistical analysis is performed on the video data to calculate the area and temperature changes of the high-temperature area in the thermal infrared video, and the brightness changes and edge detection results in the visible light video. Based on the statistical results and preset rules, it is comprehensively determined whether the conditions for the occurrence of a fire are met. If the area and temperature of the high-temperature area in the thermal infrared video continue to increase, and obvious flames or smoke appear in the visible light video, it is determined that a fire has occurred. The method based on rules and statistical models is relatively simple and intuitive, easy to implement and adjust, and suitable for fire detection in a variety of scenarios.

[0071] Methods based on rules and statistical models usually rely on preset rules and fixed statistical thresholds and lack adaptability. This means that in different scenarios or under different environmental conditions, methods based on rules and statistical models may need to be adjusted frequently, and their flexibility and adaptability are relatively low;

[0072] Therefore, in a preferred embodiment, see Figure 2 If the scene type is a non-kitchen scene, judging whether a fire occurs in the non-kitchen scene based on the visible light video and the thermal infrared video includes:

[0073] S31, segmenting each frame of the thermal infrared video according to an adaptive threshold segmentation algorithm to obtain a first segmented image sequence, wherein the first segmented image sequence includes a plurality of segmented images;

[0074] Specifically, in order to identify and extract high-temperature areas in thermal infrared videos and provide basic data for subsequent fire detection, this step applies an adaptive threshold segmentation algorithm to segment each frame of thermal infrared images into high-temperature areas and low-temperature areas. The adaptive threshold algorithm dynamically adjusts the threshold according to the image content to ensure that the high-temperature area can be effectively segmented in different frames. The segmentation results are saved as binary images, with the high-temperature area as one category and the low-temperature area as another category. The first segmented image sequence is obtained through the segmented images of the high-temperature area. Adaptive threshold segmentation can effectively segment high-temperature areas under various lighting and environmental conditions, improving segmentation accuracy and robustness.

[0075] In one embodiment, before performing adaptive threshold segmentation, the thermal infrared video frame may be preprocessed, such as denoising and contrast enhancement, to improve the segmentation effect.

[0076] S32, acquiring the contour area of ​​each segmented image according to the segmented image sequence to obtain an area sequence;

[0077] This step aims to quantify the high temperature area in the thermal infrared image and provide data support for the subsequent fire trend analysis. First, contour detection is performed on each segmented image in the segmented image sequence to identify the boundary of the high temperature area, and the contour area of ​​each high temperature area is calculated and recorded in the area sequence. Each value in the area sequence corresponds to the area of ​​the high temperature area of ​​a frame of thermal infrared video. By calculating the contour area, the size of the high temperature area can be quantified, which is convenient for the subsequent fire trend analysis and judgment.

[0078] S33, according to the area sequence, obtaining the contour areas corresponding to two adjacent frames of images in the thermal infrared video to obtain an area difference sequence;

[0079] Specifically, the area difference of two adjacent frames in the area sequence is calculated to obtain an area difference sequence. Each value in the area difference sequence represents the change in the area of ​​the high-temperature area between two adjacent frames. The area difference sequence can reflect the growth or decrease trend of the high-temperature area and is an important indicator for judging fire.

[0080] S34: If the area difference sequence shows an increasing trend, target detection is performed on the second visible light video of the non-kitchen scene to determine whether a fire occurs in the non-kitchen scene.

[0081] Specifically, the area difference sequence is analyzed to determine whether it shows an increasing trend. If the area of ​​the high temperature area continues to grow, there is a possibility of fire. At this time, in order to further judge, the second visible light video of the non-kitchen scene is subjected to target detection to determine whether a fire has occurred in the non-kitchen scene. The second visible light video can be detected frame by frame through the target detection algorithm to determine whether there is flame and / or smoke in the second visible light video, and further confirm whether a fire has occurred in the non-kitchen scene.

[0082] Preferably, the visible light camera and the thermal infrared camera are arranged on a pan / tilt platform, and if the area difference sequence shows an increasing trend, target detection is performed according to the visible light video to determine whether a fire occurs in the non-kitchen scene, including:

[0083] S341, controlling the pan / tilt head to rotate, and controlling the visible light camera to shoot the non-kitchen scene, to obtain a second visible light video;

[0084] Specifically, in this embodiment, the visible light camera and the thermal infrared camera are set on a pan-tilt head, which can ensure that the viewing angles of the two cameras are consistent. However, since the field of view of the thermal infrared camera is usually wider than that of the visible light camera, when the thermal infrared camera detects a target, the visible light camera may not be able to capture it. Therefore, in order to expand the monitoring field of view, ensure full coverage of all areas of non-kitchen scenes, and improve the comprehensiveness and accuracy of fire detection, the rotation function of the pan-tilt head is started through control instructions to rotate it at a set speed and angle, and the visible light camera is synchronously controlled to continuously shoot during the rotation of the pan-tilt head to generate a second visible light video. Ensure that the camera can cover all areas of non-kitchen scenes. The field of view of the camera is expanded by pan-tilt head rotation to ensure full coverage of the scene, avoid monitoring blind spots, and improve the coverage and accuracy of fire detection.

[0085] S342, performing frame-by-frame detection on the second visible light video according to a pre-constructed first target detection model to obtain a target image sequence, wherein the target image sequence includes a plurality of target images, and the target images include smoke and / or flames, and the first target detection model is trained by a first training set, and the first training set includes a plurality of indoor flame images and indoor smoke images;

[0086] Specifically, this step uses a trained target detection model to identify fire features (smoke and flames) to improve the accuracy and robustness of detection. The pre-trained first target detection model is loaded. The model is trained with indoor flame and smoke images and can identify smoke and flames in the image. The second visible light video is processed frame by frame, and each frame is input into the target detection model for detection. The detection results in each frame are recorded. If smoke and / or flame are detected, the frame is marked as a target image to generate a target image sequence.

[0087] In a specific embodiment, the first target detection model is built based on the YOLO model.

[0088] S343: If the number of consecutive target images in the target image sequence is greater than or equal to a preset number, a fire occurs in the non-kitchen scene.

[0089] Specifically, in order to confirm the occurrence of a fire and avoid false alarms caused by accidentally detected flames or smoke, this step analyzes the target image sequence and counts the number of continuous target images. Continuous target images refer to flames or smoke detected in multiple consecutive frames. A preset number threshold (for example, 5 frames) is set. If the number of continuous target images is greater than or equal to the threshold, it is considered that the fire detection is continuous and stable, reducing false alarms. When the number of continuous target images reaches or exceeds the preset number threshold, a fire is confirmed and the corresponding fire alarm mechanism is triggered.

[0090] By detecting and counting continuous target images, false alarms caused by accidental factors can be avoided, the accuracy and reliability of fire detection can be improved, and it can be ensured that a fire is confirmed only when flames or smoke are continuously detected.

[0091] S4. If the scene type is a kitchen scene, determine whether the stove area of ​​the kitchen scene is in an abnormal fire state according to the first visible light video;

[0092] Specifically, for the kitchen scene, the stove area is identified and judged whether it is in a normal fire state to distinguish between daily cooking flames and abnormal flames. By distinguishing between normal and abnormal fire states, false alarms can be avoided when the stove is in a normal fire state;

[0093] Identifying the stove area and determining whether it is in a normal fire state can be done by a method based on color and shape analysis. The image of the stove area is extracted through the first visible light video, and color detection is performed to identify the characteristic color range of the flame. Shape analysis is performed on the extracted color area to identify the typical shape features of the flame, such as sharpness and fluctuation. If the detected color and shape features exceed the range of normal cooking flames, it is determined to be an abnormal fire state; or it can be achieved by a method based on smoke detection, using image processing technology to detect smoke above the stove area. Common methods include edge detection, contour detection, and motion detection, etc., to calculate the concentration and diffusion range of smoke and determine whether it exceeds the smoke concentration during normal cooking.

[0094] However, smoke detection methods are easily interfered by other factors. For example, non-fire factors such as steam and oil smoke may also produce smoke, leading to false alarms. Methods based on color and shape analysis are not accurate enough and are easily interfered by changes in ambient light, resulting in a high false alarm rate.

[0095] Therefore, in a preferred embodiment, see Figure 3 If the scene type is a kitchen scene, determining whether the stove area of ​​the kitchen scene is in an abnormal fire state according to the first visible light video includes:

[0096] S41, acquiring a stove area of ​​the kitchen scene according to a pre-built second object detection model and the first visible light video, wherein the second object detection model is trained by a second training set, and the second training set includes a plurality of stove images;

[0097] Specifically, in order to accurately locate the stove area in the kitchen scene and provide a target area for subsequent flame detection and judgment, a pre-trained second target detection model is loaded. This model is trained through stove images and is specifically used to identify stoves in kitchen scenes. Each frame of the first visible light video is processed frame by frame, and the image is input into the second target detection model to detect and mark the stove area. The location information of the stove area (such as the coordinates of the rectangular bounding box) is extracted from the detection results to generate a stove area sequence. The stove area is accurately located through a dedicated model, the accuracy of stove detection is improved, and an accurate target area is provided for subsequent flame detection.

[0098] S42, detecting the flame of the stove area according to the first target detection model to obtain a flame area;

[0099] In order to detect flames in the stove area and determine whether the stove is in an abnormal ignition state, a pre-trained first target detection model is used. The model is trained with flame and smoke images and is specifically used to detect flames. The extracted stove area image is input into the first target detection model to detect whether there is a flame and mark the flame area. The flame detection results are extracted, the location information and size of the flame area are recorded, and a flame area sequence is generated.

[0100] S43: judging whether the stove area of ​​the kitchen scene is in an abnormal fire state according to the flame area and the stove area.

[0101] By analyzing the relationship between the flame area and the stove area, it is determined whether the stove is in an abnormal fire state.

[0102] Preferably, judging whether the stove area of ​​the kitchen scene is in a normal fire state according to the flame area and the stove area includes:

[0103] S431, obtaining the width of the flame area to obtain the flame width;

[0104] Specifically, based on the flame area detected in the previous step S42, the position information of the flame area, such as the width of the rectangular boundary box, is extracted.

[0105] S432 obtains the width of the stove area, and obtains the stove width;

[0106] According to the stove area detected in the previous step S41, the position information of the stove area is extracted, such as the width of the rectangular boundary box.

[0107] S433: If the ratio of the flame width to the stove width is greater than or equal to a preset ratio, the kitchen scene is in an abnormal fire-making state;

[0108] Specifically, whether there is an abnormal situation is determined based on the ratio of the width of the flame to the width of the stove. When the ratio of the width of the flame to the width of the stove is greater than or equal to a preset ratio, it indicates that the width of the flame exceeds the width of the stove, which may mean that the flame has exceeded the control range of the stove and is no longer restricted by the stove, which may lead to the occurrence of a fire or a fire has already occurred. Therefore, when the ratio of the width of the flame to the width of the stove exceeds a certain threshold, it should be considered that the stove is in an abnormal fire state.

[0109] In one embodiment, the preset ratio is 1.5. Usually, the width of the stove is relatively fixed. If the width of the flame exceeds 1.5 times the width of the stove, it may exceed the normal cooking range, which may mean that abnormal combustion occurs, such as grease splashing or burning objects, which may cause a fire. Therefore, setting the threshold to 1.5 can improve the accuracy and reliability of fire detection to a certain extent.

[0110] S434: Otherwise, the kitchen scene is in a normal fire-making state.

[0111] If the ratio of the flame width to the stove width does not reach the preset ratio, it is determined that the stove is in a normal fire state.

[0112] S5. If yes, determine whether a fire occurs in the kitchen scene based on the first visible light video and the first thermal infrared video.

[0113] Specifically, an abnormal fire-making state does not necessarily mean a fire. For example, the flame on the stove may increase briefly due to oil splashing or other cooking activities, but this does not mean that a fire has occurred. Further judgment can avoid false alarms of fires due to short-term abnormalities. Confirmation of a fire usually requires a comprehensive judgment of multiple factors (such as continued high temperature, flame area growth, etc.), rather than just a single factor (such as changes in flame size or color).

[0114] Therefore, this step further analyzes the first visible light video and the first thermal infrared video to determine whether a fire occurs in the kitchen scene.

[0115] In one embodiment, the step S5 specifically includes:

[0116] S51, obtaining a high temperature threshold according to the first visible light video and the first thermal infrared video;

[0117] Specifically, by analyzing the first visible light video and the first thermal infrared video, a high temperature threshold in the thermal infrared video is obtained for subsequent high temperature area segmentation. The first thermal infrared video is analyzed to count the temperature information of each pixel. Based on the statistical results, a high temperature threshold is determined, which can be an average temperature, a quantile, or other statistics. This provides a benchmark for separating the high temperature area from the normal temperature area in the thermal infrared video.

[0118] In one embodiment, the step S51 specifically includes: observing and analyzing the stove in the kitchen scene through a visible light camera to determine whether it is in a fire state. This can be achieved by detecting obvious flames or firelight on the stove through the first target detection model. Once it is determined that the stove is in a fire state, a thermal infrared image of the current scene is obtained from the thermal infrared camera. The acquired thermal infrared image is subjected to adaptive threshold segmentation processing. This segmentation method automatically adjusts the threshold according to the local characteristics of the image, so that the threshold of each area is different, and adapts to the illumination changes and image noise conditions in different areas. During the segmentation process, the threshold obtained is the high temperature threshold T_high. This threshold can be automatically adjusted according to the characteristics of the thermal infrared image of the stove in the fire state to ensure accurate segmentation of the high temperature area. By performing adaptive threshold segmentation on the thermal infrared image according to the fire state of the stove, the high temperature threshold obtained can better adapt to the actual situation of the current scene, thereby improving the accuracy and reliability of fire detection.

[0119] S52, performing high temperature threshold segmentation on the first thermal infrared video according to the high temperature threshold;

[0120] Specifically, the obtained high temperature threshold is used to process each frame of the first thermal infrared video, and the area with a temperature higher than the preset threshold is marked as a high temperature area, and the other areas are marked as normal temperature areas. The potential fire heat source that may exist in the thermal infrared video is highlighted, which helps to more accurately judge the fire risk.

[0121] S53: If there is no high temperature segmentation area corresponding to the high temperature threshold in the first thermal infrared video, then no fire occurs in the kitchen scene;

[0122] Specifically, it is determined whether there is a high temperature area corresponding to the high temperature threshold in the first thermal infrared video, so as to infer whether a fire has occurred in the kitchen scene, that is, a temperature area higher than the high temperature threshold. If there is no high temperature area, the possibility of a fire in the kitchen scene can be preliminarily ruled out, reducing the possibility of false alarms.

[0123] S54: if there is a high temperature segmentation area corresponding to the high temperature threshold in the first thermal infrared video, perform frame-by-frame detection on the first visible light video according to the first target detection model to obtain a flame area in each frame image;

[0124] Specifically, in the case of a high temperature area, the first visible light video is detected frame by frame using the first target detection model to determine the flame area. Through target detection, the location information of the flame in the visible light video can be obtained to further assist in the identification and location of the fire.

[0125] S55: If the total area of ​​the flame region in the first visible light video increases as the number of frames increases, a fire occurs in the kitchen scene;

[0126] This step determines whether a fire has occurred in the kitchen scene by analyzing the change in the total area of ​​the flame area over time. Specifically, for each frame of the flame area, its area is calculated and the total area is accumulated over time. If the total area of ​​the flame increases over time, it means that the fire is spreading or persisting, which helps to promptly report and respond.

[0127] S56: Otherwise, no fire occurs in the kitchen scene.

[0128] If the total area of ​​the flame region does not increase over time, it is considered that no fire has occurred in the kitchen scene.

[0129] It should be noted that all actions to obtain signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the location and with the authorization given by the owner of the corresponding device.

[0130] Example 2

[0131] See also Figure 4 The embodiment of the present invention provides a multi-modal indoor fire identification device, the device comprising:

[0132] A video acquisition module, used to control the visible light camera and the thermal infrared camera to respectively acquire a first visible light video and a first thermal infrared video of the same scene;

[0133] A scene type determination module, configured to determine the scene type of the scene according to the first visible light video;

[0134] A first fire judgment module, configured to judge whether a fire occurs in the non-kitchen scene according to the first visible light video and the first thermal infrared video if the scene type is a non-kitchen scene;

[0135] a kitchen fire risk judgment module, configured to judge whether the stove area of ​​the kitchen scene is in an abnormal fire state according to the first visible light video if the scene type is a kitchen scene;

[0136] The second fire judgment module is used to judge whether a fire occurs in the kitchen scene according to the first visible light video and the first thermal infrared video.

[0137] It should be noted that the modules and units in the multimodal indoor fire identification device in this embodiment correspond one-to-one to the steps in the multimodal indoor fire identification method in the aforementioned embodiment. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned multimodal indoor fire identification method, which will not be repeated here.

[0138] Example 3

[0139] In addition, combined Figure 1 The multi-modal indoor fire identification method according to the embodiment of the present invention described above can be implemented by an electronic device. Figure 5 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention is shown.

[0140] The electronic device may include a processor and a memory storing computer program instructions.

[0141] Specifically, the processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present invention.

[0142] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside a data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.

[0143] The processor implements any one of the multi-modal indoor fire identification methods in the above embodiments by reading and executing computer program instructions stored in the memory.

[0144] In one example, the electronic device may further include a communication interface and a bus. Figure 5As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 and communicate with each other.

[0145] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.

[0146] Bus includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In suitable cases, bus may include one or more buses. Although the embodiment of the present invention describes and shows a specific bus, the present invention considers any suitable bus or interconnection.

[0147] Example 4

[0148] In addition, in combination with the multi-modal indoor fire identification method in the above embodiment, the embodiment of the present invention can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any multi-modal indoor fire identification method in the above embodiment is implemented.

[0149] In summary, the multimodal indoor fire identification method, device, equipment and storage medium provided by the embodiments of the present invention, by controlling the visible light camera and the thermal infrared camera to respectively obtain the first visible light video and the first thermal infrared video in the same scene, ensures that both visible light and thermal infrared data are obtained at the same time, and provides the necessary information source for subsequent scene identification and fire detection. By analyzing the first visible light video, it can be determined whether the current scene is a kitchen scene or a non-kitchen scene, which helps to determine the subsequent detection strategy. In non-kitchen scenes, by combining visible light and thermal infrared video data, it is possible to more accurately determine whether a fire has occurred, avoiding the false alarm or missed alarm that may occur in a single sensor. Detecting the stove area in the kitchen scene can promptly detect whether there are abnormal flames, such as an unextinguished open flame or abnormal smoke, which helps prevent the occurrence of fire and distinguish whether the stove is in a normal fire state, reduce false alarms and missed alarms, and improve the accuracy of fire detection. Combining visible light and thermal infrared video data under abnormal fire conditions to detect fires in kitchen scenes can improve the accuracy and reliability of fire identification, promptly discover and respond to fire incidents, and ensure the safety of personnel and property.

[0150] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0151] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0152] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0153] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. A multi-modal indoor fire identification method, characterized in that: The method comprises: Controlling the visible light camera and the thermal infrared camera to respectively obtain a first visible light video and a first thermal infrared video of the same scene; determining a scene type of the scene according to the first visible light video; If the scene type is a non-kitchen scene, determining whether a fire occurs in the non-kitchen scene according to the first visible light video and the first thermal infrared video; If the scene type is a kitchen scene, determining whether a stove area of ​​the kitchen scene is in an abnormal fire state according to the first visible light video; If so, it is determined whether a fire occurs in the kitchen scene based on the first visible light video and the first thermal infrared video.

2. The multi-modal indoor fire identification method according to claim 1, characterized in that: If the scene type is a non-kitchen scene, judging whether a fire occurs in the non-kitchen scene according to the visible light video and the thermal infrared video includes: Segment each frame of the thermal infrared video according to an adaptive threshold segmentation algorithm to obtain a first segmented image sequence, wherein the first segmented image sequence includes a plurality of segmented images; Acquire the contour area of ​​each segmented image according to the segmented image sequence to obtain an area sequence; According to the area sequence, the contour areas corresponding to two adjacent frames of images in the thermal infrared video are obtained to obtain an area difference sequence; If the area difference sequence shows an increasing trend, target detection is performed on the second visible light video of the non-kitchen scene to determine whether a fire occurs in the non-kitchen scene.

3. The multi-modal indoor fire identification method according to claim 2, characterized in that: The visible light camera and the thermal infrared camera are arranged on a pan-tilt platform. If the area difference sequence shows an increasing trend, target detection is performed according to the visible light video to determine whether a fire occurs in the non-kitchen scene, including: Controlling the pan / tilt head to rotate, and controlling the visible light camera to shoot the non-kitchen scene, to obtain a second visible light video; Performing frame-by-frame detection on the second visible light video according to a pre-constructed first target detection model to obtain a target image sequence, wherein the target image sequence includes a plurality of target images, and the target images include smoke and / or flames, and the first target detection model is trained by a first training set, and the first training set includes a plurality of indoor flame images and indoor smoke images; If the number of consecutive target images in the target image sequence is greater than or equal to a preset number, a fire occurs in the non-kitchen scene.

4. The multi-modal indoor fire identification method according to claim 3, characterized in that: If the scene type is a kitchen scene, determining whether the stove area of ​​the kitchen scene is in an abnormal fire state according to the first visible light video includes: Acquire a stove area of ​​the kitchen scene according to a pre-built second object detection model and the first visible light video, wherein the second object detection model is trained by a second training set, and the second training set includes a plurality of stove images; Detecting the flame in the stove area according to the first target detection model to obtain a flame area; According to the flame area and the stove area, it is determined whether the stove area of ​​the kitchen scene is in an abnormal fire state.

5. The multi-modal indoor fire identification method according to claim 4, characterized in that: The determining, based on the flame area and the stove area, whether the stove area of ​​the kitchen scene is in a normal fire state includes: Obtaining the width of the flame area to obtain the flame width; Obtain the width of the stove area, and obtain the stove width; If the ratio of the flame width to the stove width is greater than or equal to a preset ratio, the kitchen scene is in an abnormal fire state; Otherwise, the kitchen scene is in a normal fire-making state.

6. The multi-modal indoor fire identification method according to claim 4, characterized in that: If so, determining whether a fire occurs in the kitchen scene according to the first visible light video and the first thermal infrared video includes: Obtaining a high temperature threshold according to the first visible light video and the first thermal infrared video; According to the high temperature threshold, performing high temperature threshold segmentation on the first thermal infrared video; If the high temperature segmentation area corresponding to the high temperature threshold does not exist in the first thermal infrared video, then no fire occurs in the kitchen scene; If there is a high temperature segmentation area corresponding to the high temperature threshold in the first thermal infrared video, the first visible light video is detected frame by frame according to the first target detection model to obtain a flame area in each frame image; If the total area of ​​the flame region in the first visible light video increases with the increase in the number of frames, a fire occurs in the kitchen scene; Otherwise, no fire occurs in the kitchen scene.

7. The multi-modal indoor fire identification method according to any one of claims 1 to 6, characterized in that: The determining, according to the first visible light video, the scene type of the scene includes: Inputting the first visible light video into a pre-built scene recognition model to obtain a scene recognition result, wherein the scene recognition model is a binary classification model built based on a ResNet algorithm; The scene type is obtained according to the recognition result.

8. A multi-modal indoor fire identification device, characterized in that: The device comprises: A video acquisition module, used to control the visible light camera and the thermal infrared camera to respectively acquire a first visible light video and a first thermal infrared video of the same scene; A scene type determination module, configured to determine the scene type of the scene according to the first visible light video; A first fire judgment module, configured to judge whether a fire occurs in the non-kitchen scene according to the first visible light video and the first thermal infrared video if the scene type is a non-kitchen scene; a kitchen fire risk judgment module, configured to judge whether the stove area of ​​the kitchen scene is in an abnormal fire state according to the first visible light video if the scene type is a kitchen scene; The second fire judgment module is used to judge whether a fire occurs in the kitchen scene according to the first visible light video and the first thermal infrared video.

9. An electronic device, characterized in that: include: At least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, implement the method according to any one of claims 1 to 7.

10. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images