Fire detection method, device, electronic equipment and storage medium
By conducting background modeling and deep learning model detection of thermal imaging video frame data, the high-temperature non-fire object areas are blocked, and the false alarm problem in fire detection is solved and the detection accuracy is improved.
Patent Information
- Application Number
- CN202210194329.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-01
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-03-01
AI Technical Summary
In fire detection, existing computer vision technology is prone to false alarms caused by high temperature non-real fire point background objects, especially in complex scenarios, such as when the local temperature is high during loading and unloading forklifts in indoor warehouses. Causes false alarms.
By obtaining the thermal imaging video frame data of the monitoring scene, conducting background modeling and foreground detection, using pre-trained deep learning models to detect targets, blocking areas of moving targets and preset types of targets, and reducing the fire of high-temperature non-fire objects Effects of the results of the situation.
It effectively reduces the situation of false alarms in fire conditions, improves the probability of detecting high-temperature non-fire objects, and enhances the accuracy of fire conditions detection.
Smart Images

Figure CN114549827B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a fire detection method, device, electronic device and storage medium. Background Art
[0002] With the development of computer technology, computer vision technology has also gradually developed, and applications based on computer vision have increasingly affected people's production and life. Fire detection refers to the detection of whether there is a fire in the scene through sensors such as temperature sensors and smoke sensors. With the development of computer vision technology, image-based fire detection methods have also developed rapidly.
[0003] In the current fire detection method of computer vision technology, the mapping relationship between the grayscale and temperature of the thermal imaging image is used to obtain the temperature of each area in the image. When the temperature of an area is higher than a preset temperature threshold, it is determined that a fire has occurred.
[0004] However, when using the above method in complex application scenarios, high-temperature non-real fire point background objects in the scene are likely to cause false fire alarms. For example, in application scenarios such as indoor warehouses, high local temperatures may occur during forklift operations for loading and unloading goods, which may cause false fire alarms. Summary of the invention
[0005] The purpose of the embodiments of the present application is to provide a fire detection method, device, electronic device and storage medium to reduce the situation of false fire alarms. The specific technical solution is as follows:
[0006] In a first aspect, an embodiment of the present application provides a fire detection method, the method comprising:
[0007] Acquire first thermal imaging video frame data and second thermal imaging video frame data of a monitoring scene, wherein the first thermal imaging video frame data includes a plurality of thermal imaging video frames, the second thermal imaging video frame data includes at least one thermal imaging video frame, and the video frames in the first thermal imaging video frame data and the video frames in the second thermal imaging video frame data have common video frames;
[0008] Performing background modeling and foreground detection on the first thermal imaging video frame data to obtain first trajectory information of the moving target;
[0009] Performing target detection on the second thermal imaging video frame data using a pre-trained deep learning model to obtain position information of a preset type of target in the second thermal imaging video frame data;
[0010] According to the first trajectory information and the position information of the target in the second thermal imaging video frame data, a corresponding area in the third thermal imaging video frame data is shielded to obtain shielded third thermal imaging video frame data, wherein the third thermal imaging video frame data is one or more video frames in the common video frame;
[0011] Temperature detection is performed on the shielded third thermal imaging video frame data to obtain a fire detection result.
[0012] In a possible implementation manner, performing background modeling and foreground detection on the first thermal imaging video frame data to obtain first trajectory information of the moving target includes:
[0013] Performing background modeling and foreground detection on the first thermal imaging video frame data to obtain a foreground area of the first thermal imaging video frame data;
[0014] Based on the foreground area of the first thermal imaging video frame data, associating the position areas of the same target to obtain initial trajectory information of each target;
[0015] Moving target detection is performed on the initial trajectory information of each target to obtain first trajectory information of the moving target.
[0016] In a possible implementation, the associating the position areas of the same target based on the foreground area of the first thermal imaging video frame data to obtain the initial trajectory information of each target includes:
[0017] Based on the foreground area of the first thermal imaging video frame data, the target frames of the same target are associated to obtain associated trajectory information of each of the targets;
[0018] For each target, according to the associated trajectory information of the target, the missing trajectory position in the associated trajectory information of the target is completed to obtain the initial trajectory information of the target.
[0019] In a possible implementation manner, performing moving target detection on the initial trajectory information of each target to obtain first trajectory information of the moving target includes:
[0020] For each target, based on the initial trajectory information of the target, if the position change between the current target frame to be processed in the current cycle and the previous target frame processed in the previous cycle is greater than a preset distance threshold, the current target frame in the initial trajectory information of the target is marked with a moving mark, otherwise the current target frame in the initial trajectory information of the target is marked with a stationary mark;
[0021] For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target is in continuous motion, if the current target frame of the target undergoes a target frame mutation compared with the cached target frame of the target in the cache, the current target frame of the target is fused with the cached target frame of the target in the cache, and the cached target frame of the target in the cache is updated using the fused target frame; if the current target frame of the target does not undergo a target frame mutation compared with the cached target frame of the target in the cache, the cached target frame of the target in the cache is updated using the current target frame of the target; the target is marked as a moving target; wherein the target frame mutation refers to a change in the size of the target frame exceeding a preset change threshold;
[0022] For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target is continuously stationary, the cached target frame of the target in the cache is updated using the current target frame of the target; and the target is marked as a stationary target;
[0023] For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target changes from moving to stationary, the current target frame of the target is merged with the cached target frame of the target in the cache, and the cached target frame of the target in the cache is updated using the fused target frame; the target is marked as a moving target;
[0024] For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target changes from being stationary to moving, the cached target frame of the target in the cache is updated using the current target frame of the target; and the target is marked as a moving target;
[0025] For each moving target, the cached target frame of the moving target in the cache is used as the first trajectory information of the moving target in the current cycle.
[0026] In a possible implementation, fusing the current target frame of the target with the cached target frame of the target in the cache, and updating the cached target frame of the target in the cache by using the fused target frame includes:
[0027] According to a preset fusion smoothing coefficient, a target frame area of a corresponding size is selected from the cached target frame of the target in the cache, and the selected target frame area is fused with the current target frame of the target to obtain a fused target frame of the target, wherein the fusion smoothing coefficient represents the size of the target frame area selected from the cached target frame;
[0028] Update the cached target frame of the target in the cache to the fused target frame of the target.
[0029] In a possible implementation manner, before taking the cached target frame of the moving target in the cache as the first trajectory information of the moving target in the current cycle for each moving target, the method further includes:
[0030] For a disappeared target whose target frame disappears in the current cycle, determining whether the disappeared target is a moving target;
[0031] If the disappeared target is not a moving target, the cached target frame of the disappeared target is deleted from the cache.
[0032] In a possible implementation, the method further includes:
[0033] If the disappeared target is a moving target, judging whether the temperature of a corresponding area in a thermal imaging video frame of a current period is greater than a preset temperature threshold according to the position of the cached target frame of the disappeared target in the cache;
[0034] If it is greater than a preset temperature threshold, the cached target frame of the disappeared target in the cache is retained;
[0035] If the temperature is less than the preset temperature threshold, the cached target box of the disappeared target in the cache is cleared.
[0036] In a possible implementation, the acquiring of first thermal imaging video frame data and second thermal imaging video frame data of a monitoring scene includes:
[0037] Acquire thermal imaging video data of the monitoring scene;
[0038] According to a preset first frame rate, the thermal imaging video data is frame-decimated to obtain first thermal imaging video frame data;
[0039] The thermal imaging video data is framed according to a preset second frame rate to obtain second thermal imaging video frame data, wherein the settings of the preset first frame rate and the preset second frame rate satisfy that the first thermal imaging video frame data and the second thermal imaging video frame data have common video frames.
[0040] In a possible implementation manner, after performing target detection on the second thermal imaging video frame data using a pre-trained deep learning model to obtain position information of a preset type of target in the second thermal imaging video frame data, the method further includes:
[0041] For each preset type of target, the respective position information of the target is associated according to the position information of the target in the second thermal imaging video frame data to obtain associated trajectory information of the target;
[0042] For each preset type of target, if there is a track-missing video frame of the target in the second thermal imaging video frame data, determine the position information of the target in the track-missing video frame according to the associated track information of the target, and complete the associated track information of the target based on the position information of the target in the track-missing video frame of the target to obtain the second track information of the target;
[0043] The method of shielding a corresponding area in the third thermal imaging video frame data according to the first trajectory information and the position information of the target in the second thermal imaging video frame data to obtain the shielded third thermal imaging video frame data includes:
[0044] According to the first trajectory information and the second trajectory information, the corresponding area in the third thermal imaging video frame data is shielded to obtain the shielded third thermal imaging video frame data.
[0045] In a possible implementation manner, before shielding a corresponding area in the third thermal imaging video frame data according to the first trajectory information and the position information of the target in the second thermal imaging video frame data to obtain the shielded third thermal imaging video frame data, the method further includes:
[0046] Selecting video frames with the same timestamp and / or frame number from each video frame of the first thermal imaging video frame data and each video frame of the second thermal imaging video frame data to obtain the common video frame;
[0047] The common video frame is subjected to frame extraction according to a preset third frame extraction rate to obtain third thermal imaging video frame data.
[0048] In a second aspect, an embodiment of the present application provides a fire detection device, the device comprising:
[0049] A video data acquisition module, used to acquire first thermal imaging video frame data and second thermal imaging video frame data of a monitoring scene, wherein the first thermal imaging video frame data includes a plurality of thermal imaging video frames, the second thermal imaging video frame data includes at least one thermal imaging video frame, and the video frames in the first thermal imaging video frame data and the video frames in the second thermal imaging video frame data have common video frames;
[0050] A moving target trajectory acquisition module, used to perform background modeling and foreground detection on the first thermal imaging video frame data to obtain first trajectory information of the moving target;
[0051] a target trajectory acquisition module, configured to perform target detection on the second thermal imaging video frame data using a pre-trained deep learning model, and obtain position information of a preset type of target in the second thermal imaging video frame data;
[0052] a thermal imaging video data masking module, configured to mask a corresponding area in a third thermal imaging video frame data according to the first trajectory information and the position information of the target in the second thermal imaging video frame data, so as to obtain masked third thermal imaging video frame data, wherein the third thermal imaging video frame data is one or more video frames in the common video frame;
[0053] The fire detection module is used to perform temperature detection on the shielded third thermal imaging video frame data to obtain a fire detection result.
[0054] In a possible implementation, the moving target trajectory acquisition module includes:
[0055] a foreground area acquisition submodule, configured to perform background modeling and foreground detection on the first thermal imaging video frame data to obtain a foreground area of the first thermal imaging video frame data;
[0056] An initial trajectory information acquisition submodule associates the position areas of the same target based on the foreground area of the first thermal imaging video frame data to obtain the initial trajectory information of each target;
[0057] The first trajectory information acquisition submodule is used to perform moving target detection on the initial trajectory information of each target to obtain the first trajectory information of the moving target.
[0058] In a possible implementation, the initial trajectory information acquisition submodule is specifically used to: associate target frames of the same target based on the foreground area of the first thermal imaging video frame data to obtain associated trajectory information of each target; for each target, according to the associated trajectory information of the target, complete the missing trajectory position in the associated trajectory information of the target to obtain the initial trajectory information of the target.
[0059] In a possible implementation manner, the first trajectory information acquisition submodule is specifically configured to:
[0060] For each target, based on the initial trajectory information of the target, if the position change between the current target frame to be processed in the current cycle and the previous target frame processed in the previous cycle is greater than a preset distance threshold, the current target frame in the initial trajectory information of the target is marked with a moving mark, otherwise the current target frame in the initial trajectory information of the target is marked with a stationary mark;
[0061] For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target is in continuous motion, if the current target frame of the target undergoes a target frame mutation compared with the cached target frame of the target in the cache, the current target frame of the target is fused with the cached target frame of the target in the cache, and the cached target frame of the target in the cache is updated using the fused target frame; if the current target frame of the target does not undergo a target frame mutation compared with the cached target frame of the target in the cache, the cached target frame of the target in the cache is updated using the current target frame of the target; the target is marked as a moving target; wherein the target frame mutation refers to a change in the size of the target frame exceeding a preset change threshold;
[0062] For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target is continuously stationary, the cached target frame of the target in the cache is updated using the current target frame of the target; and the target is marked as a stationary target;
[0063] For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target changes from moving to stationary, the current target frame of the target is merged with the cached target frame of the target in the cache, and the cached target frame of the target in the cache is updated using the fused target frame; the target is marked as a moving target;
[0064] For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target changes from being stationary to moving, the cached target frame of the target in the cache is updated using the current target frame of the target; and the target is marked as a moving target;
[0065] For each moving target, the cached target frame of the moving target in the cache is used as the first trajectory information of the moving target in the current cycle.
[0066] In a possible implementation manner, the first trajectory information acquisition submodule is specifically configured to:
[0067] According to a preset fusion smoothing coefficient, a target frame area of a corresponding size is selected from the cached target frame of the target in the cache, and the selected target frame area is fused with the current target frame of the target to obtain a fused target frame of the target, wherein the fusion smoothing coefficient represents the size of the target frame area selected from the cached target frame;
[0068] Update the cached target frame of the target in the cache to the fused target frame of the target.
[0069] In a possible implementation, the first trajectory information acquisition submodule is further configured to: for a disappeared target whose target frame disappears in a current cycle, determine whether the disappeared target is a moving target; if the disappeared target is not a moving target, then delete the cached target frame of the disappeared target in the cache.
[0070] In a possible implementation, the first trajectory information acquisition submodule is further configured to: if the disappeared target is a moving target, determine whether the temperature at the corresponding area in the thermal imaging video frame of the current period is greater than a preset temperature threshold according to the position of the cached target frame of the disappeared target in the cache; if the temperature is greater than the preset temperature threshold, retain the cached target frame of the disappeared target in the cache; if the temperature is less than the preset temperature threshold, clear the cached target frame of the disappeared target in the cache.
[0071] In a possible implementation, the video data acquisition module is specifically used to: acquire thermal imaging video data of a monitoring scene; perform frame extraction on the thermal imaging video data according to a preset first frame extraction frame rate to obtain first thermal imaging video frame data; perform frame extraction on the thermal imaging video data according to a preset second frame extraction frame rate to obtain second thermal imaging video frame data, wherein the settings of the preset first frame extraction frame rate and the preset second frame extraction frame rate satisfy that the first thermal imaging video frame data and the second thermal imaging video frame data have common video frames.
[0072] In a possible implementation, the device further includes:
[0073] The second trajectory information acquisition module is used to associate each position information of each preset type of target according to the position information of the target in the second thermal imaging video frame data to obtain the associated trajectory information of the target; for each preset type of target, if there is a trajectory missing video frame of the target in the second thermal imaging video frame data, then according to the associated trajectory information of the target, determine the position information of the target in the trajectory missing video frame of the target, and based on the position information of the target in the trajectory missing video frame of the target, complete the associated trajectory information of the target to obtain the second trajectory information of the target
[0074] The thermal imaging video data masking module is specifically used to: mask the corresponding area in the third thermal imaging video frame data according to the first trajectory information and the second trajectory information to obtain the masked third thermal imaging video frame data.
[0075] In a possible implementation, the device further includes:
[0076] A frame extraction module is used to select video frames with the same timestamp and / or frame number from each video frame of the first thermal imaging video frame data and each video frame of the second thermal imaging video frame data to obtain the common video frame; and extract the common video frame according to a preset third frame extraction frame rate to obtain the third thermal imaging video frame data.
[0077] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;
[0078] The memory is used to store computer programs;
[0079] The processor is used to implement any fire detection method described in the present application when executing the program stored in the memory.
[0080] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any fire detection method described in the present application.
[0081] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any fire detection method described in the present application.
[0082] Beneficial effects of the embodiments of the present application:
[0083] The fire detection method, device, electronic device and storage medium provided in the embodiments of the present application can shield the area where the moving target and the preset type of target are located in the thermal imaging video frame data, which can effectively shield the high-temperature non-fire objects in the thermal imaging video frame data, and perform temperature detection on the shielded thermal imaging video frame data, thereby effectively reducing the impact of high-temperature non-fire objects on the fire detection results, and can reduce the false alarm of fire. In addition, the target detection based on the deep learning model can effectively detect the specified target that is more than half exposed in the monitoring scene, and for the running target that is more than half blocked in the monitoring scene, it can be effectively detected by background modeling and foreground detection, which makes up for the shortcomings of the deep learning model, thereby further improving the probability of detecting high-temperature non-fire objects, and ultimately reducing the false alarm of fire. Of course, the implementation of any product or method of the present application does not necessarily require all of the advantages described above to be achieved at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0084] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and a person of ordinary skill in the art can also obtain other drawings based on these drawings.
[0085] Figure 1 A first schematic diagram of a fire detection method according to an embodiment of the present application;
[0086] Figure 2 A schematic diagram of a possible implementation of step S12 in an embodiment of the present application;
[0087] Figure 3 A second schematic diagram of the fire detection method according to an embodiment of the present application;
[0088] Figure 4 A third schematic diagram of the fire detection method according to an embodiment of the present application;
[0089] Figure 5 A schematic diagram showing that the target frame gradually decreases in the embodiment of the present application;
[0090] Figure 6 A schematic diagram of target frame tailing in an embodiment of the present application;
[0091] Figure 7 This is a schematic diagram of an improved target frame tailing in an embodiment of the present application;
[0092] Figure 8 A fourth schematic diagram of the fire detection method according to an embodiment of the present application;
[0093] Fig. 9 A schematic diagram of a fire detection device according to an embodiment of the present application. DETAILED DESCRIPTION
[0094] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.
[0095] In order to reduce the situation of false fire alarms, an embodiment of the present application provides a fire detection method, the method comprising: obtaining first thermal imaging video frame data and second thermal imaging video frame data of a monitoring scene, wherein the first thermal imaging video frame data includes multiple frames of thermal imaging video frames, the second thermal imaging video frame data includes at least one frame of thermal imaging video frame, and the video frames in the first thermal imaging video frame data and the video frames in the second thermal imaging video frame data have common video frames; performing background modeling and foreground detection on the first thermal imaging video frame data to obtain first trajectory information of a moving target; performing target detection on the second thermal imaging video frame data using a pre-trained deep learning model to obtain position information of a preset type of target in the second thermal imaging video frame data; shielding a corresponding area in the third thermal imaging video frame data according to the first trajectory information and the position information of the target in the second thermal imaging video frame data to obtain shielded third thermal imaging video frame data, wherein the third thermal imaging video frame data is one or more frames of video frames in the common video frame; performing temperature detection on the shielded third thermal imaging video frame data to obtain a fire detection result.
[0096] In an embodiment of the present application, the area where moving targets and preset types of targets are located in the third thermal imaging video frame data is shielded, which can effectively shield high-temperature non-fire objects in the third thermal imaging video frame data, and perform temperature detection on the shielded third thermal imaging video frame data, thereby effectively reducing the impact of high-temperature non-fire objects on fire detection results and reducing false fire alarms.
[0097] The fire detection method in the embodiment of the present application is specifically described below. The fire detection method in the embodiment of the present application can be implemented by electronic devices such as smart phones, smart cameras, hard disk recorders, personal computers and servers.
[0098] See also Figure 1 , Figure 1 A schematic diagram of a fire detection method according to an embodiment of the present application, the method comprising:
[0099] S11, obtaining first thermal imaging video frame data and second thermal imaging video frame data of a monitoring scene, wherein the first thermal imaging video frame data includes multiple frames of thermal imaging video frames, the second thermal imaging video frame data includes at least one frame of thermal imaging video frame, and the video frames in the first thermal imaging video frame data and the video frames in the second thermal imaging video frame data have common video frames.
[0100] The monitoring scene is a scene where fire detection is required, such as an indoor scene; in actual scenes, the real-time requirement for fire detection is very high, so the first thermal imaging video frame data and the second thermal imaging video frame data can be real-time data streams, and the real-time video frames in the first thermal imaging video frame data can be processed in real time according to the acquisition cycle.
[0101] The same video frames exist in the first thermal imaging video frame data and the second thermal imaging video frame data, which are hereinafter referred to as common video frames. For example, the first thermal imaging video frame data and the second thermal imaging video frame data may be completely identical video frames; or the second thermal imaging video frame data may be a subset of the first thermal imaging video frame data; or the first thermal imaging video frame data may be a subset of the second thermal imaging video frame data; or the second thermal imaging video frame data and the first thermal imaging video frame data have only a portion of the same video frames, while the other portion of the video frames are different, all of which are within the protection scope of the present application.
[0102] In a possible implementation, the acquiring of first thermal imaging video frame data and second thermal imaging video frame data of a monitoring scene includes:
[0103] Step 1: Obtain thermal imaging video data of the monitoring scene.
[0104] A data stream of thermal imaging video data collected by a thermal imaging camera in a monitoring scene is received in real time.
[0105] Step 2: decimating the thermal imaging video data according to a preset first decimation frame rate to obtain first thermal imaging video frame data.
[0106] Step three, extracting frames of the thermal imaging video data according to a preset second extraction frame rate to obtain second thermal imaging video frame data, wherein the settings of the preset first extraction frame rate and the preset second extraction frame rate satisfy that the first thermal imaging video frame data and the second thermal imaging video frame data have common video frames.
[0107] The preset first frame rate and the preset second frame rate can be customized according to actual conditions, but it is necessary to ensure that the first thermal imaging video frame data and the second thermal imaging video frame data have common video frames. In an example, the preset first frame rate is an integer multiple of the preset second frame rate, in which case the second thermal imaging video frame data is a subset of the first thermal imaging video frame data.
[0108] S12, performing background modeling and foreground detection on the first thermal imaging video frame data to obtain first trajectory information of the moving target.
[0109] Background modeling and foreground detection are performed on the first thermal imaging video frame data, so as to obtain the foreground area of the first thermal imaging video frame data; moving target detection is performed on the foreground area of the first thermal imaging video frame data to obtain first trajectory information of the moving target in the first thermal imaging video frame data.
[0110] After the real fire object in the monitoring scene is detected as the foreground, the size and position of its target frame will not change much, so the real fire object will not be judged as a moving target.
[0111] S13: Use a pre-trained deep learning model to perform target detection on the second thermal imaging video frame data to obtain position information of a preset type of target in the second thermal imaging video frame data.
[0112] The preset type of target is a high-temperature non-fire object that may exist in the monitoring scene, for example, a forklift in an indoor warehouse scene, or the heat dissipation port of an air conditioner or server in a computer room scene. The preset type of target can be foreseen in advance or determined during the actual monitoring process of the monitoring scene. The deep learning model is trained using sample thermal imaging images marked with preset types of targets to obtain a pre-trained deep learning model. The specific training process of the deep learning model can refer to the training process of the relevant deep learning model for target detection, which is not specifically limited here.
[0113] In the embodiments of the present application, the execution order of S12 and S13 is not limited. S12 may be executed first and then S13, S13 may be executed first and then S12, or S12 and S13 may be executed simultaneously, all of which are within the protection scope of the present application.
[0114] S14, according to the first trajectory information and the position information of the target in the second thermal imaging video frame data, the corresponding area in the third thermal imaging video frame data is shielded to obtain the shielded third thermal imaging video frame data, wherein the third thermal imaging video frame data is one or more video frames in the common video frame.
[0115] The third thermal imaging video frame data is one or more video frames in the common video frame. In one example, the third thermal imaging video frame data may be a set of common video frames. In one example, the second thermal imaging video frame data may be a subset of the first thermal imaging video frame data, and the third thermal imaging video frame data may be the same as the second thermal imaging video frame data. In one example, the common video frame may be frame extracted to obtain the third thermal imaging video frame data. In a possible implementation, before step S14, the method further includes: selecting video frames with the same timestamp and / or frame number from each video frame of the first thermal imaging video frame data and each video frame of the second thermal imaging video frame data to obtain the common video frame; and extracting the common video frame according to the preset third frame extraction frame rate to obtain the third thermal imaging video frame data. The preset third frame extraction frame rate may be set according to the situation, for example, it may be set to 4FPS (Frame Per Second, frames per second) or 5FPS, etc.
[0116] The first trajectory information represents the position area of the moving target in the first thermal imaging video frame data at multiple moments, the third thermal imaging video frame data is a subset of the first thermal imaging video frame data, and the third thermal imaging video frame data is a subset of the second thermal imaging video frame data, then the "position area of the moving target in the third thermal imaging video frame data" and the "position area of the preset type of target in the third thermal imaging video frame data" are the corresponding areas in the third thermal imaging video frame data. The corresponding areas in the third thermal imaging video frame data are shielded, and the shielded third thermal imaging video frame data is called "shielded third thermal imaging video frame data", and the shielded third thermal imaging video frame data includes a shielded area and an unshielded area; in an example, the corresponding area in the third thermal imaging video frame data can be shielded by using a preset grayscale color, so as to achieve shielding; the temperature represented by the preset grayscale color is less than the preset temperature threshold.
[0117] S15, performing temperature detection on the shielded third thermal imaging video frame data to obtain a fire detection result.
[0118] The shielded third thermal imaging video frame data shields the area where the moving target and the preset type of target are located. When performing temperature detection, the shielded area will not be detected, or the temperature detected in the shielded area is less than the preset temperature threshold. In short, the shielded area will not detect the fire, while the unshielded area in the shielded third thermal imaging video frame data can detect the fire normally.
[0119] In some scenarios, although background modeling and foreground detection are performed on the third thermal imaging video frame data, and target detection is performed on the third thermal imaging video frame data using a pre-trained deep learning model, because there are no moving targets and preset types of targets in the monitoring scene, in this case, there is no need to shield the third thermal imaging video frame data, and temperature detection can be performed directly on the third thermal imaging video frame data.
[0120] In the embodiment of the present application, the area where the moving target and the preset type of target are located in the thermal imaging video frame data is shielded, which can effectively shield the high-temperature non-fire objects in the thermal imaging video frame data, and perform temperature detection on the shielded thermal imaging video frame data, thereby effectively reducing the impact of high-temperature non-fire objects on the fire detection results, and reducing the false alarm of fire. In addition, the target detection based on the deep learning model can effectively detect the specified target that is more than half exposed in the monitoring scene, and for the running target that is more than half blocked in the monitoring scene, it can be effectively detected by background modeling and foreground detection, which makes up for the shortcomings of the deep learning model, thereby further improving the probability of detecting high-temperature non-fire objects, and ultimately reducing the false alarm of fire.
[0121] In one possible implementation, see Figure 2 , performing background modeling and foreground detection on the first thermal imaging video frame data to obtain first trajectory information of the moving target includes:
[0122] S121, performing background modeling and foreground detection on the first thermal imaging video frame data to obtain a foreground area of the first thermal imaging video frame data.
[0123] S122: Associating the position areas of the same target based on the foreground area of the first thermal imaging video frame data to obtain initial trajectory information of each target.
[0124] Target detection is performed on the foreground area of the first thermal imaging video frame data to obtain target frames of each target in the first thermal imaging video frame data. According to the target frames of each target in the first thermal imaging video frame data, target frames of the same target are associated to obtain initial trajectory information of each target.
[0125] For each target frame of the same target, the target frames of the target are associated in time sequence to obtain the initial trajectory information of the target. In the process of obtaining the initial trajectory information of the target, in an example, the position of the target at the next moment can be predicted by a related target tracking algorithm to obtain a predicted position, and the target frames of the same target can be associated based on the predicted position.
[0126] In some scenarios, due to the influence of environmental factors, the foreground may be missed. In a possible implementation, the foreground area based on the first thermal imaging video frame data is associated with the position area of the same target to obtain the initial trajectory information of each target, including:
[0127] Step 1: Based on the foreground area of the first thermal imaging video frame data, the target frames of the same target are associated to obtain the associated trajectory information of each target.
[0128] Step 2: For each target, according to the associated trajectory information of the target, the missing trajectory position in the associated trajectory information of the target is completed to obtain the initial trajectory information of the target.
[0129] In the embodiment of the present application, the missing trajectory positions in the associated trajectory information of the target are completed, so that relatively complete initial trajectory information of the target can be obtained.
[0130] It can be understood that the initial trajectory information of the target is a collection of the target position information, so no matter whether the target is moving or not, there will be initial trajectory information.
[0131] S123, performing moving target detection on the initial trajectory information of each target to obtain first trajectory information of the moving target.
[0132] After the initial trajectory information of each target is obtained, for each target, whether the target is a moving target or a stationary target can be determined according to the position change of the initial trajectory information of the target in time sequence, thereby obtaining the first trajectory information of the moving target.
[0133] In an example, for example Figure 3As shown, it includes: target detection: background modeling and foreground detection are performed on the first thermal imaging video frame data to obtain the foreground area of the first thermal imaging video frame data, and target detection is performed on the foreground area of the first thermal imaging video frame data to obtain the target frame of each target in the first thermal imaging video frame data. Target tracking: The target frames of the same target are associated through the relevant target tracking algorithm to obtain the initial trajectory information of each target. Target frame motion amount calculation: According to the temporal position change of the target frame in the initial trajectory information of the target, it is determined whether the current frame of the target moves compared with the previous frame. If movement occurs, the moving flg of the current frame is marked as 1, indicating that the target is a moving target in the current frame; if no movement occurs, the moving flg of the current frame is marked as 0, indicating that the target is a stationary target in the current frame. Target frame shielding processing judgment: It is used to judge whether the target output by the superior (target frame motion amount calculation) needs to be shielded, and the target frame that needs to be shielded is cached in the target cache pool. Subsequent processing: Each target has an ID. Based on the target ID, determine whether the target corresponding to the ID disappears in the current frame period. If it does not disappear, add the target frame of the target in the target cache pool to the first trajectory information of the target. If it disappears, determine whether the moving flg of the previous frame of the target is equal to 1. If it is not equal to 1, eliminate the target frame of the target in the target cache. If it is equal to 1, perform temperature-related tracking on the target. Specifically, temperature-related tracking can be: for moving targets whose target frames disappear, based on the target frame of the moving target in the cache, determine whether the temperature in the corresponding area of the video frame of the current period is greater than a preset temperature threshold. If it is greater than the preset temperature threshold, retain the target frame of the moving target in the cache and use it to shield the corresponding area of the current video frame until the temperature of the corresponding area in the real-time processed video frame is lower than the preset threshold, and then clear the target frame of the moving target in the cache.
[0134] In a possible implementation manner, performing moving target detection on the initial trajectory information of each target to obtain first trajectory information of the moving target includes:
[0135] Step 1: For each target, based on the initial trajectory information of the target, if the position change between the current target frame to be processed in the current cycle and the previous target frame processed in the previous cycle is greater than a preset distance threshold, the current target frame in the initial trajectory information of the target is marked with a moving mark; otherwise, the current target frame in the initial trajectory information of the target is marked with a stationary mark.
[0136] The preset distance threshold can be customized according to the actual monitoring scenario. The setting of the preset distance threshold needs to meet the following requirements: when the position change of the target frame of the same target in two adjacent cycles is greater than the preset distance threshold, the target is considered to have moved; when the position change of the target frame of the same target in two adjacent cycles is not greater than the preset distance threshold, the target is considered not to have moved.
[0137] Step 2: for each target, when the target frame identifier in the initial trajectory information of the target indicates that the target is in continuous motion, if a target frame mutation occurs between the current target frame of the target and the cached target frame of the target in the cache, the current target frame of the target is fused with the cached target frame of the target in the cache, and the cached target frame of the target in the cache is updated using the fused target frame; if a target frame mutation does not occur between the current target frame of the target and the cached target frame of the target in the cache, the cached target frame of the target in the cache is updated using the current target frame of the target; the target is marked as a moving target; wherein the target frame mutation refers to a change in the size of the target frame exceeding a preset change threshold.
[0138] Step three: for each target, when the target frame identifier in the initial trajectory information of the target indicates that the target is continuously stationary, the cached target frame of the target in the cache is updated using the current target frame of the target; and the target is marked as a stationary target.
[0139] Step 4: For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target changes from moving to stationary, the current target frame of the target is merged with the cached target frame of the target in the cache, and the cached target frame of the target in the cache is updated using the fused target frame; the target is marked as a moving target.
[0140] Step 5: for each target, when the target frame identifier in the initial trajectory information of the target indicates that the target changes from static to moving, the cached target frame of the target in the cache is updated using the current target frame of the target; and the target is marked as a moving target.
[0141] In an example, for example Figure 4 As shown, the motion flag is marked as moving flg=1, the stationary flag is marked as moving flg=0, the moving target is marked as shield fig=1, and the stationary target is marked as shield fig=0.
[0142] When a real fire object is detected as a foreground, the size and position of the target frame will not change much, so its motion flag moving flg is 0->0. Targets in this state are not shielded, reducing the risk of missing fire objects.
[0143] When a moving target (such as a forklift) goes from moving to stationary, the target gradually blends into the background, the target frame gradually decreases, or even disappears. Figure 5 As shown, its movement flag moving flg is 1->0; in this case, the current target frame of the target is fused with the cached target frame of the target in the cache, and the fused target frame is saved in the target cache pool (cache) (until the temperature of the area is lower than the threshold), so that a complete target frame of the moving target can be obtained, the detection rate of the moving target is improved, and the false alarm of fire caused by the high temperature point of the moving target outside the target frame of the moving target due to the target frame of the moving target failing to frame the complete moving target is reduced, thereby reducing the false alarm of fire.
[0144] Step six: for each moving target, use the cached target frame of the moving target in the cache as the first trajectory information of the moving target in the current cycle.
[0145] After a moving target stops moving or a fire target appears for a period of time, it will gradually blend into the background, causing the target frame to gradually decrease or even disappear. In one possible implementation, the method further includes: for a target whose target frame disappears in the current cycle (hereinafter referred to as a disappeared target), determining whether the disappeared target is a moving target; if the disappeared target is not a moving target, eliminating the cached target frame of the disappeared target in the cache. If the disappeared target is a moving target, according to the position of the cached target frame of the disappeared target in the cache, determine whether the temperature at the corresponding area in the thermal imaging video frame of the current cycle is greater than a preset temperature threshold; if it is greater than the preset temperature threshold, retain the cached target frame of the disappeared target in the cache; if it is less than the preset temperature threshold, clear the cached target frame of the disappeared target in the cache.
[0146] If there is a disappeared target in the current cycle compared with the previous cycle, the initial trajectory information of the disappeared target can be used to determine whether the target is a moving target or a stationary target. If the disappeared target is a stationary target, the target frame of the disappeared target in the cache is cleared. If the disappeared target is a moving target, according to the position of the cached target frame of the disappeared target in the cache, it is determined whether the temperature of the corresponding area in the thermal imaging video frame of the current cycle is greater than the preset temperature threshold; if it is less than the preset temperature threshold, the cached target frame of the disappeared target in the cache is cleared. If it is greater than the preset temperature threshold, the target frame of the disappeared target in the cache is retained, and the cached target frame of the disappeared target in the cache is used as the first trajectory information of the disappeared target in the current cycle.
[0147] In an embodiment of the present application, when the target frame of the target changes suddenly or the target changes from moving to stationary, the current target frame of the target is merged with the cached target frame of the target in the cache, so that a complete target frame of the moving target can be obtained, thereby improving the detection rate of the moving target and reducing the situation where the target frame of the moving target cannot fully frame the complete moving target, resulting in the high temperature point of the moving target being outside the target frame of the moving target, causing false fire alarms, thereby reducing the situation of false fire alarms.
[0148] Although the target frame after fusion processing can improve the detection rate of moving targets, there is a risk that the target frame will be too large (tailing), for example Figure 6 As shown in the white target frame in the middle, if the target frame is too large, the shielded area will exceed the area of the moving target itself, thereby causing the fire to be missed. In a possible implementation, the current target frame of the target is merged with the cached target frame of the target in the cache, and the cached target frame of the target in the cache is updated using the fused target frame, including:
[0149] Step A, according to a preset fusion smoothing coefficient, selects a target frame area of corresponding size from the cached target frame of the target in the cache, and fuses the selected target frame area with the current target frame of the target to obtain a fused target frame of the target, wherein the fusion smoothing coefficient represents the size of the target frame area selected from the cached target frame.
[0150] The fusion smoothing coefficient represents the size of the target frame area selected from the cached target frame. In one example, within the value range of the fusion smoothing coefficient, the larger the fusion smoothing coefficient is, the larger the target frame area selected from the cached target frame is. For example, the value range of the fusion smoothing coefficient is 0-1, and the fusion smoothing coefficient I can be: I = 1-a / b, where I represents the fusion smoothing coefficient, a represents the size of the target frame area, and b represents the size of the cached target frame. That is, when the fusion smoothing coefficient is 1, the target frame area is not selected from the cached target frame, and the current target frame of the target is the fused target frame of the target; when the fusion smoothing coefficient is 0, the entire cached target frame is selected as the target frame area, and the union of the current target frame of the target and the cached target frame of the target is used as the fused target frame of the target.
[0151] The user can also set the fusion sensitivity level according to the actual situation. Different sensitivity levels have different processing methods for the third thermal imaging video frame data. In one example, as shown in Table 1, the fusion sensitivity can be set to 5 levels (0-4) according to the strategy logic, where when the fusion sensitivity is level 0, background modeling, foreground detection and shielding of the area corresponding to the first trajectory information are not performed, but the corresponding area in the third thermal imaging video frame data is shielded according to the position information of the preset type of target in the second thermal imaging video frame data; when the fusion sensitivity is level 1-4, the corresponding area in the first thermal imaging video frame data is shielded according to the first trajectory information and the position information of the preset type of target in the second thermal imaging video frame data; in addition, when the fusion sensitivity is level 3-4, the temperature correlation tracking process is also performed.
[0152] When the fusion sensitivity is level 1, the corresponding fusion smoothing coefficient is 1. In this case, the current target frame of the target is the target frame after fusion. When the fusion sensitivity is level 4, the corresponding fusion smoothing coefficient is 0. In this case, the union of the current target frame of the target and the cached target frame of the target is used as the target frame after fusion (corresponding to a strict strategy). Sensitivities 2 and 3 can adjust the smoothing coefficient to achieve different fusion filtering effects. Among them, the strategy of time domain sliding window refers to: selecting part of the target frame area in the cached target frame of the target and taking the union with the current target frame of the target as the target frame after fusion.
[0153] In an example, the fusion smoothing coefficient can be set to 0.2. When the fusion smoothing coefficient is 0.2, the target box is as follows Figure 7 As shown, Figure 7 and Figure 6 In comparison, the phenomenon of the white target frame being too large (tailing) has been significantly improved, reducing the risk of missing fire reports.
[0154] Table 1
[0155]
[0156]
[0157] Step B: updating the cached target frame of the target in the cache to the fused target frame of the target.
[0158] In the embodiment of the present application, the fusion smoothing coefficient is used to effectively improve the phenomenon of the target frame being too large (tailing), which can reduce the situation where the third thermal imaging video frame data is mistakenly shielded due to the large target frame, thereby reducing the risk of missed fire reports.
[0159] Typically, the frame rate of the thermal imaging video frames used for temperature detection is lower than the frame rate of the first thermal imaging video frame data. Therefore, in order to save computing resources, target detection does not need to be performed on all thermal imaging video frames in the first thermal imaging video frame data.
[0160] In one possible implementation, see Figure 8 After performing target detection on the second thermal imaging video frame data using the pre-trained deep learning model to obtain position information of a preset type of target in the second thermal imaging video frame data, the method further includes:
[0161] S81, for each preset type of target, according to the position information of the target in the second thermal imaging video frame data, various position information of the target are associated to obtain associated trajectory information of the target.
[0162] S82, for each preset type of target, if there is a video frame with a missing trajectory of the target in the second thermal imaging video frame data, determine the position information of the target in the video frame with a missing trajectory according to the associated trajectory information of the target, and complete the associated trajectory information of the target based on the position information of the target in the video frame with a missing trajectory to obtain the second trajectory information of the target.
[0163] There may be a process of missed detection in target detection based on deep learning models. For example, if target A is detected in the 1st to 3rd video frames and also in the 5th to 8th video frames, it can be determined that target A is missed in the 4th video frame. For any preset type of target, the video frame in which the target is missed in the second thermal imaging video frame data is called a track-missing video frame. Based on the associated track information of the target, the position information of the target in the track-missing video frame of the target can be calculated through a related algorithm, so that the position information of the target in the track-missing video frame of the target can be used to complete the associated track information of the target, and the completed associated track information of the target is called the second track information of the target.
[0164] It can be understood that the second trajectory information of the target is a collection of the target position information, so the second trajectory information exists regardless of whether the target is moving or not.
[0165] The method of shielding a corresponding area in the third thermal imaging video frame data according to the first trajectory information and the position information of the target in the second thermal imaging video frame data to obtain the shielded third thermal imaging video frame data includes:
[0166] S141, shielding a corresponding area in the third thermal imaging video frame data according to the first trajectory information and the second trajectory information to obtain shielded third thermal imaging video frame data.
[0167] In the embodiment of the present application, the associated trajectory information of the target in the second thermal imaging video frame data is completed to obtain the complete trajectory information of the preset type of target.
[0168] The present application also provides a fire detection device. Fig. 9 , the device comprises:
[0169] The video data acquisition module 901 is used to acquire first thermal imaging video frame data and second thermal imaging video frame data of a monitoring scene, wherein the first thermal imaging video frame data includes multiple frames of thermal imaging video frames, the second thermal imaging video frame data includes at least one frame of thermal imaging video frame, and the video frames in the first thermal imaging video frame data and the video frames in the second thermal imaging video frame data have common video frames;
[0170] A moving target trajectory acquisition module 902 is used to perform background modeling and foreground detection on the first thermal imaging video frame data to obtain first trajectory information of the moving target;
[0171] A target trajectory acquisition module 903 is used to perform target detection on the second thermal imaging video frame data using a pre-trained deep learning model to obtain position information of a preset type of target in the second thermal imaging video frame data;
[0172] The thermal imaging video data masking module 904 is used to mask the corresponding area in the third thermal imaging video frame data according to the first trajectory information and the position information of the target in the second thermal imaging video frame data, so as to obtain the masked third thermal imaging video frame data, wherein the third thermal imaging video frame data is one or more video frames in the common video frame;
[0173] The fire detection module 905 is used to perform temperature detection on the shielded third thermal imaging video frame data to obtain a fire detection result.
[0174] In a possible implementation, the moving target trajectory acquisition module includes:
[0175] a foreground area acquisition submodule, configured to perform background modeling and foreground detection on the first thermal imaging video frame data to obtain a foreground area of the first thermal imaging video frame data;
[0176] An initial trajectory information acquisition submodule associates the position areas of the same target based on the foreground area of the first thermal imaging video frame data to obtain the initial trajectory information of each target;
[0177] The first trajectory information acquisition submodule is used to perform moving target detection on the initial trajectory information of each target to obtain the first trajectory information of the moving target.
[0178] In a possible implementation, the initial trajectory information acquisition submodule is specifically used to: associate target frames of the same target based on the foreground area of the first thermal imaging video frame data to obtain associated trajectory information of each target; for each target, according to the associated trajectory information of the target, complete the missing trajectory position in the associated trajectory information of the target to obtain the initial trajectory information of the target.
[0179] In a possible implementation manner, the first trajectory information acquisition submodule is specifically configured to:
[0180] For each target, based on the initial trajectory information of the target, if the position change between the current target frame to be processed in the current cycle and the previous target frame processed in the previous cycle is greater than a preset distance threshold, the current target frame in the initial trajectory information of the target is marked with a moving mark, otherwise the current target frame in the initial trajectory information of the target is marked with a stationary mark;
[0181] For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target is in continuous motion, if the current target frame of the target undergoes a target frame mutation compared with the cached target frame of the target in the cache, the current target frame of the target is fused with the cached target frame of the target in the cache, and the cached target frame of the target in the cache is updated using the fused target frame; if the current target frame of the target does not undergo a target frame mutation compared with the cached target frame of the target in the cache, the cached target frame of the target in the cache is updated using the current target frame of the target; the target is marked as a moving target; wherein the target frame mutation refers to a change in the size of the target frame exceeding a preset change threshold;
[0182] For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target is continuously stationary, the cached target frame of the target in the cache is updated using the current target frame of the target; and the target is marked as a stationary target;
[0183] For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target changes from moving to stationary, the current target frame of the target is merged with the cached target frame of the target in the cache, and the cached target frame of the target in the cache is updated using the fused target frame; the target is marked as a moving target;
[0184] For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target changes from being stationary to moving, the cached target frame of the target in the cache is updated using the current target frame of the target; and the target is marked as a moving target;
[0185] For each moving target, the cached target frame of the moving target in the cache is used as the first trajectory information of the moving target in the current cycle.
[0186] In a possible implementation manner, the first trajectory information acquisition submodule is specifically configured to:
[0187] According to a preset fusion smoothing coefficient, a target frame area of a corresponding size is selected from the cached target frame of the target in the cache, and the selected target frame area is fused with the current target frame of the target to obtain a fused target frame of the target, wherein the fusion smoothing coefficient represents the size of the target frame area selected from the cached target frame;
[0188] Update the cached target frame of the target in the cache to the fused target frame of the target.
[0189] In a possible implementation, the first trajectory information acquisition submodule is further configured to: for a disappeared target whose target frame disappears in a current cycle, determine whether the disappeared target is a moving target; if the disappeared target is not a moving target, then delete the cached target frame of the disappeared target in the cache.
[0190] In a possible implementation, the first trajectory information acquisition submodule is further configured to: if the disappeared target is a moving target, determine whether the temperature at the corresponding area in the thermal imaging video frame of the current period is greater than a preset temperature threshold according to the position of the cached target frame of the disappeared target in the cache; if the temperature is greater than the preset temperature threshold, retain the cached target frame of the disappeared target in the cache; if the temperature is less than the preset temperature threshold, clear the cached target frame of the disappeared target in the cache.
[0191] In a possible implementation, the video data acquisition module is specifically used to: acquire thermal imaging video data of a monitoring scene; perform frame extraction on the thermal imaging video data according to a preset first frame extraction frame rate to obtain first thermal imaging video frame data; perform frame extraction on the thermal imaging video data according to a preset second frame extraction frame rate to obtain second thermal imaging video frame data, wherein the settings of the preset first frame extraction frame rate and the preset second frame extraction frame rate satisfy that the first thermal imaging video frame data and the second thermal imaging video frame data have common video frames.
[0192] In a possible implementation, the device further includes:
[0193] The second trajectory information acquisition module is used to associate each position information of each preset type of target according to the position information of the target in the second thermal imaging video frame data to obtain the associated trajectory information of the target; for each preset type of target, if there is a trajectory missing video frame of the target in the second thermal imaging video frame data, then according to the associated trajectory information of the target, determine the position information of the target in the trajectory missing video frame of the target, and based on the position information of the target in the trajectory missing video frame of the target, complete the associated trajectory information of the target to obtain the second trajectory information of the target
[0194] The thermal imaging video data masking module is specifically used to: mask the corresponding area in the third thermal imaging video frame data according to the first trajectory information and the second trajectory information to obtain the masked third thermal imaging video frame data.
[0195] In a possible implementation, the device further includes:
[0196] A frame extraction module is used to select video frames with the same timestamp and / or frame number from each video frame of the first thermal imaging video frame data and each video frame of the second thermal imaging video frame data to obtain the common video frame; and extract the common video frame according to a preset third frame extraction frame rate to obtain the third thermal imaging video frame data.
[0197] The embodiment of the present application also provides an electronic device, including: a processor and a memory;
[0198] The memory is used to store computer programs;
[0199] When the processor is used to execute the computer program stored in the memory, any fire detection method described in the present application is implemented.
[0200] Optionally, the electronic device of the embodiment of the present application further includes a communication interface and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus.
[0201] The communication bus mentioned in the above electronic device may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0202] The communication interface is used for communication between the above electronic device and other devices.
[0203] The memory may include RAM (Random Access Memory) or NVM (Non-Volatile Memory), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0204] The above-mentioned processor can be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), etc.; it can also be a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0205] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any fire detection method described in the present application is implemented.
[0206] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any fire detection method described in the present application.
[0207] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.
[0208] It should be noted that, in this article, the technical features in each optional solution can be combined to form a solution as long as there is no contradiction, and these solutions are all within the scope disclosed in this application. Relational terms such as first and second, etc. are only used to distinguish an entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only include those elements, but also include other elements that are not clearly listed, or also include elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.
[0209] Each embodiment in this specification is described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other.
[0210] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. A fire detection method, characterized in that: The method comprises: Acquire first thermal imaging video frame data and second thermal imaging video frame data of a monitoring scene, wherein the first thermal imaging video frame data includes a plurality of thermal imaging video frames, the second thermal imaging video frame data includes at least one thermal imaging video frame, and the video frames in the first thermal imaging video frame data and the video frames in the second thermal imaging video frame data have common video frames; Performing background modeling and foreground detection on the first thermal imaging video frame data to obtain first trajectory information of the moving target; Performing target detection on the second thermal imaging video frame data using a pre-trained deep learning model to obtain position information of a preset type of target in the second thermal imaging video frame data; According to the first trajectory information and the position information of the target in the second thermal imaging video frame data, a corresponding area in the third thermal imaging video frame data is shielded to obtain shielded third thermal imaging video frame data, wherein the third thermal imaging video frame data is one or more video frames in the common video frame; Temperature detection is performed on the shielded third thermal imaging video frame data to obtain a fire detection result.
2. The method according to claim 1, characterized in that The performing background modeling and foreground detection on the first thermal imaging video frame data to obtain first trajectory information of the moving target includes: Performing background modeling and foreground detection on the first thermal imaging video frame data to obtain a foreground area of the first thermal imaging video frame data; Based on the foreground area of the first thermal imaging video frame data, associating the position areas of the same target to obtain initial trajectory information of each target; Moving target detection is performed on the initial trajectory information of each target to obtain first trajectory information of the moving target.
3. The method according to claim 2, characterized in that The method of associating the position areas of the same target based on the foreground area of the first thermal imaging video frame data to obtain the initial trajectory information of each target includes: Based on the foreground area of the first thermal imaging video frame data, the target frames of the same target are associated to obtain associated trajectory information of each of the targets; For each target, according to the associated trajectory information of the target, the missing trajectory position in the associated trajectory information of the target is completed to obtain the initial trajectory information of the target.
4. The method according to claim 2, characterized in that: The step of performing moving target detection on the initial trajectory information of each target to obtain first trajectory information of the moving target includes: For each target, based on the initial trajectory information of the target, if the position change between the current target frame to be processed in the current cycle and the previous target frame processed in the previous cycle is greater than a preset distance threshold, the current target frame in the initial trajectory information of the target is marked with a moving mark, otherwise the current target frame in the initial trajectory information of the target is marked with a stationary mark; For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target is in continuous motion, if the current target frame of the target undergoes a target frame mutation compared with the cached target frame of the target in the cache, the current target frame of the target is fused with the cached target frame of the target in the cache, and the cached target frame of the target in the cache is updated using the fused target frame; if the current target frame of the target does not undergo a target frame mutation compared with the cached target frame of the target in the cache, the cached target frame of the target in the cache is updated using the current target frame of the target; the target is marked as a moving target; wherein the target frame mutation refers to a change in the size of the target frame exceeding a preset change threshold; For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target is continuously stationary, the cached target frame of the target in the cache is updated using the current target frame of the target; and the target is marked as a stationary target; For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target changes from moving to stationary, the current target frame of the target is merged with the cached target frame of the target in the cache, and the cached target frame of the target in the cache is updated using the fused target frame; the target is marked as a moving target; For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target changes from being stationary to moving, the cached target frame of the target in the cache is updated using the current target frame of the target; and the target is marked as a moving target; For each moving target, the cached target frame of the moving target in the cache is used as the first trajectory information of the moving target in the current cycle.
5. The method according to claim 4, characterized in that The step of fusing the current target frame of the target with the cached target frame of the target in the cache, and updating the cached target frame of the target in the cache using the fused target frame, includes: According to a preset fusion smoothing coefficient, a target frame area of a corresponding size is selected from the cached target frame of the target in the cache, and the selected target frame area is fused with the current target frame of the target to obtain a fused target frame of the target, wherein the fusion smoothing coefficient represents the size of the target frame area selected from the cached target frame; Update the cached target frame of the target in the cache to the fused target frame of the target.
6. The method according to claim 4, characterized in that Before taking the cached target frame of the moving target in the cache as the first trajectory information of the moving target in the current cycle for each moving target, the method further includes: For a disappeared target whose target frame disappears in the current cycle, determining whether the disappeared target is a moving target; If the disappeared target is not a moving target, the cached target frame of the disappeared target is deleted from the cache.
7. The method according to claim 6, characterized in that The method further comprises: If the disappeared target is a moving target, judging whether the temperature of a corresponding area in a thermal imaging video frame of a current period is greater than a preset temperature threshold according to the position of the cached target frame of the disappeared target in the cache; If it is greater than a preset temperature threshold, the cached target frame of the disappeared target in the cache is retained; If the temperature is less than the preset temperature threshold, the cached target box of the disappeared target in the cache is cleared.
8. The method according to claim 1, characterized in that: The step of acquiring first thermal imaging video frame data and second thermal imaging video frame data of the monitoring scene includes: Acquire thermal imaging video data of the monitoring scene; According to a preset first frame rate, the thermal imaging video data is frame-decimated to obtain first thermal imaging video frame data; The thermal imaging video data is framed according to a preset second frame rate to obtain second thermal imaging video frame data, wherein the settings of the preset first frame rate and the preset second frame rate satisfy that the first thermal imaging video frame data and the second thermal imaging video frame data have common video frames.
9. The method according to claim 1, characterized in that: After performing target detection on the second thermal imaging video frame data using the pre-trained deep learning model to obtain position information of a preset type of target in the second thermal imaging video frame data, the method further includes: For each preset type of target, the respective position information of the target is associated according to the position information of the target in the second thermal imaging video frame data to obtain associated trajectory information of the target; For each preset type of target, if there is a track-missing video frame of the target in the second thermal imaging video frame data, determine the position information of the target in the track-missing video frame according to the associated track information of the target, and complete the associated track information of the target based on the position information of the target in the track-missing video frame of the target to obtain the second track information of the target; The method of shielding a corresponding area in the third thermal imaging video frame data according to the first trajectory information and the position information of the target in the second thermal imaging video frame data to obtain the shielded third thermal imaging video frame data includes: According to the first trajectory information and the second trajectory information, the corresponding area in the third thermal imaging video frame data is shielded to obtain the shielded third thermal imaging video frame data.
10. The method according to claim 1, characterized in that Before shielding the corresponding area in the third thermal imaging video frame data according to the first trajectory information and the position information of the target in the second thermal imaging video frame data to obtain the shielded third thermal imaging video frame data, the method further includes: Selecting video frames with the same timestamp and / or frame number from each video frame of the first thermal imaging video frame data and each video frame of the second thermal imaging video frame data to obtain the common video frame; The common video frame is subjected to frame extraction according to a preset third frame extraction rate to obtain third thermal imaging video frame data.
11. A fire detection device, characterized in that: The device comprises: A video data acquisition module, used to acquire first thermal imaging video frame data and second thermal imaging video frame data of a monitoring scene, wherein the first thermal imaging video frame data includes a plurality of thermal imaging video frames, the second thermal imaging video frame data includes at least one thermal imaging video frame, and the video frames in the first thermal imaging video frame data and the video frames in the second thermal imaging video frame data have common video frames; A moving target trajectory acquisition module, used to perform background modeling and foreground detection on the first thermal imaging video frame data to obtain first trajectory information of the moving target; a target trajectory acquisition module, configured to perform target detection on the second thermal imaging video frame data using a pre-trained deep learning model, and obtain position information of a preset type of target in the second thermal imaging video frame data; a thermal imaging video data masking module, configured to mask a corresponding area in a third thermal imaging video frame data according to the first trajectory information and the position information of the target in the second thermal imaging video frame data, so as to obtain masked third thermal imaging video frame data, wherein the third thermal imaging video frame data is one or more video frames in the common video frame; The fire detection module is used to perform temperature detection on the shielded third thermal imaging video frame data to obtain a fire detection result.
12. The device according to claim 11, characterized in that The moving target trajectory acquisition module includes: a foreground area acquisition submodule, configured to perform background modeling and foreground detection on the first thermal imaging video frame data to obtain a foreground area of the first thermal imaging video frame data; An initial trajectory information acquisition submodule associates the position areas of the same target based on the foreground area of the first thermal imaging video frame data to obtain the initial trajectory information of each target; A first trajectory information acquisition submodule is used to perform moving target detection on the initial trajectory information of each target to obtain first trajectory information of the moving target; The initial trajectory information acquisition submodule is specifically used to: associate the target frames of the same target based on the foreground area of the first thermal imaging video frame data to obtain the associated trajectory information of each target; for each target, according to the associated trajectory information of the target, complete the missing trajectory position in the associated trajectory information of the target to obtain the initial trajectory information of the target; The first trajectory information acquisition submodule is specifically used to: For each target, based on the initial trajectory information of the target, if the position change between the current target frame to be processed in the current cycle and the previous target frame processed in the previous cycle is greater than a preset distance threshold, the current target frame in the initial trajectory information of the target is marked with a moving mark, otherwise the current target frame in the initial trajectory information of the target is marked with a stationary mark; For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target is in continuous motion, if the current target frame of the target undergoes a target frame mutation compared with the cached target frame of the target in the cache, the current target frame of the target is fused with the cached target frame of the target in the cache, and the cached target frame of the target in the cache is updated using the fused target frame; if the current target frame of the target does not undergo a target frame mutation compared with the cached target frame of the target in the cache, the cached target frame of the target in the cache is updated using the current target frame of the target; the target is marked as a moving target; wherein the target frame mutation refers to a change in the size of the target frame exceeding a preset change threshold; For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target is continuously stationary, the cached target frame of the target in the cache is updated using the current target frame of the target; and the target is marked as a stationary target; For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target changes from moving to stationary, the current target frame of the target is merged with the cached target frame of the target in the cache, and the cached target frame of the target in the cache is updated using the fused target frame; the target is marked as a moving target; For each target, when the target frame identifier in the initial trajectory information of the target indicates that the target changes from being stationary to moving, the cached target frame of the target in the cache is updated using the current target frame of the target; and the target is marked as a moving target; For each moving target, using the cached target frame of the moving target in the cache as the first trajectory information of the moving target in the current cycle; The first trajectory information acquisition submodule is specifically used to: select a target frame area of corresponding size from the cached target frame of the target in the cache according to a preset fusion smoothing coefficient, and fuse the selected target frame area with the current target frame of the target to obtain a fused target frame of the target, wherein the fusion smoothing coefficient represents the size of the target frame area selected from the cached target frame; update the cached target frame of the target in the cache to the fused target frame of the target; The first trajectory information acquisition submodule is further used to: for a disappeared target whose target frame disappears in the current cycle, determine whether the disappeared target is a moving target; if the disappeared target is not a moving target, then delete the cached target frame of the disappeared target in the cache; The first trajectory information acquisition submodule is further used to: if the disappeared target is a moving target, determine whether the temperature at the corresponding area in the thermal imaging video frame of the current period is greater than a preset temperature threshold according to the position of the cached target frame of the disappeared target in the cache; if it is greater than the preset temperature threshold, retain the cached target frame of the disappeared target in the cache; if it is less than the preset temperature threshold, clear the cached target frame of the disappeared target in the cache; The video data acquisition module is specifically used to: acquire thermal imaging video data of a monitoring scene; extract frames of the thermal imaging video data according to a preset first extraction frame rate to obtain first thermal imaging video frame data; extract frames of the thermal imaging video data according to a preset second extraction frame rate to obtain second thermal imaging video frame data, wherein the settings of the preset first extraction frame rate and the preset second extraction frame rate satisfy that the first thermal imaging video frame data and the second thermal imaging video frame data have common video frames; The device further includes: a second trajectory information acquisition module, which is used to associate each position information of each preset type of target according to the position information of the target in the second thermal imaging video frame data to obtain the associated trajectory information of the target; for each preset type of target, if there is a trajectory-missing video frame of the target in the second thermal imaging video frame data, determine the position information of the target in the trajectory-missing video frame according to the associated trajectory information of the target, and complete the associated trajectory information of the target based on the position information of the target in the trajectory-missing video frame of the target to obtain the second trajectory information of the target; The thermal imaging video data masking module is specifically used to: mask the corresponding area in the third thermal imaging video frame data according to the first trajectory information and the second trajectory information to obtain the masked third thermal imaging video frame data; The device also includes: a frame extraction module, which is used to select video frames with the same timestamp and / or frame number from each video frame of the first thermal imaging video frame data and each video frame of the second thermal imaging video frame data to obtain the common video frame; and extract the common video frame according to a preset third frame extraction frame rate to obtain third thermal imaging video frame data.
