Firework recognition method and device, electronic equipment and storage medium

Abstract

The application provides a firework recognition method and device, electronic equipment and storage medium, wherein the method comprises: obtaining all first moment pictures corresponding to a target place from all cameras and receiving a position relationship corresponding to the all cameras sent by a user; generating a full scene graph of the target place according to the all first moment pictures and the position relationship; obtaining a second moment picture from the all cameras and generating a target detection area and a target segmentation area according to the second moment picture; determining whether there is a flame in the target place according to the target detection area and a thermal imaging view in the all cameras; if yes, determining again whether there is a flame in the target place according to the target segmentation area and the full scene graph. By comparing the actual pixel point coordinates of the target with the actual coordinates of the corresponding pixel points in the coordinate library and further determining through the thermal imaging image, the false detection rate of the firework recognition is effectively reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, in particular to a firework recognition generation method and device, electronic equipment and storage medium. BACKGROUND

[0002] In real life, fire prevention is an important matter. For example, if a fire breaks out near a gas station, it is easy to cause an explosion and other accidents, resulting in casualties. Therefore, it is particularly important to identify and monitor fires in a timely manner to avoid major accidents. When a fire is detected, real-time alarms should be given to inform gas station personnel to take necessary measures in a timely manner to reduce losses. Therefore, there is an urgent need for a gas station fire identification method algorithm to solve this market problem. In the past few years, target detection algorithms based on deep learning have received more attention in research. With the development of artificial intelligence and the development of some target detection algorithms based on deep convolutional neural networks (Convolutional Neural Networks), some algorithms stand out in the field of computer vision. Currently, target detection algorithms are commonly used to identify whether the anti-static chain is connected. For example, classic target detection networks such as SDD, RCNN, FasterRCNN, and YOLO series. The detection method based on deep learning has good results in normal server and indoor environments, but for outdoor environments such as gas stations, the detection effect is poor due to the influence of sunlight, car light, and other situations. The ground and walls of the gas station reflect light, often causing false positives. Moreover, the required computing power of these algorithms is also very high. Currently, most gas station detection equipment is edge terminal equipment, which is limited by computing power and cannot be directly deployed on edge terminal equipment. SUMMARY

[0003] Therefore, it is necessary to provide a firework identification method, device, electronic equipment and storage medium capable of accurately identifying firework and accurately alarming in view of the above technical problems.

[0004] In a first aspect, a firework identification method is provided, the method comprising:

[0005] obtaining all first time pictures corresponding to a target location from all cameras and receiving a position relationship corresponding to the all cameras sent by a user;

[0006] generating a full scene graph of the target location according to the all first time pictures and the position relationship;

[0007] obtaining a second time picture from the all cameras and generating a target detection region and a target segmentation region according to the second time picture;

[0008] determining whether a flame exists in the target location according to the target detection region and a thermal imaging view in the all cameras;

[0009] If yes, it is determined again whether there is fire in the target location according to the target segmentation region and the full scene graph.

[0010] In one of the embodiments, the generating the full scene graph of the target location according to the all first time pictures and the position relationship comprises:

[0011] performing gray processing, Gaussian blur processing and local binary feature processing on the all first time pictures;

[0012] performing registration processing on the all first time pictures and generating the full scene graph;

[0013] calculating point coordinates of all pixel points in the full scene graph according to the position relationship and storing the point coordinates in a coordinate database.

[0014] In one of the embodiments, the acquiring second time pictures from the all cameras and generating a target detection region and a target segmentation region according to the second time pictures comprises:

[0015] performing preprocessing and the local binary feature processing on the second time pictures and inputting the second time pictures to a neural network;

[0016] inputting the output second time pictures to a feature network and outputting a first feature map;

[0017] inputting the first feature map to a region production network and outputting a target recommendation region;

[0018] inputting the first feature map and the target recommendation region to an interesting region arrangement network and outputting a second feature map;

[0019] inputting the second feature map to a prediction layer and outputting the target detection region and the target segmentation region.

[0020] In one of the embodiments, the determining whether there is fire in the target location according to the target detection region and the thermal imaging view in the all cameras comprises:

[0021] inputting the target detection region to a full connection layer and outputting a target frame, a target classification and a target score;

[0022] determining whether there is fire in the target frame according to the target classification and the target score;

[0023] If no, the second time pictures are re-acquired from the all cameras;

[0024] If yes, it is determined whether there is fire in the target frame according to the thermal imaging view.

[0025] In one embodiment, the determining whether the flame exists in the target frame according to the thermal imaging view comprises:

[0026] acquiring an object temperature corresponding to an object in the target frame according to the thermal imaging view;

[0027] when the object temperature is greater than a temperature threshold set by a user, determining that the flame exists in the target frame.

[0028] In one embodiment, the determining whether the flame exists in the target location according to the target segmentation region and the full scene graph comprises:

[0029] repeating twice inputting the target segmentation region into a full connection layer and outputting first pixel points corresponding to the target segmentation region;

[0030] writing a target identifier of a segmentation target in the target segmentation region into a target tracking library;

[0031] calculating a first coordinate value corresponding to the first pixel points and acquiring a second coordinate value corresponding to the first pixel points from the coordinate library;

[0032] determining whether the first coordinate value and the second coordinate value are the same;

[0033] if yes, deleting the target identifier from the target tracking library and reacquiring the second time picture from the all cameras;

[0034] if no, controlling the all cameras to track the segmentation target and sending a fireworks alarm to the user.

[0035] In one embodiment, the controlling the all cameras to track the segmentation target and sending the fireworks alarm to the user comprises:

[0036] determining whether a tracking duration exceeds a time threshold set by the user;

[0037] if no, deleting the target identifier from the target tracking library and reacquiring the second time picture from the all cameras;

[0038] if yes, determining that the flame exists in the target location and sending the fireworks alarm to the user.

[0039] In another aspect, a fireworks identification device is provided, and the device comprises:

[0040] an acquisition module, configured to acquire all first time pictures corresponding to a target location from all cameras and receive a position relationship corresponding to the all cameras sent by a user.

[0041] a scene generation module configured to generate a full scene map of the target site according to the all first-time pictures and the position relationship;

[0042] a region generation module configured to acquire second-time pictures from the all cameras and generate a target detection region and a target segmentation region according to the second-time pictures;

[0043] a first determination module configured to determine whether the target site has a flame according to the target detection region and a thermal imaging view of the all cameras;

[0044] a second determination module configured to, if yes, determine again whether the target site has a flame according to the target segmentation region and the full scene map.

[0045] In one of the embodiments, the scene generation module generates a full scene map of the target site according to the all first-time pictures and the position relationship includes:

[0046] performing gray processing, Gaussian blur processing and local binary feature processing on the all first-time pictures;

[0047] performing coincidence processing on the all first-time pictures and generating the full scene map;

[0048] calculating point coordinates of all pixel points in the full scene map according to the position relationship and storing the point coordinates in a coordinate database.

[0049] In one of the embodiments, the region generation module acquires second-time pictures from the all cameras and generates a target detection region and a target segmentation region according to the second-time pictures includes:

[0050] performing preprocessing and the local binary feature processing on the second-time pictures and inputting the second-time pictures to a neural network;

[0051] inputting the output second-time pictures to a feature network and outputting a first feature map;

[0052] inputting the first feature map to a region production network and outputting a target recommended region;

[0053] inputting the first feature map and the target recommended region to an interesting region arrangement network and outputting a second feature map;

[0054] inputting the second feature map to a prediction layer and outputting the target detection region and the target segmentation region.

[0055] In one of the embodiments, the first determining module determines whether the target location has fire according to the target detection region and the thermal imaging view of the all cameras, which includes:

[0056] inputting the target detection region into a full connection layer and outputting a target frame, a target classification and a target score;

[0057] determining whether the target frame has fire according to the target classification and the target score;

[0058] if not, reacquiring the second time picture from the all cameras;

[0059] if yes, continuing to determine whether the target frame has fire according to the thermal imaging view.

[0060] In one of the embodiments, the first determining module continues to determine whether the target frame has fire according to the thermal imaging view, which includes:

[0061] acquiring an object temperature corresponding to an object in the target frame according to the thermal imaging view;

[0062] when the object temperature is greater than a temperature threshold set by a user, determining that the target frame has fire.

[0063] In one of the embodiments, the second determining module determines whether the target location has fire according to the target segmentation region and the full scene graph again, which includes:

[0064] inputting the target segmentation region into a full connection layer twice and outputting a first pixel point corresponding to the target segmentation region;

[0065] writing a target identifier of a segmentation target in the target segmentation region into a target tracking library;

[0066] calculating a first coordinate value corresponding to the first pixel point and acquiring a second coordinate value corresponding to the first pixel point from the coordinate library;

[0067] determining whether the first coordinate value and the second coordinate value are the same;

[0068] if yes, deleting the target identifier from the target tracking library and reacquiring the second time picture from the all cameras;

[0069] if not, controlling the all cameras to track the segmentation target and sending a fireworks warning to the user.

[0070] In one of the embodiments, the second determining module controls the all cameras to track the segmentation target and sends a fireworks warning to the user, which includes:

[0071] determining whether the tracking duration exceeds a time threshold set by the user;

[0072] if not, deleting the target identification from the target tracking library and reacquiring the second-time pictures from all the cameras;

[0073] if yes, determining that the target site has fire and sending the fireworks warning to the user.

[0074] In still another aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0075] acquiring all first-time pictures corresponding to the target site from all the cameras and receiving a position relationship corresponding to all the cameras sent by the user;

[0076] generating a full-scene graph of the target site according to the all first-time pictures and the position relationship;

[0077] acquiring second-time pictures from all the cameras and generating a target detection region and a target segmentation region according to the second-time pictures;

[0078] determining whether the target site has fire according to the target detection region and a thermal imaging view in all the cameras;

[0079] if yes, determining again whether the target site has fire according to the target segmentation region and the full-scene graph.

[0080] In one embodiment, the processor implements the following steps when executing the computer program:

[0081] the generating of the full-scene graph of the target site according to the all first-time pictures and the position relationship comprises:

[0082] gray processing, Gaussian blur processing, and local binary feature processing on the all first-time pictures;

[0083] coincidence processing on the all first-time pictures and generating the full-scene graph;

[0084] calculating point coordinates of all pixel points in the full-scene graph according to the position relationship and storing the point coordinates in a coordinate library.

[0085] In one embodiment, the processor implements the following steps when executing the computer program:

[0086] The second time picture is obtained from the all cameras, and a target detection region and a target segmentation region are generated according to the second time picture.

[0087] The second time picture is preprocessed and subjected to the local binary feature processing, and the second time picture is input to a neural network;

[0088] The output second time picture is input to a feature network, and a first feature map is output;

[0089] The first feature map is input to a region production network, and a target recommendation region is output;

[0090] The first feature map and the target recommendation region are input to an interesting region arrangement network, and a second feature map is output;

[0091] The second feature map is input to a prediction layer, and the target detection region and the target segmentation region are output.

[0092] In one of the embodiments, the processor implements the following steps when executing the computer program:

[0093] The target place whether the flame exists is determined according to the target detection region and the thermal imaging view in the all cameras.

[0094] The target detection region is input to a full connection layer, and a target frame, a target classification and a target score are output;

[0095] Whether the flame exists in the target frame is determined according to the target classification and the target score;

[0096] If not, the second time picture is re-obtained from the all cameras;

[0097] If yes, whether the flame exists in the target frame is determined according to the thermal imaging view.

[0098] In one of the embodiments, the processor implements the following steps when executing the computer program:

[0099] The flame whether the flame exists in the target frame is determined according to the thermal imaging view.

[0100] The object temperature corresponding to the target object in the target frame is obtained according to the thermal imaging view;

[0101] When the object temperature is greater than a temperature threshold set by a user, it is determined that the target frame has the flame.

[0102] In one of the embodiments, the processor implements the following steps when executing the computer program:

[0103] the determining whether the target location has the flame again according to the target segmentation region and the full scene graph comprises:

[0104] repeating inputting the target segmentation region to the full connection layer and outputting the first pixel point corresponding to the target segmentation region twice;

[0105] writing the target identification of the segmentation target in the target segmentation region into a target tracking library;

[0106] calculating a first coordinate value corresponding to the first pixel point and obtaining a second coordinate value corresponding to the first pixel point from the coordinate library;

[0107] determining whether the first coordinate value and the second coordinate value are the same;

[0108] if yes, deleting the target identification from the target tracking library and re-obtaining the second time picture from the all cameras;

[0109] if no, controlling the all cameras to track the segmentation target and sending the fireworks warning to the user.

[0110] In one embodiment, the processor implements the following steps when executing the computer program:

[0111] the controlling the all cameras to track the segmentation target and sending the fireworks warning to the user comprises:

[0112] determining whether the tracking duration exceeds the time threshold set by the user;

[0113] if no, deleting the target identification from the target tracking library and re-obtaining the second time picture from the all cameras;

[0114] if yes, determining that the target location has the flame and sending the fireworks warning to the user.

[0115] In another aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0116] obtaining all first time pictures corresponding to a target location from all cameras and receiving a position relationship corresponding to the all cameras sent by a user;

[0117] generating a full scene graph of the target location according to the all first time pictures and the position relationship;

[0118] obtaining a second time picture from the all cameras and generating a target detection region and a target segmentation region according to the second time picture;

[0119] determining whether the target location has flame according to the target detection region and the thermal imaging view of the all cameras;

[0120] If yes, determining again whether the target location has flame according to the target segmentation region and the full scene graph.

[0121] In one of the embodiments, the computer program is executed by the processor to realize the following steps:

[0122] The generating the full scene graph of the target location according to the all first time pictures and the position relationship comprises:

[0123] performing gray processing, Gaussian blur processing and local binary feature processing on the all first time pictures;

[0124] performing superimposition processing on the all first time pictures and generating the full scene graph;

[0125] calculating point coordinates of all pixel points in the full scene graph according to the position relationship and storing the point coordinates into a coordinate database.

[0126] In one of the embodiments, the computer program is executed by the processor to realize the following steps:

[0127] The acquiring second time pictures from the all cameras and generating target detection region and target segmentation region according to the second time pictures comprises:

[0128] performing preprocessing and the local binary feature processing on the second time pictures and inputting the second time pictures into a neural network;

[0129] inputting the output second time pictures into a feature network and outputting a first feature map;

[0130] inputting the first feature map into a region production network and outputting a target recommendation region;

[0131] inputting the first feature map and the target recommendation region into an interesting region arrangement network and outputting a second feature map;

[0132] inputting the second feature map into a prediction layer and outputting the target detection region and the target segmentation region.

[0133] In one of the embodiments, the computer program is executed by the processor to realize the following steps:

[0134] The determining whether the target location has flame according to the target detection region and the thermal imaging view of the all cameras comprises:

[0135] inputting the target detection region to a full connection layer and outputting a target frame, a target classification, and a target score;

[0136] determining whether a flame exists in the target frame according to the target classification and the target score;

[0137] if not, reacquiring the second-time picture from the all cameras;

[0138] if yes, continuing to determine whether a flame exists in the target frame according to the thermal imaging view.

[0139] In one of the embodiments, the computer program, when executed by a processor, implements the following steps:

[0140] the continuing to determine whether a flame exists in the target frame according to the thermal imaging view comprises:

[0141] acquiring an object temperature of a target object in the target frame according to the thermal imaging view;

[0142] when the object temperature is greater than a temperature threshold set by a user, determining that the target frame contains a flame.

[0143] In one of the embodiments, the computer program, when executed by a processor, implements the following steps:

[0144] the re-determining whether a flame exists in the target location according to the target segmentation region and the full scene graph comprises:

[0145] repeating twice the inputting the target segmentation region to a full connection layer and outputting a first pixel point corresponding to the target segmentation region;

[0146] writing a target identifier of a segmented target in the target segmentation region into a target tracking library;

[0147] calculating a first coordinate value corresponding to the first pixel point and acquiring a second coordinate value corresponding to the first pixel point from the coordinate library;

[0148] determining whether the first coordinate value and the second coordinate value are the same;

[0149] if yes, deleting the target identifier from the target tracking library and reacquiring the second-time picture from the all cameras;

[0150] if not, controlling the all cameras to track the segmented target and sending a fireworks warning to the user.

[0151] In one of the embodiments, the computer program, when executed by a processor, implements the following steps:

[0152] The control of the all cameras tracking the segmented target and sending the fireworks warning to the user comprises:

[0153] determining whether the tracking duration exceeds the time threshold set by the user;

[0154] If not, the target identifier is deleted from the target tracking library and the second time picture is reacquired from the all cameras;

[0155] If yes, it is determined that the target site has fire and the fireworks warning is sent to the user.

[0156] By acquiring all the first time pictures corresponding to the target site from all the cameras and receiving the position relationship corresponding to the all cameras sent by the user, a full scene graph of the target site is generated according to the all first time pictures and the position relationship. The second time picture is acquired from the all cameras and a target detection area and a target segmentation area are generated according to the second time picture. It is determined whether the target site has fire according to the target detection area and the thermal imaging view in the all cameras. If yes, it is determined again whether the target site has fire according to the target segmentation area and the full scene graph. By comparing the actual pixel point coordinates of the target with the actual coordinates of the corresponding pixel points in the coordinate library and further determining through the thermal imaging image, the false detection rate of the fireworks recognition is effectively reduced. BRIEF DESCRIPTION OF DRAWINGS

[0157] Figure 1 It is a flowchart of the fireworks recognition method;

[0158] Figure 2 It is a step schematic diagram of the fireworks recognition method;

[0159] Figure 3 It is a structural schematic diagram of the fireworks recognition device;

[0160] Figure 4 It is an internal structure diagram of the computer device in the embodiment of the application. DETAILED DESCRIPTION

[0161] In order to make the purpose, technical scheme and advantages of the application clearer, the application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0162] The fireworks recognition method provided by the application can be applied to, for example, Figure 1The flowchart of the shown pyrotechnic identification method. The target region is extracted by target detection and target segmentation. In the actual coordinate comparison of the target region pixel points, the actual pixel point coordinates of the target are compared with the actual coordinates of the corresponding pixel points in the coordinate library, and further determination is made through the thermal imaging image to accurately identify the pyrotechnics.

[0163] In one embodiment, as shown in the figure, the present application provides a pyrotechnic identification method, which comprises: Figure 2

[0164] S201, obtaining all first time pictures corresponding to the target location from all cameras and receiving the position relationship corresponding to the all cameras sent by the user;

[0165] S202, generating a full scene graph of the target location according to the all first time pictures and the position relationship;

[0166] S203, obtaining a second time picture from the all cameras and generating a target detection region and a target segmentation region according to the second time picture;

[0167] S204, determining whether there is a flame in the target location according to the target detection region and the thermal imaging view in the all cameras;

[0168] S205, if yes, determining again whether there is a flame in the target location according to the target segmentation region and the full scene graph.

[0169] Specifically, all cameras in the target location such as a gas station can be accessed or accessed, and multiple first time pictures at the same time can be intercepted from the monitoring video recorded by all cameras. The multiple pictures are preprocessed. Before intercepting the first time picture, the user needs to arrange the cameras in the target location to ensure that each position in the gas station is at least in the field of view of two cameras, so that the matching of the overlapping fields of view in different cameras is matched, and preferably, the camera can be a combined camera containing visible light and thermal imaging.

[0170] In one of the embodiments, the generating of the full scene graph of the target location according to the all first time pictures and the position relationship comprises:

[0171] The all first time pictures are subjected to gray processing, Gaussian blur processing and local binary feature processing;

[0172] The all first time pictures are subjected to superimposition processing and the full scene graph is generated;

[0173] The point coordinates of all pixel points in the full scene graph are calculated according to the position relationship and the point coordinates are stored in the coordinate library. ​

[0174] Specifically, the preprocessing includes modifying the picture into a grayscale picture first, then performing Gaussian blur processing on the grayscale picture, further reducing the noise interference of the picture and reducing the error of the pixel value coordinate calculation of the highlight part, and finally performing LBP (Local Binary Pattern) feature processing on the processed picture. The first moment picture after preprocessing can be generated by overlapping the same pixel points in each picture. The 3D coordinate axis is constructed by the position relationship sent by the user including the coordinates corresponding to each camera, and the point coordinates of each pixel point in the full scene picture are calculated by substituting the full scene picture into the coordinate axis.

[0175] In one of the embodiments, the second moment picture is obtained from the all cameras, and the target detection area and the target segmentation area are generated according to the second moment picture, which includes:

[0176] The second moment picture is preprocessed and the local binary feature processing is performed, and the second moment picture is input to the neural network;

[0177] The output second moment picture is input to the feature network and the first feature map is output;

[0178] The first feature map is input to the region production network and the target recommendation area is output;

[0179] The first feature map and the target recommendation area are input to the interest region arrangement network and the second feature map is output;

[0180] The second feature map is input to the prediction layer and the target detection area and the target segmentation area are output.

[0181] Specifically, after generating the panoramic picture, the second time picture of the same time is re-cut from the video recorded by the camera. The picture is pre-processed, and the noise points in the picture are extracted. The size of the picture is scaled to a specific size. Preferably, the specific size is set to 640*640. During the picture size scaling process, the aspect ratio of the picture is not changed, and the maximum side is scaled to 640, and the small side is filled with gray scale. Then the picture processed by the LBP feature is input into the ResNet50 (Residual Neural Network 5050-layer residual network) network. Then the picture is input into the FPN (feature pyramid networks) to obtain the extracted first feature map. Then the first feature map is input into the RPN (Region Proposal Network) to obtain the corresponding target recommendation area. Then the first feature map and the target recommendation area are input into the ROIAlign (Region of Interesting Align) network to obtain the second feature map of the final size. Finally, the second feature map is input into the head layer (prediction layer) to obtain the target detection area and the target segmentation area.

[0182] In one embodiment, determining whether the target location has a flame based on the target detection area and the thermal imaging view of all cameras includes:

[0183] Inputting the target detection area into a fully connected layer and outputting a target frame, a target classification, and a target score;

[0184] Determining whether the target frame has a flame based on the target classification and the target score;

[0185] If not, reacquiring the second time picture from all cameras;

[0186] If yes, continue to determine whether the target frame has a flame based on the thermal imaging view.

[0187] Specifically, the target detection area is input into a fully connected layer to obtain a target frame, a target classification, and a target score, i.e., the target in the target frame, the judgment of the target, and the probability of the target. For example, the score (probability) of determining that the object in the target frame is a person is 0.9, and the score of determining that the object in the target frame is a dog is 0.1. When the score of determining that the target classification in the picture is a flame is 0.5 or more (which can be set by the user), it is determined that the target frame has a flame. If the score is less than 0.5 or the generated target classification does not have a flame, other second time pictures of different times are reacquired from the camera.

[0188] In one embodiment, the determining whether the flame exists in the target frame according to the thermal imaging view comprises:

[0189] acquiring an object temperature of an object in the target frame according to the thermal imaging view;

[0190] when the object temperature is greater than a temperature threshold set by a user, determining that the target frame has the flame.

[0191] Specifically, as described above, if the value is greater than 0.5, the object temperature of the object in the target frame is acquired through the thermal imaging view of the camera, and if the value is greater than, for example, 60 degrees (set by the user), it is determined that the object in the target frame has caught fire. The object temperature of the object can be acquired by selecting a camera close to the object from the pixel point coordinates corresponding to the target frame in the picture.

[0192] In one embodiment, the determining whether the flame exists in the target frame according to the thermal imaging view comprises:

[0193] repeating twice the inputting of the target segmentation area into the full connection layer and the outputting of the first pixel point corresponding to the target segmentation area;

[0194] writing a target identifier of a segmentation target in the target segmentation area into a target tracking library;

[0195] calculating a first point coordinate corresponding to the first pixel point and acquiring a second point coordinate corresponding to the first pixel point from the coordinate library;

[0196] determining whether the first point coordinate and the second point coordinate are the same;

[0197] if yes, deleting the target identifier from the target tracking library and reacquiring the picture at the second time from the all cameras;

[0198] if no, controlling the all cameras to track the segmentation target and sending a fireworks warning to the user.

[0199] Specifically, the target segmentation region is output twice through a full connection layer to obtain a plurality of pixel points in the target segmentation region and calculate actual point coordinates (first point coordinates) of each pixel point; an ID of a segmentation target in the target segmentation region is extracted and added to a target tracking library; original coordinates (second point coordinates) of the pixel points in the target segmentation region are extracted from the target library; if the two are the same, it is indicated that the segmentation target may be a reflection of sunlight, and the actual point coordinates do not change, at which time the target ID recorded in the target tracking library is deleted and a second time image at another time is reacquired. Only when there is a new object such as a flame on the segmentation target, the point coordinates will be different, that is, when the point coordinates are different, it is indicated that there is a flame in the target segmentation region, at which time a KCF algorithm (Kernel Correlation Filter) is used to control the camera to track the segmentation target and send a fireworks warning to the user.

[0200] In one of the embodiments, the control of the all cameras to track the segmentation target and the sending of the fireworks warning to the user comprises:

[0201] determination of whether a tracking duration exceeds a time threshold set by the user;

[0202] if not, deletion of the target ID from the target tracking library and reacquisition of the second time image from the all cameras;

[0203] if yes, determination that there is a flame at the target location and sending of the fireworks warning to the user.

[0204] Specifically, when the KCF algorithm is used to control the camera to track the segmentation target, if the tracking duration does not exceed the time threshold set by the user, such as 2 minutes, it is indicated that the tracked segmentation target has disappeared, and there is no need to send a warning to the user, at which time the target ID is deleted from the target tracking library and the second time image at another time is reacquired. If the tracking duration exceeds 2 minutes, it is indicated that the tracked segmentation target still exists at the target location, at which time a fireworks warning needs to be sent to the user and the corresponding second time image is saved.

[0205] The scheme of the application has the following beneficial effects:

[0206] 1) In the actual coordinate comparison process of the pixel points in the target region, the actual pixel point coordinates of the target are compared with the actual coordinates of the corresponding pixel points in the coordinate library, and further determination is made through a thermal imaging image to determine whether it is light or fireworks, which effectively reduces the false detection rate of the fireworks recognition.

[0207] 2) The target detection and target segmentation are used to extract the fireworks region, and when the target continuously appears, the alarm is given only when a set time threshold is reached, which further reduces the false detection rate.

[0208] It should be understood that, althoughFigure 2 The steps in the flowchart are shown in sequence according to the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the steps are not strictly limited in sequence, and the steps can be executed in other sequences. Moreover, Figure 2 At least part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of the sub-steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or sub-steps or stages of other steps.

[0209] In one embodiment, as shown in Figure 3 A pyro-identification device is provided, the device comprising:

[0210] The acquisition module 301 is configured to acquire all first-time pictures corresponding to a target location from all cameras and receive a position relationship corresponding to the all cameras sent by a user.

[0211] The scene generation module 302 is configured to generate a full scene graph of the target location according to the all first-time pictures and the position relationship.

[0212] The region generation module 303 is configured to acquire second-time pictures from the all cameras and generate a target detection region and a target segmentation region according to the second-time pictures.

[0213] The first determination module 304 is configured to determine whether there is a flame in the target location according to the target detection region and a thermal imaging view in the all cameras.

[0214] If so, the second determination module 305 is configured to determine again whether there is a flame in the target location according to the target segmentation region and the full scene graph.

[0215] In one embodiment, the scene generation module generates a full scene graph of the target location according to the all first-time pictures and the position relationship, which includes:

[0216] Performing gray processing, Gaussian blur processing, and local binary feature processing on the all first-time pictures.

[0217] Performing overlapping processing on the all first-time pictures and generating the full scene graph.

[0218] Calculating point coordinates of all pixel points in the full scene graph according to the position relationship and storing the point coordinates in a coordinate database.

[0219] In one of the embodiments, the region generation module obtains a second time picture from the all cameras and generates a target detection region and a target segmentation region according to the second time picture, which comprises:

[0220] preprocessing the second time picture and the local binary feature processing and inputting the second time picture into a neural network;

[0221] inputting the output second time picture into a feature network and outputting a first feature map;

[0222] inputting the first feature map into a region production network and outputting a target recommendation region;

[0223] inputting the first feature map and the target recommendation region into an interesting region arrangement network and outputting a second feature map;

[0224] inputting the second feature map into a prediction layer and outputting the target detection region and the target segmentation region.

[0225] In one of the embodiments, the first determination module determines whether the target site has a flame according to the target detection region and a thermal imaging view in the all cameras, which comprises:

[0226] inputting the target detection region into a full connection layer and outputting a target frame, a target classification and a target score;

[0227] determining whether the target frame has a flame according to the target classification and the target score;

[0228] if not, re-obtaining the second time picture from the all cameras;

[0229] if yes, continuing to determine whether the target frame has a flame according to the thermal imaging view.

[0230] In one of the embodiments, the first determination module continues to determine whether the target frame has a flame according to the thermal imaging view, which comprises:

[0231] obtaining an object temperature corresponding to a target object in the target frame according to the thermal imaging view;

[0232] when the object temperature is greater than a temperature threshold set by a user, determining that the target frame has a flame.

[0233] In one of the embodiments, the second determination module determines whether the target site has a flame according to the target segmentation region and the full scene view again, which comprises:

[0234] inputting the target segmentation region into a full connection layer twice and outputting a first pixel point corresponding to the target segmentation region;

[0235] writing a target identity of the segmented target in the target segmentation area into a target tracking library;

[0236] calculating a first coordinate value corresponding to the first pixel point and obtaining a second coordinate value corresponding to the first pixel point from the coordinate library;

[0237] determining whether the first coordinate value and the second coordinate value are the same;

[0238] if yes, deleting the target identity from the target tracking library and re-obtaining the second time picture from the all cameras;

[0239] if no, controlling the all cameras to track the segmented target and sending a fireworks alarm to the user.

[0240] In one embodiment, the second determining module controlling the all cameras to track the segmented target and sending a fireworks alarm to the user comprises:

[0241] determining whether a tracking duration exceeds a time threshold set by the user;

[0242] if no, deleting the target identity from the target tracking library and re-obtaining the second time picture from the all cameras;

[0243] if yes, determining that there is a flame at the target location and sending the fireworks alarm to the user.

[0244] The specific limitations of the fireworks identification device can refer to the limitations of the fireworks identification method in the foregoing, which will not be repeated here. Each module in the above fireworks identification device can be realized by software, hardware and combinations thereof in whole or in part. The above each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operations corresponding to each module by the processor.

[0245] In one embodiment, a computer device is provided, which can be a terminal, and its internal structure diagram can be as shown in Figure 4As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to realize the alarm information processing method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0246] Those skilled in the art can understand that Figure 4 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0247] In one embodiment, an electronic device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0248] Obtain all first-time pictures corresponding to the target location from all cameras and receive the position relationship corresponding to the all cameras sent by the user;

[0249] Generate a full-scene graph of the target location according to the all first-time pictures and the position relationship;

[0250] Obtain second-time pictures from the all cameras and generate a target detection area and a target segmentation area according to the second-time pictures;

[0251] Determine whether there is a flame in the target location according to the target detection area and the thermal imaging view in the all cameras;

[0252] If yes, determine again whether there is a flame in the target location according to the target segmentation area and the full-scene graph.

[0253] In one embodiment, the processor implements the following steps when executing the computer program:

[0254] The generating the full scene graph of the target site according to the all first time pictures and the position relationship comprises:

[0255] The all first time pictures are subjected to gray processing, Gaussian blur processing and local binary feature processing;

[0256] The all first time pictures are subjected to registration processing and the full scene graph is generated;

[0257] The point coordinates of all pixel points in the full scene graph are calculated according to the position relationship and the point coordinates are stored in a coordinate database.

[0258] In one embodiment, the processor implements the following steps when executing the computer program:

[0259] The second time pictures are acquired from the all cameras and the target detection region and the target segmentation region are generated according to the second time pictures, which comprises:

[0260] The second time pictures are preprocessed and subjected to the local binary feature processing and the second time pictures are input to a neural network;

[0261] The output second time pictures are input to a feature network and a first feature map is output;

[0262] The first feature map is input to a region production network and a target recommendation region is output;

[0263] The first feature map and the target recommendation region are input to an interesting region arrangement network and a second feature map is output;

[0264] The second feature map is input to a prediction layer and the target detection region and the target segmentation region are output.

[0265] In one embodiment, the processor implements the following steps when executing the computer program:

[0266] The target site whether exists flame is determined according to the target detection region and the thermal imaging view in the all cameras, which comprises:

[0267] The target detection region is input to a full connection layer and a target frame, a target classification and a target score are output;

[0268] The target frame whether exists flame is determined according to the target classification and the target score;

[0269] If not, the second time pictures are re-acquired from the all cameras;

[0270] If yes, the target frame whether exists flame is determined according to the thermal imaging view.

[0271] In one embodiment, the processor implements the following steps when executing the computer program:

[0272] The determining whether the flame exists in the target frame according to the thermal imaging view comprises:

[0273] Obtaining an object temperature corresponding to an object in the target frame according to the thermal imaging view;

[0274] When the object temperature is greater than a temperature threshold set by a user, determining that the flame exists in the target frame.

[0275] In one embodiment, the processor implements the following steps when executing the computer program:

[0276] The determining whether the flame exists in the target location according to the target segmentation region and the full-scene map comprises:

[0277] Repeating twice the inputting the target segmentation region into the full connection layer and the outputting the first pixel point corresponding to the target segmentation region;

[0278] Writing a target identifier of a segmentation target in the target segmentation region into a target tracking library;

[0279] Calculating a first coordinate value corresponding to the first pixel point and obtaining a second coordinate value corresponding to the first pixel point from the coordinate library;

[0280] Determining whether the first coordinate value and the second coordinate value are the same;

[0281] If yes, deleting the target identifier from the target tracking library and re-obtaining the second-time picture from the all cameras;

[0282] If no, controlling the all cameras to track the segmentation target and sending a fireworks alarm to the user.

[0283] In one embodiment, the processor implements the following steps when executing the computer program:

[0284] The controlling the all cameras to track the segmentation target and sending the fireworks alarm to the user comprises:

[0285] Determining whether a tracking duration exceeds a time threshold set by the user;

[0286] If no, deleting the target identifier from the target tracking library and re-obtaining the second-time picture from the all cameras;

[0287] If yes, it is determined that the target site has fire and the pyrotechnic alarm is sent to the user.

[0288] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0289] All first time pictures corresponding to the target site are obtained from all cameras, and the position relationship corresponding to all cameras sent by the user is received;

[0290] A full scene graph of the target site is generated according to the all first time pictures and the position relationship;

[0291] A second time picture is obtained from the all cameras, and a target detection region and a target segmentation region are generated according to the second time picture;

[0292] It is determined whether the target site has fire according to the target detection region and the thermal imaging view in the all cameras;

[0293] If yes, it is determined again whether the target site has fire according to the target segmentation region and the full scene graph.

[0294] In one embodiment, the computer program is executed by a processor to implement the following steps:

[0295] The full scene graph of the target site is generated according to the all first time pictures and the position relationship, which includes:

[0296] The all first time pictures are subjected to gray processing, Gaussian blur processing and local binary feature processing;

[0297] The all first time pictures are subjected to superimposition processing and the full scene graph is generated;

[0298] The point coordinates of all pixel points in the full scene graph are calculated according to the position relationship, and the point coordinates are stored in a coordinate database.

[0299] In one embodiment, the computer program is executed by a processor to implement the following steps:

[0300] The second time picture is obtained from the all cameras, and a target detection region and a target segmentation region are generated according to the second time picture, which includes:

[0301] The second time picture is preprocessed and subjected to the local binary feature processing, and the second time picture is input to a neural network;

[0302] The output second time picture is input to a feature network and a first feature map is output.

[0303] inputting the first feature map into a region production network and outputting a target recommended region;

[0304] inputting the first feature map and the target recommended region into an interest region arrangement network and outputting a second feature map;

[0305] inputting the second feature map into a prediction layer and outputting the target detection region and the target segmentation region.

[0306] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0307] the determining whether the target location has a flame based on the target detection region and the thermal image view of all cameras comprises:

[0308] inputting the target detection region into a full connection layer and outputting a target frame, a target classification and a target score;

[0309] determining whether the target frame has a flame based on the target classification and the target score;

[0310] if not, reacquiring the second time picture from all cameras;

[0311] if yes, continuing to determine whether the target frame has a flame based on the thermal image view.

[0312] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0313] the continuing to determine whether the target frame has a flame based on the thermal image view comprises:

[0314] acquiring an object temperature of a target object in the target frame based on the thermal image view;

[0315] when the object temperature is greater than a temperature threshold set by a user, determining that the target frame has a flame.

[0316] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0317] the re-determining whether the target location has a flame based on the target segmentation region and the full scene map comprises:

[0318] inputting the target segmentation region into a full connection layer twice and outputting a first pixel point corresponding to the target segmentation region;

[0319] writing a target identification of a segmentation target in the target segmentation region into a target tracking library;

[0320] calculate a first coordinate value corresponding to the first pixel point and obtain a second coordinate value corresponding to the first pixel point from the coordinate library;

[0321] determine whether the first coordinate value and the second coordinate value are the same;

[0322] if yes, delete the target identifier from the target tracking library and reacquire the second time picture from all the cameras;

[0323] if no, control all the cameras to track the segmented target and send a fireworks alarm to the user.

[0324] In one of the embodiments, the computer program is executed by a processor to implement the following steps:

[0325] the control of all the cameras to track the segmented target and send a fireworks alarm to the user includes:

[0326] determine whether the tracking duration exceeds the time threshold set by the user;

[0327] if no, delete the target identifier from the target tracking library and reacquire the second time picture from all the cameras;

[0328] if yes, determine that there is a flame at the target location and send the fireworks alarm to the user.

[0329] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0330] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0331] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for identifying fireworks, comprising: Obtain all first-moment images corresponding to the target location from all cameras and receive the positional relationships of all cameras sent by the user; Generate a full scene graph of the target location based on all first-moment images and position relationships; Obtaining a second moment image from all cameras and generating a target detection region and a target segmentation region based on the second moment image, including: preprocessing and local binary feature processing the second moment image and inputting the second moment image into a neural network; inputting the output second moment image into a feature network and outputting a first feature map; inputting the first feature map into a region production network and outputting a target recommendation region; inputting the first feature map and the target recommendation region into a region of interest arrangement network and outputting a second feature map; inputting the second feature map into a prediction layer and outputting a target detection region and a target segmentation region; Determine whether there is fire at the target location based on the target detection area and thermal imaging views from all cameras; If so, determine again whether there is a flame at the target location based on the target segmentation area and the full scene map, including: repeatedly inputting the target segmentation area into the fully connected layer twice and outputting the first pixel point corresponding to the target segmentation area; writing the target identifier of the segmented target in the target segmentation area into the target tracking library; calculating the first coordinate value corresponding to the first pixel point and obtaining the second coordinate value corresponding to the first pixel point from the coordinate library; determining whether the first coordinate value and the second coordinate value are the same; if so, determine that the segmented target is a reflection of sunlight, delete the target identifier from the target tracking library and re-obtain the second moment image from all cameras; if not, determine that there is a flame in the target segmentation area, control all cameras to track the segmented target and send a fireworks alarm to the user.

2. The method according to claim 1, characterized in that Generating a full scene graph of the target location according to all the first moment images and the position relationship includes: Performing grayscale processing, Gaussian blur processing, and local binary feature processing on all the first-moment images; Performing overlap processing on all the first moment images and generating the full scene image; The point coordinates of all pixel points in the full scene graph are calculated according to the positional relationship and stored in a coordinate library.

3. The method according to claim 1, characterized in that Determining whether there is a flame at the target location based on the target detection area and the thermal imaging views of all cameras includes: Input the target detection area to the fully connected layer and output the target box, target classification and target score; Determining whether there is a flame in the target frame according to the target classification and the target score; If not, reacquire the picture at the second moment from all the cameras; If yes, continue to determine whether there is a flame in the target frame based on the thermal imaging view.

4. The method according to claim 3, characterized in that The step of continuing to determine whether there is a flame in the target frame according to the thermal imaging view includes: Acquiring an object temperature corresponding to the target object in the target frame according to the thermal imaging view; When the temperature of the object is greater than a temperature threshold set by the user, it is determined that there is a flame in the target frame.

5. The method according to claim 1, wherein The controlling all cameras to track the segmented target and sending a fire alarm to the user includes: Determining whether the tracking duration exceeds the time threshold set by the user; If not, deleting the target identifier from the target tracking library and reacquiring the second moment image from all cameras; If so, it is determined that there is fire at the target location and the fire alarm is sent to the user.

6. A fireworks identification device, the device being configured to implement the method according to any one of claims 1 to 5, comprising: An acquisition module, configured to acquire all first-moment images corresponding to a target location from all cameras and receive positional relationships corresponding to all cameras sent by a user; A scene generation module, configured to generate a full scene graph of the target location based on all the first moment images and the positional relationship; An area generation module, configured to obtain a second moment image from all cameras and generate a target detection area and a target segmentation area based on the second moment image; a first determining module, configured to determine whether a flame exists at the target location based on the target detection area and the thermal imaging views from all the cameras; The second determining module is configured to, if yes, determine again whether there is a flame at the target location based on the target segmented area and the full scene graph.

7. An electronic device, characterized in that: include: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the method according to any one of claims 1 to 5.

8. A computer storage medium, characterized in that A computer program is stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images