Intelligent fire identification method, device, computer equipment and storage medium

Abstract

This article relates to the field of artificial intelligence, and in particular to an intelligent fire identification method, device, computer equipment and storage medium, which are applied to edge servers, including: parsing the video stream data sent by the camera to obtain an image frame sequence; using a fire feature recognition model to identify images in the image frame sequence to obtain a fire feature recognition result; calculating the similarity of image frames in the image frame sequence within a preset time window to obtain an image similarity recognition result; assigning different weights to the fire feature recognition result and the image similarity recognition result, respectively, to calculate a fire recognition score; outputting a fire recognition result based on the fire recognition score and a preset alarm threshold; and sending the fire recognition result to a platform for centralized display. The fire recognition of this solution improves the real-time nature of fire warnings. It integrates multi-dimensional information such as time and scene of the image, greatly reduces false alarms and missed data, and can be better applied to complex application scenarios.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an intelligent fire identification method, device, computer equipment and storage medium. Background Art

[0002] In smart banking outlets and smart security scenarios, real-time and accurate fire identification is of great significance to the security system. All-weather monitoring of fire conditions in security scenarios and timely warning in the early stages of fires can effectively reduce the input costs of on-duty personnel and minimize the harm caused by fires to people's lives and property safety.

[0003] The existing fire identification method includes: setting up temperature, smoke and other sensors in the security monitoring scene, monitoring the status of the sensors to identify the fire. This method has a high deployment cost and is limited by the sensitivity of the sensors, and the early warning has a certain lag.

[0004] Current fire identification methods also include those based on traditional image algorithms. Traditional image algorithms identify fires through pixel value features, which can easily lead to misidentification of yellow and reddish-brown objects, and the recognition accuracy of traditional algorithms is not high. This type of solution does not fully integrate the characteristics of scene, time and space, and in some fire scenes, there is a problem that the open fire is not obvious, resulting in missed detections and false alarms.

[0005] The existing fire identification methods also include fire identification based on deep learning algorithms. Most of these solutions use target detection methods to detect whether there is flame in the image for fire identification. The algorithm of this method is deployed in the cloud, and the local camera data needs to be transmitted to the cloud through the network for identification and analysis. Multi-channel video stream data first needs to be transmitted back to the cloud for analysis and calculation in the cloud, which occupies a large bandwidth of the backhaul network, is heavily dependent on network conditions, and has a high service latency.

[0006] In view of the current problems of fire identification, such as time delay, easy missed detection and low identification accuracy, an intelligent fire identification method is needed. Summary of the invention

[0007] To solve the above problems of the prior art, the embodiments of this document provide an intelligent fire identification method, device, computer equipment and storage medium.

[0008] The embodiment of the present invention provides an intelligent fire identification method, which is applied to an edge server and includes: parsing video stream data sent by a camera to obtain an image frame sequence; using a fire feature recognition model to identify images in the image frame sequence to obtain a fire feature recognition result; calculating the image frame similarity of the image frame sequence within a preset time window to obtain an image similarity recognition result; assigning different weights to the fire feature recognition result and the image similarity recognition result respectively to calculate a fire recognition score; outputting a fire recognition result according to the fire recognition score and a preset alarm threshold; and sending the fire recognition result to a platform for centralized display.

[0009] According to one aspect of the embodiments of the present invention, parsing the video stream data sent by the camera to obtain an image frame sequence includes: decoding the video stream data to obtain an initial image frame; determining a sampling frequency and a sampling number for the initial image frame based on a frame rate of the acquired video stream data; and sampling the initial image frame based on the sampling frequency and the sampling number for the initial image frame to obtain an image frame sequence.

[0010] According to one aspect of the embodiments of the present invention, the method further includes: using a fire classification model to identify the image frame sequence to obtain a fire classification recognition result; assigning different weights to the fire feature recognition result and the image similarity recognition result, and calculating a fire recognition score, which is further: assigning different weights to the fire classification recognition result, the fire feature recognition result, and the image similarity recognition result, and calculating a fire recognition score.

[0011] According to one aspect of the embodiments of the present invention, the fire classification model training process includes: obtaining training sample images, the training sample images including historical fire images and normal images with fire type annotation information, the fire type annotation information including: whether the training sample images contain fire; using the training sample images to train a preset classification network to obtain a fire classification recognition model.

[0012] According to one aspect of the embodiment of this invention, the fire classification recognition result, the fire feature recognition result, and the image similarity recognition result are respectively assigned different weights to calculate the fire recognition score, and the fire recognition score is determined by using the following formula: total =M×score b +N×(score a +score c ) / 2, where score total is the fire identification score, score a is the fire feature recognition result, score b is the image similarity recognition result, scorec is the fire classification recognition result, M is the weight of the image similarity recognition result, and N is the weight of the fire feature recognition result and the fire classification recognition result.

[0013] According to one aspect of the embodiments of the present invention, assigning different weights to the fire classification recognition result, the fire feature recognition result, and the image similarity recognition result respectively includes: querying the weights corresponding to the fire classification result, the fire feature recognition result, and the image similarity from a preset weight table; the preset weight table stores the correspondence between fire classification and weight, the correspondence between fire feature and weight, and the correspondence between image similarity range and weight.

[0014] According to one aspect of the embodiments of the present invention, the training process of the fire feature recognition model includes: obtaining training sample images, the training sample images including historical fire images and normal images with fire feature annotation information and fire location annotation information; converting the fire feature annotation information of the training sample images into fire feature confidence scores; converting the fire location annotation information of the training sample images into fire feature coordinate information; using the training sample images as input and the fire feature confidence scores and the fire feature coordinate information as output to train parameters in a preset neural network model; and determining the fire feature recognition model according to the structure and parameters of the preset neural network model.

[0015] According to one aspect of the embodiments of the present invention, after obtaining the training sample image, it also includes: calculating the area of ​​the fire location in the training sample image; when the area of ​​the fire location is larger than a preset area, segmenting the training sample image; and using a segmented labeling method to add labeling information to the target features in the segmented training sample image.

[0016] According to one aspect of the embodiments of the present invention, the image frame similarity of an image frame sequence within a preset time window is calculated to obtain an image similarity recognition result, including: determining a preset time window size and a window sliding step; determining the similarity of the image frames within the current preset time window based on the original pixel matrix of each frame of N frames within the preset time window; and moving the preset time window according to the window sliding step to determine the similarity of the images within the next time window.

[0017] According to one aspect of the embodiments of this document, the method further includes: verifying the fire identification results within a predetermined time period to obtain the number of fire identification result errors; comparing the number of records with a preset number, and if the number of records exceeds the preset number, adjusting the preset alarm threshold.

[0018] According to one aspect of the embodiments of the present invention, the fire identification results within a predetermined time period are verified to obtain the number of wrong fire identification results, including: determining the cameras related to the fire identification area according to the fire identification area; comparing whether the fire identification results corresponding to the related cameras within the predetermined time period are consistent, recording the number of inconsistent comparison results, and using the number of inconsistent comparison results as the number of wrong fire identification results.

[0019] According to one aspect of the embodiments of this document, sending the fire identification results to a platform for centralized display also includes: obtaining the IP address of the camera; determining the geographical location of fire identification based on the IP address; associating the geographical location with the fire identification results, and sending them to the platform for display.

[0020] The embodiment of the present invention further provides an intelligent fire identification device, which includes: an image frame sequence acquisition unit, which is used to parse the video stream data sent by the camera to obtain an image frame sequence; a fire feature recognition result acquisition unit, which is used to use a fire feature recognition model to identify images in the image frame sequence to obtain a fire feature recognition result; an image similarity recognition result acquisition unit, which is used to calculate the similarity of image frames in the image frame sequence within a preset time window to obtain an image similarity recognition result; a fire identification score calculation unit, which is used to assign different weights to the fire feature recognition result and the image similarity recognition result, respectively, to calculate a fire identification score; a fire identification result output unit, which is used to output a fire identification result according to the fire identification score and a preset alarm threshold; and a sending unit, which is used to send the fire identification result to a platform for centralized display.

[0021] The embodiments of the present invention further provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0022] The embodiments of the present invention further provide a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the above method is implemented.

[0023] The intelligent fire identification method of this solution does not need to upload video stream data to the cloud server. The edge server directly calculates the fire identification algorithm, which reduces the demand for network bandwidth and improves the real-time performance of fire warning. The fire identification algorithm calculates the similarity of image frames within a preset time window and uses the fire feature recognition model to calculate the fire feature recognition results. It integrates the multi-dimensional information of the image, such as time and scene, greatly reduces false positives and missed positives, and can be better applied to complex application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of this article or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of this article. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 Shown is a flow chart of an intelligent fire identification method according to an embodiment of this invention;

[0026] Figure 2 It is a flowchart of a method for calculating a fire identification score according to an embodiment of this invention;

[0027] Figure 3 The above is a flow chart of a method for assigning different weights to fire classification recognition results, fire feature recognition results, and image similarity recognition results according to an embodiment of the present invention;

[0028] Figure 4 A flow chart of a method for training a fire feature recognition model according to an embodiment of this invention;

[0029] Figure 5 The figure is a flow chart of a method for obtaining image similarity recognition results according to an embodiment of this invention;

[0030] Figure 6 The figure is a flow chart of a method for verifying fire identification results according to an embodiment of this invention;

[0031] Figure 7 The figure is a flow chart of a method for determining an error in a fire identification result according to an embodiment of the present invention;

[0032] Figure 8 The figure is a flow chart of a method for sending fire identification results to a platform according to an embodiment of the present invention;

[0033] Fig. 9 Shown is a schematic diagram of an intelligent fire identification device according to an embodiment of this invention;

[0034] Fig.10 The figure shows a specific structural diagram of the intelligent fire identification device of this embodiment;

[0035] Fig.11 Shown is a schematic diagram of the structure of a computer device according to an embodiment of this invention.

[0036] Description of the accompanying symbols:

[0037] 901, image frame sequence acquisition unit;

[0038] 9011, initial image frame acquisition module;

[0039] 902. Fire feature recognition result acquisition unit;

[0040] 9021. Fire feature recognition model training module;

[0041] 9022, segmentation labeling module;

[0042] 903, image similarity recognition result acquisition unit;

[0043] 9031. Preset window size setting module;

[0044] 9032. Image similarity calculation module;

[0045] 904. Fire identification score calculation unit;

[0046] 9041, weight determination module;

[0047] 905, fire identification result output unit;

[0048] 9051, fire identification result verification module;

[0049] 9052. Preset alarm threshold adjustment module;

[0050] 906, sending unit;

[0051] 9061, geographic location sending module;

[0052] 1102. Computer equipment;

[0053] 1104. Processor;

[0054] 1106. Memory;

[0055] 1108, driving mechanism;

[0056] 1110, input / output module;

[0057] 1112. Input device;

[0058] 1114. Output device;

[0059] 1116. Presentation equipment;

[0060] 1118. Graphical user interface;

[0061] 1120, network interface;

[0062] 1122. Communication link;

[0063] 1124. Communication bus. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0065] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0066] This specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel.

[0067] It should be noted that the intelligent fire identification method in this article can be used in the security field, and specific scenarios include providing fire identification services for bank branches or bank self-service machines.

[0068] like Figure 1 The flowchart of an intelligent fire identification method of an embodiment of this invention is shown, and the method is applied to an edge server. The edge server can also be an edge device, in which the intelligent fire identification algorithm is mainly deployed. Furthermore, the edge server has an edge box for intelligent computing, an intelligent camera with image acquisition and computing functions, etc. The method specifically includes the following steps:

[0069] Step 101, parse the video stream data sent by the camera to obtain an image frame sequence. In this step, the edge server receives the video stream data sent by the IPC camera. Among them, the IPC cameras are distributed in an environment where fire conditions need to be identified, that is, a fire identification area. In some embodiments of this specification, the fire identification area includes but is not limited to bank branches, self-service teller machines, etc. The IPC camera collects video stream data in the on-site environment in real time, and can encode and compress the collected video stream data, and send the encoded video stream data to the edge server. In some embodiments of this specification, the IPC camera sends the acquired video stream data to a network video recorder (NVR, Network Video Recorder), and the edge server pulls the video stream of the specified camera from the NVR. In some other embodiments of this specification, the edge server can also obtain video stream data directly from the IPC camera.

[0070] In some embodiments of the present specification, after the edge server obtains the video stream data, it decodes the video stream data to obtain the initial image data. And according to the frame rate of the video stream obtained by the smart camera in the edge server, the sampling frequency and the number of samples for the initial image frame are determined, and further according to the sampling frequency and the number of samples for the initial image frame, the obtained initial image frame is sampled to obtain an image frame sequence.

[0071] Step 102, using the fire feature recognition model to recognize the images in the image frame sequence to obtain a fire feature recognition result. In this step, the fire feature recognition model is used to recognize whether there are target features related to fire in the images in the image frame sequence. Among them, flames and smoke are target features that are more related to fire. The fire feature recognition model outputs a fire feature recognition result based on the training sample images containing target features such as flames and smoke. For the specific steps of the fire feature recognition model and the model output to obtain the fire feature recognition result in this step, see Figure 2 , Figure 4 Description.

[0072] Step 103, calculate the similarity of the image frames in the image frame sequence within a preset time window to obtain an image similarity recognition result. In this step, the image frame sequence is divided according to the preset time window. Because when the video stream data changes little or the video screen does not change, the image within the preset time window tends to be static. When a fire suddenly occurs in a quiet screen, the image frames in the video stream data will mutate. Therefore, based on the similarity of the image frames in a time window before and after the fire occurs, it can be determined whether a fire has occurred within the preset time window. Further determine the information of the time dimension of the fire. In some embodiments of the present specification, the similarity is calculated for the image frames within the preset time window. For a detailed description of determining the preset time window and calculating the image similarity, see Figure 5 describe.

[0073] Step 104, assign different weights to the fire feature recognition result and the image similarity recognition result, respectively, and calculate a fire recognition score. In some embodiments of this specification, a first weight may be assigned to the fire feature recognition result according to the degree of association between the fire feature recognition result and the fire, and a second weight may be assigned to the fire feature recognition result according to the image similarity recognition result. The fire recognition score is further calculated.

[0074] Step 105, outputting a fire identification result according to the fire identification score and the preset alarm threshold. In some embodiments of the present specification, the preset alarm threshold may be determined according to historical fire identification situations. When the calculated fire identification score is greater than the preset alarm threshold, the output fire identification result is: a fire has occurred or a fire situation exists; when the calculated fire identification score is less than the preset alarm threshold, the output fire identification result is: no fire has occurred or a fire situation does not exist. In some embodiments of the present specification, the preset fire alarm threshold may be adjusted according to the actual situation of the fire identification area. The preset alarm threshold may not be suitable for outputting fire identification results in emergency situations, and may be delayed or unsuitable for emergency or temporary fire situations. Therefore, the preset alarm threshold may be incorrect in some cases. Therefore, it is necessary to further verify the fire identification results within the preset time period, compare whether the fire identification results corresponding to multiple cameras in the fire identification area are consistent, and other methods to determine whether to adjust the fire alarm threshold.

[0075] Step 106, sending the fire identification results to the platform for centralized display. In this step, the fire identification results are sent to the cloud platform for centralized display. The fire identification results and the alarm information can also be sent to the cloud platform for display. The fire identification results, the alarm information and the geographical location information of the fire identification area can also be sent to the cloud platform for display.

[0076] Figure 2 The flowchart of a fire identification score calculation method in the embodiment of this article is shown. The specific steps include:

[0077] Step 201, using a fire classification model to identify the image frame sequence to obtain a fire classification recognition result. In some embodiments of the present specification, in addition to using a fire feature recognition model to obtain a fire feature recognition result and calculating an image similarity recognition result to obtain a fire recognition score. The present application also discloses a method for using a fire recognition model to calculate a fire classification recognition result to further obtain a fire recognition score. In some embodiments of the present specification, the fire classification model can be trained using a classification model and training sample images. In this step, the training process of the fire classification model includes: obtaining a training sample image, wherein the training sample image includes a historical fire image and a normal image with fire type annotation information, and the fire type annotation information includes: whether the training sample image contains a fire. Among them, the fire type annotation information can be represented by a number for model training. For example, the number 1 can indicate that the training sample image contains a fire; the number 0 can indicate that the training sample image does not contain a fire. The present application does not limit the method of using numbers to annotate the training sample image.

[0078] The fire classification recognition model can be obtained by training the preset classification network using the training sample images. In some embodiments of the present specification, the fire classification recognition result output by the fire classification model may include: the recognition result that the image includes fire or the image does not include fire. Corresponding to the training sample, the fire classification recognition result output by the fire classification model may also be a number 1 or 0, which may respectively represent: the recognition result that the image includes fire or the image does not include fire.

[0079] Step 202, assigning different weights to the fire feature recognition results and the image similarity recognition results, and calculating the fire recognition score, further comprises: assigning different weights to the fire classification recognition results, the fire feature recognition results, and the image similarity recognition results, and calculating the fire recognition score. Corresponding to step 104, after the fire classification recognition results are obtained by using the fire classification model, the fire classification recognition results can be combined with the fire classification model. Figure 1 The fire feature recognition results and image similarity recognition results calculated by the method in are given different weights to calculate the fire recognition score.

[0080] For example, the fire identification score is determined using the following formula: total =M×score b +N×(score a +score c ) / 2, where score total is the fire identification score, score a is the fire feature recognition result, score b is the image similarity recognition result, score cis the fire classification recognition result, M is the weight of the image similarity recognition result, and N is the weight of the fire feature recognition result and the fire classification recognition result. In some embodiments of this specification, the fire feature recognition result and the fire classification recognition result are both related to the characteristics of the fire, such as flames and smoke, so the fire recognition feature result score a , fire classification recognition result score c The correlation with fire is relatively large, and the same weight N can be assigned to these two types of results. The image similarity recognition result is related to the time when the fire occurred or the fire time dimension, so the result can be assigned a weight M. In some embodiments of the present specification, N can be a value such as 0.8, 0.75, 0.7, etc., and N can be a value such as 0.2, 0.25, 0.3, etc. This application does not limit the method of assigning weights to fire classification recognition results, fire feature recognition results, and image similarity recognition results. The method of assigning different weights to fire classification recognition results, fire feature recognition results, and image similarity recognition results is not limited to the description corresponding to the formula of this step.

[0081] Figure 3 The flowchart of a method for assigning different weights to fire classification recognition results, fire feature recognition results, and image similarity recognition results according to an embodiment of the present invention is shown, and specifically includes the following steps:

[0082] Step 301, query the weights corresponding to the fire classification result, the fire feature recognition result, and the image similarity from a preset weight table. In this specification, the weights corresponding to the fire classification result, the fire feature recognition result, and the image similarity can be preset and stored in the preset weight table. In some other embodiments of this specification, the different weights assigned to different recognition results can also be adjusted in real time according to the on-site conditions of the fire recognition area.

[0083] Step 302, the preset weight table stores the correspondence between fire classification and weight, the correspondence between fire features and weight, and the correspondence between image similarity range and weight. In some embodiments of the present specification, different weights are assigned to fire classification recognition results, fire feature recognition results, and image similarity recognition results according to the correlation between fire classification, fire features, and image similarity and fire. In this step, the fire classification recognition results include: the results of sending fire and no fire, and the fire features include flames, smoke, sparks and other features. In actual scenes, the correlation between flames and fire is relatively large, so the correspondence between flames and weights is relatively large; the correlation between sparks and fire is relatively small, so the correspondence between sparks and weights is relatively small. However, there are also some situations, for example, someone uses a lighter to smoke to produce sparks and smoke, but the duration is short and only appears in a few frames of images, and it is not an actual fire. Therefore, this solution can also combine the fire feature recognition results and the image similarity recognition results to assign weights.

[0084] Figure 4 The flowchart of the method for training a fire feature recognition model in the embodiment of this article is shown. The specific steps include:

[0085] Step 401, obtaining training sample images, wherein the training sample images include historical fire images and normal images with fire feature annotation information and fire location annotation information. In this step, the fire feature recognition model is trained by the training sample images. The training sample images are a large number of images with annotation information that have been manually annotated in advance, including historical fire images and normal images. Among them, the historical fire images include images with fire feature annotation information and fire location annotation information. In this step, the initial models trained include but are not limited to: TOLOV5s model, YOLOV5F model, etc. Taking the YOLOV5F model as an example, the model can realize the detection of target objects such as flames and smoke, identify the target object features and target object categories contained in the image from the image, and output the coordinate position of the target object in the image and the category confidence score corresponding to the target object.

[0086] In some embodiments of the present specification, after obtaining a training sample image, if the main feature contained in the training sample image is flame, the model can further calculate the area of ​​the fire position in the training sample image; when the area of ​​the fire position is larger than the preset area, the training sample image is segmented; and the target features in the segmented training sample image are respectively labeled using a segmented labeling method. In some embodiments of the present specification, when the flame area in the training sample image is large, the flame may contain a large amount of background information, resulting in a large amount of noise. Using conventional feature labeling may result in large errors. Therefore, a preset area is set, and when the area of ​​the fire position in the training sample image is larger than the preset area, the training sample image is segmented, and the flame features in the segmented training sample image are further labeled using a segmented labeling method, and labeling information is added.

[0087] Step 402, converting the fire feature annotation information of the training sample image into a fire feature confidence score. In some embodiments of the present specification, the fire feature annotation information in the training sample image includes identifying specific fire target features, including but not limited to: flames, smoke, sparks, etc. Correspondingly, the fire feature annotation information is the annotation information of what target features the image corresponding to each training sample image includes. Further, the trained model converts the fire feature annotation information identified in the training sample image into a fire feature confidence score. In this step, the fire feature confidence score represents the classification score of the target feature in the training sample image. For example, if the training sample image A contains flame feature annotation information, the model can convert the flame feature annotation information into a flame feature confidence score, such as 0.8 points, during the training process; for another example, if the training sample image B contains smoke feature annotation information, the model can convert the smoke feature annotation information into a smoke feature confidence score, such as 0.6 points, etc. during the training process.

[0088] Step 403, converting the fire position annotation information of the training sample image into fire feature coordinate information. In some embodiments of the present specification, the fire position annotation information in the training sample image is the position information corresponding to the fire features identified in step 402 in each image, and the position information includes the two-dimensional target frame position information of the fire features in the image. In this step, the trained model can identify the fire position annotation information in the training sample image and convert it into fire feature coordinate information. The coordinate positions of the upper left corner and the lower right corner of the two-dimensional target frame of the fire feature in the image are calculated.

[0089] Step 404, taking the training sample image as input, the fire feature confidence score and the fire feature coordinate information as output, and training the parameters in the preset neural network model. According to the previous steps, the preset neural network model includes YOLOV5s, YOLOV5F, etc. The training sample image is the input data of the preset neural network model, and the converted fire feature annotation information and fire location annotation information become the fire feature confidence score and fire feature coordinate information respectively, which are the output of the model.

[0090] Step 405, determine the fire feature recognition model according to the structure and parameters of the preset neural network model. In this step, the relationship between the input data and the output data in step 404 is learned according to the network structure and model parameters of the preset neural network, and the preset neural network model is further optimized. When the optimization is completed, the training of the preset neural network model is completed, and a trained fire feature recognition model is obtained.

[0091] like Figure 5 The flowchart of a method for obtaining an image similarity recognition result in an embodiment of this invention is shown, which specifically includes the following steps:

[0092] Step 501, determine the preset time window size and the window sliding step. In this step, the preset time window size is determined for the image frame sequence parsed into image frames. The preset time window is used to limit the image frames of a certain time sequence, and further calculate the similarity between the images in the time sequence. For example, the preset time window size is 2 time sequences, 5 time sequences or 10 time sequences. After determining the similarity of the image frames in the preset time window, the window is moved in sequence according to the window sliding step to obtain a new time window, and the image similarity in the new time window is further calculated.

[0093] Step 502, according to the original pixel matrix of each frame of the N frames of images in the preset time window, determine the similarity of the image frames in the current preset time window. In this step, according to the original pixel matrix of each frame of the image in the preset time window, the similarity between the original pixel matrices of each image is calculated. For example, if the length of the preset time window is 2 time series, there are 2 frames of images in the preset time window, and the image similarity is calculated according to the original pixel matrix in the 2 frames of images. For another example, if the length of the preset time window is 5 time series, there are 5 frames of images in the preset time window, and the image similarity is calculated according to the original pixel matrix in the 5 frames of images. In some embodiments of the present specification, the method of calculating the image similarity includes but is not limited to: Euclidean distance, cosine similarity, Pearson similarity, Manhattan distance, modified cosine similarity, etc. Take the use of Euclidean distance to calculate image similarity as an example: when the length of the preset time window is 2, the Euclidean distance is calculated according to the original pixel matrix of the two frames of images, and the similarity is judged according to the size of the Euclidean distance.

[0094] Step 503, the preset time window is moved according to the window sliding step length, and the similarity of the images in the next time window is determined. In this step, each time the similarity of the image frames in a preset time window is calculated, the current time window is moved according to the window sliding step length, and the similarity of the images in the next time window is calculated. This continues until the image similarity calculation corresponding to all the acquired image frame sequences is completed. In this step, the window sliding step length can be 1 time sequence.

[0095] Figure 6 The flowchart of a method for verifying a fire identification result in an embodiment of this invention is shown, which specifically includes the following steps:

[0096] Step 601, verify the fire identification results within a preset time period to obtain the number of fire identification result errors. In the present application, the edge server obtains video stream data through multiple IPC cameras deployed on site, and further processes and calculates the fire identification score within the preset time period. Further according to the preset alarm threshold, the fire identification result is output. For example, the fire identification score calculated by the edge server is 0.8 points, and the preset alarm threshold is set to 0.9 points. Because the fire identification score is less than the preset alarm threshold, the fire identification result output by the edge server is: no fire occurred at outlet A within 1 hour, but after actual on-site inspection, it was found that a fire occurred at outlet A within 1 hour, and it is considered that the fire identification result is wrong. Therefore, if the preset alarm threshold is not set accurately, each fire identification result may not be completely correct. Therefore, it is necessary to verify the fire identification result calculated within the preset time period to determine whether the fire identification result is wrong. The method for verifying the fire identification result in this step is Figure 7 Described in detail in.

[0097] Step 602, compare the number of records with the preset number of times, if the number of records exceeds the preset number of times, adjust the preset alarm threshold. In this step, the number of records of the fire output result is compared with the preset number of times, if the number of errors in the fire identification result is large and exceeds the preset number of times, it is considered that the current preset alarm threshold setting is unreasonable, and the preset alarm threshold needs to be readjusted to adapt to the current real-time scene. Among them, the preset number of times is used to determine whether to adjust the preset alarm threshold. For example, the preset number of times is set to 2 times, when the number of times the fire identification result output by the edge box is recorded to be wrong is greater than 2 times, the preset alarm threshold is readjusted. For another example, the preset number of times is set to 1 time, when the number of times the fire identification result output by the edge box is recorded to be wrong is greater than 1 time, the alarm threshold needs to be readjusted.

[0098] Figure 7 The flowchart of a method for determining an error in a fire identification result according to an embodiment of the present invention is shown, which specifically includes the following steps:

[0099] Step 701, according to the fire identification area, determine the camera related to the fire identification area. As described in step 101, the IPC cameras are distributed in the fire identification area. Therefore, this step determines the camera in the fire identification area or related to the fire identification area according to the fire identification area. Each camera records the picture of at least part of the fire identification area. For example, the fire identification area is an ATM machine in a bank branch. Four IPC cameras are set in the fire identification area, which are respectively installed above the AMT cabinet, on the left side of the ATM cabinet, just above the ATM cabinet door, and on the right side of the ATM cabinet.

[0100] Step 702, compare whether the fire identification results corresponding to the relevant cameras within the predetermined time period are consistent, record the number of inconsistent comparison results, and use the number of inconsistent comparison results as the number of wrong fire identification results. In this step, if the fire identification results corresponding to different cameras in the same fire identification area are inconsistent, the fire identification result finally calculated may be wrong. For example, within the predetermined time, the fire identification result corresponding to the camera installed above the ATM cabinet is: fire occurred; while the fire identification results corresponding to the other three cameras in the fire identification area within the same time are all: no fire occurred. Based on the conclusion that the fire identification results corresponding to all relevant cameras in the area are inconsistent, it can be determined that the comparison results in the fire identification area within the predetermined time period are inconsistent. The number of inconsistent comparisons is used as the number of wrong fire identification results.

[0101] In some embodiments of the present specification, the fire identification results within a predetermined time period may be recorded according to a certain comparison period to determine the number of errors in the fire identification results. The comparison period may be 10 minutes / time, 30 minutes / time, 1 hour / time, or 2 hours / time, etc. According to the comparison period, the fire identification results are recorded regularly to effectively discover the errors in the fire identification results.

[0102] Figure 8 The flowchart of a method for sending fire identification results to a platform in an embodiment of this invention includes the following steps:

[0103] Step 801, obtaining the IP address of the camera. In this step, the camera is an IPC camera. When the edge server obtains the video stream data sent by the camera, it can also obtain the IP address of the camera.

[0104] Step 802, determining the geographical location of fire identification according to the IP address. The IP address of the camera corresponds to the server address of the camera. In this step, the server address of the camera and the geographical location of fire identification can be pre-stored in an address table. This step can further determine the geographical location of fire identification according to the server address of the camera.

[0105] Step 803, associating the geographical location with the fire identification result and sending them to the platform for display. In this step, while sending the fire identification result calculated above to the cloud platform, the geographical location of fire identification and the fire identification result can be sent to the cloud platform for centralized display.

[0106] Fig. 9 The figure shows a schematic diagram of an intelligent fire identification device according to an embodiment of the present invention, comprising:

[0107] The image frame sequence acquisition unit 901 is used to parse the video stream data sent by the camera to obtain an image frame sequence;

[0108] The fire feature recognition result acquisition unit 902 is used to recognize the images in the image frame sequence using the fire feature recognition model to obtain the fire feature recognition result;

[0109] The image similarity recognition result acquisition unit 903 is used to calculate the similarity of the image frames in the image frame sequence within a preset time window to obtain an image similarity recognition result;

[0110] A fire identification score calculation unit 904 is used to assign different weights to the fire feature identification results and the image similarity identification results to calculate a fire identification score;

[0111] The fire identification result output unit 905 is used to output the fire identification result according to the fire identification score and the preset alarm threshold;

[0112] The sending unit 906 is used to send the fire identification result to the platform for centralized display.

[0113] The intelligent fire recognition method of this solution does not need to upload video stream data to the cloud server. The edge server directly calculates the fire recognition algorithm, which reduces the demand for network bandwidth and improves the real-time nature of fire warning. The fire recognition algorithm integrates multi-dimensional information such as time and scene of the image, greatly reducing false positives and missed positives, and can be better applied to complex application scenarios.

[0114] As an embodiment of this article, you can also refer to Fig.10 The figure shows a specific structural diagram of the intelligent fire identification device of this embodiment.

[0115] As an embodiment of this article, the image frame sequence acquisition unit 901 further includes:

[0116] An initial image frame acquisition module 9011 is used to decode the video stream data to obtain an initial image frame;

[0117] The fire feature recognition result acquisition unit 902 further includes: a fire feature recognition model training module 9021, which is used to obtain training samples, fire feature annotation information and fire location annotation information, and train a preset neural network model to determine a fire feature recognition model;

[0118] The segmentation labeling module 9022 is used to add labeling information to the training sample images where the area of ​​the fire location is larger than the preset area using the segmentation labeling method;

[0119] The image similarity recognition result acquisition unit 903 further includes: a preset window size setting module 9031, which is used to determine a preset time window size and a window sliding step;

[0120] The image similarity calculation module 9032 is used to determine the similarity of the image frames in the current preset time window according to the original pixel matrix of each frame of the N frames of images in the preset time window;

[0121] The fire identification score calculation unit 904 further includes: a weight determination module 9041, which is used to assign different weights to the fire classification identification result, the fire feature identification result, and the image similarity identification result, and calculate the fire identification score;

[0122] The fire identification result output unit 905 further includes: a fire identification result verification module 9051, which is used to verify the fire identification results within a predetermined time period and obtain the number of fire identification result errors;

[0123] The preset alarm threshold adjustment module 9052 is used to compare the number of records with the preset number of times, and if the number of records exceeds the preset number of times, adjust the preset alarm threshold.

[0124] The sending unit 906 further includes: a geographic location sending module 9061, which is used to associate the geographic location of the fire identification determined according to the camera IP geology with the fire identification result, and send it to the platform for display.

[0125] like Fig.11 As shown, a computer device provided in an embodiment of the present invention, the computer device 1102 may include one or more processors 1104, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 1102 may also include any memory 1106, which is used to store any kind of information such as code, settings, data, etc. Non-restrictive, for example, the memory 1106 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 1102. In one case, when the processor 1104 executes an associated instruction stored in any memory or a combination of memories, the computer device 1102 may perform any operation of the associated instruction. The computer device 1102 also includes one or more drive mechanisms 1108 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0126] The computer device 1102 may also include an input / output module 1110 (I / O) for receiving various inputs (via input devices 1112) and for providing various outputs (via output devices 1114). A specific output mechanism may include a presentation device 1116 and an associated graphical user interface (GUI) 1118. In other embodiments, the input / output module 1110 (I / O), input device 1112, and output device 1114 may not be included, and the computer device 1102 may be used as a computer device in a network. The computer device 1102 may also include one or more network interfaces 1120 for exchanging data with other devices via one or more communication links 1122. One or more communication buses 1124 couple the components described above together.

[0127] The communication link 1122 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1122 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0128] Corresponds to Figures 1 to 8 The method in this article also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0129] The embodiment of the present invention also provides a computer readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 8 The method shown.

[0130] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0131] It should also be understood that in the embodiments of this article, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0132] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0134] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.

[0136] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

[0137] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0138] Specific embodiments are used in this article to illustrate the principles and implementation methods of this article. The description of the above embodiments is only used to help understand the methods and core ideas of this article. At the same time, for general technicians in this field, according to the ideas of this article, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this article.

Claims

1. An intelligent fire identification method, characterized in that: The method is applied to an edge server and includes: Parse the video stream data sent by the camera to obtain the image frame sequence; Using the fire feature recognition model to recognize the images in the image frame sequence, to obtain a fire feature recognition result, The image frame sequence is identified by using a fire classification model to obtain a fire classification recognition result; the fire classification model training process includes: obtaining a training sample image, the training sample image includes a historical fire image and a normal image with fire type annotation information, the fire type annotation information includes: whether the training sample image contains a fire; using the training sample image to train a preset classification network to obtain a fire classification recognition model; Calculating the image frame similarity of the image frame sequence within a preset time window to obtain an image similarity recognition result; Assigning different weights to the fire feature recognition result and the image similarity recognition result respectively, and calculating a fire recognition score, which includes: assigning different weights to the fire classification recognition result, the fire feature recognition result, and the image similarity recognition result respectively, and calculating a fire recognition score; Outputting a fire identification result according to the fire identification score and a preset alarm threshold; The fire identification results are sent to the platform for centralized display.

2. The intelligent fire identification method according to claim 1, characterized in that: The step of parsing the video stream data sent by the camera to obtain an image frame sequence includes: Decoding the video stream data to obtain an initial image frame; Determine the sampling frequency and number of samples for the initial image frame according to the frame rate of the acquired video stream data; The initial image frame is sampled according to the sampling frequency and sampling quantity of the initial image frame to obtain an image frame sequence.

3. The intelligent fire identification method according to claim 2, characterized in that: The fire classification recognition result, the fire feature recognition result, and the image similarity recognition result are respectively assigned different weights, and the fire recognition score is calculated, including: The fire identification score is determined using the following formula: score total =M×score b +N×(score a +score c ) / 2, where score total is the fire identification score, score a is the fire feature recognition result, score b is the image similarity recognition result, score c is the fire classification recognition result, M is the weight of the image similarity recognition result, and N is the weight of the fire feature recognition result and the fire classification recognition result.

4. The intelligent fire identification method according to claim 2, characterized in that: Assigning different weights to the fire classification recognition result, the fire feature recognition result, and the image similarity recognition result respectively includes: Querying the weights corresponding to the fire classification recognition result, the fire feature recognition result and the image similarity from a preset weight table; The preset weight table stores the correspondence between fire classification and weight, the correspondence between fire characteristics and weight, and the correspondence between image similarity range and weight.

5. The intelligent fire identification method according to claim 1, characterized in that: The training process of the fire feature recognition model includes: Acquire training sample images, wherein the training sample images include historical fire images and normal images with fire feature annotation information and fire location annotation information; Converting the fire feature annotation information of the training sample image into a fire feature confidence score; Converting the fire location annotation information of the training sample image into fire feature coordinate information; Taking the training sample image as input, the fire feature confidence score and the fire feature coordinate information as output, training parameters in a preset neural network model; The fire feature recognition model is determined according to the structure and parameters of the preset neural network model.

6. The intelligent fire identification method according to claim 5, characterized in that: After obtaining the training sample image, it also includes: Calculating the area of ​​the fire location in the training sample image; When the area of ​​the fire location is larger than the preset area, performing image segmentation on the training sample image; The segmented labeling method is used to add labeling information to the target features in the segmented training sample images.

7. The intelligent fire identification method according to claim 1, characterized in that: Calculating the image frame similarity of the image frame within a preset time window to obtain the image similarity recognition result includes: Determine the preset time window size and window sliding step; Determine the similarity of image frames in the current preset time window according to the original pixel matrix of each image frame in the N image frames in the preset time window; The preset time window is moved according to the window sliding step to determine the similarity of images in the next time window.

8. The intelligent fire identification method according to claim 1, characterized in that: The method further comprises: Verify the fire identification results within a predetermined time period to obtain the number of times the fire identification results are wrong; The recorded number of times is compared with the preset number of times, and if the recorded number of times exceeds the preset number of times, the preset alarm threshold is adjusted.

9. The intelligent fire identification method according to claim 8, characterized in that: The fire identification results within the predetermined time period are verified, and the number of times the fire identification results are wrong includes: According to the fire identification area, determine the cameras related to the fire identification area; Compare whether the fire identification results corresponding to the relevant cameras within a predetermined time period are consistent, record the number of times the comparison results are inconsistent, and use the number of times the comparison results are inconsistent as the number of times the fire identification results are wrong.

10. The intelligent fire identification method according to claim 1, characterized in that: The sending of the fire identification result to the platform for centralized display also includes: Obtain the IP address of the camera; Determining the geographical location of fire identification based on the IP address; The geographic location is associated with the fire identification result and sent to the platform for display.

11. An intelligent fire identification device, characterized in that: The device comprises: An image frame sequence acquisition unit is used to parse the video stream data sent by the camera to obtain an image frame sequence; a fire feature recognition result acquisition unit, configured to recognize images in the image frame sequence using a fire feature recognition model to obtain a fire feature recognition result; The image frame sequence is identified by using a fire classification model to obtain a fire classification recognition result; the fire classification model training process includes: obtaining a training sample image, the training sample image includes a historical fire image and a normal image with fire type annotation information, the fire type annotation information includes: whether the training sample image contains a fire; using the training sample image to train a preset classification network to obtain a fire classification recognition model; An image similarity recognition result acquisition unit is used to calculate the image frame similarity of the image frame sequence within a preset time window to obtain an image similarity recognition result; A fire identification score calculation unit is used to assign different weights to the fire feature identification result and the image similarity identification result, and calculate the fire identification score, which includes: assigning different weights to the fire classification identification result, the fire feature identification result, and the image similarity identification result, and calculating the fire identification score; A fire identification result output unit, configured to output a fire identification result according to the fire identification score and a preset alarm threshold; The sending unit is used to send the fire identification result to the platform for centralized display.

12. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images