Fire early warning method and device, electronic equipment and storage medium

Abstract

The invention discloses a fire early warning method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining image data information of a monitoring region, carrying out the edge feature preprocessing, and obtaining a multi-mode suspected fire image feature; performing fire image recognition processing on the multi-modal suspected fire image features based on a neural residual network structure to obtain a fire image recognition result; and carrying out visual display on the fire image recognition result based on the graphical user interface, and carrying out fire early warning according to the display result. The method can improve the response speed of the initial fire and reduce the false alarm rate and missing report rate of the fire, thereby shortening the early warning response time, and can be widely applied to the technical field of image recognition.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a fire warning method, device, electronic equipment and storage medium. Background Art

[0002] With the rapid development of technologies such as the Internet of Things, big data, and artificial intelligence, there is still room for improvement in the intelligence and automation level of existing fire warning systems. Existing fire warning systems mostly rely on fixed fire warning detection units, such as smoke alarms or temperature sensors, which monitor ambient temperature and smoke concentration to determine whether a fire has occurred. However, these devices are usually installed in key areas, such as warehouses and computer rooms, but their coverage is limited and their response speed to initial fires is slow. With the rapid development of smart homes, cameras have become an important tool for security monitoring. However, traditional cameras have limitations in fire warning. Existing monitoring systems use image recognition technology to analyze monitoring images in real time to achieve an intelligent system that can automatically identify fire risks and issue immediate warnings. Existing fire warnings based on image recognition require the intervention of lasers and other means, which are not suitable for living and office places. In addition, cameras are used to collect real-time data in the monitoring area and determine whether there are fire characteristics through image recognition algorithms. This method has all-weather and all-round monitoring capabilities, but is limited by the resolution of the camera and the accuracy of the algorithm, and there may be false alarms or missed alarms.

[0003] Therefore, the related technology has at least the following problems: the response speed to the initial fire is slow and there are false alarms or missed alarms, resulting in the inability to detect fire information and issue warnings in a timely and accurate manner. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the present invention provides a fire warning method, device, electronic device and storage medium, which can improve the response speed of the initial fire and reduce the false alarm rate and missed alarm rate of the fire, thereby shortening the warning response time.

[0005] In one aspect, an embodiment of the present invention provides a fire warning method, comprising:

[0006] Obtain image data information of the monitored area and perform edge feature preprocessing to obtain multi-modal suspected fire image features;

[0007] Based on the neural residual network structure, the multimodal suspected fire image features are processed for fire image recognition to obtain the fire image recognition result;

[0008] The fire image recognition results are visualized based on the graphical user interface and fire warnings are issued based on the displayed results.

[0009] Optionally, image data information of the monitored area is obtained and edge feature preprocessing is performed to obtain multimodal suspected fire image features, including:

[0010] Obtain image data information of the monitored area and perform frame processing to obtain image data information of several frames to be processed;

[0011] Performing image data preprocessing on a number of to-be-processed frame image data information to obtain preprocessed image data information;

[0012] Perform multi-modal edge feature extraction on the pre-processed image data information to obtain multi-modal fire image features;

[0013] The multimodal fire image features are processed by convolutional neural network to obtain multimodal suspected fire image features.

[0014] Optionally, image data information of the monitoring area is obtained and frame processing is performed to obtain a plurality of frame image data information to be processed, including:

[0015] Acquire image data information of the monitoring area through monitoring equipment;

[0016] According to the image monitoring time in the image data information, the image data information of the monitoring area is frame-processed through a computer vision platform to obtain a number of frame image data information to be processed.

[0017] Optionally, multimodal edge feature extraction is performed on the preprocessed image data information to obtain multimodal fire image features, including:

[0018] By setting a preset color threshold, flame color features are extracted from the preprocessed image data information to obtain flame color features of the fire image;

[0019] The flame edge shape feature extraction process is performed on the pre-processed image data information by using the edge detection algorithm to obtain the flame edge shape feature of the fire image;

[0020] The flame texture feature extraction process is performed on the pre-processed image data information by using a texture analysis algorithm to obtain the flame texture feature of the fire image;

[0021] The flame motion trajectory feature extraction process is performed on the pre-processed image data information by using the optical flow tracking method to obtain the flame motion feature of the fire image;

[0022] The multimodal fire image features are obtained by combining the flame color features of the fire image, the flame edge shape features of the fire image, the flame texture features of the fire image and the flame motion features of the fire image.

[0023] Optionally, the neural residual network structure includes an input layer, a convolutional layer, a maximum pooling layer, a first residual block group, a second residual block group, a third residual block group, a fourth residual block group, a global average pooling layer, a fully connected layer and an output layer, wherein:

[0024] The convolution layer includes the initial convolution layer and the maximum pooling layer;

[0025] The first residual block group, the second residual block group, the third residual block group and the fourth residual block group each have a plurality of residual blocks, and the first residual block group, the second residual block group, the third residual block group and the fourth residual block are connected via residual connections.

[0026] Optionally, fire image recognition processing is performed on multimodal suspected fire image features based on a neural residual network structure to obtain a fire image recognition result, including:

[0027] Input the multimodal suspected fire image features into the neural residual network structure;

[0028] Based on the input layer of the neural residual network structure, multi-modal suspected fire image features are obtained;

[0029] Based on the convolution layer of the neural residual network structure, the multi-modal suspected fire image features are convolved to obtain a preliminary multi-modal fire feature image;

[0030] Based on the maximum pooling layer of the neural residual network structure, the preliminary multimodal fire feature image is subjected to spatial dimensionality reduction processing to obtain the multimodal fire feature image after dimensionality reduction;

[0031] Based on the first residual block group, the second residual block group, the third residual block group and the fourth residual block group of the neural residual network structure, target feature extraction processing is performed on the multimodal fire feature image after dimensionality reduction in sequence to obtain a multimodal fire feature image;

[0032] Based on the global average pooling layer of the neural residual network structure, the multimodal fire feature image is subjected to channel compression processing to obtain the multimodal fire image feature vector;

[0033] Based on the fully connected layer of the neural residual network structure, the multimodal fire image feature vectors are identified and classified to obtain the fire image recognition results;

[0034] Based on the output layer of the neural residual network structure, the fire image recognition results are output.

[0035] Optionally, visually displaying the fire image recognition result based on a graphical user interface and issuing a fire warning according to the display result include:

[0036] Visualize the fire image recognition results based on the graphical user interface to obtain a two-dimensional visualization display result of the fire image;

[0037] The 2D visualization display result of the fire image is mapped and superimposed by a 3D rendering algorithm to obtain a 3D visualization display result of the fire image;

[0038] According to the three-dimensional visualization display results of the fire image, judgment is made to determine the location of the fire source and issue a fire warning.

[0039] On the other hand, an embodiment of the present invention provides a fire warning device, including:

[0040] The first module is used to obtain image data information of the monitoring area and perform edge feature preprocessing to obtain multi-modal suspected fire image features;

[0041] The second module is used to perform fire image recognition processing on multimodal suspected fire image features based on the neural residual network structure to obtain fire image recognition results;

[0042] The third module is used to visualize the fire image recognition results based on the graphical user interface and issue a fire warning based on the displayed results.

[0043] Optionally, the first module is specifically used for:

[0044] Obtain image data information of the monitored area and perform frame processing to obtain image data information of several frames to be processed;

[0045] Performing image data preprocessing on a number of to-be-processed frame image data information to obtain preprocessed image data information;

[0046] Perform multi-modal edge feature extraction on the pre-processed image data information to obtain multi-modal fire image features;

[0047] The multimodal fire image features are processed by convolutional neural network to obtain multimodal suspected fire image features.

[0048] Optionally, the system further comprises:

[0049] The fourth module is used to obtain image data information of the monitored area and perform edge feature preprocessing to obtain multi-modal suspected fire image features;

[0050] The fifth module is used to perform multimodal edge feature extraction processing on the pre-processed image data information to obtain multimodal fire image features;

[0051] Optionally, the fourth module is specifically used for:

[0052] Obtain image data information of the monitored area and perform frame processing to obtain image data information of several frames to be processed;

[0053] Performing image data preprocessing on a number of to-be-processed frame image data information to obtain preprocessed image data information;

[0054] Perform multi-modal edge feature extraction on the pre-processed image data information to obtain multi-modal fire image features;

[0055] The multimodal fire image features are processed by convolutional neural network to obtain multimodal suspected fire image features.

[0056] Optionally, the system further comprises:

[0057] The sixth module is used to obtain image data information of the monitoring area and perform frame processing to obtain a number of frame image data information to be processed;

[0058] Optionally, the sixth module is specifically used for:

[0059] Acquire image data information of the monitoring area through monitoring equipment;

[0060] According to the image monitoring time in the image data information, the image data information of the monitoring area is frame-processed through a computer vision platform to obtain a number of frame image data information to be processed.

[0061] Optionally, the fifth module is specifically used for:

[0062] By setting a preset color threshold, flame color features are extracted from the preprocessed image data information to obtain flame color features of the fire image;

[0063] The flame edge shape feature extraction process is performed on the pre-processed image data information by using the edge detection algorithm to obtain the flame edge shape feature of the fire image;

[0064] The flame texture feature extraction process is performed on the pre-processed image data information by using a texture analysis algorithm to obtain the flame texture feature of the fire image;

[0065] The flame motion trajectory feature extraction process is performed on the pre-processed image data information by using the optical flow tracking method to obtain the flame motion feature of the fire image;

[0066] The multimodal fire image features are obtained by combining the flame color features of the fire image, the flame edge shape features of the fire image, the flame texture features of the fire image and the flame motion features of the fire image.

[0067] Optionally, the system further includes a seventh module:

[0068] The seventh module is specifically used for the neural residual network structure including an input layer, a convolution layer, a maximum pooling layer, a first residual block group, a second residual block group, a third residual block group, a fourth residual block group, a global average pooling layer, a fully connected layer and an output layer, wherein:

[0069] The convolution layer includes the initial convolution layer and the maximum pooling layer;

[0070] The first residual block group, the second residual block group, the third residual block group and the fourth residual block group each have a plurality of residual blocks, and the first residual block group, the second residual block group, the third residual block group and the fourth residual block are connected via residual connections.

[0071] Optionally, the second module is specifically used for:

[0072] Input the multimodal suspected fire image features into the neural residual network structure;

[0073] Based on the input layer of the neural residual network structure, multi-modal suspected fire image features are obtained;

[0074] Based on the convolution layer of the neural residual network structure, the multi-modal suspected fire image features are convolved to obtain a preliminary multi-modal fire feature image;

[0075] Based on the maximum pooling layer of the neural residual network structure, the preliminary multimodal fire feature image is subjected to spatial dimensionality reduction processing to obtain the multimodal fire feature image after dimensionality reduction;

[0076] Based on the first residual block group, the second residual block group, the third residual block group and the fourth residual block group of the neural residual network structure, target feature extraction processing is performed on the multimodal fire feature image after dimensionality reduction in sequence to obtain a multimodal fire feature image;

[0077] Based on the global average pooling layer of the neural residual network structure, the multimodal fire feature image is subjected to channel compression processing to obtain the multimodal fire image feature vector;

[0078] Based on the fully connected layer of the neural residual network structure, the multimodal fire image feature vectors are identified and classified to obtain the fire image recognition results;

[0079] Based on the output layer of the neural residual network structure, the fire image recognition results are output.

[0080] Optionally, the third module is specifically used for:

[0081] Visualize the fire image recognition results based on the graphical user interface to obtain a two-dimensional visualization display result of the fire image;

[0082] The 2D visualization display result of the fire image is mapped and superimposed by a 3D rendering algorithm to obtain a 3D visualization display result of the fire image;

[0083] According to the three-dimensional visualization display results of the fire image, judgment is made to determine the location of the fire source and issue a fire warning.

[0084] On the other hand, an embodiment of the present invention provides an electronic device, including: a processor and a memory; the memory is used to store programs; the processor executes the program to implement the above-mentioned fire warning method.

[0085] On the other hand, an embodiment of the present invention provides a computer storage medium, in which a program executable by a processor is stored. The program executable by the processor is used to implement the above-mentioned fire warning method when executed by the processor.

[0086] The beneficial effects of the method, device, electronic device and medium of the present invention are as follows: first, by acquiring image data information of the monitoring area and performing edge feature preprocessing, multimodal suspected fire image features are acquired; by combining real-time image acquisition technology with feature recognition technology of a neural network, the source of the fire can be discovered in time; by using multimodal suspected fire image features, tiny fire flames can be determined in time, thereby improving the response speed of an initial fire; further, fire image recognition processing is performed on the multimodal suspected fire image features through a neural residual network structure to obtain a fire image recognition result; further, the suspected fire image is secondary recognized through a neural network to improve the processing accuracy of the image, thereby reducing the false alarm rate and missed alarm rate of the fire; finally, the fire image recognition results are visualized based on a graphical user interface and a fire warning is issued according to the displayed results; the detection results can be warned in time, the location of the fire can be determined in time through the visualization display, thereby shortening the warning response time. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0088] Figure 1 This is a schematic diagram of an implementation environment for fire warning provided by an embodiment of the present invention;

[0089] Figure 2 It is a flow chart of a fire warning method provided by an embodiment of the present invention;

[0090] Figure 3 A schematic diagram of a process for obtaining multi-modal suspected fire image features provided by an embodiment of the present invention;

[0091] Figure 4A schematic diagram of a process for obtaining frame image data information to be processed provided by an embodiment of the present invention;

[0092] Figure 5 A schematic diagram of a process for obtaining multimodal fire image features provided by an embodiment of the present invention;

[0093] Figure 6 is a schematic diagram of the structure of a neural residual network provided by an embodiment of the present invention;

[0094] Figure 7 A schematic diagram of a process for performing a fire warning according to an embodiment of the present invention;

[0095] Figure 8 A structural schematic diagram of a fire warning device provided by an embodiment of the present invention;

[0096] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0097] Fig.10 A block diagram of the computer system structure of an electronic device suitable for implementing the embodiment of the present invention is provided for the embodiment of the present invention. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0099] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0100] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0101] It is understandable that the fire warning method provided in the embodiment of the present invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this.

[0102] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0103] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0104] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0105] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present invention.

[0106] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a fire warning method, which is described below by taking the fire warning method applied in the server 101 as an example. It can be understood that the fire warning method can also be applied in the terminal 102.

[0107] First, the technical terms of the present invention are explained:

[0108] Image recognition: Image recognition is an important field of artificial intelligence, which refers to the technology of object recognition in images to identify targets and objects of various different patterns. It extracts key features from images and compares them with known patterns to identify specific content in images.

[0109] Deep learning: Deep learning is a machine learning method based on artificial neural networks that can automatically extract features and recognize patterns from large amounts of data to achieve complex tasks. In the field of image recognition, deep learning, especially convolutional neural networks (CNNs), is widely used.

[0110] ResNet50 residual neural network: ResNet50 is a type of residual network, consisting of multiple residual blocks, and is widely used in fields such as target classification. The network solves the gradient vanishing and gradient exploding problems in deep network training by introducing residual connections, and can build a deeper network structure.

[0111] Reference Figure 2 , Figure 2 The flowchart of the fire warning method applied to the server provided by the embodiment of the present invention, the execution subject of the fire warning method can be any of the aforementioned computer devices. Figure 2 , the method comprises the following steps:

[0112] S100, acquiring image data information of the monitoring area and performing edge feature preprocessing to obtain multi-modal suspected fire image features;

[0113] It should be noted that in some embodiments, Figure 3 As shown, step S100 may include S110 to S140;

[0114] S110, acquiring image data information of the monitoring area and performing frame processing to obtain a plurality of frame image data information to be processed;

[0115] like Figure 4 As shown, in some embodiments, step S110 may include but is not limited to steps S111 to S112;

[0116] S111, obtaining image data information of the monitoring area through the monitoring device;

[0117] In this embodiment, it should be noted that this embodiment provides a fire warning method for living and office places, such as warehouses, factories, residential buildings and other key areas. Further, the camera is used as an image acquisition device and deployed in the area that needs to be monitored. The fire warning of the living and office place can include a monitoring device and a processor that is communicatively connected to the monitoring device. The monitoring device is installed at at least one location in the living and office place, and the monitoring device is used to collect video surveillance images of the living and office place. The processor is used to obtain the video surveillance images of the living and office place, and perform frame processing on the video surveillance images to obtain multiple frame images to be processed.

[0118] S112. According to the image monitoring time in the image data information, the image data information of the monitoring area is processed by a computer vision platform to obtain a plurality of frame image data information to be processed.

[0119] In this embodiment, the processor can obtain the video surveillance image of the living and working place from the monitoring device, or the processor can obtain the video surveillance image of the living and working place from the monitoring device through the wireless transmission module. Among them, the monitoring device can use WiFi communication mode (or Bluetooth transmission mode) to send the video surveillance image to the wireless transmission module, so that the wireless transmission module can use WiFi communication mode (or Bluetooth transmission mode) to send the video surveillance image to the processor.

[0120] Further, it should be noted that, in the present embodiment, the processor can be implemented in various possible ways to perform frame processing on the video surveillance image to obtain a plurality of frame images to be processed. In some embodiments, the video surveillance image can be frame processed according to the image monitoring time in the video surveillance image using the computer vision platform opencv to obtain a plurality of frame images to be processed. Among them, the earlier the image monitoring time, the earlier the frame order of the corresponding frame image to be processed. OpenCV is a cross-platform computer vision and machine learning software library released based on the Apache2.0 license (open source), which can run on Linux, Windows, Android and Mac OS operating systems. OpenCV is lightweight and efficient. Specifically, OpenCV consists of a series of C functions and a small number of C++ classes, and provides interfaces for languages ​​such as Python, Ruby, and MATLAB, and implements many general algorithms in image processing and computer vision.

[0121] S120, performing image data preprocessing on a plurality of to-be-processed frame image data information to obtain preprocessed image data information;

[0122] In this embodiment, the acquired frame image data information to be processed is preprocessed to improve the accuracy of subsequent processing. The preprocessing steps include adjusting the image size, removing noise, enhancing contrast, graying, smoothing filtering (such as Gaussian blur), and color space conversion (from RGB to HSV or YCrCb), etc., matching the characteristics of suspected fire images in the data, and filtering out images with flame characteristics.

[0123] Furthermore, it should be noted that the purpose of converting the frame image to be processed from RGB to HSV or YCrCb is to better meet the specific needs of image processing, improve processing efficiency, optimize storage and transmission, and enhance visual effects. The RGB color space is not the most intuitive or effective in some image processing tasks. For example, when performing color segmentation, the RGB color space is not as intuitive as the HSV color space, because HSV uses more intuitive data to describe color, where H represents color, S represents depth, and V represents brightness. This makes HSV more useful when performing color segmentation. Through the division of thresholds, colors can be distinguished more accurately. The YCrCb color space (also known as YUV) separates the brightness signal of the color from the chrominance signal. This representation method can use the characteristics of the human eye to reduce the storage space of digital color images. The human visual system is more sensitive to brightness details than color details. Appropriately reducing the chrominance resolution will not significantly affect the image quality. Therefore, the YCrCb color space has advantages in applications such as video compression and transmission.

[0124] S130, performing multimodal edge feature extraction processing on the preprocessed image data information to obtain multimodal fire image features;

[0125] like Figure 5 As shown, in some embodiments, step S130 may include but is not limited to steps S131 to S135;

[0126] S131, extracting flame color features from the preprocessed image data information by setting a preset color threshold, to obtain flame color features of the fire image;

[0127] First of all, it should be noted that flames usually have a specific color range. In the HSV or YCrCb color space, flames tend to be in areas with higher brightness and a wider hue range. Therefore, flames can be detected by setting a color threshold.

[0128] Therefore, in this embodiment, for the HSV color space, the method of determining the threshold value can be to set the lower and upper threshold values ​​of HSV according to the color characteristics of the flame. For example, the hue (Hue) of the flame is generally between 10 and 25, and the saturation (Saturation) and brightness (Value) should be high. Similarly, for the YCrCb color space, the lower and upper limits of YCrCb are set to cover the typical colors of the flame. This range usually needs to be debugged according to the actual situation and the image, and the cv2.inRange function is further used to generate a mask. This mask will set the area that meets the color threshold range to white (255) and the remaining area to black (0), and then use the mask to extract the flame part in the image. The cv2.bitwise_and function can be used to apply the mask to the original image, so that only the flame part is retained. Finally, the original image, the mask and the extracted flame area are displayed to obtain the flame color characteristics of the fire image.

[0129] S132, performing flame edge shape feature extraction processing on the pre-processed image data information by using an edge detection algorithm to obtain flame edge shape features of the fire image;

[0130] First, it should be noted that the shape of a flame is usually irregular, but its shape features can be extracted by edge detection, contour extraction, etc. For example, an edge detection algorithm (such as Canny edge detection) is used to detect the edge of the flame, and a contour extraction algorithm is used to obtain the contour of the flame.

[0131] Therefore, in this embodiment, first, if the preprocessed image is still a color image, it can be converted into a grayscale image. Use the cv2.cvtColor function to convert the image into a grayscale mode, further perform noise removal, use Gaussian filtering or median filtering and other methods to remove noise in the image to improve the effect of edge detection, and then perform edge detection, select a suitable edge detection algorithm, such as Canny edge detection, use a suitable threshold to extract the edge of the flame, and finally perform edge feature extraction and analyze the extracted edge contour. First, use the cv2.findContours function to extract the contour of the flame from the edge image, and then analyze the extracted contour, such as calculating the contour area, perimeter, shape features (such as aspect ratio, roundness, etc.) to obtain the shape features of the flame.

[0132] S133, performing flame texture feature extraction processing on the pre-processed image data information by using a texture analysis algorithm to obtain flame texture features of the fire image;

[0133] First, it should be noted that flames have unique texture features, and texture descriptors (such as Gabor filters) or other texture analysis techniques can be used to extract flame textures.

[0134] Therefore, in this embodiment, first, similarly to step S132, the image is grayed out. If the preprocessed image is in color, it is first converted into a grayscale image for texture analysis, and then Gaussian filtering or median filtering is used to remove noise in the image to improve the accuracy of subsequent texture feature extraction, and a suitable texture analysis algorithm is further selected, such as gray level co-occurrence matrix (GLCM), local binary pattern (LBP) or other texture analysis algorithms.

[0135] The embodiment of the present invention is described by taking the gray level co-occurrence matrix as an example. First, the gray level co-occurrence matrix is ​​used to calculate the texture features of the flame. GLCM describes the texture features by calculating the gray value combination frequency between pixel pairs in the image, and then calculates different texture features from GLCM, such as contrast, correlation, energy and uniformity.

[0136] In addition, it should be noted that the local binary pattern (LBP) method can be used to obtain texture features. The LBP algorithm generates a binary pattern by comparing the local area pixels with the central pixel to reflect the local texture, and then converts the LBP features into a histogram for subsequent classification or analysis. Finally, the extracted texture features can be visualized or presented through charts, such as drawing a histogram.

[0137] S134, performing flame motion trajectory feature extraction processing on the pre-processed image data information by optical flow tracking method to obtain flame motion features of the fire image;

[0138] First of all, it should be noted that the movement of flames has a certain directionality and speed. The movement trajectory of the flame can be tracked by the optical flow method, and then its movement pattern characteristics can be extracted.

[0139] Therefore, in this embodiment, first, similarly to step S132, the image is first grayed. If the pre-processed image is in color, it is first converted into a grayscale image for texture analysis, and then Gaussian filtering or median filtering is used to remove noise in the image to improve the accuracy of subsequent texture feature extraction. An optical flow algorithm is further selected, such as the Lucas-Kanade method or the pyramid Lucas-Kanade method. The embodiment of the present invention is described by taking the Lucas-Kanade method as an example. First, the optical flow of the two frames before and after is calculated, and the optical flow method is used to calculate the optical flow vector between the two frames before and after. The motion trajectory is further calculated, and the motion trajectory and characteristics of the flame are calculated according to the optical flow vector. For example, the motion mode of the flame can be determined by calculating the size and direction of the optical flow, and then the optical flow is visualized. The calculated optical flow is visualized, and arrows can usually be used to indicate the motion direction and speed. Finally, the motion features are counted, and the motion feature statistical information of the flame is extracted, such as the mean, variance and continuity of the motion speed, etc. The motion trajectory of the flame is calculated according to the optical flow vector and the time step, and then its motion mode characteristics are extracted.

[0140] S135. Combining the fire image flame color features, the fire image flame edge shape features, the fire image flame texture features and the fire image flame motion features to obtain a multimodal fire image feature.

[0141] S140, performing image recognition processing on the multimodal fire image features through a convolutional neural network to obtain multimodal suspected fire image features.

[0142] In some specific embodiments, a convolutional neural network (CNN) is selected for the recognition and classification of flame features. The CNN model consists of multiple layers of neural networks, which can extract image features layer by layer, from simple to complex, and ultimately achieve understanding and classification of images.

[0143] It should be noted that in the specific embodiment of the present invention, a large number of labeled flame and non-flame images are first used as training data to train the CNN model. During the training process, the model will learn how to extract and identify the characteristics of flames, and gradually form a deep understanding of flame characteristics.

[0144] Furthermore, the preprocessed image is input into the trained CNN model, which extracts and classifies the features in the image and finally outputs the recognition result. If the image contains flame features, the model will recognize it as flame; otherwise, it will be recognized as a non-flame image. At this time, only preliminary multimodal fire image features, i.e., multimodal suspected fire image features, are obtained.

[0145] S200, performing fire image recognition processing on multimodal suspected fire image features based on a neural residual network structure to obtain a fire image recognition result;

[0146] First of all, it should be noted that the neural residual network in the embodiment of the present invention is a ResNet50 residual neural network. The ResNet50 network is used for early fire monitoring. ResNet50 is a deep residual network that contains 50 convolutional layers and has powerful feature extraction capabilities. The processing process is basically the same as the preliminary detection image, but the data model processed in the cloud is mature in scale and can effectively improve the accuracy. The introduction of residual connection effectively solves the problems of gradient disappearance and performance saturation in deep networks.

[0147] Furthermore, if Figure 6 As shown, the ResNet50 residual neural network structure in an embodiment of the present invention is illustrated, and the neural residual network structure includes an input layer, a convolution layer, a maximum pooling layer, a first residual block group, a second residual block group, a third residual block group, a fourth residual block group, a global average pooling layer, a fully connected layer and an output layer, wherein the convolution layer includes an initial convolution layer and a maximum pooling layer, and the first residual block group, the second residual block group, the third residual block group and the fourth residual block group all have a number of residual blocks, and the first residual block group, the second residual block group, the third residual block group and the fourth residual block are connected through residuals.

[0148] Furthermore, it should be noted that in some embodiments, ResNet50 has a total of 50 layers in depth, including the following main parts. First, the first part is the input layer, and its input image is usually an RGB three-channel image with a size of 224x224. After preprocessing (such as normalization to the [0,1] interval or standardization), the image is sent to the network. The second part is the convolution layer (ConvLayer), which specifically includes an initial convolution layer and a maximum pooling layer, wherein the initial convolution layer is a 7x7 convolution layer with a stride of 2 and an output channel number of 64, followed by Batch Normalization and ReLU activation function, and the maximum pooling layer is a 3x3 maximum pooling layer with a stride of 2. The third part is a residual block combination. ResNet50 is composed of multiple residual blocks stacked together, each residual block contains two 3x3 convolution layers, and the output of the residual block is generated by adding input (i.e., residual connection) to form a shortcut connection (skip). connection), ResNet50 is usually divided into four stages (first residual block group, second residual block group, third residual block group, fourth residual block group), each stage contains a different number of residual blocks, among which the first residual block group has 3 residual blocks, the second residual block group has 4 residual blocks, the third residual block group has 6 residual blocks, and the fourth residual block group has 3 residual blocks. The fourth part is the global pooling layer (Global Average Pooling). After the last residual block, the global average pooling layer (Global Average Pooling) is used to compress the value of each channel of the feature map into a numerical value. The sixth part is the output layer, which is the fully connected layer (Fully Connected Layer): a fully connected layer, the output is the number of categories (for example, for a two-classification task, namely fire and non-fire, the output is 2). The Softmax activation function is applied to calculate the probability distribution of each category.

[0149] Furthermore, it should be noted that, in some embodiments, step S200 may include S210 to S280:

[0150] S210, inputting the multimodal suspected fire image features into the neural residual network structure;

[0151] S220, based on the input layer of the neural residual network structure, obtaining multi-modal suspected fire image features;

[0152] S230, based on the convolution layer of the neural residual network structure, performing convolution processing on the multimodal suspected fire image features to obtain a preliminary multimodal fire feature image;

[0153] In this embodiment, it should be noted that the convolution layer of the neural residual network structure is used to automatically extract the features of the input image, and convolve the input data through the convolution kernel (filter) to generate a feature map. Each convolution layer usually increases the nonlinear expression ability of the network by using a nonlinear activation function (such as ReLU).

[0154] S240, based on the maximum pooling layer of the neural residual network structure, performing spatial dimensionality reduction processing on the preliminary multimodal fire feature image to obtain a multimodal fire feature image after dimensionality reduction;

[0155] In this embodiment, it should be noted that the role of the maximum pooling layer of the neural residual network structure is to reduce the spatial dimension of the feature map, reduce the amount of calculation, reduce the risk of overfitting, and extract the most significant features. Important features are retained by taking the maximum value in the local area.

[0156] S250, based on the first residual block group, the second residual block group, the third residual block group and the fourth residual block group of the neural residual network structure, sequentially perform target feature extraction processing on the multimodal fire feature image after dimensionality reduction to obtain the multimodal fire feature image;

[0157] S260, performing channel compression processing on the multimodal fire feature image based on a global average pooling layer of a neural residual network structure to obtain a multimodal fire image feature vector;

[0158] In this embodiment, it should be noted that the role of the global average pooling layer of the neural residual network structure is to reduce the spatial dimension of each feature map, average all spatial positions, and output a feature vector of fixed length. This method can effectively reduce the number of model parameters and reduce overfitting while maintaining the expressiveness of the model.

[0159] S270, based on the fully connected layer of the neural residual network structure, performing recognition and classification processing on the multimodal fire image feature vector to obtain the fire image recognition result;

[0160] In this embodiment, it should be noted that the fully connected layer of the neural residual network structure is used to connect the feature vector from the global average pooling layer to the output layer, which is usually used for classification tasks. In this way, the network can use the extracted features to make classification decisions.

[0161] S280, an output layer based on a neural residual network structure, outputs fire image recognition results.

[0162] In this embodiment, it should be noted that the output layer of the neural residual network structure is usually a Softmax layer, which outputs the probability distribution of each category. This layer provides the model with confidence that the input sample belongs to each class, supporting multi-class classification tasks.

[0163] S300, visually displaying the fire image recognition result based on the graphical user interface and issuing a fire warning according to the display result;

[0164] It should be noted that in some embodiments, Figure 7 As shown, step S300 may include S310 to S330:

[0165] S310, visually displaying the fire image recognition result based on the graphical user interface to obtain a two-dimensional visual display result of the fire image;

[0166] In this embodiment, the output of the output layer is used to output the content of the graphical user interface (GUI) to confirm whether there is a fire hazard. Specifically, a library for image processing (such as PIL or OpenCV) is installed for fire detection model training. A simple interface is designed based on the GUI library, including a button for loading an image, an area for displaying the original image, an area for displaying the processed image, and an area for displaying the recognition result. According to the output of the model, it is analyzed whether a fire is detected, and the recognition result is displayed in the corresponding area of ​​the GUI interface, such as "fire detected" or "fire not detected". The cloud server uses the trained fire recognition model to perform in-depth analysis of the image to confirm whether there is a fire hazard.

[0167] S320, mapping and overlaying the two-dimensional visualization display result of the fire image by a three-dimensional rendering algorithm to obtain a three-dimensional visualization display result of the fire image;

[0168] In this embodiment, after confirming the fire, AR technology is used to generate a three-dimensional model of the fire scene on the cloud server, and the real-time camera image is combined for superposition display. First, a coordinate mapping relationship is established, and a mapping relationship between two-dimensional image pixels and three-dimensional space coordinates is established based on geographic information data. This usually involves the conversion between the geographic coordinate system (such as longitude and latitude) and the three-dimensional Cartesian coordinate system, and further texture mapping is performed to map the two-dimensional image as a texture to the corresponding object or surface in the three-dimensional scene. This requires the use of texture mapping technology to ensure the correct display of the texture in the three-dimensional scene, and then build the three-dimensional scene and render it, and use three-dimensional modeling software or engines (such as Unity, UE4, etc.) to build a three-dimensional scene of the fire scene. This includes adding elements such as terrain, buildings, and vegetation, and adjusting their position, size, and shape to match the actual scene, adjusting rendering parameters such as viewing angle, lighting conditions, etc. as needed to obtain the best visualization effect, and finally performing superposition processing to superimpose the rendered three-dimensional scene with the original two-dimensional fire image. This can be achieved through image synthesis technology to ensure seamless integration between the two-dimensional image and the three-dimensional scene.

[0169] S330: Make a judgment based on the three-dimensional visualization display result of the fire image, determine the location of the fire source and issue a fire warning.

[0170] It should be noted that in the embodiment of the present invention, firefighters receive AR guidance through mobile terminals (such as smart glasses), intuitively understand the location of the fire source, the direction of fire spread and the best fire extinguishing path, and improve fire extinguishing efficiency.

[0171] In summary, in response to the related problems existing in the prior art, the embodiments of the present invention provide an implementation method for improving the response speed of an initial fire and reducing the false alarm rate and missed alarm rate of a fire, thereby shortening the time of an early warning response. By acquiring image data information of a monitoring area and performing edge feature preprocessing, multimodal suspected fire image features are acquired. By combining real-time image acquisition technology with feature recognition technology of a neural network, the source of a fire can be discovered in a timely manner. Through multimodal suspected fire image features, a small fire flame can be determined in a timely manner, thereby improving the response speed of an initial fire. The multimodal suspected fire image features are further processed by a neural residual network structure to obtain a fire image recognition result. The suspected fire image is further recognized twice by a neural network to improve the processing accuracy of the image, thereby reducing the false alarm rate and missed alarm rate of a fire. Finally, the fire image recognition result is visualized based on a graphical user interface and a fire warning is issued according to the displayed result. The detection result can be warned in a timely manner, and the location of the fire can be determined in a timely manner through a visualized display, thereby shortening the time of an early warning response.

[0172] On the other hand, Figure 8As shown, an embodiment of the present invention provides a fire warning device 800, including: a first module 810, used to obtain image data information of a monitored area and perform edge feature preprocessing to obtain multimodal suspected fire image features; a second module 820, used to perform fire image recognition processing on the multimodal suspected fire image features based on a neural residual network structure to obtain a fire image recognition result; a third module 830, used to visualize the fire image recognition result based on a graphical user interface and issue a fire warning based on the displayed result.

[0173] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0174] On the other hand, Fig. 9 As shown, an embodiment of the present invention further provides an electronic device 900, which includes at least one processor 910 and at least one memory 920 for storing at least one program; taking one processor 910 and one memory 920 as an example.

[0175] The processor 910 and the memory 920 may be connected via a bus or in other ways.

[0176] The memory 920, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory 920 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 920 may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0177] The electronic device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] Specifically, Fig.10 The structure block diagram of a computer system of an electronic device for implementing an embodiment of the present invention is schematically shown.

[0179] It should be noted that Fig.10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0180] like Fig.10 As shown, the computer system 1000 includes a central processing unit 1001 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1002 (ROM) or the program loaded from the storage part 1008 to the random access memory 1003 (RAM). Various programs and data required for system operation are also stored in the random access memory 1003. The central processing unit 1001, the read-only memory 1002 and the random access memory 1003 are connected to each other through a bus 1004. The input / output interface 1005 (Input / Output interface, i.e., I / O interface) is also connected to the bus 1004.

[0181] The following components are connected to the input / output interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0182] In particular, according to an embodiment of the present invention, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1009, and / or installed from a removable medium 1011. When the computer program is executed by the central processor 1001, various functions defined in the system of the present invention are executed.

[0183] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0184] The contents of the method embodiments of the present invention are all applicable to the system embodiments. The functions specifically implemented by the system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method.

[0185] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the above method.

[0186] The contents of the method embodiments of the present invention are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0187] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0188] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0189] It should be noted that, although several modules of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0190] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present invention.

[0191] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0192] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0193] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0194] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by an instruction execution device, apparatus or device (such as a computer-based device, a device including a processor, or other device that can fetch instructions from an instruction execution device, apparatus or device and execute the instructions), or in conjunction with such instruction execution device, apparatus or device. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution device, apparatus or device, or in conjunction with such instruction execution device, apparatus or device.

[0195] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0196] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0197] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0198] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0199] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A fire early warning method, characterized in that: include: Obtain image data information of the monitored area and perform edge feature preprocessing to obtain multi-modal suspected fire image features; Performing fire image recognition processing on the multimodal suspected fire image features based on a neural residual network structure to obtain a fire image recognition result; The fire image recognition result is visualized and displayed based on a graphical user interface, and a fire warning is issued according to the displayed result.

2. The fire warning method according to claim 1, characterized in that: The method of acquiring image data information of the monitoring area and performing edge feature preprocessing to obtain multi-modal suspected fire image features includes: Acquire the image data information of the monitoring area and perform frame processing to obtain a plurality of frame image data information to be processed; Performing image data preprocessing on the plurality of frame image data information to be processed to obtain preprocessed image data information; Performing multimodal edge feature extraction processing on the preprocessed image data information to obtain multimodal fire image features; The multimodal fire image features are subjected to image recognition processing by a convolutional neural network to obtain the multimodal suspected fire image features.

3. The fire warning method according to claim 2, characterized in that: The acquiring of the image data information of the monitoring area and performing frame processing to obtain a plurality of frame image data information to be processed includes: Acquiring image data information of the monitoring area through a monitoring device; According to the image monitoring time in the image data information, the image data information of the monitoring area is processed by frame by frame through a computer vision platform to obtain a plurality of frame image data information to be processed.

4. The fire warning method according to claim 2, characterized in that: The performing multimodal edge feature extraction processing on the pre-processed image data information to obtain multimodal fire image features includes: Extracting flame color features from the preprocessed image data information by setting a preset color threshold to obtain flame color features of the fire image; Performing flame edge shape feature extraction processing on the pre-processed image data information by using an edge detection algorithm to obtain flame edge shape features of the fire image; Performing flame texture feature extraction processing on the pre-processed image data information by using a texture analysis algorithm to obtain flame texture features of the fire image; Performing flame motion trajectory feature extraction processing on the pre-processed image data information by optical flow tracking method to obtain flame motion features of the fire image; The multimodal fire image feature is obtained by combining the fire image flame color feature, the fire image flame edge shape feature, the fire image flame texture feature and the fire image flame motion feature.

5. The fire warning method according to claim 1, characterized in that: The neural residual network structure includes an input layer, a convolutional layer, a maximum pooling layer, a first residual block group, a second residual block group, a third residual block group, a fourth residual block group, a global average pooling layer, a fully connected layer and an output layer, wherein: The convolution layer includes an initial convolution layer and a maximum pooling layer; The first residual block group, the second residual block group, the third residual block group and the fourth residual block group each have a plurality of residual blocks, and the first residual block group, the second residual block group, the third residual block group and the fourth residual block are connected via residual connections.

6. The fire warning method according to claim 5, characterized in that: The method of performing fire image recognition processing on the multimodal suspected fire image features based on the neural residual network structure to obtain a fire image recognition result includes: Inputting the multimodal suspected fire image features into the neural residual network structure; Based on the input layer of the neural residual network structure, obtaining the multimodal suspected fire image features; Based on the convolution layer of the neural residual network structure, convolution processing is performed on the multimodal suspected fire image features to obtain a preliminary multimodal fire feature image; Based on the maximum pooling layer of the neural residual network structure, the preliminary multimodal fire feature image is subjected to spatial dimensionality reduction processing to obtain a multimodal fire feature image after dimensionality reduction; Based on the first residual block group, the second residual block group, the third residual block group and the fourth residual block group of the neural residual network structure, target feature extraction processing is sequentially performed on the multimodal fire feature image after dimensionality reduction to obtain a multimodal fire feature image; Based on the global average pooling layer of the neural residual network structure, channel compression processing is performed on the multimodal fire feature image to obtain a multimodal fire image feature vector; Based on the fully connected layer of the neural residual network structure, the multimodal fire image feature vector is identified and classified to obtain the fire image recognition result; Based on the output layer of the neural residual network structure, the fire image recognition result is output.

7. The fire warning method according to claim 1, characterized in that: The method of visually displaying the fire image recognition result based on a graphical user interface and issuing a fire warning according to the display result includes: Visually displaying the fire image recognition result based on a graphical user interface to obtain a two-dimensional visualization display result of the fire image; The two-dimensional visualization display result of the fire image is mapped and superimposed by a three-dimensional rendering algorithm to obtain a three-dimensional visualization display result of the fire image; A judgment is made based on the three-dimensional visualization display result of the fire image to determine the location of the fire source and issue a fire warning.

8. A fire warning device, characterized in that: include: The first module is used to obtain image data information of the monitoring area and perform edge feature preprocessing to obtain multi-modal suspected fire image features; The second module is used to perform fire image recognition processing on the multimodal suspected fire image features based on a neural residual network structure to obtain a fire image recognition result; The third module is used to visualize the fire image recognition results based on a graphical user interface and issue a fire warning according to the displayed results.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images