Pyrotechnic detection method, device, equipment, readable storage medium and program product

Abstract

The invention relates to a smoke and fire detection method, device and equipment, a computer readable storage medium and a program product. The method comprises the steps of obtaining a to-be-detected image; performing local binary processing on the to-be-detected image to obtain a local binary statistical graph; performing time-frequency transformation processing on the to-be-detected image to obtain a frequency spectrum information feature map; taking the to-be-detected image, the local binary statistical graph and the frequency spectrum information feature graph as input of a smoke and fire classification model, and outputting based on the smoke and fire classification model to obtain a smoke and fire classification result; if the smoke and fire classification result indicates that the smoke and fire object exists in the to-be-detected image, performing optical flow analysis based on the to-be-detected image and at least one historical image to obtain an optical flow analysis result; wherein the acquisition moment corresponding to the at least one historical image is before the acquisition moment of the to-be-detected image; and based on the smoke and fire classification result and the optical flow analysis result, determining whether a smoke and fire detection result exists in the to-be-detected image. By adopting the method, the accuracy of smoke and fire detection can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, computer-readable storage medium, and program product for detecting fireworks. Background Technology

[0002] Fires cause immense damage to human society, resulting in the loss of countless lives and property every year. Real-time fire monitoring remains challenging due to various factors. However, with the development of computer technology and the widespread adoption of image recognition technology, intelligent fire monitoring via video surveillance has become a crucial means of future fire early warning.

[0003] In the field of computer vision, smoke and fire detection is a crucial application in security monitoring, playing a vital role in fire prevention and ensuring public safety. Existing smoke and fire detection technologies follow two main approaches. One approach combines target detection algorithms with video classification algorithms. This method first uses target detection to locate the smoke and fire target, then uses video classification algorithms to analyze the video sequence at the target location and identify the smoke and fire. The main function of the video classification algorithm is to further reduce false alarms. While this approach offers high accuracy, it is challenging in terms of data collection and model training, resulting in slow processing speeds and high resource consumption. The other approach combines target detection algorithms with image recognition algorithms. This method performs secondary identification of the smoke and fire target to reduce false alarms. Although the entire algorithm chain operates quickly and has higher accuracy than single detection models, it still cannot eliminate some false alarms, such as those from reflective clothing, safety helmets, ground reflections, vehicle and streetlights, and gray metal. Therefore, improving the accuracy of smoke and fire detection algorithms and reducing false alarms has become a key research focus. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, equipment, computer-readable storage medium, and program product for detecting fireworks that can improve the accuracy of fireworks detection and reduce the false detection rate, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for detecting fireworks, including:

[0006] Acquire the image to be detected;

[0007] The image to be detected is subjected to local binary processing to obtain a local binary statistical map;

[0008] The image to be detected is subjected to time-frequency transformation processing to obtain a spectral information feature map;

[0009] The image to be detected, the local binary statistical map, and the spectral information feature map are used as inputs to the fireworks classification model, and the fireworks classification result is obtained based on the output of the fireworks classification model.

[0010] If the fireworks classification result indicates that there are fireworks objects in the image to be detected, optical flow analysis is performed based on the image to be detected and at least one historical image to obtain the optical flow analysis result; wherein, the acquisition time corresponding to the at least one historical image is before the acquisition time of the image to be detected;

[0011] Based on the fireworks classification results and the optical flow analysis results, the detection result of whether fireworks exist in the image to be detected is determined.

[0012] In one embodiment, the spectral information feature map includes frequency domain feature maps corresponding to different frequency domains;

[0013] The step of performing time-frequency transformation processing on the image to be detected to obtain a spectral information feature map includes:

[0014] The image to be detected is subjected to frequency domain transformation to obtain frequency domain transformation results in different frequency domains;

[0015] The frequency domain transformation results under different frequency domains are respectively subjected to inverse frequency domain transformation to obtain frequency domain feature maps corresponding to different frequency domains.

[0016] In one embodiment, the step of using the image to be detected, the local binary statistical map, and the spectral information feature map as input to the fireworks classification model, and outputting the fireworks classification result based on the fireworks classification model, includes:

[0017] The image to be detected, the local binary statistical map, and the spectral information feature map are used as inputs to the fireworks classification model. The fireworks classification model extracts features from the image to be detected to obtain the first feature.

[0018] The second feature is obtained by extracting features from the local binary statistical graph using the fireworks classification model.

[0019] The third feature is obtained by extracting features from the spectral information feature map using the fireworks classification model.

[0020] The fireworks classification model performs feature fusion on the first feature, the second feature, and the third feature to obtain the fused feature;

[0021] The fireworks classification model performs classification prediction based on the fused features and outputs the fireworks classification result.

[0022] In one embodiment, if the fireworks classification result indicates the presence of fireworks objects in the image to be detected, optical flow analysis is performed based on the image to be detected and at least one historical image to obtain optical flow analysis results, including:

[0023] If the fireworks classification result indicates that there are fireworks objects in the image to be detected, optical flow calculation is performed on each pixel in the image to be detected and the at least one historical image to obtain the optical flow field.

[0024] Image analysis is performed based on the optical flow field to determine the motion region in the image to be detected; wherein, the image analysis includes at least one of optical flow amplitude calculation, threshold segmentation, morphological processing, and connected component analysis;

[0025] The area ratio between the bounding rectangle of the motion region and the image to be detected is used as the result of optical flow analysis.

[0026] In one embodiment, acquiring the image to be detected includes:

[0027] Obtain the original image;

[0028] The original image is subjected to target recognition to obtain the target recognition result;

[0029] Based on the target location area indicated by the target recognition result, the original image is cropped to obtain the image to be detected.

[0030] In one embodiment, determining whether there is smoke or fire in the image to be detected based on the smoke classification result and the optical flow analysis result includes:

[0031] Based on the first confidence level of the fireworks classification result, the second confidence level of the target recognition result, and the optical flow analysis result, the detection result of whether fireworks exist in the image to be detected is determined.

[0032] In one embodiment, the training process of the fireworks classification model includes:

[0033] Obtain sample images and corresponding sample labels; wherein, the sample images include fireworks images and non-firework images, and the sample labels are used to indicate the fireworks classification reference results of the sample images; wherein, the fireworks images include flame images and smoke images, and the non-firework images refer to images containing other objects similar to fireworks objects;

[0034] The sample image, the corresponding local binary statistical map of the sample image, and the corresponding spectral information feature map of the sample image are used as inputs to the base classification model, and the fireworks classification prediction result is obtained based on the output of the base classification model.

[0035] The model loss is calculated based on the fireworks classification prediction results and the fireworks classification reference results.

[0036] The parameters of the base classification model are adjusted based on the model loss until the training termination condition is met, thus obtaining the fireworks classification model.

[0037] Secondly, this application also provides a smoke detection device, comprising:

[0038] The acquisition module is used to acquire the image to be detected;

[0039] The first processing module is used to perform local binary processing on the image to be detected to obtain a local binary statistical map;

[0040] The second processing module is used to perform time-frequency transformation processing on the image to be detected to obtain a spectral information feature map;

[0041] The classification module is used to take the image to be detected, the local binary statistical map, and the spectral information feature map as inputs to the fireworks classification model, and output the fireworks classification result based on the fireworks classification model.

[0042] An optical flow analysis module is used to perform optical flow analysis based on the image to be detected and at least one historical image if the fireworks classification result indicates that there are fireworks objects in the image to be detected, and to obtain optical flow analysis results; wherein the acquisition time corresponding to the at least one historical image is located before the acquisition time of the image to be detected;

[0043] The output module is used to determine the detection result of whether there is smoke or fire in the image to be detected based on the smoke classification result and the optical flow analysis result.

[0044] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0045] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0046] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0047] The aforementioned fireworks detection method, apparatus, computer equipment, computer-readable storage medium, and program product take the image to be detected, a local binary statistical map, and a spectral information feature map as inputs to a fireworks classification model. The fireworks classification model can capture static image features based on the image to be detected; simultaneously, it can capture local texture features of the image to be detected based on the local binary statistical map; and it can capture the spectral characteristics of the image to be detected based on the spectral information feature map. Finally, the fireworks classification model combines multiple features to output a fireworks classification result, improving the accuracy of fireworks classification. Furthermore, based on the fireworks classification result indicating the presence of fireworks in the image to be detected, optical flow analysis is performed on the image to be detected and at least one historical image. This allows for further judgment based on the optical flow analysis results, thereby further reducing static and dynamic false detections through two detection processes and improving the accuracy of fireworks detection.

[0048] Furthermore, by performing frequency domain transformation on the image to be detected, image information can be efficiently decomposed into different frequency domains such as low frequency, mid frequency, and high frequency. Corresponding frequency domain feature maps are then generated through inverse frequency domain transformation, enabling accurate separation and visualization of multi-scale frequency domain features. This provides more robust frequency domain feature support for fireworks detection, significantly improving the accuracy and anti-interference capability of target recognition in complex scenes. By fusing features to obtain fused features, the fused features can more comprehensively and accurately represent the image to be detected, thereby improving the accuracy of the fireworks classification model's classification prediction based on the fused features. Optical flow analysis is used to determine the moving regions in the image to be detected, and then the area ratio between the moving regions and the image to be detected is used to further detect whether fireworks exist in the image. This fully considers the spatiotemporal correlation of the dynamic features of fireworks, achieving fireworks detection in different dimensions. By combining the first confidence level of the fireworks classification results, the second confidence level of the target recognition results, and the optical flow analysis results, the detection result of whether fireworks exist in the image to be detected is obtained. This integrates multi-dimensional information, reducing the risk of misjudgment based on a single dimension and improving the accuracy of the detection results. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a diagram illustrating the application environment of a smoke detection method in one embodiment;

[0051] Figure 2This is a flowchart illustrating a smoke detection method in one embodiment;

[0052] Figure 3 This is a schematic diagram of the training process of a fireworks classification model in one embodiment;

[0053] Figure 4 This is a local binary statistical graph corresponding to multiple different categories of sample images involved in one embodiment;

[0054] Figure 5 This is a schematic diagram illustrating time-frequency transformation processing of a sample image in one embodiment;

[0055] Figure 6 This is a schematic diagram of the network structure of a fireworks classification model involved in one embodiment;

[0056] Figure 7 This is an example of optical flow prediction and motion region extraction in one embodiment;

[0057] Figure 8 This is a schematic diagram of a smoke detection device in one embodiment;

[0058] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0061] The smoke and fire detection method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0062] Terminal 102 can acquire images and upload the acquired images to server 104. Server 104 can be deployed with a smoke and fire classification model. After receiving the image uploaded by terminal 102, server 104 executes the smoke and fire detection method provided in this application embodiment to obtain the detection result of whether there is smoke and fire in the image, and returns the detection result to terminal 102.

[0063] Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection equipment. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0064] In one exemplary embodiment, such as Figure 2 As shown, a smoke detection method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 202 to 212. Wherein:

[0065] Step 202: Obtain the image to be detected.

[0066] The image to be detected is the image from which it is necessary to determine whether there are any objects related to smoke or fire; further, objects related to smoke or fire include flames, smoke, etc.

[0067] In some embodiments, the image to be detected may be an original image captured by an image acquisition device; the image acquisition device may be a camera, a camera, etc.

[0068] In other embodiments, to improve detection efficiency, the image to be detected can also be an image obtained by cropping the original image acquired by the image acquisition device; wherein, the cropped area can be manually marked by the tester or determined by target detection of the original image.

[0069] Step 204: Perform local binary processing on the image to be detected to obtain a local binary statistical map.

[0070] Local Binary Pattern (LBP) is a method for extracting local texture features of an image. Specifically, LBP uses the center pixel as a threshold to binarize the gray values ​​of neighboring pixels, generating binary codes to obtain the LBP value of the pixel.

[0071] Local Binary Histogram (LBPH) is a histogram that calculates the LBP values ​​of all pixels in an image, thereby reflecting the frequency of occurrence of different local texture patterns.

[0072] Understandably, since local binary processing compares the gray values ​​of the middle pixel with those of its neighbors, this process relies only on the relative differences between pixels. Therefore, local binary statistical maps are insensitive to global illumination and have strong robustness.

[0073] Step 206: Perform time-frequency transformation processing on the image to be detected to obtain a spectral information feature map.

[0074] Time-frequency transformation processing refers to the analysis method of converting an image from the spatial domain to the frequency domain, or from the frequency domain to the time domain; furthermore, time-frequency transformation processing can be implemented through Fourier transform, wavelet transform, and other methods.

[0075] A spectral information feature map refers to a feature representation formed by visualizing spectral information (such as through inverse Fourier transform), used to describe the frequency domain characteristics of an image.

[0076] In some embodiments, the image to be detected can be first transformed from the spatial domain to the frequency domain to separate the frequency domain information of the image to be detected in the frequency domain. Then, the separated frequency domain information is visualized to obtain a spectral information feature map.

[0077] It is understandable that the spectral information feature map can reflect the pixel distribution pattern of the image to be detected. For example, the high-frequency information in the spectral information feature map usually reflects the details of the image to be detected; the low-frequency information in the spectral information feature map usually reflects the main body of the image to be detected, such as flames and smoke.

[0078] Step 208: The image to be detected, the local binary statistical map, and the spectral information feature map are used as inputs to the fireworks classification model, and the fireworks classification result is obtained based on the output of the fireworks classification model.

[0079] Among them, the fireworks classification model is a pre-trained classification model; the fireworks classification results are used to indicate whether there are fireworks objects in the image to be detected and the specific category of the fireworks objects.

[0080] For example, the output fireworks classification result can be a specific logical value. For instance, an output of "1" indicates that fireworks objects exist in the image to be detected, and an output of "0" indicates that fireworks objects do not exist in the image to be detected. The output fireworks classification result can also include the probability of various types of objects to be detected in the image to be detected. These types of objects to be detected are predefined. For instance, the probability of "flames" is a, the probability of "smoke" is b, etc. The object with the highest probability is determined as the target object contained in the image to be detected.

[0081] Understandably, the image to be detected allows the fireworks classification model to focus on static image features, while the local binary statistical map allows the fireworks classification model to focus on the local texture features of the image to be detected; the spectral information feature map allows the fireworks classification model to focus on the spectral characteristics of the image to be detected. Through the mutual complementarity between various features, the fireworks classification model can perform fireworks detection on the image to be detected more comprehensively and accurately, thereby improving the accuracy of fireworks detection.

[0082] Step 210: If the fireworks classification result indicates that there are fireworks objects in the image to be detected, perform optical flow analysis based on the image to be detected and at least one historical image to obtain the optical flow analysis result; wherein, the acquisition time corresponding to at least one historical image is earlier than the acquisition time of the image to be detected.

[0083] It should be noted that the acquisition time of at least one historical image is consecutive to the acquisition time of the image to be examined.

[0084] Optical flow refers to a two-dimensional vector field that describes the motion trajectory and velocity of pixels over time in a series of frames. Optical flow analysis is a technique that captures the motion of objects and displacement information between adjacent frames by analyzing the motion trajectory and velocity of pixels in at least two consecutive images, and it reflects the trend of grayscale changes.

[0085] It should be noted that flames and smoke are dynamic targets that are constantly in motion, and their pixel intensity and spatial position change rapidly over time; while static fire and smoke objects (such as helmets, car lights, streetlights, etc.) lack motion characteristics in the time dimension; optical flow analysis can efficiently distinguish between these two types of targets.

[0086] The optical flow analysis results can include whether there is a moving region, and the area ratio of the moving region in the image to be detected. Furthermore, when the optical flow analysis results indicate the presence of a moving region, or when the optical flow analysis results indicate the presence of a moving region and the area ratio of the moving region is greater than a preset ratio threshold, it indicates that there is a high probability of the presence of a fire object.

[0087] Step 212: Based on the smoke classification results and optical flow analysis results, determine the detection results of whether smoke exists in the image to be detected.

[0088] In some embodiments, a first score can be determined based on the confidence level of the fireworks classification results, and a second score can be determined based on the area ratio of the moving region in the image to be detected in the optical flow analysis results. By weighted summing of the first score and the second score, if the summation result is greater than a preset score threshold, the detection result of fireworks in the image to be detected is determined; otherwise, the detection result of fireworks not existing in the image to be detected is determined.

[0089] Furthermore, a second score is determined based on the area ratio of the moving region in the image to be detected in the optical flow analysis results. This can be achieved by pre-setting multiple second scores and corresponding ratio intervals. Based on the area ratio of the moving region in the image to be detected in the optical flow analysis results, a target ratio interval is determined, and the second score corresponding to the target ratio interval is determined as the target second score. By weighted summing of the first score and the target second score, if the summation result is greater than a preset score threshold, the detection result of the presence of smoke in the image to be detected is determined; otherwise, the detection result of the absence of smoke in the image to be detected is determined.

[0090] It should be noted that the detection result can be a logical value. For example, when the summation result is greater than the result threshold, the output is "1", which means that there is smoke in the image to be detected; otherwise, the output is "0", which means that there is no smoke in the image to be detected. The detection result can also be a direct summation result, which is used by the inspector to judge whether there is smoke.

[0091] It is worth mentioning that single optical flow analysis has problems such as optical flow miscalculation caused by sudden changes in brightness and low accuracy of optical flow analysis due to small motion scale. Therefore, the embodiments of this application combine the smoke classification results and optical flow analysis results to determine whether there are smoke in the image to be detected, thereby improving the accuracy of smoke detection.

[0092] In the aforementioned fireworks detection method, the image to be detected, a local binary statistical map, and a spectral information feature map are used as inputs to the fireworks classification model. The model captures static image features from the image to be detected; simultaneously, it captures local texture features from the local binary statistical map; and it captures the spectral characteristics of the image from the spectral information feature map. Finally, the model combines multiple features to output the fireworks classification result, improving the accuracy of fireworks classification. Furthermore, based on the fireworks classification result indicating the presence of fireworks in the image to be detected, optical flow analysis is used for further judgment. This two-stage detection process further reduces both static and dynamic false detections, improving the accuracy of fireworks detection.

[0093] In an exemplary embodiment, the spectral information feature map includes frequency domain feature maps corresponding to different frequency domains; step 206 includes steps 302 to 304. Wherein:

[0094] Step 302: Perform frequency domain transformation on the image to be detected to obtain frequency domain transformation results under different frequency domains.

[0095] Frequency domain transformation can be achieved through Discrete Cosine Transform (DCT), Fourier Transform, and Wavelet Transform.

[0096] For example, after performing a discrete cosine transform on the image to be detected, the low-frequency information of the image to be detected will be concentrated in the upper left corner, i.e., the 0-1 / 16 part, the mid-frequency information will be concentrated in the 1 / 16-1 / 8 part, and the remaining part is the high-frequency information of the image, which will be concentrated in the lower right corner. Then, by constructing three filters to extract the low, mid and high-frequency information, the frequency domain transformation results under different frequency domains are obtained.

[0097] Step 304: Perform inverse frequency domain transformation on the frequency domain transformation results under different frequency domains to obtain frequency domain feature maps corresponding to different frequency domains.

[0098] In this context, the inverse frequency domain transform is the inverse of the frequency domain transform. For example, if the frequency domain transform adopts the discrete cosine transform, then the inverse frequency domain transform is the inverse discrete cosine transform; if the frequency domain transform adopts the wavelet transform, then the inverse frequency domain transform is the inverse wavelet transform.

[0099] It is understandable that the frequency domain feature maps corresponding to different frequency domains reflect the frequency domain characteristics under different frequency domains.

[0100] In this embodiment, by performing a frequency domain transformation operation on the image to be detected, the image information can be efficiently decomposed into different frequency domains such as low frequency, mid frequency and high frequency, and the corresponding frequency domain feature map can be generated by inverse frequency domain transformation. This enables accurate separation and visualization of multi-scale frequency domain features, thereby providing more robust frequency domain feature support for fireworks detection and significantly improving the accuracy and anti-interference ability of target recognition in complex scenarios.

[0101] In an exemplary embodiment, step 208 includes steps 402 to 410. Wherein:

[0102] Step 402: The image to be detected, the local binary statistical map, and the spectral information feature map are used as inputs to the fireworks classification model. The fireworks classification model extracts features from the image to be detected to obtain the first feature.

[0103] Feature extraction refers to the process of converting high-dimensional data into low-dimensional, structured feature vectors.

[0104] The first feature, or image feature of the image to be detected, is extracted from the image to be detected.

[0105] Step 404: The fireworks classification model extracts features from the local binary statistical graph to obtain the second feature.

[0106] Similar to step 402, the difference is that since the local binary statistical map reflects the local texture features of the image to be detected, the second feature obtained based on the local binary statistical map is also the local texture feature of the image to be detected.

[0107] Step 406: The fireworks classification model extracts features from the spectral information feature map to obtain the third feature.

[0108] Similar to step 402, the difference is that, since the spectral information feature map reflects the pixel distribution pattern of the image to be detected in different frequency domains, the third feature obtained based on the spectral information feature map is also the frequency domain feature of the image to be detected.

[0109] Step 408: The fireworks classification model performs feature fusion on the first feature, the second feature, and the third feature to obtain the fused feature.

[0110] In some embodiments, feature fusion may be a feature concatenation of the first feature, the second feature, and the third feature; or it may be attention fusion of the first feature, the second feature, and the third feature, etc.

[0111] Step 410: The fireworks classification model performs classification prediction based on the fusion features and outputs the fireworks classification result.

[0112] In this embodiment, compared to a single first feature, the fused feature more comprehensively and accurately represents the image to be detected. Therefore, the fireworks classification model can improve the accuracy of classification prediction by performing classification prediction based on the fused feature.

[0113] In an exemplary embodiment, step 208 includes steps 502 to 506. Wherein:

[0114] Step 502: If the fireworks classification result indicates that there are fireworks objects in the image to be detected, perform optical flow calculation on each pixel in the image to be detected and at least one historical image to obtain the optical flow field.

[0115] Among them, optical flow calculation is to estimate the motion speed and direction of pixels in at least two adjacent images over time through an algorithm; further, the motion speed and direction of each pixel over time are combined into a two-dimensional vector field, which is the optical flow field.

[0116] Step 504: Perform image analysis based on the optical flow field to determine the motion region in the image to be detected; wherein, the image analysis includes at least one of optical flow amplitude calculation, threshold segmentation, morphological processing, and connected component analysis.

[0117] In this context, optical flow amplitude represents the motion intensity of a pixel. The larger the amplitude, the more intense the pixel motion. By calculating the optical flow amplitude and combining it with threshold segmentation, static pixels with optical flow amplitude less than the threshold and dynamic pixels with optical flow amplitude not less than the threshold can be distinguished. Dynamic pixels can be used to initially determine the original motion region. Subsequently, morphological analysis and connected component analysis are used to correct the determined original motion region to obtain the motion region.

[0118] It is understandable that, since fireworks are in constant motion, the area of ​​motion can, to some extent, represent the possible changes in fireworks.

[0119] Step 506: The area ratio between the bounding rectangle of the motion region and the image to be detected is used as the optical flow analysis result.

[0120] In some embodiments, when the area ratio is greater than a certain ratio, it indicates that there is a high probability of smoke or fire objects in the image to be detected; when the area ratio is not greater than a certain ratio, it indicates that it may be a misjudgment caused by environmental changes.

[0121] It should be noted that calculating the area of ​​the bounding rectangle of the moving region is a simplified calculation method; of course, it is also possible to directly calculate the area ratio between the area of ​​the moving region and the area of ​​the image to be detected.

[0122] In this embodiment, the area ratio between the motion region and the image to be detected is used to further detect whether there are fireworks in the image to be detected, so as to combine the fireworks classification results to determine the final detection result of the fireworks.

[0123] In an exemplary embodiment, step 202 includes steps 602 to 606. Wherein:

[0124] Step 602: Obtain the original image.

[0125] The original image, which is the image captured by the image acquisition device, not only contains the content that needs to be identified for smoke and fire, but also contains a lot of irrelevant content, such as roads and walls.

[0126] Step 604: Perform target recognition on the original image to obtain the target recognition result.

[0127] Among them, target recognition refers to identifying targets that may be fireworks objects in the original image; such as flames, smoke, car lights, streetlights, safety helmets, etc.; the target recognition results are used to indicate the image area where the target is located.

[0128] Understandably, if the target recognition result indicates that there is no area corresponding to a target that could be a firework object, the firework detection can be terminated.

[0129] In some embodiments, target recognition can be achieved through a target recognition model; specifically, the original image is used as input to the target recognition model, and the target recognition model outputs the target recognition result.

[0130] Step 606: Based on the target location area indicated by the target recognition result, crop the original image to obtain the image to be detected.

[0131] The image to be detected is an image containing a target that may be a firework object.

[0132] In this embodiment of the application, by performing target recognition and image cropping on the original image, the amount of image data that needs to be detected for fireworks is effectively reduced, thereby improving the efficiency of fireworks detection.

[0133] In an exemplary embodiment, based on steps 602 to 606, step 212 includes:

[0134] Based on the first confidence level of the fireworks classification results, the second confidence level of the target recognition results, and the optical flow analysis results, the detection results of whether fireworks exist in the image to be detected are determined.

[0135] In some embodiments, the first score may be determined based on the first confidence level of the fireworks classification result, the second score may be determined based on the area ratio of the moving region in the image to be detected in the optical flow analysis result, and the third score may be determined based on the second confidence level of the target recognition result. The first score, the second score and the third score may be weighted and summed. If the summation result is greater than a preset score threshold, the detection result of fireworks in the image to be detected is determined; otherwise, the detection result of fireworks not existing in the image to be detected is determined.

[0136] In one exemplary embodiment, such as Figure 3 As shown, the training process of the fireworks classification model includes steps 702 to 708, wherein:

[0137] Step 702: Obtain the sample image and the corresponding sample label.

[0138] The sample images include both smoke and non-smoke images, and the sample labels are used to indicate the smoke classification reference results of the sample images. The smoke images include flame images and smoke images, and the non-smoke images refer to images that contain other objects similar to the smoke objects.

[0139] Step 704: The sample image, the corresponding local binary statistical map of the sample image, and the corresponding spectral information feature map of the sample image are used as inputs to the base classification model, and the fireworks classification prediction result is obtained based on the output of the base classification model.

[0140] The process of obtaining the local binary statistical map corresponding to the sample image and the spectral information feature map corresponding to the sample image can be referred to the relevant description in the foregoing embodiments, and will not be repeated here.

[0141] Step 706: Calculate the model loss based on the fireworks classification prediction results and the fireworks classification reference results.

[0142] The model loss can be calculated using a classification loss function, such as cross-entropy loss.

[0143] Step 708: Adjust the parameters of the base classification model based on the model loss until the training termination condition is met, and obtain the fireworks classification model.

[0144] The training termination condition can be either the number of training iterations reaching a threshold or the model loss being less than a threshold; no specific restrictions are imposed here.

[0145] In this embodiment of the application, a fireworks classification model is obtained by training the base classification model, so that the fireworks classification model has the ability to classify fireworks and can be used for specific fireworks detection tasks.

[0146] The following will describe the specific implementation examples.

[0147] Step 1: Use a trained object detection model, such as the YOLO object detection model or other detection models, to perform object detection on the original image. This is used to identify possible fire and smoke objects in the original image, such as smoke, flames, yellow safety helmets, red shirts, streetlights, car lights, ground reflections, etc. After that, the original image is cropped according to the object recognition results to obtain sample images. Sample labels are added to each sample image, such as flames (category 0), smoke (category 1), car lights / streetlights (category 2), safety helmets (category 3), and others (category 4).

[0148] Step two involves extracting LBP features from the sample image to obtain an LBP feature image. This LBP feature image is then divided into m local blocks, and a histogram is extracted from each block. These histograms are then concatenated to form a statistical histogram of the LBP features. This LBP statistical histogram is then subjected to further histogram analysis, with feature values ​​ranging from 1 to 11. A bar chart is then plotted to obtain a local binary statistical graph, as shown below. Figure 4 As shown, Figure 4 Local binary statistical plots corresponding to sample images of multiple different categories are shown.

[0149] Step 3: Perform Discrete Cosine Transform (DCT) on the sample image. DCT effectively transforms image data from the spatial domain to the frequency domain, separating high-frequency information (usually details) from low-frequency information (the main part of the image). After DCT, low-frequency information is concentrated in the upper left corner, while high-frequency information is concentrated in the lower right corner. Low-frequency information is concentrated in the 0-1 / 16 range, mid-frequency information in the 1 / 16-1 / 8 range, and the remaining portion is high-frequency information. Then, three filters are constructed to extract low, mid, and high-frequency information. The obtained low, mid, and high-frequency information is then subjected to inverse DCT to obtain the low, mid, and high-frequency domain feature map of the image, i.e., the spectral information feature map, as shown below. Figure 5 As shown, Figure 5 The process of performing time-frequency transformation on the sample image is shown.

[0150] Step four: Build a fireworks classification model based on a three-input convolutional neural network (CNN), such as... Figure 6 As shown, Figure 6 The diagram illustrates the network structure of a fireworks classification model. The first input is a sample image, the second input is a local binary statistical map of the sample image, and the third input is a spectral information feature map of the sample image. For example, the sample image input is 1*3*224*224, the local binary statistical map input is 1*1*224*224, and the spectral information feature map of the sample image includes feature maps in the low, mid, and high frequency domains, with 3 channels, so the input is also 1*3*224*224. Different branches extract features through convolution, then merge the features, extract features again after merging, and output the classification result. Training the fireworks classification model with the processed classification images can effectively filter out false detections of static fireworks, improving the accuracy of fireworks detection.

[0151] Step 5: After the smoke and fire classification model detects the presence of smoke or flames, it saves the classification results and the current frame. The detected smoke or flame regions from the previous and current frames are then input into the RAFT optical flow estimation network to obtain the optical flow field. Image operations such as optical flow amplitude calculation, threshold segmentation, and morphological processing are performed on the optical flow field. The bounding rectangle of the moving region is obtained through connected component analysis. The area ratio of the bounding rectangle of the moving region to the entire ROI region is calculated, serving as one of the bases for determining false smoke and fire detections, i.e., the optical flow analysis result. Figure 7 As shown, Figure 7 An example of optical flow prediction and motion region extraction is shown.

[0152] Step six: Based on the first confidence score of the fireworks classification result, the second confidence score of the target recognition result, and the optical flow analysis result, determine whether there are fireworks in the image to be detected. Specifically, the comprehensive score Z for fireworks discrimination can be calculated using the following formula:

[0153] ;

[0154] Where w1, w2, and w3 are weight coefficients, conf cls The first confidence level for the fireworks classification results is conf det The area_ratio represents the second confidence level of the target identification result. LK The score is determined based on the ratio of the area of ​​the bounding rectangle of the motion region to the area of ​​the entire sample image region.

[0155] Furthermore, area_ratio LK It can be calculated using the following formula:

[0156]

[0157] Where area_ratio represents the ratio of the area of ​​the bounding rectangle of the motion region to the area of ​​the entire sample image region, and thresh represents a preset ratio threshold. When area_ratio is greater than the thresh threshold, area_ratio... LK Set to 1; otherwise, set to 0.

[0158] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0159] Based on the same inventive concept, this application also provides a fireworks detection device for implementing the fireworks detection method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of the one or more fireworks detection device embodiments provided below can be found in the limitations of the fireworks detection method described above, and will not be repeated here.

[0160] In one exemplary embodiment, such as Figure 8 As shown, a smoke detection device is provided, comprising: an acquisition module, a first processing module, a second processing module, a classification model, an optical flow analysis module, and an output module, wherein:

[0161] The acquisition module is used to acquire the image to be detected.

[0162] The first processing module is used to perform local binary processing on the image to be detected to obtain a local binary statistical map.

[0163] The second processing module is used to perform time-frequency transformation processing on the image to be detected to obtain a spectral information feature map.

[0164] The classification module takes the image to be detected, the local binary statistical map, and the spectral information feature map as input to the fireworks classification model, and outputs the fireworks classification result based on the fireworks classification model.

[0165] The optical flow analysis module is used to perform optical flow analysis based on the image to be detected and at least one historical image if the smoke classification result indicates that there is a smoke object in the image to be detected, and to obtain the optical flow analysis result; wherein, the acquisition time of at least one historical image is earlier than the acquisition time of the image to be detected.

[0166] The output module is used to determine the detection result of whether there is smoke or fire in the image to be detected based on the smoke classification results and optical flow analysis results.

[0167] In some embodiments, the spectral information feature map includes frequency domain feature maps corresponding to different frequency domains; the first processing module is specifically used to perform frequency domain transformation on the image to be detected to obtain frequency domain transformation results under different frequency domains; and to perform inverse frequency domain transformation on the frequency domain transformation results under different frequency domains to obtain frequency domain feature maps corresponding to different frequency domains.

[0168] In some embodiments, the classification module includes a feature extraction unit, used to take the image to be detected, a local binary statistical map, and a spectral information feature map as input to the fireworks classification model, and extract features from the image to be detected by the fireworks classification model to obtain a first feature; extract features from the local binary statistical map by the fireworks classification model to obtain a second feature; extract features from the spectral information feature map by the fireworks classification model to obtain a third feature; a feature fusion unit, used to fuse the first feature, the second feature, and the third feature by the fireworks classification model to obtain a fused feature; and a prediction unit, used to perform classification prediction by the fireworks classification model based on the fused feature, and output the fireworks classification result.

[0169] In some embodiments, the optical flow analysis module is specifically used to: if the fireworks classification result indicates that there are fireworks objects in the image to be detected, perform optical flow calculation on each pixel in the image to be detected and at least one historical image to obtain an optical flow field; perform image analysis based on the optical flow field to determine the motion region in the image to be detected; wherein, the image analysis includes at least one of optical flow amplitude calculation, threshold segmentation, morphological processing, and connected component analysis; and use the area ratio between the bounding rectangle of the motion region and the image to be detected as the optical flow analysis result.

[0170] In some embodiments, the acquisition module is specifically used to acquire the original image; perform target recognition on the original image to obtain the target recognition result; and crop the original image based on the target location area indicated by the target recognition result to obtain the image to be detected.

[0171] In some embodiments, the output module is used to determine the detection result of whether there is smoke in the image to be detected based on the first confidence level of the smoke classification result, the second confidence level of the target recognition result, and the optical flow analysis result.

[0172] In some embodiments, the fireworks detection device further includes a training module for acquiring sample images and corresponding sample labels; wherein the sample images include fireworks images and non-smoke images, and the sample labels are used to indicate the fireworks classification reference results of the sample images; wherein the fireworks images include flame images and smoke images, and the non-smoke images refer to images containing other objects similar to fireworks objects; the sample images, the corresponding local binary statistical graphs of the sample images, and the corresponding spectral information feature maps of the sample images are used as inputs to the base classification model, and the fireworks classification prediction results are obtained based on the output of the base classification model; the model loss is calculated based on the fireworks classification prediction results and the fireworks classification reference results; the parameters of the base classification model are adjusted based on the model loss until the training termination condition is met, and the fireworks classification model is obtained.

[0173] Each module in the aforementioned fireworks detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0174] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores smoke and fire detection data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a smoke and fire detection method.

[0175] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0176] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0177] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0178] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0179] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0180] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0181] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0182] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting smoke and fire, characterized in that, The method includes: Acquire the image to be detected; The image to be detected is subjected to local binary processing to obtain a local binary statistical map; The image to be detected is subjected to time-frequency transformation processing to obtain a spectral information feature map; The image to be detected, the local binary statistical map, and the spectral information feature map are used as inputs to the fireworks classification model, and the fireworks classification result is obtained based on the output of the fireworks classification model. If the fireworks classification result indicates that there are fireworks objects in the image to be detected, optical flow analysis is performed based on the image to be detected and at least one historical image to obtain the optical flow analysis result; wherein, the acquisition time corresponding to the at least one historical image is before the acquisition time of the image to be detected; Based on the fireworks classification results and the optical flow analysis results, the detection result of whether fireworks exist in the image to be detected is determined.

2. The method according to claim 1, characterized in that, The spectral information feature map includes frequency domain feature maps corresponding to different frequency domains; The step of performing time-frequency transformation processing on the image to be detected to obtain a spectral information feature map includes: The image to be detected is subjected to frequency domain transformation to obtain frequency domain transformation results in different frequency domains; The frequency domain transformation results under different frequency domains are respectively subjected to inverse frequency domain transformation to obtain frequency domain feature maps corresponding to different frequency domains.

3. The method according to claim 1, characterized in that, The process of using the image to be detected, the local binary statistical map, and the spectral information feature map as inputs to the fireworks classification model, and outputting the fireworks classification result based on the fireworks classification model, includes: The image to be detected, the local binary statistical map, and the spectral information feature map are used as inputs to the fireworks classification model. The fireworks classification model extracts features from the image to be detected to obtain the first feature. The second feature is obtained by extracting features from the local binary statistical graph using the fireworks classification model. The third feature is obtained by extracting features from the spectral information feature map using the fireworks classification model. The fireworks classification model performs feature fusion on the first feature, the second feature, and the third feature to obtain the fused feature; The fireworks classification model performs classification prediction based on the fused features and outputs the fireworks classification result.

4. The method according to claim 1, characterized in that, If the fireworks classification result indicates the presence of fireworks in the image to be detected, optical flow analysis is performed based on the image to be detected and at least one historical image to obtain optical flow analysis results, including: If the fireworks classification result indicates that there are fireworks objects in the image to be detected, optical flow calculation is performed on each pixel in the image to be detected and the at least one historical image to obtain the optical flow field. Image analysis is performed based on the optical flow field to determine the motion region in the image to be detected; wherein, the image analysis includes at least one of optical flow amplitude calculation, threshold segmentation, morphological processing, and connected component analysis; The area ratio between the bounding rectangle of the motion region and the image to be detected is used as the result of optical flow analysis.

5. The method according to claim 1, characterized in that, The acquisition of the image to be detected includes: Obtain the original image; The original image is subjected to target recognition to obtain the target recognition result; Based on the target location area indicated by the target recognition result, the original image is cropped to obtain the image to be detected.

6. The method according to claim 5, characterized in that, The step of determining whether there is smoke or fire in the image to be detected based on the smoke classification results and the optical flow analysis results includes: Based on the first confidence level of the fireworks classification result, the second confidence level of the target recognition result, and the optical flow analysis result, the detection result of whether fireworks exist in the image to be detected is determined.

7. The method according to any one of claims 1-6, characterized in that, The training process of the fireworks classification model includes: Obtain sample images and corresponding sample labels; wherein, the sample images include fireworks images and non-firework images, and the sample labels are used to indicate the fireworks classification reference results of the sample images; wherein, the fireworks images include flame images and smoke images, and the non-firework images refer to images containing other objects similar to fireworks objects; The sample image, the corresponding local binary statistical map of the sample image, and the corresponding spectral information feature map of the sample image are used as inputs to the base classification model, and the fireworks classification prediction result is obtained based on the output of the base classification model. The model loss is calculated based on the fireworks classification prediction results and the fireworks classification reference results. The parameters of the base classification model are adjusted based on the model loss until the training termination condition is met, thus obtaining the fireworks classification model.

8. A smoke and fire detection device, characterized in that, The device includes: The acquisition module is used to acquire the image to be detected; The first processing module is used to perform local binary processing on the image to be detected to obtain a local binary statistical map; The second processing module is used to perform time-frequency transformation processing on the image to be detected to obtain a spectral information feature map; The classification module is used to take the image to be detected, the local binary statistical map, and the spectral information feature map as inputs to the fireworks classification model, and output the fireworks classification result based on the fireworks classification model. An optical flow analysis module is used to perform optical flow analysis based on the image to be detected and at least one historical image if the fireworks classification result indicates that there are fireworks objects in the image to be detected, and to obtain optical flow analysis results; wherein the acquisition time corresponding to the at least one historical image is located before the acquisition time of the image to be detected; The output module is used to determine the detection result of whether there is smoke or fire in the image to be detected based on the smoke classification result and the optical flow analysis result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images