Intelligent fire identification and early warning method and system based on deep learning

Abstract

The invention discloses an intelligent fire identification and early warning method and system based on deep learning, and the method carries out the Gaussian noise reduction and quality enhancement processing before the flame contour is extracted, and can achieve the purposes of removing image noise and improving image quality. Meanwhile, when the contour is extracted, a foreground image of a target area and a scene contour image without a fire disaster are fused, so that interference of non-flame factors, especially background interference, can be removed as far as possible while flame main body edge features are reserved; therefore, the image noise can be further reduced; and finally, performing feature extraction on the extracted fire contour image, and inputting the extracted features into a fire identification model, thereby realizing fire early warning. Therefore, when the method is used for fire identification, the noise in the image is removed to the greatest extent, so that the accuracy of subsequent fire identification can be improved, and the method is very suitable for large-scale application and popularization.

Description

Technical Field

[0001] The present invention belongs to the technical field of fire warning, and particularly relates to an intelligent fire recognition and warning method and system based on deep learning. Background Art

[0002] Fire has become a major hidden danger threatening people's lives and property. Whether it is a residential scene, a flammable and explosive scene, or a warehouse scene, once a fire occurs, it will cause huge irreparable losses. Therefore, the detection and recognition of open flames and smoke in fire accidents are particularly important. Taking the warehouse scene as an example, especially in the warehouse area where flammable materials are stacked and stored, it is necessary to detect and alarm the hidden dangers of open flames and smoke in various places in the warehouse to reduce the occurrence of accidents.

[0003] Traditional fire detectors usually use smoke sensors, light sensors, and temperature sensors to detect flames, smoke, and temperature, etc. However, traditional detectors need to place the sensors near the fire and can only sense the fire and give an alarm when the fire reaches a certain level. At the same time, in the case of harsh environmental conditions and light interference, it is also prone to problems such as missed detection and false alarms. In this way, it will make it more difficult to achieve early fire warning in complex scenarios. With the rapid development of video image processing technology and deep learning technology, image-based fire detection technology based on deep learning (such as fire recognition technology based on CNN network, etc.) has been widely used and achieved good detection results.

[0004] However, the existing image-based fire detection technology based on deep learning has the following deficiencies: Since the quality of the image largely determines the recognition rate of the flame target, and the diversity of fire types and the complexity of scenes in practical applications will lead to more noise in the video image, and currently the video image processing flame recognition method does not consider image noise, so the recognition accuracy will be reduced. Therefore, how to provide a fire recognition and warning method with high recognition accuracy has become an urgent problem to be solved. Summary of the Invention

[0005] The purpose of the present invention is to provide an intelligent fire recognition and warning method and system based on deep learning to solve the problem that there are often more noises in the fire monitoring images in the prior art, resulting in low fire recognition accuracy.

[0006] To achieve the above purpose, the present invention adopts the following technical solutions:

[0007] In the first aspect, an intelligent fire recognition and warning method based on deep learning is provided, including:

[0008] Obtain the real-time fire monitoring image of the target area and the calibrated monitoring image when there is no fire;

[0009] Perform Gaussian noise reduction processing and quality enhancement processing on the real-time fire monitoring image in sequence to obtain an enhanced fire monitoring image;

[0010] Extract the flame area image in the enhanced fire monitoring image and the foreground image in the real-time fire monitoring image, and perform fusion processing on the foreground image and the flame area image to generate a fused flame area image;

[0011] Perform edge extraction on the fused flame area image to obtain an initial fire contour image;

[0012] Generate a fire-free scene contour image by using the calibrated monitoring image, and generate a fire contour image based on the initial fire contour image and the fire-free scene contour image;

[0013] Perform feature extraction processing on the fire contour image to obtain fire feature data;

[0014] Input the fire feature data into a fire recognition model to obtain a fire recognition result for the target area, and when the fire recognition result indicates a fire, send a fire alarm message to the duty end.

[0015] Based on the above disclosed content, the present invention first obtains the real-time fire monitoring image of the target area and the calibrated monitoring image when there is no fire; then, perform Gaussian noise reduction processing and quality enhancement processing on the real-time fire monitoring image in sequence to obtain an enhanced fire monitoring image. Among them, Gaussian noise reduction can convert the image from the spatial domain to the frequency domain, so that the high-frequency part in the frequency domain image can be filtered out. Therefore, it can remove or weaken the image noise and at the same time retain the edge details; and quality enhancement can improve the contrast of the image, thereby reducing the interference of the background on the recognition of the fire area and making it more convenient for subsequent recognition of the flame; thus, through the foregoing operations, the noise reduction function before contour extraction can be achieved.

[0016] Next, the present invention extracts the flame region image in the enhanced fire monitoring image and the foreground image in the real-time fire monitoring image, and fuses the two to obtain the fused flame region image. Then, edge extraction is performed on the fused flame region image to obtain the initial fire contour image. Based on this, by fusing the flame region image and the foreground image, the background interference in the extracted contour image can be further reduced. At the same time, the present invention also extracts the fire-free scene contour image based on the calibrated monitoring image, and then fuses the two to obtain the fire contour image. Based on this, the interference of the background and noise in the image can be further removed. After obtaining the fire contour image, feature extraction processing can be performed on the fire contour image to obtain fire feature data. Finally, the fire feature data is input into the fire recognition model to obtain the fire recognition result of the target area, so as to give an alarm prompt in time when the fire recognition result indicates a fire.

[0017] Through the above design, before extracting the flame contour, the present invention performs Gaussian noise reduction and quality enhancement processing, which can achieve the purpose of removing image noise and improving image quality. At the same time, when extracting the contour, the foreground image of the target area and the scene contour image without fire are fused. In this way, while retaining the edge features of the flame main body, the interference of non-flame factors, especially background interference, can be removed as much as possible. Based on this, the image noise can be further reduced. Finally, feature extraction is performed on the extracted fire contour image, and the extracted features are input into the fire recognition model, then fire early warning can be realized. Therefore, when the present invention performs fire recognition, the noise in the image is removed to the greatest extent, so the accuracy of subsequent fire recognition can be improved, and thus it is very suitable for large-scale application and promotion.

[0018] In a possible design, the real-time fire monitoring image is sequentially subjected to Gaussian noise reduction processing and quality enhancement processing to obtain an enhanced fire monitoring image, including:

[0019] Perform fast Fourier transform processing on the real-time fire monitoring image to obtain a Fourier transform signal;

[0020] Input the Fourier transform signal into a Gaussian low-pass filter for filtering processing to obtain a filtered signal;

[0021] Perform inverse fast Fourier transform processing on the filtered signal to obtain a noise-reduced fire monitoring image;

[0022] Convert the noise-reduced fire monitoring image into an HSI image, and divide the HSI image into several image blocks;

[0023] Perform quality enhancement processing on each image block to obtain each quality-enhanced image block, and use each quality-enhanced image block to generate an initial enhanced fire monitoring image;

[0024] Convert the initial enhanced fire monitoring image into an RGB image, so as to obtain the enhanced fire monitoring image after conversion.

[0025] In a possible design, perform quality enhancement processing on each image block to obtain each quality-enhanced image block, including:

[0026] Separate the luminance component image, saturation component image, and hue component image from each image block;

[0027] Perform fast Fourier transform processing on the luminance component image corresponding to each image block to obtain the frequency domain signal corresponding to each image block, and perform mean filtering processing on each frequency domain signal to obtain each filtered frequency domain signal;

[0028] Use a Gaussian high-pass filter to perform quality enhancement processing on each filtered frequency domain signal to obtain each enhanced frequency domain signal;

[0029] Perform inverse fast Fourier transform processing on each enhanced frequency domain signal to obtain each enhanced luminance component image;

[0030] Fuse each enhanced luminance component image with the saturation component image and hue component image in the corresponding image block, so as to obtain each quality-enhanced image block after the fusion processing.

[0031] In a possible design, the enhanced fire monitoring image is an RGB image, wherein, extracting the flame area image in the enhanced fire monitoring image includes:

[0032] Convert the enhanced fire monitoring image to the YCbCr color space to obtain a YCbCr image;

[0033] For any pixel point in the enhanced fire monitoring image, determine the Y component value and Cb component value of the any pixel point from the YCbCr image;

[0034] Judge whether the Y component value is greater than the Cb component value;

[0035] If so, keep the pixel value of the any pixel point in the enhanced fire monitoring image, otherwise, set the pixel value of the any pixel point in the enhanced fire monitoring image to 0, and after polling all the pixel points in the enhanced fire monitoring image, obtain the initial fire area image;

[0036] Calculate the mean value of the red component and the mean value of the luminance of the enhanced fire monitoring image;

[0037] For the k-th pixel point in the initial fire area image, determine the Y component value and the Cr component value corresponding to the k-th pixel point from the YCbCr image;

[0038] Judge whether the Y component value corresponding to the k-th pixel point is greater than the average brightness, and whether the Cr component value of the k-th pixel point is greater than the average red component;

[0039] If so, keep the pixel value of the k-th pixel point in the initial fire area image, otherwise, set the pixel value of the k-th pixel point in the initial fire area image to 0;

[0040] Increment k by 1, and re-determine the Y component value and the Cr component value corresponding to the k-th pixel point from the YCbCr image until k is equal to K, where the initial value of k is 1 and K is the total number of pixel points in the initial fire area image, to obtain the fire area image.

[0041] In a possible design, fusing the foreground image and the flame area image to generate a fused flame area image includes:

[0042] Perform median filtering on the flame area image to obtain a filtered flame area image;

[0043] Perform image AND operation on the foreground image and the filtered flame area image to obtain the fused flame area image after the image AND operation;

[0044] Correspondingly, generating a fire contour image based on the initial fire contour image and the non-fire scene contour image includes:

[0045] Perform image AND operation on the initial fire contour image and the non-fire scene contour image to obtain the fire contour image after the image AND operation.

[0046] In a possible design, performing feature extraction on the fire contour image to obtain fire feature data includes:

[0047] Perform color feature extraction on the fire contour image to obtain fire color features;

[0048] Extract the shape features of the fire points in the fire contour image to obtain fire point shape features;

[0049] Perform texture feature extraction on the fire contour image to obtain fire texture features;

[0050] Determine the flame flicker frequency in the fire contour image to obtain flame flicker frequency features;

[0051] Using the fire color feature, the fire point shape feature, the fire texture feature, and the flame flicker frequency feature, the fire feature data is composed.

[0052] In a possible design, color feature extraction processing is performed on the fire contour image to obtain the fire color feature, including:

[0053] The following formula (1) is used to extract the first color feature, the second color feature, and the third color feature;

[0054]

[0055] In the above formula (1), δ1, δ2, δ3 represent the first color feature, the second color feature, and the third color feature in sequence, and f m represents the pixel value of the m-th pixel point in the fire contour image, and M represents the total number of pixel points in the fire contour image;

[0056] Using the first color feature, the second color feature, and the third color feature, the fire color feature is composed;

[0057] Correspondingly, the shape feature of the fire point in the fire contour image is extracted to obtain the fire point shape feature, which includes:

[0058] Calculate the primitive area and perimeter of the fire contour image;

[0059] Based on the primitive area and perimeter, the fire point shape feature is determined.

[0060] In a possible design, the fire recognition model is trained in the following manner;

[0061] Obtain a training set, where the training set contains sample fire feature data corresponding to a number of sample fire monitoring images;

[0062] Initialize the model parameters of the initial fire recognition model to obtain several groups of initial model parameters;

[0063] According to several groups of initial model parameters, an initial particle population is constructed, where the initial particle population includes several initial particle individuals, each initial particle individual is configured with an initial position vector and an initial velocity vector, and the initial position vector of each initial particle individual corresponds to a group of initial model parameters respectively;

[0064] Initialize the iteration number \(t\), obtain the particle population at the \(t\)-th iteration, and construct the initial fire recognition models corresponding to each particle individual at the \(t\)-th iteration based on the position vectors of each particle individual in the particle population at the \(t\)-th iteration. Among them, when \(t = 1\), the particle population at the \(t\)-th iteration is the initial particle population;

[0065] Use the training set to train each initial fire recognition model, and use the model outputs of each initial fire recognition model to calculate the fitness of each particle individual at the \(t\)-th iteration. Among them, the fitness of any particle individual is used to characterize the classification accuracy of the initial fire recognition model corresponding to that particle individual;

[0066] Use the fitness of each particle individual at the \(t\)-th iteration to update the global extreme value and the individual extreme value of each particle individual, and obtain the updated global extreme value and the updated individual extreme value corresponding to each particle individual. Among them, the global extreme value is the position vector corresponding to the particle individual with the largest fitness during the 1st to \(t - 1\) iterations;

[0067] Judge whether the iteration stop condition is met. Among them, the iteration stop condition is that the fitness of the particle individual corresponding to the updated global extreme value is greater than or equal to the fitness threshold, or \(t\) is greater than or equal to the maximum number of iterations;

[0068] If not, calculate the population fitness according to the fitness of each particle individual at the \(t\)-th iteration;

[0069] Use the updated global extreme value and the population fitness to calculate the mutation probability;

[0070] Generate a mutation random number, and judge whether the mutation random number is less than the mutation probability. Among them, the value range of the mutation random number is \([0, 1]\);

[0071] If so, perform a mutation operation on the updated global extreme value to obtain the mutated global extreme value at the \(t\)-th iteration;

[0072] Use the mutated global extreme value to update the particle population at the \(g\)-th iteration to obtain a new particle population;

[0073] Increment \(t\) by 1, update the particle population at the \(t\)-th iteration to the new mutated population at the \((t - 1)\)-th iteration, and re-construct the initial fire recognition models corresponding to each particle individual at the \(t\)-th iteration based on the position vectors of each particle individual in the particle population at the \(t\)-th iteration until the iteration stop condition is met, so as to determine the optimal model parameters of the initial fire recognition model based on the updated global extreme value when the iteration stop condition is met;

[0074] Based on the optimal model parameters, the fire recognition model is constructed.

[0075] In a second aspect, an intelligent fire recognition and early warning system based on deep learning is provided, including:

[0076] An image processing unit, configured to perform Gaussian noise reduction processing and quality enhancement processing on the real-time fire monitoring image in sequence to obtain an enhanced fire monitoring image.

[0077] A contour extraction unit, configured to extract the flame area image in the enhanced fire monitoring image and the foreground image in the real-time fire monitoring image, and perform fusion processing on the foreground image and the flame area image to generate a fused flame area image.

[0078] A contour extraction unit, configured to perform edge extraction on the fused flame area image to obtain an initial fire contour image.

[0079] The contour extraction unit is further configured to generate a no-fire scene contour image by using the calibrated monitoring image, and generate a fire contour image based on the initial fire contour image and the no-fire scene contour image.

[0080] A fire recognition unit, configured to perform feature extraction processing on the fire contour image to obtain fire feature data.

[0081] An early warning unit, configured to input the fire feature data into the fire recognition model to obtain a fire recognition result of the target area, and send a fire alarm message to the duty end when the fire recognition result is a fire occurrence.

[0082] In a third aspect, an electronic device is provided, including a memory, a processor, and a transceiver that are communicatively connected in sequence, where the memory is configured to store a computer program, the transceiver is configured to send and receive messages, and the processor is configured to read the computer program and execute the intelligent fire recognition and early warning method based on deep learning as described in the first aspect or any possible design in the first aspect.

[0083] In a fourth aspect, a storage medium is provided, on which instructions are stored, and when the instructions are run on a computer, the intelligent fire recognition and early warning method based on deep learning as described in the first aspect or any possible design in the first aspect is executed.

[0084] In a fifth aspect, a computer program product including instructions is provided, and when the instructions are run on a computer, the computer is caused to execute the intelligent fire recognition and early warning method based on deep learning as described in the first aspect or any possible design in the first aspect.

[0085] Advantageous effects:

[0086] (1) Before extracting the flame contour, the present invention performs Gaussian noise reduction and quality enhancement processing, which can achieve the purpose of removing image noise and improving image quality. At the same time, when extracting the contour, the foreground image of the target area and the scene contour image without fire are fused. In this way, while retaining the edge features of the flame body, the interference of non-flame factors, especially background interference, can be removed as much as possible. Based on this, the image noise can be further reduced. Finally, feature extraction is performed on the extracted fire contour image, and the extracted features are input into the fire recognition model, then fire warning can be realized. Therefore, when the present invention performs fire recognition, the noise in the image is removed to the greatest extent. Therefore, the accuracy of subsequent fire recognition can be improved, and thus it is very suitable for large-scale application and promotion.

[0087] (2) The present invention uses an improved particle swarm to optimize the model parameters, that is, during the particle movement process, a mutation operation is added. In this way, it can prevent falling into local optimum and thus avoid the problem of premature convergence. Based on this, the accuracy of searching for the optimal model parameters can be improved, thus ensuring the recognition performance of the fire recognition model. Description of the Drawings

[0088] Figure 1 It is a schematic diagram of the step flow of the intelligent fire recognition and warning method based on deep learning provided by the embodiment of the present invention;

[0089] Figure 2 It is a schematic diagram of the structure of the intelligent fire recognition and warning system based on deep learning provided by the embodiment of the present invention;

[0090] Figure 3 It is a schematic diagram of the structure of the electronic device provided by the embodiment of the present invention. Detailed Embodiments

[0091] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the present invention in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. It should be noted here that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation to the present invention.

[0092] It should be understood that although terms such as first and second may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, the first unit can be called the second unit, and similarly, the second unit can be called the first unit, without departing from the scope of the exemplary embodiments of the present invention.

[0093] It should be understood that for the term "and / or" that may appear in this text, it is merely a correlation relationship describing associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, B exists alone, and both A and B exist simultaneously. For the term " / and" that may appear in this text, it describes another associated object relationship, indicating that there can be two relationships. For example, A / and B can represent: A exists alone, and both A and B exist. Additionally, for the character " / " that may appear in this text, it generally indicates that the associated objects before and after are in an "or" relationship.

[0094] Embodiment:

[0095] See Figure 1 As shown, the intelligent fire recognition and early warning method based on deep learning provided in this embodiment uses Gaussian noise reduction and quality enhancement methods to perform image denoising before contour extraction, thereby reducing the interference of noise on edge recognition. At the same time, when extracting contours, the foreground image of the target area and the scene contour image without fire are fused. In this way, while retaining the edge features of the flame body, the interference of non-flame factors can be removed as much as possible. Based on this, the image noise can be further reduced. Finally, feature extraction is performed on the extracted fire contour image, and the extracted features are input into the fire recognition model, then fire early warning can be achieved. Therefore, when this method performs fire recognition, the noise in the image is removed to the greatest extent. Therefore, the accuracy of subsequent fire recognition can be improved, and it is very suitable for large-scale application and promotion. Among them, for example, this method can be but is not limited to running on the fire recognition edge side. Optionally, the fire recognition edge can be but is not limited to a personal computer (PC), a tablet computer, or a smart phone. It can be understood that the foregoing execution subject does not constitute a limitation on the embodiments of the present application. Correspondingly, the running steps of this method can be but are not limited to the following steps S1 to S7.

[0096] S1. Obtain the real-time fire monitoring image of the target area and the calibrated monitoring image without fire. In specific implementation, for example, a camera can be installed in the target area. Then, the real-time fire monitoring image (this image is an RGB image) of the target area is collected through the camera and uploaded to the fire recognition edge. At the same time, the foregoing calibrated monitoring image can be pre-stored in the fire recognition edge and can be called when in use.

[0097] After obtaining the real-time fire monitoring image and the calibrated monitoring image, fire recognition can be carried out. Among them, due to the influence of factors such as the environment and equipment on the collected images, there is a lot of noise in the images. Therefore, in order to ensure the accuracy of subsequent fire recognition, this embodiment needs to perform denoising processing on the aforementioned collected images; among them, in this embodiment, denoising processing is performed before edge extraction and during the edge extraction process, and the process can be but is not limited to the following steps S2 to S5.

[0098] S2. Perform Gaussian denoising processing and quality enhancement processing on the real-time fire monitoring image in sequence to obtain an enhanced fire monitoring image; in specific implementation, for example, it can be but is not limited to adopting the following steps S21 to S26 to implement the Gaussian denoising and quality enhancement processing of the real-time fire monitoring image.

[0099] S21. Perform fast Fourier transform processing on the real-time fire monitoring image to obtain a Fourier transform signal; in this embodiment, fast Fourier transform is a common technique in image processing, and its principle will not be elaborated here; after completing the fast Fourier transform of the real-time fire monitoring image, the frequency-domain image (i.e., the aforementioned Fourier transform signal) can be input into a Gaussian low-pass filter to filter out the high-frequency part of the image and retain the low-frequency part, so as to achieve the purpose of denoising; among them, the Gaussian low-pass filtering process can be but is not limited to the following steps S22.

[0100] S22. Input the Fourier transform signal into a Gaussian low-pass filter for filtering processing to obtain a filtered signal; in specific implementation, for example, it can be but is not limited to adopting the following formula (2) to represent the aforementioned Gaussian low-pass filtering process.

[0101] G(r,p) = F(r,p)H(r,p) (2)

[0102] In the above formula (2), G(r,p), F(r,p), and H(r,p) represent the filtered signal, the Fourier transform signal, and the transfer function of the Gaussian low-pass filter in sequence, where:

[0103]

[0104] In formula (3), κ0 represents the cut-off frequency of the Gaussian low-pass filter, and the value in this embodiment is 50. κ(r,p) represents the distance between the origin of the frequency plane and the midpoint (r,p) of the Fourier transform signal (i.e., the frequency-domain image), and In the formula, P and B represent the rows and columns of the frequency-domain image.

[0105] Thus, through the foregoing formulas (2) and (3), the Gaussian noise reduction processing of the real-time fire monitoring image can be completed. Then, by performing an inverse Fourier transform on the filtered signal, a noise-reduced fire monitoring image can be obtained. The process can be but is not limited to the steps shown in S23 below.

[0106] S23. Perform an inverse fast Fourier transform on the filtered signal to obtain a noise-reduced fire monitoring image; in this embodiment, the inverse fast Fourier transform is a commonly used technique for image restoration, and its principle will not be elaborated here; after the Gaussian noise reduction of the image is completed, the image quality enhancement process can be carried out, and the process is shown in steps S24 to S26 below.

[0107] S24. Convert the noise-reduced fire monitoring image into an HSI image and divide the HSI image into several image blocks; in this embodiment, the conversion process between the RGB image and the HSI image is a commonly used conversion technique in the color space, and its principle will not be elaborated here. For example, the HSI image is divided into 8 image blocks; of course, the specific division data of the image blocks can be specifically set according to actual use and is not limited to the foregoing example.

[0108] After the division of the image blocks is completed, the quality enhancement process can be carried out, and the process is shown in step S25 below.

[0109] S25. Perform quality enhancement processing on each image block to obtain each quality-enhanced image block, and use each quality-enhanced image block to generate an initial enhanced fire monitoring image; in specific implementation, in this embodiment, a fast Fourier transform is performed on the luminance component of each image block, and a high-pass filter is performed on the frequency domain signal obtained after the transformation to suppress the influence of the background light luminance and improve the image contrast.

[0110] Among them, the foregoing quality enhancement process can be but is not limited to the steps shown in S25a to S25e below.

[0111] S25a. Separate the luminance component image, saturation component image, and hue component image from each image block; in this embodiment, after separating the luminance component, saturation component, and hue component images of each image block, a fast Fourier transform can be performed on the luminance component image, and the process is shown in step S25b below.

[0112] S25b. Perform fast Fourier transform processing on the luminance component images corresponding to each image block to obtain the frequency domain signals corresponding to each image block, and perform mean filtering processing on each frequency domain signal to obtain each filtered frequency domain signal; in specific implementation, for example, performing mean filtering on the frequency domain signals of each image block is to eliminate noises such as rough edges. Among them, for example, the filtering radius of the mean filtering is 3, and for any frequency domain signal (which is essentially an image), mean filtering processing can be performed on the image blocks adjacent to it horizontally and vertically; after completing the mean filtering processing of each image block, quality enhancement processing can be performed, that is, using a Gaussian high-pass filter to perform filtering processing on each filtered frequency domain signal, and its process is as shown in the following step S25c. S25c. Use a Gaussian high-pass filter to perform quality enhancement processing on each filtered frequency domain signal to obtain each enhanced frequency domain signal; in specific applications, in this embodiment, a Gaussian high-pass filter dedicated to quality enhancement of fire monitoring images is constructed, and its transfer function can be but is not limited to the following formula (4).

[0113]

[0114] In the above formula (4), U(u, v) represents the transfer function of the Gaussian high-pass filter, g h ,g l represent the high-frequency gain and low-frequency gain respectively (in this embodiment, they are 1.5 and 1 respectively), represents the sharpening coefficient (in this embodiment, the value is 1), κ(u, v) represents the distance between the origin of the frequency plane and the midpoint (u, v) of the filtered frequency domain signal (the calculation process can refer to the foregoing formula (3)), κ0 represents the cut-off frequency (in this embodiment, the value is 4), and n represents the order of the filter.

[0115] In this way, through the foregoing formula (4), the Gaussian high-pass filtering processing of each filtered frequency domain signal can be completed. Then, exponential transformation can be performed on it to obtain each enhanced frequency domain signal; finally, inverse fast Fourier transform processing is performed, and the enhanced luminance component image can be obtained, and its process is as shown in the following step S25d.

[0116] S25d. Perform inverse fast Fourier transform processing on each enhanced frequency domain signal to obtain each enhanced luminance component image; after obtaining each enhanced luminance component image, fuse it with the corresponding saturation component image and hue component image, and each quality-enhanced image block can be obtained, and its process is as shown in the following step S25e.

[0117] S25e. Fuse each enhanced luminance component image with the saturation component image and hue component image in the corresponding image block to obtain each quality-enhanced image block after the fusion processing.

[0118] In this way, through the foregoing steps S25a to S25e, the quality enhancement processing of each image block can be completed; then, by combining the quality-enhanced image blocks, the initial enhanced fire monitoring image can be obtained.

[0119] After obtaining the initial enhanced fire monitoring image, converting it into an RGB image can obtain the enhanced fire monitoring image, and the color space conversion process is as shown in the following step S26.

[0120] S26. Convert the initial enhanced fire monitoring image into an RGB image so as to obtain the enhanced fire monitoring image after conversion.

[0121] Thus, through the foregoing steps S21 to S26, Gaussian noise reduction and quality enhancement processing can be performed before fire contour extraction, so as to achieve the purpose of removing image noise and improving image quality; after completing the noise reduction processing before contour extraction, contour extraction can be performed. Among them, this embodiment provides an improved contour extraction method. During the contour extraction process, by combining foreground and background separation, image AND operation, and fusion processing of the non-fire scene contour image, etc., while retaining the edge features of the flame main body, the interference of relevant non-flame factors can be removed as much as possible, so as to extract the fire contour image with the least noise; optionally, the contour extraction process can be but is not limited to the following steps S3 to S5.

[0122] S3. Extract the flame region image in the enhanced fire monitoring image and the foreground image in the real-time fire monitoring image, and perform fusion processing on the foreground image and the flame region image to generate a fused flame region image; in specific implementation, in this embodiment, the enhanced fire monitoring image is converted into a YCbCr image, and then based on each color component in YCbCr, the foregoing flame region image is extracted.

[0123] Optionally, the extraction process of the flame region image can be but is not limited to the following steps S31 to S39.

[0124] S31. Convert the enhanced fire monitoring image to the YCbCr color space to obtain a YCbCr image; in this embodiment, for any pixel point in the enhanced fire monitoring image, the following formula (5) can be used but is not limited to it to obtain the Y component (luminance), Cb component (blue difference, that is, the difference between the blue component and the luminance), and Cr component (red difference, that is, the difference between the red component and the luminance) corresponding to the any pixel point.

[0125]

[0126] In the above formula (5), Y, Cb, and Cr respectively represent the luminance, blue difference, and red difference of any pixel point in the enhanced fire monitoring image, and R, G, and B respectively represent the red, green, and blue components of any pixel point in the enhanced fire monitoring image.

[0127] Thus, after obtaining the YCbCr image corresponding to the enhanced fire monitoring image through the foregoing formula (5), the extraction of the flame area can be performed based on the foregoing YCbCr image, and the process is as shown in the following steps S32 to S39.

[0128] S32. For any pixel point in the enhanced fire monitoring image, determine the Y component value and Cb component value of the any pixel point from the YCbCr image; after extracting the luminance component value and blue difference component value of the any pixel point, the update of the original pixel point (i.e., RGB pixel) of the any pixel point can be performed based on the two, and the process is as shown in the following steps S33 and S34.

[0129] S33. Determine whether the Y component value is greater than the Cb component value; in specific applications, if the Y component value of the any pixel point is greater than the Cb component value, the original pixel of the any pixel point can be retained, that is, the pixel value in the original RGB is retained, otherwise, it needs to be set to 0. In this way, it is equivalent to initially retaining the pixel points belonging to the flame; the pixel update process is as shown in the following step S34.

[0130] S34. If so, keep the pixel value of the any pixel point in the enhanced fire monitoring image, otherwise, set the pixel value of the any pixel point in the enhanced fire monitoring image to 0, and after polling all the pixel points in the enhanced fire monitoring image, an initial fire area image is obtained.

[0131] After completing the pixel update of the any pixel point based on the foregoing step S34, the pixel update of the remaining pixel points in the enhanced fire monitoring image can be performed in the foregoing manner. In this way, the preliminary extraction of the flame area can be completed to obtain an initial fire area image; then, the secondary extraction of the flame area can be performed, and the process is as shown in the following steps S35 to S39.

[0132] S35. Calculate the mean value of the red component and the mean value of the luminance of the enhanced fire monitoring image; in this embodiment, the mean value of the red component and the mean value of the luminance in the original RGB image are calculated (the luminance of the pixel point in the RGB image is the mean value of the sum of the three-channel component values of the pixel point), and then, based on the foregoing mean value of the red component and the mean value of the luminance, combined with the luminance and red difference of each pixel point in the initial fire area image, the secondary extraction of the flame area can be performed, and the process is as shown in the following steps S35 to S39.

[0133] S36. For the k-th pixel point in the initial fire area image, determine the Y component value and the Cr component value corresponding to the k-th pixel point from the YCbCr image; in specific applications, after obtaining the Y component value and the Cr component value corresponding to the k-th pixel point, they can be respectively compared with the brightness mean value and the red component mean value, so as to determine whether the pixel value of the k-th pixel point is updated based on the comparison result, where the judgment and pixel update processes are as shown in the following steps S37 and S38.

[0134] S37. Judge whether the Y component value corresponding to the k-th pixel point is greater than the brightness mean value, and whether the Cr component value of the k-th pixel point is greater than the red component mean value; in this embodiment, if the above two conditions are satisfied simultaneously, then keep the pixel value of the k-th pixel point, otherwise, the k-th pixel point needs to be regarded as the background, that is, set to 0; where the pixel point update process is as shown in the following step S38.

[0135] S38. If so, keep the pixel value of the k-th pixel point in the initial fire area image, otherwise, set the pixel value of the k-th pixel point in the initial fire area image to 0.

[0136] After completing the pixel update of the k-th pixel point, in the same way as above, the update of the remaining pixel points in the initial fire area image can be completed, and the process is as shown in the following step S39.

[0137] S39. Increment k by 1, and re-determine the Y component value and the Cr component value corresponding to the k-th pixel point from the YCbCr image until k is equal to K, then obtain the fire area image, where the initial value of k is 1, and K is the total number of pixel points in the initial fire area image.

[0138] Thus, through the above steps S31 - S39, the flame area can be extracted from the enhanced fire monitoring image. Then, in order to reduce background interference in this embodiment, foreground and background separation is also performed before contour extraction.

[0139] Among them, for example, but not limited to, the otsu algorithm can be used to separate the foreground and background of the enhanced fire monitoring image. Among them, the otsu algorithm is a commonly used algorithm for foreground and background separation, and its principle will not be elaborated here.

[0140] After obtaining the foreground image of the enhanced fire monitoring image, image fusion processing can be performed. Specifically, the fusion process is as follows: First, perform median filtering on the flame area image to obtain the filtered flame area image. In this way, smooth denoising of the flame area can be achieved to remove small noise points. Then, perform image AND operation on the foreground image and the filtered flame area image. After the image AND operation, the fused flame area image is obtained. Based on this, performing image AND operation on the foreground image and the flame area image can further reduce background interference in the image.

[0141] After completing the denoising process before edge extraction based on the foregoing step S3 and its sub-steps, edge extraction can be performed. The process can be but is not limited to the following step S4.

[0142] S4. Perform edge extraction on the fused flame area image to obtain the initial fire contour image. In specific implementation, an edge extraction algorithm for both images can be used to perform edge extraction on the fused flame area image, thereby obtaining the initial fire contour image (i.e., the flame contour image). For example, but not limited to, the Sobel operator can be used to complete the edge extraction. Of course, other edge extraction algorithms can also be used, and it is not limited to the foregoing example here.

[0143] After obtaining the initial fire contour image, in order to further reduce background interference in this embodiment, a calibrated monitoring image is also used to generate a fire-free scene contour image, and then the two are fused to generate a fire contour image with the least background interference information. The foregoing process can be but is not limited to the following step S5.

[0144] S5. Use the calibrated monitoring image to generate a fire-free scene contour image, and based on the initial fire contour image and the fire-free scene contour image, generate a fire contour image. In specific implementation, the generation process of the fire-free scene contour image is the same as the generation process of the foregoing initial fire contour image. First, perform Gaussian denoising and quality enhancement on the calibrated monitoring image. Then, extract the scene contour image and the foreground image, fuse the two, and finally, perform edge extraction on the fused image to obtain the fire-free scene contour image. Similarly, perform image AND operation on the foregoing initial fire contour image and the foregoing fire-free scene contour image, and the fire contour image can be obtained after the image AND operation.

[0145] In this way, through the edge extraction technology provided by the foregoing step S4 and step S5, while retaining the edge features of the flame main body, interference from non-flame factors (i.e., background interference) can be removed as much as possible. Based on this, the final binary image of the fire flame edge with the least interference information can be obtained. Then, feature extraction processing can be performed to obtain feature data for subsequent fire recognition.

[0146] Among them, the feature extraction process may but is not limited to the steps shown in step S6 below.

[0147] S6. Perform feature extraction processing on the fire contour image to obtain fire feature data; in specific applications, in this embodiment, the color feature, the fire point shape feature, the fire texture feature, and the flame flicker frequency feature of the fire contour image are extracted, so as to use the foregoing four features to form the fire feature data; specifically, the foregoing feature extraction process may but is not limited to the steps shown in steps S61 to S65 below.

[0148] S61. Perform color feature extraction processing on the fire contour image to obtain the fire color feature; in this embodiment, the binary image can be converted into an RGB image, and then the color feature is extracted. Among them, Python and the OpenCV library can be used for the conversion, which is a common technique for image color space conversion and will not be elaborated here.

[0149] Among them, one of the following common color feature extraction methods is used. For example, the following formula (1) is adopted to extract the first color feature, the second color feature, and the third color feature.

[0150]

[0151] In the above formula (1), δ1, δ2, and δ3 respectively represent the first color feature, the second color feature, and the third color feature, and f m represents the pixel value of the m-th pixel point in the fire contour image, and M represents the total number of pixel points in the fire contour image.

[0152] In this way, after extracting three different color features based on the foregoing formula (1), the first color feature, the second color feature, and the third color feature can be used to form the fire color feature.

[0153] After the extraction of the fire color feature is completed, the fire point shape feature extraction can be performed, and the process is as shown in step S62 below.

[0154] S62. Extract the shape feature of the fire point in the fire contour image to obtain the fire point shape feature; in specific implementation, it may but is not limited to first calculating the primitive area and perimeter of the fire contour image; then, based on the primitive area and perimeter, determining the fire point shape feature.

[0155] Optionally, for example but not limited to, the contourArea and arcLength functions in OpenCV can be used to calculate the area and perimeter of the contour respectively, which are common calculation methods for the area and perimeter of contour primitives, and their principles will not be calculated here; at the same time, for example but not limited to, the following formula (6) can be used to calculate the fire point shape feature.

[0156]

[0157] In the above formula (6), Q, σ, and z represent the fire point shape feature, the primitive area, and the perimeter respectively.

[0158] In this embodiment, the fire point shape feature is essentially a measure describing the similarity degree between the fire point contour and a perfect circle, and is an important basis for identifying the fire point; thus, after the extraction of the fire point shape feature is completed, the texture feature extraction can be carried out, and the process can be, for example but not limited to, as shown in the following step S63.

[0159] S63. Perform texture feature extraction processing on the fire contour image to obtain the fire texture feature; in specific implementation, for example but not limited to, the LBP (Local Binary Pattern) algorithm and the LPQ (Local Phase Quantization) algorithm can be used to extract the first texture feature and the second texture feature respectively, and then, the two are combined to obtain the fire texture feature.

[0160] After the fire texture feature of the fire contour image is extracted, the calculation of the flame flicker frequency can be carried out, and the process is as shown in the following step S64.

[0161] S64. Determine the flame flicker frequency in the fire contour image to obtain the flame flicker frequency feature; in specific implementation, for example but not limited to, based on the fire contour image, calculate the flame equivalent diameter, and then, based on the flame equivalent diameter, determine the flame flicker frequency feature.

[0162] Optionally, for example but not limited to, the following formula (7) can be used to calculate the flame equivalent diameter.

[0163]

[0164] In the above formula (7), τ represents the flame equivalent diameter, and σ represents the primitive area of the fire contour image; thus, after the flame equivalent diameter is calculated, the ratio between 2.3 and the flame equivalent can be used as the flame flicker frequency feature.

[0165] After the flame flicker frequency feature is determined based on the foregoing step S64, the three foregoing features can be combined to generate the fire feature data, and the process is as shown in the following step S65.

[0166] S65. Using the fire color feature, the fire point shape feature, the fire texture feature, and the flame flicker frequency feature, the fire feature data is formed.

[0167] Thus, through the foregoing steps S61 to S65, the fire feature data can be extracted; then, inputting it into a pre-trained fire recognition model, the fire recognition result of the target area can be obtained, and the process is as shown in the following step S7.

[0168] S7. Input the fire feature data into the fire recognition model to obtain the fire recognition result of the target area, and when the fire recognition result is a fire occurrence, send a fire alarm message to the duty end; in this embodiment, for example, the fire recognition model can be but is not limited to a trained CNN network, an SVM network, etc. Among them, the foregoing model is trained with the sample fire feature data of each sample area as the input and the fire recognition results of each sample area as the output (the specific training process is elaborated in detail in the second aspect of the embodiment), and the fire recognition result is a fire state (i.e., a fire occurs) or a normal state.

[0169] In this way, when the obtained fire recognition result is a fire occurrence, a warning message can be sent to the duty end, and at the same time, the fire monitoring image can be sent to the duty end so that the duty personnel can take fire protection measures in time; of course, a warning message can also be sent to the fire control center at the same time so that the fire fighters can arrive at the scene in time for handling, thereby preventing fire accidents.

[0170] Thus, through the intelligent fire recognition and early warning method based on deep learning detailed in the foregoing steps S1 to S7, the present invention uses Gaussian denoising and quality enhancement methods to perform image denoising processing before contour extraction, thereby reducing the interference of noise on edge recognition; at the same time, when extracting the contour, the foreground image of the target area and the scene contour image without fire are fused. In this way, while retaining the edge features of the flame main body, the interference of non-flame factors can be removed as much as possible; based on this, the image noise can be further reduced; finally, feature extraction is performed on the extracted fire contour image, and the extracted features are input into the fire recognition model, then fire early warning can be realized; thus, when the present invention performs fire recognition, the noise in the image is removed to the greatest extent, so the accuracy of subsequent fire recognition can be improved, and thus it is very suitable for large-scale application and promotion.

[0171] In a possible design, the second aspect of this embodiment provides the training process of the fire recognition model in the first aspect of the embodiment. Among them, this embodiment uses an improved particle swarm to perform parameter optimization of the fire recognition model, thereby generating an optimal model; optionally, the training process of the foregoing fire recognition model is as shown in the following steps S01 to S014.

[0172] S01. Obtain a training set, where the training set contains sample fire feature data corresponding to a number of sample fire monitoring images; in this embodiment, for the generation process of the sample fire feature data of each sample fire monitoring image, reference can be made to the first aspect of the foregoing embodiment, which will not be elaborated herein.

[0173] After obtaining the training set, the model parameters of the initial fire recognition model can be initialized, so as to subsequently construct a particle population based on this. The model initialization process is as shown in the following step S02.

[0174] S02. Perform an initialization process on the model parameters of the initial fire recognition model to obtain several groups of initial model parameters; in this embodiment, assuming that the initial fire recognition model is an SVM model, then the model parameters are the penalty parameter and the kernel parameter, and if the initial fire recognition model is a CNN model, the model parameters are the convolutional layer parameters (such as the convolutional kernel size, stride), pooling layer parameters, neuron weights, biases, regularization term weights, etc.; thus, by initializing the model parameters, different initial model parameters can be generated, and different initial model parameters can construct corresponding particle individuals, and the process is as shown in the following step S03.

[0175] S03. Construct an initial particle population according to several groups of initial model parameters, where the initial particle population includes several initial particle individuals, each initial particle individual is configured with an initial position vector and an initial velocity vector, and the initial position vector of each initial particle individual corresponds to a group of initial model parameters respectively; in this embodiment, the number of particle individuals is the same as the number of initial model parameters, the initial position vector of each particle individual is an initial model parameter, and the initial velocity vector is a preset value; thus, after constructing the initial particle population, the optimization of the model parameters can be carried out by using the movement of the population, and the process can be but is not limited to the following steps S04 to S013.

[0176] S04. Initialize the iteration number t, obtain the particle population at the t-th iteration, and construct the initial fire recognition model corresponding to each particle individual at the t-th iteration based on the position vectors of the particle individuals in the particle population at the t-th iteration, where when t is 1, the particle population at the t-th iteration is the initial particle population; in this embodiment, since it has been described above that the position vector of each particle individual corresponds to a group of model parameters, therefore, in each iteration process, the position vector of each particle individual can construct an initial fire recognition model, and in this embodiment, the foregoing training set is used to train the models corresponding to each particle individual, and based on the model output results, the fitness of each particle individual is calculated, so as to update the speed and position of the particle individual.

[0177] Among them, the calculation process of the fitness of each particle individual at the t-th iteration is as shown in the following step S05.

[0178] S05. Using the training set, train each initial fire recognition model, and use the model outputs of each initial fire recognition model to calculate the fitness of each particle individual at the t-th iteration. Among them, the fitness of any particle individual is used to characterize the classification accuracy of the initial fire recognition model corresponding to the any particle individual; in specific implementation, it is equivalent to using the sample fire feature data corresponding to each sample fire monitoring image in the training set as the input, and the fire recognition results of the corresponding areas of each sample fire monitoring image as the output to train the models corresponding to each particle individual; and use the ratio between the number of correct classifications of each model for the training set and the total number of training data as the fitness; based on this, the more accurate the classification, the higher the fitness.

[0179] After calculating the fitness of each particle individual at the t-th iteration based on step S05, the global extreme value and the individual extreme value can be updated, and the process is as shown in the following step S06.

[0180] S06. Using the fitness of each particle individual at the t-th iteration, update the global extreme value and the individual extreme value of each particle individual to obtain the updated global extreme value and the updated individual extreme value corresponding to each particle individual. Among them, the global extreme value is the position vector corresponding to the particle individual with the largest fitness during the iteration process from the 1st to the (t - 1)-th iteration; in specific implementation, if the largest fitness among the particle individuals at the t-th iteration is greater than the fitness of the particle individual corresponding to the global extreme value at the (t - 1)-th iteration, then update the global extreme value at the t-th time to the position vector of the particle individual corresponding to the largest fitness at the t-th iteration, otherwise, the global extreme value at the t-th time (i.e., the updated global extreme value) remains the global extreme value at the (t - 1)-th iteration; of course, the individual extreme value of each particle individual is also the same, that is, if the fitness of any particle individual at the t-th iteration is greater than its fitness at the (t - 1)-th iteration, then update the individual extreme value to the position vector at the t-th iteration, otherwise, keep it as the position vector at the previous iteration.

[0181] In this way, after completing the update of the global extreme value and the individual extreme value, it can be judged whether the iteration stop condition is satisfied, and the judgment process can be but is not limited to as shown in the following step S07.

[0182] S07. Judge whether the iteration stop condition is met, where the iteration stop condition is that the fitness of the particle individual corresponding to the updated global extreme value is greater than or equal to the fitness threshold, or t is greater than or equal to the maximum number of iterations; in this embodiment, the fitness threshold and the maximum number of iterations can be specifically set according to actual use and are not specifically limited here.

[0183] In specific implementation, if the iteration stop condition is not satisfied, it is necessary to mutate the updated global extreme value to prevent the updated global extreme value from falling into a local optimum, thus causing premature convergence. Among them, the mutation process of the updated global extreme value can be but is not limited to the steps shown in the following steps S08 to S11.

[0184] S08. If not, calculate the population fitness according to the fitness of each particle individual at the t-th iteration. In specific implementation, for example, but not limited to, the following formula (8) can be used to calculate the population fitness.

[0185]

[0186] In the above formula (8), represents the population fitness, R a represents the fitness of the a-th particle individual in the particle population at the t-th iteration, is the average value of the fitness of each particle individual at the t-th iteration, R is the fitness calibration factor, and A represents the total number of particle individuals.

[0187] Among them,

[0188] In this way, after calculating the population fitness through the foregoing formula (8), the mutation probability can be calculated in combination with the foregoing updated global extreme value. Among them, its calculation process is as shown in the following step S09.

[0189] S09. Calculate the mutation probability by using the updated global extreme value and the population fitness. In specific implementation, for example, but not limited to, the following formula (9) can be used to calculate the mutation probability.

[0190]

[0191] In the above formula (9), ψ t represents the mutation probability, rand() represents taking a random number, represents the population fitness threshold, represents the updated global extreme value, and B′ represents the global extreme value threshold.

[0192] After calculating the mutation probability based on the foregoing formula (9), a mutation random number can be generated and combined with the foregoing mutation probability to determine whether the mutation of the updated global extreme value can be performed. Among them, the mutation process is as shown in the following step S010 and step S011.

[0193] S010. Generate a mutated random number and determine whether the mutated random number is less than the mutation probability, where the value range of the mutated random number is [0, 1]; in this embodiment, if the mutated random number is less than the mutation probability, a mutation operation on the updated global extreme value is required, and the process is as shown in the following step S011.

[0194] S011. If so, perform a mutation operation on the updated global extreme value to obtain the mutated global extreme value at the t-th iteration; in specific applications, for example but not limited to, the following formula (10) can be used to obtain the mutated global extreme value.

[0195]

[0196] In the above formula (10), represents the mutated global extreme value, ζ represents the mutation factor, which conforms to the Gaussian (0, 1) distribution.

[0197] Thus, through the foregoing formula (10), after completing the mutation operation on the updated global extreme value, particle update can be performed, and the process can be, for example but not limited to, as shown in the following step S012.

[0198] S012. Use the mutated global extreme value to update the particle population at the g-th iteration to obtain a new particle population; in specific implementation, for any particle individual at the g-th iteration, for example but not limited to, the following formulas (11) and (12) can be used to update its velocity vector and position vector.

[0199]

[0200] X t ′ = V t ′ + X t (12)

[0201] In the above formula (11), V t ′ represents the updated velocity vector corresponding to any particle individual, V t represents the velocity vector of any particle individual (i.e., the velocity vector at the t-th iteration), X t represents the position vector of any particle individual, represents the updated individual extreme value corresponding to any particle individual, c1, c2 represent learning factors (the value in this embodiment is 2), rand1, rand2 represent random numbers between [0, 1], and ω represents the inertia weight.

[0202] Among them, In the formula, ω0 represents the initial inertia weight (a preset value), t maxdenotes the maximum number of iterations; thus, adopting a non - linear inertia weight, a larger inertia weight is assigned at the initial stage of the algorithm, which is beneficial for the particles to jump out of local extrema and expand the search space; and as the particle swarm iteratively evolves, the inertia weight gradually decreases, which is beneficial for improving the convergence ability of the particles in the local space; thus, the global and local search abilities of the particle population can be balanced.

[0203] In the above formula (12), X t ′ represents the updated position vector corresponding to any particle individual.

[0204] In this embodiment, if the aforementioned mutation random number is greater than or equal to the mutation probability, then at this time, there is no need to mutate the updated global extremum, that is, directly substitute the updated global extremum into the aforementioned formula (11) to update the particle individuals.

[0205] Thus, after obtaining the new particle population, it can be used as the particle population for the next iteration. Then, continuously repeat the aforementioned process until the iteration stop condition is met, and then the optimal model parameters can be obtained; the iterative optimization process can be but is not limited to the following steps shown in S013.

[0206] S013. Increment t by 1 and update the particle population at the t - th iteration to the new mutated population at the (t - 1)-th iteration, and re - construct the initial fire recognition model corresponding to each particle individual at the t - th iteration based on the position vectors of each particle individual in the particle population at the t - th iteration until the iteration stop condition is met, so as to determine the optimal model parameters of the initial fire recognition model based on the updated global extremum when the iteration stop condition is met.

[0207] When the particle population evolves to meet the iteration stop condition, the updated global extremum at this time is the optimal model parameter of the initial fire recognition model. Then, the fire recognition model can be constructed based on this, and the process is as shown in the following step S014.

[0208] S014. Construct the fire recognition model based on the optimal model parameters.

[0209] Thus, through the aforementioned steps S01 - S014, the improved particle swarm algorithm can be used to optimize and obtain the optimal model parameters of the CNN or SVM model. Based on this, it can prevent the global extremum from falling into local optimality, thus avoiding the problem of premature convergence; thus, it can improve the accuracy of searching for the optimal model parameters, thereby ensuring the recognition performance of the fire recognition model.

[0210] After obtaining the fire recognition model, the fire feature data of the aforementioned target area can be input into the model to obtain the fire recognition result. Then, based on the fire recognition result, a fire alarm can be issued. The process can refer to the first aspect of the embodiment and will not be elaborated here.

[0211] As Figure 2 shown, the third aspect of this embodiment provides a hardware device for implementing the intelligent fire recognition and early warning method based on deep learning described in the first aspect of the embodiment, including:

[0212] An acquisition unit for acquiring the real-time fire monitoring image of the target area and the calibrated monitoring image when there is no fire.

[0213] An image processing unit for successively performing Gaussian noise reduction processing and quality enhancement processing on the real-time fire monitoring image to obtain an enhanced fire monitoring image.

[0214] A contour extraction unit for extracting the flame area image in the enhanced fire monitoring image and the foreground image in the real-time fire monitoring image, and performing fusion processing on the foreground image and the flame area image to generate a fused flame area image.

[0215] A contour extraction unit for performing edge extraction on the fused flame area image to obtain an initial fire contour image.

[0216] The contour extraction unit is further used to generate a no-fire scene contour image by using the calibrated monitoring image, and generate a fire contour image based on the initial fire contour image and the no-fire scene contour image.

[0217] A fire recognition unit for performing feature extraction processing on the fire contour image to obtain fire feature data.

[0218] An early warning unit for inputting the fire feature data into the fire recognition model to obtain the fire recognition result of the target area, and sending a fire alarm message to the duty end when the fire recognition result is that a fire has occurred.

[0219] The working process, working details and technical effects of the device provided in this embodiment can refer to the first aspect of the embodiment and will not be elaborated here.

[0220] As Figure 3 shown, the fourth aspect of this embodiment provides an electronic device, including: a memory, a processor and a transceiver that are communicatively connected in sequence, where the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the intelligent fire recognition and early warning method based on deep learning described in the first aspect and the second aspect of the embodiment.

[0221] Specifically, the memory may include, but is not limited to, random access memory (RAM), read only memory (ROM), flash memory, first input first output (FIFO) and / or first in last out (FILO), etc.; specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). At the same time, the processor may also include a main processor and a co-processor. The main processor is a processor used to process data in the wake state, also known as the CPU (Central Processing Unit); the co-processor is a low-power processor used to process data in the standby state.

[0222] In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. For example, the processor may not be limited to using a microprocessor of the STM32F105 series, a reduced instruction set computer (RISC) microprocessor, an X86 architecture processor, or a processor integrated with an embedded neural-network processing unit (NPU); the transceiver may include, but is not limited to, a Wi-Fi wireless transceiver, a Bluetooth wireless transceiver, a General Packet Radio Service (GPRS) wireless transceiver, a ZigBee (low-power local area network protocol based on the IEEE 802.15.4 standard) wireless transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver, etc. In addition, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0223] For the working process, working details and technical effects of the electronic device provided in this embodiment, reference may be made to the first aspect and the second aspect of the embodiment, which will not be elaborated herein.

[0224] The fifth aspect of this embodiment provides a storage medium storing instructions including the deep learning-based intelligent fire recognition and early warning method described in the first aspect and the second aspect of the embodiment, that is, instructions are stored on the storage medium, and when the instructions run on a computer, the deep learning-based intelligent fire recognition and early warning method described in the first aspect and the second aspect of the embodiment is executed.

[0225] Among them, the storage medium refers to a carrier for storing data, and may include, but is not limited to, floppy disks, optical discs, hard disks, flash memories, USB flash drives, and / or memory sticks, etc. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0226] For the working process, working details and technical effects of the storage medium provided in this embodiment, reference may be made to the first aspect and the second aspect of the embodiment, which will not be elaborated herein.

[0227] The sixth aspect of this embodiment provides a computer program product containing instructions, which, when running on a computer, causes the computer to execute the deep learning-based intelligent fire recognition and early warning method described in the first aspect and the second aspect of the embodiment, where the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0228] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent fire identification and early warning method based on deep learning, characterized in that: include: Obtain real-time fire monitoring images of the target area and calibrated monitoring images when there is no fire; The real-time fire monitoring image is subjected to Gaussian noise reduction processing and quality enhancement processing in sequence to obtain an enhanced fire monitoring image; Extracting a flame area image from the enhanced fire monitoring image and a foreground image from the real-time fire monitoring image, and fusing the foreground image with the flame area image to generate a fused flame area image; Perform edge extraction on the fused flame area image to obtain the initial fire contour image; Generate a non-fire scene contour image using the calibrated monitoring image, and generate a fire contour image based on the initial fire contour image and the non-fire scene contour image; Performing feature extraction processing on the fire contour image to obtain fire feature data; The fire characteristic data is input into a fire identification model to obtain a fire identification result of a target area, and when the fire identification result indicates that a fire has occurred, a fire alarm message is sent to an on-duty terminal.

2. The method according to claim 1, characterized in that The real-time fire monitoring image is subjected to Gaussian noise reduction processing and quality enhancement processing in sequence to obtain an enhanced fire monitoring image, including: Performing fast Fourier transform processing on the real-time fire monitoring image to obtain a Fourier transform signal; Inputting the Fourier transform signal into a Gaussian low-pass filter for filtering to obtain a filtered signal; Performing inverse fast Fourier transform processing on the filtered signal to obtain a noise-reduced fire monitoring image; Converting the de-noised fire monitoring image into an HSI image, and dividing the HSI image into a plurality of image blocks; Performing quality enhancement processing on each image block to obtain each quality enhanced image block, and using each quality enhanced image block to generate an initial enhanced fire monitoring image; The initial enhanced fire monitoring image is converted into an RGB image, so as to obtain the enhanced fire monitoring image after the conversion.

3. The method according to claim 2, characterized in that Performing quality enhancement processing on each image block to obtain each quality enhanced image block includes: Separate the brightness component image, the saturation component image and the hue component image from each image block; Performing fast Fourier transform processing on the brightness component image corresponding to each image block to obtain the frequency domain signal corresponding to each image block, and performing mean filtering processing on each frequency domain signal to obtain each filtered frequency domain signal; Using a Gaussian high-pass filter, quality enhancement processing is performed on each filtered frequency domain signal to obtain each enhanced frequency domain signal; Performing inverse fast Fourier transform processing on each enhanced frequency domain signal to obtain each enhanced brightness component image; Each enhanced brightness component image is fused with the saturation component image and the hue component image in the corresponding image block to obtain each quality enhanced image block after the fusion process.

4. The method according to claim 1, characterized in that: The enhanced fire monitoring image is an RGB image, wherein extracting a flame area image from the enhanced fire monitoring image includes: Converting the enhanced fire monitoring image to a YCbCr color space to obtain a YCbCr image; For any pixel point in the enhanced fire monitoring image, determining a Y component value and a Cb component value of the any pixel point from the YCbCr image; Determine whether the Y component value is greater than the Cb component value; If yes, the pixel value of any pixel point in the enhanced fire monitoring image is maintained; otherwise, the pixel value of any pixel point in the enhanced fire monitoring image is set to 0, and after all the pixels in the enhanced fire monitoring image are polled, an initial fire area image is obtained; Calculating the red component mean and brightness mean of the enhanced fire monitoring image; For a k-th pixel point in the initial fire area image, determining a Y component value and a Cr component value corresponding to the k-th pixel point from the YCbCr image; Determine whether the Y component value corresponding to the k-th pixel is greater than the brightness mean, and whether the Cr component value of the k-th pixel is greater than the red component mean; If yes, the pixel value of the k-th pixel point in the initial fire area image is maintained; otherwise, the pixel value of the k-th pixel point in the initial fire area image is set to 0; Add 1 to k, and re-determine the Y component value and Cr component value corresponding to the k-th pixel from the YCbCr image until k is equal to K, thereby obtaining the fire area image, wherein the initial value of k is 1, and K is the total number of pixels in the initial fire area image.

5. The method according to claim 1, characterized in that The foreground image and the flame region image are fused to generate a fused flame region image, including: Performing median filtering on the flame region image to obtain a filtered flame region image; Performing image AND operation processing on the foreground image and the filtered flame region image, so as to obtain the fused flame region image after the image AND operation processing; Accordingly, generating a fire contour image based on the initial fire contour image and the non-fire scene contour image includes: The initial fire contour image and the non-fire scene contour image are processed by image AND operation, so as to obtain the fire contour image after the image AND operation processing.

6. The method according to claim 1, characterized in that Performing feature extraction processing on the fire contour image to obtain fire feature data includes: Performing color feature extraction processing on the fire contour image to obtain fire color features; Extracting shape features of fire spots in the fire contour image to obtain fire spot shape features; Performing texture feature extraction processing on the fire contour image to obtain fire texture features; Determining the flame flicker frequency in the fire contour image to obtain a flame flicker frequency feature; The fire characteristic data is composed by using the fire color characteristics, the fire point shape characteristics, the fire texture characteristics and the flame flickering frequency characteristics.

7. The method according to claim 6, characterized in that Performing color feature extraction processing on the fire contour image to obtain fire color features includes: The following formula (1) is used to extract the first color feature, the second color feature and the third color feature; In the above formula (1), δ1, δ2, δ3 represent the first color feature, the second color feature, and the third color feature respectively, and f m represents the pixel value of the mth pixel in the fire contour image, and M represents the total number of pixels in the fire contour image; Using the first color feature, the second color feature and the third color feature to form the fire color feature; Correspondingly, extracting the shape features of the fire points in the fire contour image to obtain the shape features of the fire points includes: Calculating the area and perimeter of the image element of the fire contour image; The shape characteristics of the fire point are determined based on the area and perimeter of the graphic element.

8. The method according to claim 1, characterized in that The fire recognition model is trained in the following way; Acquire a training set, wherein the training set contains sample fire feature data corresponding to a number of sample fire monitoring images; Initializing the model parameters of the initial fire identification model to obtain several groups of initial model parameters; According to several groups of initial model parameters, an initial particle population is constructed, wherein the initial particle population includes several initial particle individuals, each initial particle individual is configured with an initial position vector and an initial velocity vector, and the initial position vector of each initial particle individual corresponds to a group of initial model parameters; Initialize the number of iterations t, obtain the particle population at the t-th iteration, and construct an initial fire recognition model corresponding to each particle individual at the t-th iteration based on the position vector of each particle individual in the particle population at the t-th iteration, wherein when t is 1, the particle population at the t-th iteration is the initial particle population; Using the training set, training each initial fire identification model, and using the model output of each initial fire identification model to calculate the fitness of each particle individual at the t-th iteration, wherein the fitness of any particle individual is used to characterize the classification accuracy of the initial fire identification model corresponding to the particle individual; Using the fitness of each individual particle at the t-th iteration, the global extreme value and the individual extreme value of each individual particle are updated to obtain the updated global extreme value and the updated individual extreme value corresponding to each individual particle, wherein the global extreme value is the position vector corresponding to the individual particle with the largest fitness during the 1st to t-1th iterations; Determine whether an iteration stop condition is met, wherein the iteration stop condition is that the fitness of the individual particle corresponding to the updated global extreme value is greater than or equal to the fitness threshold, or t is greater than or equal to the maximum number of iterations; If not, the fitness of the population is calculated based on the fitness of each individual particle at the tth iteration; Calculating the mutation probability using the updated global extreme value and the population fitness; Generate a mutation random number, and determine whether the mutation random number is less than the mutation probability, wherein the value interval of the mutation random number is [0,1]; If yes, then a mutation operation is performed on the updated global extreme value to obtain a mutated global extreme value at the t-th iteration; Using the mutation global extreme value, the particle population at the g-th iteration is updated to obtain a new particle population; Increment t by 1 and update the particle population at the t-th iteration to the new mutant population at the t-1-th iteration, and re-construct the initial fire recognition model corresponding to each particle individual at the t-th iteration based on the position vector of each particle individual in the particle population at the t-th iteration, until the iteration stop condition is met, so as to determine the optimal model parameters of the initial fire recognition model based on the updated global extreme value when the iteration stop condition is met; Based on the optimal model parameters, the fire identification model is constructed.

9. An intelligent fire identification and early warning system based on deep learning, characterized in that: include: An acquisition unit, used to acquire a real-time fire monitoring image of a target area and a calibrated monitoring image when there is no fire; An image processing unit, used for sequentially performing Gaussian noise reduction processing and quality enhancement processing on the real-time fire monitoring image to obtain an enhanced fire monitoring image; a contour extraction unit, configured to extract a flame region image from the enhanced fire monitoring image and a foreground image from the real-time fire monitoring image, and fuse the foreground image with the flame region image to generate a fused flame region image; A contour extraction unit is used to extract the edge of the fused flame area image to obtain an initial fire contour image; The contour extraction unit is further used to generate a non-fire scene contour image using the calibrated monitoring image, and to generate a fire contour image based on the initial fire contour image and the non-fire scene contour image; A fire identification unit, used for performing feature extraction processing on the fire contour image to obtain fire feature data; The early warning unit is used to input the fire characteristic data into the fire recognition model to obtain the fire recognition result of the target area, and send fire alarm information to the on-duty end when the fire recognition result is a fire.

10. An electronic device, characterized in that: include: A memory, a processor and a transceiver that are sequentially connected in communication, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the deep learning-based intelligent fire identification and early warning method as described in any one of claims 1 to 8.

Images