A method and system for intelligent identification of kitchen initial fire types

Abstract

The application discloses a kind of kitchen initial fire type intelligent identification method and system, the method comprises: obtaining the multi-modal environment data in kitchen;Multi-modal environment data includes visible light image data, infrared thermal imaging data and gas concentration data;The multi-modal environment data is respectively preprocessed;Fire feature extraction is carried out to the multi-modal environment data after pre-processing, and each multi-modal initial fire feature is obtained;Multi-modal initial fire feature includes visible light image feature, infrared thermal imaging feature and gas concentration feature;Multi-modal initial fire feature is input into multi-modal feature fusion's deep learning model, and the determination result of the initial fire type of kitchen is output.The kitchen initial fire type intelligent identification method and system of the application can quickly and accurately identify the type of initial fire in the kitchen under the complex environment of kitchen, and provide the basis for subsequent targeted fire extinguishing measures.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of fire-fighting automation, and particularly relates to an intelligent identification method and system for initial fire types in a kitchen. BACKGROUND

[0002] Kitchen fires, as a high-frequency and high-hazard fire type, account for one-fifth of the total number of civil fires and cause huge economic losses every year. Unlike ordinary building fires, kitchen fires have the characteristics of complex burning materials, rapid fire spread, and high rekindling rate. In particular, commercial kitchens in catering places have high fire load density due to the concentration of open fire equipment and serious oil smoke deposition, and secondary injury caused by oil splashing during firefighting is prone to occur.

[0003] For kitchen fire automatic identification algorithms, normal cooking in the kitchen can easily cause confusion with kitchen fires, causing fire identification interference, and it is not easy to accurately and quickly identify kitchen fires. At the same time, since there can be different types of fires in the kitchen, different types of fires require different treatment methods, so accurate identification of kitchen fire types helps to more accurately match the corresponding fire extinguishing method. In addition, due to the characteristics of the kitchen environment, it is extremely easy to cause combustion and explosion, so the identification of the initial fire is particularly important.

[0004] The current automatic fire extinguishing system that can realize fire type identification mainly collects abnormal information through sensors such as smoke sensors and gas sensors, and triggers an alarm to extinguish the fire. For example, the fire identification method and device disclosed in patent CN117437741A uses a gas sensor to collect gas information in fire smoke to identify the type of fire. However, due to the significant particularity of kitchen fires compared to other types of fires, especially oil fires, which often develop at a speed of seconds, combustion and explosion are extremely easy to occur; and the method of gas or smoke analysis often requires a larger fire, obviously, this method is not suitable for identifying the initial fire type in a kitchen. Therefore, a method is needed that can quickly identify the type of fire when an initial fire occurs in a kitchen so that corresponding fire extinguishing measures can be taken. SUMMARY

[0005] In order to solve the above problems in the prior art, the present application provides an intelligent identification method and system for initial fire types in a kitchen. The technical problem to be solved by the present application is solved by the following technical scheme:

[0006] In a first aspect, the present application provides an intelligent identification method for initial fire types in a kitchen, comprising the steps of:

[0007] acquiring multi-modal environment data in the kitchen; the multi-modal environment data includes visible light image data, infrared thermal imaging data, and gas concentration data;

[0008] respectively;

[0009] extracting fire features from the pre-processed multi-modal environment data to obtain multi-modal initial fire features; the multi-modal initial fire features include visible light image features, infrared thermal imaging features and gas concentration features;

[0010] inputting the multi-modal initial fire features into a multi-modal feature fusion deep learning model to output a determination result of an initial fire type in the kitchen.

[0011] In an embodiment of the present application, in the step of pre-processing the multi-modal environment data, the method for pre-processing the visible light data comprises:

[0012] performing median filter processing on the visible light image data to eliminate salt and pepper noise in the visible light image data;

[0013] performing adaptive histogram equalization processing on the visible light image data after the salt and pepper noise is eliminated to enhance the contrast between the fire and the background;

[0014] converting the visible light image data after the contrast is enhanced from an RGB color space to an HSV color space to decouple color and brightness information;

[0015] performing component sub-band noise reduction processing on each channel of the HSV color space;

[0016] performing dynamic threshold segmentation on the HSV color space after the noise reduction processing to segment out the fire area.

[0017] In an embodiment of the present application, in the step of pre-processing the multi-modal environment data, the method for pre-processing the infrared thermal imaging data comprises:

[0018] based on a pre-calibrated blackbody radiation coefficient, converting the original signal of the infrared thermal imaging data into a calibrated temperature matrix;

[0019] performing adaptive Gaussian filtering on the calibrated temperature matrix to output a smoothed temperature matrix after adaptive Gaussian filtering processing;

[0020] performing Sobel operator edge detection on the smoothed temperature matrix after adaptive Gaussian filtering processing to output a gradient temperature matrix;

[0021] based on the smoothed temperature matrix after adaptive Gaussian filtering processing, extracting a candidate temperature point set with a temperature greater than a preset dynamic temperature threshold value according to the preset dynamic temperature threshold value;

[0022] Based on the gradient temperature matrix, the candidate temperature point set is further screened according to a preset gradient threshold value, temperature points with gradient values greater than the preset gradient threshold value are retained, and a high-temperature region is determined.

[0023] In an embodiment of the present application, in the preprocessing step of the multi-modal environment data, the method for preprocessing the gas concentration data comprises:

[0024] The time series data collected by the methane sensor is filtered to obtain a smooth methane concentration sequence;

[0025] The smoke sensor data is subjected to sliding baseline correction to obtain a relative smoke concentration sequence;

[0026] According to a preset methane concentration threshold value and a preset relative smoke concentration threshold value, the smooth methane concentration sequence and the relative smoke concentration sequence are detected respectively to generate a gas abnormal event marker and a concentration change rate curve.

[0027] In an embodiment of the present application, the component sub-band noise reduction processing of the HSV color space channels comprises:

[0028] The H channel image of the HSV color space is subjected to noise reduction processing;

[0029] The S channel image and the V channel image of the HSV color space are subjected to noise reduction processing;

[0030] The H channel image, the V channel image and the S channel image subjected to noise reduction processing are merged.

[0031] In an embodiment of the present application, the noise reduction processing of the H channel image of the HSV color space comprises:

[0032] For each image block of the H channel image, a transformation from a spatial domain to a frequency domain is performed to obtain a frequency domain transformation matrix;

[0033] The gradient values of N directions of the frequency domain transformation matrix are calculated, wherein N is greater than or equal to 4;

[0034] The maximum gradient value and the minimum gradient value among the calculated gradient values of the N directions are determined;

[0035] According to the size relationship between the maximum gradient value, the minimum gradient value and a plurality of threshold values, the number of tailing coefficients to be removed is determined;

[0036] The tailing coefficient removal is performed on the transformation matrix according to the number of tailing coefficients to be removed;

[0037] The frequency domain matrix after removing the trailing coefficient is transformed from the frequency domain to the spatial domain to obtain the H channel image after noise reduction.

[0038] In an embodiment of the present application, the noise reduction processing on the V channel image and the S channel image of the HSV color space comprises:

[0039] The V channel image and the S channel image are filtered and processed by a saturation and brightness joint bilateral filtering method, wherein the formula of the saturation and brightness joint bilateral filtering is as follows:

[0040]

[0041] In the formula, is a bilateral filtering weight, is a spatial domain weight, is a value domain weight; k , l is the coordinate of the current pixel to be processed, i , j is the coordinate of the reference point pixel, σ d is a spatial domain smoothing intensity parameter; Is k , l is the saturation value at the coordinate k , l is the saturation value at the reference point i , j is the brightness value at the coordinate i , j is the brightness value at the reference point k , l , k , l , i , j , i , j σ r is a value domain sensitivity parameter.

[0042] In an embodiment of the present application, the gradient values of N directions of the frequency domain transformation matrix are calculated, wherein when N=4, the formula used is as follows:

[0043]

[0044] In the formula, represents the gradient value of the right horizontal direction of the frequency domain matrix, represents the gradient value of the lower vertical direction of the frequency domain matrix, represents the gradient value of the lower left of the frequency domain matrix,​ This represents the gradient value in the lower right corner. Represents the first in the frequency domain matrix p The elements in row q and column q are used to form a frequency domain matrix of size m×n.

[0045] In one embodiment of the present invention, the step of extracting fire features from the preprocessed multimodal environmental data to obtain initial fire features for each multimodal mode includes:

[0046] The spatial feature map is extracted from the preprocessed visible light image data using a lightweight CNN network to obtain the visible light image features.

[0047] The spatiotemporal thermal features are extracted from the preprocessed infrared thermal imaging data using a spatiotemporal convolutional network to obtain the infrared thermal imaging features.

[0048] The gas concentration features are obtained by extracting time-series dynamic features from the preprocessed gas concentration data using bidirectional LSTM.

[0049] Secondly, embodiments of the present invention provide an intelligent identification system for the initial fire type in a kitchen, which applies the above-described method and includes:

[0050] A multi-sensor fusion sensing module is used to acquire multimodal environmental data in the kitchen; the multimodal environmental data includes visible light image data, infrared thermal imaging data, and gas concentration data;

[0051] A multimodal data preprocessing module is used to preprocess the multimodal environmental data respectively;

[0052] The fire feature extraction module is used to extract fire features from the preprocessed multimodal environmental data to obtain initial fire features for each multimodal mode; the initial fire features for each multimodal mode include visible light image features, infrared thermal imaging features, and gas concentration features;

[0053] The fire type identification module is used to input the multimodal initial fire features into a deep learning model that fuses multimodal features, and output the determination result of the initial fire type in the kitchen.

[0054] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0055] The application provides a kitchen initial fire type intelligent identification method and system, which fuses three types of heterogeneous data of visible light images, infrared thermal images and gas concentration at the same time, makes up for the limitation of a single mode, improves the comprehensiveness and accuracy of fire type identification, significantly improves the input data quality through multi-stage preprocessing optimization, greatly improves the signal-to-noise ratio and stability of the preprocessed data, and lays a reliable foundation for subsequent feature extraction, improves the accuracy of initial fire type identification in the complex kitchen environment through noise reduction processing of the HSV channel vector images, and provides key technical support for fire safety in home, catering places and other scenes. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a flowchart of a kitchen initial fire type intelligent identification method provided by an embodiment of the application.

[0057] Figure 2 is a flowchart of pre-processing of multi-modal environment data provided by an embodiment of the application.

[0058] Figure 3 is a flowchart of pre-processing of visible light data provided by an embodiment of the application.

[0059] Figure 4 is a flowchart of pre-processing of infrared thermal imaging data provided by an embodiment of the application.

[0060] Figure 5 is a module block diagram of a kitchen initial fire type intelligent identification system provided by an embodiment of the application. DETAILED DESCRIPTION

[0061] The application will be further described in detail below in combination with specific embodiments, but the embodiments of the application are not limited thereto.

[0062] Please refer to Figure 1 , Figure 1 is a flowchart of a kitchen initial fire type intelligent identification method provided by an embodiment of the application, and the kitchen initial fire type intelligent identification method provided by the embodiment of the application comprises the following steps:

[0063] S100, acquiring multi-modal environment data in a kitchen; the multi-modal environment data comprises visible light image data, infrared thermal imaging data and gas concentration data.

[0064] Due to the complex kitchen environment, the initial fire characteristics of the kitchen are affected by multiple characteristics such as flame color, temperature, smoke concentration and oil droplet splashing, and single image recognition is easily disturbed by environmental factors such as steam and light changes, therefore, the embodiment of the application fuses multiple types of sensor data to improve the robustness of initial fire identification.

[0065] The multi-sensor fusion perception module adopted by the embodiment of the present application comprises a visible light camera, an infrared thermal imaging camera and a gas sensor. The visible light camera is used to capture flame color and shape. The infrared thermal imaging camera detects high temperature areas. Since the surface temperature of oil and fat during initial combustion can reach above 300 DEG C, an infrared thermal imaging camera with a temperature measurement range of -20 DEG C to 1500 DEG C can be selected. The gas sensor adopted by the embodiment of the present application can be a catalytic combustion type methane sensor and a semiconductor type smoke sensor, which is used for a gas leakage fire and can identify the pre-combustion stage caused by gas leakage. The preferred catalytic combustion type methane sensor has a detection range of 0-100% LEL and an accuracy of ±5% LEL, and can be deployed in high-risk areas of gas leakage such as near the gas pipeline interface and below the gas stove. The catalytic element and the oxidation reaction of methane gas produce an electrical signal, which monitors the methane concentration in the environment in real time. The preferred semiconductor smoke sensor has a detection particle concentration of 0-20 mg / m³ and a response time of <10 s, and can be deployed at key points on the smoke diffusion path such as the central ceiling of the kitchen and the exhaust outlet of the extractor hood. The semiconductor material changes in resistance to perceive the adsorption of smoke particles, and outputs an electrical signal proportional to the smoke concentration.

[0066] Correspondingly, the data collected by the visible light camera in the embodiment of the present application is visible light image data, the data collected by the infrared thermal imaging camera is infrared thermal imaging data, and the data collected by the gas sensor is gas concentration data. All sensor data is transmitted synchronously to the edge computing terminal through the CAN bus, and after time stamp alignment, the next step of noise reduction preprocessing is performed.

[0067] S200, pre-process the multi-modal environment data respectively.

[0068] Due to the complex characteristics of the kitchen environment, the collected environment data is easily affected by environmental factors. For example, the visible light image can be affected by salt and pepper noise and light changes, the infrared thermal imaging can be affected by sensor noise and environmental temperature interference, and the gas concentration data can be affected by sensor drift and background gas interference. The purpose of preprocessing is to improve the data quality so that the subsequent feature extraction and model training are more effective.

[0069] Please refer to Figure 2 , Figure 2 is a flow chart for pre-processing multi-modal environment data respectively provided by the embodiment of the present application. In the embodiment of the present application, the visible light image data, the infrared thermal imaging data and the gas concentration data need to be pre-processed respectively, and the pre-processing method needs to be processed by different methods according to the characteristics of each data. The preferred method is as follows:

[0070] S210, pre-process the visible light data.

[0071] The image in the kitchen environment can be affected by steam, light reflection, etc., resulting in salt and pepper noise; if not denoised, subsequent image segmentation may appear artifacts, such as noise points being incorrectly enhanced or segmented. Therefore, preprocessing the visible light data can enhance the contrast between the flame region and the background, eliminate noise, and extract visual features such as color, shape, and motion trajectory of the flame.

[0072] See Figure 3 , Figure 3 is a flowchart for preprocessing visible light data provided by an embodiment of the present application. Specifically, the method for preprocessing visible light data comprises:

[0073] S211, the visible light image data is subjected to median filtering processing to eliminate salt and pepper noise in the visible light image data.

[0074] Salt and pepper noise is characterized by randomly appearing black and white noise points, usually caused by sensor failure, transmission error or environmental interference. This kind of noise is characterized by random distribution in the image, large amplitude, and can seriously damage the details of the image, such as blurring the edges of the flame or covering the key features. Although traditional Gaussian filtering can smooth noise, it will blur the edges, while median filtering is more effective in preserving edges.

[0075] Median filtering is a nonlinear filtering method, which uses the median value in the neighborhood of the pixel to replace the current pixel value. For a two-dimensional image, a sliding window is usually used, all pixel values in the window are sorted, and the middle value is taken as the new value of the current pixel; the sliding window can be selected as 3x3 or 5x5, for example, a sliding window smaller than 3x3 may not be able to sufficiently suppress noise, and larger than 7x7 may cause edge blurring. This can effectively suppress salt and pepper noise, because the value of a noise point is usually significantly different from the surrounding pixels, and after sorting it will be excluded from the median. The pixel value of salt and pepper noise is either very high or very low, and the difference from the surrounding normal pixels is significant. In the sorted neighborhood, these extreme values will be at both ends, while the median value usually comes from normal pixels, so replacing it can effectively eliminate noise while preserving the edges and details of the image.

[0076] This step eliminates salt and pepper noise in the visible light image data by median filtering, the edges of the flame are clearer, the color distribution is more uniform, which is conducive to extracting features such as shape and color, and provides high-quality input for subsequent feature extraction.

[0077] S212, the visible light image data after salt and pepper noise is eliminated is subjected to adaptive histogram equalization processing to enhance the contrast between the flame and the background.

[0078] After removing salt-and-pepper noise, the image may still exhibit uneven brightness, such as a darker flame area and a brighter background. Histogram equalization (HE) enhances contrast by adjusting the grayscale distribution of the image to make the grayscale values ​​more uniform. However, traditional HE is global and may amplify noise or lose details. Adaptive HE divides the image into small regions, performs HE on each region individually, and then uses bilinear interpolation to eliminate artificial artifacts at region boundaries, thereby enhancing local contrast while avoiding global over-enhancement.

[0079] Because flames typically have high brightness and a specific color, while the background may be darker, adaptive histogram equalization can enhance the local contrast of the flame region, making the edges and details of the flame more prominent and the distinction from the background more obvious. For example, specific processing steps may include image segmentation, calculating the histogram of each segment, performing histogram equalization, and merging the results using bilinear interpolation.

[0080] This step performs adaptive histogram equalization, which locally and adaptively adjusts the grayscale distribution to effectively enhance the local contrast between the flame and the background in the visible light image after eliminating salt-and-pepper noise, while preserving the details of the flame.

[0081] S213. Convert the enhanced contrast visible light image data from the RGB color space to the HSV color space to decouple color and brightness information.

[0082] The RGB color space is based on an additive color model using the three primary colors of red, green, and blue. The color of each pixel is determined by three components: R, G, and B, typically ranging from 0 to 255. The HSV (Hue, Saturation, Value) color space, on the other hand, decomposes color into hue, saturation, and value, which better aligns with human color perception. Here, H represents the color type (e.g., red, green, blue), S represents the color purity (0 for grayscale, 1 for pure color), and V represents the color brightness (0 for black, 1 for white). RGB has certain limitations; for example, brightness and color are coupled together, meaning adjusting brightness may alter the perceived color. HSV separates these three attributes, facilitating the individual processing of color and brightness. Therefore, converting visible light image data from the RGB color space to the HSV color space is essential for feature extraction in fire identification.

[0083] Visible light image data can be converted from the RGB color space to the HSV color space using methods such as geometric derivation, coordinate transformation, and segmentation definition.

[0084] S214. Perform component frequency band noise reduction processing on each channel of the HSV color space.

[0085] In fire identification, after the visible light image is converted from RGB to HSV, the H (hue), S (saturation), and V (value) channels carry key semantic information of the flame, such as "color type", "purity", and "brightness". However, each channel is susceptible to different types of noise, such as random hue shift in the H channel, saturation fluctuation in the S channel, and brightness fluctuation in the V channel. Therefore, component sub-band noise reduction processing is needed to specifically suppress noise while preserving the core features of each channel.

[0086] H (hue) represents the "type" of color, such as orange-red corresponding to H ≈ 20°~40°, and its value range is circular (0°~360°). Noise mainly manifests as random hue shift, which directly changes the color category of the flame, such as misjudging the flame as a background color. S (saturation) represents the "purity" of color (0 for gray, 1 for pure color), and its value range is [0, 1]. Noise mainly manifests as random amplitude fluctuation. S value fluctuation reduces the "vividness" of the flame, and even causes confusion between the flame and the background. V (value) represents the "brightness" of color (0 for black, 1 for white), and its value range is [0, 1]. Noise mainly manifests as brightness fluctuation. V value fluctuation blurs the "intensity" difference of the flame, such as the low brightness of the smoldering stage and the high brightness of the intense burning stage.

[0087] In the embodiments of the present application, for the identification needs of the initial fire type in the kitchen, the H channel image is preferably processed separately, and the S channel image and the V channel image can be processed jointly, and the processing methods of the two are different. The purpose of this design is that H, as the hue, is more about distinguishing the color type, and may need to be processed separately to avoid color shift. S and V are usually related to the "vividness" and "brightness" of color, and can be jointly processed when describing the same color feature of high saturation and high brightness of the flame. In addition, since H directly determines the "category" of color, its accuracy directly affects the semantic understanding of the image. The fluctuation of S only reduces the color vividness, but does not change the color category. The fluctuation of V mainly affects the brightness level, but does not change the color category or vividness. Therefore, different denoising strengths can be used for component sub-band noise reduction processing.

[0088] Specifically, the component sub-band noise reduction processing is performed on each channel of the HSV color space, including:

[0089] S2141, performing noise reduction processing on the H channel image of the HSV color space;

[0090] S2142, performing noise reduction processing on the S channel image and the V channel image of the HSV color space;

[0091] S2143, merging the H channel image, the V channel image, and the S channel image after noise reduction processing.

[0092] In step S2141, the H channel image of the HSV color space is subjected to noise reduction processing, including:

[0093] S21411, for each image block of the H channel image, a transformation from spatial domain to frequency domain is performed to obtain a frequency domain transformation matrix.

[0094] In the H channel image processing of fire identification, the image is blocked and transformed from spatial domain to frequency domain, the core purpose is to extract frequency domain features such as edge, texture and periodicity from the spatial distribution of local pixels, and the local frequency features can be extracted specifically, while the calculation complexity is reduced, so that the "color category" of the flame and the noise interference are separated more efficiently.

[0095] The transformation from spatial domain to frequency domain can be achieved by using a 53 wavelet transform method, and the formula of the 53 wavelet transform method is:

[0096]

[0097] Wherein, x is the spatial domain pixel value, that is, the pixel value in the unit block; c is the frequency domain value at the odd position, d is the frequency domain value at the even position, the frequency domain value is the value after frequency domain transformation; c is the AC coefficient, d is the DC coefficient; n is a natural number, indicating the pixel position.

[0098] Compared with the commonly used DCT (Discrete Cosine Transform) method in the prior art, the 53 wavelet transform method has the characteristics of high transformation efficiency and easy hardware implementation.

[0099] S21412, the gradient values of N directions of the frequency domain transformation matrix are calculated, wherein N≥4.

[0100] The method of calculating the gradient values of N directions of the frequency domain transformation matrix is realized by calculating the sum of absolute values of the difference values of adjacent pixels. The common directions can include horizontal, vertical, two diagonal directions, that is, the case of N=4 corresponding to this step. But N can be greater than 4, such as 8 directions, or finer angle division.

[0101] The embodiment of the application takes N=4 as an example for illustration, and the formula for calculating the gradient values of the frequency domain transformation matrix in four directions is as follows:

[0102]

[0103] In the formula, represents the gradient value of the right horizontal direction of the frequency domain matrix, represents the gradient value of the vertical direction of the lower side of the frequency domain matrix, represents the gradient value of the left lower side of the frequency domain matrix, represents the gradient value of the lower right, represents the element in the i-th row and q-th column in the frequency domain matrix, and the size of the frequency domain matrix is m x n. p

[0104] It should be noted that the edge and texture of the flame usually present multi-directional characteristics, such as: vertically rising flame edge, horizontally diffusing flame boundary, obliquely irregular edge, etc., and a single direction gradient cannot completely capture these characteristics, which may lead to edge missing or misjudgment. Therefore, the embodiment of the present application calculates the gradient values in N≥4 directions to cover the common directions of the flame edge (such as 0°, 45°, 90°, 135°, etc.), so as to ensure the comprehensiveness of the feature description.

[0105] S21413, determining the maximum gradient value and the minimum gradient value in the N direction gradient values obtained.

[0106] The multi-directional gradient value reflects the change rate of each position in the frequency domain transformation matrix in the spatial domain, however, due to noise interference or edge tailing, part of the gradient value may deviate from the normal range of the true feature; therefore, by determining the global maximum and minimum gradient values, the “reasonable interval” of the gradient value can be determined, which provides a reference for the identification of the tailing coefficient in the subsequent.

[0107] S21414, determining the number of tailing coefficients to be removed according to the size relationship of the maximum gradient value, the minimum gradient value and the plurality of threshold values.

[0108] The tailing coefficient refers to the abnormal coefficient in the frequency domain transformation matrix caused by noise or edge tailing, which will interfere with the subsequent feature extraction.

[0109] In the embodiment of the present application, the plurality of threshold values used when determining the number of tailing coefficients to be removed can be formulated according to the prior noise model of “brightness x color x noise” model, according to the noise model, a large number of sample image blocks can be obtained, and by carrying out the denoising experiment on these sample image blocks, the plurality of threshold values required by the embodiment of the present application can be determined.

[0110] Specifically, the number of tailing coefficients to be removed is determined according to the following rules:

[0111] When a < A, it means that the image block is a high-frequency noise region, and a small number of tailing coefficients need to be removed; the number of tailing coefficients to be removed is the lower limit value, and a = 0 at this time. G max T min When a = A, it means that it is a flat or low-frequency noise region, and a large number of tailing coefficients need to be removed; the number of tailing coefficients to be removed is the upper limit value, and a = A at this time.

[0112] When A < a < B, it means that the image block is a medium-frequency noise region, and a medium number of tailing coefficients need to be removed; the number of tailing coefficients to be removed is the middle limit value, and a = B at this time. G min T max ​​​When the value of the gradient value is greater than the value of the maximum gradient value, it is indicated that the region is an indistinguishable region, and the tailing coefficient needs to be removed, and at this time, a = 0.

[0113] When G min G max And G min T max And G max T min The number of tailing coefficients to be removed satisfies: 0 < a < A.

[0114] And the specific value of a is determined by the position of the overlapping interval:

[0115] The closer the overlapping interval is to T min (i.e., the left end point of [ G min , G max ]∩[ T min , T max] is closer to T min ), the greater a is; the closer the overlapping interval is to T max (i.e., the right end point of [ G min , G max ]∩[ T min , T max ] is closer to T max ), the smaller a is.

[0116] Wherein, G max is the maximum gradient value of the multi-direction gradient value; G min is the minimum gradient value of the multi-direction gradient value; T min is the minimum threshold value in the preset threshold value set; T max is the maximum threshold value in the preset threshold value set; A is the maximum number of tailing coefficients to be removed; a is the actual number of tailing coefficients to be removed (0 ≤ a ≤ A).

[0117] S21415, tailing coefficient removal is performed on the transformation matrix according to the number of tailing coefficients to be removed.

[0118] ​​​After determining the number of tailing coefficients to be removed, these coefficients are removed from the frequency domain transformation matrix by zeroing, smoothing or replacing, etc. operations to restore the true characteristics of the matrix.

[0119] S21416, the frequency domain matrix after removing the tailing coefficients is transformed from the frequency domain to the spatial domain to obtain the H channel image after noise reduction.

[0120] The embodiment of the present application determines the tailing coefficients by formulating rules, and removes the tailing coefficients, which can identify and suppress abnormal coefficients caused by noise, edge tailing or artifacts in the frequency domain, so as to retain the key frequency characteristics of the flame.

[0121] In the above step S2142, the S channel image and the V channel image of the HSV color space are subjected to noise reduction processing, which includes:

[0122] The V channel image and the S channel image are subjected to filtering and noise reduction processing by using saturation and brightness joint bilateral filtering, wherein the formula of the saturation and brightness joint bilateral filtering is as follows:

[0123]

[0124] In the formula, is a bilateral filtering weight, is a spatial domain weight, is a value domain weight; is the coordinate of the current pixel to be processed, and is the coordinate of the reference point pixel, σ d is a spatial domain smoothing intensity parameter; Is( k , l ) is the saturation value at the coordinate ( k , l ), Is( i , j ) is the saturation value at the reference point ( i , j ), Iv( k , l ) is the brightness value at the coordinate ( k , l ), and Iv( i , j ) is the brightness value at the reference point ( i , j ), σ r is a value domain sensitivity parameter.

[0125] The saturation and lightness combined bilateral filtering method adopted by the embodiment of the present application performs adaptive smoothing and feature reservation on multi-modal fire features through double constraints of a spatial domain and a value domain, solves the problems of multi-modal noise heterogeneity, edge blurring and information redundancy, and finally provides high-quality, low-noise and multi-dimensional complementary feature input for a subsequent deep learning model, thereby significantly improving the accuracy and robustness of fire identification.

[0126] After the noise reduction processing is performed on the channel images, the H channel image, the V channel image and the S channel image after the noise reduction processing are merged.

[0127] S215, performing dynamic threshold segmentation on the HSV color space after the noise reduction processing to segment out a flame region.

[0128] The HSV color space after the noise reduction processing has decoupled color, purity and brightness information, but the HSV values of the flame region may still fluctuate due to the following factors: light change: the strength change of kitchen light and natural light will change the lightness of the flame; background interference: steam, smoke or reflection may introduce pseudo-features similar to the color of the flame; flame dynamics: the brightness of the flame in the smoldering stage is low, and the brightness is high when burning fiercely, and a single global threshold cannot cover all states. The traditional global threshold is easy to misjudge these interference regions as flames or miss the real flames with low / high brightness. The dynamic threshold segmentation can adapt to the above changes by adjusting the threshold range according to the HSV statistical characteristics of the local region, and improve the accuracy of flame segmentation.

[0129] The essence of dynamic threshold segmentation is to calculate the upper and lower limits of the threshold for each pixel or neighborhood based on the statistical characteristics (such as mean, variance, histogram distribution) of the local region of the image, so that the threshold adjusts with the image content.

[0130] The embodiment of the present application performs dynamic threshold segmentation on the HSV color space after the noise reduction, accurately locates the flame region by adaptively adjusting the threshold range of each channel, and solves the misjudgment problem caused by light and background interference of the traditional global threshold. This step provides high-quality input for subsequent flame feature extraction and fire identification model.

[0131] S220, pre-processing the infrared thermal imaging data, please refer to Figure 4 , Figure 4 is a flowchart for pre-processing the infrared thermal imaging data provided by the embodiment of the present application. The specific method comprises:

[0132] S221, converting the original signal of the infrared thermal imaging data into a calibrated temperature matrix based on the pre-calibrated blackbody radiation coefficient.

[0133] In this step, the pre-calibrated blackbody radiation coefficient includes ambient temperature, infrared thermal imaging sensor response coefficient, infrared thermal imaging sensor response coefficient, and ambient temperature compensation coefficient. Combining the physical characteristics of the infrared thermal imaging sensor with the calibration data, the voltage value is mapped to the true temperature using the following formula, outputting the calibrated temperature matrix. T cal.

[0134]

[0135] Among them, R raw R is the original infrared signal matrix. raw (i,j) is the voltage value of the (i,j)th pixel of the infrared thermal imaging sensor; T env For ambient temperature, k The response coefficient of the infrared thermal imaging sensor is predetermined by a blackbody radiation calibration experiment. α The ambient temperature compensation coefficient is obtained by fitting calibration data under different ambient temperatures; β The zero-point correction factor is determined through blackbody source zero-point calibration; and k =0.12℃ / mV, α =0.003 / ℃, β= .

[0136] S222. Adaptive Gaussian filtering is used to process the calibration temperature matrix, and the smoothed temperature matrix after adaptive Gaussian filtering is output.

[0137] To suppress random noise such as sensor thermal noise and electromagnetic interference in infrared images and preserve temperature details in the high-temperature region of the flame, an adaptive Gaussian filter is needed to process the calibration temperature matrix. The adaptive Gaussian filter dynamically adjusts the filter kernel size based on the degree of local temperature fluctuations to avoid over-smoothing edge details in high-temperature regions. The specific implementation method is as follows:

[0138] (1) Define the size of the local window W The initial value is 5×5;

[0139] (2) Calculate the standard deviation of temperature within the window. σ local ;

[0140] (3) If σ local <1℃ (low noise region), using a 3×3 Gaussian kernel ( σ =1.0) Smoothing;

[0141] (4) If σ local ≥1℃ (high-noise areas, such as near range hoods), use a 7×7 Gaussian core (σ =1.5)smooth.

[0142] This step outputs the smoothed temperature matrix. By processing the calibration temperature matrix using adaptive Gaussian filtering, noise is suppressed and the edges of high-temperature regions are clarified.

[0143] S223, using Sobel operator edge detection method, processing the smoothed temperature matrix after adaptive Gaussian filtering, output gradient temperature matrix.

[0144] Using Sobel operator edge detection method, the first-order partial derivative of the temperature matrix in the x and y direction is calculated respectively, and the gradient amplitude matrix is synthesized:

[0145]

[0146] Wherein, the Sobel operator template is:

[0147]

[0148] is the gradient amplitude matrix, which represents the temperature change rate at pixel point i , j ); G x is the Sobel convolution kernel in the x direction, G y is the Sobel convolution kernel in the y direction, T ( i , j ) is the gray value of the image at pixel point i , j ).

[0149] This step outputs the temperature gradient matrix. Through this step, the noise in the infrared thermal imaging data can be effectively suppressed.

[0150] S224, based on the smoothed temperature matrix after adaptive Gaussian filtering, according to the preset dynamic temperature threshold, extracting the candidate temperature point set whose temperature is greater than the preset dynamic temperature threshold.

[0151] This step needs to set the dynamic temperature threshold T th (such as T th = ambient temperature + 100℃), extract all pixel points whose temperature is greater than T th from the smoothed temperature matrix, form a candidate high-temperature point set, denoted as S temp .

[0152] It should be noted that although the smoothed temperature matrix is output as a gradient temperature matrix in step S223, the smoothed temperature matrix will not be discarded because the temperature gradient matrix is generated, and they will be stored as independent data structures. Therefore, step S224 continues to process on the basis of step S223, and both matrices can be accessed simultaneously.

[0153] S225, based on the gradient temperature matrix, further filtering the candidate temperature point set according to the preset gradient threshold, retaining the temperature points with gradient values greater than the preset gradient threshold, and determining the high temperature region.

[0154] The candidate high temperature point set S temp Further filtering, only retaining points with gradient values > 3℃ / pixel (For example = 3℃ / pixel px ) to form a preliminary flame region set and determine the high temperature region.

[0155] This step is a key link in the accurate identification process of the high temperature region. By introducing gradient information, secondary screening of candidate temperature points is realized, which significantly improves the accuracy of edge detection and region segmentation. By combining temperature amplitude information and temperature change rate information, more accurate and robust identification of the high temperature region is realized.

[0156] S230, pre-processing the gas concentration data, the specific method comprising:

[0157] S231, filtering the time series data collected by the methane sensor to obtain a smoothed methane concentration sequence.

[0158] The time series data collected by the methane sensor is often disturbed by sensor noise, environmental interference and other noise. These noises will cause the original data to present "burr" or "jitter", affecting the accuracy of subsequent threshold detection. By filtering algorithm to suppress noise, the true trend of methane concentration is retained, and a smooth and continuous concentration sequence is obtained.

[0159] The filtering method can use moving average filtering, median filtering or Kalman filtering method. The noise peak value of the smoothed methane concentration sequence obtained after filtering is suppressed, and the concentration change trend is clearer, providing reliable "baseline data" for subsequent threshold detection.

[0160] S232, sliding baseline correction of the smoke sensor data to obtain a relative smoke concentration sequence.

[0161] The smoke sensor is susceptible to sensor aging, environmental adaptation deviation and other factors, causing the measured value to be affected by long-term drift. Direct use of the original measured value can cause "false positives", such as misjudging sensor drift as an increase in smoke concentration; or "false negatives", such as a real increase in smoke concentration being masked by drift. By calculating a dynamic baseline, the original measured value is converted into a "relative concentration", eliminating the effects of long-term drift.

[0162] The method of sliding baseline correction for smoke sensor data can use an exponential weighted sliding baseline method or an adaptive statistical baseline method, which will not be described here.

[0163] S233, according to the preset methane concentration threshold and the preset smoke relative concentration threshold, the smoothed methane concentration sequence and the relative smoke concentration sequence are detected respectively, and a gas anomaly event marker and a concentration change rate curve are generated.

[0164] The change rate is calculated by the concentration difference between adjacent time points, reflecting the change trend of the concentration over time; the methane / smoke concentration change rate curve is generated with time as the horizontal axis and the change rate as the vertical axis, directly displaying the development speed of the anomaly.

[0165] In the embodiments of the present application, the preprocessing of gas concentration data is a key link of the fire / gas leakage monitoring system. The preprocessing can eliminate noise interference, correct sensor drift, extract effective features, and provide a reliable data basis for subsequent anomaly detection.

[0166] S300, fire feature extraction is performed on the preprocessed multi-modal environment data to obtain multi-modal initial fire features; the multi-modal initial fire features include visible light image features, infrared thermal imaging features and gas concentration features.

[0167] The multi-modal initial fire features refer to the multi-dimensional physical features that appear simultaneously in the initial stage of the kitchen fire (within 0-10 seconds after the appearance of the flame) and can be captured by multi-sensor or image analysis technology, including but not limited to: visual features such as flame color and shape, thermal features such as temperature gradient and high-temperature region distribution, dynamic features such as flame area growth rate, oil droplet splashing speed and direction, material features such as oil droplet particle size and gas diffusion mode.

[0168] In order to identify the visible light image, infrared thermal imaging, gas concentration and other multi-modal initial fire features, the deep learning model needs to integrate the complementary information of different modalities through a multi-modal fusion architecture.

[0169] S310, for the preprocessed visible light image data, a lightweight CNN network is used to extract spatial feature maps to obtain visible light image features.

[0170] The spatial feature map is extracted by using a lightweight CNN, and static information such as flame shape and color distribution is reserved. The lightweight CNN can be MobileNetV3.

[0171] In S320, the spatio-temporal thermal feature is extracted from the preprocessed infrared thermal imaging data by using a spatio-temporal convolution network, and the infrared thermal imaging feature is obtained.

[0172] The spatio-temporal thermal feature is extracted by using a spatio-temporal convolution network, and the spatio-temporal mode of flame diffusion is captured. The spatio-temporal convolution network can be 3D ResNet-18.

[0173] In S330, the time sequence dynamic feature is extracted from the preprocessed gas concentration data by using a bidirectional LSTM, and the gas concentration feature is obtained.

[0174] The time sequence dynamic feature is extracted by using a bidirectional LSTM, and the mutation of the gas concentration, such as the sudden rise of the methane concentration, is modeled. The bidirectional LSTM can be BiLSTM.

[0175] In S400, the multi-modal initial fire feature is input into a multi-modal feature fusion deep learning model, and a determination result of the initial fire type in the kitchen is output.

[0176] In view of the heterogeneity of the multi-modal initial fire feature, a multi-modal feature fusion deep learning model is designed in the embodiment of the application, and through three modules of heterogeneous modal alignment, dynamic-static interaction and cross-modal attention, efficient fusion of the multi-modal feature and fire discrimination are realized.

[0177] Since the extracted visible light image feature, infrared thermal imaging feature and gas concentration feature have different modal feature dimensions and semantics, they need to be mapped to a unified dimensional semantic space through a cross-modal projection layer. Specifically:

[0178] The projection representation of the visible light image feature is:

[0179]

[0180] Wherein, F vis is the input visible light image feature vector; W vis is a visible light weight matrix; C is the channel number of the input feature; D is the target feature dimension after projection; b vis is a visible light bias vector; F vis is the projected visible light image feature.

[0181] The projection representation of the infrared thermal imaging feature is:

[0182]

[0183] wherein, GlobalAvgPool3D is a three-dimensional global average pooling operation; W ir is an infrared thermal imaging weight matrix; b ir is an infrared thermal imaging bias vector; F ir is an infrared thermal imaging feature tensor; F’ ir is a projected infrared thermal imaging feature.

[0184] The gas concentration feature projection is represented as:

[0185]

[0186] wherein, W gas is a weight parameter of linear transformation; GlobalAvgPool1D(F gas ) represents a result after one-dimensional global average pooling is performed on the gas concentration feature F gas ; b gas is a bias parameter of gas concentration; F’ gas is a projected gas concentration feature.

[0187] In view of the differences among the static visual feature (F’ vis ), the dynamic infrared thermal imaging feature (F’ ir ) and the dynamic gas concentration feature (F’ gas ), the embodiment of the present application dynamically adjusts the fusion weight by designing a gating mechanism, and the details are as follows:

[0188] The gating weight is generated by an MLP (multi-layer perceptron) g ∈[0,1] 3 , reflecting the dynamic degree of the current scene (such as g 1 controls the weight of the static feature F’ vis , g 2 / g 3 controls the weights of the dynamic features F’ ir and F’ gas respectively):

[0189]

[0190] wherein, σ is a sigmoid activation function; W g is a weight matrix; bg is a bias term; [⋅;⋅] represents splicing.

[0191] The static feature and the dynamic feature are weighted and summed according to the gating weight to obtain the fusion feature Ffusion :

[0192]

[0193] The formula dynamically models the input features through the MLP network, uses the sigmoid function to constrain the network output to reasonable weight values, and finally realizes the dynamic fusion of static and dynamic features. This mechanism enables the model to dynamically adjust feature weights according to the scene, improving the adaptability to complex environments.

[0194] In addition, in order to display the key association between modeling modalities, such as the correspondence between flame shape and high temperature area, a multi-head cross attention layer needs to be introduced:

[0195] (1) Take the fused feature F fusion as the query to extract global semantics;

[0196] (2) Use the projected features F′ vis ,F′ ir ,F′ gas of each modality as the key and value respectively;

[0197] (3) Calculate the association weight between the query and the key through multi-head attention (Multi-Head Attention), aggregate the values with weights, and enhance the complementary information between modalities:

[0198]

[0199] where Q is the query vector, and Q=F fusion Wq ; K is the key vector, and K=[ F′ vis ; F′ ir ; F′ gas ] W k ; V is the value vector, and V=[ F′ vis ; F′ ir ; F′ gas ] W v ; D is the feature dimension of Q, K, and V; W q , W k , W v ∈R D × D ; is the similarity between the query and the key.

[0200] The fused feature F fusion is subjected to global average pooling to obtain the global feature vector v global∈R D , input the full connection layer output classification probability:

[0201]

[0202] where y∈R C ( C represent the probability of each type, and the total number of fire types is N. W cls , b cls are the weights and biases of the classification layer; v global is the unified semantic feature vector after multi-modal feature fusion.

[0203] It should be noted that the feature extraction model and the deep learning network model involved in the embodiments of the present application are all models pre-trained using a large number of fire type samples. Therefore, by this step, the final output is the initial fire classification result of whether the kitchen is normal cooking or abnormal fire, and the specific initial fire type classification result of abnormal fire, such as oil fire, electrical short circuit fire, gas leakage fire, etc. In order to facilitate subsequent targeted fire extinguishing equipment or measures for fire extinguishing according to different fire types.

[0204] The intelligent identification method for the initial fire type in the kitchen provided by the embodiments of the present application simultaneously fuses three types of heterogeneous data of visible light images, infrared thermal imaging and gas concentration, which makes up for the limitations of single mode and improves the comprehensiveness of fire type identification; through a multi-stage preprocessing process, the data quality is optimized, the reliability of subsequent feature extraction is improved, and the signal-to-noise ratio and stability of the preprocessed data are greatly improved, laying a reliable foundation for subsequent feature extraction; through noise reduction processing of the HSV channel vector images respectively, the accuracy of initial fire type identification in the complex environment of the kitchen is improved.

[0205] In a second aspect, the present application provides an intelligent identification system for the initial fire type in the kitchen, please refer to Figure 5 , Figure 5 is a module block diagram of an intelligent identification system for the initial fire type in the kitchen provided by the embodiments of the present application. Corresponding to the method of the above-mentioned embodiments, the intelligent identification system for the initial fire type in the kitchen of the embodiments of the present application comprises:

[0206] A multi-sensor fusion perception module 1 is used to acquire multi-modal environment data in the kitchen; the multi-modal environment data includes visible light image data, infrared thermal imaging data and gas concentration data.

[0207] A multi-modal data preprocessing module 2 is used to preprocess the multi-modal environment data respectively.

[0208] A fire feature extraction module 3 is configured to extract fire features from the preprocessed multi-modal environment data to obtain multi-modal initial fire features, which include visible light image features, infrared thermal imaging features and gas concentration features.

[0209] A fire type identification module 4 is configured to input the multi-modal initial fire features into a multi-modal feature fusion deep learning model to output a determination result of an initial fire type in the kitchen.

[0210] The intelligent identification system of the initial fire type in the kitchen according to the embodiment of the present application, the specific implementation modes and functions of the modules correspond to the above method, and will not be described here.

[0211] Based on the multi-modal fusion kitchen initial fire type identification scheme provided by the embodiment of the present application, the effect verification is carried out through the following test.

[0212] 1. Hardware: Jetson AGX Xavier edge computing platform.

[0213] 2. Dataset: self-built kitchen fire multi-modal dataset, which contains various scene data such as oil pot fire, electric short circuit, gas leakage and normal cooking fire, wherein the normal cooking sample: 1,200 groups, containing oil smoke, steam, strong light interference scene, fire sample: 800 groups, initial flame height <10 cm, duration <10 s.

[0214] 3. Evaluation index:

[0215] Accuracy = (TP+TN) / (TP+TN+FP+FN)

[0216] False alarm rate = FP / (FP+TN)

[0217] Wherein, TP is the correct alarm of real fire, FP is the false alarm of normal scene.

[0218] By using the scheme of the present application, the initial fire identification rate is improved from 72% of the traditional smoke sensor to 96%; the average response time is shortened to 3.2 seconds (the traditional system needs more than 10 seconds); the fire extinguishing efficiency of gas leakage type fire is improved by 40%.

[0219] The above content is a further detailed description of the present application in combination with a specific preferred embodiment, and cannot be regarded as limiting the specific implementation of the present application to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be regarded as falling within the protection scope of the present application.

Claims

1. A method for intelligent identification of the initial fire type in a kitchen, characterized in that, Including the following steps: Acquire multimodal environmental data within the kitchen; the multimodal environmental data includes visible light image data, infrared thermal imaging data, and gas concentration data; The multimodal environmental data are preprocessed respectively; Fire features are extracted from the preprocessed multimodal environmental data to obtain initial fire features for each multimodal mode; the initial fire features include visible light image features, infrared thermal imaging features, and gas concentration features; The multimodal initial fire features are input into a deep learning model that fuses multimodal features, and the result of determining the type of initial fire in the kitchen is output. The method for preprocessing visible light data in the preprocessing step of the multimodal environmental data includes: The visible light image data is subjected to median filtering to eliminate salt-and-pepper noise in the visible light image data; Adaptive histogram equalization was performed on the visible light image data after removing salt and pepper noise to enhance the contrast between the flame and the background. The contrast-enhanced visible light image data is converted from the RGB color space to the HSV color space to decouple color and brightness information; Component-frequency band noise reduction is performed on each channel of the HSV color space; Dynamic threshold segmentation is performed on the noise-reduced HSV color space to segment out the flame region; The method involves inputting the multimodal initial fire features into a deep learning model that fuses multimodal features, and outputting a determination result of the initial fire type in the kitchen. Specific methods include: Visible light image feature projection is represented as: ； Among them, F vis The input is the feature vector of the visible light image; W vis This is the visible light weight matrix; C The number of channels for the input feature; D The dimension of the projected target features; b vis F' is the visible light bias vector. vis Features of the projected visible light image; Infrared thermal imaging feature projection is represented as: ； Among them, GlobalAvgPool3D is a three-dimensional global average pooling operation; W ir This is the infrared thermal imaging weight matrix; b ir F is the infrared thermal imaging bias vector; ir F' is the infrared thermal imaging feature tensor; ir These are the infrared thermal imaging features after projection; The characteristic projection of gas concentration is represented as follows: ； in, W gas The weight parameters for the linear transformation; GlobalAvgPool1D(F gas ) represents the characteristic F of gas concentration gas The result after performing one-dimensional global average pooling; b gas The bias parameter representing the gas concentration; F' gas The projected gas concentration characteristics; Gating weights are generated using a multilayer perceptron. g : ; in, σ It is the sigmoid activation function. g ∈[0,1] 3 ; W g This is the weight matrix; bg For bias terms; [⋅;⋅] indicates concatenation; The static and dynamic features are weighted and summed according to the gating weights to obtain the fused feature F. fusion : ； in, The visible light image features F' after projection vis The weight, The projected infrared thermal imaging feature F' ir The weight, The projected gas concentration feature F' gas The weight.

2. The method according to claim 1, characterized in that, The method for preprocessing infrared thermal imaging data in the preprocessing step of the multimodal environmental data includes: Based on the pre-calibrated blackbody radiation coefficient, the raw signal of the infrared thermal imaging data is converted into a calibration temperature matrix; The calibration temperature matrix is ​​processed by an adaptive Gaussian filter to output a smoothed temperature matrix after adaptive Gaussian filtering. The Sobel operator edge detection method is used to process the smoothed temperature matrix after adaptive Gaussian filtering, and the gradient temperature matrix is ​​output. Based on the smoothed temperature matrix after adaptive Gaussian filtering, a set of candidate temperature points with temperatures greater than the preset dynamic temperature threshold is extracted according to the preset dynamic temperature threshold. Based on the gradient temperature matrix, the candidate temperature point set is further filtered according to a preset gradient threshold, and temperature points with gradient values ​​greater than the preset gradient threshold are retained to determine the high-temperature region.

3. The method according to claim 1, characterized in that, The method for preprocessing gas concentration data in the preprocessing step of the multimodal environmental data includes: The time-series data collected by the methane sensor is filtered to obtain a smooth methane concentration sequence; Sliding baseline correction was performed on the smoke sensor data to obtain a relative smoke concentration sequence; Based on preset methane concentration thresholds and preset relative smoke concentration thresholds, the smoothed methane concentration sequence and the relative smoke concentration sequence are detected respectively, generating gas anomaly event markers and concentration change rate curves.

4. The method according to claim 1, characterized in that, The component-level noise reduction processing for each channel of the HSV color space includes: The H channel image in the HSV color space is subjected to noise reduction processing; The S-channel and V-channel images in the HSV color space are subjected to noise reduction processing; The H-channel image, V-channel image, and S-channel image after noise reduction are merged.

5. The method according to claim 4, characterized in that, The noise reduction processing of the H channel image in the HSV color space includes: For each image block of the H-channel image, a transformation from the spatial domain to the frequency domain is performed to obtain the frequency domain transformation matrix; Calculate the gradient values ​​of the frequency domain transformation matrix in N directions, where N≥4; Determine the maximum and minimum gradient values ​​among the N gradient values ​​to be calculated; The number of trailing coefficients to be removed is determined based on the relationship between the maximum gradient value, the minimum gradient value, and multiple thresholds. The transformation matrix is ​​subjected to tail coefficient removal based on the number of tail coefficients to be removed; The frequency domain matrix after removing the trailing coefficients is transformed from the frequency domain to the spatial domain to obtain the denoised H-channel image.

6. The method according to claim 4, characterized in that, The noise reduction processing of the V channel and S channel images in the HSV color space includes: The V-channel image and the S-channel image are subjected to filtering and noise reduction using a combined bilateral filtering method of saturation and brightness. The formula for the combined bilateral filtering of saturation and brightness is as follows: ； In the formula, For bilateral filter weights, For spatial domain weights, For range weights; k , l ) represents the coordinates of the current pixel to be processed. i , j () represents the coordinates of the reference point pixel. σ d For spatial domain smoothing intensity parameters; Is( k , l ) is the coordinate ( k , l The saturation value at () is( i , j ) is the reference point ( i , j The saturation value at () is Iv( k , l ) is the coordinate ( k , l The brightness value at () is Iv( i , j ) is the reference point ( i , j The brightness value at ) σ r This is the range sensitivity parameter.

7. The method according to claim 5, characterized in that, The gradient values ​​of the frequency domain transform matrix in N directions are obtained using the following formula when N=4: ； In the formula, This represents the gradient value in the horizontal direction on the right side of the frequency domain matrix. This represents the gradient value in the vertical direction below the frequency domain matrix. This represents the gradient value in the lower left corner of the frequency domain matrix. This represents the gradient value in the lower right corner. Represents the first in the frequency domain matrix p The elements in row q and column q are used to form a frequency domain matrix of size m×n.

8. The method according to claim 1, characterized in that, The process of extracting fire features from the preprocessed multimodal environmental data yields initial fire features for each multimodal mode, including: The spatial feature map is extracted from the preprocessed visible light image data using a lightweight CNN network to obtain the visible light image features. The spatiotemporal thermal features are extracted from the preprocessed infrared thermal imaging data using a spatiotemporal convolutional network to obtain the infrared thermal imaging features. The gas concentration features are obtained by extracting time-series dynamic features from the preprocessed gas concentration data using bidirectional LSTM.

9. An intelligent identification system for the initial fire type in a kitchen, using the method described in any one of claims 1 to 8, comprising: A multi-sensor fusion sensing module is used to acquire multimodal environmental data in the kitchen; the multimodal environmental data includes visible light image data, infrared thermal imaging data, and gas concentration data; A multimodal data preprocessing module is used to preprocess the multimodal environmental data respectively; The fire feature extraction module is used to extract fire features from the preprocessed multimodal environmental data to obtain initial fire features for each multimodal mode; the initial fire features for each multimodal mode include visible light image features, infrared thermal imaging features, and gas concentration features; The fire type identification module is used to input the multimodal initial fire features into a deep learning model that fuses multimodal features, and output the determination result of the initial fire type in the kitchen.

Images