Intelligent identification method and system for initial fire type of kitchen

Abstract

The invention discloses an intelligent identification method and system for an initial fire type of a kitchen. The method comprises the following steps: acquiring multi-modal environment data in the kitchen; the multi-modal environment data comprises visible light image data, infrared thermal imaging data and gas concentration data; the multi-mode environment data are preprocessed; performing fire feature extraction on the preprocessed multi-modal environment data to obtain each multi-modal initial fire feature; the multi-modal initial fire features comprise visible light image features, infrared thermal imaging features and gas concentration features; and inputting the multi-modal initial fire features into a deep learning model of multi-modal feature fusion, and outputting a judgment result of the kitchen initial fire type. According to the intelligent identification method and system for the initial fire type of the kitchen, the type of the initial fire of the kitchen can be rapidly and accurately identified in a complex environment of the kitchen, and a basis is provided for subsequently taking targeted fire extinguishing measures.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of fire-fighting automation, and particularly relates to an intelligent identification method and system for initial fire types in a kitchen. BACKGROUND

[0002] Kitchen fires, as a high-frequency and high-hazard fire type, account for one-fifth of the total number of civil fires and cause huge economic losses every year. Unlike ordinary building fires, kitchen fires have the characteristics of complex burning materials, rapid fire spread, and high rekindling rate. In particular, commercial kitchens in catering places have high fire load density due to the concentration of open fire equipment and serious oil smoke deposition, and secondary injury caused by oil splashing during firefighting is prone to occur.

[0003] For kitchen fire automatic identification algorithms, normal cooking in the kitchen can easily cause confusion with kitchen fires, causing fire identification interference, and it is not easy to accurately and quickly identify kitchen fires. At the same time, since there can be different types of fires in the kitchen, different types of fires require different treatment methods, so accurate identification of kitchen fire types helps to more accurately match the corresponding fire extinguishing method. In addition, due to the characteristics of the kitchen environment, it is extremely easy to cause combustion and explosion, so the identification of the initial fire is particularly important.

[0004] The current automatic fire extinguishing system that can realize fire type identification mainly collects abnormal information through sensors such as smoke sensors and gas sensors, and triggers an alarm to extinguish the fire. For example, the fire identification method and device disclosed in patent CN117437741A uses a gas sensor to collect gas information in fire smoke to identify the type of fire. However, due to the significant particularity of kitchen fires compared to other types of fires, especially oil fires, which often develop at a speed of seconds, combustion and explosion are extremely easy to occur; and the method of gas or smoke analysis often requires a larger fire, obviously, this method is not suitable for identifying the initial fire type in a kitchen. Therefore, a method is needed that can quickly identify the type of fire when an initial fire occurs in a kitchen so that corresponding fire extinguishing measures can be taken. SUMMARY

[0005] To solve the above problems in the prior art, the present application provides an intelligent identification method and system for initial fire types in a kitchen. The technical problem to be solved by the present application is solved by the following technical scheme: In a first aspect, the present application provides an intelligent identification method for initial fire types in a kitchen, comprising the steps of: obtaining multi-modal environment data in the kitchen; the multi-modal environment data includes visible light image data, infrared thermal imaging data, and gas concentration data; respectively pre-process the multi-modal environment data; extract fire features from the pre-processed multi-modal environment data to obtain multi-modal initial fire features; the multi-modal initial fire features include visible light image features, infrared thermal imaging features and gas concentration features; input the multi-modal initial fire features into a multi-modal feature fusion deep learning model to output a determination result of an initial fire type in the kitchen.

[0006] In an embodiment of the present application, in the step of pre-processing the multi-modal environment data, the method of pre-processing visible light data includes: perform median filter processing on the visible light image data to eliminate salt and pepper noise in the visible light image data; perform adaptive histogram equalization processing on the visible light image data after the salt and pepper noise is eliminated to enhance the contrast between the flame and the background; convert the visible light image data after the contrast is enhanced from an RGB color space to an HSV color space to decouple color and brightness information; perform component sub-band noise reduction processing on each channel of the HSV color space; perform dynamic threshold segmentation on the HSV color space after the noise reduction processing to segment out the flame region.

[0007] In an embodiment of the present application, in the step of pre-processing the multi-modal environment data, the method of pre-processing infrared thermal imaging data includes: convert the original signal of the infrared thermal imaging data into a calibrated temperature matrix based on a pre-calibrated blackbody radiation coefficient; perform adaptive Gaussian filter processing on the calibrated temperature matrix to output a smoothed temperature matrix after adaptive Gaussian filter processing; perform Sobel operator edge detection on the smoothed temperature matrix after adaptive Gaussian filter processing to output a gradient temperature matrix; based on the smoothed temperature matrix after adaptive Gaussian filter processing, extract a candidate temperature point set with a temperature greater than a preset dynamic temperature threshold value according to the preset dynamic temperature threshold value; based on the gradient temperature matrix, further filter the candidate temperature point set according to a preset gradient threshold value, retain temperature points with a gradient value greater than the preset gradient threshold value, and determine a high-temperature region.

[0008] In an embodiment of the present application, in the step of pre-processing the multi-modal environment data, the method of pre-processing gas concentration data includes: Filtering the time series data collected by the methane sensor to obtain a smooth methane concentration sequence; Performing sliding baseline correction on the smoke sensor data to obtain a relative smoke concentration sequence; According to the preset methane concentration threshold and the preset smoke relative concentration threshold, the smooth methane concentration sequence and the relative smoke concentration sequence are detected respectively to generate a gas abnormal event marker and a concentration change rate curve.

[0009] In an embodiment of the present application, the component sub-band noise reduction processing of the H channel image of the HSV color space includes: Performing noise reduction processing on the H channel image of the HSV color space; Performing noise reduction processing on the S channel image and the V channel image of the HSV color space; Merging the noise-reduced H channel image, V channel image and S channel image.

[0010] In an embodiment of the present application, the noise reduction processing of the H channel image of the HSV color space includes: For each image block of the H channel image, a transformation from spatial domain to frequency domain is performed to obtain a frequency domain transformation matrix; Obtaining gradient values of N directions of the frequency domain transformation matrix, where N≥4; Determining the maximum gradient value and the minimum gradient value among the obtained N direction gradient values; According to the size relationship of the maximum gradient value, the minimum gradient value and a plurality of threshold values, determining the number of tailing coefficients to be removed; According to the number of tailing coefficients to be removed, removing tailing coefficients from the transformation matrix; Performing a transformation from frequency domain to spatial domain on the tailing coefficient-removed frequency domain matrix to obtain a noise-reduced H channel image.

[0011] In an embodiment of the present application, the noise reduction processing of the V channel image and the S channel image of the HSV color space includes: The V channel image and the S channel image are filtered and noise-reduced by a saturation and brightness joint bilateral filtering method, wherein the formula of the saturation and brightness joint bilateral filtering is as follows: In the formula, is a bilateral filtering weight, is a spatial domain weight, is a value domain weight; k , l is the coordinate of the current pixel to be processed, i ,j ) is the coordinate of the reference point pixel, σ d Is the spatial domain smoothing intensity parameter; Is( k , l ) is the coordinate ( k , l ) at the saturation value, Is( i , j ) is the reference point ( i , j ) at the saturation value, Iv( k , l ) is the coordinate ( k , l ) at the brightness value, Iv( i , j ) is the reference point ( i , j ), σ r is the range sensitivity parameter.

[0012] In one embodiment of the present invention, the gradient values ​​of the frequency domain transformation matrix in N directions are obtained, wherein when N=4, the formula used is as follows:

[0013] In the formula, Represents the gradient value in the horizontal direction on the right side of the frequency domain matrix, Represents the gradient value in the vertical direction of the lower side of the frequency domain matrix, Represents the gradient value at the lower left of the frequency domain matrix, represents the gradient value at the lower right, Represents the first p The elements in the qth row and qth column, the size of the frequency domain matrix is ​​m×n.

[0014] In one embodiment of the present invention, extracting fire features from the pre-processed multimodal environmental data to obtain each multimodal initial fire feature includes: A lightweight CNN network is used to extract spatial feature maps from the preprocessed visible light image data to obtain the visible light image features; The preprocessed infrared thermal imaging data is subjected to a spatiotemporal convolutional network to extract spatiotemporal thermal features to obtain the infrared thermal imaging features; The preprocessed gas concentration data is subjected to a bidirectional LSTM to extract the temporal dynamic features and obtain the gas concentration features.

[0015] In a second aspect, an embodiment of the present invention provides an intelligent system for identifying the type of initial kitchen fire, applying the above method, including: a multi-sensor fusion perception module, configured to acquire multi-modal environment data in the kitchen; the multi-modal environment data comprises visible light image data, infrared thermal imaging data and gas concentration data; a multi-modal data preprocessing module, configured to preprocess the multi-modal environment data respectively; a fire feature extraction module, configured to perform fire feature extraction on the preprocessed multi-modal environment data to obtain multi-modal initial fire features; the multi-modal initial fire features comprise visible light image features, infrared thermal imaging features and gas concentration features; a fire type identification module, configured to input the multi-modal initial fire features into a multi-modal feature fusion deep learning model to output a determination result of an initial fire type of the kitchen.

[0016] Compared with the prior art, the present application has the following advantages: The intelligent identification method and system for the initial fire type of the kitchen provided by the present application can compensate for the limitations of a single mode by simultaneously fusing three types of heterogeneous data, namely visible light images, infrared thermal imaging and gas concentration, thereby improving the comprehensiveness and accuracy of fire type identification; the quality of input data is significantly improved through multi-stage preprocessing optimization, and the signal-to-noise ratio and stability of the data after preprocessing are greatly improved, thereby laying a reliable foundation for subsequent feature extraction; the accuracy of initial fire type identification in a complex kitchen environment is improved by respectively performing noise reduction processing on the HSV channel vector images, thereby providing key technical support for fire safety in home, catering and other scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flowchart of an intelligent identification method for the initial fire type of the kitchen provided by an embodiment of the present application.

[0018] Figure 2 is a flowchart of a method for preprocessing multi-modal environment data provided by an embodiment of the present application.

[0019] Figure 3 is a flowchart of a method for preprocessing visible light data provided by an embodiment of the present application.

[0020] Figure 4 is a flowchart of a method for preprocessing infrared thermal imaging data provided by an embodiment of the present application.

[0021] Figure 5 is a module block diagram of an intelligent identification system for the initial fire type of the kitchen provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] The present application will be further described in detail below in conjunction with specific embodiments, but the embodiments of the present application are not limited thereto.

[0023] Referring to Figure 1 , Figure 1 is a flowchart of a kitchen initial fire type intelligent identification method provided by an embodiment of the present application. The kitchen initial fire type intelligent identification method provided by the embodiment of the present application comprises the following steps: S100, acquiring multi-modal environment data in a kitchen; the multi-modal environment data comprises visible light image data, infrared thermal imaging data and gas concentration data.

[0024] Due to the complex kitchen environment, the initial fire characteristics of the kitchen are affected by multiple characteristics such as flame color, temperature, smoke concentration and oil droplet splashing. Single image recognition is easily disturbed by environmental factors such as steam and light changes. Therefore, the embodiment of the present application fuses multi-type sensor data to improve the robustness of initial fire identification.

[0025] The multi-sensor fusion perception module adopted by the embodiment of the present application comprises a visible light camera, an infrared thermal imaging camera and a gas sensor. The visible light camera is used to capture the flame color and shape. The infrared thermal imaging camera detects the high-temperature area. Since the surface temperature of the initial combustion of oil can reach more than 300 DEG C, an infrared thermal imaging camera with a temperature measurement range of -20 DEG C to 1500 DEG C can be selected. The gas sensor used in the embodiment of the present application can be a catalytic combustion type methane sensor and a semiconductor type smoke sensor, which is used for gas leakage fire and can identify the pre-combustion stage caused by gas leakage. The preferred catalytic combustion type methane sensor has a detection range of 0-100% LEL and an accuracy of ± 5% LEL, which can be deployed in high-risk areas such as near the gas pipeline interface and below the gas stove. The catalytic element and the oxidation reaction of methane gas produce an electrical signal to monitor the methane concentration in the environment in real time. The preferred semiconductor smoke sensor detects particulate matter concentration of 0-20 mg / m³ and response time < 10 s, which can be deployed at key points on the smoke diffusion path such as the central ceiling of the kitchen and the exhaust outlet of the extractor hood. The semiconductor material changes in resistance to perceive the adsorption of smoke particles and outputs an electrical signal proportional to the smoke concentration.

[0026] Correspondingly, the data collected by the visible light camera in the embodiment of the present application is visible light image data, the data collected by the infrared thermal imaging camera is infrared thermal imaging data, and the data collected by the gas sensor is gas concentration data. All sensor data is transmitted synchronously to the edge computing terminal through the CAN bus, and after the time stamp is aligned, the next step of noise reduction preprocessing is performed.

[0027] S200, respectively pre-processing the multi-modal environment data.

[0028] Due to the complex characteristics of the kitchen environment, the collected environmental data is easily affected by environmental factors, for example, the visible light image may be affected by salt and pepper noise, illumination change, etc., the infrared thermal imaging may have sensor noise, environmental temperature interference; the gas concentration data may have sensor drift, background gas interference, etc. The purpose of preprocessing is to improve the data quality, so that the subsequent feature extraction and model training are more effective.

[0029] Referring to Figure 2 , Figure 2 is a flowchart for preprocessing multi-modal environmental data provided by an embodiment of the present application. In the embodiment of the present application, visible light image data, infrared thermal imaging data and gas concentration data need to be preprocessed respectively, and the preprocessing method needs to be processed by different methods according to the characteristics of each data. The preferred method is as follows: S210, preprocessing the visible light data.

[0030] The image in the kitchen environment may be affected by steam, light reflection, etc., producing salt and pepper noise; if not denoised, the subsequent image segmentation may appear artifacts, such as noise points being falsely enhanced or segmented. Therefore, preprocessing the visible light data can enhance the contrast between the flame area and the background, eliminate noise, and extract visual features such as flame color, shape, motion trajectory, etc.

[0031] Referring to Figure 3 , Figure 3 is a flowchart for preprocessing visible light data provided by an embodiment of the present application. Specifically, the method for preprocessing the visible light data comprises: S211, the visible light image data is subjected to median filtering processing to eliminate salt and pepper noise in the visible light image data.

[0032] The characteristic of salt and pepper noise is that black and white noise points appear randomly, usually caused by sensor failure, transmission error or environmental interference. The characteristic of this noise is that it is randomly distributed in the image, with large amplitude, which can seriously damage the details of the image, such as blurring the edges of the flame or covering the key features. Although traditional Gaussian filtering can smooth the noise, it will blur the edges, while median filtering is more effective in preserving the edges.

[0033] Median filtering is a non-linear filtering method, the core of which is to replace the current pixel value with the median value in the neighborhood of the pixel. For a two-dimensional image, a sliding window is usually used, all pixel values within the window are sorted, and the middle value is taken as the new value of the current pixel; the sliding window can be selected as 3x3 or 5x5, for example, if the sliding window is less than 3x3, it may not be able to sufficiently suppress noise, and if it is greater than 7x7, it may cause edge blurring. This can effectively suppress salt and pepper noise, because the values of noise points are usually quite different from the surrounding pixels, and after sorting, they will be excluded from the median. The pixel values of salt and pepper noise are either very high or very low, and the difference with the surrounding normal pixels is significant. In the sorted neighborhood, these extreme values will be at both ends, while the median usually comes from normal pixels, so after replacement, the noise can be effectively eliminated while the edges and details of the image are preserved.

[0034] This step eliminates salt and pepper noise in the visible light image data by median filtering, making the edges of the flame clearer and the color distribution more uniform, which is conducive to extracting features such as shape and color, and provides high-quality input for subsequent feature extraction.

[0035] S212, adaptive histogram equalization processing is performed on the visible light image data after the salt and pepper noise is eliminated to enhance the contrast between the flame and the background.

[0036] After removing salt and pepper noise, the image may still have uneven brightness issues, such as darker flame areas and brighter backgrounds. Histogram equalization adjusts the gray scale distribution of the image to make the gray scale values more uniform, thereby enhancing the contrast. However, traditional histogram equalization (HE) is global, which may cause noise amplification or loss of details. Adaptive histogram equalization divides the image into small regions, performs histogram equalization on each region separately, and then performs bilinear interpolation to eliminate artificial marks at the region boundaries, thereby enhancing local contrast while avoiding global over-enhancement.

[0037] Since the flame usually has high brightness and specific colors, while the background may be dark. Adaptive histogram equalization processing can enhance the local contrast of the flame region, making the edges and details of the flame more prominent and more distinct from the background. For example, the specific processing steps can include image blocking, calculating the histogram of each block, performing histogram equalization, bilinear interpolation to merge the results, and other steps.

[0038] This step performs adaptive histogram equalization processing, which locally and adaptively adjusts the gray scale distribution, effectively enhancing the local contrast between the flame and the background in the visible light image after salt and pepper noise is eliminated, while preserving the details of the flame.

[0039] S213, convert the visible light image data with enhanced contrast from the RGB color space to the HSV color space to decouple color and brightness information.

[0040] RGB color space is a color additive model based on three primary colors of red, green and blue. The color of each pixel is determined by three components of R, G and B, and the range is usually 0-255. The HSV (Hue, Saturation, Value) color space decomposes color into hue (H), saturation (S) and value (V), which is more consistent with human perception of color. Among them, H represents the type of color (such as red, green, blue), S represents the purity of color (0 for gray, 1 for pure color), and V represents the brightness of color (0 for black, 1 for white). RGB has certain limitations, such as the coupling of brightness and color, which may change the perception of the color itself when adjusting brightness; and HSV separates the three attributes, which is convenient for processing color and brightness separately. Therefore, it is necessary to convert visible light image data from RGB color space to HSV color space for feature extraction in fire identification.

[0041] The conversion of visible light image data from RGB color space to HSV color space can use geometric derivation method, coordinate transformation method, piecewise definition method, etc.

[0042] S214, component sub-band noise reduction processing is performed on each channel of the HSV color space.

[0043] In fire identification, after the visible light image is converted from RGB to HSV, the H (hue), S (saturation), and V (value) channels carry key semantic information such as "color type", "purity", and "brightness" of the flame. However, each channel is easily disturbed by different types of noise, such as random hue shift in H channel, saturation fluctuation in S channel, and brightness fluctuation in V channel, which requires component sub-band noise reduction processing to specifically suppress noise while preserving the core features of each channel.

[0044] H (hue) represents the "type" of color, such as orange-red corresponding to H≈20°~40°, and its value range is circular (0°~360°); the noise mainly shows random hue shift, which will directly change the color category of the flame, such as misjudging the flame as background color. S (saturation) represents the "purity" of color (0 for gray, 1 for pure color), and its value range is [0, 1]; the noise mainly shows random amplitude fluctuation; S value fluctuation will reduce the "vividness" of the flame, and even cause confusion between the flame and the background. V (value) represents the "brightness" of color (0 for black, 1 for white), and its value range is [0, 1]; the noise mainly shows brightness fluctuation; V value fluctuation will blur the "intensity" difference of the flame, such as the low brightness of smoldering stage and the high brightness of intense burning.

[0045] In the embodiment of the present application, in view of the identification requirement of the initial fire type in the kitchen, the H channel image is preferably processed alone, the S channel image and the V channel image can be processed jointly, and the processing methods of the two are different. The purpose of this design is: H is the color tone, which is more the distinction of color types, and may need to be processed separately to avoid color deviation; S and V are usually related to the "vividness" and "brightness" of the color, and can be jointly processed when describing the same color characteristics of high saturation and high brightness of the flame. In addition, since H directly determines the "category" of the color, its accuracy directly affects the semantic understanding of the image; while the fluctuation of S only reduces the color vividness, but does not change the color category, and the fluctuation of V mainly affects the brightness level, but does not change the color category or vividness; therefore, different denoising strengths can be used for denoising processing of each component sub-band.

[0046] Specifically, the component sub-band denoising processing is performed on each channel of the HSV color space, including: S2141, performing denoising processing on the H channel image of the HSV color space; S2142, performing denoising processing on the S channel image and the V channel image of the HSV color space; S2143, merging the denoising processed H channel image, V channel image and S channel image.

[0047] In step S2141, the denoising processing is performed on the H channel image of the HSV color space, including: S21411, for each image block of the H channel image, performing transformation from spatial domain to frequency domain to obtain a frequency domain transformation matrix.

[0048] In the H channel image processing of fire identification, the image is divided into blocks and transformed from spatial domain to frequency domain, the core purpose of which is to extract frequency domain features such as edges, textures and periodicity from the spatial distribution of local pixels, so as to extract local frequency features and reduce the calculation complexity, thereby more efficiently separating the "color types" of the flame and noise interference.

[0049] The transformation from spatial domain to frequency domain can be performed by using the 53 wavelet transform, and the formula based on the 53 wavelet transform is: Wherein, x is the spatial domain pixel value, i.e. the pixel value in the unit block; c is the frequency domain value at the odd position, d is the frequency domain value at the even position, the frequency domain value is the value after frequency domain transformation; c is the AC coefficient, d is the DC coefficient; n is a natural number, indicating the pixel position.

[0050] Compared with the DCT (Discrete Cosine Transform) method commonly used in the prior art, the method based on 53 wavelet transform has the characteristics of high transformation efficiency and easy hardware implementation.

[0051] S21412. Obtain the gradient values ​​of the frequency domain transformation matrix in N directions, where N ≥ 4.

[0052] The method for calculating the gradient values ​​of the frequency domain transform matrix in N directions is to calculate the sum of the absolute differences between adjacent pixels. Common directions may include horizontal, vertical, and two diagonal directions, which corresponds to the case of N=4 in this step. However, N can be greater than 4, such as 8 directions, or a finer angle division.

[0053] The embodiment of the present invention is described using N=4 as an example. The formula used to obtain the gradient values ​​of the frequency domain transformation matrix in four directions is as follows:

[0054] In the formula, Represents the gradient value in the horizontal direction on the right side of the frequency domain matrix, Represents the gradient value in the vertical direction of the lower side of the frequency domain matrix, Represents the gradient value at the lower left of the frequency domain matrix, represents the gradient value at the lower right, Represents the first p The elements in the qth row and qth column, the size of the frequency domain matrix is ​​m×n.

[0055] It's important to note that flame edges and textures typically exhibit multi-directional features, such as vertically rising flame edges, horizontally spreading flame boundaries, and irregular diagonal edges. A single-directional gradient cannot fully capture these features, potentially leading to missed edge detection or misjudgment. Therefore, this embodiment of the present invention calculates gradient values ​​in N ≥ 4 directions to cover common flame edge directions (e.g., 0°, 45°, 90°, 135°, etc.), ensuring comprehensive feature description.

[0056] S21413. Determine the maximum gradient value and the minimum gradient value among the obtained gradient values ​​in the N directions.

[0057] The multi-directional gradient values ​​reflect the rate of change of each position in the frequency domain transformation matrix in the spatial domain. However, due to noise interference or edge tailing, some gradient values ​​may deviate from the normal range of the true features. Therefore, by determining the global maximum and minimum gradient values, the "reasonable range" of the gradient values ​​can be delineated, providing a benchmark for the subsequent identification of tailing coefficients.

[0058] S21414, according to the maximum gradient value, the minimum gradient value and the size relationship of multiple thresholds, the number of tail coefficients to be removed is determined.

[0059] Tail Coefficients refers to abnormal coefficients caused by noise or edge tail in the frequency domain transformation matrix, which will interfere with subsequent feature extraction.

[0060] In the embodiment of the application, the multiple thresholds used when determining the number of tail coefficients to be removed can be formulated according to the prior noise model of the model “brightness x color x noise”. According to the noise model, a large number of sample image blocks can be obtained, and by performing denoising experiments on these sample image blocks, the multiple thresholds required by the embodiment of the application can be determined.

[0061] Specifically, the number of tail coefficients to be removed is determined according to the following rules: When G max , T min , it indicates a flat or low-frequency noise area, and a large amount of denoising is required; the number of tail coefficients to be removed is the upper limit value, and a=A at this time. When G min , T max , it indicates an indeterminable area, and no tail coefficient needs to be removed, and a=0 at this time. When G min , G max and G min , T max and G max , T min , the number of tail coefficients to be removed satisfies: 0<a<A. And the specific value of a is determined by the position of the overlapping interval: The closer the overlapping interval is to T min (i.e. G min , G max )∩( T min , T max] , the closer the left endpoint of T min ), a is larger; the closer the overlapping interval is to T max (i.e. Gmin , G max ]∩[ T min , T max ] is closer to the right endpoint T max ), the smaller a is.

[0062] in, G max is the maximum gradient value of the multi-directional gradient values; G min is the minimum gradient value of the multi-directional gradient values; T min is the minimum threshold in the preset threshold set; T max is the maximum threshold in the preset threshold set; A is the maximum number of tailing coefficients to be removed; a is the actual number of tailing coefficients to be removed (0≤a≤A).

[0063] S21415. Remove the trailing coefficients of the transformation matrix according to the number of the trailing coefficients to be removed.

[0064] After determining the number of tailing coefficients to be removed, these coefficients need to be removed from the frequency domain transform matrix through operations such as zeroing, smoothing, or replacement to restore the true characteristics of the matrix.

[0065] S21416. Perform a transformation from the frequency domain to the spatial domain on the frequency domain matrix after removing the trailing coefficients to obtain a denoised H channel image.

[0066] The embodiment of the present invention determines the tailing coefficients through established rules and removes the tailing coefficients, thereby identifying and suppressing abnormal coefficients caused by noise, edge tailing or artifacts in the frequency domain, thereby retaining the key frequency characteristics of the flame.

[0067] In the above step S2142, the noise reduction process is performed on the S channel image and the V channel image in the HSV color space, including: The V channel image and the S channel image are subjected to filtering and noise reduction processing by using the saturation and brightness combined bilateral filtering method. The formula for the saturation and brightness combined bilateral filtering is as follows:

[0068] In the formula, is the bilateral filter weight, is the spatial domain weight, is the range weight; is the coordinate of the current pixel to be processed, is the coordinate of the reference point pixel, σ d Is the spatial domain smoothing intensity parameter; Is( k ,l ) are saturation values at coordinates ( k , l ) are saturation values at reference points ( i , j ) are value values at coordinates ( i , j ) are value values at reference points ( k , l ) are lightness values at coordinates ( k , l ) are lightness values at reference points ( i , j ) are value values at coordinates ( i , j ) are value values at reference points ( σ r is a value range sensitivity parameter.

[0069] The saturation and lightness joint bilateral filtering method adopted by the embodiment of the present application performs adaptive smoothing and feature reservation on multi-modal fire features through dual constraints of the spatial domain and the value domain, solves the problems of multi-modal noise heterogeneity, edge blur and information redundancy, and finally provides high-quality, low-noise and multi-dimensional complementary feature input for subsequent deep learning models, significantly improving the accuracy and robustness of fire identification.

[0070] After the noise reduction processing is performed on the images of each channel, the H channel image, the V channel image and the S channel image after the noise reduction processing are merged.

[0071] S215, performing dynamic threshold segmentation on the HSV color space after the noise reduction processing to segment out the flame region.

[0072] The HSV color space after the noise reduction processing has decoupled the color, purity and brightness information, but the HSV values of the flame region may still fluctuate due to the following factors: light change: the strength change of the kitchen light and natural light will change the lightness of the flame; background interference: steam, smoke or reflection may introduce false shadows similar to the color of the flame; flame dynamics: the brightness of the flame in the smoldering stage is lower, and the brightness is higher when burning fiercely, and a single global threshold cannot cover all states. The traditional global threshold is easy to misjudge these interference regions as flames, or miss the real flames with low / high brightness. The dynamic threshold segmentation can adapt to the above changes by adjusting the threshold range according to the statistical characteristics of the local region of the HSV, and improve the accuracy of the flame segmentation.

[0073] The essence of dynamic threshold segmentation is to calculate the upper and lower limits of the threshold for each pixel or neighborhood based on the statistical characteristics (such as mean, variance, histogram distribution) of the local region of the image, so that the threshold adjusts with the change of the image content.

[0074] The embodiment of the present application performs dynamic threshold segmentation on the noise-reduced HSV color space, accurately locates the flame region by adaptively adjusting the threshold range of each channel, and solves the misjudgment problem caused by light and background interference in the traditional global threshold. This step provides high-quality input for subsequent flame feature extraction and fire identification model.

[0075] S220, pre-process the infrared thermal imaging data, please refer to Figure 4 , Figure 4 is a flowchart for pre-processing infrared thermal imaging data provided by the embodiment of the present application. The specific method comprises: S221, convert the original signal of the infrared thermal imaging data into a calibrated temperature matrix based on the pre-calibrated blackbody radiation coefficient.

[0076] In this step, the pre-calibrated blackbody radiation coefficient includes the ambient temperature, the infrared thermal imaging sensor response coefficient, the infrared thermal imaging sensor response coefficient and the ambient temperature compensation coefficient. Combined with the physical characteristics of the infrared thermal imaging sensor and the calibration data, the voltage value is mapped to the real temperature by the following formula, and the calibrated temperature matrix is output T cal.

[0077]

[0078] Wherein, R raw is the original infrared signal matrix, R raw (i,j) is the voltage value of the (i,j) pixel of the infrared thermal imaging sensor; T env is the ambient temperature, k is the infrared thermal imaging sensor response coefficient, which is determined in advance by the blackbody radiation calibration experiment, α is the ambient temperature compensation coefficient, which is obtained by fitting the calibration data under different ambient temperatures; β is the zero-point correction coefficient, which is determined by the blackbody source zero-point calibration; and k =0.12℃ / mV, α =0.003 / ℃, β= .

[0079] S222, the calibrated temperature matrix is processed by adaptive Gaussian filtering, and a smoothed temperature matrix after adaptive Gaussian filtering is output.

[0080] In order to suppress the random noise such as sensor thermal noise and electromagnetic interference in the infrared image, and retain the temperature details of the high temperature area of the flame, it is necessary to process the calibrated temperature matrix by adaptive Gaussian filtering. By using adaptive Gaussian filtering, the filter kernel size is dynamically adjusted according to the local temperature fluctuation degree, so as to avoid excessive smoothing of the edge details of the high temperature area. The specific implementation method is: (1) Define the local window size W , initially 5x5; (2) Calculate the standard deviation of temperature within the window σ local ; (3) If σ local <1℃ (low noise area), smoothed using a 3x3 Gaussian kernel ( σ =1.0); (4) If σ local ≥1℃ (high noise area, such as near the extractor hood), smoothed using a 7x7 Gaussian kernel ( σ =1.5).

[0081] This step outputs the smoothed temperature matrix. By processing the calibrated temperature matrix using adaptive Gaussian filtering, noise is suppressed and the edges of high temperature areas are clarified.

[0082] S223, using Sobel operator edge detection method, processing the smoothed temperature matrix after adaptive Gaussian filtering, output gradient temperature matrix.

[0083] Using Sobel operator edge detection method, the first order partial derivative of the temperature matrix in x and y directions are calculated respectively, and the gradient amplitude matrix is synthesized:

[0084] Wherein, the Sobel operator template is:

[0085] is the gradient amplitude matrix, representing the temperature change rate at pixel point ( i , j ); G x is the Sobel convolution kernel in x direction, G y is the Sobel convolution kernel in y direction, T ( i , j ) is the gray value of the image at pixel point ( i , j ).

[0086] This step outputs the temperature gradient matrix, which can effectively suppress the noise in the infrared thermal imaging data.

[0087] S224, based on the adaptive Gaussian filtering processed smooth temperature matrix, according to the preset dynamic temperature threshold, extracting the candidate temperature point set whose temperature is greater than the preset dynamic temperature threshold.

[0088] This step needs to set the dynamic temperature threshold T th ( such as T th = ambient temperature + 100℃ ), extract all pixel points whose temperature is greater than T th from the smooth temperature matrix, form a candidate high temperature point set, denoted as S temp .

[0089] It should be noted that although the smooth temperature matrix is processed in step S223 and the gradient temperature matrix is output, the smooth temperature matrix will not be discarded because the temperature gradient matrix is generated, and they will be stored as independent data structures. Therefore, step S224 continues to process on the basis of step S223, and both matrices can be accessed at the same time.

[0090] S225, based on the gradient temperature matrix, further screening the candidate temperature point set according to the preset gradient threshold, retaining the temperature points whose gradient value is greater than the preset gradient threshold, and determining the high temperature region.

[0091] The candidate high temperature point set S S temp is further screened, and only the points with gradient value > 3℃ ( such as = 3℃ px ) are retained to form a preliminary flame region set and determine the high temperature region.

[0092] This step is a key link in the accurate identification process of the high temperature region. By introducing gradient information, secondary screening of candidate temperature points is realized, which significantly improves the accuracy of edge detection and region segmentation. By combining temperature amplitude information and temperature change rate information, more accurate and robust identification of the high temperature region is realized.

[0093] S230, preprocessing the gas concentration data, the specific method comprising: S231, filtering the time series data collected by the methane sensor to obtain a smooth methane concentration sequence.

[0094] The time series data collected by the methane sensor is often disturbed by sensor noise, environmental interference and other noise. These noises will cause the original data to present "burr" or "jitter", affecting the accuracy of subsequent threshold detection. By filtering algorithm to suppress noise, the true trend of methane concentration is retained, and a smooth and continuous concentration sequence is obtained.

[0095] The filtering method can adopt a moving average filtering, a median filtering or a Kalman filtering method. After filtering, the noise peak value of the smoothed methane concentration sequence is suppressed, and the concentration change trend is clearer, thereby providing reliable "baseline data" for subsequent threshold detection.

[0096] S232, performing a sliding baseline correction on the smoke sensor data to obtain a relative smoke concentration sequence.

[0097] The smoke sensor is susceptible to factors such as sensor aging and environmental adaptation deviation, so that the measured value is affected by long-term drift. Direct use of the original measured value can cause "false positives", such as misjudging the sensor drift as an increase in smoke concentration, or "false negatives", such as the real smoke concentration increase being masked by the drift. By calculating a dynamic baseline, the original measured value is converted into a "relative concentration", eliminating the influence of long-term drift.

[0098] The method of performing a sliding baseline correction on the smoke sensor data can adopt an exponential weighted sliding baseline method or an adaptive statistical baseline method, which will not be described here.

[0099] S233, detecting the smoothed methane concentration sequence and the relative smoke concentration sequence according to a preset methane concentration threshold and a preset smoke relative concentration threshold, respectively, to generate a gas anomaly event marker and a concentration change rate curve.

[0100] The change rate is calculated by the concentration difference between adjacent time points, reflecting the concentration change trend over time. The methane / smoke concentration change rate curve is generated with time as the horizontal axis and the change rate as the vertical axis, which intuitively displays the development speed of the anomaly.

[0101] In the embodiments of the present application, the preprocessing of the gas concentration data is a key link of the fire / gas leakage monitoring system. The preprocessing can eliminate noise interference, correct sensor drift and extract effective features, thereby providing a reliable data basis for subsequent anomaly detection.

[0102] S300, performing fire feature extraction on the preprocessed multi-modal environment data to obtain multi-modal initial fire features; the multi-modal initial fire features include visible light image features, infrared thermal imaging features and gas concentration features.

[0103] The multi-modal initial fire features refer to the multi-dimensional physical features that appear simultaneously in the initial stage of the kitchen fire (within 0-10 seconds after the flame appears) and can be captured by multi-sensor or image analysis technology, including but not limited to: visual features such as flame color and shape, thermal features such as temperature gradient and high-temperature region distribution, dynamic features such as flame area growth rate, oil droplet splashing speed and direction, material features such as oil droplet particle size and gas diffusion mode.

[0104] For the recognition needs of visible light images, infrared thermal imaging, gas concentration and other multi-modal initial fire characteristics, the deep learning model needs to integrate the complementary information of different modalities through a multi-modal fusion architecture.

[0105] In S310, a light CNN network is used to extract a spatial feature map from the preprocessed visible light image data, and a visible light image feature is obtained.

[0106] A light CNN is used to extract a spatial feature map, and static information such as flame morphology and color distribution is retained. The light CNN can be MobileNetV3.

[0107] In S320, a spatio-temporal convolution network is used to extract a spatio-temporal thermal feature from the preprocessed infrared thermal imaging data, and an infrared thermal imaging feature is obtained.

[0108] A spatio-temporal convolution network is used to extract a spatio-temporal thermal feature, and the spatio-temporal pattern of flame diffusion is captured. The spatio-temporal convolution network can be 3D ResNet-18.

[0109] In S330, a bidirectional LSTM is used to extract a time series dynamic feature from the preprocessed gas concentration data, and a gas concentration feature is obtained.

[0110] A bidirectional LSTM is used to extract a time series dynamic feature, and the mutation of gas concentration, such as a sudden increase in methane concentration, is modeled. The bidirectional LSTM can be BiLSTM.

[0111] In S400, the multi-modal initial fire features are input into a multi-modal feature fusion deep learning model, and a determination result of the initial fire type in the kitchen is output.

[0112] In view of the heterogeneity of multi-modal initial fire features, the embodiment of the present application designs a multi-modal feature fusion deep learning model, which realizes efficient fusion of multi-modal features and fire discrimination through three modules of heterogeneous modal alignment, dynamic-static interaction and cross-modal attention.

[0113] Since the extracted visible light image features, infrared thermal imaging features and gas concentration features have different modal feature dimensions and semantics, they need to be mapped to a unified dimensional semantic space through a cross-modal projection layer. Specifically: The projection representation of the visible light image feature is:

[0114] where F vis is the input visible light image feature vector; W vis is the visible light weight matrix; C is the number of channels of the input feature; D is the target feature dimension after projection; bvis is a visible light bias vector; F vis is a projected visible light image feature. Infrared thermal imaging feature projection is expressed as:

[0115] wherein, GlobalAvgPool3D is a three-dimensional global average pooling operation; W ir is an infrared thermal imaging weight matrix; b ir is an infrared thermal imaging bias vector; F ir is an infrared thermal imaging feature tensor; F ir is a projected infrared thermal imaging feature.

[0116] Gas concentration feature projection is expressed as:

[0117] wherein, W gas is a weight parameter of linear transformation; GlobalAvgPool1D(F gas ) represents a result after one-dimensional global average pooling is performed on the gas concentration feature F gas ; b gas is a bias parameter of gas concentration; F gas is a projected gas concentration feature.

[0118] In view of the differences among the static visual feature (F vis ), the dynamic infrared thermal imaging feature (F ir ) and the dynamic gas concentration feature (F gas ), the embodiment of the present application dynamically adjusts the fusion weight by designing a gating mechanism, and the specific implementation is as follows: Generate gating weight through MLP (multi-layer perceptron) g ∈[0,1] 3 , reflecting the dynamic degree of the current scene (such as g 1 controls the weight of the static feature F vis , g 2 / g 3 controls the weight of the dynamic feature F ir and F gas respectively):

[0119] wherein, σ is a sigmoid activation function; W g is a weight matrix; bgis the bias term; [⋅;⋅] denotes concatenation.

[0120] The static features and dynamic features are weighted and summed by the gating weight to obtain the fusion feature F fusion :

[0121] The formula dynamically models the input features by the MLP network, uses the sigmoid function to constrain the network output to reasonable weight values, and finally realizes the dynamic fusion of static and dynamic features. This mechanism enables the model to dynamically adjust feature weights according to the scene, improving the adaptability to complex environments.

[0122] In addition, to show the key association between modeling modalities, such as the correspondence between flame shape and high-temperature area, a multi-head cross-attention layer needs to be introduced: (1) Take the fusion feature F fusion as the query to extract the global semantics; (2) Use the projected features F′ vis ,F′ ir ,F′ gas of each modality as the key and value, respectively; (3) Calculate the association weight between the query and the key by multi-head attention, aggregate the values by weight, and enhance the complementary information between modalities:

[0123] where Q is the query vector, and Q=F fusion Wq ; K is the key vector, and K=[ F′ vis ; F′ ir ; F′ gas ] W k ; V is the value vector, and V=[ F′ vis ; F′ ir ; F′ gas ] W v ; D is the feature dimension of Q, K, and V; W q , W k , W v ∈R D × D ; is the similarity between the query and the key.

[0124] The fused feature F fusion is subjected to global average pooling to obtain the global feature vector vglobal ∈R D , input the full connection layer to output the classification probability:

[0125] wherein y∈R C ( C represents the probability of each type, and the total number of fire types is N. W cls , b cls are the weights and biases of the classification layer; v global is the unified semantic feature vector after multi-modal feature fusion.

[0126] It should be noted that the feature extraction model and the deep learning network model involved in the embodiments of the present application are all models pre-trained using a large number of fire type samples. Therefore, by this step, the final output is the classification result of whether the initial fire in the kitchen is normal cooking or abnormal fire, and the specific initial fire type classification result of abnormal fire, such as oil fire, electrical short circuit fire, gas leakage fire, etc. In order to facilitate subsequent targeted fire extinguishing equipment or measures for fire extinguishing according to different fire types.

[0127] The intelligent identification method for the initial fire type in the kitchen provided by the embodiments of the present application simultaneously fuses three types of heterogeneous data of visible light images, infrared thermal images, and gas concentration, which makes up for the limitations of single modal and improves the comprehensiveness of fire type identification; through a multi-stage preprocessing process, the data quality is optimized, the reliability of subsequent feature extraction is improved, and the signal-to-noise ratio and stability of the preprocessed data are greatly improved, laying a reliable foundation for subsequent feature extraction; through noise reduction processing of the HSV channel vector images respectively, the accuracy of initial fire type identification in the complex environment of the kitchen is improved.

[0128] In a second aspect, the present application provides an intelligent identification system for the initial fire type in the kitchen, please see Figure 5 , Figure 5 is a module block diagram of an intelligent identification system for the initial fire type in the kitchen provided by the embodiments of the present application. Corresponding to the method of the above-mentioned embodiments, the intelligent identification system for the initial fire type in the kitchen of the embodiments of the present application comprises: A multi-sensor fusion perception module 1 is used to acquire multi-modal environment data in the kitchen; the multi-modal environment data includes visible light image data, infrared thermal image data, and gas concentration data.

[0129] A multi-modal data preprocessing module 2 is used to preprocess the multi-modal environment data respectively.

[0130] A fire feature extraction module 3 is configured to extract fire features from the preprocessed multi-modal environment data to obtain multi-modal initial fire features, which include visible light image features, infrared thermal imaging features and gas concentration features.

[0131] A fire type identification module 4 is configured to input the multi-modal initial fire features into a multi-modal feature fusion deep learning model to output a determination result of an initial fire type in the kitchen.

[0132] The intelligent identification system of the initial fire type in the kitchen according to the embodiment of the present application, the specific implementation modes and functions of the modules correspond to the above method, and will not be described here.

[0133] Based on the multi-modal fusion kitchen initial fire type identification scheme provided by the embodiment of the present application, the effect verification is carried out through the following test.

[0134] 1. Hardware: Jetson AGX Xavier edge computing platform.

[0135] 2. Dataset: self-built kitchen fire multi-modal dataset, which contains various scene data such as oil pot fire, electric short circuit, gas leakage and normal cooking fire, wherein the normal cooking sample: 1,200 groups, containing oil smoke, steam, strong light interference scene, fire sample: 800 groups, initial flame height <10 cm, duration <10 s.

[0136] 3. Evaluation index: Accuracy = (TP+TN) / (TP+TN+FP+FN) False alarm rate = FP / (FP+TN) Wherein, TP is the correct alarm of real fire, FP is the false alarm of normal scene.

[0137] By using the scheme of the present application, the initial fire identification rate is improved from 72% of the traditional smoke sensor to 96%; the average response time is shortened to 3.2 seconds (the traditional system needs more than 10 seconds); the fire extinguishing efficiency of gas leakage type fire is improved by 40%.

[0138] The above content is a further detailed description of the present application in combination with a specific preferred embodiment, and cannot be regarded as limiting the specific implementation of the present application to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be regarded as falling within the protection scope of the present application.

Claims

1. An intelligent method for identifying the initial fire type in a kitchen, characterized in that: Including steps: Acquiring multimodal environmental data in the kitchen; the multimodal environmental data includes visible light image data, infrared thermal imaging data, and gas concentration data; Preprocessing the multimodal environmental data respectively; Extracting fire features from the pre-processed multimodal environmental data to obtain initial multimodal fire features, wherein the initial multimodal fire features include visible light image features, infrared thermal imaging features, and gas concentration features; Inputting the multimodal initial fire features into a multimodal feature fusion deep learning model to output a determination result of the initial kitchen fire type; Among them, in the step of preprocessing the multimodal environmental data, the method of preprocessing the visible light data includes: performing median filtering on the visible light image data to eliminate salt and pepper noise in the visible light image data; Adaptive histogram equalization is performed on the visible light image data after removing salt and pepper noise to enhance the contrast between the flame and the background. Convert the contrast-enhanced visible light image data from RGB color space to HSV color space to decouple color and brightness information; Performing component sub-band noise reduction processing on each channel of the HSV color space; Dynamic threshold segmentation is performed on the HSV color space after noise reduction to segment the flame area.

2. The method according to claim 1, characterized in that In the step of preprocessing the multimodal environmental data, the method of preprocessing the infrared thermal imaging data includes: Based on a pre-calibrated blackbody emissivity, the raw signal of the infrared thermal imaging data is converted into a calibration temperature matrix; Adopting an adaptive Gaussian filter to process the calibration temperature matrix and outputting a smoothed temperature matrix after the adaptive Gaussian filter processing; Using the Sobel operator edge detection method, the smoothed temperature matrix after the adaptive Gaussian filtering is processed to output a gradient temperature matrix; Based on the smoothed temperature matrix processed by the adaptive Gaussian filter, according to a preset dynamic temperature threshold, extracting a set of candidate temperature points whose temperatures are greater than the preset dynamic temperature threshold; Based on the gradient temperature matrix, the candidate temperature point set is further screened according to a preset gradient threshold, and temperature points with gradient values ​​greater than the preset gradient threshold are retained to determine a high temperature area.

3. The method according to claim 1, characterized in that In the step of preprocessing the multimodal environmental data, a method for preprocessing the gas concentration data includes: The time series data collected by the methane sensor is filtered to obtain a smoothed methane concentration series; Perform sliding baseline correction on the smoke sensor data to obtain a relative smoke concentration series; According to a preset methane concentration threshold and a preset smoke relative concentration threshold, the smoothed methane concentration sequence and the relative smoke concentration sequence are detected respectively to generate a gas anomaly event mark and a concentration change rate curve.

4. The method according to claim 1, wherein The performing component sub-band noise reduction processing on each channel of the HSV color space includes: Performing noise reduction processing on the H channel image of the HSV color space; Performing noise reduction processing on the S channel image and the V channel image of the HSV color space; Merge the H channel image, V channel image and S channel image after noise reduction.

5. The method according to claim 4, characterized in that The performing noise reduction processing on the H channel image of the HSV color space includes: For each image block of the H channel image, transform from the spatial domain to the frequency domain to obtain a frequency domain transformation matrix; Obtaining gradient values ​​of the frequency domain transformation matrix in N directions, where N ≥ 4; Determine the maximum gradient value and the minimum gradient value among the gradient values ​​obtained in the N directions; Determining the number of tailing coefficients to be removed according to a relationship between the maximum gradient value, the minimum gradient value, and a plurality of thresholds; Performing tailing coefficient removal on the transformation matrix according to the number of tailing coefficients to be removed; The frequency domain matrix after removing the tailing coefficients is transformed from the frequency domain to the spatial domain to obtain the denoised H channel image.

6. The method according to claim 4, characterized in that The performing noise reduction processing on the V channel image and the S channel image of the HSV color space includes: The V channel image and the S channel image are subjected to filtering and noise reduction processing by using a saturation and brightness combined bilateral filtering method, wherein the formula for the saturation and brightness combined bilateral filtering is as follows: ； In the formula, is the bilateral filter weight, is the spatial domain weight, is the range weight; ( k , l ) is the coordinate of the current pixel to be processed, ( i , j ) is the coordinate of the reference point pixel, σ d Is the spatial domain smoothing intensity parameter; Is( k , l ) is the coordinate ( k , l ) at the saturation value, Is( i , j ) is the reference point ( i , j ) at the saturation value, Iv( k , l ) is the coordinate ( k , l ) at the brightness value, Iv( i , j ) is the reference point ( i , j ), σ r is the range sensitivity parameter.

7. The method according to claim 5, characterized in that The gradient values ​​of the frequency domain transformation matrix in N directions are obtained, wherein when N=4, the formula used is as follows: ； In the formula, Represents the gradient value in the horizontal direction on the right side of the frequency domain matrix, Represents the gradient value in the vertical direction of the lower side of the frequency domain matrix, Represents the gradient value at the lower left of the frequency domain matrix, represents the gradient value at the lower right, Represents the first p The elements in the qth row and qth column, the size of the frequency domain matrix is ​​m×n.

8. The method according to claim 1, characterized in that The fire feature extraction is performed on the pre-processed multi-modal environmental data to obtain each multi-modal initial fire feature, including: A lightweight CNN network is used to extract spatial feature maps from the preprocessed visible light image data to obtain the visible light image features; The preprocessed infrared thermal imaging data is subjected to a spatiotemporal convolutional network to extract spatiotemporal thermal features to obtain the infrared thermal imaging features; The preprocessed gas concentration data is subjected to a bidirectional LSTM to extract the temporal dynamic features and obtain the gas concentration features.

9. An intelligent system for identifying the type of initial kitchen fire, applying the method according to any one of claims 1 to 8, comprising: A multi-sensor fusion perception module is used to obtain multimodal environmental data in the kitchen; the multimodal environmental data includes visible light image data, infrared thermal imaging data, and gas concentration data; A multimodal data preprocessing module, configured to preprocess the multimodal environmental data separately; A fire feature extraction module is used to extract fire features from the pre-processed multimodal environmental data to obtain each multimodal initial fire feature; the multimodal initial fire features include visible light image features, infrared thermal imaging features, and gas concentration features; The fire type recognition module is used to input the multimodal initial fire features into a deep learning model of multimodal feature fusion and output a determination result of the initial kitchen fire type.

Images