Low-cost intelligent fire monitoring system and method based on multi-modal data fusion

Abstract

The invention discloses a low-cost intelligent fire monitoring system and method based on multi-modal data fusion. The system comprises a data acquisition module, a data preprocessing module, a feature extraction module, a data processing module, a dynamic threshold generation module and a fire alarm module. The data acquisition module is used for acquiring data; the data preprocessing module is used for carrying out standardization, denoising and time synchronization on the acquired data of each sensor; the feature extraction module extracts features corresponding to the flame data; the data processing module obtains a classification result whether there is fire or not; the dynamic threshold generation module generates a fire alarm dynamic threshold; and the fire alarm module is used for comparing the final fire probability value with a fire alarm dynamic threshold value and sending out an alarm signal when the probability value is greater than the fire alarm dynamic threshold value, and manual fire extinguishing needs to be carried out in time. According to the method, effective fusion of multi-sensor data is realized, corresponding threshold values can be adaptively adjusted, and the fire prevention capability is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of fire detection, and in particular to a low-cost intelligent fire monitoring system and method based on multimodal data fusion. Background Art

[0002] Current fire detection technologies mainly include image-based video fire detectors, multi-band infrared detectors, multi-band ultraviolet detectors, and composite fire detectors. Image-based video fire detectors analyze images through video recognition algorithms to detect the motion, grayscale, and brightness characteristics of flames or smoke. When the analysis results match the preset fire feature library, it is determined to be a fire. However, this method is easily affected by dynamic factors on site, such as personnel movement or light source changes (such as large-screen displays), resulting in a high false alarm rate.

[0003] Multi-band infrared detectors and multi-band ultraviolet detectors are specifically designed to monitor infrared or ultraviolet radiation from flames. Although they can directly respond to flame signals, they can also cause false alarms in strong light sources such as direct sunlight or arc light from welding operations.

[0004] The composite fire detector uses a collaborative working mode of image sensors, multi-band infrared detectors and multi-band ultraviolet detectors. When any detector detects a suspected fire signal, the system cross-verifies the information from the other two detectors to ensure that the alarm is triggered only when the comprehensively assessed fire probability exceeds the preset threshold, thereby enhancing the stability and reliability of the system. However, even so, this system still has obvious limitations. The preset fire probability threshold is a static value that does not take into account the dynamic changes in environmental factors in the detection area, such as light intensity and temperature fluctuations. Under different environmental conditions, the performance of fire characteristics will be different: in a strong light environment, the visible characteristics of the flame may be masked, resulting in a detection probability below the threshold, even if a fire actually occurs; while under lower temperature conditions, although the risk of fire is reduced, the detector may be too sensitive and false alarm. Therefore, the fixed threshold setting may lead to missed or false alarms in practical applications, reducing the overall effectiveness of the system.

[0005] In summary, although existing fire detection technologies have their own advantages, they generally have defects such as high false alarm rate, lack of early warning function and inability to adapt to changing environments, especially in complex environmental conditions, such as scenes with frequent personnel flow or interference from multiple light sources. In order to overcome these limitations, it is urgent to develop more advanced and intelligent fire detection systems that can effectively integrate multiple sensor information, improve the accuracy and timeliness of detection, and especially respond in time at the initial stage of a fire, thereby minimizing the risk and damage of fire. Summary of the invention

[0006] In order to overcome the above technical problems, the purpose of the present invention is to provide a low-cost intelligent fire monitoring system and method based on multimodal data fusion. The detection system and method realize the effective fusion of multi-sensor data, and can adaptively adjust the corresponding thresholds, thereby improving the fire prevention capability.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A low-cost intelligent fire monitoring system based on multimodal data fusion, comprising a data acquisition module, a data preprocessing module, a feature extraction module, a data processing module, a dynamic threshold generation module and a fire alarm module;

[0009] The data acquisition module collects data through sensors and visible light lenses;

[0010] The data preprocessing module is used to standardize, denoise and time synchronize the collected sensor data;

[0011] The feature extraction module extracts features corresponding to the flame data collected by the sensor and the visible light lens respectively;

[0012] The data processing module inputs the four flame features extracted by the feature extraction module into the trained classification model, and finally obtains the classification result of whether there is fire;

[0013] The dynamic threshold value generation module obtains the real-time temperature and the real-time illuminance through the data acquisition module, and updates the fire alarm threshold value according to these two values ​​to generate a fire alarm dynamic threshold value;

[0014] The fire alarm module compares the final fire probability value obtained by the data processing module with the fire alarm dynamic threshold, and sends out an alarm signal when the probability value is greater than the fire alarm dynamic threshold, so as to warn the staff that there is a high possibility of fire in the detection area and manual fire extinguishing is required in time.

[0015] In the data acquisition module, the sensors and devices in the data acquisition module are all placed on the data acquisition board 1, with a visible light lens 4 placed at the top of the acquisition board 1, an IR-CUT 3 placed under the visible light lens 4, a 4.3μm infrared sensor 5 placed at the bottom left, a temperature sensor 6 and a light sensor 8 placed to the right of the infrared sensor 5, wherein the temperature sensor 6 is placed above the light sensor 8, a 3.8μm infrared sensor 7 is placed at the bottom right of the visible light lens 4, and an ultraviolet sensor 9 is placed at the bottom of the acquisition board 1. All of the above sensors and devices, except the temperature sensor 6, are placed under a special transparent protective cover 2 with stable optical performance;

[0016] The data acquisition module includes six sensors and an IR-CUT3 device, wherein the two infrared sensors are used to capture infrared rays with wavelengths of 3.8 μm and 4.3 μm in the detection area in real time, respectively; the ultraviolet sensor 9 is used to receive ultraviolet rays with wavelengths of 0.185 μm to 0.3 μm; the IR-CUT3 is used to switch the filter of the visible light lens 4; the visible light lens 4 is used to cooperate with the IR-CUT3 to collect image information of the detection area in real time; the temperature sensor 6 is used to collect the current ambient temperature; and the light sensor 8 is used to collect the current light intensity;

[0017] The data preprocessing module standardizes, removes noise and synchronizes the data collected by the infrared sensor 1 5, the infrared sensor 2 7, the ultraviolet sensor 9 and the visible light lens 4, so as to facilitate subsequent feature extraction and fusion;

[0018] Rockchip RV1126 is used as the main control chip, which is electrically connected to the data acquisition module. The data preprocessing module and the feature extraction module are both operated in the main control chip.

[0019] Multivariate scattering correction is used to preprocess the infrared and ultraviolet spectral data of the infrared sensor and the ultraviolet sensor 9, and the formula is as follows:

[0020] Get the average value of all spectral data as the "ideal spectrum":

[0021]

[0022] in, is the average value of all spectral data, Data ij is each spectral data, n is the number of spectral data collected this time;

[0023] Perform a univariate linear regression on the spectrum of each sample and the average spectrum, and solve the least squares problem to obtain the baseline shift and offset of each sample:

[0024]

[0025] Among them, Data i is the theoretical value of the spectral data at position i, k i is the baseline shift for each sample, b i is the baseline offset for each sample;

[0026] The spectrum of each sample is corrected by subtracting the obtained baseline shift and dividing by the offset to obtain the corrected spectrum:

[0027]

[0028] The image collected by the visible light lens 4 on a daily basis is an ultraviolet image, and the data of the "Y" channel is used as the grayscale value to generate a grayscale image. The feature extraction module extracts the corresponding features of the flame data in the infrared sensor 1 5, the infrared sensor 2 7, the ultraviolet sensor 9 and the grayscale image respectively;

[0029] In the feature extraction module, the analog quantities collected by the three optical sensors are converted into corresponding digital quantities, that is, the optical characteristics of the flame are extracted; the number of gray value changes in a certain area of ​​the ultraviolet image for 20 consecutive frames is counted, that is, the movement and flickering frequency of the flame ultraviolet rays are extracted;

[0030] After the two infrared sensors 1 5, the infrared sensor 2 7 and the ultraviolet sensor 9 collect infrared and ultraviolet rays of specific wavelengths, they output corresponding analog quantities, and the three sensors are respectively connected to ADC channels 0, 1, and 2 of the main control chip, so that the main control chip can read the output signals of the three sensors and convert them into corresponding voltage values, that is, extract the optical characteristics of the flame; the ultraviolet image can detect the flame by changing the gray value frequency of a certain area for several consecutive frames, that is, extract the movement and flickering characteristics of the flame ultraviolet rays;

[0031] The conversion relationship between the analog quantity collected by the infrared sensor 1 5, the infrared sensor 2 7 and the ultraviolet sensor 9 and the corresponding voltage value is as follows:

[0032]

[0033] Among them, V ref is the standard voltage, n is the number of bits for AD conversion, V result It is the collected voltage required by the user, and raw is the original data collected by AD;

[0034] The grayscale value change frequency of a certain area of ​​the grayscale image is the flickering feature of the flame. The feature is that the grayscale value of the pixel at the edge of the flame changes significantly between frames. The absolute value of the grayscale value difference of a certain pixel between frames is taken, and whether it is a flame is determined based on the absolute value and the number of changes;

[0035] The calculation steps are as follows:

[0036] First, two counting matrices map1 and map2 with the same resolution as the original image are established. Secondly, at time t, when the gray value difference between the current frame and the previous frame of a certain pixel (x, y) is greater than a threshold T, map1(x, y, t) is added by 1 on the basis of the previous frame map1(x, y, t-1) for accumulation, and map2(x, y, t) is added by 0; when the gray value difference between the current frame and the previous frame (x, y) is less than a threshold minus T, map2(x, y, t) is added by 1 and map1(x, y, t) is added by 0; if it is not the above two cases, both map1(x, y, t) and map2(x, y, t) are added by 0;

[0037]

[0038] ΔI(x,y,t)=R(x,y,t)-R(x,y,t-1)

[0039] Where T is a positive real number used to eliminate the influence of system noise and other factors on the pixel grayscale value. The value of T is generally between 5 and 20. R(x,y,t) and R(x,y,t-1) are the grayscale values ​​of the pixel (x,y) at time t and time t-1 respectively.

[0040] Finally, in the surveillance video image sequence, the grayscale value of the stationary target is basically unchanged, and the moving car lights will cause the pixel grayscale value to change greatly, but these changes are short-lived and generally continue to increase or decrease, while the grayscale value of the flame pixel will increase and decrease repeatedly, and the changes of map1(x,y,t) and map2(x,y,t) will exceed a certain threshold. If the changes of map1(x,y,t) and map2(x,y,t) of a certain pixel (x,y) exceed a certain threshold within a certain period of time, then this pixel may be the edge pixel of the flame;

[0041] The flickering characteristics of a flame are as follows:

[0042] map1(x,y,t)=1COUNT1=map1(x,y,t)-map1(x,y,tN)

[0043] map2(x,y,t)=1COUNT2=map2(x,y,t)-map2(x,y,tN)

[0044]

[0045] In the formula, MAP, MAP1 and MAP2 are binary matrices with the same resolution as the original image, and the symbol Represents a binary AND operation; N is the accumulation time window, which is equivalent to the video acquisition rate;

[0046] map1(x,y,t), map2(x,y,t) and map1(x,y,tN), map2(x,y,tN) are the count values ​​of the pixel (x,y) at time t and tN respectively, COUNT1 is the number of times the grayscale value difference of the pixel in adjacent frames within 20 frames is greater than a certain threshold T, COUNT2 is the number of times the grayscale value difference of the pixel in adjacent frames within 20 frames is less than a certain threshold minus T. The data processing module is used to input the four flame features extracted by the feature extraction module into the trained classification model, and finally obtain the classification result of whether there is fire;

[0047] In the data processing module, the gated recurrent unit (GRU) is used as the main framework of the algorithm. The core of GRU lies in its innovative gate design, including reset gate and update gate. Through the gate mechanism, GRU can automatically learn which time points are critical for predicting fires, while ignoring irrelevant or redundant information. This dynamic memory management capability enables GRU to extract meaningful patterns from sequence data, even if these patterns are distributed over a long time span, to solve the problems of RNN's inability to remember long-term and gradients in back propagation. Finally, the output of the neural network is passed to a fully connected layer to complete the process from the GRU hidden state h t The model is then quantified and deployed on the edge computing platform to complete real-time fire detection.

[0048] In the dynamic threshold generation module, the setting standard is that under the environment of preset temperature T0 and preset light intensity E0, if the output fire probability value p is higher than the preset fire alarm threshold TTH0, it indicates that there is a high possibility of fire in the detection area, and an alarm signal should be issued in time to prompt the staff to check the detection area.

[0049] In GRU, the hidden state update process is divided into three steps:

[0050] First, the reset gate is calculated, then the update gate is calculated, and finally the values ​​of these two gates are combined to update the hidden state;

[0051] Reset gate (r t ) is calculated as follows:

[0052] r t =σ(W r x t +U r h t-1 +b r )

[0053] Among them, σ is the sigmoid activation function, W r and U rare the weight matrices from input and hidden state to reset gate, b r is the bias term, x t is the input at the current moment, h t-1 is the hidden state at the previous moment;

[0054] Update gate (z t ) is calculated as follows:

[0055] z t =σ(W z x t +U z h t-1 +b z )

[0056] Among them, W z and U z are the weight matrices from input and hidden state to update gate, b z is the bias term x t ;

[0057] Candidate hidden states The calculation formula is as follows:

[0058]

[0059] Among them, tanh is the hyperbolic tangent activation function, W h and U h are the weight matrices from the hidden state adjusted by the input and reset gates to the candidate hidden state, and b h is the bias term, ⊙ represents element-wise multiplication;

[0060] The final hidden state (h t ) is calculated as follows:

[0061]

[0062] The final output of the GRU is usually associated with a hidden state, which is passed to other layers or used as the final output.

[0063] Use a fully connected layer with an activation function to complete the transformation from the GRU hidden state h t Mapping to class probabilities;

[0064] Fully connected layer:

[0065] o=W fc h T +b fc

[0066] Among them, o is the output of the fully connected layer; W fcis the weight matrix of the fully connected layer, whose dimension is H×C, where H is the dimension of the GRU hidden state and C is the number of categories; b fc is the bias vector of the fully connected layer, whose dimension is C; h T is the hidden state of GRU at the last time step of the sequence;

[0067] The activation function is the Softmax function, which converts the output into a probability distribution:

[0068] p = softmax(o)

[0069] Among them, p is a C-dimensional probability vector, and each element represents the probability of the corresponding category, that is, the fire probability.

[0070] After the temperature sensor 6 and the light sensor 8 collect the ambient temperature and light intensity, they output the corresponding analog quantities. The main control chip RV1126 used in the system includes a 6-channel ADC, where channels 0, 1, 2, 3, and 5 are used for common analog quantity input. Therefore, the signal output pins of the above two sensors are respectively connected to ADC channels 3 and 5 of the main control chip, so that the main control chip RV1126 can convert the output signals of the two sensors into corresponding voltage values, thereby representing the real-time temperature and real-time light intensity of the current environment, and update the fire alarm threshold according to these two values ​​to generate a dynamic fire alarm threshold;

[0071] The update formula is as follows:

[0072]

[0073] Wherein, T0 represents the preset temperature, E0 represents the preset light intensity, and TTH0 represents the preset fire alarm threshold.

[0074] The setting standard is that under the environment of preset temperature T0 and preset light intensity E0, if the output fire probability value p is higher than the preset fire alarm threshold TTH0, it indicates that there is a high possibility of fire in the detection area, and an alarm signal should be issued in time to prompt the staff to check the detection area;

[0075] T represents the real-time temperature, E represents the real-time light intensity, and the real-time temperature T and the real-time light intensity E are both obtained through the data acquisition module; τ represents the ratio between the temperature change degree and the light intensity change degree. The larger the ratio τ is, the greater the temperature change degree is, the greater the weight τ*β1 of the subsequent temperature change degree is; conversely, the smaller the ratio τ is, the smaller the temperature change degree is, the smaller the light intensity change degree is, the smaller the weight τ*β1 of the subsequent temperature change degree is. β1 and β2 are both preset proportional coefficients, and β1+β2=1, β1∈(0,1), β2∈(0,1).

[0076] The fire alarm module is used to compare the final fire probability value obtained by the data processing module with the fire alarm dynamic threshold, and to send out an alarm signal when the probability value is greater than the fire alarm dynamic threshold, so as to warn the staff that there is a high possibility of fire in the detection area and manual fire extinguishing is required in time.

[0077] In the fire alarm module, the alarm signal emitted by the fire alarm module includes the sound and light signal emitted by the sound and light alarm and the vibration of the alarm bracelet. The positive poles of the sound and light alarm and the alarm bracelet controller are connected to the common IO pin of the main control chip, and the negative poles are connected to the ground pin of the main control chip.

[0078] When the system detects a fire, it pulls up the IO pins connected to the sound and light alarm and the wristband controller at the same time. The sound and light alarm and the wristband controller both adopt the power-on alarm working mode. When the fire alarm module determines that an alarm needs to be sounded, the main control chip controls the corresponding IO pin to output a high level, and the sound and light alarm and the wristband controller can generate corresponding alarm signals, thereby controlling the corresponding alarm phenomenon. However, since the vibration of the alarm bracelet needs to continuously emit infrared rays for control, the IO pin that controls the wristband controller needs to be repeatedly pulled high and low at fixed time intervals, with an interval of 5s.

[0079] A detection method for a low-cost intelligent fire monitoring system based on multimodal data fusion; comprising the following steps;

[0080] Step 1: Collect raw data through the data acquisition module, including the current ambient temperature, light intensity, 3.8μm and 4.3μm infrared intensity, ultraviolet intensity and ultraviolet images of specific wavelengths;

[0081] Step 2: Send the collected data to the data preprocessing module to standardize the raw data, remove noise, and synchronize time;

[0082] Step 3: Send the processed data to the feature extraction module to extract fire features. Specifically, the optical features, motion and flicker features of the fire are extracted in this module.

[0083] Step 4: Send the extracted features to the data processing module, which contains a machine learning model GRU. The model can classify the above features in combination with time series information and output the probability value of whether a fire occurs in the current environment;

[0084] Step 5: The dynamic threshold generation module calculates the threshold of fire in the current environment based on the collected temperature and light intensity. This value is compared with the fire probability value output by the alarm module and the data processing module, and finally a conclusion is given on whether a fire has occurred, and corresponding measures are taken, that is, an alarm signal is generated or the next frame is detected.

[0085] Under normal circumstances, IR-CUT3 is in the ultraviolet filter working state, and the visible light lens 4 collects ultraviolet images. The customized ultraviolet filter can filter out some interference such as sunlight. At the same time, the two infrared sensors 1 5, infrared sensor 2 7 and ultraviolet sensor 9 are also constantly monitoring the infrared and ultraviolet rays in the environment. These data are sent to the machine learning model after data preprocessing and feature extraction modules, and the fire is judged by combining the four features. Once the system detects a fire, IR-CUT3 will immediately switch to the full-transmittance filter working state and transmit the visible light video stream to facilitate manual monitoring and fire source positioning. This state will continue until the fire is completely extinguished, and IR-CUT3 will continue to switch to the ultraviolet filter working state to perform routine fire monitoring.

[0086] Beneficial effects of the present invention:

[0087] 1. The present invention combines high-performance optical sensors, low-cost RV1126 edge computing chips, and advanced fire detection models to achieve efficient fire monitoring at low cost. In the early stages of a fire, the characteristics are often difficult to detect, but through the collaborative work of various sensors, the system can capture subtle environmental changes, thereby significantly improving the accuracy of early fire identification. Moreover, the optical sensors used in the system have long-range detection capabilities and can accurately identify the unique spectral characteristics of flames even at a long distance, greatly expanding the effective monitoring range of the system.

[0088] 2. The system uses a relatively simple gated recurrent model (GRU) network structure, which not only speeds up fire detection, but also reduces the demand for computing resources and the time and space complexity of the system. A simplified model means faster response time, which is crucial for emergency situations such as fires where every second counts.

[0089] 3. The present invention creatively introduces a dynamic threshold mechanism to automatically adjust the fire determination criteria according to different environmental conditions. This intelligent strategy can effectively filter out non-fire related interference signals, greatly reduce false alarms and missed alarms, and ensure the stability and reliability of the system.

[0090] 4. The method proposed in the present invention not only improves the accuracy and efficiency of early fire detection, but also greatly reduces the cost and complexity of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 Block diagram of a low-cost intelligent fire monitoring system based on multimodal data fusion.

[0092] Figure 2 This is the hardware distribution diagram of the data acquisition module.

[0093] Figure 3 This is a comparison chart of the spectral data after preprocessing.

[0094] Figure 4 This is the GRU network structure diagram.

[0095] Figure 5 This is the schematic diagram of the ADC part in RV1126.

[0096] Reference numerals:

[0097] 1. Data acquisition board; 2. Special transparent protective cover; 3. Customized IR-CUT; 4. Visible light lens; 5. 4.3μm infrared sensor; 6. Temperature sensor; 7. 3.8μm infrared sensor; 8. Light sensor; 9. 0.185~0.3μm UV sensor. DETAILED DESCRIPTION

[0098] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0099] This embodiment provides a low-cost intelligent fire monitoring system and method based on multimodal data fusion, such as Figure 1 As shown, it includes the following modules:

[0100] S1, data acquisition module, the data acquisition module includes six sensors and an IR-CUT3 device, and their relative positions are as follows Figure 2 As shown, the infrared sensor 1 5 and the infrared sensor 2 7 are respectively used to capture infrared rays with wavelengths of 3.8μm and 4.3μm in the detection area in real time, the ultraviolet sensor 9 is used to receive ultraviolet rays with wavelengths of 0.185μm to 0.3μm, the IR-CUT3 is used to switch the filter of the visible light lens 4, the visible light lens 4 is used to cooperate with the IR-CUT3 to collect image information of the detection area in real time, the temperature sensor 6 is used to collect the current ambient temperature, and the light sensor 8 is used to collect the current light intensity;

[0101] Specifically, the sensors and devices in the data acquisition module are placed on the data acquisition board 1, with a visible light lens 4 placed at the top of the acquisition board, a customized IR-CUT 3 placed under the lens, and a 4.3μm infrared sensor 5 placed at the bottom left. A temperature sensor 6 and a light sensor 8 are placed on the right side of the infrared sensor 5, wherein the temperature sensor 6 is placed above the light sensor 8, a 3.8μm infrared sensor 27 is placed at the bottom right of the visible light lens 4, and an ultraviolet sensor 9 is placed at the bottom of the acquisition board. Except for the temperature sensor 6, all the above sensors and devices are placed under a special transparent protective cover 2 with stable optical performance. Among them, the infrared sensor 15 and the infrared sensor 27 are RD-913FB2 and RD-913FB1 respectively, the ultraviolet sensor 9 is UVTRONR2868, the visible light lens 4 is IMX334, the temperature sensor 6 is GY-906MLX90614ESF, the light sensor 8 is B-LUX-V30G, and the IR-CUT3 is a customized device.

[0102] Among them, since natural gas, alcohol and other hydrocarbons will continuously produce carbon dioxide gas while burning, its radiation intensity in the 4.3μm band is significantly greater than that in other bands. The infrared radiation of sunlight is less intense in the 4.3μm band due to the absorption and reflection of carbon dioxide in the atmosphere and the earth, so the infrared flame sensor uses 4.3μm as the detection band. The 3.8μm band is a reference band that is sensitive to interference sources such as artificial heat sources and background light radiation. By using the mutual characteristics between the two bands to judge the flame, the problem of signal attenuation caused by the long distance can be effectively solved; the ultraviolet rays generated by the flame combustion reach a peak at a wavelength of 0.185 to 0.260μm, so a UV sensor 9 with a receiving wavelength of 0.185μm to 0.3μm is selected, and the sensor is not sensitive to visible light, and the nickel electrode used does not react to ultraviolet rays from sunlight, which can greatly reduce false alarms.

[0103] In the fire detection system, relying solely on infrared or ultraviolet sensors 9 may generate false alarms due to various interference sources, such as welding sparks, certain types of lights or transient discharges of electrical equipment, etc. In order to improve detection accuracy and reduce false alarms, the present invention proposes to use sensors for fire detection while combining the ultraviolet image characteristics of flames for comprehensive judgment.

[0104] IR-CUT3, also known as a dual filter switcher, is used to filter light between the camera lens and the photosensitive chip. It is composed of a dual filter and a power part (which can be an electromagnetic or motor). In the present invention, the dual filters in IR-CUT3 include a UV cutoff filter and a full-transmission filter. The UV filter is kept in working state on a daily basis, allowing only ultraviolet light to pass through, so that the camera can capture the unique ultraviolet image information of the flame and cooperate with the UV filter to monitor the fire; after the system detects that a fire has occurred in a certain area, the device will automatically switch to the full-transmission filter working state, so that the collected image conforms to the visual perception of the human eye, which is convenient for manual precise positioning of the fire occurrence point.

[0105] S2, data preprocessing module, the main purpose of the data preprocessing module is to standardize, remove noise and time synchronize the data collected by the infrared sensor 1 5, the infrared sensor 2 7, the ultraviolet sensor 9 and the visible light lens 4, so as to perform feature extraction and fusion later. In the present invention, Rockchip RV1126 is used as the main control chip, which is electrically connected to the data acquisition module, and the data preprocessing module, feature extraction module, etc. are all operated in the main control chip;

[0106] Specifically, the Rockchip RV1126 is an edge computing platform that supports 4K encoding and has an independent neural network processor (NPU) with a computing power of up to 2TOPs. At the same time, the chip has a 6-channel analog-to-digital converter (ADC), each channel has an AD acquisition bit of 10 bits, and the standard voltage is 1.8V.

[0107] The data collected by the infrared sensor 1 5, the infrared sensor 2 7 and the ultraviolet sensor 9 contain signals such as baseline drift and noise in addition to pure spectrum. At the same time, there is a spectral line offset caused by factors such as sample size and environment. The current spectral data preprocessing methods include standard normal variation (Standard Normal Variate Correction, SNV), multiplicative scatter correction (Multiplicative Scatter Correction, MSC), smoothing, displacement correction, trend removal correction and derivation operations. However, after comprehensively analyzing the above methods, the present invention uses multiplicative scatter correction to preprocess the infrared and ultraviolet spectral data. The formula is as follows:

[0108] Get the average value of all spectral data as the "ideal spectrum":

[0109]

[0110] in, is the average value of all spectral data, Data ijis each spectral data, and n is the number of spectral data collected this time.

[0111] Perform a univariate linear regression on the spectrum of each sample and the average spectrum, and solve the least squares problem to obtain the baseline shift and offset of each sample:

[0112]

[0113] Among them, Data i is the theoretical value of the spectral data at position i, k i is the baseline shift for each sample, b i The baseline offset for each sample.

[0114] The spectrum of each sample is corrected by subtracting the obtained baseline shift and dividing by the offset to obtain the corrected spectrum:

[0115]

[0116] like Figure 3 As shown in the figure, after the spectral data is processed by multivariate scattering correction (MSC), the nonlinear scattering effect in the spectral data can be reduced, thereby improving the data quality. The spectral data after MSC processing is more consistent and smooth, which helps to improve the accuracy of quantitative data analysis and support more effective feature classification and pattern recognition.

[0117] The images captured by the visible light lens 4 on a daily basis are ultraviolet images, but the camera in the present invention uses a color sensor with an RGB filter array. Even if an ultraviolet filter is installed, the RGB filters on the sensor will still record the information of the three color channels of red, green and blue respectively. However, since ultraviolet rays are not within the RGB visible spectrum, the RGB filters of the sensor will not accurately capture the intensity of ultraviolet rays. On the contrary, they will produce some responses based on the interaction between ultraviolet rays and the filters, which will produce different RGB values ​​on each pixel. Since the camera outputs an image encoded in the YUV format, the brightness information represented by the "Y" channel can be regarded as a grayscale representation of the ultraviolet intensity, while the "U" and "V" channels may contain pseudo-color information generated by the interaction between ultraviolet rays and the RGB filters. Therefore, in the present invention, the data of the "Y" channel is used as the grayscale value to generate a grayscale image.

[0118] Each sensor has its own clock source and different sampling frequencies. Therefore, to fuse the data they collect, it is necessary to time-align and time-synchronize each sensor. Since this system connects all sensors to the same main control chip, the main control chip can send time instructions to synchronize the internal clocks of all peripherals to ensure that there is no cumulative drift in the time difference; then the interpolation and extrapolation method is used for soft time synchronization. Specifically, the data between two time points is interpolated, and the data outside the time points is inferred using historical data. Regarding the insertion time, the current visible light data acquisition time of each frame is used as the time point to be inserted, so as to obtain the equivalent message of the other three sensors at the same time.

[0119] S3, a feature extraction module, wherein the feature extraction module extracts features corresponding to flame data from the infrared sensor 1 5, the infrared sensor 2 7, the ultraviolet sensor 9 and the grayscale image respectively;

[0120] Specifically, after the two infrared sensors 1 5, the infrared sensor 2 7 and the ultraviolet sensor 9 collect infrared and ultraviolet rays of specific wavelengths, they output corresponding analog quantities, and the three sensors are respectively connected to ADC channels 0, 1, and 2 of the main control chip, so that the main control chip can read the output signals of the three sensors and convert them into corresponding voltage values, that is, extract the optical characteristics of the flame; the ultraviolet image can detect the flame by changing the grayscale value frequency of a certain area for several consecutive frames, that is, extract the movement and flickering characteristics of the flame ultraviolet rays.

[0121] The conversion relationship between the analog quantity collected by the infrared sensor 1 5, the infrared sensor 2 7 and the ultraviolet sensor 9 and the corresponding voltage value is as follows:

[0122]

[0123] Among them, V ref is the standard voltage, n is the number of bits for AD conversion, V result is the collected voltage required by the user, and raw is the original data collected by AD.

[0124] The grayscale value change frequency of a certain area of ​​the grayscale image is the flickering feature of the flame. This feature is that the grayscale value of the pixel at the edge of the flame changes significantly between frames, and the extraction of the motion area is generally based on the change of the pixel grayscale value between frames, so the flickering feature of the flame actually also reflects the movement of the flame. Although the flickering characteristic analysis of the flame is based on continuous multi-frame images, considering that the flame is basically shaking at the same position in a short time, the flickering feature can be used to extract the motion area of ​​the flame, that is, the flickering analysis and the motion area extraction are combined into one, so that the extracted motion area also has the flickering characteristics of the flame. Specifically, the absolute value of the grayscale value difference of a certain pixel between frames is taken, and whether it is a flame is determined based on the absolute value and the number of changes.

[0125] The calculation steps are as follows:

[0126] First, establish two counting matrices map1 and map2 with the same resolution as the original image. Secondly, at time t, when the gray value difference between the current frame and the previous frame of a certain pixel (x, y) is greater than a threshold T, map1(x, y, t) is added by 1 on the basis of the previous frame map1(x, y, t-1) for accumulation, and map2(x, y, t) is added by 0; when the gray value difference between the current frame and the previous frame (x, y) is less than a threshold minus T, map2(x, y, t) is added by 1 and map1(x, y, t) is added by 0; if it is not the above two cases, both map1(x, y, t) and map2(x, y, t) are added by 0.

[0127]

[0128]

[0129] ΔI(x,y,t)=R(x,y,t)-R(x,y,t-1)

[0130] Where T is a positive real number used to eliminate the influence of system noise and other factors on the pixel grayscale value. The value of T is generally between 5 and 20. R(x,y,t) and R(x,y,t-1) are the grayscale values ​​of the (x,y) pixel at time t and time t-1, respectively.

[0131] Finally, in the surveillance video image sequence, the grayscale value of the stationary target is basically unchanged, and the moving car lights will cause the pixel grayscale value to change greatly, but these changes are short-lived. In general, continuous increase will continue to decrease, while the grayscale value of the flame pixel will repeatedly increase and decrease, and the changes of map1(x,y,t) and map2(x,y,t) will exceed a certain threshold. If the changes of map1(x,y,t) and map2(x,y,t) of a certain pixel (x,y) exceed a certain threshold within a certain period of time, then this pixel may be the edge pixel of the flame.

[0132] The flickering characteristics of a flame are as follows:

[0133] map1(x,y,t)=1COUNT1=map1(x,y,t)-map1(x,y,tN)

[0134] map2(x,y,t)=1COUNT2=map2(x,y,t)-map2(x,y,tN)

[0135]

[0136] In the formula, MAP, MAP1 and MAP2 are binary matrices with the same resolution as the original image, and the symbol Represents a binary AND operation; N is the accumulation time window, which is equivalent to the video acquisition rate;

[0137] Since the video acquisition rate is generally 25fps, and the flickering frequency of the flame is 7-12Hz, it can be judged that a fire has occurred if the flame flickers about 6 times in 20 frames, so 19 is taken in the present invention;

[0138] map1(x,y,t), map2(x,y,t) and map1(x,y,tN), map2(x,y,tN) are the count values ​​of the pixel (x,y) at time t and tN respectively. COUNT1 is the number of times the grayscale value difference of the pixel in adjacent frames within 20 frames is greater than a threshold T, and COUNT2 is the number of times the grayscale value difference of the pixel in adjacent frames within 20 frames is less than a threshold minus T.

[0139] S4, a data processing module, the data processing module is used to input the four flame features extracted by the feature extraction module into the trained classification model, and finally obtain a classification result of whether there is fire;

[0140] Specifically, the classification model is constructed with the gated recurrent unit (GRU) as the main algorithm framework, and its model framework is as follows: Figure 4 As shown, the input data x t Enter the GRU unit and reset gate r t and update gate z t Calculate their respective values ​​and reset gate r t The state of the controller at the previous moment h t-1 With the current input x t Combined approach to generate candidate hidden states Update gate z t Determine how to mix the state h of the previous moment t-1 and candidate hidden states To generate the current state h t, the current state h t It is passed to the next GRU unit for final prediction. The σ in the figure is the sigmoid activation function. This network is a type of recurrent neural network that can solve problems such as the inability of RNN to remember long-term memories and gradients in back propagation. It has a similar function to LSTM, but is simpler than LSTM and easier to train.

[0141] In fire detection systems, light sensors and cameras continuously collect environmental data. These data constitute a set of time-varying sequences that contain important information about the development of fires, such as a sharp increase in ultraviolet or infrared light of a certain wavelength. However, these data may be subject to noise interference, and key signals may appear at different positions in the sequence, which requires the model to be able to identify important time points and features. Through its unique gating mechanism, GRU can automatically learn which time points are critical for predicting fires while ignoring irrelevant or redundant information. This dynamic memory management capability enables GRU to extract meaningful patterns from sequence data, even if these patterns are distributed over long time spans.

[0142] GRU includes a reset gate and an update gate; intuitively, the update gate is responsible for deciding how to mix the new state and the old state, while the reset gate controls the degree of influence of the state at the previous moment on the current state.

[0143] Specifically, the update gate outputs a value between 0 and 1 through the sigmoid function, which determines how much information of the old state will be retained; the reset gate also outputs a value through the sigmoid function, which is used to determine how much information of the previous moment needs to be forgotten or reset. If the reset gate is set to 1 and the update gate is set to 0, the standard RNN model will be obtained again.

[0144] In GRU, the hidden state update process is divided into three steps:

[0145] First, the value of the reset gate is calculated, then the value of the update gate is calculated, and finally the values ​​of these two gates are combined to update the hidden state. This enables GRU to better capture long-term dependencies in long sequences while avoiding the problem of gradient disappearance. Compared with LSTM, GRU has a simpler structure because it simplifies the input gate, output gate, and memory unit in LSTM into update gates and reset gates, thereby reducing the number of parameters and increasing training speed.

[0146] Reset gate (r t ) is calculated as follows:

[0147] r t =σ(W r xt +U r h t-1 +b r )

[0148] Among them, σ is the sigmoid activation function, W r and U r are the weight matrices from input and hidden state to reset gate, b r is the bias term, x t is the input at the current moment, h t-1 is the hidden state at the previous moment.

[0149] Update gate (z t ) is calculated as follows:

[0150] z t =σ(W z x t +U z h t-1 +b z )

[0151] Among them, W z and U z are the weight matrices from input and hidden state to update gate, b z is the bias term x t .

[0152] Candidate hidden states The calculation formula is as follows:

[0153]

[0154] Among them, tanh is the hyperbolic tangent activation function, W h and U h are the weight matrices from the hidden state adjusted by the input and reset gates to the candidate hidden state, and b h is the bias term, and ⊙ represents element-wise multiplication.

[0155] The final hidden state (h t ) is calculated as follows:

[0156]

[0157] The final output of the GRU is usually associated with a hidden state. Depending on the specific requirements of the task, the hidden state can be passed to other layers or used as the final output.

[0158] In the present invention, a fully connected layer and an activation function are used to complete the transformation from the GRU hidden state h t Mapping from y to class probabilities.

[0159] Fully connected layer:

[0160] O=W fc h T +b fc

[0161] Among them, o is the output of the fully connected layer; W fc is the weight matrix of the fully connected layer, whose dimension is H×C, where H is the dimension of the GRU hidden state and C is the number of categories; b fc is the bias vector of the fully connected layer, whose dimension is C; h T is the hidden state of the GRU at the last time step of the sequence.

[0162] The activation function is the Softmax function, which converts the output into a probability distribution:

[0163] p = softmax(o)

[0164] Wherein, p is a C-dimensional probability vector, and each element represents the probability of the corresponding category, which is the fire probability in the present invention.

[0165] According to the above-mentioned feature extraction module, the input training data of the network model contains 4 features and 1 response value, that is, 1 output value is predicted by 4 input values. After the model is trained on the PC, the model conversion tool rknn-toolkit officially provided by Rockchip is used to convert the weight file that only supports running on the PC into a model file that can run on the RV1126 edge computing platform. The conversion process includes the quantization of the weight file. The quantization algorithm is responsible for mapping FP32 data to int8 / uint8 data, which will cause a certain loss of precision. By checking the network layer by layer, the present invention adopts a hybrid quantization method to minimize the loss of precision, that is, to fall back to FP32 for processing on the layers with greater impact.

[0166] The model is quantified and deployed on the RV1126 platform. The deployed model file is read through the C project file, and the two infrared voltage values, one ultraviolet voltage value and the number of grayscale value changes of a pixel in the grayscale image within 20 frames extracted by the above four sensors after passing through the feature extraction module are input into the model. After model classification, the probability value p of a fire at a certain moment is finally obtained.

[0167] S5, dynamic threshold generation module, the temperature sensor 6 and the light sensor 8 collect the ambient temperature and light intensity, and output the corresponding analog quantity. The main control chip RV1126 used in the system contains a 6-channel ADC, and each channel is defined as follows Figure 5As shown, channels 0, 1, 2, 3, and 5 can be used for common analog input, so the signal output pins of the above two sensors are connected to ADC channels 3 and 5 of the main control chip respectively, so that the main control chip can convert the output signals of the two sensors into corresponding voltage values, thereby representing the real-time temperature and real-time light intensity of the current environment, and update the fire alarm threshold according to these two values ​​to generate a dynamic fire alarm threshold;

[0168] The update formula is as follows:

[0169]

[0170] Wherein, T0 represents the preset temperature, E0 represents the preset light intensity, and TTH0 represents the preset fire alarm threshold.

[0171] The setting standard is that under the environment of preset temperature T0 and preset light intensity E0, if the output fire probability value p is higher than the preset fire alarm threshold TTH0, it indicates that there is a high possibility of fire in the detection area, and an alarm signal should be issued in time to prompt the staff to check the detection area;

[0172] As a real-time method, the preset temperature is 20 degrees, the preset light intensity is 400 lux, and the preset fire alarm threshold TTH0 is 0.65. Of course, the specific values ​​of the preset temperature T0, the preset light intensity E0, and the preset fire alarm threshold TTH0 can also be other reasonable values, which are not limited here.

[0173] T represents the real-time temperature, E represents the real-time light intensity, and the real-time temperature T and the real-time light intensity E are both obtained through the data acquisition module; τ represents the ratio between the temperature change degree and the light intensity change degree. The larger the ratio τ is, the greater the temperature change degree is, the greater the weight τ*β1 of the subsequent temperature change degree is; conversely, the smaller the ratio τ is, the smaller the temperature change degree is, the smaller the light intensity change degree is, the smaller the weight τ*β1 of the subsequent temperature change degree is. β1 and β2 are both preset proportional coefficients, and β1+β2=1, β1∈(0,1), β2∈(0,1). The larger the real-time temperature T is, the greater the risk of fire in the detection area due to the high temperature environment. At this time, the dynamic threshold value TTH of the fire alarm is reduced to improve the sensitivity of subsequent fire alarms and avoid the problem of subsequent fires occurring but no fire alarm is made. Similarly, the smaller the real-time temperature T is, the larger the dynamic threshold value TTH of the fire alarm is, to improve the reliability of subsequent fire alarms and avoid the problem of subsequent fires occurring but no fire alarm is made. The larger the real-time light intensity E is, the greater the possibility that the fire characteristics in the detection area are covered up due to strong light exposure. At this time, the dynamic threshold value TTH of the fire alarm is reduced to improve the sensitivity of subsequent fire alarms and avoid the problem of subsequent fires occurring but no fire alarm is made. Similarly, the smaller the real-time light intensity E is, the better the sensitivity of fire characteristic detection in the detection area is. At this time, the dynamic threshold value TTH of the fire alarm is increased to improve the reliability of subsequent fire alarms and avoid the problem of subsequent fires occurring but no fire alarm is made.

[0174] S6. Fire alarm module, which is used to compare the final fire probability value p obtained by the data processing module with the fire alarm dynamic threshold value TTH, and send out an alarm signal when the probability value is greater than the fire alarm dynamic threshold value, so as to warn the staff that there is a high possibility of fire in the detection area and manual fire extinguishing is required in time.

[0175] Specifically, the alarm signal emitted by the fire alarm module includes an audible and audible alarm and an alarm bracelet vibration. The audible and audible alarm model used in the present invention is QTXF-01KT / DC12V, and the alarm bracelet model is LM1901. The positive poles of the audible and audible alarm and the alarm bracelet controller are connected to the common IO pin of the main control chip, and the negative pole is connected to the ground of the main control chip. When the system detects a fire, the IO pins connected to the audible and audible alarm and the bracelet controller are pulled up at the same time, but because the vibration of the alarm bracelet needs to continuously emit infrared rays for control, the IO pins of the control bracelet controller need to be repeatedly pulled up and down at fixed intervals, and the interval time is 5s in the present invention. However, when the system is just powered on, especially during the time when the visible light lens 4 collects the first 20 frames of data, the system is not stable, and the collected data is not of reference value, so the system will not generate an alarm within 1s after powering on. The above embodiments are only preferred embodiments cited to fully illustrate the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or modification made by a person skilled in the art based on the present invention is within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A low-cost intelligent fire monitoring system based on multimodal data fusion, characterized in that: It includes a data acquisition module, a data preprocessing module, a feature extraction module, a data processing module, a dynamic threshold generation module and a fire alarm module; The data acquisition module is used to collect data through sensors and visible light lenses; The data preprocessing module is used to standardize, denoise and time synchronize the collected sensor data; The feature extraction module extracts features corresponding to the flame data collected by the sensor and the visible light lens respectively; The data processing module inputs the four flame features extracted by the feature extraction module into the trained classification model, and finally obtains the classification result of whether there is fire; The dynamic threshold value generation module obtains the real-time temperature and the real-time illuminance through the data acquisition module, and updates the fire alarm threshold value according to these two values ​​to generate a fire alarm dynamic threshold value; The fire alarm module compares the final fire probability value obtained by the data processing module with the fire alarm dynamic threshold, and sends out an alarm signal when the probability value is greater than the fire alarm dynamic threshold, so as to warn the staff that there is a high possibility of fire in the detection area and manual fire extinguishing is required in time.

2. A low-cost intelligent fire monitoring system based on multimodal data fusion according to claim 1, characterized in that: In the data acquisition module, the sensors and devices in the data acquisition module are all placed on a data acquisition board (1), a visible light lens (4) is placed at the top of the acquisition board (1), an IR-CUT (3) is placed under the visible light lens (4), a 4.3 μm infrared sensor (5) is placed at the bottom left, a temperature sensor (6) and a light sensor (8) are placed to the right of the infrared sensor (5), wherein the temperature sensor (6) is placed above the light sensor (8), a 3.8 μm infrared sensor (7) is placed at the bottom right of the visible light lens (4), and an ultraviolet sensor (9) is placed at the bottom of the acquisition board (1), and all the above sensors and devices, except the temperature sensor (6), are placed under a special transparent protective cover (2) with stable optical properties.

3. A low-cost intelligent fire monitoring system based on multimodal data fusion according to claim 2, characterized in that: The data preprocessing module standardizes, removes noise and synchronizes the time of the data collected by the infrared sensor 1 (5), the infrared sensor 2 (7), the ultraviolet sensor (9) and the visible light lens (4) for subsequent feature extraction and fusion; Rockchip RV1126 is used as the main control chip, which is electrically connected to the data acquisition module. The data preprocessing module and the feature extraction module are both operated in the main control chip; Multivariate scattering correction is used to preprocess the infrared and ultraviolet spectral data of infrared sensor 1 (5), infrared sensor 2 (7) and ultraviolet sensor (9). The formula is as follows: Get the average value of all spectral data as the "ideal spectrum": in, is the average value of all spectral data, Data ij is each spectral data, n is the number of spectral data collected this time; Perform a univariate linear regression on the spectrum of each sample and the average spectrum, and solve the least squares problem to obtain the baseline shift and offset of each sample: Among them, Data i is the theoretical value of the spectral data at position i, k i is the baseline shift for each sample, b i is the baseline offset for each sample; Correct the spectrum of each sample by subtracting the obtained baseline shift and dividing by the offset to obtain the corrected spectrum: The image collected by the visible light lens (4) on a daily basis is an ultraviolet image, and the data of the "Y" channel is used as the grayscale value to generate a grayscale image.

4. A low-cost intelligent fire monitoring system based on multimodal data fusion according to claim 3, characterized in that: The feature extraction module extracts features corresponding to flame data from the infrared sensor 1 (5), the infrared sensor 2 (7), the ultraviolet sensor (9) and the grayscale image respectively; In the feature extraction module, the analog quantities collected by the three optical sensors are converted into corresponding digital quantities, that is, the optical characteristics of the flame are extracted; the number of gray value changes in a certain area of ​​the ultraviolet image for 20 consecutive frames is counted, that is, the movement and flickering frequency of the flame ultraviolet rays are extracted; After the infrared sensor 1 (5), the infrared sensor 2 (7) and the ultraviolet sensor (9) collect infrared and ultraviolet rays of specific wavelengths, they output corresponding analog quantities, and the three sensors are respectively connected to ADC channels 0, 1, and 2 of the main control chip. The main control chip reads the output signals of the three sensors and converts them into corresponding voltage values, thereby extracting the optical characteristics of the flame; the ultraviolet image detects the flame by changing the gray value frequency of a certain area in several consecutive frames, thereby extracting the movement and flickering characteristics of the flame ultraviolet rays; The conversion relationship between the analog quantity collected by the infrared sensor 1 (5), the infrared sensor 2 (7) and the ultraviolet sensor (9) and the corresponding voltage value is as follows: Among them, V ref is the standard voltage, n is the number of AD conversion bits, V result It is the collected voltage required by the user, and raw is the original data collected by AD; The grayscale value change frequency of a certain area of ​​the grayscale image is the flickering feature of the flame. The feature is that the grayscale value of the pixel at the edge of the flame changes significantly between frames. The absolute value of the grayscale value difference of a certain pixel between frames is taken, and whether it is a flame is determined based on the absolute value and the number of changes; The calculation steps are as follows: First, two counting matrices map1 and map2 with the same resolution as the original image are established. Secondly, at time t, when the gray value difference between the current frame and the previous frame of a certain pixel (x, y) is greater than a threshold T, map1(x, y, t) is added by 1 on the basis of the previous frame map1(x, y, t-1) for accumulation, and map2(x, y, t) is added by 0; when the gray value difference between the current frame and the previous frame (x, y) is less than a threshold minus T, map2(x, y, t) is added by 1 and map1(x, y, t) is added by 0; if it is not the above two cases, both map1(x, y, t) and map2(x, y, t) are added by 0; ΔI(x,y,t)=R(x,y,t)-R(x,y,t-1) Where T is a positive real number used to eliminate the influence of system noise and other factors on the pixel grayscale value. The value of T is generally between 5 and 20. R(x,y,t) and R(x,y,t-1) are the grayscale values ​​of the pixel (x,y) at time t and time t-1 respectively. Finally, in the surveillance video image sequence, the grayscale value of the stationary target remains basically unchanged, while the moving car lights will cause a large change in the pixel grayscale value. The changes of map1(x,y,t) and map2(x,y,t) will exceed a certain threshold. If the changes of map1(x,y,t) and map2(x,y,t) of a certain pixel (x,y) exceed a certain threshold within a certain period of time, then this pixel may be the edge pixel of the flame; The flickering characteristics of a flame are as follows: map1(x,y,t)=1COUNT1=map1(x,y,t)-map1(x,y,tN) map2(x,y,t)=1COUNT2=map2(x,y,t)-map2(x,y,tN) In the formula, MAP, MAP1 and MAP2 are binary matrices with the same resolution as the original image, and the symbol Represents a binary AND operation; N is the accumulation time window, which is equivalent to the video acquisition rate; map1(x,y,t), map2(x,y,t) and map1(x,y,tN), map2(x,y,tN) are the count values ​​of the pixel (x,y) at time t and tN respectively. COUNT1 is the number of times the grayscale value difference of the pixel in adjacent frames within 20 frames is greater than a threshold T, and COUNT2 is the number of times the grayscale value difference of the pixel in adjacent frames within 20 frames is less than a threshold minus T.

5. A low-cost intelligent fire monitoring system based on multimodal data fusion according to claim 4, characterized in that: The data processing module is used to input the four flame features extracted by the feature extraction module into the trained classification model, and finally obtain the classification result of whether there is fire; In the data processing module, a gated recurrent unit (GRU) is used as the main framework of the algorithm, including a reset gate and an update gate; In GRU, the hidden state update process is divided into three steps: First, the reset gate is calculated, then the update gate is calculated, and finally the values ​​of these two gates are combined to update the hidden state; Reset gate (r t ) is calculated as follows: r t =σ(W r x t +U r h t-1 +b r ) Among them, σ is the sigmoid activation function, W r and U r are the weight matrices from input and hidden state to reset gate, b r is the bias term, x t is the input at the current moment, h t-1 is the hidden state at the previous moment; Update gate (z t ) is calculated as follows: z t =σ(W z x t +U z h t-1 +b z ) Among them, W z and U z are the weight matrices from input and hidden state to update gate, b z is the bias term x t ; Candidate hidden states The calculation formula is as follows: Among them, tanh is the hyperbolic tangent activation function, W h and U h are the weight matrices from the hidden state adjusted by the input and reset gates to the candidate hidden state, and b h is the bias term, ⊙ represents element-wise multiplication; The final hidden state (h t ) is calculated as follows: The final output of the GRU is usually associated with a hidden state, which is passed to other layers or used as the final output. Use a fully connected layer with an activation function to complete the transformation from the GRU hidden state h t Mapping to class probabilities; Fully connected layer: O=W fc h T +b fc Among them, o is the output of the fully connected layer; W fc is the weight matrix of the fully connected layer, whose dimension is H×C, where H is the dimension of the GRU hidden state and C is the number of categories; b fc is the bias vector of the fully connected layer, whose dimension is C; h T is the hidden state of GRU at the last time step of the sequence; The activation function is the Softmax function, which converts the output into a probability distribution: p = softmax(o) Among them, p is a C-dimensional probability vector, and each element represents the probability of the corresponding category, that is, the fire probability.

6. A low-cost intelligent fire monitoring system based on multimodal data fusion according to claim 3, characterized in that: In the dynamic threshold generation module, the setting standard is that under the environment of preset temperature T0 and preset light intensity E0, if the output fire probability value p is higher than the preset fire alarm threshold TTH0, it indicates that there is a high possibility of fire in the detection area, and an alarm signal should be issued in time to prompt the staff to check the detection area; After the temperature sensor (6) and the light sensor (8) collect the ambient temperature and light intensity, they output corresponding analog quantities. The main control chip RV1126 used in the system includes a 6-channel ADC, wherein channels 0, 1, 2, 3, and 5 are used for common analog quantity input. The signal output pins of the above two sensors are respectively connected to ADC channels 3 and 5 of the main control chip, and the fire alarm threshold is updated according to these two values ​​to generate a dynamic fire alarm threshold; The update formula is as follows: Where, T0 represents the preset temperature, E0 represents the preset light intensity, and TTH0 represents the preset fire alarm threshold; T represents the real-time temperature, E represents the real-time light intensity, and the real-time temperature T and the real-time light intensity E are both obtained through the data acquisition module; τ represents the ratio between the temperature change degree and the light intensity change degree. The larger the ratio τ is, the greater the temperature change degree is, the greater the weight τ*β1 of the subsequent temperature change degree is; conversely, the smaller the ratio τ is, the smaller the temperature change degree is, the smaller the light intensity change degree is, the smaller the weight τ*β1 of the subsequent temperature change degree is. β1 and β2 are both preset proportional coefficients, and β1+β2=1, β1∈(0,1), β2∈(0,1).

7. A low-cost intelligent fire monitoring system based on multimodal data fusion according to claim 1, characterized in that: The fire alarm module is used to compare the final fire probability value obtained by the data processing module with the fire alarm dynamic threshold, and send out an alarm signal when the probability value is greater than the fire alarm dynamic threshold; In the fire alarm module, the alarm signal emitted by the fire alarm module includes an audible and visual signal emitted by the audible and visual alarm and a vibration of the alarm bracelet. The positive electrodes of the audible and visual alarm and the alarm bracelet controller are connected to the common IO pin of the main control chip, and the negative electrodes are connected to the ground pin of the main control chip. When the system detects a fire, it pulls up the IO pins connected to the sound and light alarm and the wristband controller at the same time. The sound and light alarm and the wristband controller both adopt the power-on alarm working mode. When the fire alarm module determines that an alarm needs to be sounded, the main control chip controls the corresponding IO pin to output a high level, and the sound and light alarm and the wristband controller can generate corresponding alarm signals, thereby controlling the corresponding alarm phenomenon. However, since the vibration of the alarm bracelet needs to continuously emit infrared rays for control, the IO pin that controls the wristband controller needs to be repeatedly pulled high and low at fixed intervals, with an interval of 5s.

8. A detection method for a low-cost intelligent fire monitoring system based on multimodal data fusion according to any one of claims 1 to 7, characterized in that: The steps include: Step 1: Collect raw data through the data acquisition module, including the current ambient temperature, light intensity, 3.8μm and 4.3μm infrared intensity, ultraviolet intensity and ultraviolet images of specific wavelengths; Step 2: Send the collected data to the data preprocessing module to standardize the raw data, remove noise, and synchronize time; Step 3: Send the processed data to the feature extraction module to extract fire features. Specifically, the optical features, motion and flicker features of the fire are extracted in this module. Step 4: Send the extracted features to the data processing module, which contains a machine learning model GRU. The model classifies the above features in combination with time series information and outputs the probability value of whether a fire occurs in the current environment; Step 5: The dynamic threshold generation module calculates the threshold of fire in the current environment based on the collected temperature and light intensity. This value is compared with the fire probability value output by the alarm module and the data processing module, and finally a conclusion is given on whether a fire has occurred, and corresponding measures are taken, that is, an alarm signal is generated or the next frame is detected.

9. The detection method of a low-cost intelligent fire monitoring system based on multimodal data fusion according to claim 8 is characterized in that: Under normal circumstances, IR-CUT (3) is in the ultraviolet filter working state, and the visible light lens (4) collects ultraviolet images. The customized ultraviolet filter can filter out some interference such as sunlight. At the same time, the two infrared sensors 1 (5), infrared sensor 2 (7) and ultraviolet sensor (9) are also constantly monitoring the infrared and ultraviolet rays in the environment. These data are sent to the machine learning model after data preprocessing and feature extraction modules, and the four features are combined to make a fire judgment. Once the system detects a fire, IR-CUT (3) will immediately switch to the full-transmittance filter working state and transmit the visible light video stream to facilitate manual monitoring and fire source location. This state will continue until the fire is completely extinguished, and IR-CUT (3) will continue to switch to the ultraviolet filter working state to perform routine fire monitoring.

Images