Kitchen open fire hidden danger precursor sensing method based on multiple modes
By using multimodal data fusion and dynamic weight adjustment, the problem of poor anti-interference capability in kitchen open flame monitoring was solved, enabling accurate early warning and adaptive monitoring during peak oil fume periods, thus improving kitchen safety.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG GONGSHANG UNIVERSITY
- Filing Date
- 2026-01-20
- Publication Date
- 2026-04-17
AI Technical Summary
Existing kitchen open flame monitoring technologies suffer from poor anti-interference capabilities in single-modal monitoring and lack of dynamic weight adjustment mechanisms in multi-modal fusion schemes, making them unsuitable for special scenarios such as peak periods of cooking fumes, resulting in poor monitoring performance.
A multimodal data fusion method is adopted, including the simultaneous acquisition of visible light images, infrared thermal imaging images and multi-parameter sensor data. Through preprocessing such as Gaussian filtering, histogram equalization and threshold segmentation, visual, thermodynamic and temporal dynamic features are extracted. A random forest model is constructed to evaluate the confidence level, and weighted fusion is performed based on a dynamic weight allocation function. Finally, a graded risk assessment and early warning are carried out.
It effectively avoids interference from the complex kitchen environment, improves the accuracy of early warning of open flame hazards, significantly reduces false alarm and missed alarm rates, and achieves adaptive adjustment to adapt to different kitchen environments and user habits.
Smart Images
Figure CN121884518A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of kitchen safety monitoring technology, specifically to a multimodal method for detecting early signs of open flame hazards in kitchens. Background Technology
[0002] As a high-risk area for fires, the early and accurate detection of open flame hazards in the kitchen is crucial to ensuring personal and property safety. Early warning signs of open flame hazards, such as flammable gas leaks, abnormal localized temperature increases, and abnormal accumulation of cooking fumes, are often subtle and easily affected by the complex kitchen environment, leading to significant limitations in traditional monitoring methods.
[0003] Existing kitchen open flame monitoring technologies suffer from the following shortcomings: First, single-modal monitoring schemes have poor anti-interference capabilities. For example, monitoring based on visible light images is easily obscured by cooking fumes, leading to feature extraction failure. Monitoring based on a single sensor is prone to false alarms and missed alarms due to environmental interference. Second, some multimodal fusion schemes lack dynamic weight adjustment mechanisms, making them unsuitable for special scenarios such as peak cooking fume periods, resulting in poor fusion effects. In response, this paper proposes a multimodal method for detecting early warning signs of kitchen open flame hazards. Summary of the Invention
[0004] To address the aforementioned technical issues, this technical solution provides a multimodal method for detecting early signs of open flame hazards in kitchens. This solution resolves the problems of poor anti-interference capability of the single-modal monitoring scheme, the lack of dynamic weight adjustment mechanism in some multimodal fusion schemes, and the inability to adapt to special scenarios such as peak periods of cooking fumes, resulting in poor fusion effects.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0006] A multimodal method for detecting early warning signs of open flame hazards in kitchens includes:
[0007] S1. Within the kitchen monitoring area, simultaneously collect multimodal data including visible light image data, infrared thermal imaging image data, and multi-parameter sensor data of the target area;
[0008] S2. Preprocess and analyze the data of each modality, extract feature information related to the precursors of open flame hazards, and evaluate the real-time confidence of each feature;
[0009] S3. Based on real-time confidence, dynamically determine the fusion weight of each feature information, and perform weighted fusion to obtain a comprehensive risk assessment value;
[0010] S4. Classify and judge based on comprehensive risk assessment value, and issue a warning signal when it is determined that there are signs of potential open flame hazards.
[0011] Preferably, step S1 specifically includes:
[0012] The time synchronization device controls the visible light camera, infrared thermal imaging equipment, and multi-parameter sensor module deployed at the same physical location in the kitchen monitoring area to simultaneously trigger data acquisition with the same acquisition cycle; the multi-parameter sensor module includes at least a combustible gas sensor and a volatile organic compound sensor.
[0013] Each time it is triggered, a visible light image, an infrared temperature distribution image, and a set of environmental sensor readings are obtained synchronously, and the same timestamp is assigned to the set of data, which is then encapsulated into a spatiotemporally aligned structured data packet.
[0014] Preferably, the specific steps for preprocessing the modal data in step S2 are as follows:
[0015] Based on visible light image data, a Gaussian filtering algorithm is used to remove random noise; a histogram equalization algorithm is used to enhance the image and improve contrast; based on the preset boundary of the kitchen monitoring area, a threshold segmentation algorithm is used to extract the region of interest and eliminate interference from non-monitoring areas.
[0016] Temperature calibration is performed based on infrared thermal imaging data and infrared thermal imager calibration parameters; median filtering algorithm is used to filter noise and eliminate salt-and-pepper noise; based on temperature gradient differences, region growing algorithm is used to segment the region and divide it into different temperature ranges.
[0017] Preprocessing is performed on multi-parameter sensor data. Outliers are removed using the 3σ criterion to eliminate abnormal data caused by sensor malfunctions or external interference. The data is smoothed using a moving average algorithm to reduce data fluctuations. The min-max standardization method is used to map each parameter data to the [0,1] interval to obtain standardized sensor data.
[0018] Preferably, the specific steps for extracting feature information in step S2 are as follows:
[0019] Visual features are extracted from the region of interest in the preprocessed visible light image.
[0020] The visual features include color features, shape features, and texture features. Color features are represented by the mean of HSV color space components, shape features are represented by rectangularity and circularity, and texture features are represented by the energy and entropy values of the gray-level co-occurrence matrix.
[0021] Thermodynamic features are extracted from the preprocessed infrared thermal imaging images;
[0022] The thermodynamic features include the average pixel temperature within each segmented connected region, the temperature gradient magnitude of pixels at the region edges, the pixel area of the region, and the rate of change of the region area over consecutive time frames.
[0023] Based on preprocessed multi-parameter sensor data, temporal dynamic features are extracted;
[0024] The time-series dynamic characteristics include the rate of change of smoke sensor readings per unit time, the duration and extent of combustible gas concentration exceeding the safety threshold, and the rate of increase of ambient temperature sensor readings per unit time.
[0025] Preferably, the specific steps for S2 to assess real-time confidence are as follows:
[0026] A confidence evaluation model based on random forest is constructed. The random forest model contains 100 decision trees, each with a maximum depth of 15. The feature sampling method is random sampling. The input is a multimodal feature vector composed of visual features, thermodynamic features, and temporal dynamic features extracted from data at the same time.
[0027] The confidence assessment model was trained using training samples, which included 1,000 sets of normal kitchen scenario samples and 800 sets of scenario samples showing early signs of open flame hazards. Each sample was labeled with corresponding feature information and a true confidence label.
[0028] The training process uses 5-fold cross-validation to optimize model parameters, and the evaluation metric for cross-validation is accuracy.
[0029] The extracted feature information is input into the trained confidence assessment model, and the output consists of real-time confidence scores corresponding to three sets of features: visual features, thermodynamic features, and temporal dynamic features. The scoring range is [0,1].
[0030] Preferably, S3 specifically includes:
[0031] Real-time confidence scores based on three sets of features: visual features, thermodynamic features, and temporal dynamic features. Input to a dynamic weight allocation function;
[0032] The dynamic weight allocation function normalizes the confidence scores to obtain the corresponding dynamic fusion weights. ;
[0033] The weight allocation function is configured to: when it is determined from the temporal dynamic features that the current period is a peak period for cooking fumes, reduce the dynamic fusion weight corresponding to the visual features. .
[0034] Preferably, S3 further includes:
[0035] Based on the extracted visual features, thermodynamic features, and temporal dynamic features, the three sets of feature information are respectively input into three independent primary risk scoring models;
[0036] The primary risk scoring model is a binary classification model constructed based on logistic regression or support vector machine, and its training objective is to map the input feature vector into a primary risk score representing the likelihood of the presence of precursors of open fire hazards in the current modality. The scoring range is [0,1].
[0037] Preferably, the specific implementation of the comprehensive risk assessment value in step S3 is as follows:
[0038] Based on the obtained dynamic fusion weights and the calculated corresponding primary risk scores perform weighted linear summation, and the calculation formula is:
[0039] .
[0040] Preferably, step S4 specifically includes:
[0041] Based on the comprehensive risk assessment value , compare it with a preset hierarchical risk threshold set including two levels;
[0042] The hierarchical risk threshold set includes at least a first threshold T1 and a second threshold T2, and 0 < T1 < T2 ≤ 1; the values of T1 and T2 are initialized according to the kitchen environment characteristics;
[0043] Judge the current risk level by comparison: if < T1, it is determined to be a risk-free state; if T1 ≤ R < T2, it is determined to be a primary warning state, and local audible and visual warnings are executed; if ≥ T2, it is determined to be a high-level alarm state, and an emergency alarm including remote notification is executed;
[0044] After the system issues a warning, continuously monitor the change of the comprehensive risk assessment value R;
[0045] If the R value drops below T1 within the preset continuous monitoring time and the modal feature information of each modality returns to normal, the alarm is automatically stopped and a risk解除 notice is sent;
[0046] Preferably, the method further includes:
[0047] The system continuously records the historical comprehensive risk assessment value R and the event label of whether a real hidden danger finally occurs; regularly use the recorded historical data to optimize and adjust the hierarchical risk threshold set T1, T2,... through machine learning algorithms;
[0048] Update the optimized new threshold to the system to achieve the adaptive adjustment of the warning threshold according to different kitchen environments and user habits.
[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0050] This invention proposes a dynamic weight allocation mechanism that automatically reduces the weight of visually disturbed features and increases the weight of infrared and sensor data during peak periods of cooking fumes. This effectively avoids interference from the complex kitchen environment, ensures the accuracy of early warning of potential hazards, and significantly reduces false alarms and missed alarms. The initial value of the graded risk threshold of this invention can be flexibly adapted to different types of stoves such as gas stoves and induction cookers, as well as the size of the kitchen space. Furthermore, it can be periodically optimized and adaptively adjusted through a gradient descent algorithm, which can accurately match the monitoring needs of different scenarios such as home kitchens and restaurant kitchens. Attached Figure Description
[0051] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation
[0052] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.
[0053] Reference Figure 1 As shown, the method for detecting early warning signs of open flame hazards in kitchens based on multimodal approaches includes:
[0054] S1. Within the kitchen monitoring area, simultaneously collect multimodal data including visible light image data, infrared thermal imaging image data, and multi-parameter sensor data of the target area;
[0055] Step S1 specifically includes:
[0056] The time synchronization device controls the visible light camera, infrared thermal imaging equipment, and multi-parameter sensor module deployed at the same physical location in the kitchen monitoring area to simultaneously trigger data acquisition with the same acquisition cycle; the multi-parameter sensor module includes at least a combustible gas sensor and a volatile organic compound sensor.
[0057] Each time it is triggered, a visible light image, an infrared temperature distribution image, and a set of environmental sensor readings are obtained synchronously, and the same timestamp is assigned to the set of data, which is then encapsulated into a spatiotemporally aligned structured data packet.
[0058] In summary, the time synchronization device is used to achieve precise synchronous acquisition of multimodal data. This device is a master clock source, which is directly connected to the trigger ports of each acquisition module through hardware trigger signal lines (such as GPIO). The master program sends instructions to the clock source according to a preset acquisition cycle (e.g., 10Hz), and the clock source generates a high-precision synchronization pulse signal, which is simultaneously sent to all cameras and sensors to drive them to perform one acquisition at the same physical moment.
[0059] S2. Preprocess and analyze the data of each modality, extract feature information related to the precursors of open flame hazards, and evaluate the real-time confidence of each feature;
[0060] The specific steps for preprocessing the modal data in step S2 are as follows:
[0061] Based on visible light image data, a Gaussian filtering algorithm is used to remove random noise; a histogram equalization algorithm is used to enhance the image and improve contrast; based on the preset boundary of the kitchen monitoring area, a threshold segmentation algorithm is used to extract the region of interest and eliminate interference from non-monitoring areas.
[0062] Temperature calibration is performed based on infrared thermal imaging data and infrared thermal imager calibration parameters; median filtering algorithm is used to filter noise and eliminate salt-and-pepper noise; based on temperature gradient differences, region growing algorithm is used to segment the region and divide it into different temperature ranges.
[0063] Preprocessing is performed on multi-parameter sensor data. Outliers are removed using the 3σ criterion to eliminate abnormal data caused by sensor malfunctions or external interference. The data is smoothed using a moving average algorithm to reduce data fluctuations. The min-max standardization method is used to map each parameter data to the [0,1] interval to obtain standardized sensor data.
[0064] Based on the above, visible light image preprocessing specifically includes:
[0065] For the acquired RGB image, perform the following operations in sequence:
[0066] Gaussian filtering for noise reduction: A 5x5 Gaussian convolution kernel is used to convolve the image, with the standard deviation σ set to 1.5; the formula is as follows:
[0067]
[0068] In the formula, Indicates the position of the smoothed image. Pixel intensity value at; Double summation means summing over the product of the two products. Traverse all pixels within a local neighborhood of size (2k+1)×(2k+1) centered at the center, where k is an integer that determines the radius of the convolution kernel. Here, a 5x5 convolution kernel is used, i.e., k=2. For Gaussian convolution kernels at offset Weighting coefficients at each location; For the input image at position The original pixel intensity value at that location;
[0069] The weighting coefficients are obtained by discretizing and normalizing a two-dimensional Gaussian function, and their calculation formula is as follows:
[0070]
[0071] In the formula, variables This represents the two-dimensional coordinate offset of the current computation position within the convolution kernel relative to the center point (0, 0); in this invention, a preferred embodiment for a kitchen monitoring scenario is selected. =1.5; This value can effectively suppress image sensor noise, particle noise caused by uneven lighting, and subtle coding noise, while ensuring that key visual features of the early signs of open flame, such as the edge of the stove, the outline of the pot, and the initial smoke pattern, are not excessively blurred. The normalization coefficients ensure that the volume of the two-dimensional Gaussian function is divided into 1 across the entire plane; The weight decreases exponentially with the square of the Euclidean distance from the center point.
[0072] Infrared thermal imaging image preprocessing specifically includes:
[0073] Taking into account ambient temperature drift, the system automatically performs an online temperature calibration every 24 hours. During the calibration process, the stove is turned off and there is no heat source, and the average temperature of the stable area representing the ambient object in the infrared image is read. Simultaneously read the readings from the ambient temperature sensor. The formula for calculating the offset is:
[0074]
[0075] A 3x3 sliding window is used to perform median filtering on the calibrated temperature matrix; this operation can effectively filter out "salt and pepper noise" caused by uneven pixel response or transient interference, especially around high temperature points; for the 9 temperature values within the window, the median value is taken as the new value of the center pixel;
[0076] To identify potential abnormal heat sources, a threshold-based region growing algorithm is used for segmentation. First, a base temperature threshold is set (e.g., 50°C higher than the ambient temperature). All pixels with temperatures higher than the base temperature threshold are marked as "high-temperature seed points." Then, starting from each seed point, the algorithm grows outwards to its four neighbors (top, bottom, left, and right). If the temperature difference between a neighboring pixel and the seed point is less than 5°C, it is included in the same high-temperature region. This process is iterated until no new pixels can be included. Finally, several disconnected high-temperature connected regions and their pixel coordinates are output.
[0077] The specific steps for extracting feature information in step S2 are as follows:
[0078] Visual features are extracted from the region of interest in the preprocessed visible light image.
[0079] The visual features include color features, shape features, and texture features. Color features are represented by the mean of HSV color space components, shape features are represented by rectangularity and circularity, and texture features are represented by the energy and entropy values of the gray-level co-occurrence matrix.
[0080] Thermodynamic features are extracted from the preprocessed infrared thermal imaging images;
[0081] The thermodynamic features include the average pixel temperature within each segmented connected region, the temperature gradient magnitude of pixels at the region edges, the pixel area of the region, and the rate of change of the region area over consecutive time frames.
[0082] Based on preprocessed multi-parameter sensor data, temporal dynamic features are extracted;
[0083] The time-series dynamic characteristics include the rate of change of smoke sensor readings per unit time, the duration and extent of combustible gas concentration exceeding the safety threshold, and the rate of increase of ambient temperature sensor readings per unit time.
[0084] Based on the above, feature extraction specifically includes:
[0085] The feature extraction step of this invention is based on preprocessed multimodal data, extracting quantitative features that are highly correlated with early signs of open flame hazards from visible light images, infrared thermal imaging images and environmental sensor data, respectively, in order to construct a feature vector that comprehensively describes the safety status.
[0086] For visible light images, visual feature analysis is performed within the extracted regions of interest. First, color feature extraction is performed: the image is converted from the RGB color space to the HSV space, and the statistical mean of the hue and saturation components of suspected smoke areas or the core area of the cookware is calculated to capture color shifts caused by high temperatures or the unique color attributes of cooking fumes. Second, shape feature extraction is performed: potential anomalous regions are obtained through image segmentation, and their rectangularity and circularity are calculated to quantify the diffusion pattern of flames or smoke. Finally, texture feature extraction is performed: the energy and entropy values of the regions are calculated based on the gray-level co-occurrence matrix to characterize the changes in texture uniformity and complexity caused by smoke diffusion or flame combustion.
[0087] For infrared thermal imaging images, thermodynamic features are extracted from the segmented high-temperature connected regions. The average temperature value of all pixels within each connected region is calculated as the core thermal intensity index. Simultaneously, by analyzing the gradient field of the infrared image, the average temperature gradient amplitude at the edges of high-temperature regions is extracted to reflect the sharpness of heat diffusion. Furthermore, the pixel area of the high-temperature region and its rate of area change between consecutive frames are calculated to dynamically assess the expansion trend of the heat source, a key spatiotemporal dynamic indicator of the thermal runaway process.
[0088] For multi-parameter sensor data, temporal dynamic features are extracted from their smoothed and standardized time-series signals. The linear regression slope of volatile organic compound concentration within a short time window is calculated as a characteristic of the abnormal growth rate of smoke concentration, which is highly sensitive to the sudden generation of cooking fumes. Combustible gas concentration data is analyzed to extract Boolean indicators of whether it exceeds the safety threshold, the duration of continuous exceedance, and the magnitude of exceedance, collectively characterizing the severity and persistence of gas leaks. The rate of increase of ambient temperature sensor readings per unit time is calculated to capture abnormally rapid temperature rises in the overall kitchen thermal environment.
[0089] Ultimately, the system integrates and encapsulates all the aforementioned visual, thermodynamic, and temporal dynamic features into a unified multimodal feature vector. This vector constitutes a digital panoramic description of the current safety status of the kitchen, providing a standardized and measurable input data foundation for subsequent confidence assessment and multimodal weighted fusion decision-making.
[0090] The specific steps for evaluating real-time confidence level using S2 are as follows:
[0091] A confidence evaluation model based on random forest is constructed. The random forest model contains 100 decision trees, each with a maximum depth of 15. The feature sampling method is random sampling. The input is a multimodal feature vector composed of visual features, thermodynamic features, and temporal dynamic features extracted from data at the same time.
[0092] The confidence assessment model was trained using training samples, which included 1,000 sets of normal kitchen scenario samples and 800 sets of scenario samples showing early signs of open flame hazards. Each sample was labeled with corresponding feature information and a true confidence label.
[0093] The training process uses 5-fold cross-validation to optimize model parameters, and the evaluation metric for cross-validation is accuracy.
[0094] The extracted feature information is input into the trained confidence assessment model, and the output consists of real-time confidence scores corresponding to three sets of features: visual features, thermodynamic features, and temporal dynamic features. The scoring range is [0,1].
[0095] S3. Based on real-time confidence, dynamically determine the fusion weight of each feature information, and perform weighted fusion to obtain a comprehensive risk assessment value;
[0096] S3 specifically includes:
[0097] Real-time confidence scores based on three sets of features: visual features, thermodynamic features, and temporal dynamic features. Input to a dynamic weight allocation function;
[0098] The dynamic weight allocation function normalizes the confidence scores to obtain the corresponding dynamic fusion weights. ;
[0099] The weight allocation function is configured to: when it is determined from the temporal dynamic features that the current period is a peak period for cooking fumes, reduce the dynamic fusion weight corresponding to the visual features. .
[0100] Specifically, S3 also includes:
[0101] Based on the extracted visual features, thermodynamic features, and temporal dynamic features, the three sets of feature information are respectively input into three independent primary risk scoring models;
[0102] The primary risk scoring model is a binary classification model based on logistic regression or support vector machine. Its training objective is to map the input feature vector into a primary risk score that represents the likelihood of an early sign of an open flame hazard in the current modality. The scoring range is [0,1].
[0103] The comprehensive risk assessment value in step S3 is specifically implemented as follows:
[0104] Based on the obtained dynamic fusion weights and the corresponding primary risk score calculated. The weighted linear summation is calculated using the following formula:
[0105] ;
[0106] In the formula, This is the comprehensive risk assessment value of the target area of the kitchen at the current moment, and its value range is [0,1]. These represent independent risk assessment outputs for three modalities: visual, infrared thermodynamic, and sensor timing, respectively. These represent the weighting coefficients of the three primary risk scores mentioned above; in the formula... It is a dynamic variable; during peak periods of cooking fumes, the visible light image is interfered with, and its confidence level... The decrease led to its weight The weights are automatically reduced during calculations, while infrared and sensor data, being unaffected by visible light interference, have their weights adjusted accordingly. and It increases automatically relative to the target.
[0107] S4. Classify and judge based on comprehensive risk assessment value, and issue a warning signal when it is determined that there are signs of potential open flame hazards.
[0108] Step S4 specifically includes:
[0109] Based on the comprehensive risk assessment value , compare it with a preset two-level hierarchical risk threshold set;
[0110] The hierarchical risk threshold set at least includes a first threshold T1 and a second threshold T2, and satisfies 0 < T1 < T2 ≤ 1;
[0111] Judge the current risk level by comparison: If < T1, it is determined to be a risk-free state; if T1 ≤ R < T2, it is determined to be a primary warning state, and local audible and visual warnings are executed; if ≥ T2, it is determined to be a high-level alarm state, and an emergency alarm including remote notification is executed;
[0112] After the system issues a warning, continuously monitor the change of the comprehensive risk assessment value R;
[0113] If the R value drops below T1 within the preset continuous monitoring time and all modal feature information returns to normal, automatically stop the alarm and send a risk cancellation notice;
[0114] The method further includes:
[0115] The system continuously records the historical comprehensive risk assessment value R and the event label of whether a real hidden danger finally occurs; regularly use the recorded historical data to optimize and adjust the hierarchical risk threshold set T1, T2,... through machine learning algorithms;
[0116] Update the optimized new threshold to the system to achieve adaptive adjustment of the warning threshold according to different kitchen environments and user habits.
[0117] Combined with the above content, the detailed explanations and determination methods of the thresholds T1 and T2 are as follows:
[0118] T1 and T2 are not fixed values set based on experience, but scientific parameters derived from historical data analysis and system performance trade-offs. The first warning threshold T1 aims to define the boundary between "normal operating noise" and "clear hidden danger signals". Its setting principle is to allow an extremely low false alarm rate while tolerating a certain amount of false alarms (which can be handled gently through reminder-level warnings). The second alarm threshold T2 is used to identify the critical point between "hidden dangers requiring manual intervention" and "emergency states requiring automatic emergency response". Its setting is extremely strict and requires an extremely high degree of confidence to ensure that the probability of erroneous action when triggering linkage control is extremely low. During initial deployment, the typical method for determining T1 and T2 is as follows: collect a large amount of historical multimodal data covering normal cooking and simulated potential hazard scenarios, and calculate the distribution of the corresponding comprehensive risk assessment value R; use receiver operating characteristic curve (ROC curve) analysis or optimization based on cost matrix to find the optimal balance between "missed detection risk" and "false alarm cost" to determine T1 (e.g., corresponding to a risk probability of 30%) and T2 (e.g., corresponding to a risk probability of 75%); more importantly, after the system is established, T1 and T2 can be fine-tuned through an online learning mechanism based on actual operating data (e.g., false alarm records, user feedback) to achieve adaptive thresholds according to specific kitchen environments and user habits.
[0119] The complete workflow is as follows: The system calculates the R value in real time and first compares it with T1 and T2 to determine the risk level. If the R value is between T1 and T2, it is determined to be a primary warning state. The system executes local audible and visual prompts (such as a slow flashing yellow indicator light) and sends a warning notification to the user's mobile APP to remind the user to take initiative in checking. If the R value exceeds the higher T2, it is immediately determined to be a high-level alarm state. The system will initiate the highest level response: a piercing alarm and a flashing red light are issued locally, and an emergency alarm is sent to the user and the remote security platform simultaneously. At the same time, the system will automatically execute a preset safety linkage operation sequence, the core of which is "cut off fuel and increase ventilation," such as immediately closing the smart gas valve and forcibly switching the range hood to maximum power operation. This automatic handling mechanism is crucial when the user cannot respond in time.
[0120] After an alert is issued, the system enters an enhanced monitoring and feedback phase. It continuously monitors changes in the R value and underlying characteristics. If, due to proper handling, the R value continues to decrease within a preset time period (e.g., 60 seconds) and falls below T1, and all sensor data return to normal, the system determines that the risk has been eliminated, automatically stops the alarm, and sends a cancellation notification. The entire process forms a closed loop of "perception-assessment-decision-action-verification," completing the entire process from early warning to emergency response and state recovery without human intervention, greatly improving the reliability and timeliness of kitchen safety response.
[0121] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.
Claims
1. A multi-modal based kitchen open flame hazard precursor perception method, characterized in that, include: S1. Within the kitchen monitoring area, simultaneously collect multimodal data including visible light image data, infrared thermal imaging image data, and multi-parameter sensor data of the target area; S2. Preprocess and analyze the data of each modality, extract feature information related to the precursors of open flame hazards, and evaluate the real-time confidence of each feature; S3. Based on real-time confidence, dynamically determine the fusion weight of each feature information, and perform weighted fusion to obtain a comprehensive risk assessment value; S4. Classify and judge based on comprehensive risk assessment value, and issue a warning signal when it is determined that there are signs of potential open flame hazards.
2. The multi-modal based kitchen open flame hazard precursor perception method according to claim 1, wherein, Step S1 specifically includes: The time synchronization device controls the visible light camera, infrared thermal imaging equipment, and multi-parameter sensor module deployed at the same physical location in the kitchen monitoring area to simultaneously trigger data acquisition with the same acquisition cycle; the multi-parameter sensor module includes at least a combustible gas sensor and a volatile organic compound sensor. Each time it is triggered, a visible light image, an infrared temperature distribution image, and a set of environmental sensor readings are obtained synchronously, and the same timestamp is assigned to the set of data, which is then encapsulated into a spatiotemporally aligned structured data packet.
3. The method for detecting early warning signs of open flame hazards in kitchens based on multimodal methods according to claim 1, characterized in that, The specific steps for preprocessing the modal data in step S2 are as follows: Based on visible light image data, a Gaussian filtering algorithm is used to remove random noise; a histogram equalization algorithm is used to enhance the image and improve contrast; based on the preset boundary of the kitchen monitoring area, a threshold segmentation algorithm is used to extract the region of interest and eliminate interference from non-monitoring areas. Temperature calibration is performed based on infrared thermal imaging data and infrared thermal imager calibration parameters; median filtering algorithm is used to filter noise and eliminate salt-and-pepper noise. Based on the temperature gradient difference, a region growing algorithm is used to segment the region and divide it into different temperature ranges; Preprocessing is performed on multi-parameter sensor data. Outliers are removed using the 3σ criterion to eliminate abnormal data caused by sensor malfunctions or external interference. The data is smoothed using a moving average algorithm to reduce data fluctuations. The min-max standardization method is used to map each parameter data to the [0,1] interval to obtain standardized sensor data.
4. The method for detecting early warning signs of open flame hazards in kitchens based on multimodal methods according to claim 1, characterized in that, The specific steps for extracting feature information in step S2 are as follows: Visual features are extracted from the region of interest in the preprocessed visible light image. The visual features include color features, shape features, and texture features. Color features are represented by the mean of HSV color space components, shape features are represented by rectangularity and circularity, and texture features are represented by the energy and entropy values of the gray-level co-occurrence matrix. Thermodynamic features are extracted from the preprocessed infrared thermal imaging images; The thermodynamic features include the average pixel temperature within each segmented connected region, the temperature gradient magnitude of pixels at the region edges, the pixel area of the region, and the rate of change of the region area over consecutive time frames. Based on preprocessed multi-parameter sensor data, temporal dynamic features are extracted; The time-series dynamic characteristics include the rate of change of smoke sensor readings per unit time, the duration and extent of combustible gas concentration exceeding the safety threshold, and the rate of increase of ambient temperature sensor readings per unit time.
5. The method for detecting early warning signs of open flame hazards in kitchens based on multimodal methods according to claim 1, characterized in that, The specific steps for the S2 to evaluate the real-time confidence are as follows: Construct a confidence evaluation model based on a random forest. The random forest model contains 100 decision trees, the maximum depth of each decision tree is 15, and the feature sampling method is random sampling. The input is a multi-modal feature vector composed of visual features, thermodynamic features, and temporal dynamic features extracted from the data at the same moment. The confidence evaluation model is trained with training samples. The training samples include 1000 sets of normal kitchen scene samples and 800 sets of precursor scene samples of open fire hazards. Each sample is labeled with corresponding feature information and true confidence labels. The 5-fold cross-validation is used in the training process to optimize the model parameters, and the evaluation index of the cross-validation is the accuracy rate. The extracted feature information is input into the trained confidence assessment model, and the output consists of real-time confidence scores corresponding to three sets of features: visual features, thermodynamic features, and temporal dynamic features. The scoring range is [0,1].
6. The method for detecting early warning signs of open flame hazards in kitchens based on multimodal methods according to claim 1, characterized in that, The S3 specifically includes: Real-time confidence scores based on three sets of features: visual features, thermodynamic features, and temporal dynamic features. Input to a dynamic weight allocation function; The dynamic weight allocation function normalizes the confidence scores to obtain the corresponding dynamic fusion weights. ; The weight allocation function is configured to: when it is determined from the temporal dynamic features that the current period is a peak period for cooking fumes, reduce the dynamic fusion weight corresponding to the visual features. .
7. The method for detecting early warning signs of open flame hazards in kitchens based on multimodal methods according to claim 1, characterized in that, The S3 specifically further includes: Based on the three groups of feature information of the extracted visual features, thermodynamic features, and temporal dynamic features, they are respectively input into three independent primary risk scoring models. The primary risk scoring model is a binary classification model based on logistic regression or support vector machine. Its training objective is to map the input feature vector into a primary risk score that represents the likelihood of an early sign of an open flame hazard in the current modality. The scoring range is [0,1].
8. The method for detecting early warning signs of open flame hazards in kitchens based on multimodal methods according to claim 7, characterized in that... , The specific implementation of the comprehensive risk assessment value in step S3: Based on the obtained dynamic fusion weights and the corresponding primary risk score calculated. The weighted linear summation is calculated using the following formula: 。 9. The method for detecting early warning signs of open flame hazards in kitchens based on multimodal methods according to claim 1, characterized in that, The step S4 specifically includes: Based on the aforementioned comprehensive risk assessment value It is compared with a preset set of graded risk thresholds that includes two levels; The set of hierarchical risk thresholds includes at least a first threshold T1 and a second threshold T2, and satisfies 0 < T1 < T2 ≤ 1. The values of T1 and T2 are initialized according to the kitchen environment characteristics. Determine the current risk level by comparison: If < T1, it is determined to be in a risk-free state; if T1 ≤ R < T2, it is determined to be in a primary warning state, and local acoustic and optical warnings are executed; if ≥ T2, it is determined to be in a high-level alarm state, and an emergency alarm including remote notification is executed; After the system issues an alarm, it continuously monitors the change of the comprehensive risk assessment value R. If the R value drops below T1 within the preset continuous monitoring time, and all modal feature information returns to normal, the alarm is automatically stopped, and a risk解除 notice is sent.
10. The method for detecting early warning signs of open flame hazards in kitchens based on multimodal methods according to claim 9, characterized in that, The method further includes: The system continuously records the historical comprehensive risk assessment value R and the event label of whether a real hidden danger finally occurs. Regularly use the recorded historical data to optimize and adjust the set of hierarchical risk thresholds T1, T2,... through machine learning algorithms. Update the optimized new threshold to the system to achieve the adaptive adjustment of the warning threshold according to different kitchen environments and user habits.