A gas use behavior recognition method and system based on multi-modal feature fusion

Abstract

This invention discloses a gas usage behavior recognition method and system based on multimodal feature fusion, belonging to the field of gas safety monitoring technology. The invention simultaneously acquires vibration and temperature signals. A one-dimensional residual network is used to extract vibration feature vectors from the vibration signals, and a network including convolutional neural networks and bidirectional long short-term memory networks is used to extract temperature feature vectors from the temperature signals. A cross-modal attention mechanism is used to mutually enhance the vibration and temperature modes, generating temperature-enhanced vibration features and vibration-enhanced temperature features. These are then concatenated to obtain a fused feature vector, which is used to identify the gas usage behavior category. The behavior category is then combined with gas concentration to output multi-level alarm reminders. This significantly reduces the false alarm rate caused by normal cooking and provides proactive detection capabilities for dangerous conditions such as dry burning.

Description

Technical Field

[0001] This invention relates to the field of gas safety monitoring technology, and in particular to a gas consumption behavior recognition method and system based on multimodal feature fusion. Background Technology

[0002] In home cooking environments, accurately distinguishing between normal cooking activities and actual gas leaks or dangerous open flames is a core technological challenge in the current gas safety monitoring field. Currently, most household gas alarms employ a single gas sensor-based monitoring solution. This solution integrates a catalytic combustion or semiconductor gas sensor within the alarm to collect real-time data on the concentration of combustible gas in the cooking area and compare it to a preset, fixed alarm threshold. When the detected concentration exceeds the threshold, the alarm triggers an audible and visual alarm. Some solutions, to address instantaneous gas concentration fluctuations during cooking, incorporate hardware delay circuits in the alarm circuitry or use software to set a fixed delay judgment window to filter out short-term concentration spikes.

[0003] However, the above-mentioned solutions suffer from insurmountable false alarm defects in real cooking environments. During cooking methods such as frying, stir-frying, and deep-frying, the stove releases trace amounts of combustible gas upon ignition. Combined with the cross-interference from oil fumes, steam, and the high temperature and humidity generated during cooking, this easily causes the sensor to output concentration signals exceeding the alarm threshold. Because a single gas sensor solution relies solely on gas concentration information, it lacks the ability to perceive the current activity state of the cooking area and cannot distinguish whether the increased concentration is due to normal cooking or a dangerous leak. Therefore, even with the addition of a delay circuit, it can only passively wait within a fixed time window, unable to dynamically adapt to different cooking durations and environmental changes. This leads to frequent false alarms during normal cooking, severely reducing user trust and the actual usability of the alarm system.

[0004] Existing technologies have attempted to incorporate vibration or temperature monitoring into the field of gas safety. For example, vibration sensors are used for vibration monitoring and periodic leak detection in gas pipelines, or fixed temperature probes are used for measuring the temperature of pot bottoms to prevent dry burning. However, in these solutions, vibration and temperature signals are used as independent alarm dimensions, without any collaborative fusion or cross-validation mechanism. Since a single vibration signal is insufficient to distinguish the operating status of various devices such as range hoods and blenders, and a single temperature signal is easily affected by ambient temperature fluctuations, this independent approach fails to fundamentally solve the reliability problem of identifying the activity status of the cooking area, and also cannot provide gas alarms with an effective decision context to dynamically adjust alarm strategies. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a gas usage behavior recognition method and system based on multimodal feature fusion. This invention deploys vibration and temperature sensors in the cooking area to collect vibration signals from cooking equipment and ambient temperature data. The vibration signals are used to identify the operating status of range hoods, gas stoves, and other cooking equipment, and temperature changes are combined to collaboratively determine the activity status of the cooking area. Based on this, this invention not only helps gas alarms reduce false alarm rates but also provides low-cost, privacy-free intelligent sensing services for smart home integration, cooking health management, equipment fault early warning, elderly care, and safety linkage control.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is: a gas consumption behavior recognition method based on multimodal feature fusion, comprising: acquiring synchronously collected vibration signals and temperature signals; The vibration signal is preprocessed, and a one-dimensional residual network is used to extract the vibration feature vector. h v ; The temperature signal is preprocessed, and a network including a convolutional neural network and a bidirectional long short-term memory network is used to extract the temperature feature vector. h t ; Based on the cross-modal attention mechanism, according to the vibration feature vector h v and the temperature feature vector h t The vibration eigenvector enhanced by temperature is calculated according to the following formula. h' v Temperature eigenvectors enhanced by vibration h' t : in, W vt , W tv The weight matrix is ​​a learnable matrix. b vt , b tv For bias terms, As the attention weight of temperature on vibration, The attention weight of vibration to temperature; Will h' v and h' t Concatenate the vectors to obtain the fused feature vector; Based on the fused feature vector, the gas consumption behavior category is identified; The cross-modal attention mechanism enables the characteristics of vibrational modes and temperature modes to mutually enhance each other.

[0007] Preferably, the preprocessing includes: bandpass filtering and frame windowing of the vibration signal, and moving average smoothing and Z-score normalization of the temperature signal.

[0008] Preferably, the one-dimensional residual network is 1D-ResNet18, the convolutional neural network includes two convolutional layers with a kernel size of 3 and a max pooling layer, and the bidirectional long short-term memory network is a single bidirectional LSTM with 64 hidden units per direction.

[0009] Preferably, the gas usage behavior categories include cooking, stewing soup, equipment idling, other equipment running, dry burning, no one operating, abnormal conditions, and non-cooking states.

[0010] Preferably, the gas consumption behavior recognition method further includes: acquiring a gas concentration signal; The gas consumption behavior category is combined with the gas concentration signal, and multi-level alarm reminders are output according to preset rules. The multi-level alarm notification includes four levels: normal monitoring, status prompt, observation, and danger alert.

[0011] Preferably, the triggering rules for the multi-level alarm notification include: A hazard warning is triggered when the gas concentration exceeds 25% of the lower explosive limit. A hazard alert is triggered when the gas concentration exceeds the standard and the behavior category is dry burning or no one is abnormal. When the gas concentration exceeds the standard and the behavior category is equipment idling, other equipment running, or non-cooking state, a monitoring alert will be triggered. When the gas concentration exceeds the standard and the behavior category is normal cooking or stewing soup, a status prompt is triggered.

[0012] Preferably, the gas consumption behavior recognition method further includes: caching the vibration signal and temperature signal corresponding to samples with a confidence level greater than 0.9; uploading the samples to the cloud; using the elastic weight consolidation algorithm to incrementally update the parameters of the one-dimensional residual network, the network containing the convolutional neural network and the bidirectional long short-term memory network, and the cross-modal attention mechanism; and distributing the updated parameters.

[0013] A gas consumption behavior recognition system based on multimodal feature fusion includes: a vibration sensor for collecting vibration signals; Temperature sensor, used to collect temperature signals; A signal preprocessing module is used to preprocess the vibration signal and the temperature signal; The vibration feature extraction module is used to extract vibration feature vectors from the preprocessed vibration signal using a one-dimensional residual network. h v ; The temperature feature extraction module is used to extract temperature feature vectors from the preprocessed temperature signal using a network that includes a convolutional neural network and a bidirectional long short-term memory network. h t ; A cross-modal attention fusion module is used to perform the following based on the vibration feature vector. h v and the temperature feature vector h t The vibration eigenvector enhanced by temperature is calculated according to the following formula. h' v Temperature eigenvectors enhanced by vibration h' t : in, W vt , W tv The weight matrix is ​​a learnable matrix. b vt , b tv For bias terms, As the attention weight of temperature on vibration, The attention weight of vibration to temperature; And h' v and h' t The fused feature vector is obtained by concatenating the features. The behavior recognition module is used to identify the gas consumption behavior category based on the fused feature vector; The cross-modal attention fusion module enables the mutual enhancement of features between vibration modes and temperature modes.

[0014] Preferably, it also includes a gas alarm and a collaborative decision-making module; The collaborative decision-making module is used to acquire gas concentration signals and output multi-level alarm reminders based on the gas usage behavior category.

[0015] Preferably, the collaborative decision-making module is specifically used to execute the following triggering rules: A hazard warning is triggered when the gas concentration exceeds 25% of the lower explosive limit. A hazard alert is triggered when the gas concentration exceeds the standard and the behavior category is dry burning or no one is abnormal. When the gas concentration exceeds the standard and the behavior category is equipment idling, other equipment running, or non-cooking state, a monitoring alert will be triggered. When the gas concentration exceeds the standard and the behavior category is normal cooking or stewing soup, a status prompt is triggered.

[0016] By adopting the above technical solution, the present invention has the following beneficial effects.

[0017] (1) This invention overcomes the shortcomings of existing gas alarm schemes that treat vibration and temperature as independent alarm dimensions and cannot distinguish between normal cooking and actual leakage by introducing a cross-modal attention fusion mechanism of vibration mode and temperature mode. This scheme enables vibration features and temperature features to enhance each other during the identification process: vibration features focus on temperature changes to confirm the actual heating state of the stove, and temperature features focus on vibration patterns to eliminate interference from ambient temperature fluctuations. The two modes cross-validate in the feature space, realizing accurate differentiation between normal gas usage behaviors such as "cooking" and "stewing soup" and dangerous states such as "dry burning" and "abnormal absence of people". This fundamentally solves the technical problem of difficulty in distinguishing between normal cooking activities and gas leaks in the cooking area.

[0018] (2) This invention constructs a multi-level alarm strategy by coordinating behavior recognition results with gas concentration signals. This strategy uses behavior category as the control factor for dynamically adjusting alarm logic. For two different behavior contexts, "normal cooking" and "dry burning", different levels of reminders are triggered even if the gas concentration is the same. This enables the alarm system to have scene adaptive capability, significantly reduces the false alarm rate, and achieves for the first time the proactive perception and reminder of non-leakage dangerous events such as dry burning.

[0019] (3) This invention achieves continuous evolution of the recognition model after deployment through an incremental learning mechanism. The edge device transmits high-confidence samples back to the cloud, and uses an elastic weight consolidation algorithm to incrementally update the cross-modal attention mechanism and each neural network branch, so that the model can continuously adapt to sensor drift, equipment replacement and environmental noise changes, maintain long-term stable recognition accuracy, and break through the limitations of traditional gas alarm equipment firmware solidification and performance degradation with the use time.

[0020] (4) The present invention can achieve fine-grained perception of the activity status of the cooking area by using only vibration sensors and non-contact temperature sensors. It does not require cameras or gas sensor arrays, which greatly reduces hardware costs and system power consumption. It completely avoids privacy disputes in the cooking area from the perspective of physical perception, making it possible to deploy intelligent perception functions on a large scale in ordinary households.

[0021] (5) This invention deploys vibration sensors and temperature sensors in the cooking area to collect vibration signals from the cooking area equipment and ambient temperature data. The vibration signals are used to identify the working status of the range hood, gas stove, and other cooking area equipment, and combined with temperature changes to collaboratively determine the activity status of the cooking area. Based on this, this invention not only helps gas alarms reduce false alarm rates but also provides low-cost, privacy-free intelligent sensing services for smart home integration, cooking health management, equipment fault early warning, elderly care, and safety linkage control. Attached Figure Description

[0022] The following provides a detailed discussion of the manufacture and application of preferred embodiments of the present invention. However, it should be understood that the present invention provides many applicable inventive concepts that can be embodied in various specific environments. The specific embodiments discussed are merely illustrative of specific ways of manufacturing and using the present invention and do not limit the scope of the invention. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort.

[0023] Figure 1 This is a schematic diagram of the method of the present invention.

[0024] Figure 2 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0025] The following provides a detailed discussion of the manufacture and application of preferred embodiments of the present invention. However, it should be understood that the present invention provides many applicable inventive concepts that can be embodied in various specific environments. The specific embodiments discussed are merely illustrative of specific ways of manufacturing and using the invention and do not limit the scope of the invention.

[0026] This invention overcomes the shortcomings of existing gas alarm schemes that treat vibration and temperature as independent alarm dimensions and cannot distinguish between normal cooking and actual leaks by introducing a cross-modal attention fusion mechanism of vibration and temperature modes. This scheme enables vibration and temperature features to mutually reinforce each other during the identification process: vibration features focus on temperature changes to confirm the actual heating status of the stove, while temperature features focus on vibration patterns to eliminate interference from ambient temperature fluctuations. The two modes cross-validate in the feature space, achieving accurate differentiation between normal gas usage behaviors such as "cooking" and "simmering soup" and dangerous states such as "dry burning" and "abnormal absence of personnel." This fundamentally solves the technical problem of distinguishing between normal cooking activities and gas leaks in cooking areas. The following section, in conjunction with the appendix... Figure 1-2 To elaborate further.

[0027] Figure 1 This is a flowchart illustrating a gas consumption behavior recognition method based on multimodal feature fusion, provided in an embodiment of the present invention. (Refer to...) Figure 1The method in this embodiment mainly includes the following steps: Step S1: Acquire synchronously collected vibration signals and temperature signals.

[0028] In this embodiment, a multimodal sensor integrating a vibration sensing module, a temperature sensing module, and a gas concentration sensing module is preferably installed on the casing of the range hood near the cooktop, rather than directly on the cooktop itself. This installation method has the following advantages: First, the built-in high-sensitivity fiber optic vibration sensor can simultaneously cover the vibration signal acquisition range of multiple types of kitchen equipment. It can clearly capture the low-frequency vibrations generated during the ignition and combustion process of the gas stove, and simultaneously acquire the medium- and high-frequency vibrations generated during the operation of the range hood, as well as the operating vibration characteristics of other cooking area equipment such as blenders, microwave ovens, dishwashers, and electric kettles, providing comprehensive vibration data support for judging the overall activity status of the kitchen. Furthermore, this installation location is compatible with various convenient installation methods such as magnetic bases, adhesive backing, or screw fixing, requiring no modification to the cooktop itself, and possessing good engineering feasibility and user acceptance. Specifically, in this embodiment, the vibration signals are collected by a vibration sensor installed on the casing of the cooking area equipment. The vibration sensor is a high-sensitivity fiber optic vibration sensor, which is installed on the range hood casing near the cooktop using a magnetic base, adhesive backing, or screws. The bottom of the casing fits snugly against the mounting surface to maximize vibration transmission efficiency. The vibration signal is sampled at a frequency of 8kHz.

[0029] The temperature signal is acquired by a non-contact temperature sensor that continuously monitors the ambient temperature of the cooking area at a sampling frequency of 2Hz.

[0030] The vibration sensor and temperature sensor are sampled by a timer of the same microcontroller, ensuring that the data collected by both have the same global timestamp, providing a basis for subsequent multimodal time alignment.

[0031] Step S2: Preprocess the vibration signal and extract the vibration feature vector using a one-dimensional residual network. h v .

[0032] The preprocessing of the vibration signal includes: performing bandpass filtering on the original vibration signal from 0.5kHz to 3kHz to remove circuit noise and irrelevant low-frequency environmental interference; then performing frame-by-frame windowing processing, with the frame length set to 10 seconds and each frame containing 80,000 sampling points at an 8kHz sampling rate.

[0033] The preprocessed single-frame vibration data has a dimension of [80000, 1] and is input into a one-dimensional residual network for feature extraction. The specific structure of this one-dimensional residual network is as follows: First, it passes through an initial convolutional layer with a kernel size of 7, a stride of 2, and 64 output channels, followed by a ReLU activation function. Then, it passes through four residual blocks in sequence. Residual block 1 contains two convolutional layers, each with a kernel size of 3, a stride of 1, and 64 channels. Residual block 2 contains two convolutional layers, where the first convolutional layer has a stride of 2 to perform downsampling, and the second layer has a stride of 1, increasing the number of channels to 128. Residual block 3 has a first convolutional layer with a stride of 2, a second layer with a stride of 1, increasing the number of channels to 256. Residual block 4 has a first convolutional layer with a stride of 2, a second layer with a stride of 1, increasing the number of channels to 512. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The multi-channel feature map output from residual block 4 is fed into a global average pooling layer, which compresses the feature map of each channel into a scalar, resulting in a 512-dimensional feature vector. This vector is then mapped to 256 dimensions through a fully connected layer, ultimately outputting the vibration feature vector. h v .

[0034] Step S3: Preprocess the temperature signal by extracting temperature feature vectors using a network comprising a convolutional neural network and a bidirectional long short-term memory network. h t .

[0035] The preprocessing of the temperature signal includes: smoothing the original temperature sequence by moving average with a window length of 3 sampling points to filter out instantaneous measurement noise; and then performing Z-Score normalization to make the data distribution conform to a standard normal distribution.

[0036] Since the temperature sampling frequency is 2Hz, a total of 20 temperature sampling points are obtained within a 10-second analysis frame. Therefore, the dimension of the preprocessed single-frame temperature data is [20,1].

[0037] The temperature data is input into a network containing a convolutional neural network (CNN) and a bidirectional long short-term memory (LSTM) network. The CNN structure is as follows: a first convolutional layer with a kernel size of 3, a stride of 1, and 16 output channels, followed by a ReLU activation function; a second convolutional layer with a kernel size of 3, a stride of 1, and 32 output channels, followed by a ReLU activation function; and a max-pooling layer with a kernel size of 2, a stride of 2, and an output feature map dimension of approximately [10, 32]. The pooling output is flattened by a dimension transformation layer and mapped to a sequence of [10, 64] through a linear layer to adapt to subsequent network operations. The bidirectional LSTM network is a single-layer bidirectional LSTM with 64 hidden units per direction and a total output dimension of 128. The output of the last time step is taken and mapped to 256 dimensions through a fully connected layer to obtain the temperature feature vector. ht .

[0038] Step S4: Based on the cross-modal attention mechanism, according to the vibration feature vector h v and the temperature feature vector h t The vibration eigenvector enhanced by temperature is calculated according to the following formula. h' v Temperature eigenvectors enhanced by vibration h' t and will h' v and h' t The features are concatenated to obtain a fused feature vector.

[0039] The calculation process in this step satisfies the following relationship: in, W vt , W tv The weight matrix is ​​a learnable matrix with dimensions [1, 256]. b vt , b tv This is the bias term. The attention weight of vibration to temperature is calculated using the first relation. This weight characterizes the distribution of correlation strength between the current vibration characteristics and temperature characteristics across different dimensions. The attention weight of temperature on vibration is calculated using the second relationship. This weight represents the distribution of the correlation strength between the current temperature characteristics and the various dimensions of the vibration characteristics.

[0040] After obtaining the attention weights, Compared with the original vibration characteristics h v Element-wise multiplication yields the vibrational characteristic vector with enhanced temperature. h' v This operation significantly amplifies the components of the vibration characteristics that are highly coupled with temperature change trends, while suppressing isolated vibration components unrelated to ambient temperature fluctuations. Similarly, it... Compared with the original temperature characteristics h t Element-wise multiplication yields the temperature eigenvector of enhanced vibration. h' t This strengthens the dynamic components in the temperature characteristics that co-evolve with the vibration modes.

[0041] Will h' v and h't By concatenating the features along the feature dimension, a fused feature vector with a dimension of 512 is obtained.

[0042] The core mechanism behind the outstanding technical effect of this embodiment lies in the fact that the aforementioned cross-modal attention fusion is not a simple feature splicing, but rather achieves mutual enhancement of vibration and temperature modes at the feature level. Specifically, in real cooking scenarios, there is a stable temporal coupling relationship between the vibration signal generated by the ignition and continuous operation of the gas stove and the rising temperature of the surrounding environment. Simultaneously, users typically activate the range hood when using the gas stove, and the continuous mid-to-high frequency vibration generated by the range hood provides crucial auxiliary features for determining active cooking behavior. The cross-modal attention weights can automatically capture and amplify the aforementioned co-evolutionary correlation between vibration and temperature. For interference signals such as environmental noise, isolated vibrations, or natural temperature fluctuations caused by non-cooking factors, the corresponding feature expressions are suppressed by the attention weights because they lack the aforementioned co-evolutionary correlation. Through cross-validation and mutual contextualization of the two modes, boundary samples that were previously difficult to distinguish in a single mode exhibit clear separability in the fused feature space, fundamentally improving the robustness and accuracy of gas usage behavior recognition.

[0043] Step S5: Identify the gas usage behavior category based on the fused feature vector.

[0044] Specifically, the 512-dimensional fused feature vector is input into a fully connected layer, mapped to a 7-dimensional output vector, and then the probability distribution of various gas usage behaviors is calculated using the Softmax function. This embodiment defines seven categories of gas usage behaviors: cooking, stewing, equipment idling, other equipment running, dry burning, no one present (abnormal situation), and non-cooking state. The system selects the category corresponding to the highest probability as the gas usage behavior recognition result for the current moment.

[0045] In a preferred embodiment, to further enhance the model's adaptability during long-term deployment, the method further includes an incremental learning step. Specifically, after each inference, it is determined whether the confidence level of the current recognition result is greater than a preset threshold of 0.9. If so, the vibration signal, temperature signal, and recognition result corresponding to the sample are stored as pseudo-labels in the local cache. The local cache size is set to 2000 entries. Every morning, the system randomly samples 500 high-confidence samples from the cache and uploads them to the cloud server via the communication module. After the cloud aggregates data from multiple terminal devices, the elastic weight consolidation algorithm is used to incrementally train all learnable parameters in the one-dimensional residual network, the network containing convolutional neural networks and bidirectional long short-term memory networks, and the cross-modal attention mechanism. The loss function of the elastic weight consolidation algorithm, based on the cross-entropy loss, adds a regularization constraint term for important parameters of the old task, with the regularization coefficient... λThe value is set to 1000. After incremental training is completed in the cloud, the updated model parameters are incrementally downloaded to each terminal device via over-the-air (OTA). This mechanism enables the recognition model to continuously adapt to data distribution shifts caused by factors such as sensor aging, replacement of cooking area equipment, and seasonal environmental changes, maintaining long-term stable recognition accuracy.

[0046] In a preferred embodiment, the above method further includes a multi-level alarm decision-making step. Specifically, the gas concentration signal collected by the gas alarm is acquired, the gas usage behavior category identified in step S5 is combined with the gas concentration signal, and a multi-level alarm reminder is output according to preset rules. The multi-level alarm reminder is divided into four levels: L0 is normal monitoring, with the indicator light remaining solid green; L1 is status indication, with the yellow warning light on, outputting the current device status information; L2 is observation, with the buzzer sounding intermittently at a low frequency and the indicator light flashing slowly red, outputting a reminder message; L3 is danger warning, with the buzzer sounding continuously at a high frequency and the indicator light flashing rapidly red, sending a notification to the user terminal through the communication module, and sending a gas shut-off command to the linkage solenoid valve.

[0047] The specific configuration of the multi-level alarm trigger rules is as follows: When the gas concentration exceeds 25% of the lower explosive limit, an L3 hazard warning is triggered regardless of the current behavior category; when the gas concentration exceeds the limit and the behavior category output is dry burning or no one is present, an L3 hazard warning is triggered; when the gas concentration exceeds the limit and the behavior category output is equipment idling, other equipment running, or non-cooking state, an L2 observation warning is triggered; when the gas concentration exceeds the limit and the behavior category output is normal cooking or stewing, only an L1 status prompt is triggered; when the gas concentration does not exceed the limit but the behavior category output is dry burning or no one is present, an L2 observation warning is triggered; when the gas concentration does not exceed the limit but the behavior category output is equipment idling, an L1 status prompt is triggered; in other cases, L0 normal monitoring is maintained. The core of this multi-level alarm decision mechanism is that the behavior recognition result serves as a control factor for dynamically adjusting the alarm logic, enabling the same gas concentration to trigger drastically different warning levels in different behavior contexts, giving the alarm system scene perception and adaptive decision-making capabilities.

[0048] Furthermore, embodiments of the present invention also provide a gas consumption behavior recognition system based on multimodal feature fusion, such as... Figure 2 As shown in the diagram, the system includes a multi-source signal acquisition module, a signal preprocessing module, a deep learning recognition and fusion module, a collaborative decision-making module, and a communication interface module. The functions and connections of each module are as follows.

[0049] The multi-source signal acquisition module includes a vibration sensor, a temperature sensor, and a gas sensor; these are used to measure vibration and temperature within the cooking area, as well as the gas concentration of the gas alarm itself. Vibration measurement employs a high-sensitivity fiber optic vibration sensor, installed on the outer casing of the kitchen equipment, preferably on the casing of the range hood near the cooktop. This sensor collects vibration signals generated during the operation of the kitchen equipment. It can simultaneously detect low-frequency vibrations generated during gas stove ignition and combustion, medium-to-high-frequency vibrations generated during range hood operation, and vibration characteristics of other kitchen equipment (blenders, microwave ovens, dishwashers, electric kettles, etc.). Temperature measurement uses a non-contact temperature sensor to continuously monitor the kitchen ambient temperature. The vibration sampling frequency is 8kHz-16kHz, and the temperature sampling frequency is 0.1-2Hz.

[0050] The signal preprocessing module performs bandpass filtering (0.5-3kHz) and frame windowing on the vibration signal (10-second frame length, 80,000 sampling points per frame). It also performs moving average smoothing (window length 3) and Z-score normalization on the temperature signal. The high-frequency vibration feature vector and low-frequency temperature feature vector are aligned by timestamp using linear interpolation. For the start and end timestamps of each vibration analysis frame, the corresponding two temperature sampling points are found in the temperature time series data, and linear interpolation is performed to obtain 20 temperature values ​​corresponding to that vibration frame (20 points in 10 seconds due to a 2Hz sampling rate), forming a synchronous sample pair. Its data structure is [Vibration data:(80000,1), Temperature data:(20,1)].

[0051] Deep learning recognition and fusion module: This module adopts a dual-branch neural network and cross-modal attention fusion architecture, as detailed below.

[0052] (1) Vibration Feature Extraction Branch: One-dimensional temporal features are extracted using 1D-ResNet18. The specific network parameters are as follows: Input layer dimension [80000,1] (corresponding to a single-channel vibration signal sampled for 10 seconds and 8kHz). Initial convolutional layer: kernel size 7, stride 2, output channels 64, followed by ReLU activation function. Residual block configuration: Residual block 1 contains 2 convolutional layers, each with kernel size 3, stride 1, and 64 channels; Residual block 2 contains 2 convolutional layers, the first convolutional layer stride 2 (downsampling), the second layer stride 1, and 128 channels; Residual block 3 contains 2 convolutional layers, the first convolutional layer stride 2, the second layer stride 1, and 256 channels; Residual block 4 contains 2 convolutional layers, the first convolutional layer stride 2, the second layer stride 1, and 512 channels. Batch normalization (BatchNorm) and ReLU activation function are used after each convolutional layer. Global average pooling layer: The kernel size depends on the feature map size output by residual block 4. It globally averages each channel of the 512-channel feature map to output a 512-dimensional feature. Fully connected layer: Maps the 512-dimensional feature to 256 dimensions to obtain the vibration feature vector. h v The dimension is 256.

[0053] (2) Temperature Feature Extraction Branch: A CNN+BiLSTM network structure is used to extract temperature data features. The specific network parameters are: input layer dimension [20, 1] (20 temperature points within 10 seconds). CNN part: Convolutional layer 1: kernel size 3, stride 1, output channels 16, followed by ReLU activation; Convolutional layer 2: kernel size 3, stride 1, output channels 32, followed by ReLU activation; Max pooling layer: kernel size 2, stride 2, output feature map dimension approximately [10, 32] (time step becomes 10, number of channels 32). Dimension transformation layer: The CNN output is flattened and mapped to [10, 64] (time step 10, feature dimension 64) through a linear layer to adapt to the BiLSTM input. BiLSTM part: 1-layer bidirectional LSTM, 64 hidden units per direction, total output dimension 128, taking the output of the last time step. Output fully connected layer: Maps 128 dimensions to 256 dimensions to obtain temperature feature vector h_t with dimensions of 256.

[0054] (3) Multimodal fusion layer: A cross-modal collaborative attention mechanism is adopted to calculate the attention weight of vibration on temperature. Attention weights of vibration and temperature The calculation formula is as follows: in, W vt , W tv The weight matrix is ​​a learnable matrix.b vt , b tv The bias term has dimensions [1, 256]. After calculating the attention weights, the original features are enhanced, and the temperature-enhanced vibration feature vector is calculated according to the following formula. h' v Temperature eigenvectors enhanced by vibration h' t :

[0055] Perform the enhancement on the two feature vectors. h' v and h' t By concatenating the features, a fused feature vector is obtained. h fusion =[ h' v , h ' t ], with a dimension of 512.

[0056] (4) Behavior recognition layer: The fully connected layer maps the 512-dimensional fused features to 7 dimensions, and then outputs the probability distribution of various gas usage behaviors through the Softmax function. The following behavior categories are defined: cooking; stewing soup; equipment idling; other equipment running; dry burning; no one is present and there is an abnormality; non-cooking state.

[0057] (5) Incremental Learning Module: After deployment, the edge device caches high-confidence inference results and raw data, and periodically uploads them back to the cloud. The high-confidence threshold is set to 0.9, meaning that only when the model's maximum prediction probability for a sample is greater than 0.9 will the sample (vibration data, temperature data, and predicted category pseudo-label) be stored in the local cache. The local cache buffer size is set to 2000 entries. Every morning, the system randomly samples 500 high-confidence samples from the buffer and uploads them to the cloud. The cloud uses the Experience Playback + Elastic Weight Consolidation (EWC) method to incrementally update the model. The loss function of EWC is: in L new ( θ The cross-entropy loss is represented by the cross-entropy loss on the new data. θ i These are the current model parameters. θ' i For the old model parameters, F i The diagonal elements of the Fisher information matrix (measuring parameter importance), regularization coefficients λ=1000. After completing incremental training every weekend, the updated model parameters are incrementally distributed to the terminal devices via OTA (Over-The-Air).

[0058] Collaborative Decision Module: Combines the behavior categories and confidence levels output by deep learning with the gas concentration signal to output multi-level reminder decisions.

[0059] Communication interface module: Supports Wi-Fi and 4G / 5G wireless communication, and can connect to gas alarms, smart home gateways, mobile apps and other terminals.

[0060] To verify the technical effectiveness of this invention, the applicant conducted a 30-day continuous comparative test in a typical residential cooking area of ​​approximately 6 square meters. The test cooking area was equipped with a side-suction range hood and a built-in gas stove. The baseline scheme A used only a single catalytic combustion gas sensor, with the alarm threshold set at 8% of the lower explosive limit. Scheme B, the scheme of this invention, employed a vibration sensor and a temperature sensor and executed the complete method described above. Approximately 90 real cooking activities were recorded during the test, covering various cooking methods such as frying, stir-frying, steaming, stewing, and deep-frying, and five simulated gas leaks and nine simulated dry-burning incidents were included. Existing single-sensor schemes and independent vibration and temperature alarm schemes both have significant false alarm defects and cannot effectively distinguish between normal cooking behavior and real dangerous conditions. For example, when using only gas sensors, the concentration of volatile organic compounds in cooking fumes rises instantaneously during stir-frying, and its equivalent methane concentration can exceed the alarm threshold, triggering a false alarm. Similarly, when using only temperature sensors, a rapid temperature rise during normal cooking can also trigger an abnormal alarm. Furthermore, if vibration and temperature are treated as independent alarm dimensions, the lack of a cross-verification mechanism between modes means that vibrations generated by a range hood operating alone can easily be misinterpreted as cooking activity, and gas leaks accompanying the range hood's operation can be masked as normal gas usage. In other words, under a single dimension, the signal characteristics of normal cooking behavior and actual hazardous conditions overlap, making effective differentiation impossible. This invention, through a cross-modal fusion mechanism of vibration and temperature, achieves mutual enhancement and cross-verification of the two modal characteristics, effectively distinguishing between normal cooking activities and actual hazardous conditions, significantly reducing the false alarm rate, and thus effectively solving the aforementioned core deficiencies. As shown in Table 1, the test results show that: Scheme A triggered 12 alarms in 30 days, of which 7 were false alarms caused by normal cooking activities, resulting in an extremely high false alarm rate; Scheme B triggered 14 alarms, including 6 L3 level alarms and 8 L2 level alarms, of which only 1 was a false alarm, reducing the number of false alarms by 85.7% compared to Scheme A, and can also provide proactive reminders for dry burning events.

[0061] Table 1: Comparison of test results between the present invention and a single gas sensor.

[0062]

[0063] Regarding the accuracy of gas usage behavior recognition, Scheme A lacks any behavior recognition capability, while Scheme B achieves an accuracy rate of over 85% for all seven types of behavior, with a 92% accuracy rate for non-cooking states, as shown in Table 2. This comparison clearly demonstrates that the present invention, through cross-modal attention fusion of vibration and temperature, can accurately distinguish between normal cooking activities and real dangerous states—an effect that cannot be achieved by a single modality or simple modal splicing.

[0064] Table 2: Accuracy Comparison of Single Gas Sensor Scheme and Gas Behavior Recognition Scheme of the Present Invention.

[0065]

[0066] Although the specification has provided a detailed description, it should be understood that various changes, substitutions, and modifications can be made without departing from the spirit and scope of the invention as defined by the appended claims. Furthermore, the specific embodiments described are not intended to limit the scope of the invention, and those skilled in the art will readily understand based on this invention that existing or future-developed processes, machines, manufactures, compositions of matter, means, methods, or steps can perform substantially the same functions or achieve substantially the same results as the embodiments of the invention. Therefore, the appended claims are intended to include such processes, machines, manufactures, compositions of matter, means, methods, or steps within their scope.

Claims

1. A gas use behavior recognition method based on multi-modal feature fusion, characterized in that, include: Acquire synchronously collected vibration and temperature signals; The vibration signal is preprocessed, and a one-dimensional residual network is used to extract a vibration feature vector h v ; preprocessing the temperature signal, extracting a temperature feature vector using a network comprising a convolutional neural network and a bidirectional long short-term memory network h t ; based on a cross-modal attention mechanism, according to the vibration feature vector h v and the temperature feature vector h t , the vibration-enhanced temperature feature vector is calculated according to the following relationship h' v and the vibration-enhanced temperature feature vector h' t : wherein, W vt , W tv is a learnable weight matrix, b vt , b tv is a bias term, is a temperature-to-vibration attention weight, is a vibration-to-temperature attention weight; Will h' v With h' t Splicing in feature dimension, obtaining fusion feature vector; The fused feature vector is input into the fully connected layer and mapped to the output vector. The probability distribution of various gas consumption behaviors is calculated by the Softmax function, and the category corresponding to the maximum probability is selected as the gas consumption behavior recognition result. The cross-modal attention mechanism enables the characteristics of vibrational modes and temperature modes to mutually enhance each other.

2. The method of claim 1, wherein, The preprocessing includes: bandpass filtering and frame windowing of the vibration signal, and moving average smoothing and Z-score normalization of the temperature signal.

3. The method of claim 1, wherein, The one-dimensional residual network is 1D-ResNet18, the convolutional neural network includes two convolutional layers with a kernel size of 3 and a max pooling layer, and the bidirectional long short-term memory network is a single bidirectional LSTM with 64 hidden units per direction.

4. The method according to any one of claims 1 to 3, characterized in that, The gas usage behavior categories include cooking, stewing soup, equipment idling, other equipment running, dry burning, no one operating, abnormal conditions, and non-cooking states.

5. The method of claim 1, wherein, The gas consumption behavior recognition method further includes: Acquire gas concentration signal; The gas consumption behavior category is combined with the gas concentration signal, and multi-level alarm reminders are output according to preset rules. The multi-level alarm notification includes four levels: normal monitoring, status prompt, observation, and danger alert.

6. The method of claim 5, wherein, The triggering rules for the multi-level alarm notification include: A hazard warning is triggered when the gas concentration exceeds 25% of the lower explosive limit. A hazard alert is triggered when the gas concentration exceeds the standard and the behavior category is dry burning or no one is abnormal. When the gas concentration exceeds the standard and the behavior category is equipment idling, other equipment running, or non-cooking state, a monitoring alert will be triggered. When the gas concentration exceeds the standard and the behavior category is normal cooking or stewing soup, a status prompt is triggered.

7. The method of claim 1, wherein, The gas consumption behavior recognition method further includes: The vibration and temperature signals corresponding to samples with a confidence level greater than 0.9 in the cached recognition results; The samples are uploaded to the cloud, and the parameters of the one-dimensional residual network, the network containing the convolutional neural network and the bidirectional long short-term memory network, and the cross-modal attention mechanism are incrementally updated using the elastic weight consolidation algorithm, and the updated parameters are then distributed.

8. A gas use behavior recognition system based on multi-modal feature fusion, adopting the gas use behavior recognition method according to any one of claims 1-7, characterized in that, include: Vibration sensors are used to collect vibration signals; Temperature sensor, used to collect temperature signals; A signal preprocessing module is used to preprocess the vibration signal and the temperature signal; a vibration feature extraction module, configured to extract a vibration feature vector from the preprocessed vibration signal by using a one-dimensional residual network h v ; The temperature feature extraction module is used to extract temperature feature vectors from the preprocessed temperature signal using a network that includes a convolutional neural network and a bidirectional long short-term memory network. h t ; A cross-modal attention fusion module is used to perform the following based on the vibration feature vector. h v and the temperature feature vector h t The vibration eigenvector enhanced by temperature is calculated according to the following formula. h' v Temperature eigenvectors enhanced by vibration h' t : in, W vt , W tv The weight matrix is ​​a learnable matrix. b vt , b tv For bias terms, As the attention weight of temperature on vibration, The attention weight of vibration to temperature; And h' v and h' t The fused feature vector is obtained by concatenating the features along the feature dimension. The behavior recognition module is used to input the fused feature vector into the fully connected layer, map it to the output vector, calculate the probability distribution of various gas consumption behaviors through the Softmax function, and select the category corresponding to the maximum probability as the gas consumption behavior recognition result. The cross-modal attention fusion module enables the mutual enhancement of features between vibration modes and temperature modes.

9. The system according to claim 8, characterized in that, The gas consumption behavior recognition system also includes a gas alarm and a collaborative decision-making module; The collaborative decision-making module is used to acquire gas concentration signals and output multi-level alarm reminders based on the gas usage behavior category.

10. The system according to claim 9, characterized in that, The collaborative decision-making module is specifically used to execute the following triggering rules: A hazard warning is triggered when the gas concentration exceeds 25% of the lower explosive limit. A hazard alert is triggered when the gas concentration exceeds the standard and the behavior category is dry burning or no one is abnormal. When the gas concentration exceeds the standard and the behavior category is equipment idling, other equipment running, or non-cooking state, a monitoring alert will be triggered. When the gas concentration exceeds the standard and the behavior category is normal cooking or stewing soup, a status prompt is triggered.

Images