Intelligent electric appliance cooking category detection method based on infrared and audio dynamic fusion

Abstract

The application relates to an intelligent electric appliance cooking category detection method based on infrared and audio dynamic fusion, which is applied to the field of intelligent electric appliances and comprises the following steps: judging whether an infrared sensor is blocked when a spatial temperature gradient disappears; the disappearance of the spatial temperature gradient is determined according to temperature data of a measured object collected by the infrared sensor; under the condition that the infrared sensor is judged to be blocked, recursively estimating temperature data by using a temperature-based state transition equation and an observation equation in a preset autoregressive filter to obtain temperature prediction values and corresponding confidence degrees which are obtained by constraining a temperature rise curve of an occlusion object and a material type of the occlusion object; performing multi-modal feature fusion on a multi-dimensional feature vector and corresponding confidence degrees in audio data collected by an audio acquisition device and the temperature prediction values and corresponding confidence degrees to output a category probability of a cooking heat state based on the intelligent electric appliance. Through the application, the problem that the cooking heat state detection based on the intelligent electric appliance is not accurate enough is solved.

Description

Technical Field

[0001] This application relates to the field of smart appliances, and in particular to a method for detecting the cooking category of smart appliances based on the dynamic fusion of infrared and audio. Background Technology

[0002] With the improvement of people's living standards and the promotion and popularization of technologies such as the Internet, big data, artificial intelligence, and voice interaction, more and more traditional lifestyles are gradually changing, and the use of home appliances is gradually moving towards intelligence. While bringing more convenience to users, the functions of various home appliances are also becoming more diversified. At present, some high-end range hoods have begun to integrate infrared temperature sensors to monitor the temperature of the stove and prevent dry burning and excessive oil temperature.

[0003] However, in actual cooking, the detection path of the infrared temperature sensor to the object being measured is blocked. Therefore, the temperature value of the object detected by the infrared temperature sensor is not accurate enough, and the evaluation angle of temperature for cooking thermal state detection is relatively singular, resulting in inaccurate cooking thermal state detection based on smart appliances.

[0004] There is currently no effective solution to the problem that the detection of cooking thermal status in smart appliances is not accurate due to the infrared temperature sensor being blocked and the evaluation angle being singular. Summary of the Invention

[0005] This embodiment provides a method for detecting the cooking category of smart appliances based on the dynamic fusion of infrared and audio, in order to solve the problem that the detection of the cooking thermal state of smart appliances is not accurate enough in related technologies.

[0006] Firstly, this embodiment provides a method for detecting the cooking category of a smart appliance based on the dynamic fusion of infrared and audio data. The smart appliance is equipped with an infrared sensor and an audio acquisition device. The method includes:

[0007] When the spatial temperature gradient disappears, it is determined that the infrared sensor is blocked; the disappearance of the spatial temperature gradient is determined by the temperature data of the object being measured collected by the infrared sensor.

[0008] If the infrared sensor is determined to be blocked, the temperature data is recursively estimated using the temperature-based state transition equation and observation equation in the preset autoregressive filter to obtain the temperature prediction value and its corresponding confidence level; the temperature prediction value and its corresponding confidence level are obtained by the temperature rise curve of the blocking object and the material type of the blocking object.

[0009] Multimodal feature fusion is performed on the multidimensional feature vector and its corresponding confidence level in the audio data collected by the audio acquisition device, and the temperature prediction value and its corresponding confidence level, to obtain a comprehensive feature vector;

[0010] The output is a category probability of the cooking thermal state of the smart appliance based on the comprehensive feature vector.

[0011] Through the above steps, temperature and audio data during the cooking process are collected using infrared sensors and audio acquisition devices, respectively. Based on the spatial temperature gradient of the temperature data, accurate judgment of occlusion is achieved. When occlusion is detected, the temperature data is recursively estimated using an autoregressive filter to obtain a predicted temperature value and its corresponding confidence level. Simultaneously, the estimated temperature prediction value and its corresponding confidence level are constrained by combining the temperature rise curve and material type of the obstructing object, achieving accurate temperature prediction under occlusion conditions and avoiding inaccurate detection of the thermal state of the tested object due to occlusion. Subsequently, by combining the multi-dimensional feature vector and confidence level corresponding to the audio data, as well as the predicted temperature value and confidence level, a comprehensive feature vector is obtained from both sound and temperature perspectives. Based on this comprehensive feature vector, the probability of the current cooking category is determined, achieving accurate detection of the thermal state of cooking based on smart appliances.

[0012] In some embodiments, the method further includes:

[0013] Based on the temperature-based state vectors of the previous and current time moments, and combined with the time difference between the previous and current time moments and a preset process noise, a temperature-based state transition equation is constructed; the process noise is used to quantify the uncertainty in the construction of the equation.

[0014] Based on the average temperature at the current moment and the observation matrix of temperature measurement by the infrared sensor, and combined with the measurement noise of the infrared sensor, a temperature-based observation equation is constructed.

[0015] Through the above steps, the state transition equation explicitly models the temporal evolution of temperature state using time difference and preset process noise, achieving a priori prediction of temperature change trends at unobserved times and quantifying the inherent uncertainties in model construction. The observation equation, by incorporating the current average temperature and the observation matrix of the infrared sensor, combines measurement noise to ensure the reliability of real-time measurement information. These two constructed equations adaptively balance prior evolution information and posterior measured information during the recursive estimation process, effectively suppressing measurement fluctuations caused by random errors in the infrared sensor and environmental interference. This significantly enhances the continuity and robustness of temperature estimation in dynamic scenarios, thereby achieving high-precision, low-latency dynamic tracking of the true temperature state.

[0016] In some embodiments, determining that the infrared sensor is blocked when the spatial temperature gradient disappears includes:

[0017] The temperature data of different regions in the region of interest of the measured object are processed by partial derivative to determine the spatial temperature gradient.

[0018] Within a preset time window, the spatial temperature gradient is calculated by an exponential moving average to determine the time baseline of the spatial temperature gradient.

[0019] Based on the time baseline and the preset occlusion factor, the spatial occlusion baseline value is determined;

[0020] If the spatial temperature gradient does not exceed the spatial occlusion baseline value, the spatial temperature gradient is determined to conform to the exponential movement range.

[0021] Through the above steps, when judging whether the spatial temperature gradient conforms to the exponential moving range based on temperature data, the spatial temperature gradient obtained by calculating the partial derivative of the temperature data within the sliding preset time window is exponentially averaged to obtain the time baseline. In actual scenarios, the thermal field distribution of the tested object is relatively uniform during normal cooking. Therefore, by comparing the results of the spatial temperature gradient and the time baseline, occlusion misjudgment under different cooking conditions can be avoided, thus improving the accuracy of occlusion judgment.

[0022] In some of these embodiments, the preset occlusion factor is set to 0.5.

[0023] By following the steps above, and considering the high temperatures involved in cooking, setting the preset occlusion factor to 0.5 can more accurately distinguish between actual occlusion and the situation caused by the increase in cooking temperature due to high-temperature steam, thereby further improving the accuracy of occlusion judgment.

[0024] In some embodiments, determining that the infrared sensor is blocked when the spatial temperature gradient disappears further includes:

[0025] Determine the single-frame temperature difference and average temperature of the temperature data, as well as the average temperature peak value in the average temperature;

[0026] If, within a preset time window, the temperature difference of a single frame of the temperature data does not exceed a preset temperature difference threshold, the difference between the peak and average temperatures meets a preset threshold range, and the spatial temperature gradient does not exceed the spatial occlusion baseline value, then the infrared sensor is determined to be occluded.

[0027] Through the above steps, when determining whether the object being measured by the infrared sensor is blocked, the judgment is made based on multiple conditions, such as the peak value of the temperature data, the instantaneous drop, and the change of the spatial temperature gradient, rather than relying on the change of the absolute temperature value. Among these, the change of the spatial temperature gradient can further improve the accuracy of distinguishing between real blockage and other cooking conditions, avoiding misjudgments and omissions.

[0028] In some embodiments, the method further includes:

[0029] If it is determined that the infrared sensor is blocked, the temperature rise curve and material type of the blocking object are obtained;

[0030] The thermal conduction time constant of the shield is determined based on the material type;

[0031] Based on the heat exchange process between the object under test and the shielding object, the estimated temperature of the object under test is determined according to the temperature rise curve of the shielding object and the heat conduction time constant.

[0032] Based on the estimated temperature of the object being measured, the predicted temperature value and its corresponding confidence level are constrained.

[0033] Through the above steps, before the occlusion occurs, the occluding object is usually not in the field of view of the infrared sensor, and the system cannot obtain the temperature of the occluding object in advance. Therefore, in this embodiment, by continuously tracking the temperature rise curve of the surface temperature of the occluding object from the moment the occlusion occurs, and based on the dynamic characteristics of the temperature rise curve and the thermal conduction time constant corresponding to the material type, the temperature range of the measured object can be calculated from the occluding device. Based on this temperature range, it serves as an auxiliary constraint for the predicted value of the autoregressive filter, further improving the prediction accuracy of the temperature prediction value and its corresponding confidence level.

[0034] In some embodiments, the method further includes:

[0035] If it is determined that the infrared sensor is not blocked, the predicted state vector and error covariance matrix of the previous sampling period are estimated based on the state transition equation.

[0036] Based on the predicted state vector and error covariance matrix in the previous sampling period, the prior state vector and prior error covariance matrix of the current sampling period are determined by using a preset autoregressive filter.

[0037] The filter gain is calculated based on the prior error covariance matrix and the measurement residual of the current sampling period; the measurement residual is determined by the measurement state vector obtained from the observation equation and the prior state vector.

[0038] The update state for the current sampling period is estimated based on the prior state vector and the filter gain.

[0039] Through the above steps, when the infrared sensor is not blocked, a preset autoregressive filter is used to predict the prior state vector and prior error covariance matrix in the current sampling period based on the predicted state vector and error covariance matrix in the previous sampling period. This enables the prediction of the temperature value in the current sampling period, effectively compensating for the information gap caused by the sensor's physical response delay, so as to achieve rapid tracking response when the temperature changes abruptly. Furthermore, the optimal weighted fusion between the prior predicted value and the current observed value is performed to estimate the updated state, so that the updated state estimate has the smallest statistically significant error variance.

[0040] In some embodiments, when it is determined that the infrared sensor is blocked, the step of recursively estimating the temperature data using the temperature-based state transition equation and observation equation in a preset autoregressive filter to obtain the predicted temperature value and its corresponding confidence level includes:

[0041] Determine the moment when the infrared sensor is blocked; obtain the prior state estimate and the prior error covariance matrix at the moment before the blocking occurs;

[0042] In the case where the infrared sensor is blocked, the state estimate and error covariance matrix at the current time of blocking are determined based on the prior state estimate at the moment before the blocking occurs.

[0043] Based on the state estimate at the current occlusion moment, the temperature prediction at the current occlusion moment is estimated using the temperature-based state transition equation and observation equation in the preset autoregressive filter.

[0044] Based on the error covariance matrix at the current occlusion time, the uncertainty quantization value at the current occlusion time is predicted, and the confidence level corresponding to the temperature prediction value is determined based on the uncertainty quantization value.

[0045] Through the above steps, when it is determined that the infrared sensor is blocked, the preset autoregressive filter predicts the state estimate and the confidence level corresponding to the quantified value of the prediction uncertainty at the current blocking moment based only on the prior state estimate and the prior error covariance matrix. At this time, the filter stops updating and only retains the prediction step. Combining the predicted temperature value and its corresponding confidence level, it provides the best prediction of the current thermal state, realizing a reference temperature prediction value that conforms to the temperature law under the blocking state. Moreover, the output error covariance matrix quantitatively characterizes the confidence range of the prediction, improving the accuracy of the final output.

[0046] In some embodiments, the multimodal feature fusion of the multidimensional feature vectors and their corresponding confidence scores in the audio data acquired by the audio acquisition device, and the temperature prediction value and its corresponding confidence score, to obtain a comprehensive feature vector includes:

[0047] Obtain the multidimensional feature vectors of the data frames in the audio data, and determine the total energy of the multidimensional feature vectors;

[0048] The sound confidence level of the data frame is determined based on the comparison result between the total energy and the preset silence threshold.

[0049] The multi-dimensional feature vector and sound confidence corresponding to the audio data, as well as the temperature prediction value and its corresponding confidence, are subjected to multi-modal feature weighting fusion to obtain a comprehensive feature vector; the comprehensive feature vector is used to input to a preset state classifier to output the cooking category probability.

[0050] Through the above steps, multimodal features including sound and temperature are extracted in real time, and a state classifier is used to achieve multimodal fusion classification, thereby realizing the technical effect of hierarchical safety early warning. Furthermore, there is no need to distinguish whether occlusion factors affect the accuracy of temperature data, improving the accuracy of the fused comprehensive feature vector and further enhancing the output accuracy of the cooking category probability.

[0051] Secondly, this embodiment provides a smart appliance that uses the smart appliance cooking state detection method based on the dynamic fusion of infrared and audio as described in the first aspect to output the category probability of the current cooking thermal state of the smart appliance. The smart appliance is one of a range hood or a combined range hood and stove.

[0052] Compared with related technologies, the smart appliance cooking category detection method based on dynamic fusion of infrared and audio provided in this embodiment collects temperature data and audio data during the cooking process through infrared sensors and audio acquisition devices, respectively. Based on the spatial temperature gradient of the temperature data, it accurately determines the presence of occlusion. When occlusion is detected, the temperature data is recursively estimated using an autoregressive filter to obtain a predicted temperature value and its corresponding confidence level. Simultaneously, the estimated predicted temperature value and its corresponding confidence level are constrained by combining the temperature rise curve and material type of the occluding object, achieving accurate temperature prediction under occlusion conditions and avoiding inaccurate detection of the thermal state of the tested object due to occlusion. Subsequently, the multi-dimensional feature vector and confidence level corresponding to the audio data, along with the predicted temperature value and confidence level, are fused from both sound and temperature perspectives to obtain a comprehensive feature vector. Based on this comprehensive feature vector, the probability of the current cooking category is determined, achieving accurate detection of the cooking thermal state of smart appliances.

[0053] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0055] Figure 1 This is a flowchart of a smart appliance cooking category detection method based on dynamic fusion of infrared and audio provided in an embodiment of this application;

[0056] Figure 2 This is a flowchart of a method for auxiliary constraint prediction based on the temperature of an obstruction, provided in an embodiment of this application.

[0057] Figure 3 This is a hardware structure block diagram of the terminal for the intelligent appliance cooking category detection method based on dynamic fusion of infrared and audio provided in this embodiment;

[0058] Figure 4 This is a flowchart of the method for tracking cooking thermal state provided in this specific embodiment;

[0059] Figure 5 This is a schematic diagram of the range hood structure provided in this specific embodiment;

[0060] Figure 6 This is a flowchart of a cooking safety early warning method based on dynamic fusion of infrared and audio provided in this specific embodiment. Detailed Implementation

[0061] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0062] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0063] Currently, some high-end range hoods have begun integrating infrared temperature sensors to monitor stovetop temperature and prevent dry burning and overheating of oil. Microphones are also used for voice control and basic cooking sound recognition. However, infrared temperature sensors become completely ineffective when obstructed by pot lids or affected by steam. Relying solely on sound for temperature judgment is difficult to accurately quantify and is susceptible to environmental noise. Existing infrared and sound fusion solutions use simple logical AND or OR operations, representing shallow fusion. When the infrared sensor fails due to obstruction, existing solutions lack a contingency plan and cannot continue to provide reliable safety warnings.

[0064] Therefore, to address the issue of infrared temperature sensors failing due to obstruction by objects such as pot lids, this embodiment provides a smart appliance equipped with an infrared sensor and an audio acquisition device. The smart appliance executes the following cooking category detection method based on dynamic fusion of infrared and audio data to output the category probability of the current cooking thermal state based on the smart appliance, thereby achieving accurate classification of the thermal state based on the smart appliance.

[0065] Based on the aforementioned smart appliances, this embodiment provides a method for detecting the cooking category of smart appliances based on the dynamic fusion of infrared and audio signals. Figure 1This is a flowchart of a smart appliance cooking category detection method based on dynamic fusion of infrared and audio provided in an embodiment of this application, as shown below. Figure 1 As shown, the process includes the following steps:

[0066] Step S110: When the spatial temperature gradient disappears, it is determined that the infrared sensor is blocked; the disappearance of the spatial temperature gradient is determined by the temperature data of the object being measured collected by the infrared sensor.

[0067] This method is mainly applied to the cooking process of smart appliances to detect the thermal state category of cooking based on smart appliances, including but not limited to categories such as normal heating, violent boiling, risk of overflowing, dry burning or other heating conditions.

[0068] In step S120, if it is determined that the infrared sensor is blocked, the temperature data is recursively estimated using the temperature-based state transition equation and observation equation in the preset autoregressive filter to obtain the temperature prediction value and its corresponding confidence level. The temperature prediction value and its corresponding confidence level are obtained by the temperature rise curve of the blocking object and the material type constraint of the blocking object.

[0069] Here, the autoregressive filter is a linear estimation algorithm that can achieve high-precision prediction, such as the Kalman filter and the optimal linear estimator. No specific limitations are imposed on the autoregressive filter here.

[0070] Step S130: Multimodal feature fusion is performed on the multidimensional feature vector and its corresponding confidence level in the audio data collected by the audio acquisition device, and the temperature prediction value and its corresponding confidence level, to obtain a comprehensive feature vector.

[0071] The multidimensional feature vectors of the audio data here include, but are not limited to, the feature vectors corresponding to the audio data in the time domain, frequency domain, wavelet domain, and auditory perception. By obtaining multidimensional feature vectors, the accuracy of audio data analysis can be achieved.

[0072] Step S140: Output the category probability of the cooking thermal state based on the comprehensive feature vector.

[0073] Through the above steps, temperature and audio data during the cooking process are collected using infrared sensors and audio acquisition devices, respectively. Based on the spatial temperature gradient of the temperature data, accurate judgment of occlusion is achieved. When occlusion is detected, the temperature data is recursively estimated using an autoregressive filter to obtain a predicted temperature value and its corresponding confidence level. Simultaneously, the estimated temperature prediction value and its corresponding confidence level are constrained by combining the temperature rise curve and material type of the obstructing object, achieving accurate temperature prediction under occlusion conditions and avoiding inaccurate detection of the thermal state of the tested object due to occlusion. Subsequently, by combining the multi-dimensional feature vector and confidence level corresponding to the audio data, as well as the predicted temperature value and confidence level, a comprehensive feature vector is obtained from both sound and temperature perspectives. Based on this comprehensive feature vector, the probability of the current cooking category is determined, achieving accurate detection of the thermal state of cooking based on smart appliances.

[0074] In some embodiments, the smart appliance cooking category detection method based on dynamic fusion of infrared and audio also includes:

[0075] Based on the temperature-based state vectors of the previous and current time moments, and combined with the time difference between the previous and current time moments and the preset process noise, a temperature-based state transition equation is constructed; the process noise is used to quantify the uncertainty in the construction of the equation.

[0076] Based on the average temperature at the current moment and the observation matrix of temperature measurement by infrared sensors, and combined with the measurement noise of infrared sensors, a temperature-based observation equation is constructed.

[0077] Through the above steps, the state transition equation explicitly models the temporal evolution of temperature state using time difference and preset process noise, achieving a priori prediction of temperature change trends at unobserved times and quantifying the inherent uncertainties in model construction. The observation equation, by incorporating the current average temperature and the observation matrix of the infrared sensor, combines measurement noise to ensure the reliability of real-time measurement information. These two constructed equations adaptively balance prior evolution information and posterior measured information during the recursive estimation process, effectively suppressing measurement fluctuations caused by random errors in the infrared sensor and environmental interference. This significantly enhances the continuity and robustness of temperature estimation in dynamic scenarios, thereby achieving high-precision, low-latency dynamic tracking of the true temperature state.

[0078] Furthermore, the heat capacity parameter can be incorporated into the state vector as a third state variable, and the type of the object being measured can be automatically identified through the temperature response in the first few sampling cycles, thereby further improving the prediction accuracy.

[0079] In some embodiments, step S110, determining that the infrared sensor is blocked when the spatial temperature gradient disappears, includes:

[0080] The temperature data of different regions in the region of interest of the measured object are processed by partial derivative to determine the spatial temperature gradient.

[0081] Within a preset time window, the spatial temperature gradient is calculated using an exponential moving average to determine the time baseline of the spatial temperature gradient.

[0082] The spatial occlusion baseline value is determined based on the temporal baseline and the preset occlusion factor;

[0083] If the space temperature gradient does not exceed the space occlusion baseline value, the space temperature gradient is determined to conform to the exponential shift range.

[0084] The region of interest (ROI) is determined using a connected component analysis algorithm based on a temperature threshold. During the cooking process, temperature data is collected by an infrared sensor mounted on the smart appliance. The temperature matrix, after background subtraction, is then determined based on this data. The comparison between the temperature matrix and the hotspot detection threshold is binarized. Next, connected component analysis is performed on the binary image using a multi-adjacency connectivity criterion, such as the 8-adjacency connectivity criterion. If each pixel and its four adjacent pixels (top, bottom, left, right, and four corners) are all equal to 1, they belong to the same connected component. Finally, based on the number of pixels in each connected component, the area of ​​each component is calculated. The connected component with the largest area is taken as the hotspot region corresponding to the object being measured, i.e., the ROI.

[0085] Preferably, the spatial temperature gradient value is the average temperature difference between the edge region and the center region within the region of interest, expressed by the partial derivative formula as follows:

[0086] ;

[0087] in, The temperature gradient is represented by M, which represents the number of locations within the region of interest (ROI), such as the number of pixels within the ROI. T represents the temperature data, and x and y represent the horizontal and vertical coordinates of different locations within the ROI, such as different pixel column and pixel row positions, respectively.

[0088] Subsequently, a sliding time window of a preset number of seconds is maintained to determine the changes in the spatial temperature gradient; specifically, the spatial temperature gradient is obtained by performing an exponential moving average on the spatial temperature gradient within the time window. This can be expressed as a formula:

[0089] ;

[0090] Where t represents the current time, and (t-1) represents the time before the current time. This is the default value, usually set to 0.05. This represents the spatial temperature gradient of the region of interest at time t. The time baseline represents the spatial temperature gradient at time (t-1).

[0091] In some embodiments, the preset occlusion factor is set to 0.5. The determination of when occlusion occurs can be expressed by the formula:

[0092] ;

[0093] in, Represents the spatial temperature gradient. The time baseline representing the spatial temperature gradient.

[0094] By following the steps above, and considering the high temperatures involved in cooking, setting the preset occlusion factor to 0.5 can more accurately distinguish between actual occlusion and the situation caused by the increase in cooking temperature due to high-temperature steam, thereby further improving the accuracy of occlusion judgment.

[0095] When determining whether the spatial temperature gradient conforms to the exponential moving range based on temperature data, the spatial temperature gradient obtained by partial derivative calculation of temperature data within a sliding preset time window is exponentially averaged to obtain the time baseline. In actual scenarios, the thermal field distribution of the tested object is relatively uniform during normal cooking. Therefore, by comparing the results of spatial temperature gradient and time baseline, occlusion misjudgment under different cooking conditions can be avoided, thus improving the accuracy of occlusion judgment.

[0096] In some embodiments, step S110, which determines that the infrared sensor is blocked when the spatial temperature gradient disappears, further includes: determining the single-frame temperature difference and average temperature of the temperature data, as well as the average temperature peak in the average temperature.

[0097] If, within a preset time window, the temperature difference in a single frame of temperature data does not exceed a preset temperature difference threshold, the difference between the peak and average temperatures meets a preset threshold range, and the spatial temperature gradient does not exceed the spatial occlusion baseline value, then the infrared sensor is determined to be occluded.

[0098] Preferably, if the temperature difference of a single frame of temperature data is simultaneously satisfied... average temperature and If so, it can be determined that the infrared sensor is blocked.

[0099] in, This represents the average temperature within the region of interest at time t; Indicates the average temperature within the preset time window. The average peak temperature in the middle; Represents the spatial temperature gradient; The time baseline representing the spatial temperature gradient.

[0100] Through the above steps, when determining whether the object being measured by the infrared sensor is blocked, the judgment is made based on multiple conditions, such as the peak value of the temperature data, the instantaneous drop, and the change of the spatial temperature gradient, rather than relying on the change of the absolute temperature value. Among these, the change of the spatial temperature gradient can further improve the accuracy of distinguishing between real blockage and other cooking conditions, avoiding misjudgments and omissions.

[0101] In some of these embodiments, Figure 2 This is a flowchart of a method for auxiliary constraint prediction based on the temperature of obstructing objects, provided in an embodiment of this application. (Refer to...) Figure 2 The method for detecting the cooking category of smart appliances based on the dynamic fusion of infrared and audio also includes:

[0102] Step S210: If it is determined that the infrared sensor is blocked, obtain the temperature rise curve and material type of the blocking object; determine the heat conduction time constant of the blocking object based on the material type.

[0103] Step S220: Based on the heat exchange process between the object being tested and the shielding object, the estimated temperature of the object being tested is determined according to the temperature rise curve and heat conduction time constant of the shielding object.

[0104] Step S230: Based on the estimated temperature of the object being measured, constrain the predicted temperature value and its corresponding confidence level.

[0105] Taking a pot as the test object and the lid as the shield as an example, the higher the temperature inside the pot, the faster the lid is heated. The closer the lid temperature is to the pot's internal temperature, the slower the temperature rise rate (approaching thermal equilibrium). If the pot suddenly heats up drastically (such as by dry-heating), the lid's temperature rise rate will also suddenly accelerate. Therefore, the lid's temperature rise curve contains indirect information about the pot's internal thermal state.

[0106] In this embodiment, the estimated temperature of the object under test is indirectly determined by the temperature rise rate and material type of the pot lid, and then the predicted value of the autoregressive filter is constrained based on the estimated temperature.

[0107] Through the above steps, the obstructing object is usually not within the field of view of the infrared sensor before the obstruction occurs, making it impossible to obtain the temperature of the obstructing object in advance. Therefore, in this embodiment, by continuously tracking the temperature rise curve of the obstructing object's surface temperature during the rising process from the moment the obstruction occurs, and based on the dynamic characteristics of the temperature rise curve and the thermal conduction time constant corresponding to the material type, the temperature range of the measured object can be calculated from the obstruction device. This temperature range is then used as an auxiliary constraint for the predicted value of the autoregressive filter, further improving the accuracy of the predicted temperature value and its corresponding confidence level.

[0108] In some embodiments, the smart appliance cooking category detection method based on dynamic fusion of infrared and audio also includes:

[0109] Assuming the infrared sensor is not blocked, the predicted state vector and error covariance matrix of the previous sampling period are estimated based on the state transition equation.

[0110] Based on the predicted state vector and error covariance matrix of the previous sampling period, the prior state vector and prior error covariance matrix of the current sampling period are determined by using a preset autoregressive filter.

[0111] The filter gain is calculated based on the prior error covariance matrix and the measurement residual of the current sampling period; the measurement residual is determined by the measurement state vector and the prior state vector obtained from the observation equation.

[0112] The update state for the current sampling period is estimated based on the prior state vector and the filter gain.

[0113] In cases where the infrared sensor is not obstructed, such as when cooking begins or the object being measured is changed, the autoregressive filter needs to be initialized based on the first valid temperature measurement value obtained by the infrared sensor.

[0114] Subsequently, based on the predicted state vector and error covariance matrix of the previous sampling period, the prior state vector and prior error covariance matrix of the current sampling period are predicted, thus achieving prediction. Simultaneously, based on the prior error covariance matrix and the measurement residual of the current sampling period, the filter gain is calculated, and the state and error covariance matrices are updated according to this filter gain.

[0115] Through the above steps, when the infrared sensor is not blocked, a preset autoregressive filter is used to predict the prior state vector and prior error covariance matrix in the current sampling period based on the predicted state vector and error covariance matrix in the previous sampling period. This enables the prediction of the temperature value in the current sampling period, effectively compensating for the information gap caused by the sensor's physical response delay, so as to achieve rapid tracking response when the temperature changes abruptly. Furthermore, the optimal weighted fusion between the prior predicted value and the current observed value is performed to estimate the updated state, so that the updated state estimate has the smallest statistically significant error variance.

[0116] In some embodiments, when it is determined that the infrared sensor is blocked, the temperature data is recursively estimated using the temperature-based state transition equation and observation equation in a preset autoregressive filter to obtain the predicted temperature value and its corresponding confidence level, including:

[0117] Determine the moment when the infrared sensor is blocked; obtain the prior state estimate and prior error covariance matrix at the moment before the blocking occurs;

[0118] When the infrared sensor is blocked, the state estimate and error covariance matrix at the current time of blocking are determined based on the prior state estimate at the moment before the blocking occurs.

[0119] Based on the state estimate at the current occlusion moment, the temperature prediction at the current occlusion moment is estimated using the temperature-based state transition equation and observation equation in the preset autoregressive filter.

[0120] Based on the error covariance matrix at the current occlusion time, the uncertainty quantification value at the current occlusion time is predicted, and the confidence level corresponding to the temperature prediction value is determined based on the uncertainty quantification value.

[0121] In cases where the infrared sensor is blocked, effective measurement values ​​cannot be obtained. At this point, the filter stops updating steps, retains only the prediction steps, and combines the confidence level to obtain the final predicted temperature value.

[0122] Through the above steps, when it is determined that the infrared sensor is blocked, the preset autoregressive filter predicts the state estimate at the current blocking moment and the confidence level corresponding to the quantified value of the prediction uncertainty based only on the prior state estimate and the prior error covariance matrix. At this time, the filter stops updating and only retains the prediction step. Combining the predicted temperature value and its corresponding confidence level, it provides the best prediction of the current thermal state, realizing a reference temperature prediction value that conforms to the temperature law under the blocking state. Moreover, the output error covariance matrix quantitatively characterizes the confidence range of the prediction, improving the accuracy of the final output temperature prediction value.

[0123] Additionally, after determining that the occlusion has been removed, the first valid measurement value is obtained. A hard reset is performed on the autoregressive filter, directly overriding the predicted state with the measured temperature, resetting the uncertainty to the initial level and ensuring that the filter converges quickly to the true thermal state. Then, the normal prediction-update cycle is resumed.

[0124] In some embodiments, multimodal feature fusion is performed on the multidimensional feature vectors and their corresponding confidence scores in the audio data acquired by the audio acquisition device, and the temperature prediction values ​​and their corresponding confidence scores, to obtain a comprehensive feature vector, including:

[0125] Obtain the multidimensional feature vectors of data frames in the audio data and determine the total energy of the multidimensional feature vectors;

[0126] The sound confidence level of the data frame is determined based on the comparison between the total energy and the preset silence threshold.

[0127] Multimodal feature weighting and fusion are performed on the multidimensional feature vector and sound confidence corresponding to the audio data, as well as the temperature prediction value and its corresponding confidence, to obtain a comprehensive feature vector. The comprehensive feature vector is used as input to a preset state classifier to output the cooking category probability.

[0128] Here, the audio data is output as a single-channel time-domain signal after beamforming enhancement and wind noise suppression. For the preprocessed signal Perform frame segmentation, add a Hamming window to each frame, and obtain For each frame, multidimensional features are extracted to form a multidimensional feature vector F. For example, the frame length is 25ms and the frame shift is 10ms. Preferably, the multidimensional features are 10-dimensional, including 2-dimensional time-domain features, 3-dimensional frequency-domain features, 3-dimensional wavelet-domain features, and auditory perception features including the first two reflection parameters after 12th-order LPC analysis. The features of all current frames are then combined by calculating the mean and standard deviation of each dimension to form a 20-dimensional statistical feature vector. This refers to a multi-dimensional feature vector. Here, using 10-dimensional features can comprehensively cover the information dimensions of cooking sounds, significantly improving robustness in complex kitchen noise environments, and is computationally efficient, making it suitable for embedded deployment.

[0129] In addition, the multidimensional feature vectors here can also include nonlinear dynamic features (such as fractal dimension, entropy features), statistical distribution features (sound intensity kurtosis and skewness, crack sound detection features), and cepstral domain features, etc.

[0130] Subsequently, the total energy E of the multidimensional feature vector can be determined by summing the squares of the amplitudes of all sampling points within a frame in the time domain, or by summing the distribution of signal energy across the frequency band. After obtaining the total energy, its corresponding confidence level is determined. If the total energy exceeds a preset silence threshold, it indicates that the total energy E is too high and exceeds the clipping threshold, which may be impulse noise.

[0131] The method for fusing the multidimensional feature vectors and sound confidence scores corresponding to the audio data, as well as the temperature prediction values ​​and their corresponding confidence scores, can be as follows: the infrared channel and the sound channel can each independently complete the state classification, each outputting a posterior probability vector of the cooking state, and then the data can be fused at the decision layer; alternatively, a lightweight neural network with two input branches can be used, with one branch processing infrared features and the other processing acoustic features, and feature fusion can be performed in the middle layer of the network. No specific limitations are made here.

[0132] Through the above steps, multimodal features including sound and temperature are extracted in real time, and a state classifier is used to achieve multimodal fusion classification, thereby realizing the technical effect of hierarchical safety early warning. Furthermore, there is no need to distinguish whether occlusion factors affect the accuracy of temperature data, improving the accuracy of the fused comprehensive feature vector and further enhancing the output accuracy of the cooking category probability.

[0133] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 3 This is a hardware structure block diagram of the terminal for the intelligent appliance cooking category detection method based on dynamic fusion of infrared and audio provided in this embodiment. Figure 3 As shown, a terminal may include one or more ( Figure 3 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 3 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown are illustrated.

[0134] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the intelligent appliance cooking category detection method based on infrared and audio dynamic fusion in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0135] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0136] The present embodiment will be described and explained below through specific examples.

[0137] Taking a range hood as an example, a cookware as the test object, a pot lid as the obstruction, and a Kalman filter as the autoregressive filter. Figure 4 This is a flowchart of a method for tracking the thermal state of cooking provided in this specific embodiment. (Reference) Figure 4 The method includes:

[0138] Step S410: Determine the occlusion situation.

[0139] Specifically, Figure 5 This is a schematic diagram of the range hood structure provided in this specific embodiment. (Reference) Figure 5 The infrared sensor's temperature probe is deployed on the range hood and faces the cooking appliance, i.e., the pot, to collect infrared data in real time during the cooking process, i.e., the temperature data in the aforementioned embodiment; the pot is placed on the stovetop of the gas stove; and the microphone is set on the range hood panel to collect audio data during the cooking process.

[0140] First, by finding the largest connected hot spot, the bottom region of the pot is defined, thus obtaining the temperature set of pixels within the Region of Interest (ROI). Subsequently, infrared characteristic parameters are calculated, including the spatial temperature gradient, single-frame temperature difference, and average temperature in the aforementioned embodiments.

[0141] Among them, the average temperature within the ROI, i.e., the region of interest. Expressed as a formula:

[0142] ;

[0143] in, This represents the temperature value of the i-th pixel in the region of interest, and N represents the total number of temperature values ​​for the i-th pixel.

[0144] Space temperature gradient Expressed as a formula:

[0145] ;

[0146] The formula calculates the average temperature difference between the edge and center of the ROI, where T represents the temperature data, M represents the number of locations within the region of interest (ROI), such as the number of pixels within the ROI, and x and y represent the horizontal and vertical coordinates of different locations within the ROI, such as different pixel column and pixel row positions.

[0147] Subsequently, the occlusion status was monitored in real time using an occlusion detection algorithm, specifically as follows: a sliding window was maintained for 3 seconds, and the average temperature of the ROI was calculated. The peak value (i.e., the peak value within the past window time) maximum value ), instantaneous descent and spatial gradient change.

[0148] If the following conditions are met simultaneously: single-frame temperature difference , and If so, then occlusion is determined to have occurred.

[0149] in, This represents the average temperature within the region of interest at time t; Indicates the average temperature within the preset time window. The average peak temperature in the middle. The time baseline representing the spatial temperature gradient; This represents the spatial temperature gradient.

[0150] When the infrared temperature rises back to above 100°C within 2 seconds and If restored, the occlusion is determined to have been removed.

[0151] Furthermore, a mechanism to prevent false positives is set up, if the average temperature of the ROI... If the temperature is already below 80°C, such as an initial cold pot, a drop in temperature will not trigger shading. If the condition is met but the temperature recovers after less than 1 second, it may be due to a brief hand brush or smoke disturbance, in which case filtering will occur and it will not be considered stable shading.

[0152] To further improve the robustness of the detection, the following grading judgment logic can be added:

[0153] High-temperature zone shading (cookware temperature > 150°C): mainly relies on the condition of vanishing spatial temperature gradient. In this case, even if the temperature does not drop, gradient collapse will trigger shading.

[0154] Low-temperature zone shading (cookware temperature < 150°C): This depends on both the conditional temperature drop and the change in the spatial temperature gradient. In this case, the temperature drop is the primary characteristic, while the change in the spatial temperature gradient serves as a secondary confirmation.

[0155] If the conditional spatial temperature gradient change is met but the temperature does not decrease, the system enters a "suspected occlusion" state. In this state, the acoustic channel is still activated as an auxiliary judgment, but the infrared confidence level is temporarily maintained until the conditional spatial temperature gradient change is met for more than a set time (e.g., 2 seconds) before occlusion is confirmed.

[0156] Regardless of the pot lid temperature, once obstruction is detected, the "prediction-only mode" will be activated, and the subsequent fusion logic remains completely consistent. The accuracy of the predicted temperature is independent of the pot lid temperature in the early stages of obstruction (the prediction is based on the temperature rise trend of the pot bottom before obstruction), therefore the warning judgment is not affected by the pot lid temperature.

[0157] Step S420: Track the heat status of the cookware in real time.

[0158] Specifically, state transition equations and observation equations are constructed, and the prior thermal state of the cookware is tracked in real time based on the state transition equations and observation equations.

[0159] State transition equation It can be represented as:

[0160] ;

[0161] ;

[0162] Where k represents the sequence number corresponding to the sampling time. For the k-th sampling time (total sampling time) , The average temperature of the object being measured. The temperature rise rate at the k-th sampling time . This represents the state transition vector corresponding to the state transition equation at the k-th sampling time. Let represent the state transition vector corresponding to the state transition equation at the (k-1)th sampling time.

[0163] in, , Indicates the sampling interval time; The process noise is represented by a Gaussian distribution with a mean of 0 and a covariance of Q. A typical value for Q is... .

[0164] Process noise is used to quantify the uncertainty of a model. It reflects the extent to which a mathematical model cannot perfectly describe a real physical process, including unmodeled dynamic characteristics, external disturbances, and model simplification errors. The diagonal elements in Q correspond to the noise intensity of different state variables.

[0165] The observation equation can be expressed by the following formula:

[0166] ;

[0167] in, This represents the average temperature observation value obtained by the infrared sensor at the k-th sampling time. This represents the state transition vector corresponding to the state transition equation at the k-th sampling time. The observation matrix indicates that the infrared sensor only directly measures the temperature and does not measure the rate of temperature rise. For measuring noise, R is the measurement noise variance of the infrared sensor. A typical value is... .

[0168] Step S430: Initialize the filter.

[0169] Specifically, assuming the infrared sensor is not obstructed, the filter is initialized when cooking begins or when the cookware is changed, as expressed by the formula:

[0170] ;

[0171] ;

[0172] in, Let be the initial state vector. The initial measured temperature, i.e., the first valid temperature measurement value obtained from the infrared sensor, is used to initialize the temperature state of the filter. The initial error covariance matrix, This represents the initial error variance of the temperature state estimation, which is set to 100 here. This represents the initial error variance of the temperature rise rate state estimation, which is set to 10 here. In each subsequent sampling period, a prediction step and an update step are executed sequentially. The prediction step is expressed by the formula:

[0173] ;

[0174] ;

[0175] in, This is a priori state estimate (based solely on model predictions). Let be the prior error covariance matrix, representing the uncertainty of the predicted state. This represents the posterior state estimate, or predicted state vector, at the (k-1)th sampling time of the previous sampling period. Let represent the posterior error covariance matrix at the (k-1)th sampling time of the previous sampling period. Q represents the covariance, typically with a value of . , , This indicates the sampling interval.

[0176] The update step includes Kalman gain and updated state estimation. Specifically, the Kalman gain is calculated. and update state estimation They are expressed by the following formulas:

[0177] ;

[0178] ;

[0179] in, For Kalman gain, The prior error covariance matrix; To update the state estimate, The residual is used to measure the deviation between the measured and predicted values. For prior state estimation, This represents the average temperature observation value obtained by the infrared sensor at the k-th sampling time. Let R be the observation matrix, and R be the measurement noise variance of the infrared sensor. , This indicates the sampling interval.

[0180] Updated error covariance matrix Expressed as a formula:

[0181] ;

[0182] in, It is a 2×2 identity matrix. Let be the prior error covariance matrix. For Kalman gain, This is the observation matrix.

[0183] At this point, assuming the infrared sensor is not obstructed, the temperature prediction value output by the filter is... The corresponding confidence weight =1; where, This indicates an update to the state estimate.

[0184] In step S440, if the infrared sensor is blocked, the filter stops updating and only retains the prediction step.

[0185] Specifically, when the infrared sensor is blocked, effective measurement values ​​cannot be obtained. (The average temperature observation of the ROI acquired by the infrared sensor at time k), the filter stops updating the step and only retains the prediction step.

[0186] For each sampling period (from the moment the occlusion occurs) (Start counting) to make predictions and obtain the state estimate at the current occlusion time. And error covariance matrix :

[0187] ;

[0188] ;

[0189] in, This indicates the moment when occlusion occurs, and j represents the sampling period. , Indicates the sampling interval; Q represents the covariance, typically a value of ; The previous occlusion time is ( The state estimate of the posterior state estimate at the sampling time. The previous occlusion time is ( The posterior state estimation error covariance matrix at the sampling time.

[0190] At this point, with the infrared sensor blocked, the predicted temperature output by the filter is: And the quantified value of the prediction uncertainty is: ;in, This indicates an update to the state estimate. This represents the updated error covariance matrix.

[0191] Subsequently, this uncertainty quantification value is mapped to a confidence weight. :

[0192] ;

[0193] in This is a preset scale parameter, with a typical value set to 50; This represents the quantified value of the uncertainty in the forecast.

[0194] Step S450: After the occlusion is removed, perform a hard reset on the filter.

[0195] Specifically, after the occlusion is removed, the first valid measurement value is obtained. State estimates of the filter And error covariance matrix Perform a hard reset, expressed by the formula:

[0196] ;

[0197] ;

[0198] in, This represents the first valid measurement after the occlusion is removed, and R represents the measurement noise variance of the infrared sensor. Indicates the initial measured temperature Estimation of the error variance corresponding to the temperature rise rate.

[0199] Subsequently, the predicted state is directly overwritten with the measured temperature, and the uncertainty quantization value is reset to the initial level to ensure that the filter converges quickly to the true thermal state. Then, the normal prediction-update cycle resumes. By acquiring infrared temperature data in real time and using a Kalman filter to track the infrared temperature, the system achieves real-time output of infrared temperature data and confidence weights.

[0200] In one specific embodiment, a method for inferring the pot's internal temperature based on the temperature rise curve of the pot lid after obstruction is provided. The initial temperature of the pot lid is immediately recorded the instant the obstruction detection algorithm determines that the infrared sensor is obstructed. The final state of the pot bottom (including the average temperature of the ROI of the pot bottom in the last frame before occlusion). The rate of temperature rise at the bottom of the pot in the last frame before occlusion ).

[0201] Subsequently, the initial temperature difference was calculated. :

[0202] ;

[0203] in, This represents the initial temperature of the pot lid at time t0. t0 represents the average temperature of the ROI at the bottom of the pot in the last frame before occlusion; t0 represents the initial time.

[0204] This initial temperature difference reflects the current thermal state of the pot lid. If the pot lid was originally placed near the stove (at room temperature). It will be very large; if the pot lid has just been preheated, It will be smaller.

[0205] During the duration of the shading, the temperature of the pot lid was recorded every 0.1 seconds to construct a temperature rise curve for the pot lid. The temperature rise rate of the pot lid is constructed by the temperature rise curve of the pot lid. The rate of temperature rise of the pot lid Expressed as a formula:

[0206] ;

[0207] in, This represents the temperature rise curve of the pot lid at time t. This represents the temperature rise curve of the pot lid at time (t-1). Let represent the time difference between time t and time (t-1), where t represents time.

[0208] And calculate the acceleration of the temperature rise of the pot lid (the rate of change of the temperature rise rate). :

[0209] ;

[0210] in, This represents the rate of temperature rise of the pot lid at time t. This represents the rate of temperature rise of the pot lid at time (t-1). This represents the time difference between time t and time (t-1); t represents time.

[0211] Temperature rise curve of the pot lid Temperature rise rate and temperature rise acceleration Together, these factors constitute the dynamic characteristics of the pot lid's temperature rise curve.

[0212] Furthermore, the heat exchange between the lid and the internal thermal environment of the pot can be approximated as a first-order inertial process:

[0213] ;

[0214] in, This represents the temperature of the outer surface of the pot lid at time t. This represents the average temperature of the bottom ROI of the pot at time t, i.e., the temperature inside the pot. The time constant for heat conduction is represented by t, where t represents time.

[0215] Based on the formula for the first-order inertial process described above, the estimated temperature inside the pot at time t can be derived by reverse calculation. The expression, that is, the expression using the association model, is:

[0216] ;

[0217] That is, the estimated value of the pot temperature at time t obtained by reverse calculation. =Temperature of the outer surface of the pot lid at time t +Time constant of heat conduction Temperature rise rate of the pot lid at time t The faster the lid heats up, the higher the temperature inside the pot is compared to the current outer surface temperature of the lid; as the lid heats up to near 0°C, the two temperatures tend to equalize.

[0218] in, This represents the temperature of the outer surface of the pot lid at time t. This represents the estimated temperature inside the pot at time t obtained through reverse calculation. This represents the time constant of heat conduction, used to represent the thermal inertia of the pot lid. The larger the value, the slower the pot lid heats up. This represents the rate of temperature rise of the pot lid at time t.

[0219] The time constant for heat conduction can be determined by typical values ​​for the type of lid. For example, the time constant is 60-90s when the lid is made of glass; 30-50s when the lid is made of thin metal; and 80-120s when the lid is made of thick cast iron.

[0220] If the shading duration is long enough, the temperature of the pot lid will eventually stabilize; at this point, neglecting the slight temperature difference between the outer and inner surfaces of the pot lid, the temperature of the outer surface of the pot lid at time t is... Approximately equal to the temperature inside the pot Therefore, the optimal value of the time constant can be fitted by the least squares method using the temperature rise curve of the pot lid during the entire shading period.

[0221] The time constant can also be estimated using the initial temperature difference. At the instant of occlusion t0, the temperature inside the pot in the last frame before occlusion is known. and the initial temperature of the pot lid at time t0 In the first sampling period after shading, the initial temperature rise rate of the pot lid was measured. The time constant of heat conduction can be obtained. The estimated value is expressed as:

[0222] ;

[0223] in, This represents the initial temperature of the pot lid at time t0. This represents the temperature inside the pot at time t0, which is the last frame before the occlusion. This represents the initial temperature difference. express The initial temperature rise rate of the pot lid at any given time. t0 represents the time of the first sampling period after occlusion, and t0 represents the initial time.

[0224] In the initial stage of shading (such as the first 30 seconds), the temperature rise rate of the pot lid is still unstable, and the estimation of the time constant may have a large error. Therefore, the correlation model is not used for the time being, and the pure prediction of Kalman filtering is relied upon entirely.

[0225] When the duration of shading exceeds the preset time tmin, the temperature rise rate of the pot lid has stabilized, and the time constant is reasonably estimated, the correlation model is enabled.

[0226] Using the aforementioned correlation model as a soft constraint, the temperature prediction value of the Kalman filter is calculated. It then determines whether the predicted temperature value is within the estimation range of the correlation model. If it is within the range, the normal prediction uncertainty quantification value is maintained. If the growth exceeds the range, the process noise Q is temporarily increased to accelerate the growth of uncertainty and reduce the infrared weight.

[0227] The above correlation model can also be used as an intermittent pseudo-measurement, with the estimated temperature inside the pot from the correlation model every N cycles. As a measurement, the error variance of the correlation model is estimated. To measure noise Perform Kalman filter update; measurement noise of spurious measurements. The accuracy of the model should decrease over time, meaning that as the lid approaches thermal equilibrium, the model's accuracy improves. Reduce the error variance estimation of the correlation model. Let t be the rate of temperature rise of the pot lid. and the time constant of heat conduction The sum of the square of the product of the standard deviations of the estimated values ​​and the square of the standard deviation of the infrared thermometry noise.

[0228] Errors in estimating the time constant, approximation errors in the first-order model regarding actual heat conduction, or noise in the lid temperature measurement lead to variations in the estimated temperature inside the pot. There is an error involved; therefore, in order to improve the above-mentioned estimated temperature inside the pot... The accuracy of the estimation requires the confidence level of the model that approximates the heat exchange between the lid and the internal thermal environment of the pot as a first-order inertial process, and the uncertainty of the estimation result is quantified based on the model confidence level.

[0229] All data in this method for back-calculating the pot temperature based on the pot lid temperature rise curve after shading are collected from the moment of shading, without relying on any pot lid information before shading; the back-calculation formula, i.e. the correlation model, is simple to calculate and reduces resource consumption; and as shading continues and the pot lid approaches thermal equilibrium, the model accuracy automatically improves, effectively making up for the problem of Kalman filter pure prediction diverging over time.

[0230] Figure 6 This is a flowchart of a cooking safety early warning method based on dynamic fusion of infrared and audio data provided in this specific embodiment. By extracting multimodal features in real time and utilizing a lightweight MLP state classifier, the problem of multimodal fusion is solved, thereby achieving the technical effect of graded safety early warning. (Reference) Figure 6 The method includes:

[0231] Step S610: Extract the multidimensional feature vector of the audio data.

[0232] For details, please refer to Figure 5 The microphone array is deployed on the control panel of the smoke machine to collect audio data in real time. After beamforming enhancement and wind noise suppression, a single-channel time-domain signal is output. For the preprocessed signal The process involves frame segmentation, with a frame length of 25ms and a frame shift of 10ms. A Hamming window is added to each frame, resulting in... Ten-dimensional features are extracted from each frame to form a multi-dimensional feature vector of the audio data.

[0233] Furthermore, the multidimensional feature vectors of audio data include: time-domain features (2D), frequency-domain features (3D), wavelet-domain features (3D), and 2D auditory perception features.

[0234] In the time-domain features (2D), the zero-crossing rate ZCR is expressed as:

[0235] ;

[0236] Where N represents the total number of audio signal frames, N-1 represents the last frame in the audio signal, and n represents the current audio signal frame number. This indicates an audio signal frame with a Hamming window added. This represents the nth audio signal frame with a Hamming window applied. This represents the (n-1)th frame of the audio signal with a Hamming window applied. `sgn[]` represents the sign function, used to map continuous numerical values ​​to three discrete states, such as positive, negative, and zero. This represents the discrete state value obtained by mapping the nth audio signal frame. This represents the discrete state value obtained by mapping the (n-1)th frame of the audio signal.

[0237] The short-time energy logarithm is expressed as:

[0238] ;

[0239] Where E represents the total energy of the current audio data across different domains, N represents the total number of audio signal frames corresponding to the audio data, and n represents the current audio signal frame number. This represents the nth audio signal frame with a Hamming window applied.

[0240] In the frequency domain features (3D), the spectral centroid SC is represented as:

[0241] ;

[0242] in, The FFT amplitude spectrum at the k-th frequency point, Let k be the frequency of the k-th frequency point, where k represents the frequency point.

[0243] The spectrum bandwidth SB is represented as:

[0244] ;

[0245] in, The FFT amplitude spectrum at the k-th frequency point, Let be the frequency of the k-th frequency point, where k represents the frequency point and SC represents the centroid of the spectrum.

[0246] The spectral roll-off point is represented as: equal The minimum frequency R in.

[0247] in, The FFT amplitude spectrum at the k-th frequency point, Let R be the frequency of the k-th frequency point, where k represents the frequency point, R represents the frequency, and N / 2 represents the termination frequency of the summation.

[0248] In the wavelet domain feature (3D) analysis, the Daubechies-4 wavelet is used to perform a 3-level wavelet packet decomposition on each frame, resulting in 8 sub-band signals. The normalized energy of each sub-band is calculated, that is, the ratio of the energy of the three sub-bands 250-500Hz, 1000-2000Hz, and 3000-4000Hz to the total energy is calculated.

[0249] In auditory perception features, a 12th-order LPC (Linear Predictive Coding) analysis is performed on each frame to extract the first two reflection coefficients (or the first two dimensions of LSP parameters).

[0250] The mean and standard deviation of all the frame features mentioned above are calculated for each dimension, and then combined to form a 20-dimensional statistical feature vector. .

[0251] Step S620: Calculate the total energy of the current audio frame.

[0252] Specifically, the total energy E of the current audio data in different domains is calculated. If E is lower than the silence threshold E_silence, then the sound confidence level is... =0.1. If E is too high and exceeds the clipping threshold, it may be impulse noise, then the sound confidence level is... =0.5, which is within the normal range for sound confidence. =1.

[0253] Step S630: Weighted fusion of multimodal features.

[0254] Specifically, construct the comprehensive feature vector. .in, When the infrared sensor is unobstructed, , This represents the predicted temperature output of the filter when the infrared sensor is unobstructed; when the infrared sensor is obstructed by the pot lid, it represents... , This represents the predicted temperature output by the filter when the infrared sensor is blocked. This indicates the confidence level of the temperature prediction. This represents the predicted temperature value; Indicates the confidence level of the sound. This represents the multidimensional feature vector corresponding to the audio data.

[0255] In addition, multimodal feature weighted fusion can be performed in the following ways.

[0256] 1. Decision-level Fusion.

[0257] The infrared and audio channels independently perform state classification, each outputting a posterior probability vector of the cooking state, and then they are fused at the decision layer.

[0258] Specifically, the fusion result is obtained by weighting and averaging the two probability vectors. The weights are their respective confidence levels. and :

[0259] ;

[0260] in, and These are the probability vectors for the infrared channel and the audio channel, respectively. Indicates the confidence level of the temperature prediction; Indicates the confidence level of the sound.

[0261] The two channels are completely decoupled and can be developed and optimized independently; the system can still operate when either channel fails; however, it cannot capture complementary information between the two channels, and the classification accuracy is slightly lower than that of feature-level fusion.

[0262] 2. Model-level fusion, such as a two-stream neural network. This method employs a lightweight neural network with two input branches: one processing infrared features and the other processing acoustic features. Feature fusion is performed in the network's intermediate layers. The network can automatically learn the optimal fusion method and intermediate representations without requiring manual design of feature construction rules.

[0263] Specifically, the infrared branch converts 2D features into 8D features, the sound branch converts 20D features into 8D features, and the features from the infrared and sound branches are concatenated into 16D features. After passing through a fully connected layer, the fused features are output.

[0264] 3. Dynamic fusion based on attention mechanism: By introducing an attention mechanism, the network can automatically learn "whether to focus on infrared or sound" at each moment and make a judgment based on the externally set confidence level.

[0265] Specifically, an attention module is added to the input layer of the MLP to automatically calculate the attention weights for infrared and sound based on the current input features. These weights can be considered as "confidence levels learned by the network," avoiding dependence on external confidence functions. The network can automatically discover the optimal weight allocation strategy. Furthermore, the attention weights can be output for debugging and analysis, providing interpretability.

[0266] Step S640, thermal state classification.

[0267] Specifically, a lightweight MLP state classifier is used. The input layer receives the feature vector Z. The first hidden layer contains 16 neurons, each performing a weighted summation of the input, followed by a ReLU activation function to introduce non-linearity. The second hidden layer contains 8 neurons, also performing weighted summation and ReLU activation. The output layer contains 4 neurons, and the output is converted into probabilities using a Softmax function. . Category 1: Normal heating. Category 2: Risk of violent boiling / overflow. Corresponding category 3: burnt / dry burning. Corresponding category 4: Other / Silent.

[0268] Furthermore, to achieve stable decision-making, an exponential moving average is applied to the scorching / dry burning process to obtain the exponential moving average probability value at time t. :

[0269] ;

[0270] in, This represents the exponential moving average probability value of category 3 at time t. , This represents the probability of the current category 3 (burnt / dry burning) scenario. This represents the exponential moving average probability value of category 3 at time (t-1).

[0271] When applying the exponential moving average to the probability value at time t obtained from the exponential moving average of the burnt / dry burnt material. If the threshold (0.8) is exceeded and remains so for 0.5 seconds, an alarm is triggered. Similarly, the probability of "violent boiling / overflow risk" is smoothed to reduce the transient false alarm rate.

[0272] By extracting audio and infrared feature data in real time to form feature vectors, and using a lightweight MLP state classifier for multimodal fusion, the occurrence probability of each category is output, achieving the technical effect of graded early warning.

[0273] It should be noted that the steps shown in the above process or in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions.

[0274] This embodiment also provides an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0275] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0276] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0277] S1, when the spatial temperature gradient disappears, it is determined that the infrared sensor is blocked; the disappearance of the spatial temperature gradient is determined by the temperature data of the measured object collected by the infrared sensor.

[0278] S2, when it is determined that the infrared sensor is blocked, the temperature data is recursively estimated using the temperature-based state transition equation and observation equation in the preset autoregressive filter to obtain the temperature prediction value and its corresponding confidence level; the temperature prediction value and its corresponding confidence level are obtained by the temperature rise curve of the blocking object and the material type constraint of the blocking object.

[0279] S3. Multimodal feature fusion is performed on the multidimensional feature vectors and their corresponding confidence scores in the audio data collected by the audio acquisition device, and the temperature prediction values ​​and their corresponding confidence scores, to obtain a comprehensive feature vector.

[0280] S4 outputs the category probability of the cooking thermal state based on the comprehensive feature vector.

[0281] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0282] Furthermore, in conjunction with the intelligent appliance cooking category detection method based on dynamic fusion of infrared and audio provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the intelligent appliance cooking category detection methods based on dynamic fusion of infrared and audio provided in the above embodiments.

[0283] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0284] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0285] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0286] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for detecting the cooking category of intelligent appliances based on dynamic fusion of infrared and audio signals, characterized in that, The smart appliance is equipped with an infrared sensor and an audio acquisition device; the method includes: When the spatial temperature gradient disappears, it is determined that the infrared sensor is blocked; the disappearance of the spatial temperature gradient is determined by the temperature data of the object being measured collected by the infrared sensor. If the infrared sensor is determined to be blocked, the temperature data is recursively estimated using the temperature-based state transition equation and observation equation in the preset autoregressive filter to obtain the temperature prediction value and its corresponding confidence level; the temperature prediction value and its corresponding confidence level are obtained by the temperature rise curve of the blocking object and the material type of the blocking object. Multimodal feature fusion is performed on the multidimensional feature vector and its corresponding confidence level in the audio data collected by the audio acquisition device, and the temperature prediction value and its corresponding confidence level, to obtain a comprehensive feature vector; The output is a category probability of the cooking thermal state of the smart appliance based on the comprehensive feature vector.

2. The method for detecting the cooking category of intelligent appliances based on dynamic fusion of infrared and audio as described in claim 1, characterized in that, The method further includes: Based on the temperature-based state vectors of the previous and current time moments, and combined with the time difference between the previous and current time moments and a preset process noise, a temperature-based state transition equation is constructed; the process noise is used to quantify the uncertainty in the construction of the equation. Based on the average temperature at the current moment and the observation matrix of temperature measurement by the infrared sensor, and combined with the measurement noise of the infrared sensor, a temperature-based observation equation is constructed.

3. The method for detecting the cooking category of intelligent appliances based on dynamic fusion of infrared and audio as described in claim 1, characterized in that, The step of determining that the infrared sensor is blocked when the spatial temperature gradient disappears includes: The temperature data of different regions in the region of interest of the measured object are processed by partial derivative to determine the spatial temperature gradient. Within a preset time window, the spatial temperature gradient is calculated by an exponential moving average to determine the time baseline of the spatial temperature gradient. Based on the time baseline and the preset occlusion factor, the spatial occlusion baseline value is determined; If the spatial temperature gradient does not exceed the spatial occlusion baseline value, the spatial temperature gradient is determined to conform to the exponential movement range.

4. The method for detecting the cooking category of intelligent appliances based on dynamic fusion of infrared and audio as described in claim 3, characterized in that, The preset occlusion factor is set to 0.

5.

5. The method for detecting the cooking category of intelligent appliances based on dynamic fusion of infrared and audio as described in claim 3, characterized in that, The step of determining that the infrared sensor is blocked when the spatial temperature gradient disappears also includes: Determine the single-frame temperature difference and average temperature of the temperature data, as well as the average temperature peak value in the average temperature; If, within a preset time window, the temperature difference of a single frame of the temperature data does not exceed a preset temperature difference threshold, the difference between the peak and average temperatures meets a preset threshold range, and the spatial temperature gradient does not exceed the spatial occlusion baseline value, then the infrared sensor is determined to be occluded.

6. The method for detecting the cooking category of intelligent appliances based on dynamic fusion of infrared and audio as described in claim 1, characterized in that, The method further includes: If it is determined that the infrared sensor is blocked, the temperature rise curve and material type of the blocking object are obtained; The thermal conduction time constant of the shield is determined based on the material type; Based on the heat exchange process between the object under test and the shielding object, the estimated temperature of the object under test is determined according to the temperature rise curve of the shielding object and the heat conduction time constant. Based on the estimated temperature of the object being measured, the predicted temperature value and its corresponding confidence level are constrained.

7. The method for detecting the cooking category of intelligent appliances based on dynamic fusion of infrared and audio as described in claim 1, characterized in that, The method further includes: If it is determined that the infrared sensor is not blocked, the predicted state vector and error covariance matrix of the previous sampling period are estimated based on the state transition equation. Based on the predicted state vector and error covariance matrix in the previous sampling period, the prior state vector and prior error covariance matrix of the current sampling period are determined by using a preset autoregressive filter. The filter gain is calculated based on the prior error covariance matrix and the measurement residual of the current sampling period; the measurement residual is determined by the measurement state vector obtained from the observation equation and the prior state vector. The update state for the current sampling period is estimated based on the prior state vector and the filter gain.

8. The method for detecting the cooking category of intelligent appliances based on dynamic fusion of infrared and audio as described in claim 7, characterized in that, When it is determined that the infrared sensor is blocked, the temperature data is recursively estimated using the temperature-based state transition equation and observation equation in a preset autoregressive filter to obtain the predicted temperature value and its corresponding confidence level, including: Determine the moment when the infrared sensor is blocked; obtain the prior state estimate and the prior error covariance matrix at the moment before the blocking occurs; In the case where the infrared sensor is blocked, the state estimate and error covariance matrix at the current time of blocking are determined based on the prior state estimate at the moment before the blocking occurs. Based on the state estimate at the current occlusion moment, the temperature prediction at the current occlusion moment is estimated using the temperature-based state transition equation and observation equation in the preset autoregressive filter. Based on the error covariance matrix at the current occlusion time, the uncertainty quantization value at the current occlusion time is predicted, and the confidence level corresponding to the temperature prediction value is determined based on the uncertainty quantization value.

9. The method for detecting the cooking category of intelligent appliances based on dynamic fusion of infrared and audio as described in claim 1, characterized in that, The process of fusing multi-modal features from the multi-dimensional feature vectors and their corresponding confidence scores in the audio data acquired by the audio acquisition device, and the predicted temperature value and its corresponding confidence score, to obtain a comprehensive feature vector includes: Obtain the multidimensional feature vectors of the data frames in the audio data, and determine the total energy of the multidimensional feature vectors; The sound confidence level of the data frame is determined based on the comparison result between the total energy and the preset silence threshold. The multi-dimensional feature vector and sound confidence corresponding to the audio data, as well as the temperature prediction value and its corresponding confidence, are subjected to multi-modal feature weighting fusion to obtain a comprehensive feature vector; the comprehensive feature vector is used to input to a preset state classifier to output the cooking category probability.

10. A smart appliance, characterized in that, The cooking category detection method for smart appliances based on dynamic fusion of infrared and audio, as described in any one of claims 1 to 9, is used to output the category probability of the current thermal state of the smart appliance, wherein the smart appliance is one of a range hood and a combined range hood and stove.

Images