Intelligent kitchen cooking detection system and method based on multi-source data

Abstract

The invention relates to the technical field of intelligent kitchens, and particularly discloses an intelligent kitchen cooking detection system and method based on multi-source data, and the system employs an artificial intelligence technology based on deep learning to carry out the signal analysis of a smell signal and a smoke signal collected by an electronic nose and a smoke sensor in a kitchen. Time-frequency characteristics of the smell signal and the smoke signal are captured for representation, and then the kitchen oil smoke degree is represented based on interactive fusion characteristics between the smell signal and the smoke signal, so that intelligent control of the kitchen range hood is realized. Therefore, the air speed and the running state of the range hood can be automatically adjusted according to the lampblack degree generated in the cooking process, freshness of kitchen air is ensured, and potential threats of lampblack to health of family members are avoided.

Description

Technical Field

[0001] The present application relates to the field of smart kitchen technology, and more specifically, to a smart kitchen cooking detection system and method based on multi-source data. Background Art

[0002] With the rapid development of modern science and technology, the Internet of Things and artificial intelligence technologies have deeply penetrated into every aspect of our lives, especially in the field of smart home. As an important part of it, smart kitchen is gradually becoming a hot spot for home intelligent transformation. In modern families, the control of kitchen fume during cooking is an important issue. The fume not only affects the cleanliness of the kitchen environment, but also may affect the health of family members, such as increasing the risk of respiratory diseases.

[0003] Traditional range hood control usually relies on manual operation. Users manually adjust the working state of the range hood through switches or buttons on the range hood, such as starting, stopping, adjusting the wind speed, etc. However, this manual control method often cannot adapt to the different levels of oil smoke generated by various cooking methods and the changes in oil smoke during the cooking process in real time and reasonably, resulting in low efficiency in oil fume treatment. Therefore, a smart kitchen cooking detection solution is expected. Summary of the invention

[0004] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides an intelligent kitchen cooking detection system and method based on multi-source data, which uses artificial intelligence technology based on deep learning to perform signal analysis on odor signals and smoke signals collected by electronic noses and smoke sensors in the kitchen, so as to capture the time-frequency feature representation of odor signals and smoke signals, and then characterize the degree of kitchen fume based on the interactive fusion characteristics between the two, thereby realizing intelligent control of kitchen range hoods. In this way, the wind speed and operating state of the range hood can be automatically adjusted according to the degree of fume generated during the cooking process, ensuring the freshness of the kitchen air and avoiding the potential threat of fume to the health of family members.

[0005] Accordingly, according to one aspect of the present application, a smart kitchen cooking detection system based on multi-source data is provided, which includes:

[0006] A kitchen cooking detection module, used to obtain kitchen cooking odor signals collected by the electronic nose and kitchen smoke signals collected by the smoke sensor;

[0007] A signal time-frequency analysis module, used for performing time-frequency analysis on the kitchen cooking smell signal and the kitchen smoke signal to obtain a kitchen cooking smell time-frequency feature graph and a kitchen smoke time-frequency feature graph;

[0008] A multimodal representation module for the degree of cooking fume, used for performing feature correlation fusion enhancement on the time-frequency feature map of the kitchen cooking smell and the time-frequency feature map of the kitchen smoke to obtain a kitchen fume degree representation feature map;

[0009] The range hood control instruction generating module is used to generate a range hood control instruction based on the kitchen fume degree characterization characteristic diagram, wherein the range hood control instruction indicates that the range hood should be turned on, the wind speed should be increased, the wind speed should be decreased, or it should be turned off.

[0010] In the above-mentioned intelligent kitchen cooking detection system based on multi-source data, the signal time-frequency analysis module includes: a wavelet transformation unit, which is used to perform wavelet transformation on the kitchen cooking odor signal and the kitchen smoke signal respectively to obtain a kitchen cooking odor signal time-frequency graph and a kitchen smoke signal time-frequency graph; a time-frequency feature extraction unit, which is used to pass the kitchen cooking odor signal time-frequency graph and the kitchen smoke signal time-frequency graph through a time-frequency feature extractor based on a convolutional neural network model to obtain the kitchen cooking odor time-frequency feature graph and the kitchen smoke time-frequency feature graph.

[0011] In the above-mentioned intelligent kitchen cooking detection system based on multi-source data, the multimodal characterization module of oil fume degree is used to: arrange the kitchen cooking odor time-frequency feature map and the kitchen smoke time-frequency feature map along the channel dimension into a multimodal kitchen cooking state time-frequency feature map, and then obtain the kitchen oil fume degree characterization feature map through a feature bidirectional enhancement module including a channel attention layer and a spatial attention layer.

[0012] In the above-mentioned intelligent kitchen cooking detection system based on multi-source data, the multimodal representation module of the degree of oil smoke is used to: pass the multimodal kitchen cooking state time-frequency feature map through the spatial attention layer of the feature bidirectional enhancement module to obtain a local spatial enhanced kitchen cooking state time-frequency feature map; pass the multimodal kitchen cooking state time-frequency feature map through the channel attention layer of the feature bidirectional enhancement module to obtain a global channel enhanced kitchen cooking state time-frequency feature map; and perform position point-by-position analysis on the local spatial enhanced kitchen cooking state time-frequency feature map and the global channel enhanced kitchen cooking state time-frequency feature map to obtain the kitchen oil smoke degree representation feature map.

[0013] In the above-mentioned intelligent kitchen cooking detection system based on multi-source data, the range hood control instruction generation module includes: a feature sparsification unit, which is used to perform feature sparsification processing on the kitchen oil fume degree characterization feature map to obtain a kitchen oil fume degree characterization feature vector; a feature optimization unit, which is used to perform feature optimization on the kitchen oil fume degree characterization feature vector to obtain an optimized kitchen oil fume degree characterization feature vector; and a classification decision unit, which is used to input the optimized kitchen oil fume degree characterization feature vector into a range hood controller based on a classifier to obtain the range hood control instruction.

[0014] In the above-mentioned intelligent kitchen cooking detection system based on multi-source data, the feature sparseness unit is used to: use the principal component analysis method to process the kitchen fume degree characterization feature map to obtain the kitchen fume degree characterization feature vector.

[0015] In the above-mentioned intelligent kitchen cooking detection system based on multi-source data, the feature optimization unit is used to: input the kitchen fume degree characterization feature vector into the viewpoint metric correlation network based on backward learning sensitive preference to obtain the optimized kitchen fume degree characterization feature vector.

[0016] In the above-mentioned intelligent kitchen cooking detection system based on multi-source data, the feature optimization unit includes: a probabilistic activation subunit, which is used to perform probabilistic activation on the kitchen fume degree characterization feature vector based on the Sigmoid function to obtain a probabilistic kitchen fume degree characterization feature vector; a pre-classification subunit, which is used to input the kitchen fume degree characterization feature vector into a classifier to obtain an initial class label probability value; an error measurement subunit, which is used to calculate the absolute value of the difference between the feature value of each position in the probabilistic kitchen fume degree characterization feature vector and the initial class label probability value to obtain a backward machine learning fine-grained error measurement vector; an attention optimization subunit, which is used to calculate the absolute value of the difference between the feature value of each position in the probabilistic kitchen fume degree characterization feature vector and the initial class label probability value to obtain a backward machine learning fine-grained error measurement vector; and a pre-classification subunit, which is used to calculate the absolute value of the difference between the feature value of each position in the probabilistic kitchen fume degree characterization feature vector and the initial class label probability value to obtain a backward machine learning fine-grained error measurement vector. The quantity is used as a key vector, the kitchen fume degree characterization feature vector is used as a query vector and a value vector, and the key vector, the query vector and the value vector are input into a converter structure to obtain a backward learning type attention optimized kitchen fume degree characterization feature vector; an optimization scaling factor calculation subunit is used to calculate the ratio between the sum of the backward machine learning fine-grained error metric vector and the length of the backward machine learning fine-grained error metric vector, and calculate the natural exponential function value with the ratio as the power as the optimization scaling factor; a feature scaling subunit is used to calculate the product between the optimization scaling factor and the backward learning type attention optimized kitchen fume degree characterization feature vector to obtain the optimized kitchen fume degree characterization feature vector.

[0017] According to another aspect of the present application, a smart kitchen cooking detection method based on multi-source data is provided, which includes:

[0018] Acquire kitchen cooking odor signals collected by the electronic nose and kitchen smoke signals collected by the smoke sensor;

[0019] Performing time-frequency analysis on the kitchen cooking smell signal and the kitchen smoke signal to obtain a kitchen cooking smell time-frequency feature graph and a kitchen smoke time-frequency feature graph;

[0020] Performing feature correlation fusion enhancement on the kitchen cooking odor time-frequency feature map and the kitchen smoke time-frequency feature map to obtain a kitchen fume degree representation feature map;

[0021] Based on the kitchen fume degree characterization characteristic diagram, a range hood control instruction is generated, wherein the range hood control instruction indicates that the range hood should be turned on, the wind speed should be increased, the wind speed should be decreased, or it should be turned off.

[0022] Compared with the prior art, the intelligent kitchen cooking detection system and method based on multi-source data provided by the present application uses artificial intelligence technology based on deep learning to analyze the odor signals and smoke signals collected by the electronic nose and smoke sensor in the kitchen, so as to capture the time-frequency feature representation of the odor signal and the smoke signal, and then characterize the kitchen fume level based on the interactive fusion characteristics between the two, thereby realizing intelligent control of the kitchen range hood. In this way, the wind speed and operating state of the range hood can be automatically adjusted according to the degree of fume generated during the cooking process, ensuring the freshness of the kitchen air and avoiding the potential threat of fume to the health of family members. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0024] Figure 1 4 is a block diagram of an intelligent kitchen cooking detection system based on multi-source data according to an embodiment of the present application.

[0025] Figure 2 Schematic diagram of the architecture of an intelligent kitchen cooking detection system based on multi-source data according to an embodiment of the present application.

[0026] Figure 3 4 is a block diagram of a signal time-frequency analysis module in a smart kitchen cooking detection system based on multi-source data according to an embodiment of the present application.

[0027] Figure 4 This is a block diagram of a range hood control instruction generation module in a smart kitchen cooking detection system based on multi-source data according to an embodiment of the present application.

[0028] Figure 5 It is a block diagram of a feature optimization unit in a smart kitchen cooking detection system based on multi-source data according to an embodiment of the present application.

[0029] Figure 6 The present invention is a flowchart of a smart kitchen cooking detection method based on multi-source data according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] Below, the embodiments of the present application will be described in more detail in conjunction with the accompanying drawings, and the above and other purposes, features and advantages of the present application will become more apparent. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein.

[0031] Figure 1 4 is a block diagram of an intelligent kitchen cooking detection system based on multi-source data according to an embodiment of the present application. Figure 2 FIG. 1 is a schematic diagram of the architecture of an intelligent kitchen cooking detection system based on multi-source data according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to an embodiment of the present application, the intelligent kitchen cooking detection system 100 based on multi-source data includes: a kitchen cooking detection module 110, which is used to obtain a kitchen cooking odor signal collected by an electronic nose and a kitchen smoke signal collected by a smoke sensor; a signal time-frequency analysis module 120, which is used to perform time-frequency analysis on the kitchen cooking odor signal and the kitchen smoke signal to obtain a kitchen cooking odor time-frequency feature map and a kitchen smoke time-frequency feature map; an oil fume degree multimodal characterization module 130, which is used to perform feature correlation fusion enhancement on the kitchen cooking odor time-frequency feature map and the kitchen smoke time-frequency feature map to obtain a kitchen oil fume degree characterization feature map; a range hood control instruction generation module 140, which is used to generate a range hood control instruction based on the kitchen oil fume degree characterization feature map, wherein the range hood control instruction indicates that it should be turned on, the wind speed should be increased, the wind speed should be reduced, or it should be turned off.

[0032] As mentioned in the above background technology, traditional range hood control usually relies on manual operation, and the user manually adjusts the working state of the range hood through the switch or button on the range hood, such as starting, stopping, adjusting the wind speed, etc. However, this manual control method often cannot adapt to the different degrees of oil smoke generated by various cooking methods and the changes in oil smoke during the cooking process in real time and reasonably, resulting in low efficiency in oil fume treatment. In response to the above technical problems, the technical concept of the present application is to use artificial intelligence technology based on deep learning to perform signal analysis on the odor signals and smoke signals collected by the electronic nose and smoke sensor in the kitchen to capture the time-frequency feature representation of the odor signal and the smoke signal, and then characterize the kitchen oil fume degree based on the interactive fusion characteristics between the two, thereby realizing intelligent control of the kitchen range hood. In this way, the wind speed and operating state of the range hood can be automatically adjusted according to the degree of oil fume generated during the cooking process, ensuring the freshness of the kitchen air and avoiding the potential threat of oil fume to the health of family members.

[0033] In the above-mentioned intelligent kitchen cooking detection system 100 based on multi-source data, the kitchen cooking detection module 110 is used to obtain the kitchen cooking odor signal collected by the electronic nose and the kitchen smoke signal collected by the smoke sensor. It should be understood that various odors and smoke are usually generated during the cooking process, and different cooking methods and ingredients will produce different odor and smoke characteristics. For example, high-oil smoke cooking methods such as frying and deep-frying will quickly produce a large amount of smoke, while low-oil smoke cooking methods such as stewing will have less smoke but the odor may be stronger. By simultaneously collecting odor and smoke signals, information such as the state of cooking activities, the type of food, the cooking method, and the degree of oil smoke generation can be obtained, so as to more comprehensively understand the real-time environmental conditions in the kitchen, provide a rich data basis for subsequent oil smoke level assessment, and thus achieve more refined control of the range hood wind speed. For example, when the oil smoke just starts to be generated, the system can respond quickly and adjust the range hood to a suitable wind speed; when the oil smoke is reduced, the wind speed can be reduced in time to save energy.

[0034] In the above-mentioned intelligent kitchen cooking detection system 100 based on multi-source data, the signal time-frequency analysis module 120 is used to perform time-frequency analysis on the kitchen cooking odor signal and the kitchen smoke signal to obtain a kitchen cooking odor time-frequency feature graph and a kitchen smoke time-frequency feature graph. Figure 3 FIG. 1 is a block diagram of a signal time-frequency analysis module in a smart kitchen cooking detection system based on multi-source data according to an embodiment of the present application. Figure 3 As shown, the signal time-frequency analysis module 120 includes: a wavelet transform unit 121, which is used to perform wavelet transform on the kitchen cooking odor signal and the kitchen smoke signal respectively to obtain a kitchen cooking odor signal time-frequency diagram and a kitchen smoke signal time-frequency diagram; a time-frequency feature extraction unit 122, which is used to pass the kitchen cooking odor signal time-frequency diagram and the kitchen smoke signal time-frequency diagram through a time-frequency feature extractor based on a convolutional neural network model to obtain the kitchen cooking odor time-frequency feature diagram and the kitchen smoke time-frequency feature diagram.

[0035] Specifically, the wavelet transform unit 121 is used to perform wavelet transform on the kitchen cooking odor signal and the kitchen smoke signal respectively to obtain a time-frequency diagram of the kitchen cooking odor signal and a time-frequency diagram of the kitchen smoke signal. That is, in order to effectively capture the dynamic changes of the kitchen cooking odor signal and the kitchen smoke signal in the time domain and the frequency domain, reveal their time-frequency characteristics, so as to understand the generation and diffusion process of oil smoke, in the technical solution of the present application, the kitchen cooking odor signal and the kitchen smoke signal are processed respectively by a wavelet transform method. Those of ordinary skill in the art should know that wavelet transform is a multi-resolution analysis tool that can perform localized analysis on non-stationary signals, can reflect both the frequency characteristics of the signal and the time characteristics of the signal, and is particularly suitable for processing signals such as cooking odor and smoke signals that change dynamically with the cooking process. Here, the kitchen cooking odor signal and the kitchen smoke signal are converted from the original time domain representation to the time-frequency domain representation by the wavelet transform method, revealing the local characteristics of the signal in the time domain and frequency domain, and intuitively showing the frequency components of the signal changing over time, such as the instantaneous changes in the generation of oil smoke or the gradual accumulation of odor, etc., so as to facilitate the subsequent feature extraction and accurate characterization of the degree of oil smoke. In addition, in practical applications, the collected kitchen cooking odor signal and the kitchen smoke signal may contain noise and interference, and the wavelet transform can suppress the noise by selecting the appropriate wavelet basis and decomposition layer number, and extract the feature information useful for the evaluation of the degree of oil smoke, further improving the accuracy of oil smoke detection.

[0036] Specifically, the time-frequency feature extraction unit 122 is used to obtain the kitchen cooking odor signal time-frequency graph and the kitchen smoke signal time-frequency graph through a time-frequency feature extractor based on a convolutional neural network model. It should be understood that the time-frequency features in the kitchen cooking odor signal time-frequency graph and the kitchen smoke signal time-frequency graph are closely related to the evaluation of the degree of oil smoke. For example, certain specific frequency components or time variation patterns may correspond to specific oil smoke generation states. In order to effectively extract features related to the degree of oil smoke from the time-frequency graph, in the technical solution of the present application, a time-frequency feature extractor based on a convolutional neural network (CNN) is used to process the kitchen cooking odor signal time-frequency graph and the kitchen smoke signal time-frequency graph respectively. The CNN model shows strong capabilities in image processing and feature extraction, and can automatically learn and extract high-level abstract features in the input image without manually designing complex feature engineering. Here, the time-frequency feature extractor based on the convolutional neural network model includes structures such as convolution layers, pooling layers, and activation layers. Through multi-layer nonlinear transformations, the features in the time-frequency graph are abstracted and screened layer by layer, and the local correlation features in the kitchen cooking odor signal time-frequency graph and the kitchen smoke signal time-frequency graph can be mined, such as the peak value of a specific frequency or the temporal change trend, thereby obtaining the key feature representation of the degree of oil smoke, providing strong support for subsequent range hood control.

[0037] In the above-mentioned intelligent kitchen cooking detection system 100 based on multi-source data, the oil fume degree multimodal characterization module 130 is used to perform feature association fusion enhancement on the kitchen cooking odor time-frequency feature map and the kitchen smoke time-frequency feature map to obtain a kitchen oil fume degree characterization feature map. In a specific example of the present application, the method of performing feature association fusion enhancement on the kitchen cooking odor time-frequency feature map and the kitchen smoke time-frequency feature map is to arrange the kitchen cooking odor time-frequency feature map and the kitchen smoke time-frequency feature map along the channel dimension into a multimodal kitchen cooking state time-frequency feature map, and then pass through a feature bidirectional enhancement module including a channel attention layer and a spatial attention layer to obtain the kitchen oil fume degree characterization feature map. That is, in order to effectively integrate the feature information in the time-frequency feature map of the kitchen cooking smell and the time-frequency feature map of the kitchen smoke, strengthen the correlation between different modal features, highlight key features, and suppress noise interference, first, the time-frequency feature map of the kitchen cooking smell and the time-frequency feature map of the kitchen smoke are stacked in the channel dimension to form a multi-modal feature joint representation, and then the feature is enhanced by a feature bidirectional enhancement module including a channel attention layer and a spatial attention layer. Among them, the channel attention layer can dynamically adjust the weight of each channel according to the importance of different modal features, enhance the relevance and consistency of the features, and the spatial attention layer can capture the spatial distribution characteristics of the time-frequency features, focus on key areas, and suppress irrelevant or noise features. This feature association fusion enhancement mechanism can adaptively explore the potential connections between different modal data, improve the expressiveness and robustness of the features, and thus more accurately characterize the degree of kitchen fume.

[0038] Specifically, the multimodal representation module 130 of the degree of oil smoke is used to: pass the multimodal kitchen cooking state time-frequency feature map through the spatial attention layer of the feature bidirectional enhancement module to obtain a local spatial enhanced kitchen cooking state time-frequency feature map; pass the multimodal kitchen cooking state time-frequency feature map through the channel attention layer of the feature bidirectional enhancement module to obtain a global channel enhanced kitchen cooking state time-frequency feature map; and perform position point-by-position analysis on the local spatial enhanced kitchen cooking state time-frequency feature map and the global channel enhanced kitchen cooking state time-frequency feature map to obtain the kitchen oil smoke degree representation feature map.

[0039] In the above-mentioned intelligent kitchen cooking detection system 100 based on multi-source data, the range hood control instruction generation module 140 is used to generate a range hood control instruction based on the kitchen oil fume degree characterization characteristic diagram, wherein the range hood control instruction indicates that it should be turned on, the wind speed should be increased, the wind speed should be reduced, or it should be turned off. Figure 4 FIG. 1 is a block diagram of a range hood control instruction generation module in a smart kitchen cooking detection system based on multi-source data according to an embodiment of the present application. Figure 4As shown, the range hood control instruction generation module 140 includes: a feature thinning unit 141, used for performing feature thinning processing on the kitchen oil fume degree characterization feature map to obtain a kitchen oil fume degree characterization feature vector; a feature optimization unit 142, used for performing feature optimization on the kitchen oil fume degree characterization feature vector to obtain an optimized kitchen oil fume degree characterization feature vector; a classification decision unit 143, used for inputting the optimized kitchen oil fume degree characterization feature vector into a range hood controller based on a classifier to obtain the range hood control instruction.

[0040] Specifically, the feature thinning unit 141 is used to perform feature thinning processing on the kitchen fume degree characterization feature map to obtain a kitchen fume degree characterization feature vector. In a specific example of the present application, the processing method for performing feature thinning processing on the kitchen fume degree characterization feature map is to use the principal component analysis method to process the kitchen fume degree characterization feature map to obtain the kitchen fume degree characterization feature vector. It should be understood that, considering that the feature dimension of the kitchen fume degree characterization feature map is high, in order to further reduce the computational complexity, remove redundant information, and retain key features, in the technical solution of the present application, the principal component analysis (PCA) method is further used to perform feature thinning processing on the kitchen fume degree characterization feature map. Specifically, PCA can map the kitchen fume degree characterization feature map from a high-dimensional feature space to a low-dimensional space by performing a linear transformation on the kitchen fume degree characterization feature map, so that most of the feature information is concentrated on a few principal components, thereby obtaining a kitchen fume degree characterization feature vector. In this way, not only can the computational burden of subsequent processing be reduced, but it can also help to improve the distinguishability of the features, so that the system can judge the fume degree more accurately.

[0041] Specifically, the feature optimization unit 142 is used to perform feature optimization on the kitchen fume degree characterization feature vector to obtain an optimized kitchen fume degree characterization feature vector. In particular, in the technical solution of the present application, it is considered that the signals collected by the electronic nose and the smoke sensor may be affected by factors such as the kitchen environment, equipment accuracy and installation location, resulting in deviations between the collected signals and the actual fume degree. Wavelet transform is used to extract the time-frequency features of the signal, but the characteristics of the odor and smoke signals generated by different cooking processes may vary greatly, and the wavelet transform may not be able to fully adapt to all types of signals. The CNN-based time-frequency feature extractor may not be able to capture all features related to the kitchen fume degree. Some important time-frequency features may not be learned by the model. If the number of samples of some range hood control instructions in the data set is much larger than that of other instructions, the model may tend to recognize these common instructions, but the recognition of uncommon instructions is inaccurate, that is, the kitchen fume degree characterization feature vector has a class harmony offset relative to the predetermined class label of the classifier. In order to reduce the category harmony deviation of the kitchen fume degree characterization feature vector, in a specific example of the present application, the kitchen fume degree characterization feature vector is input into a viewpoint metric correlation network based on backward learning sensitive preference to obtain the optimized kitchen fume degree characterization feature vector.

[0042] Figure 5 FIG. 1 is a block diagram of a feature optimization unit in a smart kitchen cooking detection system based on multi-source data according to an embodiment of the present application. Figure 5As shown, the feature optimization unit 142 includes: a probabilistic activation subunit 1421, which is used to perform probabilistic activation on the kitchen fume degree characterization feature vector based on the Sigmoid function to obtain a probabilistic kitchen fume degree characterization feature vector; a pre-classification subunit 1422, which is used to input the kitchen fume degree characterization feature vector into a classifier to obtain an initial class label probability value; an error measurement subunit 1423, which is used to calculate the absolute value of the difference between the feature value of each position in the probabilistic kitchen fume degree characterization feature vector and the initial class label probability value to obtain a backward machine learning fine-grained error measurement vector; an attention optimization subunit 1424, which is used to use the backward machine learning fine-grained error measurement vector as a key vector, the kitchen fume degree characterization feature vector as a query vector and a value vector, the key vector, the query vector and the value vector are input into a converter structure to obtain a backward learning type attention optimized kitchen fume degree characterization feature vector; an optimization scaling factor calculation subunit 1425 is used to calculate the ratio between the sum of the backward machine learning fine-grained error metric vector and the length of the backward machine learning fine-grained error metric vector, and calculate the natural exponential function value with the ratio as the power as the optimization scaling factor; a feature scaling subunit 1426 is used to calculate the product between the optimization scaling factor and the backward learning type attention optimized kitchen fume degree characterization feature vector to obtain the optimized kitchen fume degree characterization feature vector.

[0043] That is, the feature distribution optimization unit 142 is used to process the kitchen fume degree characterization feature vector using the following feature distribution optimization formula to obtain the optimized kitchen fume degree characterization feature vector, wherein the feature distribution optimization formula is:

[0044]

[0045] Among them, V c represents the probabilistic kitchen fume degree characterization feature vector obtained after the kitchen fume degree characterization feature vector is probabilistically activated, P represents the initial class label probability value, Indicates subtraction by position, V e represents the fine-grained error metric vector for backward machine learning, v i represents the eigenvalue of the ith position of the backward machine learning fine-grained error metric vector, L represents the length of the backward machine learning fine-grained error metric vector, (·) T represents the transpose of a vector, represents vector multiplication, Softmax represents the normalized exponential function, ⊙ represents point multiplication by position, V' c Represents the optimized kitchen fume degree characterization feature vector.

[0046] Based on this, in the technical solution of the present application, the kitchen fume degree characterization feature vector is input into the viewpoint metric correlation network based on backward learning sensitive preference, which first implements the probabilistic activation based on the Sigmoid function on the kitchen fume degree characterization feature vector to obtain the probabilistic kitchen fume degree characterization feature vector, and inputs the kitchen fume degree characterization feature vector into the classifier to obtain the initial class label probability value. Then, the error between the probabilistic kitchen fume degree characterization feature vector and the initial class label probability value is calculated through backward error analysis to generate a fine-grained error metric vector, which quantifies the viewpoint metric contribution of each feature position to the classification error.

[0047] Then, the backward machine learning fine-grained error metric vector is used as a key vector, the kitchen fume degree characterization feature vector is used as a query vector and a value vector, and the key vector, the query vector and the value vector are input into a transformer structure to perform error metric-based attention-based backward machine learning on the kitchen fume degree characterization feature vector through a Transformer-like architecture. In this way, the kitchen fume degree characterization feature vector is subjected to a backward interpretability analysis so that the obtained backward learning-like attention-optimized kitchen fume degree characterization feature vector has a backward learning sensitive preference relative to the predetermined class label of the classifier. And further, the ratio of the sum of the error metric vector and the vector length is calculated, and the optimized scaling factor obtained by calculating the natural exponential function value with this ratio as an exponent is used to perform backward scaling optimization on the backward learning-like attention-optimized kitchen fume degree characterization feature vector. In this way, the sensitivity preference and robustness of the kitchen fume degree characterization feature vector relative to the predetermined class label of the classifier can be effectively improved, thereby improving the accuracy of the classification result.

[0048] Specifically, the classification decision unit 143 is used to input the optimized kitchen fume level characterization feature vector into the range hood controller based on the classifier to obtain the range hood control instruction. That is, the pre-trained classifier is used to perform feature analysis on the optimized kitchen fume level characterization feature vector, identify the fume state characteristics of the current cooking environment, and combine the classification knowledge learned in the training process to map the optimized kitchen fume level characterization feature vector to the predefined fume control instruction space, generate the corresponding range hood control instruction, and instruct the range hood to turn on, increase the wind speed, reduce the wind speed or turn off, so as to effectively remove the fume in the kitchen, maintain the comfort of the kitchen environment, and save energy.

[0049] In summary, the intelligent kitchen cooking detection system based on multi-source data according to the embodiment of the present application is explained, which uses artificial intelligence technology based on deep learning to perform signal analysis on the odor signal and smoke signal collected by the electronic nose and smoke sensor in the kitchen to capture the time-frequency feature representation of the odor signal and the smoke signal, and then characterize the kitchen fume level based on the interactive fusion characteristics between the two, thereby realizing intelligent control of the kitchen range hood. In this way, the wind speed and operating state of the range hood can be automatically adjusted according to the degree of fume generated during the cooking process, ensuring the freshness of the kitchen air and avoiding the potential threat of fume to the health of family members.

[0050] Figure 6 FIG. 1 is a flow chart of a smart kitchen cooking detection method based on multi-source data according to an embodiment of the present application. Figure 6 As shown, according to the embodiment of the present application, the intelligent kitchen cooking detection method based on multi-source data includes the following steps: S110, obtaining a kitchen cooking odor signal collected by an electronic nose and a kitchen smoke signal collected by a smoke sensor; S120, performing time-frequency analysis on the kitchen cooking odor signal and the kitchen smoke signal to obtain a kitchen cooking odor time-frequency feature map and a kitchen smoke time-frequency feature map; S130, performing feature association fusion enhancement on the kitchen cooking odor time-frequency feature map and the kitchen smoke time-frequency feature map to obtain a kitchen fume degree characterization feature map; S140, generating a range hood control instruction based on the kitchen fume degree characterization feature map, wherein the range hood control instruction indicates that the range hood should be turned on, the wind speed should be increased, the wind speed should be reduced, or it should be turned off.

[0051] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned intelligent kitchen cooking detection method based on multi-source data have been referred to above. Figures 1 to 5 The invention has been introduced in detail in the description of the intelligent kitchen cooking detection system based on multi-source data, and therefore, its repeated description will be omitted.

[0052] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above embodiments are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0053] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. In the several embodiments provided by the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiment described above is only schematic. For example, the module division is only a logical function division, and there may be other division methods in actual implementation. The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0054] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0055] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0056] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference to a figure in a claim should not be considered as limiting the claim to which it relates.

[0057] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units stated in the system claims can also be implemented by one unit through software or hardware.

[0058] Finally, it should be noted that the above description has been given for the purpose of illustration and description. In addition, the above embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. An intelligent kitchen cooking detection system based on multi-source data, characterized in that: include: A kitchen cooking detection module, used to obtain kitchen cooking odor signals collected by the electronic nose and kitchen smoke signals collected by the smoke sensor; A signal time-frequency analysis module, used for performing time-frequency analysis on the kitchen cooking smell signal and the kitchen smoke signal to obtain a kitchen cooking smell time-frequency feature graph and a kitchen smoke time-frequency feature graph; A multimodal representation module for the degree of cooking fume, used for performing feature correlation fusion enhancement on the time-frequency feature map of the kitchen cooking smell and the time-frequency feature map of the kitchen smoke to obtain a kitchen fume degree representation feature map; The range hood control instruction generating module is used to generate a range hood control instruction based on the kitchen fume degree characterization characteristic diagram, wherein the range hood control instruction indicates that the range hood should be turned on, the wind speed should be increased, the wind speed should be decreased, or it should be turned off.

2. The intelligent kitchen cooking detection system based on multi-source data according to claim 1 is characterized in that: The signal time-frequency analysis module comprises: A wavelet transform unit, used for performing wavelet transform on the kitchen cooking smell signal and the kitchen smoke signal respectively to obtain a time-frequency diagram of the kitchen cooking smell signal and a time-frequency diagram of the kitchen smoke signal; The time-frequency feature extraction unit is used to obtain the kitchen cooking smell time-frequency feature map and the kitchen smoke time-frequency feature map by a time-frequency feature extractor based on a convolutional neural network model.

3. The intelligent kitchen cooking detection system based on multi-source data according to claim 2 is characterized in that: The oil smoke degree multimodal characterization module is used for: The kitchen cooking odor time-frequency feature map and the kitchen smoke time-frequency feature map are arranged along the channel dimension into a multimodal kitchen cooking state time-frequency feature map, and then a feature bidirectional enhancement module including a channel attention layer and a spatial attention layer is used to obtain the kitchen fume degree characterization feature map.

4. The intelligent kitchen cooking detection system based on multi-source data according to claim 3 is characterized in that: The oil smoke degree multimodal characterization module is used for: The multimodal kitchen cooking state time-frequency feature map is passed through the spatial attention layer of the feature bidirectional enhancement module to obtain a local spatial enhanced kitchen cooking state time-frequency feature map; Passing the multimodal kitchen cooking state time-frequency feature map through the channel attention layer of the feature bidirectional enhancement module to obtain a global channel enhanced kitchen cooking state time-frequency feature map; The local space enhanced kitchen cooking state time-frequency feature map and the global channel enhanced kitchen cooking state time-frequency feature map are processed according to the position points to obtain the kitchen fume degree characterization feature map.

5. The intelligent kitchen cooking detection system based on multi-source data according to claim 4 is characterized in that: The range hood control instruction generating module comprises: A feature thinning unit, used for performing feature thinning processing on the kitchen fume degree characterization feature map to obtain a kitchen fume degree characterization feature vector; A feature optimization unit, used for performing feature optimization on the kitchen fume degree characterization feature vector to obtain an optimized kitchen fume degree characterization feature vector; The classification decision unit is used to input the optimized kitchen fume degree characterization feature vector into a range hood controller based on a classifier to obtain the range hood control instruction.

6. The intelligent kitchen cooking detection system based on multi-source data according to claim 5 is characterized in that: The feature sparsification unit is used to: The kitchen fume degree characterization feature map is processed using a principal component analysis method to obtain the kitchen fume degree characterization feature vector.

7. The intelligent kitchen cooking detection system based on multi-source data according to claim 6 is characterized in that: The feature optimization unit is used to: The kitchen fume degree characterization feature vector is input into a viewpoint metric correlation network based on backward learning sensitive preference to obtain the optimized kitchen fume degree characterization feature vector.

8. The intelligent kitchen cooking detection system based on multi-source data according to claim 7 is characterized in that: The feature optimization unit comprises: A probabilistic activation subunit, used for performing probabilistic activation on the kitchen fume degree characterization feature vector based on a Sigmoid function to obtain a probabilistic kitchen fume degree characterization feature vector; A pre-classification subunit, used for inputting the kitchen fume degree characterization feature vector into a classifier to obtain an initial class label probability value; An error metric subunit, used to calculate the absolute value of the difference between the feature value of each position in the probabilistic kitchen fume degree characterization feature vector and the initial class label probability value to obtain a backward machine learning fine-grained error metric vector; an attention optimization subunit, configured to use the backward machine learning fine-grained error metric vector as a key vector, the kitchen fume degree characterization feature vector as a query vector and a value vector, and input the key vector, the query vector and the value vector into a converter structure to obtain a backward learning class attention optimized kitchen fume degree characterization feature vector; an optimization scaling factor calculation subunit, configured to calculate a ratio between a sum of the backward machine learning fine-grained error metric vector and a length of the backward machine learning fine-grained error metric vector, and to calculate a natural exponential function value raised to a power of the ratio as an optimization scaling factor; The feature scaling subunit is used to calculate the product between the optimization scaling factor and the backward learning type attention optimized kitchen fume degree characterization feature vector to obtain the optimized kitchen fume degree characterization feature vector.

9. An intelligent kitchen cooking detection method based on multi-source data, characterized in that: include: Acquire kitchen cooking odor signals collected by the electronic nose and kitchen smoke signals collected by the smoke sensor; Performing time-frequency analysis on the kitchen cooking smell signal and the kitchen smoke signal to obtain a kitchen cooking smell time-frequency feature graph and a kitchen smoke time-frequency feature graph; Performing feature correlation fusion enhancement on the kitchen cooking odor time-frequency feature map and the kitchen smoke time-frequency feature map to obtain a kitchen fume degree representation feature map; Based on the kitchen fume degree characterization characteristic diagram, a range hood control instruction is generated, wherein the range hood control instruction indicates that the range hood should be turned on, the wind speed should be increased, the wind speed should be decreased, or it should be turned off.

10. The intelligent kitchen cooking detection method based on multi-source data according to claim 9, characterized in that: Performing time-frequency analysis on the kitchen cooking smell signal and the kitchen smoke signal to obtain a kitchen cooking smell time-frequency feature graph and a kitchen smoke time-frequency feature graph, including: Performing wavelet transform on the kitchen cooking smell signal and the kitchen smoke signal respectively to obtain a time-frequency diagram of the kitchen cooking smell signal and a time-frequency diagram of the kitchen smoke signal; The kitchen cooking odor signal time-frequency diagram and the kitchen smoke signal time-frequency diagram are passed through a time-frequency feature extractor based on a convolutional neural network model to obtain the kitchen cooking odor time-frequency feature diagram and the kitchen smoke time-frequency feature diagram.

Images