Computer vision-based cooking oil fume air pollution feature representation method and system

Abstract

The application discloses a cooking oil fume air pollution feature representation method and system based on computer vision, and the method comprises the following steps: obtaining cooking picture video information and performing data preprocessing to obtain preprocessed cooking information; performing fusion processing on the preprocessed cooking information based on a cross-modal fusion method of a motion enhancement attention mechanism to obtain cooking feature information; constructing a two-dimensional time sequence model based on cooking multi-periodicity and predicting the cooking feature information to obtain an indoor air particulate matter pollution level. The system comprises an acquisition module, a fusion module and a prediction module. By using the application, the indoor air particulate matter pollution level of cooking can be predicted. The application can be widely applied to the field of computer vision recognition technology as a cooking oil fume air pollution feature representation method and system based on computer vision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision recognition, and particularly relates to a cooking oil fume air pollution feature representation method and system based on computer vision. BACKGROUND

[0002] High-temperature cooking produces a large amount of dangerous air pollutants, including a variety of carcinogens recognized by international cancer organizations. High-temperature cooking is common in Chinese households, and the related indoor air pollution is serious. According to preliminary estimates, for non-smoking urban households in China, cooking sources and air pollution sources contribute equally to the total exposure of fine particulate matter to the population. Therefore, representing the cooking oil fume air pollution feature has important practical significance for further improving indoor air quality and protecting people's health in China. The level of indoor air pollution caused by cooking oil fume is largely affected by cooking methods, time, exhaust hood working status, and ventilation conditions. For example, under normal circumstances, cooking methods such as frying and stir-frying will produce more air pollutants than frying and stewing. The air pollution level produced by cooking in a well-ventilated kitchen is relatively low. In addition, the above information will change greatly due to changes in the external environment, and there are large differences in kitchens in different households and restaurants. Therefore, it is necessary to accurately record the air pollution level and the above related information for a long time in order to more accurately represent the cooking oil fume air pollution feature. The commonly used representation method at present is based on short-term household testing. Professional technicians use relatively high-end and expensive equipment to measure the air pollution level during cooking, without recording related cooking information such as cooking time and cooking method, exhaust hood working status, and building ventilation conditions. Or rely on staff and subjects to record the above information. This method greatly depends on human resources, has high cost, and is difficult to implement long-term measurement of air pollution levels and cooking-related information in multiple environments. SUMMARY

[0003] To solve the above technical problems, the purpose of the present application is to provide a cooking oil fume air pollution feature representation method and system based on computer vision, which improves the prediction of indoor air particulate matter pollution levels during cooking by using a cross-modal fusion method based on a motion-enhanced attention mechanism and a two-dimensional time series model based on cooking periodicity.

[0004] The first technical solution adopted by the present application is: a cooking oil fume air pollution feature representation method based on computer vision, comprising the following steps:

[0005] Obtain cooking picture video information and perform data preprocessing to obtain preprocessed cooking information;

[0006] A cross-modal fusion method based on a motion-enhanced attention mechanism is used to fuse and process the preprocessed cooking information to obtain cooking feature information;

[0007] A two-dimensional timing model based on cooking multi-periodicity is constructed, and cooking feature information is predicted to obtain the air particulate matter pollution level in the cooking chamber.

[0008] Further, the step of obtaining cooking picture video information and performing data preprocessing to obtain preprocessed cooking information specifically includes:

[0009] Obtain the cooking picture video information of the kitchen through the camera module and the environment monitor;

[0010] Split the cooking picture video information of the kitchen based on Python to obtain corresponding audio information and video information;

[0011] Disassemble the audio information based on FFMPEG to obtain corresponding loudness information and text information;

[0012] Integrate the text information and the video information to construct the preprocessed cooking information.

[0013] Further, it further includes obtaining the working current information of the oil smoke machine in the kitchen through the intelligent electric meter, judging the working state of the oil smoke machine according to the working current information of the oil smoke machine, and further judging the pollution level generated in the kitchen during cooking.

[0014] Further, the step of the cross-modal fusion method based on the motion enhanced attention mechanism for fusing the preprocessed cooking information to obtain the cooking feature information specifically includes:

[0015] Based on the text information feature extractor, the text information in the preprocessed cooking information is extracted and processed to obtain language text feature information;

[0016] Introduce the motion attention mechanism, construct the Cooking-Transformer module, and extract and process the video information in the preprocessed cooking information to obtain video feature information;

[0017] Fuse the language text feature information and the video feature information to obtain the cooking feature information.

[0018] Further, the step of the text information feature extractor for extracting and processing the text information in the preprocessed cooking information to obtain language text feature information specifically includes:

[0019] Input the text information in the preprocessed cooking information into the text information feature extractor, and the text information feature extractor includes a self-attention module and a multi-head attention module;

[0020] The self-attention module is used to assign scores to the word correlation between the input text information and to perform weighted average calculation according to the score results, so as to obtain the context-related representation of the words between the text information.

[0021] The multi-head attention module is used to obtain information in different dimensions of the context-related representation of the words between the text information, and to construct language text feature information.

[0022] Further, the cooking-motion attention mechanism is introduced, and the Cooking-Transformer module is constructed to perform feature extraction processing on the video information in the preprocessed cooking information, so as to obtain the video feature information, which specifically includes:

[0023] The video information in the preprocessed cooking information is input into the Cooking-Transformer module.

[0024] Each frame of the video information in the preprocessed cooking information is divided into non-overlapping patches.

[0025] The non-overlapping patches are combined to obtain video frame blocks.

[0026] The spatial attention module is used to perform spatial dimension extraction processing on the video frame blocks, so as to obtain corresponding spatial dimension object actions.

[0027] The time attention module is used to perform time dimension extraction processing on the video frame blocks, so as to obtain corresponding time dimension object actions.

[0028] The I3D and CNN fusion network is used to fuse the spatial dimension object actions and the time dimension object actions, so as to obtain the video feature information.

[0029] Further, the two-dimensional time sequence model based on cooking multi-periodicity is constructed, and the cooking feature information is predicted to obtain the cooking indoor air particulate matter pollution level, which specifically includes:

[0030] The cooking feature information is sequentially subjected to cleaning, data regularization and data set division processing to obtain one-dimensional cooking feature information.

[0031] The one-dimensional cooking feature information is input into the two-dimensional time sequence model based on cooking multi-periodicity.

[0032] The periodicity of the one-dimensional cooking feature information is calculated by fast Fourier transform.

[0033] The original time sequence of the cooking feature information is folded based on the periodicity of the one-dimensional cooking feature information, and the time sequence feature is converted from one dimension to two-dimensional tensor to obtain a feature matrix.

[0034] The feature matrix is subjected to two-dimensional convolution feature extraction processing through a convolution layer to obtain two-dimensional time sequence features;

[0035] The two-dimensional time sequence features are subjected to dimension reduction processing to obtain one-dimensional frequency intensity information;

[0036] The one-dimensional frequency intensity information is subjected to weighted summation to obtain the air particulate matter pollution level in the cooking chamber.

[0037] Further, the expression for calculating the periodicity of the one-dimensional cooking feature information through fast Fourier transform is specifically as follows:

[0038] A = Avg (Amp (FFT (X 1D )))

[0039] In the above formula, X 1D represents a one-dimensional time sequence of cooking feature information, FFT(·) represents fast Fourier transform, A represents the periodicity of one-dimensional cooking feature information, Amp(·) represents a function that obtains the absolute value of each complex value element in the frequency domain signal, i.e., provides the amplitude of each frequency component, and Avg(·) represents a function that calculates the average of all amplitudes obtained.

[0040] Further, it further includes a process of predicting cooking feature information in combination with the loudness information of the cooking scene video information of the kitchen.

[0041] The second technical solution adopted by the present application is: a cooking oil fume air pollution feature representation system based on computer vision, comprising:

[0042] An acquisition module is configured to acquire cooking scene video information and perform data preprocessing to obtain preprocessed cooking information;

[0043] A fusion module is configured to perform fusion processing on the preprocessed cooking information based on a cross-modal fusion method of a motion-enhanced attention mechanism to obtain cooking feature information;

[0044] A prediction module is configured to construct a two-dimensional time sequence model based on cooking multi-periodicity and predict the cooking feature information to obtain the air particulate matter pollution level in the cooking chamber.

[0045] The method and system have the advantages that the method extracts cooking time and mode by acquiring cooking picture video information, processes the cooking picture video information based on a cross-modal fusion method of a motion-enhanced attention mechanism, introduces a motion attention mechanism to construct a Cooking-Transformer module, considers adjacent patches from different frames in the attention module, captures relative patch motion, further improves the performance of feature extraction, records the working current of the smoke extractor, the current information mainly acquires the working state of the smoke extractor, the larger the current is, the higher the power of the smoke extractor is, and it is approximately considered that the pollutant level is lower, a two-dimensional time sequence model is further established based on the cooking multi-periodicity, one-dimensional time sequence data of cooking is expanded to two-dimensional space for analysis, the periodicity of the data can be fully utilized, and the indoor air particulate matter pollution level during cooking is predicted. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a step flow chart of the cooking oil fume air pollution feature representation method based on computer vision of the present application;

[0047] Figure 2 is a structural block diagram of the cooking oil fume air pollution feature representation system based on computer vision of the present application;

[0048] Figure 3 is a step flow chart of the cooking oil fume air pollution feature representation method of the specific embodiment of the present application;

[0049] Figure 4 is a flow chart of the Cooking-Transformer module processing of the specific embodiment of the present application;

[0050] Figure 5 is a device structure diagram of the kitchen sensor network construction of the present application;

[0051] Figure 6 is a result data diagram of the simulation experiment using the method of the present application;

[0052] The drawings show that: 1, a network camera; 2, an environment monitor; 3, a smoke extractor working current sensor. DETAILED DESCRIPTION

[0053] The present application will be further described in detail below in combination with the drawings and specific embodiments. For the step numbers in the following embodiments, only the setting is for the convenience of description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0054] Reference Figure 1 and Figure 3The application provides a cooking oil fume air pollution feature characterization method based on computer vision, which comprises the following steps:

[0055] S1, obtaining cooking picture video information and performing data preprocessing to obtain preprocessed cooking information;

[0056] S11, sensor selection;

[0057] Specifically, through preliminary investigation, the application selects particulate matter sensors, carbon dioxide sensors, temperature and humidity sensors, range hood working current sensors, network cameras, and corresponding power supply, data storage and transmission modules with excellent performance.

[0058] S12, sensor network construction;

[0059] Specifically, referring to Figure 5 , the network camera is installed around the kitchen stove, and is required to clearly capture the cooking picture on the stove. The environmental monitor is provided with particulate matter sensors, carbon dioxide sensors, temperature and humidity sensors, and corresponding power supply, data storage and transmission modules, and is placed in the kitchen, and is required to be more than 1 meter away from the stove and each wall. The range hood working current sensor is connected to the range hood to measure and record the working current of the range hood. The equipment works uninterruptedly, and the data is transmitted to the cloud storage through the wireless network. Among them, the current information is mainly obtained from the working state of the range hood. The larger the current, the higher the power of the range hood, and it is approximately considered that the pollutant level is lower.

[0060] S2, a cross-modal fusion method based on motion enhancement attention mechanism is used to fuse the preprocessed cooking information to obtain cooking feature information;

[0061] Specifically, first, the video information collected by the network camera is split into audio and image information using Python+FFMPEG to realize video splitting. The audio information is split into loudness information and text information;

[0062] The text information is extracted by a traditional feature extractor based on a self-attention module. The core of the self-attention mechanism is to calculate the correlation score (attention score) between each word in the input sequence and other words, and then perform weighted average on the input sequence to obtain the context-related representation of each word. We also use the multi-head attention technology, which allows the model to focus on information in different dimensions at the same time, improving the ability and efficiency of feature extraction;

[0063] Referring to Figure 4, video information will be extracted through two network models at the same time, first through two Cooking-Transformer modules in succession for feature extraction, there are at least two challenges to be overcome in video classification: how to represent spatio-temporal information in video and how to use spatio-temporal information for classification. Spatio-temporal information contains two aspects: spatial information (such as objects in the frame) and temporal information (such as correlation in different frames), which is very important for video classification. However, the current spatio-temporal attention mechanism has a problem, the current spatio-temporal attention divides the query patch and the patch located in the same position in other frames. Suppose these patches are well aligned so that they can jointly model the motion information of some parts in the video, but due to the movement of video objects or the movement of the camera, there is always patch motion, which will cause the query patch and the patch in other frames to be misaligned, which will reduce the performance of feature extraction, obviously, in the attention module, adjacent patches from different frames should be considered, so that we can capture relative patch motion, inspired by this insight, the present application proposes a Cooking-Transformer module that can capture motion information, Cooking-Transformer divides each frame into non-overlapping patches, several adjacent patches form a block, and the attention in the time dimension is calculated on the same spatial block in different frames, called motion attention, in order to avoid the cost of calculation, Cooking-Transformer does not use spatio-temporal attention to jointly learn the spatial information and temporal information of each layer. Instead, it first calculates the spatial features and then calculates the spatio-temporal features, at the same time, the video frame will also be extracted through a layer of I3D and CNN fusion network, the features extracted by the two feature extractors will be input into the final classification network through feature fusion. The classification network will integrate the loudness feature, language text feature and video information feature to classify the cooking method, therefore, the present application proposes a Cooking-Transformer module that can capture motion information, Cooking-Transformer first divides each frame into non-overlapping patches, and several adjacent patches form a block. For each block, we first extract the action of the object in this block in the frame through a spatial attention module, and then extract the action of the object in the time dimension through a temporal attention module to obtain the correlation in different frames, finally, feature fusion is performed to input the next module;

[0064] Therefore, the self-attention module of the present application is used to calculate the correlation scores between each word in the input sequence and other words, and the input sequence is weighted and averaged using these scores to obtain a context-related representation of each word, the multi-head attention of the present application is used to allow the model to focus on information in different dimensions at the same time, improve the ability and efficiency of feature extraction, and the feature fusion device of the present application is used to fuse the features extracted by the text information feature extractor and the video information feature extractor into the classification network for cooking method classification.

[0065] S3, a two-dimensional time sequence model based on cooking multi-periodicity is constructed, and cooking feature information is predicted to obtain the cooking room air particulate matter pollution level.

[0066] Specifically, the image and video data in the data set are preprocessed, including cleaning, data regularization and data set division, and an innovative method is adopted in the model construction aspect, which extends one-dimensional time sequence data to two-dimensional space for analysis (in this way, the periodicity characteristics of the data can be fully utilized, and at the same time, convolution can be used to extract features in linear data). We fold the time series according to multiple periods to obtain multiple two-dimensional tensors. The columns and rows of each two-dimensional tensor reflect the time sequence changes within and between periods. Specifically, first, the periodicity of the feature is calculated using fast Fourier transform (FFT), for a one-dimensional time sequence X 1D , the formula for calculating the periodicity is:

[0067] A = Avg (Amp (FFT (X 1D ))

[0068] In the above formula, X 1D represents a one-dimensional time sequence of cooking feature information, FFT(·) represents fast Fourier transform, A represents the periodicity of one-dimensional cooking feature information, Amp(·) represents a function that obtains the absolute value of each complex value element in the frequency domain signal, that is, provides the amplitude of each frequency component, and Avg(·) represents a function that calculates the average of all amplitudes obtained;

[0069] Based on this periodicity, the original time sequence is folded, and the time sequence feature is converted from one-dimensional to two-dimensional tensor to form a feature matrix. Then, two-dimensional convolution is used for feature extraction through the convolution layer. After that, the extracted time sequence feature is reduced in dimension, which is degraded from two-dimensional to one-dimensional to achieve information aggregation. Then, the one-dimensional representation obtained by the present application is weighted and summed according to the strength of its corresponding frequency to obtain the final output.

[0070] Referring to Figure 2 , the cooking oil fume air pollution feature representation system based on computer vision comprises:

[0071] An acquisition module is configured to acquire cooking picture video information and perform data preprocessing to obtain preprocessed cooking information;

[0072] A fusion module is configured to perform fusion processing on the preprocessed cooking information based on a cross-modal fusion method of a motion-enhanced attention mechanism to obtain cooking feature information.

[0073] A prediction module is configured to construct a two-dimensional time sequence model based on cooking multi-periodicity and predict the cooking feature information to obtain an indoor air particulate matter pollution level.

[0074] In summary, the technical scheme of the present application comprises three steps, and the third step of analysis and prediction based on multi-modal sensor data comprises constructing a comprehensive database using data collected by a multi-sensor network, predicting and extracting cooking information (cooking time and method) using the computer vision model obtained in step two and the cooking image or video obtained in step one, and predicting the indoor air particulate matter pollution level caused by cooking using the hood working information (current level captured by a current sensor) and ventilation information (represented by indoor carbon dioxide level) in combination with the two-dimensional time sequence model constructed based on cooking multi-periodicity.

[0075] Reference Figure 6 The present application proposes a cross-modal fusion method based on a motion-enhanced attention mechanism for cooking method classification, which has an accuracy of more than 98% in the cooking method classification task.

[0076] The contents in the above method embodiments are all applicable to the present system embodiment, the present system embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0077] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above-mentioned embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A computer vision-based method for characterizing air pollution features from cooking fumes, characterized in that, Includes the following steps: The cooking video information is acquired and preprocessed to obtain the preprocessed cooking information. A cross-modal fusion method based on motion-enhanced attention mechanism is used to fuse preprocessed cooking information to obtain cooking feature information; A two-dimensional time-series model based on the multi-period nature of cooking was constructed and cooking characteristic information was used to predict the level of indoor air particulate pollution in cooking rooms; The cross-modal fusion method based on motion-enhanced attention mechanism performs fusion processing on preprocessed cooking information to obtain cooking feature information. This step specifically includes: Based on the text information feature extractor, feature extraction processing is performed on the text information in the preprocessed cooking information to obtain language text feature information; A motion attention mechanism is introduced, and a Cooking-Transformer module is constructed to extract features from the preprocessed cooking information to obtain video feature information. By fusing language text features with video features, cooking features are obtained. The step of constructing a two-dimensional time-series model based on the multi-period nature of cooking and predicting the level of particulate matter pollution in the cooking room based on cooking characteristic information specifically includes: The cooking feature information is sequentially cleaned, data normalized, and dataset partitioned to obtain one-dimensional cooking feature information. One-dimensional cooking feature information is input into a two-dimensional time-series model based on the multi-periodity of cooking; The periodicity of one-dimensional cooking feature information is calculated using Fast Fourier Transform; Based on the periodicity of one-dimensional cooking feature information, the original time series of cooking feature information is folded to transform the time series features from one-dimensional to two-dimensional tensors, thus obtaining the feature matrix. Two-dimensional temporal features are obtained by performing two-dimensional convolutional feature extraction on the feature matrix through convolutional layers; The two-dimensional time series features are reduced in dimensionality to obtain one-dimensional representation of frequency intensity information. The level of particulate matter pollution in the indoor air is obtained by weighted summation of one-dimensional frequency intensity information.

2. The method for characterizing the air pollution features of cooking fumes based on computer vision according to claim 1, characterized in that, The step of acquiring cooking video information and performing data preprocessing to obtain preprocessed cooking information specifically includes: The cooking video information in the kitchen is obtained through a camera module and an environmental monitoring device; The video information of cooking scenes in the kitchen is split and processed using Python to obtain the corresponding audio and video information; Based on FFMPEG, audio information is decomposed and processed to obtain corresponding loudness information and text information; Integrate text and video information to construct pre-processed cooking information.

3. The method for characterizing the air pollution features of cooking fumes based on computer vision according to claim 2, characterized in that, It also includes obtaining the operating current information of the range hood during kitchen cooking through a smart meter, judging the working status of the range hood based on the operating current information, and then judging the level of pollutants generated during kitchen cooking.

4. The method for characterizing the air pollution features of cooking fumes based on computer vision according to claim 3, characterized in that, The step of extracting features from the preprocessed cooking information using a text information feature extractor to obtain language text feature information specifically includes: The text information in the preprocessed cooking information is input into a text information feature extractor, which includes a self-attention module and a multi-head attention module. Based on the self-attention module, the word relevance between the input text information is scored and a weighted average is calculated based on the score results to obtain the contextual relevance representation of the words between the text information. Based on the multi-head attention module, information on different dimensions of the contextual relationship between words in the text is obtained to construct language text feature information.

5. The method for characterizing the air pollution features of cooking fumes based on computer vision according to claim 4, characterized in that, The step of introducing a motion attention mechanism and constructing a Cooking-Transformer module to extract features from the preprocessed cooking information to obtain video feature information specifically includes: Input the video information from the pre-processed cooking information into the Cooking-Transformer module; Each frame of the video information in the preprocessed cooking information is divided into non-overlapping patches; Non-overlapping patches are combined to obtain video frame blocks; The spatial attention module is used to extract the spatial dimension of video frame blocks to obtain the corresponding spatial dimension object action. The temporal attention module is used to extract the temporal dimension of video frame blocks to obtain the corresponding temporal dimension object action. Based on the fusion network of I3D and CNN, the actions of objects in the spatial dimension and the actions of objects in the temporal dimension are fused to obtain video feature information.

6. The method for characterizing the air pollution features of cooking fumes based on computer vision according to claim 5, characterized in that, The expression for calculating the periodicity of one-dimensional cooking feature information using Fast Fourier Transform is as follows: In the above formula, A one-dimensional time series representing cooking feature information. Represents the Fast Fourier Transform. This indicates the periodicity of one-dimensional cooking feature information. The function retrieves the absolute value of each complex element in the frequency domain signal, thus providing the amplitude of each frequency component. This represents the average value of all amplitudes obtained from the function calculation.

7. The method for characterizing the air pollution features of cooking fumes based on computer vision according to claim 6, characterized in that, It also includes the process of predicting cooking feature information by combining loudness information from kitchen cooking video footage.

8. A computer vision-based system for characterizing air pollution from cooking fumes, characterized in that, The method for characterizing cooking fume air pollution based on computer vision as described in claim 1 includes the following modules: The acquisition module is used to acquire cooking video information and perform data preprocessing to obtain preprocessed cooking information. The fusion module uses a cross-modal fusion method based on motion-enhanced attention mechanism to fuse preprocessed cooking information to obtain cooking feature information; The prediction module is used to construct a two-dimensional time series model based on the multi-period nature of cooking and predict cooking characteristic information to obtain the indoor air particulate pollution level.

Images