Oven control method and system based on food image

Abstract

The invention provides an oven control method and system based on food image, which belongs to the technical field of ovens, and comprises the following steps: S101: obtaining preset information, constructing a CNN-LSTM hybrid model, and performing area division; S102: obtaining multiple frames of baking images of different unit times, performing image data processing, and obtaining image data matrices of different unit times; S103: importing the image data matrices of different unit times into the CNN-LSTM hybrid model one by one; S104: performing iterative calculation based on the CNN-LSTM hybrid model, and deriving a result CNN-LSTM hybrid model; S105: importing the image data matrix of the current unit time, and obtaining a predicted baking time based on the result CNN-LSTM hybrid model; S106: comparing and judging the baking time with the preset time, and performing corresponding operations; the oven can predict the baking time based on the food image and the CNN-LSTM hybrid model, and adjust the baking time accordingly, so as to prevent the oven from over-baking or poorly baking food, and improve the baking effect of the food.

Description

Technical Field

[0001] The present invention belongs to the technical field of ovens, and more specifically, particularly relates to an oven control method and system based on food image. Background Art

[0002] In people's daily life, ovens have become one of the important cooking tools and are widely used to bake all kinds of food. With the development of technology, ovens are gradually evolving towards intelligence. The new generation of smart ovens not only have traditional cooking modes, including baking, thawing and other functions, but also integrate more technologies, such as automatic identification of ingredients and intelligent adjustment of cooking parameters, such as humidity and temperature. However, when baking food, traditional ovens often bake the food in the oven based on a preset time and do not have the function of self-adjusting the baking time, which will cause the food in the oven to be over-baked or poorly baked. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides an oven control method and system based on food images to solve the technical problem in the prior art that traditional ovens, when baking food, often bake the food in the oven based on a preset time and do not have the function of self-adjusting the baking time, resulting in over-baking or poor baking of the food in the oven.

[0004] The purpose and effect of the oven control method and system based on food image of the present invention are achieved by the following specific technical means:

[0005] An oven control method based on food image comprises the following steps:

[0006] S101: After the oven starts baking, preset information is obtained, wherein the preset information includes a preset time for food to be converted from raw food to cooked food, a CNN-LSTM hybrid model is constructed, and an internal area of ​​the oven is divided into multiple sub-areas based on the CNN-LSTM hybrid model;

[0007] S102: Acquire multiple frames of baking images at different unit times, where the unit time is 1 minute, and the baking images are real-time images of baking inside the oven. Perform image data processing on the baking images at different frames, and acquire image data matrices at different unit times based on the multiple frames of baking images at different unit times. The image data matrix is:

[0008]

[0009] Wherein, a, b, c...z are image feature data, m is the mth image feature data in different rows in the image data matrix, x is the region mark matching the image feature data in the row of the image data matrix, and y is the time corresponding to the image feature data in the column of the image data matrix;

[0010] S103: importing the image data matrices of different unit times into the CNN-LSTM hybrid model one by one, and matching the image feature data in the image data matrices of different unit times with the plurality of sub-regions one by one based on the CNN-LSTM hybrid model;

[0011] S104: performing iterative calculation based on the CNN-LSTM hybrid model, and when the calculation result meets a preset condition, exporting the CNN-LSTM hybrid model as a result CNN-LSTM hybrid model;

[0012] S105: importing the image data matrix of the current unit time into the result CNN-LSTM hybrid model, and obtaining a predicted baking time based on the result CNN-LSTM hybrid model, wherein the predicted baking time is a predicted time for converting the food from raw food to cooked food predicted by the result CNN-LSTM hybrid model;

[0013] S106: Compare and judge the predicted baking time with the preset time, and perform corresponding operations, including:

[0014] S1061: If the predicted baking time is higher than the preset time, the oven continues to operate to bake the food until the baking time reaches the predicted baking time;

[0015] S1062: If the predicted baking time is equal to the preset time, the oven continues to operate to bake the food until the baking time reaches the preset time;

[0016] S1063: If the predicted baking time is less than the preset time, the oven continues to operate to bake the food until the redundant time is reached.

[0017] As a further solution of the present invention, a plurality of frames of baking images of different unit times are obtained, wherein the unit time is 1 minute, and the baking images are real-time images of baking inside the oven, and image data processing is performed on the baking images of different frames, and image data matrices of different unit times are obtained based on the plurality of frames of baking images of different unit times, including:

[0018] S201: acquiring a single frame of the baking image inside the oven every 1 second, and capturing a total of 60 frames of the baking image per unit time;

[0019] S202: performing equal-area cutting on the baking image of each frame, cutting the baking image of each frame into a plurality of block images, dividing the plurality of block images into different regions, and marking the different regions;

[0020] S203: grayscale processing is performed on the multiple block images of each frame, the multiple color block images of each frame are converted into multiple black and white block images, and Gaussian filtering is applied to remove noise from the multiple block images of each frame;

[0021] S204: Based on the multiple block images of each frame, a plurality of image feature data of each frame are obtained, the multiple image feature data of each frame are matched with different region marks, timestamps are marked on the multiple image feature data based on different frames, and the multiple image feature data of the same frame are marked with the same timestamp, multiple block data arrays are established based on different region marks and the image feature data of different frames, region markings are performed on the multiple block data arrays based on the regions where the block images are located, and the multiple image feature data in the multiple block data arrays are arranged in positive order according to the corresponding timestamps. The block data arrays of different frames are:

[0022]

[0023] Wherein, A…Z are multiple block data arrays marked with different regions, a, b, c…z are the image feature data in the multiple block data arrays, and n is the number of image feature data contained in different block data arrays;

[0024] S205: Screening the plurality of block data arrays based on an edge detection algorithm:

[0025] If the image feature data in the block data array includes any food feature data, the block data array is retained;

[0026] If the image feature data in the block data array does not include any food feature data, deleting the block data array;

[0027] S206: Establishing the image data matrix based on the retained plurality of block data arrays.

[0028] As a further solution of the present invention, screening the plurality of block data arrays based on an edge detection algorithm includes:

[0029] A plurality of food feature regions are obtained based on an edge detection algorithm, and a plurality of food feature data are obtained based on the plurality of food feature regions.

[0030] As a further solution of the present invention, a plurality of frames of baking images of different unit times are obtained, wherein the unit time is 1 minute, and the baking images are real-time images of baking inside the oven, and image data processing is performed on the baking images of different frames, and image data matrices of different unit times are obtained based on the plurality of frames of baking images of different unit times, including:

[0031] S301: acquiring a single frame of the baking image inside the oven every 0.5 seconds, and capturing a total of 120 frames of the baking image per unit time;

[0032] S302: preprocessing each frame of the baking image, wherein the preprocessing includes grayscale processing of each frame of the baking image, converting each frame of the color baking image into a black-and-white baking image, and applying Gaussian filtering to remove noise from each frame of the baking image;

[0033] S303: Obtaining comparison parameters of each frame of the baking image, and comparing and judging each frame of the baking image based on the comparison parameters and a preset brightness threshold:

[0034] If the comparison parameter is lower than the preset brightness threshold, deleting the baking image corresponding to the comparison parameter;

[0035] If the comparison parameter is higher than or equal to the preset brightness threshold, retaining the baking image corresponding to the comparison parameter;

[0036] S304: establishing a data plane for one of the retained baking images, acquiring a plurality of plane regions of equal area based on the data plane, marking the plurality of plane regions, and acquiring a feature data vector of each frame of the retained baking image;

[0037] S305: Based on a clustering processing algorithm, a plurality of cluster centers located in different plane areas are established on the data plane, a plurality of feature data vectors are assigned to clusters of each cluster center, a plurality of different clusters are matched with a plurality of different area labels, and the image feature matrix is ​​established based on the plurality of different clusters.

[0038] As a further solution of the present invention, obtaining a comparison parameter of the baking image of each frame, and comparing and judging each baking image based on the comparison parameter and a preset brightness threshold, including:

[0039] The grayscale value of each frame of the baking image is obtained, and the comparison parameter of each frame of the baking image is obtained based on the grayscale value. The calculation formula of the comparison parameter is:

[0040]

[0041] Wherein, L is the width of each frame of the baking image, H is the width of each frame of the baking image, GR is the grayscale value of each frame of the baking image, MGR is the average brightness of each frame of the baking image, is the comparison parameter of the baked image for each frame.

[0042] As a further solution of the present invention, iterative calculation is performed based on the CNN-LSTM hybrid model, and when the calculation result meets the preset conditions, the CNN-LSTM hybrid model is exported as the result CNN-LSTM hybrid model, including:

[0043] Based on the CNN-LSTM hybrid model, multiple sub-regions that fail to match the image feature data are marked as vacant, and based on the CNN-LSTM hybrid model, multiple peripheral sub-regions that match the image feature data are marked as edges. When performing iterative calculations, the weights of the image feature data that match the edge-marked sub-regions are reset to γ, where γ is greater than 0 and less than 1.

[0044] As a further solution of the present invention, if the predicted baking time is less than the preset time, the oven continues to operate to bake the food until the redundant time is reached, including:

[0045] The redundant time is the time that the oven prolongs the baking time of the food inside. The calculation formula of the redundant time is:

[0046]

[0047] Where, T re is the redundant time, T ac is the preset time, T pr The predicted baking time is given.

[0048] An oven control system based on food image, comprising:

[0049] A preset module, the preset module is used to obtain preset information, the preset information includes a preset time for food to be converted from raw food to cooked food;

[0050] An acquisition module is used to acquire multiple frames of baking images per unit time, where the unit time is 1 minute, and the baking images are real-time images of baking inside the oven, including:

[0051] A single frame of the baking image inside the oven is acquired every 1 second, and a total of 60 frames of the baking image are captured per unit time;

[0052] Or, a single frame of the baking image inside the oven is acquired every 0.5 s, and a total of 120 frames of the baking image are captured per unit time;

[0053] The baking processing module includes a prediction module, an analysis module and a judgment module;

[0054] An analysis module is used to perform image data processing on the baking images of different frames, wherein the image data processing is to delete redundant image data, obtain image data matrices of different unit times based on multiple frames of the baking images of different unit times, and import the image data matrices of different unit times into the CNN-LSTM hybrid model one by one;

[0055] A prediction module, used to construct the CNN-LSTM hybrid model, and to obtain the predicted baking time through the CNN-LSTM hybrid model;

[0056] The judgment module is used to compare and judge the predicted baking time with the preset time and perform corresponding operations, including:

[0057] If the predicted baking time is higher than the preset time, the oven continues to operate to bake the food until the baking time reaches the predicted baking time;

[0058] If the predicted baking time is equal to the preset time, the oven continues to operate to bake the food until the baking time reaches the preset time;

[0059] If the predicted baking time is less than the preset time, the oven continues to operate to bake the food until a redundant time is reached, where the redundant time is the time the oven extends the baking time of the food inside.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] First, preset information is obtained based on the preset module, and the preset information includes the preset time for the food to be converted from raw food to cooked food. A CNN-LSTM hybrid model is constructed based on the prediction module. The internal area of ​​the oven is divided into multiple sub-areas through the CNN-LSTM hybrid model. Multi-frame baking images of different unit times are obtained based on the acquisition module. Image data processing is performed on the baking images based on the analysis module, and image data matrices of different unit times are obtained. Image data processing of the baking images can make the image feature data in the image data matrix more convenient for subsequent calculations. Subsequently, the image data matrices are imported into the CNN-LSTM hybrid model one by one, and iterative calculations are performed through the CNN-LSTM hybrid model. When the preset conditions are met, the resulting CNN-LSTM hybrid model is exported, and then the image data of the current unit time is converted into the image data matrix. The matrix is ​​imported into the result CNN-LSTM hybrid model, and the predicted baking time is obtained based on the result CNN-LSTM hybrid model. The predicted baking time is compared and judged with the preset time based on the judgment module. When the predicted baking time is higher than the preset time, the oven runs until the baking time reaches the predicted baking time. When the predicted baking time is equal to the preset time, the oven runs until the preset time is reached. When the predicted baking time is less than the preset time, the oven runs until the redundant time is reached. The oven can predict the baking time based on the food image and the CNN-LSTM hybrid model, and adjust the baking time accordingly, which can prevent the oven from over-baking or poorly baking the food, improve the baking effect of the food, and solve the problem that traditional ovens often do not have the function of self-adjusting the baking time, resulting in over-baking or poorly baking the food in the oven. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flowchart of the steps of an oven control method and system based on food image of the present invention;

[0063] Figure 2 It is a flowchart of comparison and judgment in step S106 of an oven control method and system based on food image of the present invention;

[0064] Figure 3 It is a flowchart of the steps of image data processing in step S102 of an embodiment 1 of an oven control method and system based on food image of the present invention;

[0065] Figure 4 It is a flowchart of the steps of image data processing in step S102 of a second embodiment of an oven control method and system based on food image of the present invention;

[0066] Figure 5 It is a schematic diagram of the area division of the internal area of ​​the oven in an oven control method based on food image and a system of the present invention. DETAILED DESCRIPTION

[0067] The following is a further detailed description of the embodiments of the present invention in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the technical solutions of the present invention, but cannot be used to limit the scope of protection of the present invention.

[0068] Embodiment 1:

[0069] As attached Figure 1 , Figure 2 , Figure 3 and Figure 5 As shown:

[0070] The present invention provides an oven control method based on food image, which is applicable to an oven and includes the following steps:

[0071] S101: After the oven starts baking, preset information is obtained, including a preset time for the food 102 to be converted from raw food to cooked food, a CNN-LSTM hybrid model is constructed, and the internal area 101 of the oven is divided into a plurality of sub-areas based on the CNN-LSTM hybrid model, and the area labels are matched with the sub-areas;

[0072] It can be understood that the CNN-LSTM hybrid model is a hybrid model that combines the convolutional neural network (CNN) with the long short-term memory network (LSTM). The CNN-LSTM hybrid model can capture the temporal and spatial features in the data. The convolutional neural network (CNN) performs convolution operations on the image feature data in the image data matrix, and the long short-term memory network (LSTM) can extract the time information in the image data matrix and perform operations based on the convolutional neural network (CNN), thereby obtaining the predicted baking time.

[0073] S102: Acquire multiple frames of baking images of different unit time, where the unit time is 1 minute, and the baking images are real-time images of baking inside the oven, and perform image data processing on the baking images of different frames. By processing the image data of the baking images, the image feature data in the image data matrix can be more convenient for subsequent calculations, wherein the image data processing includes the following steps:

[0074] S201: acquiring a single frame baking image inside the oven every 1 second, and capturing a total of 60 frames of baking images per unit time;

[0075] S202: performing equal-area cutting on each frame of the baking image, cutting each frame of the baking image into a plurality of block images, dividing the plurality of block images into different regions, and marking the different regions;

[0076] It can be understood that performing equal-area cutting on the baked image and cutting the baked image into multiple block images helps reduce the complexity of regional processing. By image cutting, the data contained in an image is divided into multiple groups of image data, which reduces the amount of data in a single image and is conducive to the subsequent extraction of image feature data.

[0077] S203: grayscale processing is performed on the multiple block images of each frame, the multiple color block images of each frame are converted into multiple black and white block images, and Gaussian filtering is applied to remove noise from the multiple block images of each frame;

[0078] It can be understood that grayscale processing of multiple block images can convert the block images into black and white images, and black and white images can reduce the interference of color information. Applying Gaussian filtering to the grayscale processed block images can remove noise in the block images, thereby enhancing the image quality of the block images.

[0079] S204: Based on the multiple block images of each frame, multiple image feature data of each frame are obtained, the multiple image feature data of each frame are matched with different regional marks, and the multiple image feature data are marked with timestamps based on different frames, and the multiple image feature data of the same frame are marked with the same timestamp. After aligning the timestamps, multiple block data arrays are established based on different regional marks and image feature data of different frames, and the block data arrays contain data contained in multiple block images with the same regional mark. The multiple block data arrays are marked with regional marks based on the regions where the block images are located, and the multiple image feature data in the multiple block data arrays are arranged in positive order according to the corresponding timestamps, so that it is easier for the analysis module to read the image feature data contained in the block data array, and there is no need for the analysis module to re-sort the image feature data when reading. The block data arrays of different frames are:

[0080]

[0081] Wherein, A…Z are multiple block data arrays marked with different regions, a, b, c…z are image feature data in multiple block data arrays, and n is the number of image feature data contained in different block data arrays;

[0082] S205: Screening multiple block data arrays based on edge detection algorithm:

[0083] Acquire multiple food feature regions based on an edge detection algorithm, and acquire food feature data based on the multiple food feature regions;

[0084] It can be understood that, in an image, an edge is usually a pixel area with a large change in grayscale value. The edge detection algorithm can detect the change in grayscale value in the image. When a large change in grayscale value occurs in a block image, the area where the block image is located is marked as a food feature area. The food feature area is the area in the block image that contains food 102. For block images that contain food 102 in some images, a large change in grayscale value will occur at the edge of the food 102, and the area where the block image is located is marked as a food feature area. For block images that contain food 102 in the entire image, the grayscale value will also change accordingly due to the texture of the food 102, and the area where the block image is located is marked as a food feature area, and multiple food feature data in multiple food feature areas are extracted, while the grayscale value of the block image that does not contain food 102 remains basically unchanged;

[0085] If the image feature data in the block data array includes any set of food feature data, the block data array is retained; if the image feature data in the block data array does not include any set of food feature data, the block data array is deleted;

[0086] S206: Establishing an image data matrix based on the retained plurality of block data arrays.

[0087] After processing the image data of the baked image, the image data matrix of different unit times is obtained based on multiple frames of baked images of different unit times. The image data matrix is:

[0088]

[0089] Among them, a, b, c...z are image feature data, m is the mth image feature data in different rows in the image data matrix, x is the area mark matching the image feature data in the row of the image data matrix, and y is the time corresponding to the image feature data in the column of the image data matrix.

[0090] S103: importing the image data matrices of different unit times into the CNN-LSTM hybrid model one by one, and matching the image feature data in the image data matrices of different unit times with the multiple sub-regions one by one based on the CNN-LSTM hybrid model;

[0091] It can be understood that each image feature data has been region-marked. When matching with a sub-region, it is only necessary to match the region mark with the sub-region. After a successful match, the image feature data can be classified into the sub-region.

[0092] S104: Based on the CNN-LSTM hybrid model, multiple sub-regions that fail to match the image feature data are marked as vacant, and the vacant marking can make the CNN-LSTM hybrid model not operate on the vacant marked sub-regions. Based on the CNN-LSTM hybrid model, multiple peripheral sub-regions that match the image feature data are marked at the edge, and only part of the image in the peripheral sub-region contains food 102. When performing iterative calculations, the weight of the image feature data matching the sub-region marked with the edge is reset to γ, and γ is greater than 0 and less than 1, and the weight of the image feature data in the peripheral sub-region is reduced, so that when the CNN-LSTM hybrid model performs iterative calculations, the predicted baking time is more accurate. Based on the CNN-LSTM hybrid model, iterative calculations are performed. When performing iterative calculations, the CNN-LSTM hybrid model can be optimized by a gradient descent algorithm and a regularization algorithm, so as to update the weights of the image feature data of different sub-regions, thereby improving the accuracy of the CNN-LSTM hybrid model. When the calculation result meets the preset condition, the preset condition is that when the error value of multiple iterative calculations is less than the preset value, the iterative calculation is stopped, and the CNN-LSTM hybrid model is exported as the result CNN-LSTM hybrid model;

[0093] It can be understood that when the error value calculated by multiple iterations is less than the preset value, the baking time predicted by the CNN-LSTM hybrid model is accurate enough, and the training of the CNN-LSTM hybrid model can be stopped. Continuing to train the CNN-LSTM hybrid model through iterative calculation will have little effect;

[0094] Among them, since the iterative calculation of the CNN-LSTM hybrid model has a more complex mathematical relationship, based on this, iterative calculation can be performed through artificial intelligence, so as to more accurately determine and describe the relationship between them.

[0095] S105: Import the image data matrix of the current unit time into the result CNN-LSTM hybrid model, and obtain the predicted baking time based on the result CNN-LSTM hybrid model. The predicted baking time is the predicted time for converting the food 102 from raw food to cooked food predicted by the result CNN-LSTM hybrid model.

[0096] S106: Compare the predicted baking time with the preset time and perform corresponding operations, including:

[0097] S1061: If the predicted baking time is higher than the preset time, the oven continues to operate to bake the food 102 until the baking time reaches the predicted baking time. When the predicted baking time is higher than the preset time, it indicates that the food 102 is not fully cooked when it is baked in the oven and reaches the preset time. Therefore, the baking time needs to be extended to the baking time so that the food 102 is cooked in the oven.

[0098] S1062: If the predicted baking time is equal to the preset time, the oven continues to operate to bake the food 102 until the baking time reaches the preset time. When the predicted baking time is equal to the preset time, it indicates that the food 102 is baked in the oven and is cooked when the preset time is reached, so no additional operation is required;

[0099] S1063: If the predicted baking time is less than the preset time, the oven continues to operate to bake the food 102 until the redundant time is reached. The redundant time is the time for the oven to extend the baking time of the food 102 inside. By setting the redundant time, it is possible to prevent the situation where the food 102 is not fully cooked when the oven finishes baking. The redundant time is calculated as:

[0100]

[0101] Where, T re is the redundant time, T ac is the preset time, T pr To predict baking time;

[0102] When the predicted baking time is less than the preset time, it indicates that the food 102 is being baked in the oven. When the preset time is reached, the food 102 will be overbaked, so the baking time is adjusted to a redundant time.

[0103] An oven control system based on food image, comprising:

[0104] A preset module, the preset module is used to obtain preset information, the preset information includes a preset time for the food 102 to be converted from raw food to cooked food;

[0105] The acquisition module is used to acquire multiple frames of baking images per unit time. The unit time is 1 minute. The baking image is a real-time image of baking inside the oven, including:

[0106] A single-frame baking image of the oven is acquired every 1 second, and a total of 60 frames of baking images are captured per unit time;

[0107] Or, a single frame baking image of the oven is acquired every 0.5s, and a total of 120 frames of baking images are captured per unit time;

[0108] The baking processing module includes a prediction module, an analysis module and a judgment module;

[0109] The analysis module is used to process the image data of the baking images of different frames. The image data processing is to delete the redundant image data, obtain the image data matrix of different unit time based on the multiple baking images of different unit time, and import the image data matrix of different unit time into the CNN-LSTM hybrid model one by one;

[0110] The prediction module is used to build a CNN-LSTM hybrid model, and the predicted baking time can be obtained through the CNN-LSTM hybrid model;

[0111] The judgment module is used to compare the predicted baking time with the preset time and make corresponding operations, including:

[0112] If the predicted baking time is higher than the preset time, the oven continues to operate to bake the food 102 until the baking time reaches the predicted baking time;

[0113] If the predicted baking time is equal to the preset time, the oven continues to operate to bake the food 102 until the baking time reaches the preset time;

[0114] If the predicted baking time is less than the preset time, the oven continues to run to bake the food 102 until the redundant time is reached. The redundant time is the time that the oven extends the baking time of the food 102 inside. By setting the redundant time, it can prevent the situation where the food 102 is not fully cooked when the oven finishes baking.

[0115] An oven control system based on food image proposed in an embodiment of the present invention first obtains preset information based on a preset module, the preset information includes a preset time for food 102 to be converted from raw food to cooked food, a CNN-LSTM hybrid model is constructed based on a prediction module, and an internal area 101 of the oven is divided into multiple sub-areas by the CNN-LSTM hybrid model, and multiple frames of baking images of different unit times are obtained based on an acquisition module, and image data processing is performed on the baking images based on an analysis module, and image data matrices of different unit times are obtained. By processing the image data of the baking images, image feature data in the image data matrix can be made more convenient for subsequent calculations, and then the image data matrices are imported into the CNN-LSTM hybrid model one by one, and iterative calculations are performed by the CNN-LSTM hybrid model. When the preset conditions are met, the result CNN-LSTM hybrid model is exported, and then The image data matrix of the current unit time is imported into the result CNN-LSTM hybrid model, and the predicted baking time is obtained based on the result CNN-LSTM hybrid model. The predicted baking time is compared and judged with the preset time based on the judgment module. When the predicted baking time is higher than the preset time, the oven runs until the baking time reaches the predicted baking time. When the predicted baking time is equal to the preset time, the oven runs until the preset time is reached. When the predicted baking time is less than the preset time, the oven runs until the redundant time is reached. The oven can predict the baking time based on the food image and the CNN-LSTM hybrid model, and adjust the baking time accordingly, which can prevent the oven from over-baking or poorly baking the food 102, improve the baking effect of the food 102, and solve the problem that traditional ovens often do not have the function of self-adjusting the baking time, resulting in over-baking or poorly baking the food 102 in the oven.

[0116] Embodiment 2:

[0117] Based on the food image-based oven control method and system provided in Example 1 of the present application, Example 2 of the present application proposes an food image-based oven control method and system. This Example 2 is merely a preferred method of Example 1, and the implementation of Example 2 will not affect the independent implementation of Example 1. The second Example 2 of the present invention will be further described below.

[0118] Please refer to the attached Figure 4 As shown, performing image data processing on the baking image to obtain image data matrices of different unit times includes the following steps:

[0119] S301: acquiring a single frame baking image inside the oven every 0.5 seconds, and capturing a total of 120 frames of baking images per unit time;

[0120] S302: preprocessing each frame of the baking image, the preprocessing is grayscale processing of each frame of the baking image, converting each frame of the color baking image into a black and white baking image, and applying Gaussian filtering to remove noise from each frame of the baking image;

[0121] S303: Obtaining comparison parameters of each frame of the baking image;

[0122] It can be understood that the grayscale value of each frame of the baking image is obtained, and the comparison parameter of each frame of the baking image is obtained based on the grayscale value. The comparison parameter calculation formula is:

[0123]

[0124] Where L is the width of each baking image frame, H is the width of each baking image frame, GR is the grayscale value of each baking image frame, MGR is the average brightness of each baking image frame, Bake the image comparison parameters for each frame;

[0125] Compare and judge each frame of baking image based on the comparison parameters and the preset brightness threshold:

[0126] If the comparison parameter is lower than the preset brightness threshold, the baking image corresponding to the comparison parameter is deleted. When the comparison parameter is lower than the preset brightness threshold, it means that the definition of the baking image is not good, and it is deleted;

[0127] If the comparison parameter is higher than or equal to the preset brightness threshold, the baking image corresponding to the comparison parameter is retained. When the comparison parameter is higher than the preset brightness threshold, it means that the definition of the baking image is better and it is retained.

[0128] Furthermore, the number of the retained baked images is set between 40 and 60 to prevent the decrease of calculation accuracy due to too little data, or the excessive complexity of calculation due to too much data.

[0129] S304: establishing a data plane for one of the retained baking images, acquiring a plurality of plane regions of equal area based on the data plane, marking the plurality of plane regions, and acquiring a feature data vector of each frame of the retained baking image;

[0130] S305: establishing a plurality of cluster centers in different plane regions on the data plane based on a cluster processing algorithm. When the cluster processing algorithm establishes the initial cluster centers, the initial cluster centers are established in different plane regions at equal intervals, and different plane regions are used as constraint conditions for updating the positions of the cluster centers for iterative calculation. Based on the iterative calculation, updated cluster centers are obtained, and a plurality of feature data vectors are assigned to clusters of each cluster center. A plurality of different clusters are matched with a plurality of different region labels, and different clusters include different image data features. Different image data features are matched with different region labels, and an image feature matrix is ​​established based on the plurality of different clusters.

[0131] Compared with the method of establishing the image data matrix by using image cutting, edge detection algorithm and block data array in the first embodiment, the second embodiment establishes the image data matrix by using comparison parameters and clustering processing algorithm. The main difference is that the baking image is compared and judged by comparing parameters and preset brightness thresholds to filter images, which can reduce the amount of data and calculation of the filtered images. The image data matrix is ​​established based on the clustering processing algorithm, which can establish the image data matrix more quickly. The remaining conditions are consistent with those in the first embodiment, so they are not repeated in this embodiment.

[0132] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by a wireless (such as infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0133] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0134] It should be understood that in the embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0135] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An oven control method based on food image, characterized in that: The following steps are included: S101: After the oven starts baking, preset information is obtained, the preset information including a preset time for the food (102) to be converted from raw food to cooked food, a CNN-LSTM hybrid model is constructed, and based on the CNN-LSTM hybrid model, an internal area (101) of the oven is divided into multiple sub-areas; S102: Acquire multiple frames of baking images at different unit times, where the unit time is 1 minute, and the baking images are real-time images of baking inside the oven. Perform image data processing on the baking images at different frames, and acquire image data matrices at different unit times based on the multiple frames of baking images at different unit times. The image data matrix is: Wherein, a, b, c...z are image feature data, m is the mth image feature data in different rows in the image data matrix, x is the region mark matching the image feature data in the row of the image data matrix, and y is the time corresponding to the image feature data in the column of the image data matrix; S103: importing the image data matrices of different unit times into the CNN-LSTM hybrid model one by one, and matching the image feature data in the image data matrices of different unit times with the plurality of sub-regions one by one based on the CNN-LSTM hybrid model; S104: performing iterative calculation based on the CNN-LSTM hybrid model, and when the calculation result meets a preset condition, exporting the CNN-LSTM hybrid model as a result CNN-LSTM hybrid model; S105: importing the image data matrix of the current unit time into the result CNN-LSTM hybrid model, and obtaining a predicted baking time based on the result CNN-LSTM hybrid model, wherein the predicted baking time is a predicted time for the food (102) to be converted from raw food to cooked food predicted by the result CNN-LSTM hybrid model; S106: Compare and judge the predicted baking time with the preset time, and perform corresponding operations, including: S1061: If the predicted baking time is higher than the preset time, the oven continues to operate to bake the food (102) until the baking time reaches the predicted baking time; S1062: If the predicted baking time is equal to the preset time, the oven continues to operate to bake the food (102) until the baking time reaches the preset time; S1063: If the predicted baking time is less than the preset time, the oven continues to operate to bake the food (102) until the redundant time is reached.

2. The oven control method based on food image according to claim 1, characterized in that: Acquire multiple frames of baking images at different unit times, the unit time being 1 minute, the baking images being real-time images of baking inside the oven, perform image data processing on the baking images at different frames, and acquire image data matrices at different unit times based on the multiple frames of baking images at different unit times, including: S201: acquiring a single frame of the baking image inside the oven every 1 second, and capturing a total of 60 frames of the baking image per unit time; S202: performing equal-area cutting on the baking image of each frame, cutting the baking image of each frame into a plurality of block images, dividing the plurality of block images into different regions, and marking the different regions; S203: grayscale processing is performed on the multiple block images of each frame, the multiple color block images of each frame are converted into multiple black and white block images, and Gaussian filtering is applied to remove noise from the multiple block images of each frame; S204: Based on the multiple block images of each frame, a plurality of image feature data of each frame are obtained, the multiple image feature data of each frame are matched with different region marks, timestamps are marked on the multiple image feature data based on different frames, and the multiple image feature data of the same frame are marked with the same timestamp, multiple block data arrays are established based on different region marks and the image feature data of different frames, region markings are performed on the multiple block data arrays based on the regions where the block images are located, and the multiple image feature data in the multiple block data arrays are arranged in positive order according to the corresponding timestamps. The block data arrays of different frames are: Wherein, A…Z are multiple block data arrays marked with different regions, a, b, c…z are the image feature data in the multiple block data arrays, and n is the number of image feature data contained in different block data arrays; S205: Screening the plurality of block data arrays based on an edge detection algorithm: If the image feature data in the block data array includes any food feature data, the block data array is retained; If the image feature data in the block data array does not include any food feature data, deleting the block data array; S206: Establishing the image data matrix based on the retained plurality of block data arrays.

3. The oven control method based on food image according to claim 2, characterized in that: Screening the plurality of block data arrays based on an edge detection algorithm includes: A plurality of food feature regions are obtained based on an edge detection algorithm, and a plurality of food feature data are obtained based on the plurality of food feature regions.

4. The oven control method based on food image according to claim 1, characterized in that: Acquire multiple frames of baking images at different unit times, the unit time being 1 minute, the baking images being real-time images of baking inside the oven, perform image data processing on the baking images at different frames, and acquire image data matrices at different unit times based on the multiple frames of baking images at different unit times, including: S301: acquiring a single frame of the baking image inside the oven every 0.5 seconds, and capturing a total of 120 frames of the baking image per unit time; S302: preprocessing each frame of the baking image, wherein the preprocessing includes grayscale processing of each frame of the baking image, converting each frame of the color baking image into a black-and-white baking image, and applying Gaussian filtering to remove noise from each frame of the baking image; S303: Obtaining comparison parameters of each frame of the baking image, and comparing and judging each frame of the baking image based on the comparison parameters and a preset brightness threshold: If the comparison parameter is lower than the preset brightness threshold, deleting the baking image corresponding to the comparison parameter; If the comparison parameter is higher than or equal to the preset brightness threshold, retaining the baking image corresponding to the comparison parameter; S304: establishing a data plane for one of the retained baking images, acquiring a plurality of plane regions of equal area based on the data plane, marking the plurality of plane regions, and acquiring a feature data vector of each frame of the retained baking image; S305: Based on a clustering processing algorithm, a plurality of cluster centers located in different plane areas are established on the data plane, a plurality of feature data vectors are assigned to clusters of each cluster center, a plurality of different clusters are matched with a plurality of different area labels, and the image feature matrix is ​​established based on the plurality of different clusters.

5. The oven control method based on food image according to claim 4, characterized in that: Acquiring comparison parameters of each frame of the baking image, and comparing and judging each frame of the baking image based on the comparison parameters and a preset brightness threshold, including: The grayscale value of each frame of the baking image is obtained, and the comparison parameter of each frame of the baking image is obtained based on the grayscale value. The calculation formula of the comparison parameter is: Wherein, L is the width of each frame of the baking image, H is the width of each frame of the baking image, GR is the grayscale value of each frame of the baking image, MGR is the average brightness of each frame of the baking image, is the comparison parameter of the baked image for each frame.

6. The oven control method based on food image according to claim 1, characterized in that: Iterative calculation is performed based on the CNN-LSTM hybrid model, and when the calculation result meets the preset conditions, the CNN-LSTM hybrid model is exported as a result CNN-LSTM hybrid model, including: Based on the CNN-LSTM hybrid model, multiple sub-regions that fail to match the image feature data are marked as vacant, and based on the CNN-LSTM hybrid model, multiple peripheral sub-regions that match the image feature data are marked as edges. When performing iterative calculations, the weights of the image feature data that match the edge-marked sub-regions are reset to γ, where γ is greater than 0 and less than 1.

7. The oven control method based on food image according to claim 1, characterized in that: If the predicted baking time is less than the preset time, the oven continues to operate to bake the food (102) until the redundant time is reached, including: The redundant time is the time that the oven prolongs the baking time of the food (102) inside. The calculation formula of the redundant time is: Where, T re is the redundant time, T ac is the preset time, T pr The predicted baking time is given.

8. An oven control system based on food image, characterized in that: include: A preset module, the preset module being used to obtain preset information, the preset information including a preset time for the food (102) to be converted from raw food to cooked food; An acquisition module is used to acquire multiple frames of baking images per unit time, where the unit time is 1 minute, and the baking images are real-time images of baking inside the oven, including: A single frame of the baking image inside the oven is acquired every 1 second, and a total of 60 frames of the baking image are captured per unit time; Or, a single frame of the baking image inside the oven is acquired every 0.5 s, and a total of 120 frames of the baking image are captured per unit time; The baking processing module includes a prediction module, an analysis module and a judgment module; An analysis module is used to perform image data processing on the baking images of different frames, wherein the image data processing is to delete redundant image data, obtain image data matrices of different unit times based on multiple frames of the baking images of different unit times, and import the image data matrices of different unit times into the CNN-LSTM hybrid model one by one; A prediction module, used to construct the CNN-LSTM hybrid model, and to obtain the predicted baking time through the CNN-LSTM hybrid model; The judgment module is used to compare and judge the predicted baking time with the preset time and perform corresponding operations, including: If the predicted baking time is higher than the preset time, the oven continues to operate to bake the food (102) until the baking time reaches the predicted baking time; If the predicted baking time is equal to the preset time, the oven continues to operate to bake the food (102) until the baking time reaches the preset time; If the predicted baking time is less than the preset time, the oven continues to operate to bake the food (102) until a redundant time is reached, wherein the redundant time is the time for the oven to extend the baking time of the food (102) inside.

Images