Method, apparatus, kitchen electric appliance and server for determining ingredient maturity

Abstract

The present application relates to the technical field of intelligent devices, and discloses a method, a device, a kitchen electric appliance, and a server for determining the maturity of food ingredients. The method includes: obtaining a current frame image of the food ingredient being cooked in the electric appliance and a starting frame image, where the starting frame image is an image of the food ingredient when cooking is started; performing fusion processing on the current frame image and the starting frame image to obtain a current fusion processed image; and inputting the current fusion processed image into a configured maturity machine learning training algorithm model to determine the maturity corresponding to the food ingredient. In this way, by extracting the features of the fusion processed image after double-image fusion and model prediction, the reliability of determining the maturity of the food ingredient is improved.

Description

Technical Field

[0001] This application relates to the technical field of intelligent devices, for example, methods, devices, kitchen appliances, and servers for determining the ripeness of food ingredients. Background Art

[0002] With the progress of science and technology and the development of artificial intelligence, intelligent algorithms are increasingly applied to intelligent household appliances, such as refrigerators, air conditioners, ovens, etc. Among them, kitchen appliances such as ovens, microwave ovens, and air fryers can intelligently determine cooking time, cooking power, etc. according to information such as the type and weight of food ingredients.

[0003] Currently, for some baking kitchen appliances, the ripeness of food ingredients can also be judged. After detecting the temperature inside the food ingredients through a temperature sensor, the ripeness of the food ingredients can be judged according to the temperature. However, during the baking process, the temperature of the food ingredients is unstable, making the ripeness judgment not reliable enough. Moreover, detecting the temperature through a contact sensor may damage the appearance of the food ingredients and may also contaminate the food ingredients. Summary of the Invention

[0004] To provide a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a comprehensive review, nor is it intended to identify key / important elements or delineate the scope of protection of these embodiments. Instead, it serves as a preface to the following detailed description.

[0005] Embodiments of the present disclosure provide a method, device, kitchen appliance, and server for determining the ripeness of food ingredients to solve the technical problem of low reliability in determining the ripeness of food ingredients.

[0006] In some embodiments, the method includes:

[0007] Obtain a current frame image of the food ingredient being cooked and a starting frame image in the electric device, where the starting frame image is an image of the food ingredient when cooking starts;

[0008] Fuse the current frame image and the starting frame image to obtain a current fused image;

[0009] Input the current fused image into a configured ripeness machine learning training algorithm model to determine the ripeness corresponding to the food ingredient.

[0010] In some embodiments, the device includes:

[0011] An information acquisition module configured to obtain a current frame image of the food ingredient being cooked and a starting frame image in the kitchen appliance, where the starting frame image is an image of the food ingredient when cooking starts;

[0012] A feature extraction module, configured to perform a fusion process on the current frame image and the starting frame image to obtain a current fusion processed image;

[0013] A prediction determination module, configured to input the current fusion processed image into a configured maturity machine learning training algorithm model to determine the maturity of the food ingredient.

[0014] In some embodiments, the kitchen electrical appliance includes: the device for determining the maturity of the food ingredient as described above.

[0015] In some embodiments, the server includes: the device for determining the maturity of the food ingredient as described above.

[0016] The method, device, kitchen electrical appliance, and server for determining the maturity of a food ingredient provided by the embodiments of the present disclosure can achieve the following technical effects:

[0017] Obtain the current frame image and the starting frame image during the cooking process of the food ingredient, perform a fusion process on the dual images to obtain the corresponding current fusion processed image, and input it into a configured maturity machine learning training algorithm model to determine the maturity of the food ingredient. In this way, through feature extraction and model prediction of the fusion processed image after fusing the dual images, according to the difference degree between the dual images in the fusion processed image, the maturity of the food ingredient is determined, improving the reliability of determining the maturity of the food ingredient. Moreover, it is not necessary to contact the food ingredient to determine the reliability, reducing the probability of damaging the appearance of the food ingredient and contaminating the food ingredient.

[0018] The above general description and the following description are only exemplary and explanatory, and are not used to limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] One or more embodiments are exemplarily illustrated by corresponding drawings. These exemplary illustrations and the drawings do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation, and among them:

[0020] Figure 1 is a flowchart of a method for determining the maturity of a food ingredient provided by an embodiment of the present disclosure;

[0021] Figure 1-1 is a schematic diagram of a fusion processed image provided by an embodiment of the present disclosure;

[0022] Figure 2 is a flowchart of a method for determining the maturity of a food ingredient provided by an embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of the structure of a system for determining the maturity of a food ingredient provided by an embodiment of the present disclosure;

[0024] Figure 4 It is a schematic flowchart of a method for determining the maturity of food materials provided by an embodiment of the present disclosure;

[0025] Figure 5 It is a schematic structural diagram of a device for determining the maturity of food materials provided by an embodiment of the present disclosure;

[0026] Figure 6 It is a schematic structural diagram of a device for determining the maturity of food materials provided by an embodiment of the present disclosure;

[0027] Figure 7 It is a schematic structural diagram of a device for determining the maturity of food materials provided by an embodiment of the present disclosure;

[0028] Figure 8 It is a schematic structural diagram of a device for determining the maturity of food materials provided by an embodiment of the present disclosure. Detailed implementation manners

[0029] In order to be able to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are only for reference and illustration, and are not used to limit the embodiments of the present disclosure. In the following technical description, for the sake of explanation, numerous details are provided to give a thorough understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be shown in a simplified manner to simplify the drawings.

[0030] In the embodiments of the present disclosure, the current frame image and the starting frame image during the cooking process of the food material are obtained, the two images are fused to obtain the corresponding current fused image, and the fused image is input into the configured maturity machine learning training algorithm model to determine the maturity of the food material. In this way, by extracting the features of the fused image after the fusion of the two images and model prediction, the maturity of the food material is determined according to the difference degree between the two images in the fused image, which improves the reliability of determining the maturity of the food material. Moreover, when configuring the maturity machine learning training algorithm model, through the fusion processing of the two images, the problem of non-convergence of the maturity regression network caused by differences in the shape, color, texture, etc. of different food materials can be effectively avoided. In addition, it is not necessary to contact the food material to determine the reliability, which reduces the probability of damaging the appearance of the food material and contaminating the food material.

[0031] Figure 1 It is a schematic flowchart of a method for determining the maturity of food materials provided by an embodiment of the present disclosure. As Figure 1 shown, the process of determining the maturity of the food material includes:

[0032] Step 101: Obtain the current frame image and the starting frame image of the food material being cooked in the kitchen appliance.

[0033] In the embodiments of the present disclosure, the kitchen electric appliance may include: an oven, a microwave oven, an air fryer, etc., which are enclosed appliances capable of baking food materials. Generally, an image acquisition device may be configured in the kitchen electric appliance. For example, a camera is configured at the top, and image or video information in the kitchen electric appliance can be acquired through the image acquisition device.

[0034] The image acquisition device can be started regularly to obtain images in the kitchen electric appliance or record video information for a set time. Alternatively, the image acquisition device can be turned on or off under the control of a trigger signal, so as to obtain corresponding images or video information. The trigger signal can be sent by the kitchen electric appliance, or sent by the server through the kitchen electric appliance. For example, when it is determined that the kitchen electric appliance starts cooking, the kitchen electric appliance or the server sends a start instruction to start the configured image acquisition device to collect video information; and when it is determined that the door in the kitchen electric appliance is opened, or when the set time is reached, the kitchen electric appliance or the server sends a shutdown instruction to turn off the image acquisition device and stop collecting video information.

[0035] Therefore, in some embodiments, when this method is applied to a kitchen electric appliance, obtaining the current frame image and the starting frame image of the food material being cooked in the kitchen electric appliance may include: when cooking starts, controlling the camera to start recording a cooking video, obtaining and saving the starting frame image corresponding to the food material, and obtaining the current frame image of the food material through the camera.

[0036] In some embodiments, a server communicating with the kitchen electric appliance is used to obtain the current frame image and the starting frame image of the food material in the kitchen electric appliance. That is, when this method is applied to the server, the current frame image of the food material sent by the kitchen electric appliance and the saved starting frame image of the food material are received.

[0037] Step 102: Perform fusion processing on the current frame image and the starting frame image to obtain a current fusion processed image.

[0038] In this embodiment, it is necessary to perform dual-image fusion processing on the current frame image and the starting frame image to fuse them into one frame image. In some embodiments, it may include: the current frame image and the starting frame image can be fused by arranging pixel rows of the two images alternately to obtain a current fusion processed image. Alternatively, the current frame image and the starting frame image are fused by splicing the two images vertically to obtain a current fusion processed image.

[0039] Figure 1-1 It is a schematic diagram of a fusion processed image provided by an embodiment of the present disclosure. In this embodiment, the kitchen electric appliance can be an oven, and the food material can be a cake. The current frame image of the cake during the process of baking the cake in the oven is obtained. Then, the current frame image and the starting frame image corresponding to when the cake is placed in the oven and started are spliced vertically to obtain, as shown in...Figure 1-1 The current fused processed image shown. Of course, the present disclosure is not limited to this, and other methods that can fuse two frames of images into one frame of image can also be applied here.

[0040] Step 103: Input the current fused processed image into the configured maturity machine learning training algorithm model to determine the maturity of the ingredients.

[0041] Convolutional Neural Networks (CNN) can automatically learn the features of an image at various levels through convolution and pooling operations, which conforms to the common sense of people's understanding of images. When people recognize an image, they abstract it layer by layer. First, they understand the color and brightness, then the local detail features such as edges, corners, and straight lines, followed by more complex information and structures such as textures and geometric shapes, and finally form the concept of the entire object. Each convolutional layer contains multiple convolutional kernels. These convolutional kernels scan the entire image from left to right and top to bottom in sequence to obtain the output data called the feature map, that is, the image feature information. In a convolutional neural network, the previous convolutional layers capture the local and detailed information of the image and have a small receptive field, that is, each pixel of the output image only utilizes a very small range of the input image; while the receptive field of the subsequent convolutional layers increases layer by layer to capture more complex and abstract information of the image. After the operations of multiple convolutional layers, the abstract representations of the image at various different scales are finally obtained.

[0042] In a convolutional neural network, each convolutional layer corresponding to a convolutional kernel is actually a system, a system for judging a certain feature in an image. When all individual convolutional layers, that is, individual feature judgment systems, can effectively complete the task of feature judgment, the complex system CNN composed of these numerous convolutional layers can complete the complex tasks required by humans for the convolutional neural network.

[0043] Therefore, in the embodiments of the present disclosure, based on the convolutional neural network CNN and through a large amount of sample data, a maturity machine learning training algorithm model is configured.

[0044] In some embodiments, the configuration process of the maturity machine learning training algorithm model includes: based on the convolutional neural network CNN, extracting the feature image information of multiple sample images, where the sample images are generated after fusing the ingredient images with calibrated maturity and the corresponding starting ingredient images; and through a regression network, performing supervised training on each piece of feature image information to generate the maturity machine learning training algorithm model.

[0045] The maturity of the food ingredient images has been calibrated, and each food ingredient image corresponds to the starting image of the food ingredient. Thus, the fusion processing of the two images is also required. That is, the food ingredient image with calibrated maturity can be fused with the corresponding starting image of the food ingredient by interleaving the pixel rows of the two images or by splicing the two images vertically to obtain the fused processed image, i.e., the sample image. Then, based on the convolutional neural network CNN, the feature image information of the image can be extracted, and the feature image information can be supervised and trained through the regression network to generate the machine learning training algorithm model for maturity.

[0046] After the two images are fused, when extracting the feature image information of the fused processed image based on the convolutional neural network CNN, the convolutional operation well considers the positional relationship between pixels in the two images. Thus, the result of judging the food maturity can fuse the information between the food ingredient image and the starting image and measure the difference degree between the two images. In this way, the extracted feature information utilizes the difference degree between the images. If the difference between the food ingredient image and the starting image is greater, then its maturity value is higher; otherwise, it is lower.

[0047] Moreover, when the feature image information of the fused processed image after the two-image fusion is supervised and trained through the regression network, the non-convergence of the maturity regression network caused by "the differences in the shapes, colors, textures, etc. of different food ingredients" can be effectively avoided. Among them, the differences between the same foods with different maturities are amplified; at the same time, the differences between different foods with the same maturity are reduced. In this way, the characteristics for judging the food maturity are more obvious, making it easier for the network to converge.

[0048] In some embodiments, to make the convergence of the regression network for supervised training better, the loss function corresponding to formula (1) can be adopted:

[0049]

[0050] where ε is the convergence coefficient, and Δx is the difference between the predicted value and the calibrated value.

[0051] In this embodiment, the regression network is used to predict the maturity of the food ingredient. Therefore, the predicted value can be the predicted maturity, and the calibrated value is the calibrated maturity. Currently, there is no convergence coefficient ε in the loss function adopted by the regression network. In this way, when the absolute value of Δx is greater than 1, it can converge faster, and a single linear derivative can make the network converge quickly; while when the absolute value of Δx is less than 1, the original smooth L1 is not sensitive to outliers and abnormal values, and the gradient change is relatively smaller, and it is not easy to go astray during training. However, the resulting problem is slow convergence, especially for large networks and large-sample training. Therefore, in this embodiment, it can be smooth L1Plus(Δx) introduces ε, where ε is a dynamic coefficient. When the absolute value of Δx is less than 1, ε can be adjusted to ensure that when the training converges, the gradient of the mean absolute error (MAE, also known as the L1 loss) is not too small, and at the same time, the mean square error (MSE, also known as the L2 loss) can prevent the network from diverging.

[0052] In the process of configuring a machine learning training algorithm model for maturity, the sample images are fused with the ingredient images with calibrated maturity, that is, each ingredient image has a corresponding calibrated maturity. However, the calibration of maturity is a difficult problem. Generally, only very experienced cooks can accurately judge the doneness of a certain food among various foods, and this is also a prerequisite for the supervised network training of deep learning. In some embodiments, the process of calibrating maturity includes: when the cooking of the ingredient starts, controlling the camera to start recording the cooking video, and when the maturity of the ingredient reaches the set maturity, controlling the camera to stop recording the cooking video; determining the total number of frames of the recorded cooking video; and determining the maturity corresponding to the first image according to the frame number corresponding to the first image, the total number of frames, and the set maturity.

[0053] Among them, the process of determining that the maturity of the ingredient reaches the set maturity can be determined by a cook observing the ingredient. Of course, it is not limited to this, and it can also be determined by image comparison, etc.

[0054] For example: when the ingredient to be calibrated is placed in the oven, start recording the video, and observe the critical state of the food baking and roasting manually. When it is determined that it has reached the state, stop recording the video. Thus, the total number of frames nall of the video can be obtained, and the formula for the doneness of the nth frame can be doneness = n / nall.

[0055] After configuring the machine learning training algorithm model for maturity, the currently fused processed image after the fusion process in step 102 can be input. In this way, based on the convolutional neural network CNN, the current image feature information of the currently fused processed image is extracted, and through the regression network, the current image feature information is supervised and trained to predict the maturity of the ingredient, that is, through the model for prediction, the prediction result is obtained, that is, the maturity corresponding to the ingredient is determined.

[0056] It can be seen that in the embodiments of the present disclosure, the current frame image and the starting frame image during the cooking process of the food material are obtained, the dual images are fused, and the fused image after processing is input into the configured maturity machine learning training algorithm model to determine the maturity of the food material. In this way, by using the fused image that fuses the features of the dual images for model prediction, and according to the difference degree between the dual images in the fused image, the maturity of the food material is determined, which improves the reliability of determining the maturity of the food material. Moreover, when configuring the maturity machine learning training algorithm model, through the dual-image fusion processing, the problem of non-convergence of the maturity regression network caused by differences in the shape, color, texture, etc. of different food materials can be effectively avoided. In addition, it is not necessary to contact the food material to determine the reliability, reducing the probability of damaging the appearance of the food material and contaminating the food material.

[0057] In the embodiments of the present disclosure, the kitchen electric appliance can determine the maturity of the food material locally, or send the current frame image and the starting frame image to the server, so that the server determines the maturity of the food material. Therefore, in some embodiments, when this method is applied to a kitchen electric appliance, after determining the corresponding maturity of the food material, information reminder of the maturity can also be performed. It may include at least one of the following methods: displaying the maturity information on the display screen of the kitchen electric appliance, playing the maturity information through a voice broadcast device; sending the maturity information to the terminal for display and reminder. In this way, the kitchen electric appliance determines the maturity of the food material locally, which improves the speed of determining the maturity and also reduces the occupation of network resources.

[0058] In some embodiments, when this method is applied to the server, after determining the corresponding maturity of the food material, the maturity is sent to the kitchen electric appliance for information reminder of the maturity. The specific reminder method of the kitchen electric appliance can be as described above. In this embodiment, the server determines the maturity of the food material, which reduces the occupation of the resources of the kitchen electric appliance, saves memory, and improves the multi-control function of the kitchen electric appliance.

[0059] Next, the operation process will be incorporated into specific embodiments to illustrate the process of determining the maturity of the food material provided by the embodiments of the present invention.

[0060] In an embodiment of the present disclosure, the kitchen electric appliance can be an oven, and a camera is configured on the top of the oven. Moreover, the kitchen electric appliance has configured a maturity machine learning training algorithm model through machine learning.

[0061] Figure 2 It is a schematic flowchart of a method for determining the maturity of a food material provided by the embodiments of the present disclosure. As Figure 2 shown, the process of determining the maturity of the food material includes:

[0062] Step 201: Obtain the current frame image of the food material being cooked in the oven.

[0063] Step 202: Determine whether the current frame image is the starting frame image? If so, execute Step 203; otherwise, execute Step 204.

[0064] Step 203: Save the current frame image as the starting frame image and return to Step 201.

[0065] Step 204: By interleaving the pixel rows of the two images, fuse the current frame image and the starting frame image to obtain the current fused image.

[0066] Step 205: Input the current fused image into the configured maturity machine learning training algorithm model to determine the maturity of the food ingredient.

[0067] Step 206: Display the maturity information on the display screen of the oven.

[0068] It can be seen that in this embodiment, the oven can obtain the current frame image and the starting frame image during the baking process of the food ingredient, fuse the two images to obtain the corresponding current fused image, and input it into the configured maturity machine learning training algorithm model to determine the maturity of the food ingredient. In this way, by extracting the features of the fused image after fusing the two images and making model predictions, and based on the difference degree between the two images in the fused image, the maturity of the food ingredient is determined, which improves the reliability of determining the maturity of the food ingredient. Moreover, it is not necessary to contact the food ingredient to determine the reliability, reducing the probability of damaging the appearance of the food ingredient and contaminating the food ingredient. In addition, the oven determines the maturity of the food ingredient locally, which improves the speed of determining the maturity and also reduces the occupation of network resources.

[0069] In an embodiment of the present disclosure, the maturity of the food ingredient can be determined by the server.

[0070] Figure 3 is a schematic structural diagram of a food ingredient maturity determination system provided by an embodiment of the present disclosure. As Figure 3 shown, the system includes: a kitchen electric appliance 100, an image acquisition device 200 configured on the kitchen electric appliance, and a server 300 communicating with the kitchen electric appliance 100.

[0071] Among them, the kitchen electric appliance 100 can obtain the current frame image and the starting frame image of the food ingredient being cooked in the kitchen electric appliance through the image acquisition device 200 and send them to the server 300. The server 300 is configured with a maturity machine learning training algorithm model through machine learning. Thus, the server 300 can perform feature extraction and prediction on the current frame image and the current fused image after fusing the starting frame image based on the maturity machine learning training algorithm model to obtain the maturity of the food ingredient, and can send the maturity to the kitchen electric appliance 100 for prompt processing.

[0072] Figure 4 is a schematic flowchart of a method for determining the maturity of food materials provided by an embodiment of the present disclosure. The food material maturity determination system can be as follows Figure 3 , such as Figure 4 shown, the process of determining the maturity of food materials includes:

[0073] Step 401: The kitchen electric appliance device acquires the current frame image of the food material being cooked in the kitchen electric appliance device through the image acquisition device.

[0074] Step 402: The kitchen electric appliance device determines whether the current frame image is the starting frame image? If so, execute Step 403, otherwise, execute Step 404.

[0075] Step 403: The kitchen electric appliance device saves the current frame image as the starting frame image and returns to Step 401.

[0076] Step 404: The kitchen electric appliance device sends the current frame image and the starting frame image to the server.

[0077] Step 405: The server fuses the current frame image and the starting frame image by the method of splicing the two images up and down to obtain the current fusion processed image.

[0078] Step 406: The server inputs the current fusion processed image into the configured maturity machine learning training algorithm model to determine the maturity of the food material.

[0079] Step 407: The server sends the maturity to the kitchen electric appliance device.

[0080] Step 408: The kitchen electric appliance device displays the maturity information on the display screen and, in the case where the maturity is greater than the set value, performs voice broadcast.

[0081] It can be seen that in this embodiment, the server can obtain the current frame image and the starting frame image during the cooking process of the food material through the kitchen electric appliance device, perform fusion processing on the two images to obtain the corresponding current fusion processed image, and input it into the configured maturity machine learning training algorithm model to determine the maturity of the food material. In this way, through feature extraction and model prediction of the fusion processed image after fusing the two images, according to the difference degree between the two images in the fusion processed image, the maturity of the food material is determined, which improves the reliability of determining the maturity of the food material. Moreover, it is not necessary to contact the food material to determine the reliability, reducing the probability of damaging the appearance of the food material and contaminating the food material. In addition, the server determines the maturity of the food material, reducing the occupation of resources of the kitchen electric appliance device and improving the multi-control function of the kitchen electric appliance device.

[0082] According to the above process of determining the maturity of food materials, a device for determining the maturity of food materials can be constructed.

[0083] Figure 5It is a schematic structural diagram of a device for determining the maturity of food materials provided by an embodiment of the present disclosure. As Figure 5 shown, the device for determining the maturity of food materials includes: an information acquisition module 510, an image fusion module 520, and a prediction determination module 530.

[0084] The information acquisition module 510 is configured to acquire the current frame image of the food material being cooked in the kitchen appliance and the starting frame image, where the starting frame image is the image of the food material when the cooking starts.

[0085] The image fusion module 520 is configured to perform fusion processing on the current frame image and the starting frame image to obtain the current fusion processed image.

[0086] The prediction determination module 530 is configured to input the current fusion processed image into the configured maturity machine learning training algorithm model to determine the maturity corresponding to the food material.

[0087] In some embodiments, it further includes: a model configuration module, configured to extract the feature image information of multiple sample images based on a convolutional neural network (CNN), where the sample images are obtained by performing fusion processing on the food material images with calibrated maturity and the corresponding starting images of the food materials; and, through a regression network, performing supervised training on each piece of feature image information to generate a maturity machine learning training algorithm model; the loss function in the regression network includes:

[0088]

[0089] where ε is the convergence coefficient, and Δx is the difference between the predicted value and the calibrated value.

[0090] In some embodiments, it further includes: a calibration module, configured to control the camera to start recording the cooking video when the cooking of the food material starts, and control the camera to stop recording the cooking video when the maturity of the food material reaches the set maturity; determining the total number of frames of the recorded cooking video; and determining the maturity corresponding to the first image according to the frame number corresponding to the first image, the total number of frames, and the set maturity.

[0091] The following is an example to illustrate the device for determining the maturity of food materials provided by the embodiments of the present disclosure.

[0092] Figure 6 It is a schematic structural diagram of a device for determining the maturity of food materials provided by an embodiment of the present disclosure. As Figure 6 shown, the device for determining the maturity of food materials can be applied to a kitchen appliance, and includes: an information acquisition module 510, an image fusion module 520, a prediction determination module 530, and further includes a model configuration module 540, a calibration module 550, and a processing module 560.

[0093] Among them, the calibration module 550 can calibrate the maturity of the sample food materials, that is, when the cooking of the food materials is started, it controls the camera to start recording the cooking video, and when the maturity of the food materials reaches the set maturity, it controls the camera to stop recording the cooking video; determines the total number of frames of the recorded cooking video; and determines the maturity corresponding to the first image according to the frame number corresponding to the first image, the total number of frames, and the set maturity.

[0094] In this way, the model configuration module 540 can extract the feature image information of multiple sample images based on the convolutional neural network CNN, where the sample images are obtained after the fusion processing of the food material images with calibrated maturity and the corresponding starting images of the food materials; and, through the regression network, supervise and train each piece of feature image information to generate a maturity machine learning training algorithm model; the loss function in the regression network includes:

[0095]

[0096] Among them, ε is the convergence coefficient, and Δx is the difference between the predicted value and the calibrated value.

[0097] When the cooking is started, the camera starts recording the cooking video. The information acquisition module 510 can acquire and save the starting frame image corresponding to the food materials, and through the camera, the information acquisition module 510 can acquire the current frame image of the food materials.

[0098] In this way, through the image fusion module 520, the current frame image and the starting frame image can be fused to obtain the current fusion processed image.

[0099] The prediction determination module 530 can input the current fusion processed image into the maturity machine learning training algorithm model configured by the model configuration module 540 to determine the maturity corresponding to the food materials. The processing module 560 can display the maturity information on the display interface.

[0100] It can be seen that in this embodiment, the device for determining the maturity of food materials applied to the kitchen electric appliance can acquire the current frame image and the starting frame image during the cooking process of the food materials, fuse the two images to obtain the corresponding current fusion processed image, and input it into the configured maturity machine learning training algorithm model to determine the maturity of the food materials. In this way, through the feature extraction and model prediction of the fusion processed image after the fusion of the two images, according to the difference degree between the two images in the fusion processed image, the maturity of the food materials is determined, which improves the reliability of determining the maturity of the food materials. Moreover, it is not necessary to contact the food materials to determine the reliability, reducing the probability of damaging the appearance of the food materials and contaminating the food materials. In addition, the kitchen electric appliance locally determines the maturity of the food materials, which improves the speed of determining the maturity and also reduces the occupation of network resources.

[0101] Figure 7This is a schematic structural diagram of a device for determining the maturity of food materials provided by an embodiment of the present disclosure. As Figure 7 shown, the device for determining the maturity of food materials can be applied to a server, including: an information acquisition module 510, an image fusion module 520, a prediction determination module 530, and further including a model configuration module 540, a calibration module 550, and a sending module 570.

[0102] Among them, the calibration module 550 can calibrate the maturity of the sample food materials, that is, when the cooking of the food materials starts, control the camera to start recording the cooking video, and when the maturity of the food materials reaches the set maturity, control the camera to stop recording the cooking video; determine the total number of frames of the recorded cooking video; and determine the maturity corresponding to the first image according to the frame number corresponding to the first image, the total number of frames, and the set maturity.

[0103] In this way, the model configuration module 540 can extract the feature image information of multiple sample images based on the convolutional neural network CNN, where the sample images are obtained by fusing the food material images with calibrated maturity and the corresponding starting images of the food materials; and, through the regression network, perform supervised training on each piece of feature image information to generate a maturity machine learning training algorithm model; the loss function in the regression network includes:

[0104]

[0105] where ε is the convergence coefficient and Δx is the difference between the predicted value and the calibrated value.

[0106] When the cooking starts, the camera starts recording the cooking video, and the kitchen electrical appliance can obtain the starting frame image and the current frame image corresponding to the food materials. In this way, the information acquisition module 510 can receive the current frame image of the food materials sent by the kitchen electrical appliance and the saved starting frame image of the food materials.

[0107] Similarly, through the image fusion module 520, the current frame image and the starting frame image can be fused to obtain the current fusion processed image.

[0108] The prediction determination module 530 can input the current fusion processed image into the maturity machine learning training algorithm model configured by the model configuration module 540 to determine the maturity corresponding to the food materials. The sending module 570 can send the maturity to the kitchen electrical appliance for maturity information reminder.

[0109] It can be seen that in this embodiment, the device for determining the maturity of food materials applied to the server can obtain the current frame image and the starting frame image during the cooking process of the food materials through the kitchen electric appliance, perform fusion processing on the dual images to obtain the corresponding current fusion processed image, and input it into the configured maturity machine learning training algorithm model to determine the maturity of the food materials. In this way, by extracting the features of the fusion processed image after fusing the dual images and predicting with the model, according to the difference degree between the dual images in the fusion processed image, the maturity of the food materials is determined, which improves the reliability of determining the maturity of the food materials. Moreover, without contacting the food materials, the reliability can be determined, reducing the probability of damaging the appearance of the food materials and contaminating the food materials. In addition, the server determines the maturity of the food materials, reducing the occupation of resources of the kitchen electric appliance and improving the multi-control functions of the kitchen electric appliance.

[0110] An embodiment of the present disclosure provides a device for determining the maturity of food materials, and its structure is as Figure 8 shown, including:

[0111] A processor 1000 and a memory 1001, and may further include a communication interface 1002 and a bus 1003. Among them, the processor 1000, the communication interface 1002, and the memory 1001 can complete mutual communication through the bus 1003. The communication interface 102 can be used for information transmission. The processor 1000 can call the logical instructions in the memory 1001 to execute the method for determining the maturity of food materials in the above embodiment.

[0112] In addition, when the logical instructions in the above-mentioned memory 1001 are implemented in the form of software function units and sold or used as an independent product, they can be stored in a computer-readable storage medium.

[0113] The memory 1001, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of the present disclosure. The processor 1000 executes functional applications and data processing by running the program instructions / modules stored in the memory 1001, that is, implements the method for determining the maturity of food materials in the above method embodiments.

[0114] The memory 1001 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 101 may include a high-speed random access memory and may also include a non-volatile memory.

[0115] An embodiment of the present disclosure provides a kitchen electric appliance including the above-mentioned device for determining the maturity of food materials.

[0116] An embodiment of the present disclosure provides a server, which includes the above-mentioned food ingredient maturity determination device.

[0117] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, and the computer-executable instructions are configured to execute the above-mentioned food ingredient maturity determination method.

[0118] An embodiment of the present disclosure provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer is caused to execute the above-mentioned food ingredient maturity determination method.

[0119] The above-mentioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transient computer-readable storage medium.

[0120] The technical solution of an embodiment of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The foregoing storage medium may be a non-transient storage medium, including: various media capable of storing program codes such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, or may also be a transient storage medium.

[0121] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure, enabling those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, process, and other changes. The embodiments merely represent possible variations. Unless explicitly required, individual components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of the present disclosure includes the entire scope of the claims and all available equivalents of the claims. When used in this application, although terms such as "first", "second", etc. may be used in this application to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without changing the meaning of the description, the first element may be called the second element, and similarly, the second element may be called the first element, as long as all occurrences of "the first element" are consistently renamed and all occurrences of "the second element" are consistently renamed. The first element and the second element are both elements, but they may not be the same element. Moreover, the terms used in this application are only used to describe the embodiments and do not limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to also include the plural forms. Similarly, as used in this application, the term "and / or" refers to any and all possible combinations including one or more of the associated listed items. Additionally, when used in this application, the term "comprise" and its variants "comprises" and / or "comprising" etc. mean the presence of the stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups of these. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, or device including the element. In this document, each embodiment may focus on the differences from other embodiments, and the same or similar parts between the embodiments may be referred to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, the relevant parts may refer to the description of the method part.

[0122] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner may depend on the specific application and design constraints of the technical solution. The skilled person can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the embodiments of the present disclosure. The skilled person can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0123] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units can be merely a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Additionally, the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to implement this embodiment. Additionally, in the embodiments of the present disclosure, the various functional units can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. Each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for determining the maturity of food ingredients, characterized in that, Including: Obtain the current frame image of the food ingredient being cooked in the kitchen appliance and the starting frame image, where the starting frame image is the image of the food ingredient when cooking is started; Perform fusion processing on the current frame image and the starting frame image to obtain a current fusion processed image; Input the current fusion processed image into a configured maturity machine learning training algorithm model to determine the maturity corresponding to the food ingredient; Wherein, before obtaining the current frame image of the food ingredient being cooked in the kitchen appliance and the starting frame image, it further includes: Based on a convolutional neural network (CNN), extract the feature image information of multiple sample images, where the sample images are obtained after fusion processing of the food ingredient images with calibrated maturity and the corresponding starting food ingredient images; Through a regression network, perform supervised training on each piece of feature image information to generate the maturity machine learning training algorithm model, and the loss function in the regression network includes: Where ε is the convergence coefficient and Δx is the difference between the predicted value and the calibrated value.

2. The method according to claim 1, characterized in that, The calibration process of the maturity includes: When cooking of the food ingredient is started, control the camera to start recording the cooking video, and when the maturity of the food ingredient reaches the set maturity, control the camera to stop recording the cooking video; Determine the total number of frames of the recorded cooking video; According to the frame sequence number corresponding to the first image, the total number of frames, and the set maturity, determine the maturity corresponding to the first image.

3. The method according to claim 1, wherein The obtaining of the current frame image of the food ingredient being cooked in the kitchen appliance and the starting frame image includes: When the method is applied to a kitchen appliance, at the start of cooking, control the camera to start recording the cooking video, obtain and save the starting frame image corresponding to the food ingredient, and obtain the current frame image of the food ingredient through the camera; When the method is applied to a server, receive the current frame image of the food ingredient sent by the kitchen appliance and the saved starting frame image of the food ingredient.

4. The method according to claim 1 or 3, characterized in that, After determining the maturity corresponding to the food ingredient, it further includes: When the method is applied to a kitchen appliance, perform information reminder of the maturity; When the method is applied to a server, send the maturity to the kitchen appliance for information reminder of the maturity.

5. An apparatus for determining the maturity of an ingredient, characterized in that Including: An information acquisition module, configured to obtain the current frame image of the food ingredient being cooked in the kitchen appliance and the starting frame image, where the starting frame image is the image of the food ingredient when cooking is started; An image fusion module, configured to perform fusion processing on the current frame image and the starting frame image to obtain a current fusion processed image; A prediction determination module, configured to input the current fusion processed image into a configured maturity machine learning training algorithm model to determine the maturity corresponding to the food ingredient; It further includes: A model configuration module, configured to extract feature image information of multiple sample images based on a convolutional neural network (CNN), wherein the sample images are obtained by fusing a calibrated ingredient image with a corresponding starting ingredient image; and, through a regression network, performing supervised training on each piece of the feature image information to generate the maturity machine learning training algorithm model; the loss function in the regression network includes: where ε is a convergence coefficient and Δx is the difference between the predicted value and the calibrated value.

6. The device according to claim 5, characterized in that It further includes: A calibration module, configured to control the camera to start recording a cooking video when the ingredient cooking starts, and control the camera to stop recording the cooking video when the ingredient maturity reaches the set maturity; Determine the total number of frames of the recorded cooking video; according to the frame number corresponding to the first image, the total number of frames, and the set maturity, determine the maturity corresponding to the first image.

7. A kitchen electrical appliance, characterized in that, It includes the device according to any one of claims 5 to 6.

8. A server, characterized in that, It includes the device according to any one of claims 5 to 6.

Images