Cooking process control

Abstract

In an embodiment, a computer-implemented method (100) of controlling a cooking process implemented by a cooking device is described. The method comprises receiving (102): first image data corresponding to a view of a first time of a cooking process; and second image data corresponding to a view of the cooking process at a second time. The method further comprises determining (104): a first image parameter value from a portion of the first image data corresponding to the region of interest; a second image parameter value from a portion of second image data corresponding to the region of interest; and an indication of a cooking state change based on a comparison of the first image parameter value and the second image parameter value. The region of interest includes a first area mapped to a portion of the food visible in the view, and a second area mapped to a portion of the background of the food visible in the view. In response to the indication that the range specified for the cooking process is exceeded between the first time and the second time, the method further includes providing (106) a modified cooking parameter to the cooking device for modifying the cooking process to compensate for the indication that the range is exceeded.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method, a non-transitory machine-readable medium, and a cooking device for controlling a cooking process. Background Art

[0002] Starchy foods such as bread, cakes, and cookies can expand during the cooking process. Other foods such as vegetables and meats can shrink during the cooking process. There are multiple factors that can affect the outcome of the cooking process. For example, the cooking temperature specified by a recipe or a consumer may not result in the food being cooked to the desired quality (e.g., the desired quality in terms of the size, shape, color, texture, taste, juiciness, etc. of the food). Additionally, the cooking device may not reach the precise cooking temperature and may not take into account various other factors such as the ambient temperature, the size of the chamber of the cooking device, the location where the food is placed within the chamber, etc. In one example, if the temperature is too high, a food such as a cake can rise too quickly, causing the cake to burst on its surface and / or become too dark, which is contrary to the consumer's expectations. Summary of the Invention

[0003] A camera can be integrated as part of a cooking device, such as an air fryer or an oven, and computer vision techniques, such as based on certain machine learning techniques, can be used to monitor the cooking process based on images acquired by such a camera. However, some cooking devices may have limited computational resources, which means that certain computer vision techniques may not be suitable for implementation by such cooking devices.

[0004] Certain aspects or embodiments described herein relate to controlling a cooking process based on image data, such as can be acquired by a camera of a cooking device. Certain aspects or embodiments can reduce or eliminate certain problems associated with using computer vision techniques to monitor or control a cooking process.

[0005] In a first aspect, a computer-implemented method of controlling a cooking process implemented by a cooking device is described. The method includes receiving: first image data corresponding to a view at a first time of the cooking process; and second image data corresponding to a view at a second time of the cooking process. The method further includes determining a first image parameter value from a portion of the first image data corresponding to a region of interest. The region of interest is selected to include a portion of the view. The region of interest includes a first area mapped to a portion of the food visible in the view and a second area mapped to a portion of the background of the food visible in the view. The first image parameter value represents the cooking state of the food at the first time of the cooking process. The method further includes determining a second image parameter value from a portion of the second image data corresponding to the region of interest. The second image parameter value represents the cooking state of the food at the second time of the cooking process. The method further includes determining an indication of a change in the cooking state based on a comparison of the first image parameter value and the second image parameter value. In response to an indication outside a specified range between the first time and the second time for the cooking process, the method further includes providing modified cooking parameters to the cooking device for modifying the cooking process to compensate for the indication outside the range.

[0006] Some embodiments related to the first aspect and other aspects are described below. In some embodiments, if the indication is within the range, the indication represents a prediction that the food is expected to meet a specified quality due to the cooking process.

[0007] In some embodiments, if the indication is outside the range, the indication represents a prediction that the food is not expected to meet the specified quality due to the cooking process. In such a case, the cooking parameters will compensate for the indication outside the range so that the food meets the specified quality due to the cooking process.

[0008] In some embodiments, the food expands or contracts at a higher rate during a first stage of the cooking process than during a second stage of the cooking process. The food darkens at a higher rate during the second stage than during the first stage. The specified range used during the first stage may be different from the specified range used during the second stage. The expansion or contraction of the food may be indicated by the movement of a portion of the food relative to the background.

[0009] In some embodiments, the first image parameter value is determined based on a ratio of the first area to the second area in the region of interest of the first image data. The second image parameter value may be determined based on a ratio of the first area to the second area in the region of interest of the second image data.

[0010] In some embodiments, a first image parameter value is determined based on a color parameter value derived from a portion of first image data corresponding to a first area. A second image parameter value may be determined based on a color parameter value derived from a portion of second image data corresponding to the first area.

[0011] In some embodiments, this portion of the food includes an edge of the food visible in the view relative to a background of the food visible in the view.

[0012] In some embodiments, the first image parameter value is an average pixel intensity value recorded by a set of pixels corresponding to a region of interest in the first image data. The second image parameter value may be an average pixel intensity value recorded by a set of pixels corresponding to a region of interest in the second image data.

[0013] In some embodiments, the indication is proportional to a difference between the first image parameter value and the second image parameter value.

[0014] In some embodiments, the indication is proportional to a difference in cooking speed between a first time and a second time. The cooking speed at the first time may be proportional to a difference between the first image parameter value and a reference image parameter value determined from reference image data obtained at a reference time during the cooking process. The reference image parameter value may be determined from a portion of the reference image data corresponding to the region of interest. The reference image parameter value may represent a cooking state of the food at the reference time during the cooking process. The cooking speed at the second time may be proportional to a difference between the second image parameter value and the reference image parameter value.

[0015] In some embodiments, the cooking speed at the first time is inversely proportional to a duration between the first time and the reference time. The cooking speed at the second time may be inversely proportional to a duration between the second time and the reference time.

[0016] In some embodiments, the region of interest is selected to include corresponding pixels of the first image data and the second image data that are within a boundary enclosing at least a portion of the food visible in the view.

[0017] In some embodiments, pixels along or enclosed by a perimeter of the region of interest in the first image data and the second image data correspond to designated pixels along or enclosed by the boundary.

[0018] In some embodiments, the region of interest has an area up to 50% of a total number of pixels of a camera imaging sensor used to obtain the first image data and the second image data.

[0019] In some embodiments, the region of interest is selected to include a portion of the view in which, from a comparison of corresponding portions of first and second image data corresponding to the region of interest, movement of a portion of the food relative to the background of the food is evident during the cooking process. Both the portion and the background can be within the region of interest. In some embodiments, the comparison can indicate that the portion of the food has moved relative to the background as a result of the cooking process.

[0020] In a second aspect, a non-transitory machine-readable medium is described. The non-transitory machine-readable medium stores instructions readable and executable by a processor to implement the method of any aspect of the first aspect or related embodiments.

[0021] In a third aspect, a cooking apparatus for implementing a cooking process is described. The cooking apparatus includes a cooking chamber for receiving food. The cooking apparatus further includes a housing defining the cooking chamber. The cooking apparatus further includes an air circulation system for circulating an air flow inside the cooking chamber. The cooking apparatus further includes a camera for capturing an image during the cooking process. The cooking apparatus includes a controller. The controller is configured to implement the method of any aspect of the first aspect or related embodiments.

[0022] Certain aspects or embodiments described herein can provide various technical benefits such as: facilitating control of the cooking process using computationally lightweight techniques suitable for implementation by resource-constrained hardware such as that integrated in certain cooking apparatuses; improving the result of the cooking process, e.g., achieving food of a certain quality; and / or increasing the reliability of controlling the cooking process by simplifying the techniques for monitoring the cooking state of the food.

[0023] These and other aspects of the invention will become apparent and be elucidated with reference to the embodiments (one or more) described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Exemplary embodiments of the invention will now be described, by way of example only, with reference to the following drawings, in which:

[0025] Figure 1 relates to a method of controlling a cooking process according to an embodiment;

[0026] Figure 2 is a schematic diagram of a cooking ecosystem according to an embodiment;

[0027] Figure 3 is a schematic diagram of a cooking apparatus for implementing a cooking process according to an embodiment;

[0028] Figure 4 (A) to Figure 4 (B) are schematic diagrams of views of food at different times during a cooking process;

[0029] Figure 5 (A) to Figure 5 (B) are graphs of experimental data obtained when baking a cake in different scenarios;

[0030] Figure 6 is a schematic diagram of a machine-readable medium for implementing various embodiments; and

[0031] Figure 7 is a schematic diagram of a device for implementing various embodiments. Detailed implementation

[0032] As described above, computer vision technology can be used to monitor the cooking process. However, such technology can be not easily implemented and can produce inconsistent results. An example scenario involving baking a cake is outlined below.

[0033] Machine learning-based computer vision technology can be used to monitor the size of a cake during the cooking process. Changes in the size of the cake can indicate the speed at which the cake is cooking. Techniques such as segmenting the cake from the background and / or finding the edges of the cake can be used to detect any size changes. However, there can be several factors that make these solutions not optimal. These factors can include: the cake mixture can have various colors, which means that threshold segmentation can work differently for different types of cakes (e.g., a chocolate cake has a different color from a vanilla cake); during cooking, the color of the cake can change as the surface can brown; and / or the cake can be reflected by some surfaces as different baking pans can have different colors, lines, textures, etc.

[0034] If segmentation is used (e.g., based on threshold segmentation techniques), the contrast between the cake and the background can vary between different cake types, baking pan types, and during the cooking process. This can make it difficult to maintain precise segmentation during the baking process. Inaccurate segmentation can result in incorrect settings being used in the case of controlling the cooking process based on the segmentation.

[0035] Otherwise, highly complex segmentation or edge detection algorithms or machine learning techniques can be required. However, the low-power computing resources of the cooking device may not be able to run such algorithms. Although cloud-based solutions with higher computing capabilities can be used, this involves a large amount of data processing and signaling over the network, as well as additional costs such as installing a network interface card on the cooking device and paying for network usage fees.

[0036] Therefore, a simple and efficient computer vision solution is needed.

[0037] Figure 1Relates to a computer-implemented method 100 for controlling a cooking process. The cooking process is implemented by a cooking device.

[0038] As referred to herein, a "cooking process" refers to heating food to cause changes in the food. The application of such heat can cause the food to merely warm up, or greater changes can occur to the food, such as can be achieved by using cooking methods such as baking, roasting, grilling, frying, air frying, etc.

[0039] There can be various main stages during the cooking process, such as a growth / expansion stage, a shrinkage / contraction stage, and / or a browning stage. For example, in the case of baking a cake mixture, the first cooking stage can be dominated by a growth stage during which the cake mixture ferments rapidly. There can still be some limited browning during the first cooking stage. The second cooking stage can be controlled by a browning stage during which the fermentation rate decreases or stops and the browning rate increases compared to the first cooking stage.

[0040] As referred to herein, a "cooking device" refers to any device capable of heating food to complete the above cooking process. Heat can be applied to the food by the cooking device in one or more ways, such as by conduction, convection, or radiation. Examples of cooking devices include: ovens, microwave ovens, stove tops, air fryers, etc.

[0041] Method 100 can be implemented by, for example, a processor of a cooking device or another entity, as described in more detail below.

[0042] Method 100 includes, at block 102, receiving first image data and second image data (either of which image data can be referred to herein as "image data"). The first image data corresponds to a view of the food at a first time of the cooking process. The second image data corresponds to a view of the food at a second time of the cooking process. The view of the food can refer to the field of view of a camera used to acquire the image data.

[0043] The following sections provide additional details of the image data referred to in block 102, after which additional details of method 100 are described.

[0044] The second image data can be received after the first image data (e.g., when or shortly after the corresponding image data is obtained). Although in some cases, the first image data and the second image data can be received simultaneously.

[0045] As described below, reference image data can be obtained at a reference time of the cooking process (i.e., time = t_0). In some cases, the reference time refers to the time when the cooking device starts heating the food (i.e., t_0 = 0). In some cases, the reference time refers to the time when the food starts to undergo physical changes due to the cooking process (e.g., t_0 > 0 minutes, such as 5 minutes after the start of the cooking process). In some cases, the first image data can correspond to the reference image data. Although in some cases, the first image data can be obtained after the reference image data. That is, the first time can refer to a certain time after the start of the cooking process. This can be related to the situation where little physical change occurs to the food within a certain time after the start of the cooking process and the food does not need to be monitored during this time.

[0046] As will be explained in more detail below, during the cooking process, each image can be obtained by a camera at a specified time interval. Such intervals can be of a fixed duration or different durations throughout the cooking process. In some cases, the duration of such intervals can depend on the type of food being cooked and / or the stage of the cooking process. Therefore, in some cases, the duration of such intervals can change at least once during the cooking process. This can be useful in cases where there are different cooking stages, where the intervals used may need to be different depending on the cooking stage. In some cases, the duration of the intervals can be fixed throughout the cooking process, which can simplify the implementation of the image data acquisition process. The experimental data provided below indicates example time intervals that can be used to implement Method 100.

[0047] As will be explained in more detail below, the control of the cooking process can be based on the analysis of image data obtained at two times during the cooking process. Therefore, the first image data and the second image data can be any two image data in the set of image data obtained during the cooking process. For example, if the cooking process lasts for 20 minutes and there is a 2-minute time interval, a total of 11 images can be obtained (i.e., for the i-th image, the reference image data at t_0 plus 10 subsequent images obtained at time t(i)). Any two (e.g., consecutive) of the eleven images can be analyzed to determine how to control the cooking process. In other words, the first image data and the second image data can refer to any two of the eleven images. The form of this example is as follows.

[0048] In some cases, the acquisition time of the i-th image can be given by the expression: t(i) = t_0 + i*k, where k is the duration of the time interval (assuming a fixed interval duration). Those skilled in the art will understand how to appropriately modify such expressions in the case where the interval changes at least once during the cooking process.

[0049] Each time new image data is received, the analysis of the image data can be repeated. In the case discussed above, the first image data (i.e., the "i" - th image) can be obtained at t(i)=t_0 + i*k, and the second image data (i.e., the "(i + 1)" - th image) can be obtained at t(i)=t_0+(i + 1)*k.

[0050] The image data can be obtained by a camera (as described in more detail below). In some cases, the image data can be the raw imaging data provided by the camera. For example, the received image data can be without any image processing and can be in the same format output by the camera's circuitry. In some cases, the image data can be processed (e.g., processed from the raw format, compressed, or modified by the camera's circuitry or any other available processing circuitry).

[0051] In some cases, the image data can indicate the red - green - blue (RGB) values (or another color space) of each pixel of the imaging data. In some cases, the image data can be in a different color space, such as the hue - saturation - value (HSV) color space. The HSV color space (and similar types of color spaces) can be more convenient for certain analyses, such as certain segmentation operations, as described below.

[0052] The following section describes additional details of method 100.

[0053] Method 100 further includes, at block 104, determining a first image parameter value from a portion of the first image data corresponding to the region of interest. The region of interest is selected to include a portion of the view. The region of interest includes a first area mapped to a portion of the food visible in the view and a second area mapped to a portion of the background of the food visible in the view. In some embodiments, the region of interest is selected to include a portion of the view in which movement of a portion of the food is apparent from a comparison of corresponding portions of the first image data and the second image data corresponding to the region of interest during the cooking process. The first image parameter value represents the cooking state of the food at a first time of the cooking process.

[0054] In the case of growing or shrinking food, movement of that portion of the food can be observed within the region of interest. The region of interest is selected to include the portion of the food expected to move during the cooking process. For example, in the case of cake batter, the region of interest can include portions of the food such as the edges of the cake batter and the background of those edges (which are apparent within the field of view of the camera). As the cake batter rises, the edges of the cake batter move relative to the background. By including the edges of the cake batter and the adjacent background within the region of interest, any movement of the edges of the cake batter relative to the background is observable within the region of interest. For example, the proportion of the region of interest that includes the cake batter can vary between a first time and a second time. In some embodiments, the portion of the food includes the edges of the food visible within the view relative to the background of the food visible within the view.

[0055] Block 104 of method 100 further includes determining a second image parameter value from a portion of the second image data corresponding to the region of interest. The second image parameter value represents the cooking state of the food at a second time during the cooking process.

[0056] The same region of interest is used for the first image data and the second image data. The first image data and the second image data can be based on pixel values (e.g., pixel intensity and / or color values) recorded by a set of pixels of the camera (where the set of pixels corresponds to the entire area of the camera imaging sensor). The region of interest can include a subset of the pixel values (such that the subset of pixels corresponds to a sub-area of the camera imaging sensor). The pixel values of the region of interest can represent the appearance of the food (and background) within the region of interest. The data size of the region of interest can be less than the data size of the entire image acquired by the set of pixels. The appearance of the food and background can depend on the illumination used and the spectral reflectance of the food and background. During cooking, the spectral reflectance of the food itself can change, e.g., due to browning. Additionally, as the proportion of the food that constitutes the background changes over time, the overall appearance of the food and background (over the region of interest) can change.

[0057] The first image parameter value and the second image parameter value can be derived respectively from subsets of pixel values of the region of interest corresponding to the first image data and the second image data. For example, a function can be applied to the subset of pixel values in order to derive the image parameter value. By way of example, the function can be the average pixel value from the subset of pixel values (e.g., mean, median, or mode). Due to the food browning and / or portions of the food moving during the cooking process, the image parameter value can have a corresponding change between a first time and a second time.

[0058] Block 104 of method 100 further includes determining an indication of a change in the cooking state based on a comparison of the first image parameter value and the second image parameter value.

[0059] As indicated above, the first image parameter value and the second image parameter value respectively represent the cooking states of the food at the first time and the second time (e.g., depending on how much a portion of the food has moved and / or the overall browning of the food). Thus, by comparing the first image parameter value and the second image parameter value, an indication representing the change in cooking state between the first time and the second time can be determined (e.g., a quantity such as cooking speed or rate of change of cooking speed).

[0060] In response to an indication that is outside a range specified for the cooking process between the first time and the second time, method 100 further includes, at block 106, providing modified cooking parameters to the cooking device for modifying the cooking process to compensate for the out-of-range indication.

[0061] In some cases, the cooking parameter can refer to the cooking temperature of the cooking device. In some cases, the cooking parameter can refer to the cooking time. In some cases, the cooking parameter can be any parameter that can otherwise affect the cooking process.

[0062] The indication can be monitored during the cooking process, e.g., at each time interval. The indication indicates the speed at which the food is being cooked. Certain foods need to be cooked fast enough to ensure that the food is cooked in time and / or to ensure that the food has desired physical properties, such as in terms of size, texture, taste, etc. Similarly, certain foods need to be cooked slow enough to prevent the food from having undesired physical properties, such as excessive swelling / shrinking and / or a burnt surface. Acceptable results can include the food being cooked within an acceptable time range and / or having desired physical properties.

[0063] In some cases, the range can be pre-determined based on experimental data for the type of food being cooked. That is, experiments can be conducted to determine the indications associated with different cooking parameters used for cooking the food. In the case where the indication is associated with an unacceptable result of the cooking process, the indication can be outside the range specified at block 106. On the other hand, in the case where the indication is associated with an acceptable result of the cooking process, the indication can be within the range specified at block 106. The range can extend between an upper threshold indication and a lower threshold indication. An indication above the upper threshold can indicate that the cooking process is too fast to achieve an acceptable result. An indication below the lower threshold can indicate that the cooking process is too slow to achieve an acceptable result.

[0064] Compensating for out-of-range indications can include indicating that a cooking parameter will be used to change the expected outcome of a cooking process. For example, if the expected outcome based on the indication is that the food is being cooked too fast or too slow, the cooking parameter can be identified to decrease or increase the cooking rate, respectively. Such cooking parameters can be provided to the cooking device itself or a user interface (e.g., a user interface of the cooking device itself or another user device). The cooking device can automatically take action to implement the indicated cooking parameter for the cooking process. If the indication is out of range during a first time period, providing the cooking parameter for a second time period (after the first time period) can compensate for the effect of the indication during the first time period, which would otherwise be expected to result in an unacceptable outcome. In other words, the provided cooking parameter can increase the chance that the cooking process has an acceptable outcome.

[0065] Accordingly, method 100 and certain other embodiments described herein can provide one or more technical benefits, such as those described below, and can be understood with reference to the entire disclosure.

[0066] A technical benefit of method 100 and / or related embodiments can be to facilitate control of a cooking process using computationally lightweight techniques suitable for implementation by resource-constrained hardware such as that integrated in certain cooking devices. Although higher-powered computing resources (e.g., processing and / or memory) may be available in, for example, cloud-based computing systems, this can mean that the cloud computing system requires additional signaling, power consumption, time, complexity, and / or cost to control the cooking process. The relative simplicity of method 100 and / or related embodiments can reduce the need to consume such high-performance computing resources. This simplicity can be to the extent that relatively low-cost computing resources of a cooking device or other user device can be used to implement method 100 and / or related embodiments.

[0067] Another technical benefit of method 100 and / or related embodiments can be to improve the outcome of a cooking process, e.g., achieve a specified quality of food (e.g., quality in terms of the size, texture, taste, color, etc. of the food). By indicating the cooking parameters used by the cooking device (e.g., in the case of an out-of-range indication), method 100 and / or related embodiments can increase the likelihood that the cooking process has an acceptable outcome in terms of food quality. In other words, method 100 and / or related embodiments can increase the likelihood that the food has a quality consistent with a recipe or user-specified quality.

[0068] Another technical benefit of method 100 and / or related embodiments may be to improve the reliability of controlling the cooking process. Method 100 and / or related embodiments may represent a relatively simple technique for monitoring the cooking state of food. In contrast, more complex methods, such as machine learning-based methods, may work well in some scenarios, but may require a large amount of training, which can introduce biases that are difficult to overcome.

[0069] Some embodiments related to method 100 are now described.

[0070] In some embodiments, if the indication is within the range, the indication represents a prediction that the food is expected to meet a specified quality (e.g., have an acceptable outcome) due to the cooking process.

[0071] In some embodiments, if the indication is outside the range, the indication represents a prediction that the food is not expected to meet a specified quality (e.g., an acceptable outcome) due to the cooking process. In such embodiments, cooking parameters are used to compensate for the out-of-range indication so that the food meets the specified quality due to the cooking process.

[0072] The indicated cooking parameters can be obtained from a memory (e.g., a cooking device, other user equipment, or other online available) that stores information about cooking parameters for use depending on the indication. For example, the information can be in the form of a lookup table that includes a set of cooking parameters and an associated set of indications (e.g., a set of quantities, such as cooking speed and / or rate of change of cooking speed associated with each cooking parameter in the set of cooking parameters). Such information can be predetermined, for example, based on experimental data obtained by cooking food at different temperatures. Such experiments can be performed by an expert (e.g., the manufacturer of the cooking device) or the user of the cooking device. In some cases, multiple food types can be analyzed. The information can depend on the food type because the nature of the cooking process can be different for different food types. By determining the indication, the information can be looked up to determine the appropriate cooking parameters to use depending on the indication.

[0073] In some embodiments, the food expands or contracts at a higher rate during a first phase of the cooking process than during a second phase of the cooking process. Additionally, the food darkens (e.g., browns) at a higher rate during the second phase than during the first phase. The specified range used during the first phase is different from the specified range used during the second phase. The expansion or contraction of the food can be indicated by the movement of a portion of the food relative to the background.

[0074] There can be scenarios where the cooking process during one cooking phase is faster than another. Monitoring the cooking process at shorter time intervals during the faster cooking phase can be useful, which can reduce the risk that the cooking process proceeds in a way that has an adverse effect on the result of the cooking process.

[0075] Additional technical benefits will be apparent in view of the entirety of the present disclosure and with reference to the embodiments described herein.

[0076] Figure 2 FIG. 200 is a schematic diagram of a cooking ecosystem 200 according to an embodiment. Certain embodiments described herein (e.g., method 100) may be implemented in certain parts of the cooking ecosystem 200. The cooking ecosystem 200 depicts various devices and entities that may be deployed as parts of the cooking ecosystem 200. As described below, in some scenarios, and not every device or entity depicted is required.

[0077] The cooking ecosystem 200 includes a cooking device 202 for cooking food 204. The cooking device 202 includes a controller 206 for controlling the cooking process. For example, the controller 206 may control a heating element (not shown) of the cooking device 202 (e.g., control the cooking temperature of the cooking device 202). The controller 206 is communicatively coupled to a camera 208 for capturing images. The camera 208 has an imaging sensor (not shown) for acquiring imaging data. The camera imaging sensor has a total (active / sensing) area defined by a set of pixels. In some embodiments, the imaging data acquired by a subset of the set of pixels is mapped to a region of interest used in certain embodiments described herein. The camera 208 is positioned such that the region of interest associated with the food 204 is within the field of view of the camera 208. This particular configuration is an example. For example, the camera 208 may or may not be inside the cooking device 202, but even if the camera 208 is outside the cooking device 202, the food 204 may still be within its field of view.

[0078] In some cases, the cooking ecosystem 200 includes a cloud computing service 210 communicatively coupled to the controller 206. The cloud computing service 210 may provide data storage and / or data processing services. The cloud computing service 210 may provide computing resources in cases where there are not sufficient available computing resources in any of the connected devices. In some cases, the cloud computing service 210 may provide updates and other services to the cooking device 202.

[0079] In some cases, the cooking ecosystem 200 includes a user device 212 communicatively coupled to the controller 206. The user device 212 can refer to any computing device associated with a user (e.g., a user of the cooking device 202). Examples of the user device 212 include: smart phones, smart watches, tablets, Internet of Things (IoT) devices, etc. In some cases, the user device 212 can be communicatively coupled to a cloud computing service 210.

[0080] Any one or combination of the controller 206, the cloud computing service 210, and the user device 212 can be used to implement the method 100 and other embodiments described herein. For example, in some cases, the controller 206 can implement the method 100 and related embodiments. In this regard, the controller 206 can include a processor (not shown) for implementing the method 100 and related embodiments. In other cases, the processing circuitry associated with the various devices and entities of the cooking ecosystem 200 can implement the method 100 and related embodiments.

[0081] Figure 3 is a schematic diagram of a cooking device 300 for implementing a cooking process according to an embodiment. The cooking device 300 can implement the functions of certain embodiments described herein, such as those described Figure 1 for the method 100. Certain features of the cooking device 300 can correspond to Figure 2 the features of the cooking device 202 or have functions similar to those of Figure 2 the features of the cooking device 202.

[0082] The cooking device 300 includes a cooking chamber 302 for receiving food 304. The cooking device 300 also includes a housing 306 that defines the cooking chamber 302. The cooking device 300 also includes an air circulation system 308 for circulating an air flow within the cooking chamber 302. Thus, in this regard, the cooking device 300 can have a form similar to a fan oven or an air fryer. The cooking device 300 also includes a camera 310 for capturing an image (a "view" associated with the food 304) during the cooking process. The captured image can correspond to or be used to derive first image data and second image data.

[0083] The cooking device 300 also includes a controller 312, such as the controller 206 corresponding to Figure 2 The controller 312 is configured to implement the method 100 in this embodiment. In additional embodiments, the controller 312 is configured to implement embodiments associated with the method 100.

[0084] Accordingly, in the case of implementing method 100, the controller 312 is configured to: receive first image data and second image data. The first image data corresponds to a view (of the food) at a first time of the cooking process. The second image data corresponds to a view (of the food) at a second time of the cooking process.

[0085] The controller 312 is further configured to determine a first image parameter value from a portion of the first image data corresponding to the region of interest. The region of interest is selected to include a portion of the view. The region of interest includes a first area mapped to a portion of the food visible in the view and a second area mapped to a portion of the background of the food visible in the view. The first image parameter value represents the cooking state of the food at the first time of the cooking process.

[0086] The controller 312 is further configured to determine a second image parameter value from a portion of the second image data corresponding to the region of interest. The second image parameter value represents the cooking state of the food at the second time of the cooking process.

[0087] The controller 312 is further configured to determine an indication of a change in the cooking state based on a comparison of the first image parameter value and the second image parameter value.

[0088] The controller 312 is further configured to, in response to an indication that is outside a specified range between the first time and the second time for the cooking process, determine to provide modified cooking parameters to the cooking device for modifying the cooking process to compensate for the indication that is outside the range.

[0089] Although Figure 3 the controller 312 of the cooking device 300 implementing method 100 is described, in some cases, other devices or entities (such as Figure 2 depicted) may implement at least some of the functions of method 100 (and related embodiments).

[0090] Figure 4 (A) to Figure 4 (B) are schematic views of a view 400 of food such as a cake mixture 402 (represented by the shaded area) at different times of a cooking process used in different embodiments. Although the type of food is a cake mixture 402, similar principles apply to other types of food, whether they grow or shrink during the cooking process.

[0091] The view 400 may be represented by image data, such as image data that can be obtained by a camera as described above. Figure 4 (A) refers to the view 400 at the first time, and Figure 4(B) refers to the view 400 at a second time. The view of the cake mixture 402 is a perspective view, and this is also the case for a camera positioned at the top side of the chamber of the cooking device and facing the center of the chamber.

[0092] As Figure 4 (A) to Figure 4 (B) depicts, the cake mixture 402 is in a circular baking pan. The portion of the inner surface of the baking pan visible in the view 400 is the background 404 of the cake mixture 402. It can be clearly seen from the view 400 that there is a contrast between the reflectance (e.g., spectral reflectance) of the cake mixture 402 (the shaded area) and the background 404 (i.e., the non - shaded area corresponding to the surface of the baking pan).

[0093] The image data can be segmented to identify at least a portion of the image data that includes at least a portion of the cake mixture 402 and the background 404. In some cases, this segmentation can be based on identifying pixel value changes in the image data that exceed a threshold corresponding to the edges of the objects in the view 400. This segmentation can be based on techniques such as the Otsu method known to those skilled in the art. For example, the edge of the baking pan can form a distinct contrast with the surrounding objects in the cooking device and can be segmented from these objects. The boundary 406 depicted by Figure 4 (A) to Figure 4 (B) depicts the segmented portion of the image data. The boundary 406 is depicted as intersecting the upper, lower, left, and right points of the edge of such a baking pan, as visible in the view 400.

[0094] A region of interest 408 is selected from the segmented portion of the image data. Figure 4 (A) to Figure 4 (B) also shows a magnified view of the region of interest 408 in the view 400.

[0095] In some embodiments, the region of interest 408 is selected to include corresponding pixels of first image data and second image data within the boundary 406 that enclose at least the portion of the food (i.e., the cake mixture 402) visible in the surrounding view. In some cases, the region of interest 408 can be entirely within the boundary 406. In some cases, as described below, the region of interest 408 can intersect the boundary 406.

[0096] In some embodiments, the pixels along or enclosed by the perimeter of the region of interest 408 in the first image data and the second image data correspond to the designated pixels along or enclosed by the boundary 406. For example, as Figure 4 (A) to Figure 4As depicted in (B), a section (e.g., a pixel row) of the perimeter of the region of interest 408 can be aligned with a section (e.g., a pixel row) of the boundary 406. In another example, pixels along the perimeter of the region of interest 408 can intersect with a section of the boundary 406. In this example, the region of interest 408 can include pixels both inside and outside the boundary 406. In another example, pixels along the perimeter of the region of interest 408 can correspond to pixels enclosed by the boundary 406. Thus, at least some of the pixels within the region of interest 408 can be mapped to at least some of the pixels within the boundary 406. Not all pixels in the region of interest 408 need to be within the boundary 406. The boundary 406 can help identify the appropriate location of the region of interest 408. In some cases, a segmentation operation may not be required. Instead, the region of interest 408 can be selected without reference to the segmentation boundary 406 (e.g., if the region of interest 408 corresponds to predetermined pixels of the image data).

[0097] The area (i.e., the number of pixels) of the region of interest 408 is less than the area (i.e., the number of pixels) of the segmented portion of the image data. The area of the segmented portion of the image data can be less than the area of the image data. Thus, the region of interest 408 in the view 400 is mapped to a number of pixels (or an area) that is less than the total number of pixels (or the total area) of the entire image data (corresponding to the entire view 400). Up to any one of the following specified percentages: 90%, 80%, 70%, 60%, 50%, 40%, 30%, 20%, 10%, 5%, 4%, 3%, 2%, 1% of the total number of pixels (or the total area) of the image data (corresponding to the entire view 400) can be used for the region of interest. Thus, the region of interest has an area corresponding to up to the specified percentage (listed above) of the total number of pixels of the camera imaging sensor used to obtain the first image data and the second image data.

[0098] In other words, a set of pixels of the camera can be used to generate the image data. A first subset of the set of pixels can form the area of the segmented portion. A second subset of the pixels can form the area of the region of interest 408. The second pixel subset can include some of the first pixel subset. In some cases, the second pixel subset can include pixels that are not in the first pixel subset (i.e., pixels from outside the boundary 406). Generally, the region of interest 408 can be selected to have a size suitable for analysis by low-power computing resources of a cooking device or other user equipment. It has been found that image data of a relatively small size, such as 60×360 pixels (e.g., from an image of size 1920×1080 pixels), is sufficient to provide the functions described herein (i.e., highly accurate cooking state detection even with limited available computing resources). However, fewer or more pixels (as a proportion of the overall image size) can still provide the functions described herein.

[0099] The set of pixels corresponding to the region of interest 408 remains the same between the first image data and the second image data. By comparing Figure 4 (A) through Figure 4 (B), it can be seen that the proportion of the region of interest 408 that includes the shaded area of the cake mixture changes between the first time and the second time because the cake mixture 402 has risen. The proportion of the region of interest 408 that includes the background 404 changes by an opposite corresponding amount between the first time and the second time.

[0100] The region of interest 408 includes a first area and a second area. The first area maps to the portion of the food visible in the view 400 (i.e., the shaded area of the region of interest 408). The second area maps to the portion of the background of the food visible in the view 400 (i.e., the blank / non - shaded area of the region of interest). Such areas refer to portions of the image data. In the case of this depiction, the edge of the food (which is obvious in the view) corresponds to the boundary between the first area and the second area.

[0101] As the cake mixture 402 rises, the ratio of the first area to the second area changes. In other words, the proportion of the region of interest 408 that maps to the cake mixture 402 increases, while the proportion of the region of interest 408 that maps to the background 404 decreases. Depending on the position of the camera and how the food changes in size (e.g., increases or decreases), the proportion of the region of interest 408 that maps to the food can decrease. Since the average apparent intensity and / or color recorded by the pixels corresponding to the region of interest 408 changes between the first time and the second time, the first image parameter value and the second image parameter value can indicate the cooking state of the food.

[0102] Thus, in some embodiments, the first image parameter value is determined based on the ratio of the first area to the second area in the region of interest of the first image data. Additionally, the second image parameter value can be determined based on the ratio of the first area to the second area in the region of interest of the second image data.

[0103] In some embodiments, the first image parameter value is determined based on color parameter values derived from the portion of the first image data corresponding to the first area. Additionally, the second image parameter value can be determined based on color parameter values derived from the portion of the second image data corresponding to the first area. The color parameter values can refer to components of a color space, such as the RGB color space or the HSV color space derived from the pixel intensity values of the pixels of the region of interest 408. Changes in the color parameter values can represent changes in the reflectance (including spectral reflectance) of the food, which indicates the cooking state of the food.

[0104] Therefore, the change in the image parameter value between the first time and the second time can depend on the following factors: (i) the change in the ratio of the first area to the second area in the region of interest; (ii) the change in the reflectivity of the food itself between the first time and the second time, or (iii) a combination of factors (i) and (ii).

[0105] From Figure 4 (A) to Figure 4 (B) depicts the movement of the cake mixture 402 indicating the first cooking stage of the cooking process, during which the cake mixture 402 rises.

[0106] However, a second different cooking stage involves browning of the surface of the cake mixture 402. In either cooking stage, the image parameter value (e.g., average pixel intensity and / or color value) can change between the first time and the second time. However, for the first cooking stage and the second cooking stage, the main causes of this change can be different. This is because, in the first cooking stage, the change in the ratio of the first area to the second area between the first time and the second time can be the main cause of the change in the image parameter value. In the second cooking stage, the change in the reflectivity of the cake mixture 402 itself can be the main cause of the change in the image parameter value. In the first cooking stage, the reflectivity of the cake mixture can change, but its contribution to the change in the image parameter value can be less than the change in the ratio. Accordingly, in the second cooking stage, the ratio can change, but its contribution to the change in the image parameter value can be less than the change in the reflectivity.

[0107] Method 100 refers to comparing a first image parameter value and a second image parameter value to determine an indication of a change in cooking state between a first time and a second time. If the indication (i.e., the quantity as described herein) is outside a (predetermined) range, cooking parameters for a cooking device for cooking food are provided (e.g., in response to looking up cooking parameters from information in a look-up table and based on the indication determined by method 100). The cooking parameters can compensate for the indication being out of range. The indication indicates a cooking speed (as described herein). For example, the quantity corresponding to the indication can be the cooking speed itself. In this example, the range specified by method 100 can refer to a range of cooking speeds. In another example, the quantity corresponding to the indication can be a rate of change of the cooking speed. In this example, the range specified by method 100 can refer to a range of rate of change values (i.e., a range of the rate of change of cooking speed values). In some cases, the ranges for different cooking stages can be different. The heating temperature of the cooking device can be adjusted automatically by the cooking device (based on the provided cooking parameters) and / or manually by alerting a user to take some action in response to the indication being out of range (based on the provided cooking parameters). Such a method can prevent a cake from rising too fast during a first cooking stage, where if the heat is too high, cracks can be generated. Additionally, such a method can avoid a cake growing too slowly during a first cooking stage, which can result in insufficient fermentation or wasted time. Further, such a method can prevent too much or too little browning during a second cooking stage.

[0108] Since the main contribution to the change in the image parameter value between the first time and the second time depends on which factor is most relevant to the region of interest (i.e., the area or reflectivity of the food in the region of interest), the range can be different for different cooking stages. However, in some cases, the range can be consistent throughout the cooking process.

[0109] Now refer to Figure 4 (A) to Figure 4 (B) for a description of an implementation for monitoring the progress of cooking a cake mixture 402. Experiments were conducted using this implementation, and the results of these experiments will be discussed below. Similar principles can be applied to other food types, although the implementation can vary depending on the settings of the cooking device and the type of food.

[0110] Reference image data is acquired before or at the start of the cooking process. A region of interest 408 is selected. The pixels of the image data corresponding to the region of interest 408 can be predetermined for the settings of the cooking device or determined based on a segmented portion of the reference image data. In the present embodiment, the region of interest 408 includes the top portion of the cake mixture 402 to detect the rise of the cake mixture 402 during the cooking process. The size of the region of interest 408 is 60×360 pixels (width×height). The cooking speed and / or surface color change can be detected based on the analysis of the image.

[0111] From a reference time t_0 (such as 5 minutes after the start of the cooking process), the image parameter value is calculated every k (e.g., k = 2 minutes) starting from t_0. The image parameter value is referred to as g(t) and corresponds to the average gray-level pixel intensity value in the region of interest 408. The difference between the first time and the second time is equal to the duration k of the interval.

[0112] In some embodiments, for a specific time (i.e., the first time or the second time) during the cooking process, the cooking speed v(t) can be calculated as: v(t) = c_1((g(t) - g(t_0)) / (t - t_0)). The parameter c_1 is a constant (e.g., equal to 1), although different parameter values of c_1 can be used for controlling the scaling. Thus, to calculate the cooking speed, the difference between the image parameter value at time t and the reference time t_0 is calculated. This difference is divided by the time elapsed since the reference time t_0. v(t) is an example quantity corresponding to the indication used in method 100.

[0113] In some embodiments, the rate of change a(t) (of the cooking speed) is calculated as: a(t) = c_2(v(t) - v(t - k)). The parameter c_2 is a constant (e.g., equal to 1) for controlling the scaling. v(t) corresponds to the cooking speed at the second time. v(t - k) corresponds to the cooking speed at the first time. Thus, to calculate the rate of change, the difference between the cooking speeds at the first time and the second time is calculated. Thus, the rate of change is proportional to the difference between the first image parameter value and the second image parameter value. a(t) is another example quantity corresponding to the indication used in method 100.

[0114] Thus, in some embodiments, the first image parameter value is the average pixel intensity value (e.g., average value, median value, or mode value) registered by a set of pixels corresponding to the region of interest in the first image data. Additionally, the second image parameter value is the average pixel intensity value (e.g., average value, median value, or mode value) recorded by a set of pixels corresponding to the region of interest in the second image data.

[0115] In some embodiments, the indication (e.g., a(t)) is proportional to the difference between the first image parameter value and the second image parameter value. Similarly, in some embodiments, the indication is proportional to the difference in cooking speed between the first time and the second time.

[0116] In some embodiments, the cooking speed v(t-k) at the first time (t-k) is proportional to the difference between the first image parameter value g(t-k) and the reference image parameter value g(t_0), which is determined from reference image data obtained at the reference time t0 of the cooking process. The reference image parameter value g(_0) is determined from a portion of the reference image data corresponding to the region of interest 408. The reference image parameter value g(_0) represents the cooking state of the food at the reference time of the cooking process. The cooking speed v(t) at the second time (t) is proportional to the difference between the second image parameter value g(t) and the reference image parameter value g(_0).

[0117] In some embodiments, the cooking speed at the first time is inversely proportional to the duration between the first time and the reference time. Additionally, the cooking speed at the second time is inversely proportional to the duration between the second time and the reference time.

[0118] In some embodiments, the range can extend between an upper threshold and a lower threshold of a(t) during a cooking phase. The upper threshold and the lower threshold can be determined experimentally, in which experts conduct multiple experiments to cook food at different temperatures. Such experiments can generate a set of curves representing v(t) and / or a(t) for each cooked food sample. The experts can determine whether the selected temperature cooks the food to an appropriate quality and calculate the acceptable range of the rate of change to exclude v(t) and / or a(t) values that result in the food not having an acceptable quality (e.g., excessive or insufficient growth or shrinkage, or excessive or insufficient browning). As previously mentioned, different cooking phases can be associated with different ranges (for each of v(t) and a(t)). In some cases, the experts can also determine the range of acceptable rate-of-change values for each cooking phase. Similarly, the range can extend between an upper threshold and a lower threshold of v(t) during a cooking phase. Whether to use v(t) or a(t) as the indication depends on whether one of these quantities is particularly useful for identifying that the cooking speed is too fast or too slow to achieve an acceptable result.

[0119] In some embodiments, the region of interest is selected to include a portion of the view in which the movement of a portion of the food relative to the background of the food is apparent from a comparison of corresponding portions of the first image data and the second image data corresponding to the region of interest during the cooking process. Both the portion and the background can be within the region of interest. In some embodiments, the comparison can indicate that the portion of the food has moved relative to the background due to the cooking process.

[0120] Method 100 and related embodiments can control the cooking process depending on the values of v(t) and / or a(t), e.g., by providing modified cooking parameters for the cooking device to cook food. The cooking parameters can compensate for effects indicating out of range. For example, if the cooking process is determined to be too fast (e.g., causing too much food to grow or brown), an indication of the cooking parameters to be used can inform the consumer to lower the temperature setting of the cooking device or automatically control the cooking device to lower the cooking temperature. In this way, food that would otherwise have an unacceptable quality (if the cooking process were carried out without modification) can be cooked to an acceptable quality because the cooking parameters can effect a modification of the cooking process.

[0121] Figure 5 (A) to Figure 5 (B) are graphs of experimental data obtained when cooking a cake in different scenarios. Figure 5 (A) refers to the cooking speed v(t) of an exemplary cake in two scenarios (labeled #1 and #2, respectively). Figure 5 (B) refers to the rate of change a(t) of the rise of the exemplary cake in these two scenarios #1 and #2.

[0122] In scenario #1, the cooking temperature is acceptable, which means that the cooking speed (as Figure 5 depicted in A) and the rate of change (as Figure 5 depicted in B) are within the ranges specified for the cooking speed and the rate of change, respectively. The smooth curve representing the cooking speed associated with scenario #1 is within the range: 0 ≤ v(t) < 75%. For scenario #1, the rate of change a(t) derived from the cooking speed v(t) is within the range -9% < a(t) < 26%. The cake cooked under scenario #1 has acceptable results in terms of the degree of cake expansion and the degree of browning.

[0123] In scenario #2, the cooking temperature is unacceptable (too high), which means that the cooking speed (as Figure 5 depicted in A) and the rate of change (as Figure 5Those depicted by B are outside the specified ranges of cooking speed and rate of change, respectively. Contrary to Scenario #1, the steeper curve representing the cooking speed associated with Scenario #2 is within the range: 0 ≤ v(t) < 285%. In Scenario #2, due to the excessive rise, the surface of the cake cracked, and its surface color became substantially browned. Since in Scenario #2, the cake rose rapidly within the first 20 to 30 minutes of the cooking process, the ratio of the first area to the second area in the region of interest changed rapidly within the first 20 to 30 minutes, which caused the value of v(t) to increase sharply within this time range. In addition, since the cake darkened rapidly after about 30 minutes, it darkened rapidly within the region of interest, which caused the value of v(t) to drop sharply after 30 minutes.

[0124] Experimental data obtained by cooking food under different scenarios (e.g., at different temperatures) can be used to identify the ranges. In some cases, an expert such as a human or machine learning model can select which experimental data is associated with acceptable results in terms of food quality and select one or more ranges based on the selected experimental data.

[0125] For example, referring to Figure 5 (A) to Figure 5 (B), a first range can be specified for the first cooking stage (cake fermentation stage), and a second range can be specified for the second cooking stage (browning stage).

[0126] The first range can be that if v(t) > 50% within the first 20 minutes, cooking parameters are provided to compensate for the indicated v(t) exceeding 50% within this time range. Obviously, the data of Scenario #1 did not trigger the provision of cooking parameters. However, the data of Scenario #2 did trigger the provision of cooking parameters (i.e., at t = 15 minutes).

[0127] The second range can be that if |a(t)| > 10% after 35 minutes, cooking parameters are provided to compensate for the indicated |a(t)| exceeding 10% within this time range. Obviously, the data of Scenario #1 did not trigger the provision of cooking parameters. However, the data of Scenario #2 did trigger the provision of cooking parameters (i.e., at t = 35 minutes).

[0128] Figure 6 is a schematic diagram of a non-transitory machine-readable medium 600 for implementing various embodiments described herein. As used herein, the term "non-transitory" does not include transient propagated signals. The machine-readable medium 600 stores instructions 602 that can be read and executed by a processor 604 to implement the methods of any of the embodiments described herein (e.g., Method 100 and / or related embodiments). The machine-readable medium 600 and / or the processor 604 can be byFigure 2 or Figure 3 implemented by any one of the controller 206, cloud computing service 210, user device 212, and / or controller 312 of Figure 3 .

[0129] Figure 7 is a schematic diagram of an apparatus 700 for implementing various embodiments described herein. The apparatus 700 may be implemented by Figure 2 or Figure 3 any one of the controller 206, cloud computing service 210, user device 212, and / or controller 312 of Figure 3 .

[0130] The apparatus 700 includes a processor 702. The processor 702 is configured to communicate with an interface 704. The interface 704 may be any interface (wireless or wired) that implements a communication protocol to facilitate the exchange of data (e.g., image data, cooking device control instructions, etc.) with other devices such as another part of the cooking ecosystem 200.

[0131] The apparatus 700 further includes a memory 706 (e.g., non-transitory or otherwise) that stores instructions 708 readable and executable by the processor 702 to implement various embodiments described herein (e.g., any one of method 100 or associated embodiments).

[0132] The present disclosure includes the subject matter of the following numbered paragraphs.

[0133] Paragraph 1. A computer-implemented method for controlling a cooking process implemented by a cooking device, the method comprising:

[0134] Receiving:

[0135] first image data corresponding to a view of food at a first time of the cooking process; and

[0136] second image data corresponding to a view of food at a second time of the cooking process; and

[0137] Determining:

[0138] a first image parameter value from a portion of the first image data corresponding to a region of interest, wherein the region of interest is selected to include a portion of the view in which movement of a portion of the food during the cooking process is apparent from a comparison of corresponding portions of the first image data and the second image data corresponding to the region of interest, and wherein the first image parameter value represents the cooking state of the food at the first time of the cooking process;

[0139] A second image parameter value, the second image parameter value being from a portion of second image data corresponding to a region of interest, wherein the second image parameter value represents the cooking state of food at a second time of a cooking process; and

[0140] A comparison of the first image parameter value and the second image parameter value indicates a change in the cooking state; and

[0141] In response to an indication that is outside a specified range between a first time and a second time for the cooking process, providing cooking parameters to a cooking device for cooking the food so as to compensate for the indication that is outside the range.

[0142] Paragraph 2. The method according to paragraph 1, wherein:

[0143] If the indication is within the range, the indication represents a prediction that the food is expected to meet a specified quality due to the cooking process; or

[0144] If the indication is outside the range, the indication represents a prediction that the food is not expected to meet a specified quality due to the cooking process, and wherein the cooking parameters compensate for the indication that is outside the range so that the food meets the specified quality due to the cooking process.

[0145] Paragraph 3. The method according to any one of paragraphs 1 to 2, wherein:

[0146] The food expands or contracts at a higher rate during a first phase of the cooking process than during a second phase of the cooking process;

[0147] The food darkens at a higher rate during the second phase than during the first phase; and

[0148] The specified range used during the first phase is different from the specified range used during the second phase.

[0149] Paragraph 4. The method according to any one of paragraphs 1 to 3, wherein the region of interest includes:

[0150] A first area mapped to a portion of the food visible in the view; and a second area mapped to a portion of the background of the food visible in the view.

[0151] Paragraph 5. The method according to paragraph 4, wherein:

[0152] Determining the first image parameter value based on a ratio of the first area to the second area in the region of interest of the first image data; and

[0153] Determining the second image parameter value based on a ratio of the first area to the second area in the region of interest of the second image data.

[0154] Paragraph 6. The method according to any one of paragraphs 4 to 5, wherein:

[0155] Determine a first image parameter value based on a color parameter value derived from a portion of first image data corresponding to a first area; and

[0156] Determine a second image parameter value based on a color parameter value derived from a portion of second image data corresponding to the first area.

[0157] Paragraph 7. The method according to any one of paragraphs 1 to 6, wherein the portion of the food includes an edge of the food visible in the view relative to the background of the food visible in the view.

[0158] Paragraph 8. The method according to any one of paragraphs 1 to 7, wherein:

[0159] The first image parameter value is an average pixel intensity value recorded by a set of pixels corresponding to a region of interest in the first image data; and

[0160] The second image parameter value is an average pixel intensity value recorded by a set of pixels corresponding to a region of interest in the second image data.

[0161] Paragraph 9. The method according to any one of paragraphs 1 to 8, wherein the indication is proportional to the difference between the first image parameter value and the second image parameter value.

[0162] Paragraph 10. The method according to paragraph 9, wherein the indication is proportional to the difference in cooking speed between a first time and a second time, and wherein:

[0163] The cooking speed at the first time is proportional to the difference between the first image parameter value and a reference image parameter value, the reference image parameter value being determined from reference image data obtained at a reference time during the cooking process, wherein the reference image parameter value is determined from a portion of the reference image data corresponding to the region of interest, and wherein the reference image parameter value represents the cooking state of the food at the reference time during the cooking process; and

[0164] The cooking speed at the second time is proportional to the difference between the second image parameter value and the reference image parameter value.

[0165] Paragraph 11. The method according to paragraph 10, wherein:

[0166] The cooking speed at the first time is inversely proportional to the duration between the first time and the reference time; and

[0167] The cooking speed at the second time is inversely proportional to the duration between the second time and the reference time.

[0168] Paragraph 12. The method according to any one of paragraphs 1 to 11, wherein the region of interest is selected to include corresponding pixels of the first image data and the second image data, and the corresponding pixels are within the boundary enclosing at least a portion of the food visible in the surrounding view.

[0169] Paragraph 13. The method according to paragraph 12, wherein the pixels along or enclosed by the perimeter of the region of interest in the first image data and the second image data correspond to the specified pixels along or enclosed by the boundary.

[0170] Paragraph 14. A non-transitory machine-readable medium storing instructions readable and executable by a processor to implement the method according to any one of paragraphs 1 to 13.

[0171] Paragraph 15. A cooking device for performing a cooking process, the cooking device comprising:

[0172] A cooking chamber for receiving food;

[0173] A housing that defines the cooking chamber;

[0174] An air circulation system for circulating an air flow inside the cooking chamber; a camera for capturing an image during the cooking process; and a controller configured to:

[0175] Receive:

[0176] First image data corresponding to a view of the food at a first time of the cooking process; and

[0177] Second image data corresponding to a view of the food at a second time of the cooking process; and

[0178] Determine:

[0179] A first image parameter value from a portion of the first image data corresponding to the region of interest, wherein the region of interest is selected to include a portion of the view in which movement of a portion of the food is apparent from a comparison of corresponding portions of the first image data and the second image data corresponding to the region of interest during the cooking process, and wherein the first image parameter value represents the cooking state of the food at the first time of the cooking process;

[0180] A second image parameter value from a portion of the second image data corresponding to the region of interest, wherein the second image parameter value represents the cooking state of the food at the second time of the cooking process;

[0181] Indicate a change in the cooking state based on a comparison of the first image parameter value and the second image parameter value; and

[0182] In response to an indication that is outside a specified range between a first time and a second time for a cooking process, provide cooking parameters to a cooking device for cooking food so as to compensate for the indication that is outside the range.

[0183] Any one of the models described herein can be implemented by a processing circuitry for implementing the methods described herein. Accordingly, some of the blocks of the methods can involve using such models in order to provide the stated functionality. The models can be (machine learning) ML-based or non-ML-based. However, some of the embodiments described herein involve using non-ML-based models, which can avoid the need to use substantial computational resources and / or implement local processing.

[0184] Although the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.

[0185] One or more features described in one embodiment can be combined with or substituted for features described in another embodiment.

[0186] Embodiments in the present disclosure can be provided as a method, a system, or as a combination of machine-readable instructions and processing circuitry. Such machine-readable instructions can be included on or in a non-transitory machine (e.g., a computer) readable storage medium having computer-readable program code thereon (including but not limited to disk storage, CD-ROM, optical storage, flash memory, etc.).

[0187] The present disclosure is described with reference to the flowcharts and block diagrams of methods, apparatuses, and systems according to embodiments of the present disclosure. Although the above flowcharts illustrate a specific order of execution, the order of execution can be different from that depicted. Blocks described with respect to one flowchart can be combined with blocks of another flowchart. It should be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by machine-readable instructions.

[0188] Machine-readable instructions can be executed, for example, by a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to implement the functions described in the specification and the drawings. Specifically, the processor or processing circuitry or its modules can execute the machine-readable instructions. Thus, the functional modules of the apparatuses and other devices described herein can be implemented by a processor that executes machine-readable instructions stored in a memory or by a processor that operates according to instructions embedded in logic circuitry. The term 'processor' should be construed broadly to include a CPU, a processing unit, an ASIC, a logic unit, or a programmable gate array, etc. These methods and functional modules can be executed by a single processor or divided among several processors.

[0189] Such machine-readable instructions can also be stored in a computer-readable storage medium, which can direct a computer or other programmable data processing device to operate in a specific mode.

[0190] Such machine-readable instructions can also be loaded onto a computer or other programmable data processing device, such that the computer or other programmable data processing device performs a series of operations to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device implement the functions specified by the block(s) in the flowchart and / or block diagram.

[0191] In addition, the teachings herein can be implemented in the form of a computer program product stored in a storage medium and including multiple instructions for causing a computer device to implement the methods described in the embodiments of the present disclosure.

[0192] The elements or steps described with respect to one embodiment can be combined with or replaced by the elements or steps described with respect to another embodiment. Other variations of the disclosed embodiments can be understood and implemented by those skilled in the art in practicing the claimed invention by studying the drawings, the disclosure, and the appended claims. In the claims, the word 'comprising' does not exclude other elements or steps, and the indefinite article 'a' or 'an' does not exclude a plurality. A single processor or other unit can implement the functions of several items recited in the claims. The fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously. A computer program can be stored or distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A computer-implemented method (100) for controlling a cooking process implemented by a cooking device, the method comprising: Receiving (102): First image data corresponding to a view at a first time of the cooking process; And Second image data corresponding to a view at a second time of the cooking process; And Determining (104): A first image parameter value from a portion of the first image data corresponding to a region of interest, where the region of interest is selected to include a portion of the view, where the region of interest includes a first area mapped to a portion of the food visible in the view and a second area mapped to a portion of the background of the food visible in the view, and where the first image parameter value represents the cooking state of the food at the first time of the cooking process; A second image parameter value from a portion of the second image data corresponding to the region of interest, where the second image parameter value represents the cooking state of the food at the second time of the cooking process; And Indicating a change in the cooking state based on a comparison of the first image parameter value and the second image parameter value; And In response to an indication outside a specified range between the first time and the second time for the cooking process, providing (106) modified cooking parameters to the cooking device for modifying the cooking process to compensate for the indication outside the range.

2. The method according to claim 1, wherein: If the indication is within the range, the indication represents a prediction that the food is expected to meet a specified quality due to the cooking process; or If the indication is outside the range, the indication represents a prediction that the food is not expected to meet the specified quality due to the cooking process, and wherein the cooking parameters compensate for the indication outside the range so that the food meets the specified quality due to the cooking process.

3. The method according to any one of claims 1 to 2, wherein: The food expands or contracts at a higher rate during a first stage of the cooking process than during a second stage of the cooking process, where the expansion or contraction of the food is indicated by the movement of the portion of the food relative to the background; The food darkens at a higher rate during the second stage than during the first stage; And The specified range used during the first stage is different from the specified range used during the second stage.

4. The method according to claim 3, wherein during the first stage of the cooking process: Determining the first image parameter value based on a ratio of the first area to the second area in the region of interest of the first image data; and Determining the second image parameter value based on a ratio of the first area to the second area in the region of interest of the second image data.

5. The method according to claim 3, wherein during the second stage of the cooking process: determining the first image parameter value based on a color parameter value derived from a portion of the first image data corresponding to the first area; and determining the second image parameter value based on the color parameter value derived from a portion of the second image data corresponding to the first area.

6. The method according to any one of claims 1 to 5, wherein the portion of the food includes an edge of the food visible in the view relative to a background of the food visible in the view.

7. The method according to any one of claims 1 to 6, wherein: the first image parameter value is an average pixel intensity value recorded by a set of pixels corresponding to a region of interest in the first image data; and the second image parameter value is an average pixel intensity value recorded by a set of pixels corresponding to a region of interest in the second image data.

8. The method according to any one of claims 1 to 7, wherein the indication is proportional to a difference between the first image parameter value and the second image parameter value.

9. The method according to claim 8, wherein the indication is proportional to a difference in cooking speed between the first time and the second time, and wherein: the cooking speed at the first time is proportional to a difference between the first image parameter value and a reference image parameter value determined from reference image data obtained at a reference time during the cooking process, wherein the reference image parameter value is determined from a portion of the reference image data corresponding to the region of interest, and wherein the reference image parameter value represents a cooking state of the food at the reference time during the cooking process; and the cooking speed at the second time is proportional to a difference between the second image parameter value and the reference image parameter value.

10. The method according to claim 9, wherein: the cooking speed at the first time is inversely proportional to a duration between the first time and the reference time; and the cooking speed at the second time is inversely proportional to a duration between the second time and the reference time.

11. The method according to any one of claims 1 to 10, wherein the region of interest is selected to include corresponding pixels of the first image data and the second image data within a boundary enclosing at least a portion of the food visible in the view.

12. The method according to any one of claims 1 to 11, wherein the region of interest has an area up to 50% of a total number of pixels of a camera imaging sensor used to obtain the first image data and the second image data.

13. The method according to any one of claims 1 to 12, wherein the region of interest is selected to include a portion of the view in which, from a comparison of corresponding portions of the first image data and the second image data corresponding to the region of interest, movement of a portion of the food relative to the background of the food is apparent during the cooking process, wherein both the portion and the background are within the region of interest, and wherein the comparison indicates that the portion of the food has moved relative to the background due to the cooking process.

14. A non-transitory machine-readable medium (600) storing instructions (602) readable and executable by a processor (604) to implement the method according to any one of claims 1 to 13.

15. A cooking apparatus (300) for implementing a cooking process, comprising: a cooking chamber (302) for receiving food (304); a housing (306) defining the cooking chamber; an air circulation system (308) for circulating an air flow inside the cooking chamber; a camera (310) for capturing images during the cooking process; and a controller (312) configured to: receive (102): first image data corresponding to a view at a first time of the cooking process; and second image data corresponding to a view at a second time of the cooking process; and determine (104): a first image parameter value from a portion of the first image data corresponding to a region of interest, wherein the region of interest is selected to include a portion of the view, wherein the region of interest includes a first area mapped to a portion of the food visible in the view and a second area mapped to a portion of the background of the food visible in the view, and wherein the first image parameter value represents the cooking state of the food at the first time of the cooking process; a second image parameter value from the portion of the second image data corresponding to the region of interest, wherein the second image parameter value represents the cooking state of the food at the second time of the cooking process; a change in the cooking state indicated based on a comparison of the first image parameter value and the second image parameter value; and in response to the indication beyond a range specified for the cooking process between the first time and the second time, provide (106) modified cooking parameters to the cooking apparatus to modify the cooking process so as to compensate for the indication beyond the range.

Images