Oven with imaging system for food preparation

Abstract

The oven may include an image processing system configured to capture an image of an interior of the oven cavity. An image of the interior of the oven cavity captured by the image processing system is input into the trained machine learning model to identify food items in the image. The oven uses the identity of the food output by the machine learning model to determine a cooking procedure for the food item and cook the food item according to the cooking procedure. The oven may use an image processing system to monitor a cooking process of a food item and update a cooking procedure during the cooking process based on an observed progress.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for ovens. More particularly, the disclosed embodiments relate to ovens utilizing an imaging system.

[0002] introduce

[0003] In recent years, camera systems have been implemented in many common household appliances to help monitor and automate processes that previously required the presence or input of a human user. Summary of the invention

[0004] The present disclosure provides systems, apparatus, and methods related to ovens including image processing systems utilizing machine learning.

[0005] In some embodiments, the disclosed method for cooking multiple food items includes: generating an image of the interior of an oven cavity containing multiple food items using an image sensor; outputting an identity and count of each food item based on the image of the interior of the oven cavity using one or more trained machine learning models; determining a primary food item among the multiple food items disposed in the oven cavity; selecting a cooking program based on cooking the primary food item; and cooking the multiple food items according to the cooking program selected for the primary food item.

[0006] In some embodiments, the oven of the present disclosure includes: an oven body, including an oven cavity having one or more heating elements and a selectively openable door; an imaging device, coupled to the oven body, such that the imaging device is configured to generate an image of the interior of the oven cavity; and an electronic controller, configured to selectively control the one or more heating elements to heat the oven cavity, the electronic controller including processing logic configured to: output an identity and count of each of several food items set in the oven cavity based on one or more images of the interior of the oven cavity generated by the imaging device using one or more trained machine learning models; determine a primary food item among the multiple food items set in the oven cavity; select a cooking program based on cooking the primary food item; and cook the several food items according to the cooking program selected for the primary food item.

[0007] The various features, functions, and advantages can be achieved independently in various embodiments of the present disclosure or may be combined in yet other embodiments, further details of which can be seen with reference to the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic diagram of an illustrative machine learning model according to aspects of the present disclosure.

[0009] Figure 2is a schematic diagram of selected controller inputs and outputs of an illustrative oven according to aspects of the present disclosure.

[0010] Figure 3 is a schematic diagram of a side view of an illustrative oven according to aspects of the present disclosure.

[0011] Figure 4 is a flow chart depicting the steps of an illustrative method for identifying food items using a machine learning model in accordance with aspects of the present disclosure.

[0012] Figure 5 is a flow chart depicting the steps of an illustrative method for determining cooking programs for similar food items in accordance with aspects of the present disclosure.

[0013] Figure 6 is a flow chart depicting the steps of an illustrative method for determining cooking procedures for dissimilar food items in accordance with aspects of the present disclosure.

[0014] Figure 7 is a flow chart depicting the steps of an illustrative method for monitoring a food item being cooked, according to aspects of the present disclosure.

[0015] Figure 8 is a flow chart depicting the steps of an illustrative method for activating a self-cleaning cycle of an oven in accordance with aspects of the present disclosure.

[0016] Fig. 9 is a flow chart depicting the steps of an illustrative method for detecting debris accumulation according to aspects of the present disclosure. DETAILED DESCRIPTION

[0017] Various aspects and examples of ovens utilizing machine learning, and related methods are described below and illustrated in the associated drawings. Unless otherwise stated, an oven and / or its various components according to the present teachings may contain at least one of the structures, components, functions, and / or variations described, illustrated, and / or incorporated herein. In addition, unless specifically excluded, the process steps, structures, components, functions, and / or variations described, illustrated, and / or incorporated herein in conjunction with the present teachings may be included in other similar devices and methods, including being interchangeable between the disclosed embodiments. The following description of various examples is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or use. In addition, the advantages provided by the examples and embodiments described below are illustrative in nature, and not all examples and embodiments provide the same advantages or the same level of advantages.

[0018] The detailed description includes the following sections, which are immediately below: (1) Definitions; (2) Overview; (3) Examples, Components, and Alternatives; (4) Advantages, Features, and Benefits; and (5) Conclusion. The Examples, Components, and Alternatives section is further divided into subsections, each of which is labeled accordingly.

[0019] definition

[0020] Unless otherwise indicated, the following definitions apply herein.

[0021] "Comprising," "including," and "having" (and variations thereof) are used interchangeably to mean including but not necessarily limited to, and are open-ended terms that are not intended to exclude additional, unrecited elements or method steps.

[0022] Terms such as "first," "second," and "third" are used to distinguish or identify various members of a group or analogue and are not intended to indicate sequence or numerical limitations.

[0023] "AKA" means "also known as," and may be used to indicate alternative or corresponding terms for one or more given elements.

[0024] "Coupled" means permanently or releasably connected, whether directly or indirectly through intermediate components.

[0025] "Processing logic" describes any suitable device(s) or hardware configured to process data by performing one or more logical and / or arithmetic operations (e.g., executing coded instructions). For example, processing logic may include one or more processors (e.g., central processing units (CPUs) and / or graphics processing units (GPUs)), microprocessors, clusters of processing cores, FPGAs (field programmable gate arrays), artificial intelligence (AI) accelerators, digital signal processors (DSPs), and / or any other suitable combination of logic hardware.

[0026] A "controller" or "electronic controller" includes processing logic programmed with instructions for performing control functions with respect to a control element. For example, an electronic controller may be configured to receive an input signal, compare the input signal to a selected control value or set point value, and determine an output signal to a control element (e.g., a motor or actuator) to provide corrective action based on the comparison. In another example, an electronic controller may be configured to interface between a host device (e.g., a desktop computer, a mainframe, etc.) and a peripheral device (e.g., a memory device, an input / output device, etc.) to control and / or monitor input and output signals to and from the peripheral device.

[0027] Directional terms such as "up," "down," "vertical," "horizontal," etc. should be understood in the context of the particular object being discussed. For example, an object may be oriented about defined X, Y, and Z axes. In those examples, the XY plane would define the horizontal, with "up" defined as the positive Z direction and "down" defined as the negative Z direction.

[0028] In the context of a method, "providing" may include receiving, obtaining, purchasing, manufacturing, generating, processing, pre-processing, etc., so that the provided object or material is in a state and configuration for other steps to be performed.

[0029] In the present disclosure, one or more publications, patents and / or patent applications may be incorporated by reference. However, such materials are only incorporated to the extent that there is no conflict between the incorporated materials and the statements and drawings set forth herein. In the event of any such conflict, including any conflict in terminology, the present disclosure has control.

[0030] Overview

[0031] In general, an oven according to the present teachings utilizes an image processing system that includes an image sensor disposed within an oven cavity, and one or more machine learning (ML) models that are trained to recognize food items in images of the oven cavity taken by the image sensor. The oven matches the food items identified by the trained ML models with one or more corresponding cooking programs for the food items, and suggests a cooking program to a user of the oven. The oven uses the image processing system to monitor the food items during the cooking process in order to update the cooking program based on the observed progress of the items being cooked.

[0032] The image processing system has at least one image sensor or camera configured to capture images of the interior of the oven cavity. The image sensor can be permanently or removably coupled to the interior of the oven cavity or to the door of the oven. In some examples, the image sensor is disposed outside the oven cavity but has a field of view covering the interior of the oven cavity. The camera is configured and oriented so that the image sensor can obtain a three-dimensional (3D) view, such as a cubic view, an orthogonal view, or an oblique view of the interior of the oven cavity. For example, the image sensor can be pointed at an oblique angle relative to the general cubic interior of the oven. In some examples, the image processing system includes multiple image sensors each configured to capture images of the interior of the oven cavity.

[0033] An image of the interior of the oven cavity is input into a trained ML model, and the trained ML model detects items in the image and outputs the identity of one or more food items detected in the image. The trained ML model can identify the cookware containing or holding the food items and any foreign objects within the oven cavity (i.e., items that the ML model does not identify as food items or cooking vessels). The oven uses information about the identity of the food items in the oven cavity and / or the (one or more) cooking vessels containing the food items to determine an appropriate cooking program for the food items. Determining a cooking program for a food item may include determining the following for the food item: a cooking mode (baking, broiling, grilling, etc.), a cooking temperature, a cooking duration, a time-based cooking algorithm (e.g., a temperature profile that changes over time), and / or a condition-based cooking algorithm (e.g., with a step-by-step triggering criteria such as a browning level). In some cases, such as when a covered cookware is used, the food items within the cookware are not visible to the ML model for recognition. In these cases, the oven can identify the cookware and store information about food items that are typically cooked in a specific covered cookware. A cooking program may then be suggested based on the food items typically contained in the covered cookware.

[0034] The image processing system is configured to identify the food items present in the oven and determine the number or count of each individual food item. In some examples, the image processing system is configured to distinguish between similar and dissimilar food items in the oven cavity. "Similar" or "same" food items include, for example, multiple (e.g., three) chicken breasts or two potatoes (e.g., one small and one large). "Dissimilar" or "different" food items include, for example, steak and potatoes or chicken breasts and two corn cobs. This allows the user to place multiple identical food items and / or different food items into the oven cavity and bake or cook them at the same time. The image processing system and the ML model determine the identity and count of each item, as well as the appropriate cooking procedure for each item. If dissimilar food items are placed into the oven cavity at the same time for cooking, the image processing system can identify each food item individually and determine which of them are primary food items (e.g., food items that may cause illness if not cooked, and / or food items that require long cooking times), and which of them are secondary food items (e.g., food items that are not dangerous if not cooked and / or require less cooking time than primary food items). In this example, the oven determines a cooking program for the primary food item and determines a completion time for the secondary food item based on the cooking program for the primary food item.

[0035] In some examples, the image processing system determines the quantity based on the mass and / or volume of the individual food items in the oven cavity and uses this information to determine the appropriate cooking procedure for the given food item. To facilitate this feature, a reference geometric object (RGO) (also referred to as a reference object and / or a geometric reference object) is disposed within the oven cavity to provide the system with a reference having known dimensions. Using the known dimensions of the RGO in the field of view of the image sensor and the apparent dimensions of the RGO, the image processing system determines the precise location of the RGO within the oven cavity. The RGO may be mounted on a shelf of the oven, a dedicated cooker mounted to the oven, and / or any other suitable surface within the oven cavity such that the RGO is disposed adjacent to any food item to be cooked in the oven cavity. In some examples, the oven uses the known distance between the grilles in the oven shelf as the RGO. In some examples, the RGO is a known object mounted at a known location (e.g., on a shelf of the oven), and the oven shelf has a limited set of known locations within the oven cavity such that the image processing system determines the oven shelf position based on the apparent dimensions of the RGO in the field of view of the image sensor. By imaging the RGO and comparing the known size and position of the RGO with the apparent size and position of (one or more) food items within the field of view of the image sensor, the image processing system estimates the volume of each food item. The oven uses the determined volume and identity of the food item to estimate the mass of the food item. For example, the mass of the food item may be useful in determining the cooking time and cooking temperature of the food item. In some examples, the oven includes a mass sensor that is configured to directly measure the mass of the food item.

[0036] The oven may be configured to request from the user their (one or more) completion preferences for food items, which are also referred to herein as one or more completion criteria. The oven uses the identity and quality of the food items to determine an appropriate cooking program that will achieve the user's (one or more) completion preferences. The completion preferences may include a level of doneness (e.g., rare, medium, well done), a level of browning, a specific temperature of the food item, and / or any other suitable indicator configured to measure the progress of the food item being cooked. If multiple food items are identified in the oven cavity, the oven may request the user's completion preferences for each food item separately.

[0037] After the food item begins cooking, the oven can monitor the cooking progress of the food item. Throughout the cooking process, some food items experience predictable changes in volume, quality and / or browning level. The image processing system is configured to measure or detect the browning level, volume and / or quality of the food item and detect the changes throughout the cooking process. In some examples, the oven has a temperature sensor or temperature probe that is configured to be inserted into the food item to monitor the temperature changes of the food item throughout the cooking process. Based on the changes in the browning level, volume, quality and / or temperature of the food item during the cooking process, the cooking temperature and / or cooking time of the food item can be dynamically and automatically adjusted.

[0038] In some examples, the live images provided by the image sensor during the cooking process can be displayed on a user interface (UI) provided on the oven and / or on a user's mobile device. In some examples, the oven includes a lighting element configured to provide illumination within the oven cavity so that a user of the oven can accurately determine the browning level or color of the (one or more) food items in the live image of the oven cavity. Displaying the live color image of the (one or more) food items during the cooking process allows the user of the oven to view the cooking progress of the (one or more) food items and determine when the (one or more) food items are cooked to the user's done preference without opening the oven door.

[0039] In some examples, the image processing system and the trained ML model are configured to detect foreign objects in the oven cavity. For example, the image processing system may detect an item in the oven cavity, but the item may not be recognized as a food item by the trained ML model. In these cases, the oven may be configured to take some action, such as notifying a user of the detected foreign object and / or pausing operation until the item is removed or manually identified. In some examples, the oven stores a list of items left in the oven after a previous cooking program and notifies the user of these items before starting a subsequent cooking program. In some examples, if a foreign object in the oven cavity has been detected, the oven will not start a cooking program until the foreign object is removed or otherwise confirmed by the user as an item to be cooked by the oven.

[0040] In some examples, the image processing system is configured to detect the accumulation of debris inside the oven cavity, such as caused by food spillage or general use of the oven. If left in place, the debris may burn and adhere to the interior of the oven cavity, causing damage to the oven. Using the image processing system and the trained ML model, the oven is configured to detect the accumulation of debris, and in response, activate a self-cleaning cycle to clean the interior of the oven. In response to detecting that the accumulation of debris is at a threshold level, the oven can automatically activate the self-cleaning cycle, and / or can notify the user of the oven that there is debris and prompt the user to activate the self-cleaning cycle. In some examples, the self-cleaning cycle includes a pyrolytic cleaning cycle, in which the interior of the oven is heated to a temperature of more than 800 degrees Fahrenheit (F). Due to the high temperatures in the oven cavity during these self-cleaning cycles, items (such as oven racks and cookware) left in the oven cavity can be deformed or otherwise damaged. In order to avoid damage to items left in the oven cavity, the image processing system and the trained ML model are configured to detect any such items before activating the self-cleaning cycle. The oven may warn the user of any items in the oven cavity and request that the items be removed before activating the self-cleaning cycle. The oven may be configured to activate the self-cleaning cycle only after the items are removed from the oven cavity. In some examples, the user is allowed to override this feature, assuming that any damage to items left in the oven cavity will be the user's responsibility.

[0041] The oven may store data for one or more machine learning modules and cooking programs locally in a memory of the oven's data processing system and / or in the cloud. The oven may also store information about user food preferences, cooking utensils and cookware typically used by the user, time elapsed since the last self-clean cycle, accumulation of debris in the oven cavity, etc.

[0042] The user of the oven can interact with the oven through a human-machine interface (HMI) or user interface (UI) provided on the oven, on an app on a mobile device, and / or in any other suitable medium. In some examples, the oven reminds the user when the food item is finished cooking and / or provides the user with a current image of the food item through an app on a mobile device, so that the user can monitor the cooking process while away from the oven.

[0043] Examples, Parts, and Alternatives

[0044] The following sections describe selected aspects of illustrative ovens and related systems and / or methods. The examples in these sections are intended to illustrate and should not be construed as limiting the scope of the present disclosure. Each section may include one or more different embodiments or examples, and / or context or related information, functions and / or structures.

[0045] A. Illustrative Machine Learning Algorithms

[0046] like Figure 1 As shown in , this section describes the training or use of an illustrative machine learning algorithm or model 100 suitable for use with the oven of the present disclosure.

[0047] In general, machine learning (ML) models (also called ML algorithms, ML tools, or ML programs) can be used to generate estimates, predictions, or decisions that are useful in themselves and / or in the service of a more comprehensive program. Based on existing sample data, ML algorithms "learn" by example and generate trained models. Using the trained model, predictions or decisions about new data can be made without explicit programming. Thus, machine learning involves algorithms or tools that learn from existing data and make predictions about new data.

[0048] Training data 102 (e.g., labeled training data) is used to build a trained ML model 100 so that the ML model can produce a desired output 104 when presented with new data 106. Typically, the ML model uses labeled training data 102, which includes values ​​of input variables and values ​​of known correct outputs, to determine relationships and correlations between variables or features 108 to produce an algorithm that maps input values ​​to outputs.

[0049] For the purpose of producing a classification or regression algorithm, supervised learning methods can be utilized. Classification algorithms are often used in situations where the target is a classification (e.g., whether a photo contains a cat or a dog). Regression algorithms are often used in situations where the target is a numerical value (e.g., the market value of a house).

[0050] Features 108 may include any suitable characteristics that can be measured and configured to provide some level of information about an input scenario, situation, or phenomenon. For example, if the goal is to provide an output related to the market value of a house, then features may include variables such as square footage, zip code, year built, lot size, number of bedrooms, etc. Although these example features are numeric, other feature types may also be included, such as strings, Boolean values, etc.

[0051] Depending on the application, different ML techniques may be used. For example, trained ML models may be generated using artificial neural networks, decision trees, support vector machines, regression analysis, Bayesian networks, genetic algorithms, random forests, and / or the like.

[0052] Based on the identified features 108 and the training data 102, a trained ML model 110 is generated by a training process 112. The trained ML model 110 can then be utilized to predict a class or determine an output value 104 based on new data 106.

[0053] In connection with the present disclosure, one or more ML models may be trained and used to detect and identify food items, cooking vessels containing food items, (one or more) foreign objects, and / or debris accumulation in an oven cavity. In some examples, the oven includes multiple ML models, each ML model being configured to detect and identify one or more different types of items in the oven cavity. In some examples, the oven may include one ML model that is configured to detect and identify several different types of items expected in the oven cavity. The training data for the one or more ML models may include labeled images of food items, common cooking vessels containing food items, non-food items, and / or debris accumulation in the oven cavity, so that the ML models are trained to recognize the presence and identity of separate food items, cooking vessels, non-food items, and / or debris accumulation in the image of the oven cavity. For example, one or more trained ML models may receive an image of the oven cavity as input and output the identity of the food items in the image of the oven cavity, the identity of the cooking vessels containing food items, the detection results of non-food items, and / or the detection results of the selected level of debris accumulation.

[0054] B. Illustrative Oven Controls

[0055] like Figure 2 As shown in the schematic diagram of , this section describes the control system of an illustrative oven 200. Oven 200 is an example of the oven described in the overview above.

[0056] Oven 200 includes oven controller 202, oven sensor 204, heating element 206, lighting element 208, oven display 210, and wireless communication module 212. Oven controller 202 communicates with oven sensor 204, heating element 206, lighting element 208, oven display 210, and wireless communication module 212, and oven controller 202 is configured to provide control outputs and / or receive information inputs from these components.

[0057] The oven controller 202 has a central processing unit 214 and a memory 216. The memory 216 stores information for the operation of the oven, and the central processing unit 214 uses the information in the memory 216 to selectively control the features of the oven 200. The memory 216 can store user information 218, one or more cooking programs 220, one or more self-cleaning programs 222, and one or more machine learning modules 224. Each machine learning module 224 includes processing logic that includes one or more machine learning models that are trained to identify and / or count food items, foreign objects, cooking utensils, and / or debris accumulation in the oven cavity. In some examples, the machine learning module 224 is configured to receive image data of the oven cavity captured by the image sensor or camera 226 and input the image data into one or more trained machine learning models. The one or more trained machine learning models output the items identified in the image of the oven cavity, and the one or more machine learning modules 224 can transmit the output to various components of the oven 200.

[0058] The oven sensor 204 may include an image sensor 226 and a temperature sensor 228. The image sensor 226 may include any device suitable for imaging the interior cavity of the oven, such as a digital camera. The temperature sensor 228 may include a temperature probe configured to be inserted into a food item in the oven and provide an internal temperature of the food item. The heating element 206 is configured to heat the oven cavity, and the lighting element 208 is configured to illuminate the oven cavity. The lighting element 208 is configured to provide lighting within the oven cavity so that a user of the oven can view an image of the food item in the oven cavity taken by the image sensor 226 and determine the browning level of the food item without opening the door of the oven.

[0059] The oven display 210 may include a user interface 230 (UI) provided on the oven. The user interface 230 is configured to display information about food items identified by the machine learning module, recommended cooking programs for the identified food items, cooking progress of the food items, etc. In some examples, the user interface 230 is a touch screen, and the user of the oven can interact with the oven through the user interface. In some examples, the user can interact with the oven using voice or gesture commands or through an app on a mobile device 232. The wireless communication module 212 is connected to an app on the user's mobile device via a network 234. In some examples, the user receives and sends information about the identified food items and the recommended cooking programs for the food items to the oven 200 through the app on the mobile device 232.

[0060] C. Illustrative Oven

[0061] like Figure 3 As shown in , this section describes an illustrative oven 300 , which is an example of the ovens described above.

[0062] Oven 300 includes oven cavity 302, oven door 304, and oven shelf 306. Figure 3 308A. However, oven shelf 306 may be located in any of positions 308B-D. Oven 300 also includes heating element 310, lighting element 312, and image processing system 314. Heating element 310 is configured to heat the interior of oven cavity 302, and lighting element 312 is configured to provide standardized lighting within oven cavity 302.

[0063] Image processing system 314 has an image sensor 316 that is configured to capture images of the interior of oven cavity 302. Figure 3 , an image sensor 316 is mounted to the interior of the oven cavity 302, but in some examples, the image sensor can be mounted to the oven door 304. The image sensor 316 has a field of view 317 and is positioned so that the image sensor 316 has a 3D field of view of the interior of the oven cavity 302. In some examples, the image processing system has multiple image sensors that are each configured to capture images of the interior of the oven cavity. Set on the oven shelf 306 are food items 318 and reference geometry objects (RGO) 320. The RGO 320 is mounted to the shelf 306 and has a known size and a limited number of known positions within the oven cavity. In some examples, the RGO is mounted to the interior surface of the cooking vessel or oven cavity. The food item 318 can be a food item, a plurality of food items, or a cooking vessel or cookware containing one or more food items.

[0064] In some examples, the image processing system 314 uses the known position and size of the RGO 320 to determine the volume of the food item 318. The image processing system 314 first determines a distance 322 to the RGO 320 and a reference angle 324 representing the height of the RGO in the field of view 317 of the image sensor 316. Because the height of the RGO is known, the image processing system can determine the oven shelf position based on the reference angle 324 and the distance to the RGO 320. Using the determined shelf position, the image processing system determines a distance 326 to the food item 318 and a reference angle 328 representing the height of the food item above the oven shelf. By comparing the distance 322 and the reference angle 324 of the RGO 320 with the distance 326 and the reference angle 328 of the food item 318, the image processing system estimates the volume of the food item 318. This information is useful in a cooking procedure involving determining an appropriate cooking procedure for the item 318.

[0065] Oven 300 includes controller 330, user interface 332, and wireless communication module 334. User interface 332 is configured to display cooking information to a user and receive input from an oven user regarding the cooking of item 318. Wireless communication module 334 is configured to connect to a network to receive information about a cooking program or to a user's mobile device. In some examples, wireless communication module 334 connects to a user's mobile device to display information about the cooking of item 318 to the user via a mobile app and receive information about the cooking of item 318 from the user.

[0066] D. Illustrative Methods for Food Item Identification

[0067] This section describes the steps of an illustrative method 400 for identifying food items within an oven cavity using an image processing system and a machine learning model; see Figure 4 Various aspects of oven 300 may be used in the method steps described below. Where appropriate, reference may be made to components and systems that may be used in performing each step. These references are for illustration purposes and are not intended to limit possible ways of performing any particular step of the method.

[0068] Figure 4 The flowcharts are flowcharts illustrating the steps performed in the illustrative method and may not describe the complete process or all steps of the method. Figure 4 Various steps of method 400 are depicted in , but not all steps need to be performed, and in some cases the steps may be performed simultaneously or in an order different from that shown.

[0069] Step 402 of method 400 includes imaging the interior of an oven cavity using an image processing system. The image processing system includes an image sensor or camera configured to capture an image of the interior of the oven cavity, and a lighting element may be used to illuminate the oven cavity.

[0070] Step 404 of method 400 includes using one or more machine learning (ML) models to identify food items, foreign objects, and / or cooking utensils in the oven cavity. The image of the interior of the oven cavity captured in step 402 of method 400 is input into one or more ML models, and the one or more ML models are trained to output the identity and / or count of any food items, foreign objects (objects in the oven cavity that are not identified as food items or cooking utensils), and / or the identity of any cooking utensils containing food items in the image. In some examples, the oven includes multiple independent ML models, each of which is trained to detect a specific item or a specific category of items in the oven cavity. In some examples, a single ML model is trained to detect several different types of items in the oven cavity.

[0071] The identified cooking vessel can be a flat baking sheet, a pot, a stew pot, etc. In some examples, the cooking vessel is not detected and the ML model only identifies the food item in the oven cavity. The one or more ML models trained to identify food items in the oven cavity are configured to identify single food items and multiple food items in the oven cavity. If multiple food items are detected in the oven cavity, the one or more ML models identify each food item separately. The one or more trained ML models can be stored locally in the oven's memory and / or stored in the cloud and connected via the oven's wireless communication module.

[0072] Step 406 of method 400 includes determining the volume and mass of each food item. The image processing system is configured to determine the volume of each food item using a reference geometric object (RGO) disposed in the oven cavity and having known dimensions. The image processing system determines the position of the RGO based on the known dimensions of the RGO and the apparent dimensions of the RGO in the field of view of the image sensor. The image processing system determines the volume of each food item in the oven cavity by comparing the position and size of the RGO in the field of view of the image sensor with the apparent position and size of each food item relative to the RGO. The mass of each item is determined using the volume of the food item and the identity of the food item determined in step 404 of method 400. The RGO may be mounted on an oven shelf, mounted on an interior surface of the oven cavity, and / or mounted at any other suitable location in the oven cavity so that the RGO is adjacent to (one or more) food items. In some examples, the RGO is the distance between the grilles in the oven shelf.

[0073] Step 408 of method 400 includes receiving confirmation of each identified food item, foreign object, and / or cooking vessel in the oven cavity. The oven may request the user of the oven to confirm that the trained ML model has accurately identified the food items, foreign objects, and cooking vessels in the oven cavity. In some examples, the oven requests confirmation of the mass and / or volume of each food item determined by the image processing system. The mass or volume may be estimated in standard units (e.g., pounds, ounces, kilograms, etc.) and / or in size categories (e.g., small, medium, large). The oven may request the user of the oven to remove the foreign objects identified in the oven cavity to avoid cooking non-food items. The oven may receive user confirmation via a user interface on the oven, and / or the oven may be wirelessly connected to the user's mobile device and receive confirmation via an app on the mobile device.

[0074] E. Illustrative methods of cooking similar food items together

[0075] This section describes the steps of an illustrative method 500 for utilizing an image processing system and a machine learning model to determine a cooking program for (one or more) similar food items identified in an oven; see Figure 5 If, for example, multiple similar food items are identified in the oven cavity in step 404 of method 400 (i.e., several food items in the oven are identified as the same type of food items, e.g., three chicken breasts), then method 500 is an extension of method 400 described above. Various aspects of oven 300 may be used in the method steps described below. Where appropriate, reference may be made to components and systems that may be used in performing each step. These references are for illustration and are not intended to limit the possible ways in which any particular step of the method may be performed.

[0076] Figure 5 The flowcharts are flowcharts illustrating the steps performed in the illustrative method and may not describe the complete process or all steps of the method. Figure 5 Various steps of method 500 are depicted in FIG. 5 , but not all steps need to be performed, and in some cases the steps may be performed simultaneously or in an order different from that shown.

[0077] Step 502 of method 500 includes receiving a doneness preference or doneness criteria from a user of the oven for each of the similar food items identified in the oven cavity. The user of the oven may input their doneness criteria for each identified food item through a user interface provided on the oven or through an app on the user's mobile device. The doneness criteria may include a desired doneness level (e.g., rare, well-done) of the food item, a desired browning level of the food item, and / or a finished temperature of the food item. For example, two steaks are identified in the oven cavity, and the oven receives a doneness preference from the user to cook one of the steaks to rare and the other steak to well-done.

[0078] Step 504 of method 500 includes determining an appropriate cooking program for similar food items. Determining a cooking program for similar food items may include determining the following for the food items: cooking mode (baking, broiling, grilling, etc.), cooking temperature, cooking duration, time-based cooking algorithm (e.g., temperature distribution that varies over time) and / or condition-based cooking algorithm (e.g., with step-by-step trigger criteria such as browning level). The oven uses the identity, quality, volume, and / or user-entered completion criteria of the food items to determine an appropriate cooking program for the food items. Based on the differences in quality, volume, and / or user-entered completion criteria of a particular food item, individual food items in the identified similar food items may require different cooking times, and the oven uses this information to determine the appropriate cooking time for each food item. The oven may be connected to a cooking program in the cloud, or the cooking program may be stored locally in a computer's memory. After determining an appropriate cooking program for the food item, the oven may request the user to confirm the cooking program before proceeding to step 506.

[0079] Step 506 of method 500 includes cooking food items according to the cooking program determined in step 504. In some examples, the oven prompts the user of the oven to insert a temperature probe into one or more food items before starting the cooking process. The temperature probe can be wired to the oven or wirelessly connected to the oven to provide information about the internal temperature of the food items during the cooking process. In some examples, the oven uses an image processing system to monitor the cooking progress of the food items. The oven can monitor the browning level, internal temperature, quality and / or volume of each individual food item. The oven can update the cooking program or completion time for the individual food items based on the observed cooking progress of the food items. In some examples, the oven displays an instant image of the food items in the oven cavity on the user interface of the oven and / or the user's mobile device, so that the user can observe the cooking progress of the food items without opening the door of the oven cavity. The user can manually update the cooking program based on the observed cooking progress of the food items and the user's completion preference for the food items.

[0080] Step 508 of method 500 includes reminding the user when each individual food item is finished cooking so that the user can remove the finished food item from the oven cavity. The oven can remind the user of the oven of the finished food item through a user interface on the oven and / or through an app on the user's mobile device. In some examples, the oven determines that the individual food item is finished cooking when the completion criteria of the measured food item matches the completion criteria entered by the user for the food item. For example, if the completion criteria entered by the user for the food item is an internal temperature of 150 degrees Fahrenheit, then when the temperature probe inserted into the food item measures the internal temperature of the food item to be 150 degrees Fahrenheit, the oven will remind the user that the food item is finished cooking. In some examples, when the browning level, volume and / or quality of the food item matches the completion criteria entered by the user for the food item, the oven reminds the user that the food item is finished.

[0081] F. Illustrative Methods for Cooking Non-similar Food Items

[0082] This section describes the steps of an illustrative method 600 for utilizing an image processing system and a trained machine learning model to determine a cooking program for (one or more) dissimilar food items in an oven; see Figure 6If multiple food items are identified in the oven cavity in step 404 of method 400, method 600 is an extension of method 400 described above when the food items are dissimilar food items (i.e., not all food items are identified as the same type of food items, e.g., chicken breasts and potato casserole). Various aspects of oven 300 may be used in the method steps described below. Where appropriate, reference may be made to components and systems that may be used in performing each step. These references are for illustration and are not intended to limit the possible ways in which any particular step of the method may be performed.

[0083] Figure 6 The flowcharts are flowcharts illustrating the steps performed in the illustrative method and may not describe the complete process or all steps of the method. Figure 6 Various steps of method 600 are depicted in FIG. 6 , but not all steps need to be performed, and in some cases the steps may be performed simultaneously or in an order different from that shown.

[0084] Step 602 of method 600 includes determining a primary food item from the dissimilar food items identified in the oven cavity. The primary food item can be a food item that is potentially dangerous to consume if undercooked, and / or a food item that has a longer cooking time relative to (one or more) other food items identified in the oven cavity. For example, chicken breast and potatoes are identified in the oven cavity, and the oven identifies the chicken breast as the primary food item. In some examples, a list of common primary food items is stored in a memory of the oven, and the oven determines the primary food item by matching one of the identified dissimilar food items with an item in the list. In some examples, the oven receives user input as to which food items in the oven cavity are primary food items.

[0085] Step 604 of method 600 includes determining an appropriate cooking program for the main food item. Because the main food item is potentially dangerous to eat when undercooked, the oven determines an appropriate cooking program for dissimilar food items in the oven based on the main food item. Basing the suggested cooking program on the main food item ensures that the main food item will be fully cooked without being dangerous when eaten. Determine the appropriate cooking program for the main food item based on the identity, quality and / or volume of the main food item, and / or based on the cooking vessel containing the main food item. In some examples, the oven receives a completion preference for the main food item from a user of the oven and uses the user's completion preference when determining an appropriate cooking program. Determining an appropriate cooking program for the main food item may include determining the following for the main food item: cooking mode (baking, broiling, grilling, etc.), cooking temperature, cooking duration, time-based cooking algorithm (e.g., temperature distribution over time) and / or condition-based cooking algorithm (e.g., with step-by-step trigger criteria such as browning level).

[0086] Step 606 of method 600 includes determining a completion time for a secondary food item (i.e., a non-primary food item) in the oven cavity based on the cooking program for the primary food item determined in step 604 of method 600. The oven determines a completion time for the secondary food item in the oven cavity based on the cooking mode and / or cooking temperature of the cooking program for the primary food item. After determining the appropriate cooking program for the primary food item and the completion time for the secondary food item, the oven may prompt a user of the oven to confirm the cooking program and completion time determined by the oven before proceeding to step 608.

[0087] Step 608 of method 600 includes cooking dissimilar food items according to the determined cooking program. In some examples, before starting the cooking process, the oven prompts the user of the oven to insert a temperature probe into the main food item. The temperature probe can be wired to the oven or wirelessly connected to the oven to provide information about the internal temperature of the main food item during the cooking process. In some examples, the oven uses an image processing system to monitor the cooking progress of dissimilar food items. The oven can update the cooking program or completion time of a separate food item based on the progress of the observed food items. In some examples, the oven displays an instant image of the oven cavity on the user interface of the oven and / or on the user's mobile device so that the user can observe the cooking progress of the food item without opening the door of the oven cavity. The user can manually update the cooking program based on the cooking progress of the observed food item and the user's completion preference for the food item.

[0088] Step 610 of method 600 includes alerting the user when the secondary food item is finished cooking. The user of the oven may remove the secondary food item from the oven cavity while continuing to cook the primary food item.

[0089] Step 612 of method 600 includes alerting the user when the primary food item has finished cooking.

[0090] G. Illustrative Methods for Monitoring Food Items During Cooking

[0091] This section describes the steps of an illustrative method 700 for monitoring a food item during its cooking process using an image processing system and a machine learning model; see Figure 7 Various aspects of oven 300 may be used in the method steps described below. Where appropriate, reference may be made to components and systems that may be used in performing each step. These references are for illustration and are not intended to limit the possible ways of performing any particular step of the method.

[0092] Figure 7 The flowcharts are flowcharts illustrating the steps performed in the illustrative method and may not describe the complete process or all steps of the method. Figure 7 Various steps of method 700 are depicted in FIG. 7 , but not all steps need to be performed, and in some cases the steps may be performed simultaneously or in an order different from that shown.

[0093] Step 702 of method 700 includes imaging a food item using an image processing system of an oven. The image processing system has an image sensor configured to capture an image of an interior of an oven cavity and a lighting element configured to illuminate the oven cavity. The lighting element is configured to provide illumination in the oven cavity so that a browning level of the food item can be determined by the image processing system and compared to browning levels of food items measured at different times.

[0094] Step 704 of method 700 includes measuring the browning level of the food item using color analysis of the image captured by the image sensor in step 702. The measured browning level is stored in a memory of the oven for comparison with the browning level of the food item later in the cooking process.

[0095] Step 706 of method 700 includes determining the volume and / or mass of the food item. As described above, the volume of the food item is determined by comparing the size and position of the food item within the field of view of the image sensor with the known size and position of a reference geometric object within the field of view of the image sensor. The mass of the food item is determined based on the volume and identity of the food item. The determined volume and / or mass of the food item is stored in a memory of the oven for comparison with the volume and / or mass of the food item later in the cooking process.

[0096] Step 708 of method 700 includes monitoring the change in browning level, quality and / or volume of the food item during the cooking process. The volume, quality and / or browning level of some food items may change throughout the cooking process. The oven may continuously or intermittently monitor the current browning level, volume and / or quality of the food item in the oven cavity and compare the current value with the value determined for the food item in steps 704 and 706 of method 700. In some examples, the oven only proceeds to step 710 of method 700 after the oven detects that the change in browning level, volume and / or quality of the food item is at a threshold level.

[0097] Step 710 of method 700 includes updating a cooking program based on a detected change in the browning level, mass, and / or volume of a food item. During the cooking process, some food items change in volume, mass, and / or browning level in a predictable manner such that the change provides information about the progress of cooking of the food item. In some examples, if the change in browning level, mass, and / or volume of a food item occurs faster or slower than expected, the oven reevaluates the cooking program for the food item. The oven may update the cooking program and / or completion time for the food item based on the changes in the food item detected during the cooking process.

[0098] H. Illustrative Method for Self-Cleaning Cycle Activation

[0099] This section describes the steps of an illustrative method 800 for activating a self-cleaning cycle of an oven; see Figure 8 Various aspects of oven 300 may be used in the method steps described below. Where appropriate, reference may be made to components and systems that may be used in performing each step. These references are for illustration and are not intended to limit the possible ways of performing any particular step of the method.

[0100] Figure 8 The flowcharts are flowcharts illustrating the steps performed in the illustrative method and may not describe the complete process or all steps of the method. Figure 8Various steps of method 800 are depicted in , but not all steps need to be performed, and in some cases the steps may be performed simultaneously or in an order different from that shown.

[0101] Step 802 of method 800 includes receiving input to activate a self-clean cycle of the oven. In some examples, the self-clean cycle includes a pyrolytic cleaning cycle in which the oven cavity is heated to a temperature in excess of 800 degrees Fahrenheit. A user of the oven may provide input to activate the self-clean cycle of the oven, the oven may automatically activate the self-clean cycle in response to detecting an accumulation of debris, and / or the oven may automatically activate the self-clean cycle after a predetermined period of time has passed since a previous self-clean cycle of the oven.

[0102] Step 804 of method 800 includes imaging the interior of the oven cavity using an image processing system of the oven. The image processing system has an image sensor configured to capture an image of the interior of the oven cavity.

[0103] Step 806 of method 800 includes identifying any items in the oven cavity that are to be removed prior to activating the self-cleaning cycle. The image of the interior of the oven cavity is input into one or more machine learning models that are trained to detect items or objects within the oven cavity. Due to the high temperatures in the oven cavity during the self-cleaning cycle, many items will deform if left in the oven cavity during the self-cleaning cycle of the oven. Items that may need to be removed prior to activating the self-cleaning cycle to avoid deformation include cookware, oven racks, and / or any foreign objects disposed in the oven cavity.

[0104] Step 808 of method 800 includes reminding the user of the oven of any items to be removed from the oven cavity before activating the self-cleaning cycle. The user may be reminded through a user interface on the oven, through an app on the user's mobile device, and / or any other suitable medium.

[0105] Step 810 of method 800 includes activating a self-cleaning cycle of the oven after removing the item from the oven. After notifying the user of the item to be removed from the oven, the oven can monitor the oven cavity using the image processing system and wait until the item is no longer detected in the oven cavity before activating the self-cleaning cycle. In some examples, the user of the oven can override the need to remove the item before activating the self-cleaning cycle, with the understanding that any damage to the item left in the oven cavity during the self-cleaning cycle is the responsibility of the user.

[0106] I. Illustrative Methods for Detecting Debris Accumulation

[0107] This section describes the steps of an illustrative method 900 for detecting the accumulation of debris in an oven cavity and activating a self-cleaning cycle; see Fig. 9 Various aspects of oven 300 may be used in the method steps described below. Where appropriate, reference may be made to components and systems that may be used in performing each step. These references are for illustration and are not intended to limit the possible ways of performing any particular step of the method.

[0108] Fig. 9 The flowcharts are flowcharts illustrating the steps performed in the illustrative method and may not describe the complete process or all steps of the method. Fig. 9 Various steps of method 900 are depicted in , but not all steps need to be performed, and in some cases the steps may be performed simultaneously or in an order different from that shown.

[0109] Step 902 of method 900 includes imaging an interior of an oven cavity using an image processing system. The image processing system has an image sensor configured to capture an image of the interior of the oven cavity.

[0110] Step 904 of method 900 includes detecting debris accumulation on the interior of the oven cavity using one or more machine learning models. The accumulation of debris is detected by inputting images captured by the image sensor into one or more machine learning models, which are trained to identify the accumulation of debris on surfaces in the interior of the oven cavity. If the debris is left undisturbed, the debris may become charred and attached to surfaces inside the oven cavity, causing permanent damage to the oven. In some examples, the machine learning model is trained to detect debris accumulation on the interior of the oven cavity above a predetermined threshold level.

[0111] Step 906 of method 900 includes identifying items to be removed from the oven cavity prior to activating a self-cleaning cycle. An image of the oven cavity is input into one or more machine learning models that are trained to detect items or objects in the image of the oven cavity. The one or more machine learning models trained to detect items or objects may be the same or different from the machine learning models used to detect debris accumulation in the oven cavity. Certain objects identified by the machine learning models may be deformed if left in the oven cavity during the self-cleaning cycle. In some examples, the oven may store a list of common items in the oven's memory that may be damaged if left in the oven cavity during the oven's self-cleaning cycle. Prior to activating the self-cleaning cycle, the oven may alert a user to the items detected by the one or more machine learning models to be removed from the oven cavity.

[0112] Step 908 of method 900 includes activating a self-clean cycle after any item detected in step 906 has been removed from the oven cavity. In some examples, the oven is configured to begin a self-clean cycle only after an item detected by the image processing system has been removed from the oven. In some examples, the user may disregard removal of items in the oven and activate a self-clean cycle, understanding that any items left in the oven cavity during the self-clean cycle and damaged are the user's responsibility.

[0113] Illustrative combinations and additional examples

[0114] This section describes additional aspects and features of the oven, presented without limitation as a series of paragraphs, some or all of which may be designated alphanumerically for clarity and efficiency. Each of these paragraphs may be combined in any suitable manner with one or more other paragraphs and / or with disclosures from elsewhere in this application (including materials incorporated by reference in cross-references). Some of the following paragraphs explicitly reference and further limit other paragraphs, providing without limitation some examples of suitable combinations.

[0115] A0. A method for determining a cooking program for a food item disposed in a cavity of an oven, comprising;

[0116] imaging the interior of the oven cavity using an image sensor;

[0117] inputting an image of the interior of the oven cavity into a machine learning (ML) model trained to detect a food item in the oven cavity and output an identity of the food item;

[0118] determining a volume of the identified food item based on the image of the interior of the oven cavity and estimating a mass of the food item based on the volume of the food item and the identity of the food item;

[0119] receiving completion preferences for cooking parameters for the food item;

[0120] using the food item identity, quality, and finishing preferences of cooking parameters to determine an appropriate cooking procedure for the food item; and

[0121] The food item is cooked according to the appropriate cooking program.

[0122] A1. The method described in paragraph A0 also includes receiving user input, wherein the user input confirms the identity of the food item determined by the machine learning model.

[0123] A2. The method of paragraph A0 or A1, further comprising monitoring a cooking parameter of the food item during the cooking process.

[0124] A3. The method of paragraph A2, wherein the cooking parameter is the level of browning of the food item.

[0125] A4. The method of paragraph A2, wherein the cooking parameter is the volume of the food item.

[0126] A5. The method of paragraph A2 further includes updating the cooking program in response to detecting a change in the cooking parameter of the food item, and cooking the food item according to the updated cooking program.

[0127] A6. A method as described in any of paragraphs A0 to A5, wherein the machine learning model is also trained to detect non-food items in the oven cavity and inhibit cooking of the food items until the non-food items are removed from the oven cavity.

[0128] A7. A method according to any of paragraphs A0 to A6, wherein the volume of the food item is determined based on the apparent geometric dimensions of the food item in the image of the interior of the oven cavity and the known geometric dimensions of a reference object arranged in the image of the interior of the oven cavity.

[0129] A8. The method described in any of paragraphs A0 to A7 further includes using a machine learning model to identify multiple similar food items in the oven cavity and receiving completion preferences for cooking parameters for each individual food item from a user of the oven.

[0130] A9. The method according to any of paragraphs A0 to A7, further comprising:

[0131] Using a machine learning model to identify dissimilar food items in an oven cavity; and

[0132] A primary food item is automatically selected from the dissimilar food items.

[0133] A10. The method of paragraph A9, further comprising determining an appropriate cooking program for the primary food item and cooking the dissimilar food items together according to the cooking program selected for the primary food item.

[0134] A11. The method of any of paragraphs A0 to A10, further comprising using the trained ML model to identify a cooking vessel containing a food item.

[0135] A12. The method of paragraph A11, further comprising determining the appropriate cooking procedure for the food item based on the identity of the food item and based on cooking properties of a cooking vessel containing the food item.

[0136] A13. The method of any of paragraphs A0 to A12 further includes using a machine learning model to detect debris accumulation on the interior of the oven cavity and activating a self-cleaning cycle of the oven in response to detecting the debris accumulation.

[0137] B0. A method for determining a cooking program for a plurality of food items disposed in an oven cavity, the method comprising;

[0138] imaging the interior of the oven cavity using an image sensor;

[0139] inputting an image of the interior of the oven cavity into a machine learning model trained to detect food items in the oven cavity and output an identity of each food item;

[0140] determining a primary food item from a plurality of food items disposed in the oven cavity;

[0141] selecting a cooking program based on cooking the primary food item; and

[0142] A plurality of food items are cooked according to a cooking program selected for the primary food item.

[0143] B1. The method of paragraph B0 further includes receiving a finishing preference for a cooking parameter of the primary food item and determining an appropriate cooking procedure based on the finishing preference for the cooking parameter.

[0144] B2. The method of paragraph B0 or ​​B1, further comprising monitoring a cooking parameter of the primary food item during the cooking process.

[0145] B3. The method of paragraph B2, wherein the cooking parameter of the primary food item is a level of browning of the primary food item.

[0146] B4. The method of paragraph B2, wherein the cooking parameter of the primary food item is the volume of the primary food item.

[0147] B5. The method of paragraph B2 further includes updating an appropriate cooking program in response to detecting a change in a cooking parameter of the primary food item, and cooking the plurality of food items according to the updated cooking program.

[0148] B6. The method of any of paragraphs B0 to B5, further comprising identifying a non-food item in the oven cavity and commencing cooking of the food item after removing the non-food item from the oven cavity.

[0149] B7. The method described in any of paragraphs B0 to B6 further includes using an image of the interior of the oven cavity to determine the volume of the main food item and estimating the mass of the main food item based on the volume and the identity of the main food item.

[0150] C0. A method for cooking a plurality of food items, the method comprising:

[0151] generating, using an image sensor, an image of an interior of an oven cavity containing a plurality of food items;

[0152] outputting an identity and a count of each food item based on the image of the interior of the oven cavity using the trained one or more machine learning models;

[0153] determining a primary food item among a plurality of food items disposed in the oven cavity;

[0154] selecting a cooking program based on cooking the primary food item; and

[0155] The plurality of food items are cooked according to the cooking program selected for the primary food item.

[0156] C1. The method according to C0 also includes receiving user preferences for cooking parameters of the main food item and determining an appropriate cooking program based on the user preferences for the cooking parameters.

[0157] C2. The method of C1, wherein the user preferences include a final internal temperature of the primary food item.

[0158] C3. The method of any of paragraphs C0 to C2, further comprising monitoring a cooking parameter of the primary food item during the cooking process.

[0159] C4. The method of C3, wherein the cooking parameter of the main food item is a browning level of the main food item.

[0160] C5. The method of C3, wherein the cooking parameter of the main food item is the volume of the main food item.

[0161] C6. The method according to C3, further comprising:

[0162] updating the cooking program in response to detecting a change in the cooking parameter of the primary food item; and

[0163] The plurality of food items are cooked according to the updated cooking program.

[0164] C7. The method of any of paragraphs C0 to C6, further comprising identifying a non-food item in the oven cavity and inhibiting cooking of the food item if the non-food item is present in the oven cavity.

[0165] C8. The method of any of paragraphs C0 to C7 further includes determining the volume of the main food item using an image of the interior of the oven cavity and estimating the mass of the main food item based on the volume and the identity of the main food item.

[0166] C9. The method of any of paragraphs C0 to C8, wherein the plurality of food items comprises more than one identical food item.

[0167] D0. An oven, comprising:

[0168] an oven body including an oven cavity having one or more heating elements and a selectively openable door;

[0169] an imaging device coupled to the oven body such that the imaging device is configured to generate an image of an interior of the oven cavity; and

[0170] an electronic controller configured to selectively control the one or more heating elements to heat the oven cavity, the electronic controller comprising processing logic configured to:

[0171] outputting, using the trained one or more machine learning models, an identity and a count of each of several food items disposed in the oven cavity based on one or more images of the interior of the oven cavity generated by the imaging device;

[0172] determining a primary food item among the plurality of food items disposed in the oven cavity;

[0173] selecting a cooking program based on cooking the primary food item; and

[0174] The several food items are cooked according to the cooking program selected for the primary food item.

[0175] D1. The oven according to D0, wherein the processing logic is further configured to:

[0176] receiving user preferences for cooking parameters for the primary food item; and

[0177] An appropriate cooking program is determined based on the user preferences for the cooking parameters.

[0178] D2. The oven of D1, wherein the user preferences include a final internal temperature of the primary food item.

[0179] D3. An oven as described in any of paragraphs D0 to D2, wherein the processing logic is further configured to monitor cooking parameters of the primary food item during the cooking process.

[0180] D4. The oven of D3, wherein the cooking parameter of the primary food item is a browning level of the primary food item.

[0181] D5. An oven according to D3, wherein the cooking parameter of the main food item is the volume of the main food item.

[0182] D6. The oven according to D3, wherein the processing logic is further configured to:

[0183] updating the cooking program in response to detecting a change in the cooking parameter of the primary food item; and

[0184] The plurality of food items are cooked according to the updated cooking program.

[0185] D7. An oven as described in any of paragraphs D0 to D6, wherein the processing logic is further configured to identify non-food items in the oven cavity and inhibit cooking of the food item if the non-food item is present in the oven cavity.

[0186] D8. The oven of any of paragraphs D0 to D7, wherein the oven further comprises a reference geometric object disposed in the oven cavity and having at least one known size and position, such that the reference geometric object is present in the image of the interior of the oven cavity; and

[0187] The processing logic is further configured to:

[0188] determining a volume of the primary food item using the image of the interior of the oven cavity;

[0189] and estimating a mass of the primary food item based on the volume and the identity of the primary food item.

[0190] D9. The method of any of paragraphs D0 to 8, wherein the plurality of food items comprises more than one identical food item.

[0191] Advantages, Features and Benefits

[0192] The various embodiments and examples of ovens described herein provide several advantages over known solutions for ovens that provide image processing systems and food recognition. For example, the illustrative embodiments and examples described herein allow an oven to use an image processing system to capture an image of the interior of an oven cavity, and input the image captured by the image processing system into a machine learning model that is trained to identify (one or more) food items, (one or more) cooking utensils, (one or more) foreign objects, and / or debris accumulation in the oven cavity based on the image of the oven cavity captured by the image processing system. Using a trained machine learning model to identify (one or more) food items, (one or more) cooking utensils, (one or more) foreign objects, and / or debris accumulation in the oven cavity facilitates the automation of various oven processes.

[0193] Furthermore, and among other benefits, the illustrative embodiments and examples described herein allow the oven to suggest a cooking program for a food item to a user of the oven based on the identity of the food item, the properties of the cooking vessel containing the food item, the volume and mass of the food item, and / or the user's finishing preferences for the food item. The oven determines a cooking program for a single food item, multiple similar food items, and / or dissimilar food items to be cooked simultaneously. Having the oven automatically determine a cooking program for an identified food item can simplify the cooking process for the user of the oven and improve the quality of the cooked food item.

[0194] In addition, and among other benefits, the illustrative embodiments and examples described herein include an oven that automatically determines an appropriate cooking program to cook dissimilar food items simultaneously in an oven cavity. In some examples, the oven identifies a primary food item from the dissimilar food items present and determines an appropriate cooking program based on the primary food item. In this example, the oven determines a completion time for a non-primary food item based on the appropriate cooking program for the primary food item. In this example, the oven alerts a user of the oven when the non-primary food item is finished cooking, and when the primary food item is finished cooking. By identifying the appropriate cooking program for the primary food item and the completion time for the secondary food item, the oven can successfully cook various combinations of dissimilar food items simultaneously.

[0195] In addition, and among other benefits, the illustrative embodiments and examples described herein include an oven that is configured to cook similar food items simultaneously with different doneness preferences for each item. For example, two steaks are identified in an oven cavity, and the oven receives a doneness preference from a user to cook one steak rare and the other steak well-done. The oven determines the appropriate cooking program and individual cooking times for each steak to cook the two steaks simultaneously based on the user's doneness preferences. The oven can alert the user when the steak to be cooked rare is finished cooking and when the steak to be cooked well-done is finished cooking. By determining a finish time for each food item individually, many similar food items can be cooked simultaneously, and each individual food item can be cooked according to different doneness preferences for the food items.

[0196] In addition, and among other benefits, the illustrative embodiments and examples described herein allow an oven to determine the volume and mass of a food item using only an image processing system. The oven determines the mass of the food item based on the volume and identity of the food item determined using an image captured by an image sensor of the image processing system. The oven does not require an additional sensor (e.g., a scale) to determine the mass of the food item. The mass of the food item is useful in determining the appropriate cooking program for the food item, and the volume of the food item is useful in monitoring changes to the food item during the cooking process. If the oven determines and displays the correct mass of the food item, the user of the oven may be more likely to trust the cooking program suggested by the oven.

[0197] In addition, and among other benefits, the illustrative embodiments and examples described herein monitor the internal temperature, browning level, and / or volume and quality of a food item during the cooking process, and update the cooking program for the food item based on the observed cooking progress of the food item. For example, if the oven detects that the internal temperature of the food item is increasing faster than expected, the oven can update the cooking program to lower the temperature of the oven, and / or reduce the expected cooking time of the food item. Updating the cooking program based on the observed cooking progress of the food item ensures that the food item is cooked properly, and can improve the quality of the cooked food item.

[0198] In addition, and among other benefits, the illustrative embodiments and examples described herein allow the oven to detect a foreign object or an object in the oven cavity that is unlikely to need to be cooked. If a foreign object is detected in the oven cavity, the oven may not start a cooking program and / or a self-cleaning cycle. In some examples, the oven prompts the user of the oven to remove the foreign object before starting the cooking program or the self-cleaning cycle. Inhibiting the start of a cooking program and / or a self-cleaning cycle before the foreign object is removed from the oven can improve the safety of the oven and avoid damage to the foreign object.

[0199] Furthermore, and among other benefits, the illustrative embodiments and examples described herein allow an oven to detect debris accumulation on interior surfaces of an oven cavity. In response to detecting debris accumulation in the oven cavity, the oven can prompt a user of the oven to activate a self-cleaning cycle, and / or automatically activate a self-cleaning cycle before the debris becomes charred and adheres to interior surfaces of the oven cavity and causes permanent damage.

[0200] There is no known system or device that can perform these functions.However, not all embodiments and examples described herein provide the same advantages or the same degree of advantages.

[0201] in conclusion

[0202] The present disclosure set forth above may include multiple different examples with independent utility. Although each of these embodiments is disclosed in its (one or more) preferred form, the specific embodiments thereof disclosed and illustrated herein should not be considered to have a restrictive meaning, because many variations are possible. To the extent that the various section titles are used within the present disclosure, such titles are only used for organizational purposes. The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various elements, features, functions and / or properties disclosed herein. The following claims specifically point out certain combinations and sub-combinations that are considered to be novel and non-obvious. In an application claiming priority to the present application or a related application, other combinations and sub-combinations of features, functions, elements and / or properties may be claimed. Such claims, whether or not they are broader, narrower, equivalent or different in scope than the original claims, are also considered to be included in the subject matter of the present disclosure.

Claims

1. A method for cooking a plurality of food items, the method comprising: generating, using an image sensor, an image of an interior of an oven cavity containing a plurality of food items; outputting an identity and a count of each food item based on the image of the interior of the oven cavity using the trained one or more machine learning models; determining a primary food item of the plurality of food items disposed in the oven cavity; selecting a cooking program based on cooking the primary food item; as well as The plurality of food items are cooked according to the cooking program selected for the primary food item.

2. The method of claim 1, further comprising receiving user preferences for cooking parameters of the primary food item and determining an appropriate cooking program based on the user preferences for the cooking parameters.

3. The method of claim 2, wherein the user preferences include a final internal temperature of the primary food item.

4. The method of claim 1 further comprising monitoring the cooking parameters of the primary food item during a cooking process.

5. The method of claim 4, wherein the cooking parameter of the primary food item is a level of browning of the primary food item.

6. The method of claim 4, wherein the cooking parameter of the primary food item is a volume of the primary food item.

7. The method according to claim 4, further comprising: updating the cooking program in response to detecting a change in the cooking parameter of the primary food item; as well as The plurality of food items are cooked according to the updated cooking program.

8. The method of claim 1 further comprising identifying a non-food item in the oven cavity and inhibiting cooking of the food item if the non-food item is present in the oven cavity.

9. The method of claim 1 further comprising determining a volume of the primary food item using the image of the interior of the oven cavity and estimating a mass of the primary food item based on the volume and the identity of the primary food item.

10. The method of claim 1, wherein the plurality of food items comprises more than one identical food item.

11. An oven, comprising: an oven body including an oven cavity having one or more heating elements and a selectively openable door; an imaging device coupled to the oven body such that the imaging device is configured to generate an image of an interior of the oven cavity; as well as an electronic controller configured to selectively control the one or more heating elements to heat the oven cavity, the electronic controller comprising processing logic configured to: outputting, using the trained one or more machine learning models, an identity and a count of each of several food items disposed in the oven cavity based on one or more images of the interior of the oven cavity generated by the imaging device; determining a primary food item among a plurality of food items disposed in the oven cavity; selecting a cooking program based on cooking the primary food item; as well as The several food items are cooked according to the cooking program selected for the primary food item.

12. The oven of claim 11, wherein the processing logic is further configured to: receiving user preferences for cooking parameters for the primary food item; and An appropriate cooking program is determined based on the user preferences for the cooking parameters.

13. The oven of claim 12, wherein the user preferences include a final internal temperature of the primary food item.

14. The oven of claim 11, wherein the processing logic is further configured to monitor cooking parameters of the primary food item during a cooking process.

15. The oven of claim 14, wherein the cooking parameter of the primary food item is a level of browning of the primary food item.

16. The oven of claim 14, wherein the cooking parameter of the primary food item is a volume of the primary food item.

17. The oven of claim 14, wherein the processing logic is further configured to: updating the cooking program in response to detecting a change in the cooking parameter of the primary food item; and The plurality of food items are cooked according to the updated cooking program.

18. The oven of claim 11, wherein the processing logic is further configured to identify a non-food item in the oven cavity and to inhibit cooking of the food item if the non-food item is present in the oven cavity.

19. The oven of claim 11, wherein the oven further comprises a reference geometric object disposed in the oven cavity and having at least one known dimension and position such that the reference geometric object is present in the image of the interior of the oven cavity; and The processing logic is further configured to: determining a volume of the primary food item using the image of the interior of the oven cavity; and estimating a mass of the primary food item based on the volume and the identity of the primary food item.

20. The method of claim 11, wherein the plurality of food items comprises more than one identical food item.

Images