System and method for monitoring and management of comestible preparation

Abstract

A kitchen vision system for a restaurant includes a camera configured to record image data of a workstation and one or more processing circuits including one or more memory devices coupled to one or more processors. The one or more memory devices are configured to store instructions that, when executed by the one or more processors, cause the one or more processors to receive an indication of a food product to be prepared, determine an ingredient required to produce the food product, and determine, based on the image data, whether the ingredient has been retrieved from a storage portion of the workstation.

Description

SYSTEM AND METHOD FOR MONITORING AND MANAGEMENTOF COMESTIBLE PREPARATIONCROSS-REFERENCE TO RELATED PATENT APPLICATION

[0001] This application claims the benefit of and priority to U.S. Provisional Application No. 63 / 652,116, filed on May 27, 2024, the entire disclosure of which is hereby incorporated by reference herein.BACKGROUND

[0002] The present disclosure relates generally to systems for food preparation. More specifically, the present disclosure relates to systems for facilitating accurate preparation of desired food products. In restaurants, it is desirable to quickly and accurately prepare food products according to a customer’s desired order.SUMMARY

[0003] At least one embodiment relates to a kitchen vision system for a restaurant includes a camera configured to record image data of a workstation and one or more processing circuits including one or more memory devices coupled to one or more processors. The one or more memory devices are configured to store instructions that, when executed by the one or more processors, cause the one or more processors to receive an indication of a food product to be prepared, determine an ingredient required to produce the food product, and determine, based on the image data, whether the ingredient has been retrieved from a storage portion of the workstation.

[0004] Another embodiment relates to a method of preparing a food product within a kitchen. The method comprising receiving an indication of the food product to be prepared, determining an ingredient required to produce the food product, and determining, based on image data from a camera that observes a workstation within the kitchen, whether the ingredient has been retrieved from a storage portion of the workstation.

[0005] Another embodiment relates to a non-transitory computer readable medium configured to store instructions, which, when executed by a processor, cause the processor to receive an indication of a food product to be prepared, determine an ingredient required to produce the food product, and determine, based on image data from a camera that observes a workstation within a kitchen, whether the ingredient has been retrieved from a storage portion of the workstation.

[0006] This summary is illustrative only and is not intended to be in any way limiting. Other aspects, inventive features, and advantages of the devices or processes described herein will become apparent in the detailed description set forth herein, taken in conjunction with the accompanying figures, wherein like reference numerals refer to like elements.BRIEF DESCRIPTION OF THE FIGURES

[0007] The disclosure will become more fully understood from the following detailed description, taken in conjunction with the accompanying figures, wherein like reference numerals refer to like elements, in which:

[0008] FIG. l is a block diagram of a kitchen vision system, according to an exemplary embodiment.

[0009] FIG. 2 is a block diagram of a control system for the kitchen vision system of FIG. 1, according to an exemplary embodiment.

[0010] FIG. 3 is a perspective view of the kitchen vision system of FIG. 1, according to an exemplary embodiment.

[0011] FIG. 4 is image data including an overlaid interface and illustrating an appliance being used to cook various ingredients, according to an exemplary embodiment.

[0012] FIGS. 5-11 are images including an overlaid interface and illustrating a user preparing a chicken sandwich, according to an exemplary embodiment.

[0013] FIGS. 12-25 are images including an overlaid interface and illustrating a user preparing a chicken sandwich, according to an exemplary embodiment.

[0014] FIGS. 26-30 are a graphical user interfaces produced by the kitchen vision system of FIG. 1, according to various exemplary embodiments.DETAILED DESCRIPTION

[0015] Before turning to the figures, which illustrate certain exemplary embodiments in detail, it should be understood that the present disclosure is not limited to the details or methodology set forth in the description or illustrated in the figures. It should also be understood that the terminology used herein is for the purpose of description only and should not be regarded as limiting.

[0016] Referring generally to the figures, a system is provided for monitoring and management of consumable products, such as comestible goods. In particular, the depicted exemplary systems and exemplary methods can aid in monitoring and management of preparation of comestibles including food and beverage items. An area where such goods are prepared can be a kitchen or portion thereof, for example. It should be understood that the terms “consumable,” “comestible,” “edible,” “food,” “beverage,” and similar terms are used in a broad sense. To the extent that exemplary embodiments in the present disclosure focus on food for consumption by human patrons, it should be understood that the disclosure is not so limited and can encompass other items (e.g., novelty items for pets).

[0017] A kitchen vision system is shown and described according to an exemplary embodiment. The kitchen vision system may be used with or include in a food preparation environment, such as a kitchen for a restaurant (e.g., a fast-food restaurant). The kitchen vision system may be used when preparing any type of food or drink (e.g., sandwiches, wraps, steaks, noodle dishes, coffee, ice cream, shakes, alcoholic beverages, etc.). The kitchen vision system may monitor a food preparation area (e.g., a table or a grill) using a camera and provide input, guidance, instructions or other feedback to a user (e.g., an employee) based on the image data for order accuracy and training purposes.

[0018] A camera and a user interface are provided at a food preparation area. The camera may be positioned above the food preparation area to capture image data of the food preparation area. By way of example, the camera may be fixed in position above the foodpreparation area. The camera may utilize visible light (e.g., light within a wavelength range visible to the human eye). Additionally or alternatively, the camera may utilize infrared light to determine temperatures of ingredients in the food preparation area (e.g., to determine if a particular component of a product, e.g., a patty, a bun, etc., has finished cooking).

[0019] The user interface may be or include projector that projects light to form an image within the food preparation area. Beneficially, a projector may be positioned away from the food preparation area to avoid physical contact with the food products. Additionally, a projector may be able to project onto other objects. By way of example, the projector may project a colored light onto a specific ingredient to indicate that that ingredient should be added.

[0020] A controller (e.g., including a processor and a memory) receives image data from the camera. The controller may perform image recognition to identify items within the food preparation area, as well their shape, size, position and / or orientation. The controller may identify ingredients (e.g., buns, vegetables, proteins, condiments, etc.), cooking implements (e.g., tools such as spatulas or tongs), users (e.g., a gloved hand of a user assembling a food product), and / or other items within the food preparation area.

[0021] The controller may monitor a storage area within the food preparation area. Additionally or alternatively, cameras may be used in food storage areas (e.g., freezers, refrigerators, pantries, etc.) to monitor the amount of ingredients stored in those areas. The storage area may include bins of ingredients. The controller may identify which ingredients are contained within each bin. The positions of the bins may be fixed (e.g., predetermined and provided to the controller) or movable. The controller may identify a quantity of the ingredients contained within each bin. Upon identifying that an ingredient is running low (e.g., the amount of an ingredient falls below a threshold), the controller may prompt preparation and / or gathering of additional ingredients. This may include swapping the bin for a fresh or full bin, prompting a user (e.g., an employee) to begin preparing (e.g., cutting, frying, etc.) more of the ingredient, and / or automatically ordering (e.g., purchasing) additional ingredients for delivery to the restaurant.

[0022] In some embodiments, the controller monitors removal of ingredients from the storage areas. The controller may monitor an amount of each ingredient that is used. The controller may analyze the use of each ingredient over time. By way of example, the controller may determine whether the user is adding more or less than a desired amount of the ingredient and take corrective action. By way of another example, the controller may estimate when a bin will be emptied based on the current use rate.

[0023] In some embodiments, the controller monitors a food product being assembled within the food preparation area. Using the image data from the camera, the controller may track which ingredients have been added to the food product being assembled. The controller may receive an order (e.g., from a point of sale system) and determine based on the order which ingredients should be added to the food product being assembled. By comparing the ingredients that have been added with the ingredients from the order, the controller may determine a current stage of the assembly process of the food product. The controller may indicate (e.g., through the user interface) which ingredients still have to be added before the food product is complete.

[0024] In some embodiments, the controller varies the timing of instructions to facilitate completing one or more food products at a desired time. By way of example, if an order is received that contains two items having different preparation times, the controller may delay (e.g., cause an elapse in transmission) the instructions to prepare the item having the shorter cooking time to facilitate completing both items simultaneously. By way of another example, the controller may predict a high-demand or congestion period when a large number of orders will be received (e.g., based on historical order data). The high-demand time period may be, for example, a breakfast “rush hour” period, a lunch period, etc., when there are a relatively higher number of orders than other times of the day. In advance of the high-demand time period, the controller may initiate preparation of one or more food products to have the food products ready for predicted orders within that period.

[0025] If the controller identifies an ingredient that has been added in error (e.g., an ingredient that is not part of the order for the desired product), the controller may indicate that an error has occurred and instruct a user (e.g., through the user interface) to correct the error. The controller may instruct the user to remove a particular ingredient. Alternatively,the controller may instruct the user to dispose of (e.g., trash) the current food product and remake the food product completely. In response to determining that the current food product has to be remade, the controller may determine if additional ingredients need to be prepared to complete the order (e.g., if more vegetables need to be cut, additional proteins need to be grilled, etc.). The controller may instruct another user to begin preparing any additional ingredients needed. By way of example, in response to a determination that mustard has been mistakenly added to a hamburger, an expediter may be ordered to dispose of the mistakenly-created food product, a grill cook may be ordered to grill another burger patty, and an assembler may be instructed to assemble another hamburger.

[0026] In some embodiments, the controller identifies the specific employee performing a given action (e.g., based on facial recognition, badge scans or swipes, key card data, employee logins, employee schedules, etc.). The controller may monitor the performance of each employee individually. By way of example, the controller may determine if the employee typically dispenses more or less than the desired amount of a specific ingredient. By way of another example, the controller may track how many errors (e.g., ingredients added in error, overcooked food products, etc.) are made by each employee. For example, the controller may monitor and record how many “re-dos” are required, e.g., over a given time period and / or for a specific product. The controller may assign each employee a score indicative of past or current performance and / or indicative of whether the employee is predicted to be able to carry out a future task in accordance with a performance expectation. In some embodiments, the scores are assigned for specific tasks or stations. The controller may assign each employee a task based on the likelihood of that employee performing the task in a satisfactory manner. By way of example, if an employee has a history of errors on the grill, the controller may instead assign that employee to an assembly station.

[0027] In some embodiments, the kitchen vision system is used in a bar application to monitor serving of alcoholic drinks (e.g., liquor, wine, beer, etc.). The controller may utilize image data from the camera to monitor bottles of alcohol. Based on the image data, the controller may identify which liquid are being served by a bartender (e.g., based on the labels of the bottles held by the bartender, based on which beer taps are pulled by the bartender, etc.). The controller may monitor the amount of liquid dispensed (e.g., based onthe amount of time that a bottle is determined to be inverted, based on a pour rate, an amount based on the fill level of a glass, etc.). The amount of liquid dispensed may be used to control portion sizes (e.g., by instructing the bartender to stop pouring when the correct amount of liquid has been dispensed). The amount of liquid dispensed may be used to automatically add the correct volume of alcohol onto a customer’s bill.Kitchen Vision System

[0028] Referring to FIGS. 1 and 2, a kitchen vision system, kitchen management system, kitchen tracking system, or camera system is shown as kitchen vision system 10 according to an exemplary embodiment. The kitchen vision system 10 may utilize image data to monitor the production of food products within a food preparation environment. Based on the image data, the kitchen vision system 10 may monitor the storage, production, and distribution of food products. The kitchen vision system 10 may provide guidance to personnel (e.g., employees) to facilitate accurate and timely production of the food products, may facilitate inventory management (e.g., ordering and preparation of ingredients), and may generate operational statistics and performance metrics to facilitate management of the food preparation environment.

[0029] As shown in FIG. 1, the kitchen vision system 10 is utilized with (e.g., utilized within, includes, etc.) a food preparation environment, shown as kitchen 20. The kitchen 20 may be a portion of a restaurant, bar, or other environment that prepares food products (e.g., food and / or beverages). The kitchen 20 may produce food products for nearby consumption (e.g., served within a sit-down restaurant) or may produce food products for external consumption (e.g., through a drive through or delivery service. In some embodiments, the kitchen 20 is part of a fast-food restaurant.

[0030] As used herein, the term “food product” may include any type of food and / or beverage. The kitchen 20 may produce one or more different food products (e.g., sandwiches, wraps, steaks, noodle dishes, coffee, ice cream, shakes, alcoholic beverages, etc.). The kitchen 20 may offer a set menu of food products or may produce food products upon request by a customer. The types of food products produced may vary based on the time of day (e.g., providing different breakfast and dinner menus), based on the time of year(e.g., serving cold beverages during the summer and hot beverages during the winter), based on the availability of ingredients (e.g., offering seasonal items), or for other reasons.

[0031] As shown in FIG. 1, the kitchen 20 includes a variety of different areas, each equipped to facilitate performance of a particular task. As shown, the kitchen 20 includes a pantry, larder, storeroom, back of house, cold storage, or dry storage, shown as storage area 22. The storage area 22 may store ingredients in bulk to facilitate other operations of the kitchen 20. The kitchen 20 includes a food production area, processing area, cooking area, or work area, shown as preparation area 24. The preparation area 24 may facilitate one or more personnel processing (e.g., cooking, slicing, chopping, combining, etc.) ingredients from the storage area 22 in preparation for final assembly into a desired food product. The kitchen 20 further includes a food preparation area, food production area, or final assembly area, shown as assembly area 26. The assembly area 26 may facilitate one or more personnel assembling, arranging, or otherwise combining one or more ingredients (e.g., ingredients from the storage area 22, ingredients prepared in the preparation area 24, etc.) to produce a desired food product.

[0032] FIG. 1 illustrates one potential layout of the storage area 22, the preparation area 24, and the assembly area 26. In other embodiments, the kitchen 20 is otherwise arranged. In yet other embodiments, two or more of the areas are combined (e.g., the functionalities of the preparation area 24 and the assembly area 26 are combined into a single area).

[0033] As shown in FIG. 1, the storage area 22 includes one or more food storage devices 30 for storing ingredients (e.g., in bulk). The one or more food storage devices 30 include one or more shelves, racks, pantries, cabinets, cellars, baskets, drums, jugs, or other containers, shown as room-temperature storage 32. The room -temperature storage 32 may store ingredients at room temperature (e.g., in an un-insulated sealed container, in fluid communication with the surrounding atmosphere, etc.). The room-temperature storage 32 may be used to store ingredients that are not temperature-sensitive, such as buns, vegetables, fruits, grains, or canned goods.

[0034] As shown in FIG. 1, the one or more food storage devices 30 further includes one or more temperature-controlled storage devices, shown as refrigerators 34 and freezers 36.The refrigerators 34 and the freezers 36 are each configured to store ingredients at a temperature other than room temperature (e.g., below room temperature). In some embodiments, the refrigerators 34 store ingredients at a temperature between room temperature and freezing, and the freezers 36 store ingredients at a temperature below freezing. When use of the ingredients from the freezers 36 is desired, the ingredients may be moved to the refrigerators 34 to permit the ingredients to thaw.

[0035] As shown in FIGS. 1 and 3, the preparation area 24 includes one or more work areas or workstations, shown as preparation workstations 40, that facilitate preparing one or more ingredients for use in a food product. Preparation may include cooking (e.g., grilling, frying, baking, etc.), chopping, slicing, blending, mixing, or other actions. Each preparation workstations 40 may be operated by one or more personnel. The prepared ingredients may be stored for later use in desired food products.

[0036] As shown in FIGS. 1 and 3, the preparation workstations 40 include one or more cooking implements, shown as appliances 42, that facilitate one or more preparation actions. By way of example, the one or more appliances 42 may include grills, griddles, ovens, blenders, microwaves, fryers, or other types of appliances. The one or more appliances 42 may have one or more cooking surfaces 44. The one or more appliances 42 may heat the one or more cooking surfaces 44 such that the ingredients are heated for cooking when in contact with the one or more cooking surfaces 44. By way of example, the appliances 42 may include a griddle having a heated flat top surface that serves as the one or more cooking surfaces 44.

[0037] As shown in FIGS. 1 and 3, the preparation workstations 40 may include one or more support surfaces, shown as work surfaces 46, that facilitate one or more preparation actions. The work surfaces 46 may include flat or curved surfaces that support ingredients when performing one or more food preparation actions or between preparation actions. By way of example, the work surfaces 46 may be or include a countertop that supports ingredients (e.g., directly or within another container, such as a bowl) when chopping, mixing, or performing another preparation action.

[0038] As shown in FIGS. 1 and 3, the preparation area 24 and the assembly area 26 may each include one or more refuse storage areas, shown as waste receptacles 48. The waste receptacles 48 may be or include any container that stores items (e.g., ingredients, food products, containers, etc.) that are no longer desired and should be disposed of (e.g., refuse). By way of example, the waste receptacles 48 may store trimmings of ingredients, food products made incorrectly, or containers that were used to package an ingredient. The waste receptacles 48 may include trash cans, trash bins, or other containers. The waste receptacles 48 may identify a particular type of refuse that should be stored within a particular waste receptacle 48 (e.g., to facilitate recycling or composting).

[0039] As shown in FIGS. 1 and 3, the preparation area 24 includes one or more work areas or workstations, shown as assembly workstations 50, that facilitate assembling one or more ingredients to form a desired food product. The ingredients used may have previously been prepared on a preparation workstation 40. The prepared ingredients may be combined or assembled according to a recipe to form the desired food product. Each assembly workstation 50 may be operated by one or more personnel.

[0040] As shown in FIGS. 1 and 3, the assembly workstations 50 each include one or more containers, shown as storage bins 52. Each storage bin 52 is configured to contain a volume of a prepared ingredient. The storage bins 52 may facilitate keeping a quantity of each prepared ingredient at hand to facilitate assembly of the desired food products. The storage bins 52 may each contain a different ingredient, or one or more of the storage bins 52 may contain the same ingredient (e.g., when a particular ingredient is used frequently or in large amounts). The storage bins 52 may be arranged in a grid pattern (e.g., in one or more rows or columns) to facilitate locating storage bins 52 corresponding to particular ingredients. In some embodiments, the storage bins 52 are removable from the assembly workstations 50 (e.g., to facilitate swapping an empty storage bin 52 with a full one). Storage bins 52 may be filled with prepared ingredients by personnel working in the preparation area 24 and transported to the assembly workstations 50.

[0041] As shown in FIGS. 1 and 3, the assembly workstations 50 may include one or more support surfaces, shown as work surfaces 54, that facilitate one or more assembly actions. The work surfaces 54 may be similar to the work surfaces 46. The work surfaces54 may include flat or curved surfaces that support ingredients being assembled to form a desired food product. By way of example, the storage bins 52 may be arranged in a grid that extends along the work surface 54. The work surfaces 54 may be or include a countertop that supports a food product being assembled at multiple points along a length of the work surface 54, such that the food product can be moved in front of particular storage bins 52 to minimize the distance between the food product and the storage bins 52 (e.g., to prevent accidental dropping of ingredients at a position between the storage bin 52 and the food product being assembled).

[0042] The assembly workstations 50 each further include one or more staging areas or storage areas, shown as holding areas 56. The holding areas 56 may be used to store desired food products that have been assembled but are not yet packaged for delivery to a customer. By way of example, the holding areas 56 may be used to store a quantity of hamburgers that have been assembled but have not yet been bagged to fulfill a customer order. The holding areas 56 may include shelves, bins, or other components that facilitate temporarily storing the completed food products. In some embodiments, the holding areas 56 include temperature control devices (e.g., heat lamps, heated trays, refrigerated containers, etc.) that facilitate holding the completed food products at a desired serving temperature.

[0043] The assembly area 26 further includes one or more staging areas or storage areas, shown as delivery areas 58. The delivery areas 58 may include desired food products that have been completed and packaged as specified in a customer’s order. The delivery areas 58 may serve to hold completed orders before they are picked up by a customer. The customer may retrieve the completed order from a delivery area 58 themselves, or personnel may hand the completed order from a delivery area 58 to a customer.

[0044] Referring to FIG. 2, the kitchen vision system 10 includes a control system 100 that controls operation of the kitchen vision system 10. The control system 100 may be completely contained within a restaurant or may be distributed between multiple locations (e.g., a restaurant and a remote data center).

[0045] As shown in FIG. 2, the control system 100 includes a local controller or processing circuitry, shown as system controller 110, that controls operation of the control system 100. The system controller 110 includes at least one processing circuit, shown as processor 112, and at least one memory device, shown as memory 114. The memory 114 may contain one or more instructions that, when executed by the processor 112, cause the system controller 110 to perform one or more of the operations (e.g., processes and methods) discussed herein. By way of example, the memory 114 may be or include a non- transitory computer readable medium configured to store instructions, which, when executed by the processor 112, cause the processor 112 to perform the various operations discussed herein. The system controller 110 may be a local controller that is located within or nearby the kitchen 20. Accordingly, the system controller 110 may be capable of controlling operation of the kitchen vision system 10 even if an external network connection (e.g., to the Internet) degrades or fails (e.g., so as to result in at least one period of dysconnectivity or reduced connectivity).

[0046] The control system 100 includes one or more image sensors, shown as cameras 120, operatively coupled to the system controller 110. The cameras 120 each provide image data (e.g., pictures, video, etc.) capturing a field of view of the camera 120. The image data may show or contain people (e.g., personnel), equipment (e.g., the preparation workstations 40, the assembly workstations 50, knives, bowls, appliances, etc.), ingredients, food products, or other items. The cameras 120 may provide the image data in color, black and white, or other colors. In some embodiments, the cameras 120 generate the image data using visible light. In other embodiments, the cameras 120 generate the image data using lights non-visible portions of the electromagnetic spectrum (e.g., infrared light).

[0047] As shown in FIGS. 1 and 3, the cameras 120 may be strategically positioned throughout the kitchen 20 to provide image data showing particular features. As shown in FIGS. 1 and 3, the control system 100 includes cameras 120 positioned within the storage area 22, the preparation area 24, and the assembly area 26. Each of the cameras 120 may observe a different part of the food production process, depending upon where the camera 120 is located and oriented.

[0048] The cameras 120 within the storage area 22 may monitor inventory (e.g., ingredients) within the one or more food storage devices 30. By way of example, the cameras 120 may view ingredients stored within the room-temperature storage 32, the refrigerators 34, and / or the freezers 36. In some embodiments, one or more of the cameras 120 are positioned within the one or more food storage devices 30 to facilitate the field of view of the cameras 120 having an uninterrupted path (line of sight) to the ingredients.

[0049] The cameras 120 within the preparation area 24 may monitor the preparation of ingredients for food service. By way of example, the cameras 120 may monitor the personnel, the appliances 42, the ingredients, the waste receptacles 48, or other items within the preparation area 24. The cameras 120 may be positioned such that a field of view of at least one camera 120 includes the cooking surfaces 44 and the work surfaces 46 to monitor the preparation of ingredients. The cameras 120 may be positioned such that a field of view of at least one camera 120 includes the waste receptacle 48 to monitor handling and / or waste of ingredients. In some embodiments (e.g., as shown in FIG. 3), the camera 120 is positioned directly above a preparation workstation 40 to provide a direct, unobstructed line of sight between the camera 120 and the preparation workstation 40.

[0050] The cameras 120 within the assembly area 26 may monitor the assembly of desired food products. By way of example, the cameras 120 may monitor the personnel, the storage bins 52, the ingredients, the food products undergoing assembly, the waste receptacles 48, or other items within the assembly area 26. The cameras 120 may be positioned such that a field of view of at least one camera 120 includes the work surfaces 54 to monitor the production of desired food products. The cameras 120 may be positioned such that a field of view of at least one camera 120 includes the holding areas 56 to monitor completion of desired food products. The cameras 120 may be positioned such that a field of view of at least one camera 120 includes the waste receptacle 48 to monitor handling and / or waste of ingredients. In some embodiments (e.g., as shown in FIG. 3), the camera 120 is positioned directly above an assembly workstation 50 to provide a direct, unobstructed line of sight between the camera 120 and the assembly workstation 50.

[0051] As shown in FIGS. 1-3, the control system 100 further includes one or more temperature sensors, shown as temperature sensors 122, operatively coupled to the systemcontroller 110. The temperature sensors 122 provide sensor data indicating a measurement of a temperature of one or more objects within the kitchen 20. The temperature sensors 122 may include thermocouples, resistance temperature detectors, infrared temperature detectors, or other types of temperature sessors. The temperature sensors 122 may measure the temperatures of the appliances 42, the one or more cooking surfaces 44, the work surfaces 46, ingredients, or other objects. In some embodiments, one or more of the cameras 120 are infrared cameras that are capable of measuring temperature. In some such embodiments, the camera 120 can act as (can serve as) the temperature sensor 122 (e.g., to provide an imaging capability and a temperature-indicating capability), and the kitchen 20 does not include a separate, dedicated temperature sensor 122.

[0052] As shown in FIGS. 2 and 3, the temperature sensors 122 are positioned within the preparation area 24. By way of example, the temperature sensors 122 may be used to monitor the cooking status (e.g., internal temperature, surface temperature, cooking progress, how well done the ingredient is, etc.) of one or more ingredients. By way of another example, the temperature sensors 122 may monitor a status of the appliances 42 (e.g., to determine if a cooking surface 44 is at a desired temperature). The temperature sensors 122 may be away from the preparation workstations 40 (e.g., coupled to a ceiling above a preparation workstation 40) or incorporated into the preparation workstations 40.

[0053] As shown in FIG. 2, the control system 100 further includes one or more input / output devices or user interfaces, shown as employee user interfaces 130, operatively coupled to the system controller 110. The employee user interfaces 130 are configured to provide information (e.g., status information, commands, etc.) to one or more personnel. The kitchen vision system 10 may utilize the employee user interfaces 130 to guide personnel to complete tasks quickly and accurately.

[0054] As shown in FIGS. 2 and 3, the employee user interfaces 130 include one or more output devices or displays, shown as screens 132, operatively coupled to the system controller 110. The screens 132 may visually display information (e.g., a graphical user interface) to one or more personnel. The screens 132 may include a touchscreen, such that the screens 132 may receive inputs from the personnel. The system controller 110 may convey information such as the locations of ingredients, the next steps that should be takento prepare ingredients or assemble food products, the specific food products that are desired, or other information.

[0055] As shown in FIG. 2, the employee user interfaces 130 may include one or more output devices, shown as speakers 134, operatively coupled to the system controller 110. The speakers 134 may audibly convey information to one or more personnel. By way of example, the speakers 134 may provide alarms, chimes, beeps, or spoken words and phrases. The speakers 134 may convey alerts, notifications, commands, or other information to the personnel.

[0056] As shown in FIGS. 2 and 3, the employee user interfaces 130 include one or more output devices, spotlights, projectors, or light emitters, shown as projectors 136, operatively coupled to the system controller 110. The projectors 136 are configured to emit a beam of light to form an image onto one or more surfaces of the kitchen 20. The projectors 136 may form the images on the food storage devices 30, the preparation workstations 40, the waste receptacles 48, and / or the assembly workstations 50. Accordingly, the projectors 136 may be present in the storage area 22, the preparation area 24, and / or the assembly area 26. The projectors 136 may be elevated (e.g., coupled to a ceiling of the kitchen 20) to provide the projectors 136 with a line of sight to one or more target surfaces (e.g., the cooking surfaces 44, the work surfaces 54, etc.) that is generally unobstructed by the personnel.

[0057] The images may serve as commands or indicators to one or more personnel. By way of example, presence or content of an image may indicate that the employee has performed an error that requires a corrective action. By way of another example, a projector 136 may form an image onto a particular ingredient to indicate that that ingredient should be used.

[0058] The projectors 136 may be able to vary certain characteristics of the formed images. In some embodiments, the projectors 136 are capable of varying the location of the formed image. By way of example, a projector 136 may be able to move the image between different storage bins 52 of the assembly workstations 50 (e.g., such that the image illuminates a first storage bin 52 but not a second storage bin 52 of the same assembly workstation 50). In some embodiments, the projectors 136 are capable of varying the shapeand size of the formed image. By way of example, a projector 136 may change the shape and size of the formed image to match the shape and size of a storage bins 52. By way of another example, the projectors 136 may change the shape of the image to form text (e.g., indicating a command, such as “error - remake” or “lettuce”). In some embodiments, the projectors 136 are capable of varying the color of the formed image. By way of example, the formed image may be a full color image formed from a grid of pixels that each have a separately variable color. In some embodiments, any one or more of the location, shape, size or color can be controlled, e.g., to be varied.

[0059] In the exemplary embodiment shown in FIG. 3, the control system 100 includes a projector 136 associated with a preparation workstation 40. The projector 136 emits a first beam of light Bl toward the work surface 46 that forms a first image II in the shape of a circle atop the work surface 46. The first image II may indicate, for example, an ingredient supported on the work surface 46 that an employee is being instructed to prepare. The projector 136 emits a second beam of light B2 toward the appliance 42 that forms a second image 12 in the shape of a square atop the cooking surface 44. The second image 12 may indicate, for example, an ingredient (e.g., a burger patty) on the cooking surface 44 that the employee is being instructed to flip or remove.

[0060] As shown in FIG. 2, the system controller 110 is in communication with an external controller, server, or third-party system, shown as cloud computing system 140. In some embodiments, the cloud computing system 140 is positioned remotely from the system controller 110 (e.g., remote from the kitchen 20). The system controller 110 may include a communication interface (e.g., a wired network interface, a wireless network interface, a cellular receiver, etc.) that facilitates communication between the system controller 110 and the cloud computing system 140. By way of example, the system controller 110 may communicate with the cloud computing system 140 through the Internet.

[0061] In some embodiments, the system controller 110 and the cloud computing system 140 share data between one another. By way of example, the system controller 110 may provide data regarding operation of the kitchen 20 to the cloud computing system 140 where the data is combined with similar data from other kitchens. By way of anotherexample, the cloud computing system 140 may provide externally gathered data, such as data regarding the operation of other kitchens, to the system controller 110.

[0062] For ease of description, various processes are discussed herein as being performed by the system controller 110. However, it should be understood that such processes may be performed by the system controller 110, by the cloud computing system 140, or by a combination of both the system controller 110 and the cloud computing system 140. By way of example, a determination may be made by the cloud computing system 140 regarding a desired operation of the kitchen 20, and the cloud computing system 140 may provide a command to the system controller 110 that the system controller 110 executes locally. In some embodiments, the system controller 110 and / or the cloud computing system 140 include multiple devices (e.g., multiple processors 112 and / or multiple memories 114) that cooperate to perform the processes described herein.

[0063] As shown in FIG. 2, the control system 100 further includes one or more user interfaces, point of sale devices, registers, ordering kiosks, or user devices, shown as point of sale devices 142, operatively coupled to the system controller 110. The point of sale devices 142 may receive orders from one or more customers. By way of example, the point of sale devices 142 may provide a user interface (e.g., a touch screen, one or more buttons, etc.) through which a user may select one or more desired food products to form, indicate, or generate an order. A point of sale device 142 may be a kiosk operated directly by a customer, a register operated by an employee on behalf of a customer, or a user device (e.g., a smartphone, a tablet, a laptop computer, etc.) in communication with the kitchen vision system 10.

[0064] As shown in FIG. 2, the control system 100 further includes one or more user interface devices or system management devices, shown as manager interface devices 144, operatively coupled to the system controller 110. The manager interface devices 144 may facilitate a manager of the kitchen vision system 10 (e.g., an owner or operator of the kitchen 20) interacting with the kitchen vision system 10. By way of example, the manager interface devices 144 may include a user device, such as a smartphone, tablet, laptop computer, or other device. The manager interface devices 144 may communicate with thesystem controller 110 and / or the cloud computing system 140 through a network connection (e.g., the Internet).

[0065] The manager interface devices 144 may provide information to a manager. By way of example, the manager interface devices 144 may provide operational statistics (e.g., amount of food products sold, profit, waste, etc.) and performance metrics (e.g., individual employee performance, etc.) for review by a manager. In some embodiments, the manager interface devices 144 permit a manager to adjust operation of the kitchen vision system 10 (e.g., adding new products, changing how the kitchen vision system 10 reacts in response to certain situations, etc.).Stored Data

[0066] The control system 100 stores a variety of data for use in the processes described herein. The data may be generated by the control system 100 or generated externally and provided to the control system 100. As shown in FIG. 2, the data is stored in the memory 114. In other embodiments, the data is stored in the cloud computing system 140 or distributed across the system controller 110 and / or the cloud computing system 140.A. Menu Data

[0067] As shown in FIG. 2, the memory 114 contains product data, product availability data, product instruction data, or menu data, shown as menu data 150. The menu data 150 indicates the types of food products that the kitchen 20 is permitted to produce or offer (e.g., capable of offering, intended to offer, etc.). The menu data 150 may be generated or edited through a manager interface device 144 or by the cloud computing system 140. By way of example, the menu data 150 may be common across multiple franchises of a restaurant chain. In one such example, all of the franchises start with a common set of menu data 150 that is adjusted to meet the needs of a specific kitchen 20 through a manager interface device 144.

[0068] The menu data 150 may include a list of food products that the kitchen 20 is permitted to produce. Additionally, the menu data 150 may include information about specific food products. By way of example, the menu data 150 may include a list ofingredients required to produce a food product. By way of another example, the menu data 150 may include an amount of each ingredient required to produce a food product. By way of another example, the menu data 150 may include a set of instructions or steps (e.g., a recipe) that should be followed to produce the food product. The instructions or steps may have a predetermined order required to produce the food product. By way of another example, the menu data 150 may indicate a price of the food product.B. Order Data

[0069] As shown in FIG. 2, the memory 114 contains customer order data, shown as order data 152. The order data 152 indicates the content of one or more orders for food products placed by customers. The point of sale devices 142 may generate the order data 152 based on selections provided through the point of sale devices 142. By way of example, each set of order data 152 may correspond to a particular order placed by a customer (e.g., placed directly by the customer or communicated through an employee).

[0070] The order data 152 may include a list of desired food products that are required to be provided to the customer in order for the order to be fulfilled. To minimize waste, it may be desirable for the control system 100 to fulfill the order completely and accurately without providing any additional items that were not specified in the order data 152. The order data 152 may include a type of the desired food product (e.g., cheeseburger, fried chicken, fries, cola, etc.). The order data 152 may include a quantity of the desired food product (e.g., one, three, five, etc.). The order data 152 may include a size of the desired food product (e.g., large, small, 16 fluid ounces, 1 pound, etc.). The order data 152 may include requested customizations for the desired food product (e.g., removing a particular ingredient, adding extra of a particular ingredient, requesting a particular cook temperature, etc.). The order data 152 may include a requested time of delivery (e.g., 5:00 PM, etc.). The order data 152 may include a unique identifier for the customer that placed the order (e.g., a customer number, a customer name, etc.). The order data 152 may include a total cost of the order. The order data 152 may include a payment status of the order (e.g., paid in advance, awaiting payment, etc.).c. Employee Data

[0071] As shown in FIG. 2, the memory 114 contains personnel data, employee identification data, or shown as employee data 154. The employee data 154 identifies the personnel assigned to the kitchen 20 as well as information about the performance of the individual employees. The employee data 154 may be provided by the cloud computing system 140. Additionally or alternatively, the employee data 154 may be provided by a manager through the manager interface devices 144.

[0072] The employee data 154 may include a list of employees, as well as identifiers for each employee (e.g., employee names, employee numbers, etc.). The employee data 154 may include a work schedule for each employee. The employee data 154 may include an indication of which tasks are assigned to each employee (e.g., food preparation, griddle, food product assembly, etc.). The employee data 154 may include a record of employee performance (e.g., an amount of successful food products produced, a list of errors, an amount of waste, etc.).D. Image Data

[0073] As shown in FIG. 2, the memory 114 contains sensor data or camera data, shown as image data 156. The image data 156 may be provided by the cameras 120. The image data 156 may provide a visual representation of the environment within a field of view of each camera 120. Accordingly, the cameras 120 may be placed to provide image data 156 showing objects of interest. The image data 156 may be stored in the memory 114 directly or processed prior to storage in the memory 114. By way of example, the image data 156 may be stored as individual frames (e.g., static images) recorded at regular intervals. By way of another example, images provided by the cameras 120 may be analyzed to identify objects within the images, and the analyzed images and data summarizing the analysis may be stored as the image data 156. By way of yet another example, information regarding objects recognized in the images (e.g., pose, object type, etc.) may be stored as the image data 156 without storing the raw images from the cameras 120.Analysis of Image Data

[0074] The system controller 110 may analyze the image data 156 to determine information regarding objects of interest within the kitchen 20. In some embodiments, the system controller 110 utilizes a machine learning model trained to detect and identify objects within an image. By way of example, the machine learning model may be trained using a dataset optimized for ingredients, food products, and humans. In some embodiments, the system controller 110 utilizes a You Only Look Once (YOLO) machine learning model. See J. Terven et al., “A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOvl to YOLOv8 and YOLO-NAS,” Machine Learning and Knowledge Extraction. 5 (4): 1680-1716. arXiv:2304.00501. doi: 10.3390 / make5040083. ISSN 2504-4990 (Nov. 2023). In some embodiments, a different machine learning model can be used. The machine learning model can employ at least one regional proposal algorithm and / or at least one convolutional neural network to detect and localize at least one object in an image. In some embodiments, the machine learning model employs a two- stage detection algorithm and can employ rCNN, Fast rCNN, rFCN or another convolutional neural network model. In some embodiments, the machine learning model employs a one-stage detection algorithm, such as YOLO, as mentioned above, or singleshot detection (SSD).

[0075] The machine learning model may provide an output with bounding boxes around each detected object, an assigned identity for the object (e.g., a type of ingredient, a particular employee, a human hand, or another type of object), and a confidence score indicating a level of confidence with which the system controller 110 was able to detect and identify the object. A high confidence score may indicate a high probability that the object was correctly detected and identified. In some embodiments, the machine learning model further provides an orientation of the detected object. In some such embodiments, the machine learning model provides both the position and orientation of each object (e.g., a pose). By way of example, the system controller 110 may determine a direction that an employee is facing and / or a field of view of the employee.

[0076] FIGS. 4-25 illustrate image data 154 that has been processed by the system controller 110 (e.g., using a YOLO machine learning model) to identify various objects.Specifically, FIG. 4 illustrates image data showing an appliance 42 of a preparation workstation 40 that is in the process of cooking various food products. FIGS. 5-11 illustrate image data showing an assembly workstation 50 that is being used to prepare a chicken sandwich. FIGS. 12-26 illustrate image data showing an assembly workstation 50 that is being used to prepare a cheeseburger.

[0077] FIGS. 4-25 include various objects that have been identified by the system controller 110. The objects include ingredients 160 used to form desired food products. The ingredients 160 may include raw ingredients, processed ingredients (e.g., cooked, chopped, mixed, etc.), and ingredients being processed (e.g., being heated). The system controller 110 may determine a preparation status, state, or condition of the ingredient (e.g., whole, chopped, sliced, raw, cooked, etc.).

[0078] The system controller 110 may identify ingredients 160 directly (e.g., where a direct line of sight is available between the cameras 120 and the ingredients 160). By way of example, the system controller 110 may identify an ingredient 160 that is laid out on a cooking surface 44, on a work surface 54, or within an open storage bin 52. The system controller 110 may additionally or alternatively identify ingredients 160 within a container (e.g., a bottle, ajar, a can, a storage bin 52 that is closed, etc.). When identifying ingredients 160 within a container, the system controller 110 may utilize the shape of the container, the size of the container, the color of the container, information contained on a label, or other information to determine a type of food product that is present within the container.

[0079] The system controller 110 may identify a variety of different ingredients 160. The ingredients 160 may include packaging or containers for food products 164, such as bags, wrappers, cups, lids, straw, napkins, or other types of packaging for the food products 164. The ingredients 160 may include various food ingredients, such as bread (e.g., buns, wraps, rolls, pancakes, waffles, etc.), meats (e.g., beef, chicken, pork, lamb, bacon, etc.), chicken pieces (e.g., whole chickens, legs, thighs, wings, breasts, etc.), deli meats (e.g., ham, roast beef, chicken, bologna, salami, etc.), fruits and vegetables (e.g., lettuce, tomatoes, olives, onions, peppers, pickles, etc.), condiments (e.g., mayonnaise, ketchup, mustard, etc.), cheese (e.g., mozzarella, parmesan, gouda, American, cheddar, Swiss, etc.), or other typesof food ingredients. The ingredients 160 may include various beverage ingredients, such as water (e.g., tap water, sparkling water, etc.), soda (e.g., cola, orange soda, energy drinks, etc.), fruit juices (e.g., orange juice, cranberry juice, apple juice, etc.), alcoholic beverages (e.g., beer, wine, rum, whiskey, etc.), ice, or other beverage ingredients.

[0080] The objects identified by the system controller 110 further include personnel, shown as employees 162. The system controller 110 may identify an entire employee 162 (e.g., from head to toe) or a portion of the employee 162 (e.g., an appendage, such as a hand, etc.). The objects identified by the system controller 110 may further include food products 164. The food products 164 may be assembled and ready for serving to a customer or food products that are in the process of being assembled.

[0081] FIGS. 4-25 have been annotated to show certain elements of the analysis performed by the system controller 110. The elements include boundaries or borders, shown as boundary boxes 170, that illustrate the identified boundaries of a detected object. Accordingly, the boundary boxes 170 may indicate the position, size, shape, and / or orientation of a detected object. While the boundary boxes 170 are shown as being rectangular, in other embodiments the boundary boxes 170 conform to the shapes of the corresponding objects. The elements further include identifiers, shown as labels 172, that contain determined information regarding the identified objects. By way of example, the labels 172 may include an object type, a confidence level, or other information. The label 172 may represent this information as a text overlay for viewing by a user. As shown, the confidence level may represented as a number between 0 and 1, with 1 being the highest confidence level.

[0082] The labels 172 may additionally include information from other sources, such as the temperature sensors 122. By way of example, FIG. 4 includes labels showing a measured grill temperature for a left side of the cooking surface 44 (e.g., “Left Side Grill Temp: 402°F”) and a measured grill temperature for a right side of the cooking surface 44 (e.g., “Right Side Grill Temp: 310°F”). The temperatures may be measured by the temperature sensors 122 (e.g., multiple temperature sensors 122 coupled to the appliance 42). Alternatively, the cameras 120 may be configured to sense the temperatures (e.g., using a camera 120 configured as an infrared camera).

[0083] FIGS. 5-11 further include a list of ingredients, shown as ingredient list 174, that is overlaid onto the image data 156. The ingredient list 174 may include a list of ingredients that the system controller 110 has determined have been added to a food product that is being assembled on the work surface 54. The ingredient list 174 may be updated as more ingredients are added to the food product.

[0084] In addition to the type, shape, size, position, and orientation of an identified object, the system controller 110 may determine other information about the identified objects (e.g., using the YOLO machine learning model). The information may be included as part of the labels 172 or otherwise included as part of image data 156.A. Ingredient Amounts

[0085] In some embodiments, the system controller 110 determines an amount of material shown in the image data 156. By way of example, the system controller 110 may determine an amount of an ingredient 160 stored in a particular storage bin 52, an amount of an ingredient 160 in a food storage device 30 (e.g., in room-temperature storage 32, in refrigerators 34, in freezers 36, etc.), an amount of an ingredient 160 that has been added to a food product 164, an amount of an ingredient 160 placed into a waste receptacle 48, or other amounts of ingredients 160.

[0086] The system controller 110 may store a predetermined relationship between a size of an ingredient 160 in an image (e.g., a number of pixels corresponding to an ingredient 160 or a collection of ingredients, such as a pile) and an amount of the ingredient 160 (e.g., a quantity of the ingredient 160, a volume of the ingredient 160, a weight of the ingredient 160, etc.). By way of example, a slice of tomato may have an average thickness corresponding to a given number of pixels in an image, so the system controller 110 may determine the amount of tomatoes shown based on how many pixels correspond to the tomatoes. Alternatively, the system controller 110 may store a predetermined relationship between an amount of free space in a storage container (e.g., a storage bin 52) and an amount of an ingredient 160 stored within the storage container. By way of example, the system controller 110 may determine an amount of tomatoes within a storage bin 52 basedon what portion of a wall of the storage bin 52 is visible (e.g., is not obscured by the tomatoes within the storage bin 52).

[0087] In some embodiments, the system controller 110 determines an amount of an ingredient 160 that has been dispensed into a container based on an amount of time that the ingredient 160 has been dispensed. By way of example, the kitchen 20 may serve fluid ingredients 160, such as beverages or soups, that are dispensed from a container. Such a container may include a visible flow control member, such as a tap handle or valve handle, that controls when the fluid is being dispensed. Using the image data 156, the system controller 110 may determine (a) which ingredient 160 is being dispensed (e.g., based on which tap handle is being opened) and (b) a length of time for which the ingredient 160 was dispensed. A flow rate of the ingredient 160 through the flow control member may be predetermined, such that the length of time for which the ingredient 160 was dispensed may be used to calculate the amount of the ingredient 160 that was dispensed.

[0088] In some embodiments, the system controller 110 determines a price paid by the customer based on the amount of an ingredient 160 that was dispensed. By way of example, a given ingredient 160 may be assigned a particular cost per volume. Based on the amount of the ingredient 160 that is being served to a customer and the cost per volume, the system controller 110 may assign a price to the resultant food product. Such an arrangement may be useful in, for example, frozen yogurt shops where a soft serve product is dispensed from a container or bars where beer or other beverages are dispensed from taps. The system controller 110 may update the order data 152 to include the assigned price.B. Ingredient Preparation State

[0089] In some embodiments, the system controller 110 determines a preparation state, status, or condition of an ingredient 160 shown in the image data 156. The preparation state of an ingredient 160 may include whether the ingredient 160 has been chopped, sliced, mixed, or otherwise reshaped. By way of example, the system controller 110 may analyze the size and shape of the ingredient 160 to determine whether the ingredient 160 has been cut into pieces. By way of example, an onion may have an average shape and size. A sliced onion may have a smaller size than a full onion and may have an elongated shape. Adiced onion may have a smaller size than a full onion and my have a generally rectangular shape. By way of another example, the system controller 110 may analyze the shape of an ingredient 160 to determine whether the ingredient 160 has been mixed. In one such example, the system controller 110 may determine that an egg has not been beaten when the shape of an egg yolk is clearly visible.

[0090] The preparation state of an ingredient 160 may include a cooked state of the ingredient 160 (e.g., indicating to what degree the ingredient 160 has been cooked). The cooked state may include whether the ingredient 160 is frozen, thawed, raw, or cooked and / or a doneness level (e.g., rare, medium, well-done, etc.). The system controller 110 may determine a cooked state of an ingredient 160 based on a location of the ingredient 160 and how long the ingredient 160 has been in that location. By way of example, the system controller 110 may determine that an ingredient 160 that has been in a freezer 36 for greater than a threshold period of time (e.g., more than 12 hours) is frozen. By way of another example, the system controller 110 may determine that an ingredient 160 that has been in a refrigerator 34 for greater than a threshold period of time (e.g., more than 12 hours) is thawed. By way of another example, the system controller 110 may determine that an ingredient 160 that is on a cooking surface 44 is being cooked. The system controller 110 may determine a doneness level of the ingredient 160 based on at least one of (a) how long the ingredient 160 has been present on the cooking surface 44 or (b) a temperature of the cooking surface 44.

[0091] Additionally or alternatively, the system controller 110 may use sensor data from the temperature sensors 122 to determine a cooked state of the ingredient 160 (e.g., by measuring the temperature of the ingredient 160 directly). The system controller 110 may store a predetermined relationship between temperature and cooked state for a given type of food product (e.g., chicken thighs, steak, ground beef, etc.). The system controller 110 may compare the current temperature of the ingredient 160 with the predetermined relationship to determine the cooked state.

[0092] Additionally or alternatively, the system controller 110 may determine a cooked state of the ingredient 160 based on a color of the ingredient 160. An ingredient 160 may change color as the ingredient 160 cooks, such that a given doneness level of the ingredient160 may correspond to a predetermined color. The ingredient 160 may have a first color when raw, a second color when cooked to a desired doneness level, and a third color when overcooked. The system controller 110 may compare the color of the ingredient with these predetermined colors to determine the cooked status of the ingredient 160.

[0093] Additionally or alternatively, the system controller 110 may determine a cooked state of the ingredient 160 based on a color of the surroundings of the ingredient 160. When cooking, an ingredient 160 may emit steam or smoke that is visible to the cameras 120. The system controller 110 may differentiate between steam and smoke based on a color detected by the camera 120. The system controller 110 may determine the cooked state based on the presence of smoke or steam.

[0094] The preparation state may include whether the ingredient 160 has been agitated while cooking. During cooking, certain ingredients 160 require agitation (e.g., flipping, stirring, etc.) to ensure an even cooking. The system controller 110 may compare subsequent frames of image data 156 to detect movement of ingredients 160. Based on how the position and / or orientation of the ingredient 160 has changed, the system controller 110 may determine whether the ingredient 160 has been agitated. By way of example, the system controller 110 may monitor a beef patty and determine that the beef patty has been flipped when the system controller 110 detects an inversion of the orientation of the beef patty.C. Food Product Preparation State

[0095] In some embodiments, the system controller 110 determines a preparation state, status, or condition of a food product 164 shown in the image data 156. The preparation state of the food products 164 may indicate which ingredients 160 have been added to the food products 164. By way of example, the system controller 110 may identify a type and an amount of an ingredient 160 present on a food product 164 that is being assembled directly. By way of another example, the system controller 110 may monitor removal of ingredients 160 from storage bins 52 by an employee 162 that is assembling the food products 164. When the employee 162 removes an ingredient 160 and moves their handtoward the food product 164, the system controller 110 may determine that the ingredient 160 has been added to the food product 164.D. Examples of Analyzed Image Data

[0096] FIG. 4 illustrates an example of image data 154 including an image of a griddle or grill that has been analyzed by the kitchen vision system 10. The image data 154 of FIG. 4 shows ingredients 160 including two beef patties, onions, bacon, an egg, and a pancake. A pair of labels 172 indicate that the left and right sides of the grill are at 402°F and 310°F, respectively. Because all of the ingredients 160 are present on a cooking surface 44, the system controller 110 may determine that all of the ingredients 160 are currently being cooked.

[0097] A first label 172 identifies a beef patty with a confidence level of 0.8. The label 172 indicates that the beef patty has been flipped and has been cooking for 2 minutes and 35 seconds. A second label 172 identifies a beef patty with a confidence level of 0.95. The label 172 indicates that the beef patty has not been flipped and has been cooking for 1 minute and 5 seconds. A third label 172 identifies onions with a confidence level of 0.30.A fourth label 172 identifies bacon with a confidence level of 0.87. The label 172 indicates that the bacon has been cooking for 10 seconds and has a temperature of 100°F. A fifth label 172 identifies an egg with a confidence level of 0.96. The label 172 indicates that the egg was not beaten before being added to the cooking surface 44 and suggests a corrective action of scrambling the egg. The labels 172 indicates that the egg should be scrambled immediately (e.g., based on the current cook time of the egg). A sixth label 172 identifies a pancake with a confidence level of 0.83. The label 172 indicates that the pancake is overdone and should be removed and placed in the waste receptacles 48.

[0098] FIGS. 5-11 illustrate image data showing an assembly workstation 50 that is being used to prepare a food product 164, specifically a chicken sandwich. The image data illustrates various ingredients 160 arranged atop a work surface 54 and within a series of storage bins 52 adjacent the work surface 54. The storage bins 52 contain lettuce, pickles, sliced tomatoes, cheese, sliced onions, mayonnaise, and ketchup. Additionally, buns, apaper wrapper, and a piece of crispy chicken are placed onto the work surface 54 from outside of the field of view of the image data (e.g., from offscreen storage bins 52).

[0099] The image data includes a variety of boundary boxes 170 and labels 172 generated by the system controller 110. Each label 172 indicates a type of ingredient and a corresponding confidence level. The system controller 110 may generate the boundary boxes 170 and the labels 172 in real time as the process of FIGS. 5-11 are performed.

[0100] Each of FIGS. 5-11 may illustrate a different step performed to assemble the chicken sandwich. In FIG. 5, an employee 162 places a paper wrapper onto the work surface 54, and the employee 162 places a pair of buns onto the paper wrapper. In FIG. 6, the employee 162 places mayonnaise from a bottle onto one of the buns. In FIG. 7, the employee 162 places lettuce onto one of the buns. In FIG. 8, the employee 162 places a piece of crispy chicken onto one of the buns. In FIG. 9, the employee 162 places tomatoes onto the lettuce. In FIG. 10, the employee 162 puts the sandwich together with the chicken atop the tomatoes. In FIG. 11, the employee 162 wraps the sandwich with the paper wrapper. At this point, the food product 164 is complete and ready for order fulfillment (e.g., to be delivered and / or served to a customer).

[0101] FIGS. 12-26 illustrate image data showing an assembly workstation 50 that is being used to prepare a food product 164, specifically a cheeseburger. The image data illustrates various ingredients 160 arranged atop a work surface 54 and within a series of storage bins 52 adjacent the work surface 54. The storage bins 52 contain mayonnaise, lettuce, sliced tomatoes, cheese, ketchup, and beef patties. Additionally, buns and a paper wrapper are placed onto the work surface 54 from outside of the field of view of the image data (e.g., from offscreen storage bins 52).

[0102] The image data includes a variety of boundary boxes 170 and labels 172 generated by the system controller 110. Each label 172 indicates a type of ingredient and a corresponding confidence level. The system controller 110 may generate the boundary boxes 170 and the labels 172 in real time as the process of FIGS. 12-26 are performed.

[0103] Each of FIGS. 12-26 may illustrate a different step performed to assemble the chicken sandwich. In FIG. 12, an employee 162 places a paper wrapper onto the work surface 54, the employee 162 places a pair of buns onto the paper wrapper, and the employee spreads mayonnaise onto one of the buns. In FIG. 13, the employee 162 places lettuce onto one of the buns. In FIGS. 14 and 15, the employee 162 places tomatoes onto the lettuce. In FIG. 16, the employee 162 retrieves a beef patty from a storage bin 52. In FIG. 17, the employee 162 places the beef patty onto one of the buns. In FIGS. 18 and 19, the employee 162 places cheese onto the beef patty. In FIG. 20, the employee 162 places pickles onto the cheese. In FIG. 21, the employee 162 places sliced onions onto the pickles. In FIG. 22, the employee 162 places ketchup onto the onions and lifts one of the buns, the lettuce, and the tomatoes. In FIG. 23, the employee 162 puts the sandwich together with the tomatoes atop the ketchup. In FIGS. 24 and 25, the employee 162 wraps the cheeseburger with the paper wrapper. At this point, the food product 164 is complete and ready to be served to a customer.Demand Predictions

[0104] Referring to FIGS. 1-3, the kitchen vision system 10 may predict a future demand for ingredients and / or food products. The kitchen 20 may experience periods of low demand (e.g., nights, midday, weekdays, etc.) and periods of high demand (e.g., breakfast rushes, dinner rushes, lunch rushes, periods of promotions or limited-time offerings, weekends, etc.). During periods of high demand, more food products are ordered by customers than during periods of low demand, increasing the amount of food products that are desired to be produced and the amount of ingredients required to produce the desired food products. Accordingly, the system controller 110 may attempt to predict the demand to facilitate stocking an inventory of the kitchen 20 sufficiently to meet the demand without excess ingredients or food products that are wasted.

[0105] The system controller 110 may a demand for a restaurant including the kitchen 20 based on a variety of different factors. In some embodiments, the system controller 110 predicts the demand based on time. Certain times may be associated with periods of high demand or low demand. By way of example, the system controller 110 may predict the demand based on the time of day, the day of the week (e.g., Monday, Tuesday, etc.), themonth of the year (e.g., January, June, September, etc.), the season, or other times. By way of another example, the system controller 110 may predict the demand based on holidays. In some embodiments, the system controller 110 predicts the demand based on the location of the restaurant. By way of example, certain areas, cities, states, or countries may have different preferences that drive demand at different times or for different items.

[0106] In some embodiments, the system controller 110 predicts demand based on historical order data (e.g., order data 152 for past fulfilled orders). Order data 152 corresponding to a past order at a similar time may be used to predict demand at a future time. By way of example, the system controller 110 may predict the demand for a particular type of food product on a Friday during the summer at 5:00PM based on the average demand for that type of food product on past Fridays during the summer at similar times. By way of another example, the system controller 110 may identify trends in order data 152 (e.g., a particular food product becoming more popular) and predict that the demand will increase or decrease according to the trend. Order data 152 corresponding to other restaurants in nearby or similar locations may be used to predict demand. By way of example, the system controller 110 may predict the demand for a particular food product based on the average demand for that type of food product at nearby restaurants.

[0107] In some embodiments, the system controller 110 predicts an amount of food products that will be desired at a given day and time based on the predicted demand. The system controller 110 may predict a specific amount of each type of food product that will be desired (e.g., a specific amount of each menu item of the menu data 150). Based on the predicted amount of each type of food product and the menu data 150, the system controller 110 may predict the amount of each prepared ingredient that will be required for the given day and time. By way of example, the system controller 110 may utilize the menu data 150 to determine the type and amount of prepared ingredients required to produce each food product that is predicted to be desired.

[0108] Based on the predicted amount of desired food products and the corresponding amount of prepared ingredients required to produce the desired food products, the system controller 110 may determine threshold levels for ingredients and food products that will satisfy the predicted demand. The threshold levels may indicate a minimum acceptableamount of the ingredient or food product to have in inventory at a given time. By way of example, the system controller 110 may predict that the kitchen 20 will experience demand for 20 cheeseburgers from 5:00PM to 5:30PM. In response, the system controller 110 may set a threshold amount of cheeseburgers to ensure that at least 20 cheeseburgers are available at this time. Additionally, the system controller 110 may set a threshold amount of each ingredient required to produce the demanded cheeseburgers. The system controller 110 may set the threshold amount of each ingredient to be prepared before the demand is predicted to occur in order to ensure that the ingredients are prepared on time. The system controller 110 may utilize the threshold levels when performing inventory management, as described herein.Inventory Management

[0109] Referring to FIGS. 1-3, the kitchen vision system 10 may track the inventory of food products and / or ingredients within the kitchen 20. As the kitchen 20 operates, raw ingredients are taken from the storage area 22 to the preparation area 24 where the ingredients are prepared for service. The prepared ingredients are then taken to the assembly area 26 and stored in the storage bins 52. The prepared ingredients are removed from the storage bins 52 and used to form desired food products within the assembly area 26, which are then stored in the holding areas 56. The food products are taken from the holding areas 56 and bagged for delivery to a customer in the delivery areas 58.Throughout this process, if the inventory is not managed properly, the customer may experience delays while additional ingredients or food products are prepared, or the kitchen 20 may be unable to fulfill orders for certain food products. Beneficially, the kitchen vision system 10 may ensure that the ingredients and food products are available as needed to reduce customer wait times and an prevent food products from becoming unavailable.

[0110] The system controller 110 may utilize the image data 156 to perform the inventory management. The system controller 110 may utilize the image data 156 to determine current inventory levels. The inventory levels may include a current amount of ingredients or food products available in certain areas. The determined inventory levels may differentiate between inventory that is present and ready for use and inventory that is present but not yet prepared for use. By way of example, the determined inventory levelsmay differentiate between thawed ingredients that are ready for use and frozen ingredients that require thawing before use.[OHl] In some embodiments, the system controller 110 uses the image data 156 to determine current inventory levels within the storage area 22. The system controller 110 may determine the types and amounts of ingredients within the room-temperature storage 32, the refrigerators 34, and the freezers 36. By way of example, when an ingredient is added to one of the food storage devices 30, the system controller 110 may analyze the image data 156 from the cameras 120 within the storage area 22 and determine (a) which types of ingredients have been added, (b) how much of each type of ingredient have been added, (c) a date and time when the ingredients were added, and (d) which type of food storage device 30 the ingredients were added to (e.g., whether the ingredients are at room temperature, refrigerated, or frozen).

[0112] In some embodiments, the system controller 110 uses the image data 156 to determine current inventory levels within the assembly area 26. The system controller 110 may determine the types and amounts of prepared ingredients within the storage bins 52. By way of example, the system controller 110 may monitor the image data 156 and determine when a prepared ingredient is provided from the preparation area 24 to the assembly area 26. When a prepared ingredient is added to a storage bins 52, the system controller 110 may analyze the image data 156 from the cameras 120 within the preparation area 24 and / or the assembly area 26 and determine (a) which types of prepared ingredients have been added, (b) which storage bins 52 the prepared ingredients were added to, (c) how much of each type of prepared ingredient was added, and (d) a date and time when the ingredients were prepared.

[0113] In some embodiments, the system controller 110 determines whether an ingredient is ready to be prepared for service. Specifically, the system controller 110 may determine whether an ingredient is thawed or still frozen. Certain ingredients (e.g., meats, vegetables, etc.) may be provided to the kitchen 20 frozen (e.g., to facilitate stability during transportation). Such ingredients may be kept frozen for longevity until shortly before use. At that point, an employee may move the ingredients from the freezers 36 to the refrigerators 34 to begin thawing of the ingredients. The ingredients may require a periodof time to fully thaw before being prepared for service. Accordingly, it may be desirable for the system controller 110 to determine which ingredients are frozen, which are thawing, and which are fully thawed.

[0114] In some embodiments, the system controller 110 determines whether an ingredient is frozen based on (a) the type of the ingredient, (b) the type of food storage device 30 the ingredient was added to, and (c) the time that has elapsed since the ingredient was added to the food storage device 30. By way of example, the system controller 110 may determine that an ingredient 160 that has been in a freezer 36 for greater than a threshold period of time (e.g., more than 12 hours) is frozen. By way of another example, the system controller 110 may determine that an ingredient 160 that has been in a refrigerator 34 for greater than a threshold period of time (e.g., more than 12 hours) is thawed. The system controller 110 may vary the threshold periods of time based on the type of the ingredient. By way of example, a frozen vegetable may thaw more quickly than a steak.

[0115] In some embodiments, the system controller 110 determines an expiration status for an ingredient or food product. Specifically, the system controller 110 may determine whether an ingredient or food product is expired or when the ingredient or food product will expire. Ingredients and food products may have a predetermined “lifespan,” or recommended consumable period, after which the ingredient or food product should be disposed of (e.g., placed in the waste receptacles 48). The consumable period may begin when an ingredient is delivered to the kitchen 20, when an ingredient is thawed, when an ingredient is prepared for service, or when a food product is assembled. The length of the consumable period may be predetermined for a given type of ingredient or food product, and can correspond to food product labeling, product usage recommendations, a policy, etc. The system controller 110 may use the determined type of ingredient or food product to determine the consumable period and may use the image data 156 to determine when the consumable period of the ingredient or food product began. Based on this information, the system controller 110 may determine when the ingredient or food product will expire or has already expired.

[0116] By way of example, the system controller 110 may determine, based on the image data 156, that container of chicken was thawed on a first date and time (e.g., April 4 at1 :00PM) and that the recommended consumable period of thawed chicken is a given time period (e.g., 48 hours). Based on this information, the system controller 110 may determine that that container of chicken will expire at a second time period (e.g., April 6 at 1 :00PM). By way of another example, the system controller 110 may determine, based on the image data 156, that a cheeseburger was assembled at 2:30PM and that the consumable period for a cheeseburger is 30 minutes after assembly. Based on this information, the system controller 110 may determine that the cheeseburger will expire at 3:00PM.

[0117] In response to a determination that an ingredient or food product has expired, the system controller 110 may provide a command to an employee (e.g., through an employee user interface 130 or a manager interface device 144). The system controller 110 may then remove the ingredient or food product from the available inventory and mark the ingredient or food product as waste.

[0118] The system controller 110 may determine threshold levels for ingredients and food products, that indicate a minimum acceptable amount of the ingredient or food product to have in inventory. In some embodiments, the threshold level is a fixed, predetermined amount. In other embodiments, the threshold levels vary over time (e.g., based on demand). By way of example, the system controller 110 may increase the threshold levels in preparation for a predicted period of high demand. By way of example, the system controller 110 may increase the threshold level of food products required in advance of a predicted order rush (e.g., a breakfast, lunch or dinner rush).

[0119] In response to an amount of a food product available in the holding areas 56 falling below a threshold amount, the system controller 110 may command an employee to prepare more of the food product. By way of example, the system controller 110 may indicate a quantity and type of the desired food product to an employee within the assembly area 26 through an employee user interface 130. In response, the employee may retrieve prepared ingredients from the storage bins 52 and prepare the desired food products.

[0120] In response to an amount of a prepared ingredient within one of the storage bins 52 falling below a threshold amount, the system controller 110 may command an employee to prepare more of the ingredient and restock the storage bin 52. By way of example, thesystem controller 110 may indicate an amount and type of the desired prepared ingredient to an employee within the preparation area 24 through an employee user interface 130 within the preparation area 24. In response, the employee may retrieve ingredients from the storage area 22, prepare the ingredients for service, and provide the prepared ingredients to the assembly area 26 in a storage bin 52.

[0121] In response to an amount of a thawed ingredient within the storage area 22 falling below a threshold amount, the system controller 110 may command an employee to begin thawing more of the ingredient. By way of example, the system controller 110 may indicate an amount and type of the desired ingredient to an employee through an employee user interface 130. In response, the employee may move the ingredient from a freezer 36 to a refrigerator 34.

[0122] In response to an amount of an ingredient (e.g., a frozen ingredient, a roomtemperature ingredient, or a refrigerated ingredient where no additional frozen product is available) within the storage area 22 falling below a threshold amount, the system controller 110 my initiate a delivery request for more of the ingredient to restock the storage area 22. The delivery request may cause an external delivery service (e.g., associated with the restaurant of the kitchen 20 or a third-party service) to deliver more of the product to the storage area 22. By way of example, the delivery request may be provided to a manager through a manager interface device 144, and the manager may coordinate purchase and delivery of the ingredient. By way of another example, the system controller 110 may provide the delivery request directly to the external delivery service (e.g., over the Internet). The delivery request may indicate a type of ingredient to be delivered and an amount of the ingredient to be delivered.Facilitating Ingredient Preparation and Food Product Assembly

[0123] Referring to FIGS. 1-3, the kitchen vision system 10 may provide various features that facilitate production of food products by personnel. The system controller 110 may monitor performance of food production tasks by the personnel and provide instructions to the personnel. The instructions may indicate what food products to produce, what ingredients to prepare, and what steps to perform in order to accomplish these tasks. Byinstructing the personnel in this way, the kitchen vision system 10 may ensure that the desired food products are produced accurately and on schedule to meet demand.

[0124] In some embodiments, the kitchen vision system 10 assigns tasks to each employee. The tasks may include tasks to be performed at preparation workstations 40 (e.g., ingredient preparation operations, such as chopping, mixing, and grilling) and tasks to be performed at assembly workstations 50 (e.g., assembling desired food products from prepared ingredients). Each employee, each preparation workstations 40, or each assembly workstations 50 may be assigned to a queue of tasks to be performed, as well as an order in which the tasks should be performed (e.g., based on which prepared ingredients or food products are needed and when).

[0125] In some embodiments, the tasks are determined based on the current available inventory. By way of example, the system controller 110 may assign an employee at a preparation workstation 40 the task of preparing a particular ingredient in response to the amount of that prepared ingredient in a storage bin 52 falling below a threshold level. By way of another example, the system controller 110 may assign an employee at an assembly workstation 50 the task of assembling a desired food product in response to the amount of that food product falling below a threshold level. In some embodiments, the tasks are determined based on demand. By way of example, the system controller 110 may predict demand and identify food products or ingredients that should be prepared to meet the predicted demand, as described herein.

[0126] In some embodiments, the system controller 110 varies the timing when the task is performed (e.g., when the task is assigned to a particular employee, preparation workstation 40, or assembly workstation 50) based on a preparation time required to prepare an ingredient or assemble a food product. An ingredient may have an associated preparation time required for the ingredient to be prepared (e.g., a cooking time, an estimated amount of time required to chop the ingredient, etc.). Similarly, a food product may have an associated amount of preparation time required for the food product to be prepared (e.g., an estimated amount of time required to assemble the food product). If multiple different ingredients or food products are desired, these differences in preparation time may cause them all to be completed at different times.

[0127] In some situations, it is desirable to have multiple different ingredients or food products completed at the same time. By way of example, a particular order may include both a chicken sandwich and a cheeseburger that are desired by a particular customer. Accordingly, it may be desirable for production of the chicken sandwich and the cheeseburger to be completed at the same time. However, a chicken patty required for the chicken sandwich may have preparation time (e.g., a cooking time) that is longer than a preparation time (e.g., a cooking time) for a beef patty of the cheeseburger. Accordingly, the system controller 110 may delay assignment of a task associated with the cheeseburger (e.g., cooking the beef patty) such that the chicken sandwich and the cheeseburger are completed at the same time.

[0128] The preparation times for ingredients and food products may be predetermined and stored by the system controller 110 (e.g., in the menu data 150). Based on the preparation times associated with a given order or a given demand (e.g., a dinner rush), the system controller 110 may determine a delay (an elapse) for a given task that is intended to provide for completion of the various food products simultaneously. The system controller 110 may initiate a timer upon beginning of the first task, and the system controller 110 may use the timer to track progress of the desired elapse. In some embodiments, the system controller 110 initiates the timer in response to an ingredient being removed from a storage area (e.g., a food storage device 30, a storage bin 52, etc.). In response to the timer exceeding the desired elapse, the system controller 110 may initiate the second task (e.g., by commanding an employee to begin the second task).

[0129] Once a task is assigned to an employee, the system controller 110 may provide guidance to facilitate the employee performing the task. The guidance may indicate actions that the employee should take to complete the task (e.g., what ingredients to use, what to do with the ingredients, etc.). The guidance may serve to train a new employee, may serve as a reminder for an experienced employee, or may otherwise facilitate fast and accurate completion of the task regardless of the employee’s experience level.

[0130] In some embodiments, the system controller 110 utilizes the employee user interfaces 130 to provide the guidance. The system controller 110 may control the screens 132 to provide the guidance. By way of example, the system controller 110 may providecommands to an employee on the screen 132 in the form of text instructions (e.g., in a user- selectable language). By way of another example, the system controller 110 may provide commands to an employee on the screen 132 in the form of an image (e.g., showing a desired result) or video (e.g., showing the task being performed). Beneficially, an imagebased command may be understood regardless of the language spoken by the employee.

[0131] The system controller 110 may control the speakers 134 to provide the guidance. By way of example, the system controller 110 may provide commands to an employee through the speakers 134 in the form of beeps, buzzes, spoken instructions, or other audible instructions.

[0132] As shown in FIG. 3, the system controller 110 may control the projectors 136 to provide the guidance. By way of example, the system controller 110 may control a projector 136 to emit a beam of light that forms an image indicating an instruction. The location of the image may indicate the instruction (e.g., the image overlaps an ingredient that should be used to perform the current step of a task, etc.). The shape of the image may indicate the instruction (e.g., the image forms a circle over a desired ingredient, the image forms an X over an ingredient that should not be used, etc.). The color of the image may indicate the instruction (e.g., a green image indicates a correct action, a red image indicates an incorrect action, etc.).

[0133] Referring to FIGS. 1-3, the system controller 110 may adjust the guidance based on the image data 156. The system controller 110 may determine a current action being performed by the employee or a current step of a task based on the image data 156. By way of example, the system controller 110 may monitor the movement of the employee to determine an action being performed by the employee (e.g., an employee’s hand moving toward a particular ingredient). By way of example, the system controller 110 may determine a current cooked state of an ingredient. By way of example, the system controller 110 may determine a current stage of assembly of a food product (e.g., based on which ingredients have been added to a food product). When a task includes multiple steps, the system controller 110 may automatically determine the current step of the task (e.g., using the image data 156) and adjust the guidance based on the determined step.

[0134] Referring to FIGS. 1-4, the kitchen vision system 10 may be used to provide guidance to an employee using an appliance 42 to cook ingredients. By way of example, the kitchen vision system 10 may guide an employee cooking the ingredients 160 on a griddle as shown in FIG. 4. The image data 156 may be used to monitor the ingredients 160 atop the cooking surface 44. The system controller 110 may determine the type, size, and location of the ingredients 160 on the cooking surface 44. Additionally or alternatively, the system controller 110 may determine a type and quantity of ingredient 160 that that should be added to the cooking surface 44 to fulfill the task assigned to the employee.

[0135] Based on the type of ingredients 160 identified, the system controller 110 may determine one or more actions required to properly cook the ingredients (e.g., which ingredients should be added to the cooking surface 44, a desired cooking temperature, a desired cooking time, whether the ingredient needs to be flipped during cooking, etc.). The actions required to properly cook the ingredient may be included in the menu data 150 for the ingredient.

[0136] Based on the image data 156 and / or the sensor data from the temperature sensors 122, the system controller 110 may determine the current cooking conditions being experienced by each ingredient. By way of example, the system controller 110 may determine the temperature at different locations on the cooking surface 44 (e.g., a temperature gradient across the cooking surface 44). The system controller 110 may use the image data 156 to determine the locations of the ingredients and compare the locations with the temperature gradient to determine the current cooking temperature being experienced by the ingredient. Additionally or alternatively, the system controller 110 may use the image data 156 to determine open locations on the cooking surface 44 and use the temperature gradient to select an open location having the most appropriate temperature for cooking an ingredient to be newly added to the cooking surface 44.

[0137] Based on the current cooking temperature, the actions required to cook the ingredient, and the times when the ingredients were added to the cooking surface 44, the system controller 110 may track the cooked state of the ingredients. By way of example, the system controller 110 may determine a remaining cooking time before an ingredient should be flipped or a remaining cooking time before the ingredient should be removed.Based on the cooked state of the ingredient, the system controller 110 may determine one or more actions to be performed by the employee to complete their assigned task. By way of example, the system controller 110 may determine that the ingredient should be flipped or removed from the cooking surface 44. By way of another example, the system controller 110 may determine that an ingredient should be added to the cooking surface 44 according to the assigned task.

[0138] When the system controller 110 determines an action to be performed by the employee, the system controller 110 may command the employee to perform the action. The system controller 110 may indicate the command using the employee user interfaces 130. By way of example, the system controller 110 may use the screens 132 to provide an image showing an ingredient along with an instruction to flip or remove the ingredient. By way of another example, the system controller 110 may use the screens 132 to show an ingredient that should be added to the cooking surface 44 and where the ingredient can be found (e.g., in the storage area 22). By way of another example, the system controller 110 may use a projector 136 to emit a beam of light that forms an image on an ingredient. The resultant image may indicate that the ingredient should be removed or flipped. By way of another example, the system controller 110 may use a projector 136 to emit a first beam of light that forms an image (e.g., the image II in FIG. 3) on an ingredient to be added to the cooking surface 44 and a second beam of light that forms a second image (e.g., the image 12 in FIG. 3) on the cooking surface 44 where the ingredient should be placed.

[0139] Referring to FIGS. 1-3 and 5-25, the kitchen vision system 10 may be used to provide guidance to an employee assembling food products at an assembly workstation 50. By way of example, the kitchen vision system 10 may guide an employee producing a chicken sandwich as shown in FIGS. 5-11 or an employee producing a cheeseburger as shown in FIGS. 12-25.

[0140] The task assigned to an employee may indicate a type of food product to be assembled by the employee. The system controller 110 may utilize the menu data 150 to determine the ingredients required to form the food product and / or any specific instructions to follow when assembling the food product. The instructions may include an order inwhich to assemble the food product, an amount of each ingredient to include, or other information.

[0141] The system controller 110 may use the image data 156 to monitor the ingredients 160 within the storage bins 52 and a food product being prepared atop the work surface 54. When a container (e.g., a paper wrapper, a box, etc.) is placed onto the work surface 54, the system controller 110 may determine that assembly of a new food product 164 has been initiated. The system controller 110 may monitor the addition of ingredients 160 to the food product 164 and determine an amount and type of ingredients 160 that have been added. The system controller 110 may compare the added ingredients 160 to the ingredients specified in the menu data 150. Based on the comparison, the system controller 110 may determine a current stage of assembly of the food product 164.

[0142] Based on the current stage of assembly and the menu data 150, the system controller 110 may determine one or more actions to be performed by the employee to complete the food product 164. By way of example, the system controller 110 may determine the next ingredient 160 that should be added to the food product 164. By way of another example, the system controller 110 may determine where the ingredient 160 should be retrieved from (e.g., which storage bin 52 contains the ingredient 160). By way of another example, system controller 110 may determine a location on the food product 164 where the ingredient 160 should be placed.

[0143] When the system controller 110 determines an action to be performed by the employee, the system controller 110 may command the employee to perform the action. The system controller 110 may indicate the command using the employee user interfaces 130. By way of example, the system controller 110 may use a screen 132 to provide an image showing an ingredient 160, a location of the storage bins 52 where the ingredient 160 is stored, and an amount of the ingredient 160 that should be added to the food product 164. By way of another example, the system controller 110 may use a projector 136 to emit a beam of light that forms an image on a storage bin 52 containing an ingredient. The resultant image may indicate that the ingredient should be added to the food product 164.

[0144] Once the system controller 110 determines that the food product 164 is complete, the system controller 110 may command the employee to place the food product 164 in a holding area 56. By way of example, the system controller 110 may control a screen 132 to produce text stating that the food product 164 is completed and should be placed in the holding area 56. By way of another example, the system controller 110 may control a projector 136 to emit a beam of light that forms an image on the holding area 56 indicating a location where the food product 164 should be placed. Once the system controller 110 determines that the food product 164 has been placed in the holding area 56 (e.g., based on the image data 156), the system controller 110 may update the inventory to indicate that the food product 164 is available.

[0145] Referring to FIG. 26, a graphical user interface (GUI), shown as employee GUI 200, is an example of a user interface generated by the system controller 110. The employee GUI 200 may provide real-time information, commands, and other guidance to an employee operating an assembly workstation 50. The system controller 110 may control a screen 132 positioned nearby the assembly workstation 50 to display the employee GUI 200 while the employee assembles food product. In other embodiments, the system controller 110 provides a similar GUI for employees at preparation workstations 40.

[0146] The employee GUI 200 includes a first section or storage bin status indictor, shown as storage bin section 210, that is displayed as a portion of the employee GUI 200. The storage bin section 210 indicates a current status of each storage bin 52 of the assembly workstation 50. As shown, the storage bin section 210 includes a grid (e.g., arranged in rows and columns) formed of first indicators, shown as empty bin indicators 212, and second indicators, shown as filled bin indicators 214.

[0147] The empty bin indicators 212 and the filled bin indicators 214 illustrate the layout of the storage bins 52 on the assembly workstation 50. By way of example, the system controller 110 may use the image data 156 to determine the size, location, and contents of each storage bin 52. The system controller 110 may update the empty bin indicators 212 and the filled bin indicators 214 to match the determined size, location, and contents of the storage bins 52. If the storage bins 52 of the assembly workstation 50 change, the systemcontroller 110 may automatically update the empty bin indicators 212 and / or the filled bin indicators 214 to match the updated arrangement of the storage bins 52.

[0148] The empty bin indicators 212 indicate storage bins 52 that do not contain any prepared ingredients. The filled bin indicators 214 indicate storage bins 52 that contain prepared ingredients. As shown, the filled bin indicators 214 each include a label that indicates the type of ingredient contained within the corresponding storage bin 52. Beneficially, the employee may refer to the employee GUI 200 to quickly identify which storage bins 52 contain each ingredient and where the storage bins 52 are located on the assembly workstation 50.

[0149] By way of example, “BOLO” may indicate that the corresponding storage bin 52 contains bologna. By way of example, “B ACN” may indicate that the corresponding storage bin 52 contains bacon. By way of example, “JALA” may indicate that the corresponding storage bin 52 contains jalapeno peppers. By way of example, “HAM” may indicate that the corresponding storage bin 52 contains ham. By way of example, “MORT” may indicate that the corresponding storage bin 52 contains mortadella. By way of example, “LETT” may indicate that the corresponding storage bin 52 contains lettuce. By way of example, “ONIO” may indicate that the corresponding storage bin 52 contains onions. By way of example, “TOMA” may indicate that the corresponding storage bin 52 contains tomatoes.

[0150] The filled bin indicators 214 may be colored, textured, or otherwise visually distinguished from one another to indicate an amount of the prepared ingredient in the corresponding storage bin 52. As shown in FIG. 26, a first subset of the filled bin indicators 214 are hatched (e.g., indicating the color green), and a second subset of the filled bin indicators 214 are stippled (e.g., indicating the color green). The hatching on the filled bin indicators 214 indicates that the corresponding storage bins 52 contain more than the threshold amount of prepared ingredients. The stippling on the filled bin indicators 214 indicates that the corresponding storage bins 52 contain less than the threshold amount of prepared ingredients. Beneficially, instructs the employee which of the storage bins 52 should be refilled.

[0151] The employee GUI 200 further includes a predicted ingredient usage section, shown as prediction section 220, that is displayed as a portion of the employee GUI 200. The prediction section 220 is arranged as a table including a first column, shown as ingredient column 222, a second column, shown as short-term column 224, and a third column, shown as long-term column 226. The ingredient column 222 includes a list of prepared ingredients used by (e.g., contained within, available to, etc.) the assembly workstation 50. The ingredient column 222 may include the prepared ingredients from the filled bin indicators 214, as well as other prepared ingredients (e.g., salad, cheddar cheese, gouda cheese, parmesan cheese, pickles, etc.) that are used by the assembly workstation 50.

[0152] The short-term column 224 indicates an amount of each prepared ingredient that is predicted to be used by the assembly workstation 50 in a short-term period (e.g., the next 30 minutes). The long-term column 226 indicates an amount of each prepared ingredient that is predicted to be used by the assembly workstation 50 in a long-term period (e.g., the next hour). The system controller 110 may generate the short-term column 224 and the longterm column 226 based on the predicted demand. Beneficially, the prediction section 220 may facilitate the employee knowing what prepared ingredients are expected to be in high demand (e.g., higher than a nominal demand, a standard demand, an average demand, etc.) in the future and preparing to acquire more of those prepared ingredients if it appears that the storage bins 52 will not be sufficient to meet the demand.

[0153] The employee GUI 200 further includes a series of food product instruction sections or assigned task sections, shown as desired food product sections 230, that are displayed as a portion of the employee GUI 200. The desired food product sections 230 each indicate a desired food product that has been assigned to the employee at the assembly workstation 50 for assembly. Each desired food product section 230 includes a name of the desired food product and / or another identifier for the food product (e.g., an identification number or code), as well as a list of ingredients required to make the desired food product. The system controller 110 may determine the required ingredients based on the menu data 150.

[0154] The system controller 110 may update the desired food product sections 230 to indicate a status of each ingredient added to the corresponding food product. As shown, afirst indicator (e.g., a blank box next to the ingredient name) indicates that the ingredient has not yet been added to the food product. A second indicator (e.g., a check mark next to the ingredient name) indicates that the ingredient has been successfully added to the food product. A third indicator (e.g., an X next to the ingredient name) indicates that the order requested a modification to the standard recipe for the food product to remove an ingredient. In the desired food product sections 230 shown to the left in FIG. 26, the food product is a BLT sandwich that has successfully had lettuce added, still requires bacon, tomato, and pickles, and has been modified to request that no onions are added.Error Identification

[0155] Referring to FIGS. 1-3, the system controller 110 may detect one or more errors made by employees when performing assigned tasks. By way of example, the system controller 110 may utilize the image data 156 to detect the errors. If the errors were to go unnoticed or uncorrected, the errors could result in inaccurate orders, wasted ingredients, delayed order completion, or other negative effects. The system controller 110 may perform actions to correct the errors or otherwise mitigate the negative effects of the errors.

[0156] In some embodiments, the system controller 110 detects errors made while preparing ingredients (e.g., cooking, chopping, etc.). By way of example, the system controller 110 may detect when an employee selects a wrong ingredient for preparation. In one such example, an employee begins chopping red onions when assigned to prepare yellow onions. By way of another example, the system controller 110 may detect when an employee selects a wrong amount of an ingredient for preparation. In one such example, an employee cooks only one beef patty when assigned to prepare ten beef patties. By way of another example, the system controller 110 may detect when an employee fails to prepare an ingredient properly before proceeding to the next step in an assigned task. In one such example, an employee forgets to beat an egg prior to adding the egg to a cooking surface 44 when assigned to make scrambled eggs. By way of another example, the system controller 110 may detect when an employee fails to prepare an ingredient to a desired cooked status. In one such example, an employee overcooks a chicken patty on the cooking surface 44. In another such example, an employee removes a beef patty before the beef patty has reached a fully cooked temperature.

[0157] In some embodiments, the system controller 110 detects errors made when assembling food products. By way of example, the system controller 110 may detect when an employee adds a wrong ingredient to a food product. In one such example, an employee adds onions to a chicken sandwich whose menu data 150 indicates that the chicken sandwich should not contain onions. By way of another example, the system controller 110 may detect when an employee fails to add an ingredient to a food product. In one such example, an employee forgets to add pickles to a sandwich whose menu data 150 indicate that the sandwich requires pickles. By way of another example, the system controller 110 may detect when an employee adds a wrong amount of an ingredient to a food product. In some such examples, the employee adds too much of a particular ingredient or too little of a particular ingredient (e.g., more or less than specified in the menu data 150). By way of another example, the system controller 110 may detect when an employee adds an ingredient in a wrong order. In one such example, an employee adds a bun between two other ingredients instead of placing the bun on the outside of a sandwich.

[0158] In some embodiments, the system controller 110 determines whether a solution to the error is possible or available. The system controller 110 may store a predetermined list of possible errors and corresponding solutions. By way of example, if an employee adds too much of an ingredient to a food product or adds a wrong ingredient to the food product, the solution may include removing the ingredient from the food product. By way of another example, if an employee adds a wrong ingredient to a food product, the solution may include setting aside the food product and saving the partially-assembled food product for use in a different food product that requires that ingredient. By way of another example, if an employee adds an ingredient to the cooking surface 44 without mixing, the solution may include mixing the ingredient after the ingredient is added to the cooking surface 44. By way of another example, if an employee removes an ingredient from a cooking surface 44 before the ingredient reaches a desired cooked state, the solution may be to add the ingredient back to the cooking surface 44. If the predetermined list of possible errors indicates that the error does not have a corresponding solution, or if the system controller 110 is unable to locate the error within the predetermined list, the system controller 110 may determine that there is no solution for the error.

[0159] If the system controller 110 determines that a solution is available, the system controller 110 may control the employee user interface 130 to provide a notification indicating that an error has occurred and commanding an employee to perform the corresponding solution. By way of example, the system controller 110 may control a screen 132 to display a description of the error and a corresponding solution. By way of another example, the system controller 110 may control a projector 136 to emit a beam of red light toward an employee’s hand and emit a beam of green light toward a desired ingredient when a wrong ingredient is grabbed by a user. If the system controller 110 determines that a solution is not available, the system controller 110 may control the employee user interface 130 to indicate that an error has occurred and the ingredient or food product should be discarded (e.g., into a waste receptacle 48). The system controller 110 may control the employee user interface 130 to command the employee to remake the ingredient or food product that was discarded.

[0160] FIG. 27 illustrates an example of a notification indicating that an error has occurred. Specifically, FIG. 27 illustrates a variation of the employee GUI 200 including a notification, shown as popup 240. The system controller 110 may update the employee GUI 200 to include the popup 240 in response to a determination that an error has occurred. Specifically, the popup 240 illustrates that a wrong ingredient has been added to a food product. The popup 240 specifies that the wrong ingredient added was onions, and the affected product was a BLT sandwich. The popup 240 may indicate to an employee that the error is present and should be corrected (e.g., by removing the onions or disposing of the BLT sandwich in a waste receptacle 48).

[0161] When an error occurs, the system controller 110 may associate the error with employee data 154 corresponding to the employee that made the error. By way of example, the system controller 110 may identify the employee based on image data 156 showing the employee. By way of another example, the system controller 110 may identify the employee based on an identity of an employee assigned to the preparation workstation 40 or the assembly workstation 50 where the error occurred.

[0162] When detecting an error, the system controller 110 may determine error data including the date and time of the error, the action that resulted in the error, the action thatshould have been performed instead of the error, whether the error was corrected, and a consequence of making the error (e.g., an amount of wasted ingredients). The system controller 110 may store the error data with the employee data 154 for future reference regarding that particular employee.

[0163] The collective error data for a given employee may be utilized to develop an error profile for that employee. The error profile may indicate what errors the employee has made in the past. The error profile may predict what errors the employee is predicted to make in the future. By way of example, if an employee consistently experiences errors when performing a particular task (e.g., producing a particular food product), the system controller 110 may indicate in the error profile that the employee is likely to make similar errors in the future. In some embodiments, the system controller 110 assigns tasks to employees based on the error profiles for those employees. By way of example, a system controller 110 may seek to avoid assigning a task to an employee that is known to have a high likelihood to make an error for that task when another employee without the high likelihood for error is available instead. Accordingly, the system controller 110 may assign the task to the second employee instead of the first employee based on the error profiles.

[0164] In some embodiments, the system controller 110 tracks wasted ingredients (e.g., ingredients that are thrown away instead of being made into a food product and sold). The system controller 110 may record both what types of ingredients are wasted and how much of each type is wasted. In some embodiments, the system controller 110 determines that an ingredient or the ingredients within a partially-assembled food product are wasted when the system controller 110 determines that there is no solution to an error made with respect to the ingredient or food product. In some embodiments, the system controller 110 uses the image data 156 to monitor the placement of ingredients into the waste receptacles 48. The system controller 110 may determine that an ingredient is wasted when the image data 156 shows the ingredient or a food product containing the ingredient being placed into a waste receptacle 48. Each instance of waste may be associated with a responsible employee and added to the error profile of that employee. By way of example, the responsible employee may be the individual that makes the error that causes the waste or the employee that places the ingredients in the waste receptacles 48.Manager Interface

[0165] The kitchen vision system 10 may provide a manager with various information about the performance of the kitchen vision system 10 through the manager interface devices 144. The manager interface devices 144 may provide information about individual employees, information about the kitchen 20 as a whole, and information showing comparisons between the kitchen 20 and other locations (e.g., other stores). The manager interface devices 144 may facilitate the manager assessing the performance of the kitchen vision system 10 and completing ordering or other tasks.

[0166] In some embodiments, the manager interface devices 144 provide the manager with information regarding the performance of individual employees. The system controller 110 may use the image data 156 to monitor the performance of each employee and store various information regarding the employees as employee data 154. In some embodiments, the employee data 154 identifies specific individuals (e.g., by name). In other embodiments, the employee data 154 is anonymized. For example, the data can be anonymized and then aggregated for statistical purposes, e.g., analyzing performance over time.

[0167] In some embodiments, the system controller 110 determines an amount of time that an employee takes to complete a task (e.g., a completion time, etc.). The system controller 110 may count the completion time from the when the task is initiated (e.g., when the employee picks up the first ingredient associated with the task) to when the task is completed (e.g., an ingredient is prepared, a food product is placed in the holding area 56, etc.). The system controller 110 may store completion times for different tasks within the employee data 154 for an employee.

[0168] The system controller 110 may evaluate the performance of an employee based on the employee data 154. By way of example, the system controller 110 may evaluate the employee based on the completion times and the error profile of the employee. An employee having higher average completion times may be rated more poorly. An employee having a greater likelihood of producing errors may be rated more poorly. Based on the evaluation, each employee may be assigned a grade rating the performance of the employee.

[0169] The employee GUI 200 of FIGS. 26 and 27 each include a rating or score, shown as grade 242, for a corresponding employee. As shown in FIGS. 26 and 27, the grade 242 is A+, indicating high performance. In other embodiments, the grade 242 utilizes different grading scales (e.g. a percentage grade instead of a letter grade). In some embodiments, the employee GUI 200 is available through the manager interface devices 144 to facilitate a manager reviewing the grade 242 and other information available on the employee GUI 200.

[0170] FIG. 28 illustrates a GUI, shown as grading GUI 250, that is displayed on the manager interface devices 144. The grading GUI 250 illustrates employee data 154 that evaluates the performance of an employee. The grading GUI 250 includes a series of rows, each corresponding to a different task performed by the employee. The tasks include, for example, producing a burger, producing fries, etc. Each task may include a completion status (e.g., on time, cook delay, never cooked, etc.), as wall as a completion time for the task. Based on the completion status and the completion time, the system controller 110 assigns a score for the employee. The composite (e.g., average score) for the employee may be considered the grade.

[0171] In some embodiments, the manager interface devices 144 provide the manager with information regarding the performance of the kitchen 20. Throughout operation, the system controller 110 may track the amount of ingredients used and the amount of ingredients wasted (e.g., based on the image data 156). The system controller 110 may determine the actual sales of products based on the order data 152. The system controller 110 may compare the predicted sales of food products and predicted demand for ingredients with the actual sales of products, the actual amount of ingredients used, and the actual amount of ingredients wasted. Based on this comparison, the manager may determine how accurate the prediction was and may receive new predictions for the future.

[0172] Additionally, the manager interface devices 144 may facilitate comparing the performance of the kitchen 20 with the performance of kitchens in other stores or locations. The manager interface devices 144 may permit the manager to compare employee performance, total sales, the amount of particular food products sold, the amount of ingredients wasted, and other information between different stores. Based on thiscomparison, the manager may evaluate how one store is performing relative to other stores and determine if a change in procedure is warranted.

[0173] FIG. 29 illustrates a manager GUI, shown as sales comparison GUI 260. The sales comparison GUI 260 includes a series of rows each corresponding to a different food product produced by the kitchen 20. Each row includes a predicted demand for the food product (e.g., “planned”) as well as the actual sales numbers for that food product (e.g., “adjusted”). The sales comparison GUI 260 further includes a comparison of the planned and adjusted numbers (e.g., “change”). Based on the percent change, the manager may evaluate the accuracy of the predicted demand and whether the procedures need to change to accommodate future changes in demand.

[0174] FIG. 30 illustrates a manager GUI, shown as food product summary GUI 270. The food product summary GUI 270 may illustrates statistics regarding the production and sales of a particular food product on a particular day. The food product summary GUI 270 includes a series of rows each corresponding to a different time. For each time, the food product summary GUI 270 includes a predicted demand for the food product, a quantity of the food product sold, an amount of the food product wasted, and a number of the food product remaining. The food product summary GUI 270 may facilitate the manager determining the current production status of each food product.

[0175] As utilized herein with respect to numerical ranges, the terms “approximately,” “about,” “substantially,” and similar terms generally mean + / - 10% of the disclosed values. When the terms “approximately,” “about,” “substantially,” and similar terms are applied to a structural feature (e.g., to describe its shape, size, orientation, direction, etc.), these terms are meant to cover minor variations in structure that may result from, for example, the manufacturing or assembly process and are intended to have a broad meaning in harmony with the common and accepted usage by those of ordinary skill in the art to which the subject matter of this disclosure pertains. Accordingly, these terms should be interpreted as indicating that insubstantial or inconsequential modifications or alterations of the subject matter described and claimed are considered to be within the scope of the disclosure as recited in the appended claims.

[0176] It should be noted that the term “exemplary” and variations thereof, as used herein to describe various embodiments, are intended to indicate that such embodiments are possible examples, representations, or illustrations of possible embodiments (and such terms are not intended to connote that such embodiments are necessarily extraordinary or superlative examples).

[0177] The term “coupled” and variations thereof, as used herein, means the joining of two members directly or indirectly to one another. Such joining may be stationary (e.g., permanent or fixed) or moveable (e.g., removable or releasable).

[0178] References herein to the positions of elements (e.g., “top,” “bottom,” “above,” “below”) are merely used to describe the orientation of various elements in the FIGURES. It should be noted that the orientation of various elements may differ according to other exemplary embodiments, and that such variations are intended to be encompassed by the present disclosure.

[0179] The aforementioned techniques can be utilized in connection with one or more drive-through ordering systems, point-of-sale systems, and drive-through interfaces for facilitating orders. In addition, the aforementioned techniques can be utilized in connection with order preparation at other locations, e.g., entertainment venues, ballparks, stadiums, arenas, and elsewhere. Suitable ordering systems, methods, and techniques are set forth, for example, in U.S. Patent No. 11,244,681, granted on February 8, 2022, U.S. Patent No.11,741,529, granted on August 29, 2023, U.S. Patent No. 11,328,278, granted May 10, 2022, U.S. Provisional Patent Application No. 63 / 452,218, filed on March 15, 2023, U.S. Provisional Patent Application No. 63 / 529,850 filed on July 31, 2023, U.S. Provisional Patent Application No. 63 / 539,920 filed on September 22, 2023, U.S. Provisional Patent Application No. 63 / 587,611 filed on October 3, 2023, U.S. Provisional Patent Application No. 63 / 606,510 filed December 5, 2023, U.S. Provisional Patent Application No.63 / 627,710 filed January 31, 2024, U.S. Patent Application No. 18 / 606,958 filed March 15, 2024, U.S. Patent Application No. 18 / 607,065 filed March 15, 2024, and U.S. Patent Application No. 18 / 607,011 filed March 15, 2024. The entire contents of the aforementioned patents and patent applications are incorporated herein by reference forbackground information and the systems, components, processes and techniques disclosed therein.

[0180] The hardware and data processing components used to implement the various processes, operations, illustrative logics, logical blocks, modules and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose single- or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or, any conventional processor, controller, microcontroller, or state machine. A processor also may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some embodiments, particular processes and methods may be performed by circuitry that is specific to a given function. The memory (e.g., memory, memory unit, storage device) may include one or more devices (e.g., RAM, ROM, Flash memory, hard disk storage) for storing data and / or computer code for completing or facilitating the various processes, layers and modules described in the present disclosure. The memory may be or include volatile memory or non-volatile memory, and may include database components, object code components, script components, or any other type of information structure for supporting the various activities and information structures described in the present disclosure. According to an exemplary embodiment, the memory is communicably connected to the processor via a processing circuit and includes computer code for executing (e.g., by the processing circuit or the processor) the one or more processes described herein.

[0181] The present disclosure contemplates methods, systems and program products on any machine-readable media for accomplishing various operations. The embodiments of the present disclosure may be implemented using existing computer processors, or by a special purpose computer processor for an appropriate system, incorporated for this or another purpose, or by a hardwired system. Embodiments within the scope of the presentdisclosure include program products comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine- readable media can be any available media that can be accessed by a general purpose or special purpose computer or other machine with a processor. By way of example, such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code in the form of machine-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer or other machine with a processor. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions.

[0182] Although the figures and description may illustrate a specific order of method steps, the order of such steps may differ from what is depicted and described, unless specified differently above. Also, two or more steps may be performed concurrently or with partial concurrence, unless specified differently above. Such variation may depend, for example, on the software and hardware systems chosen. All such variations are within the scope of the disclosure. Likewise, software implementations of the described methods could be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps, and decision steps.

[0183] The following enumerated paragraphs set forth various exemplary non-limiting embodiments that can be combined, arranged, or modified in a variety of manners. For example, the paragraphs below identified alphanumerically and relating to exemplary systems set forth embodiments or aspects of embodiments that can be performed according to exemplary methods in keeping with the principles of the present disclosure.Paragraph Al . A kitchen vision system for a restaurant, comprising: a camera configured to record image data of a workstation; andone or more processing circuits comprising one or more memory devices coupled to one or more processors, the one or more memory devices configured to store instructions that, when executed by the one or more processors, cause the one or more processors to: receive an indication of a food product to be prepared; determine an ingredient required to produce the food product; and determine, based on the image data, whether the ingredient has been retrieved from a storage portion of the workstation.Paragraph A2. The kitchen vision system of Paragraph Al, wherein the instructions cause the one or more processors to: determine, based on the image data, whether the ingredient has been added to a food product being produced at the workstation; and in response to a determination that the ingredient has been added to the food product being produced at the workstation, provide a notification indicating a stage of assembly of the food product.Paragraph A3. The kitchen vision system of Paragraph A3, further comprising a user interface configured to provide information to a user, wherein the instructions cause the one or more processors to provide, through the user interface, a command identifying the ingredient required to produce the food product.Paragraph A4. The kitchen vision system of Paragraph A3, wherein the ingredient is a first ingredient of a plurality of ingredients required to produce the food product, and wherein the instructions cause the one or more processors to provide, through the user interface, a command identifying a second ingredient of the plurality of ingredients in response to a determination that the first ingredient has been retrieved from the storage portion of the workstation and added to a food product being produced at the workstation.Paragraph A5. The kitchen vision system of Paragraph A4, wherein the instructions cause the one or more processors to: determine, based on the image data, whether a third ingredient that is not one of the plurality of ingredients required to prepare the food product been added to the food product being produced at the workstation; andprovide, through the user interface, an indication that an error has occurred in response to a determination that the third ingredient has been added to the food product being produced at the workstation.Paragraph A6. The kitchen vision system of Paragraph A5, wherein the instructions cause the one or more processors to provide, through the user interface, a command to remake the food product in response to a determination that the third ingredient has been added to the food product being produced at the workstation.Paragraph A7. The kitchen vision system of Paragraph A5, wherein the instructions cause the one or more processors to provide, through the user interface, a command to remove the third ingredient from the food product being produced at the workstation in response to a determination that the third ingredient has been added to the food product being produced at the workstation.Paragraph A8. The kitchen vision system of Paragraph A3, wherein the user interface includes a projector configured to project an image onto the workstation.Paragraph A9. The kitchen vision system of Paragraph A8, wherein the command identifying the ingredient is a projection of the image onto the storage portion of the workstation.Paragraph A10. The kitchen vision system of Paragraph A9, wherein the ingredient is a first ingredient, wherein the storage portion of the workstation is a first bin containing the first ingredient, wherein the workstation further includes a second bin containing a second ingredient, and wherein the image is not projected onto the second bin.Paragraph Al l. The kitchen vision system of Paragraph A3, wherein the food product is a first food product and the ingredient is a first ingredient, wherein the command further identifies a second food product, and wherein the instructions cause the one or more processors to: determine an elapse based on a first preparation time associated with the first food product and a second preparation time associated with the second food product;initiate a timer in response to a determination that the first ingredient has been retrieved from the storage portion; and in response to a determination that the timer has exceeded the elapse, provide, through the user interface, a command identifying a second ingredient required to produce the second food product.Paragraph A12. The kitchen vision system of Paragraph A3, further comprising a point of sale device, wherein the command to prepare the food product is an order identifying the food product, and wherein the point of sale device is configured to receive the order from at least one of a customer or an employee.Paragraph A13. The kitchen vision system of Paragraph A3, wherein the instructions cause the one or more processors to generate the command to prepare the food product in response to entering a period of predicted demand higher than a nominal demand.Paragraph A14. The kitchen vision system of Paragraph A13, wherein the instructions cause the one or more processors to predict the period of predicted demand higher than a nominal demand based on historical order data corresponding to past orders including the food product.Paragraph Al 5. The kitchen vision system of Paragraph Al, further comprising a user interface configured to provide information to a user, wherein the instructions cause the one or more processors to: determine, based on the image data, an amount of the ingredient within the storage portion of the workstation; and provide, through the user interface, an instruction to restock the storage portion of the workstation with the ingredient in response to a determination that the amount of the ingredient within the storage portion is below a threshold amount.Paragraph Al 6. The kitchen vision system of Paragraph Al, wherein the camera is a first camera and the image data is first image data, wherein the kitchen vision system further comprises a second camera configured to record second image data of a storage area of the restaurant, and wherein the instructions cause the one or more processors to:determine, based on the second image data, an amount of the ingredient within the storage area; and provide a command to restock the storage area with the ingredient in response to a determination that the amount of the ingredient within the storage portion is below a threshold amount.Paragraph Al 7. The kitchen vision system of Paragraph Al 6, wherein the command to restock the storage area is a request for delivery of the ingredient to the restaurant from a delivery service outside of the restaurant.Paragraph A18. The kitchen vision system of Paragraph Al, wherein the storage portion is a first storage portion and the ingredient is a first ingredient, and wherein the instructions cause the one or more processors to: determine, based on the image data, that a second ingredient has been retrieved from a second storage portion of the workstation; determine, based on the image data, that the second ingredient has been wasted; and provide a notification indicating that the second ingredient has been wasted.Paragraph Al 9. The kitchen vision system of Paragraph Al 8, wherein the instructions cause the one or more processors to determine, based on the image data, an amount of the second ingredient that has been wasted, and wherein the notification indicates the amount of the second ingredient that has been wasted.Paragraph A20. The kitchen vision system of Paragraph Al, wherein the instructions cause the one or more processors to: determine, based on the image data, that an error has occurred in preparing the food product; identify an employee that made the error; and provide, based on the error, an evaluation of the employee.Paragraph A21. A method of preparing a food product within a kitchen, the method comprising: receiving an indication of the food product to be prepared;&.... ...&.^v..ent required to produce the food product; and determining, based on image data from a camera that observes a workstation within the kitchen, whether the ingredient has been retrieved from a storage portion of the workstation.Paragraph A22. A non-transitory computer readable medium configured to store instructions, which, when executed by a processor, cause the processor to: receive an indication of a food product to be prepared; determine an ingredient required to produce the food product; and determine, based on image data from a camera that observes a workstation within a kitchen, whether the ingredient has been retrieved from a storage portion of the workstation.

[0184] It is important to note that the construction and arrangement of the kitchen vision system 10 as shown in the various exemplary embodiments is illustrative only.Additionally, any element disclosed in one embodiment may be incorporated or utilized with any other embodiment disclosed herein. For example, the overlay of the exemplary embodiment shown in at least FIGS. 5-11 may be utilized with the image data of the exemplary embodiment shown in at least FIGS. 12-26. Although only one example of an element from one embodiment that can be incorporated or utilized in another embodiment has been described above, it should be appreciated that other elements of the various embodiments may be incorporated or utilized with any of the other embodiments disclosed herein.

Claims

WHAT IS CLAIMED IS:

1. A kitchen vision system for a restaurant, comprising: a camera configured to record image data of a workstation; and one or more processing circuits comprising one or more memory devices coupled to one or more processors, the one or more memory devices configured to store instructions that, when executed by the one or more processors, cause the one or more processors to: receive an indication of a food product to be prepared; determine an ingredient required to produce the food product; and determine, based on the image data, whether the ingredient has been retrieved from a storage portion of the workstation.

2. The kitchen vision system of Claim 1, wherein the instructions cause the one or more processors to: determine, based on the image data, whether the ingredient has been added to a food product being produced at the workstation; and in response to a determination that the ingredient has been added to the food product being produced at the workstation, provide a notification indicating a stage of assembly of the food product.

3. The kitchen vision system of Claim 1, further comprising a user interface configured to provide information to a user, wherein the instructions cause the one or more processors to provide, through the user interface, a command identifying the ingredient required to produce the food product.

4. The kitchen vision system of Claim 3, wherein the ingredient is a first ingredient of a plurality of ingredients required to produce the food product, and wherein the instructions cause the one or more processors to provide, through the user interface, a command identifying a second ingredient of the plurality of ingredients in response to a determination that the first ingredient has been retrieved from the storage portion of the workstation and added to a food product being produced at the workstation.

5. The kitchen vision system of Claim 4, wherein the instructions cause the one or more processors to: determine, based on the image data, whether a third ingredient that is not one of the plurality of ingredients required to prepare the food product been added to the food product being produced at the workstation; and provide, through the user interface, an indication that an error has occurred in response to a determination that the third ingredient has been added to the food product being produced at the workstation.

6. The kitchen vision system of Claim 5, wherein the instructions cause the one or more processors to provide, through the user interface, a command to remake the food product in response to a determination that the third ingredient has been added to the food product being produced at the workstation.

7. The kitchen vision system of Claim 5, wherein the instructions cause the one or more processors to provide, through the user interface, a command to remove the third ingredient from the food product being produced at the workstation in response to a determination that the third ingredient has been added to the food product being produced at the workstation.

8. The kitchen vision system of Claim 3, wherein the user interface includes a projector configured to project an image onto the workstation.

9. The kitchen vision system of Claim 8, wherein the command identifying the ingredient is a projection of the image onto the storage portion of the workstation.

10. The kitchen vision system of Claim 9, wherein the ingredient is a first ingredient, wherein the storage portion of the workstation is a first bin containing the first ingredient, wherein the workstation further includes a second bin containing a second ingredient, and wherein the image is not projected onto the second bin.

11. The kitchen vision system of Claim 3, wherein the food product is a first food product and the ingredient is a first ingredient, wherein the command further identifies a second food product, and wherein the instructions cause the one or more processors to: determine an elapse based on a first preparation time associated with the first food product and a second preparation time associated with the second food product; initiate a timer in response to a determination that the first ingredient has been retrieved from the storage portion; and in response to a determination that the timer has exceeded the elapse, provide, through the user interface, a command identifying a second ingredient required to produce the second food product.

12. The kitchen vision system of Claim 3, further comprising a point of sale device, wherein the command to prepare the food product is an order identifying the food product, and wherein the point of sale device is configured to receive the order from at least one of a customer or an employee.

13. The kitchen vision system of Claim 3, wherein the instructions cause the one or more processors to generate the command to prepare the food product in response to entering a period of predicted demand higher than a nominal demand.

14. The kitchen vision system of Claim 13, wherein the instructions cause the one or more processors to predict the period of predicted demand higher than a nominal demand based on historical order data corresponding to past orders including the food product.

15. The kitchen vision system of Claim 1, further comprising a user interface configured to provide information to a user, wherein the instructions cause the one or more processors to: determine, based on the image data, an amount of the ingredient within the storage portion of the workstation; and provide, through the user interface, an instruction to restock the storage portion of the workstation with the ingredient in response to a determination that the amount of the ingredient within the storage portion is below a threshold amount.

16. The kitchen vision system of Claim 1, wherein the camera is a first camera and the image data is first image data, wherein the kitchen vision system further comprises a second camera configured to record second image data of a storage area of the restaurant, and wherein the instructions cause the one or more processors to: determine, based on the second image data, an amount of the ingredient within the storage area; and provide a command to restock the storage area with the ingredient in response to a determination that the amount of the ingredient within the storage portion is below a threshold amount.

17. The kitchen vision system of Claim 16, wherein the command to restock the storage area is a request for delivery of the ingredient to the restaurant from a delivery service outside of the restaurant.

18. The kitchen vision system of Claim 1, wherein the storage portion is a first storage portion and the ingredient is a first ingredient, and wherein the instructions cause the one or more processors to: determine, based on the image data, that a second ingredient has been retrieved from a second storage portion of the workstation; determine, based on the image data, that the second ingredient has been wasted; and provide a notification indicating that the second ingredient has been wasted.

19. The kitchen vision system of Claim 18, wherein the instructions cause the one or more processors to determine, based on the image data, an amount of the second ingredient that has been wasted, and wherein the notification indicates the amount of the second ingredient that has been wasted.

20. The kitchen vision system of Claim 1, wherein the instructions cause the one or more processors to: determine, based on the image data, that an error has occurred in preparing the food product; identify an employee that made the error; and provide, based on the error, an evaluation of the employee.

21. A method of preparing a food product within a kitchen, the method comprising: receiving an indication of the food product to be prepared; determining an ingredient required to produce the food product; and determining, based on image data from a camera that observes a workstation within the kitchen, whether the ingredient has been retrieved from a storage portion of the workstation.

22. A non-transitory computer readable medium configured to store instructions, which, when executed by a processor, cause the processor to: receive an indication of a food product to be prepared; determine an ingredient required to produce the food product; and determine, based on image data from a camera that observes a workstation within a kitchen, whether the ingredient has been retrieved from a storage portion of the workstation.

Images