Integrated robotic kitchen system, and robotic kitchen system

Abstract

The present disclosure relates to methods, computer program products, and computer systems for instructing a robot to prepare a food dish by replacing the movements and actions of a human chef. [Solution] Monitoring the human chef is performed in an instrumented, application-specific environment, in this case a standardized robotic kitchen, and involves using sensors and a computer to watch, monitor, record, and interpret the motions and actions of the human chef to develop a robot-executable command set that is robust to variations and changes in the environment that enables the robot or automated system in the robotic kitchen to prepare dishes that are the same in terms of specification and quality as dishes prepared by the human chef.

Description

Technical Field

[0001] [Cross - reference to related applications] This application is a compilation of U.S. Provisional Patent Application No. 62 / 116,563, filed on February 16, 2015, entitled "Method and System for Food Preparation in a Robotic Cooking Kitchen," U.S. Provisional Patent Application No. 62 / 113,516, filed on February 8, 2015, entitled "Method and System for Food Preparation in a Robotic Cooking Kitchen," U.S. Provisional Patent Application No. 62 / 109,051, filed on January 28, 2015, entitled "Method and System for Food Preparation in a Robotic Cooking Kitchen," and a compilation of applications filed on January 16, 2015, entitled "Method and System for Robotic Cooking U.S. Provisional Patent Application No. 62 / 104,680 titled "Method and System for Robotic Cooking Kitchen," filed on December 10, 2014; U.S. Provisional Patent Application No. 62 / 090,310 titled "Method and System for Robotic Cooking Kitchen," filed on November 22, 2014; U.S. Provisional Patent Application No. 62 / 083,195 titled "Method and System for Robotic Cooking Kitchen," filed on October 31, 2014; U.S. Provisional Patent Application No. 62 / 073,846 titled "Method and System for Robotic Cooking Kitchen," filed on September 26, 2014;U.S. Provisional Patent Application No. 62 / 044,677, filed on September 2, 2014, entitled "Method and System for Robotic Cooking Kitchen"; U.S. Provisional Patent Application No. 62 / 024,948, filed on July 15, 2014, entitled "Method and System for Robotic Cooking Kitchen"; U.S. Provisional Patent Application No. 62 / 013,691, filed on June 18, 2014, entitled "Method and System for Robotic Cooking Kitchen"; U.S. Provisional Patent Application No. 62 / 013,502, filed on June 17, 2014, entitled "Method and System for Robotic Cooking Kitchen"; U.S. Provisional Patent Application No. 62 / 013,502, filed on June 17, 2014, entitled "Method and System for Robotic Cooking Kitchen" U.S. Provisional Patent Application No. 62 / 013,190 titled "Method and System for Robotic Cooking Kitchen," filed on May 8, 2014; U.S. Provisional Patent Application No. 61 / 990,431 titled "Method and System for Robotic Cooking Kitchen," filed on May 1, 2014; U.S. Provisional Patent Application No. 61 / 987,406 titled "Method and System for Robotic Cooking Kitchen," filed on March 16, 2014; U.S. Provisional Patent Application No. 61 / 953 titled "Method and System for Robotic Cooking Kitchen," filed on March 16, 2014.Patent No. 930 claims priority to U.S. Provisional Patent Application No. 61 / 942,559, titled "Method and System for Robotic Cooking Kitchen," filed on February 20, 2014, and the disclosures of these applications are incorporated herein by full quotation.

[0002] The present invention relates to the interdisciplinary fields of robotics and artificial intelligence in general, and more specifically to a computerized robotic food preparation system for food preparation by digitizing the food preparation processes of professional and non-professional chefs, and then reproducing the chef's cooking movements, processes, and techniques with real-time electronic adjustments. [Background technology]

[0003] Research and development in robotics has been ongoing for decades, but its progress has largely been in heavy industrial applications such as automotive manufacturing automation or military use. While simpler robotic systems have been designed for the consumer market, broad applications in the home consumer robot space have been largely unseen so far. The combination of technological advancements and a high-income population suggests this market is ripe for creating opportunities for technological advancements that improve people's lives. Robotics has continued to improve automation technology through advanced artificial intelligence and the imitation of many forms of human skills and tasks.

[0004] Since robots were first developed in the 1970s, the concept of robots replacing humans in certain fields and performing tasks that humans would normally do has been a continuously evolving conceptual form. The manufacturing sector has long used robots in teach-and-playback mode, where robots are taught which motions to continuously replicate without modification or deviation by generating and downloading fixed trajectories while the robot is idle or offline. Companies have incorporated pre-programmed trajectory execution and robot motion playback of computer-taught trajectories into application domains such as beverage mixing, welding, or automotive painting. However, all of these conventional applications have aimed solely at getting robots to faithfully execute motion commands, almost always using the one-to-one computer-to-robot or teach-and-playback principle where the robot follows a taught / pre-calculated trajectory without deviation.

[0005] Gastronomy is the art of fine dining, where gourmet recipes skillfully blend high-quality ingredients with flavors that appeal to all human senses. Gastronomy can be extremely elaborate, requiring specialized skills and techniques, and in some cases, following rules based on technique that necessitate extensive training. In recent years, the demand for gourmet food has risen dramatically, driven by rapidly increasing incomes and a generational shift in culinary awareness. However, diners still need to visit certain restaurants or venues for gourmet dishes prepared by their favorite chefs. Conversely, it is considered advantageous to watch a chef prepare one's favorite dishes in real time, or to experience the preparation of a dish while recalling childhood meals made by one's grandmother. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] U.S. Provisional Patent Application No. 62 / 116,563 [Patent Document 2] U.S. Provisional Patent Application No. 62 / 113,516 [Patent Document 3] U.S. Provisional Patent Application No. 62 / 109,051 [Patent Document 4] U.S. Provisional Patent Application No. 62 / 104,680 [Patent Document 5] U.S. Provisional Patent Application No. 62 / 090,310 [Patent Document 6] U.S. Provisional Patent Application No. 62 / 083,195 [Patent Document 7] U.S. Provisional Patent Application No. 62 / 073,846 [Patent Document 8] U.S. Provisional Patent Application No. 62 / 055,799 [Patent Document 9] U.S. Provisional Patent Application No. 62 / 044,677 [Patent Document 10] U.S. Provisional Patent Application No. 62 / 024,948 [Patent Document 11] U.S. Provisional Patent Application No. 62 / 013,691 [Patent Document 12] U.S. Provisional Patent Application No. 62 / 013,502 [Patent Document 13] U.S. Provisional Patent Application No. 62 / 013,190 [Patent Document 14] U.S. Provisional Patent Application No. 61 / 990,431 [Patent Document 15] U.S. Provisional Patent Application No. 61 / 987,406 [Patent Document 16] U.S. Provisional Patent Application No. 61 / 953,930 [Patent Document 17] U.S. Provisional Patent Application No. 61 / 942,559 [Non-patent literature]

[0007] [Non-Patent Document 1] Garagnani, M. (1999) "Improving the Efficiency of Processed Domain-axioms Planning," PLANSIG-99 Proceedings, Manchester, UK, pp. 190-192. [Non-Patent Document 2] Leake's book (1996), "Case-Based Reasoning: Experiences, Lessons, and Future Directions," http: / / journals.cambridge.org / action / displayAbstract?fromPage=online&aid=4068324&fileId=S0269888900006585dl.acm.org / citation.cfm?id=524680 [Non-Patent Document 3] Carbonell's document (1983), "Learning by Analogy: Formulating and Generalizing Plans from Past Experience," http: / / link.springer.com / chapter / 10.1007 / 978-3-662-12405-5_5 [Non-Patent Document 4] Kamakura, Noriko, Michiko Matsuo, Harumi Ishii, Fumiko Mitsuboshi, and Yoriko Miura, "Patterns of static prehension in normal hands," American Journal of Occupational Therapy 34, No. 7 (1980): pp. 437-445. [Non-Patent Document 5] A. Kapandji, "The Physiology of the Joints, Volume 1: Upper Limb, 6e", Churchill Livingstone, 6th edition, 2007 [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] Therefore, it is desirable to have a system and method for conveniently preparing and delivering gourmet dishes prepared by chefs to consumers in their own homes, without the need to travel around the world to visit various restaurants to enjoy specific gourmet dishes. [Means for solving the problem]

[0009] Embodiments of the present disclosure relate to a method, computer program product, and computer system for a robotic apparatus that uses robotic instructions to reproduce a food dish with substantially the same results as when a chef prepares the food dish. In the first embodiment, the robotic apparatus in a standardized robotic kitchen includes two robotic arms and hands that reproduce the detailed movements of a chef preparing a food dish in the same sequence (or substantially the same sequence) and at the same timing (or substantially the same timing) based on a previously recorded software file (recipe script) of the chef's detailed movements when preparing the same food dish. In the second embodiment, a computer-controlled cooking apparatus prepares the same food dish based on sensing curves such as temperature over time recorded in a software file when a chef previously prepared the food dish, using a cooking apparatus equipped with sensors on which a computer records sensor values ​​over time as a chef prepares the food dish. In a third embodiment, the kitchen apparatus includes a robotic arm in the first embodiment and a cooking apparatus having sensors in the second embodiment for preparing food, thereby combining both the robotic arm and one or more sensing curves, allowing the robotic arm to perform quality checks on characteristics such as taste, smell, and appearance during the cooking process, and enabling any cooking adjustments to the food preparation stage. In a fourth embodiment, the kitchen apparatus includes a food storage system using computer-controlled containers and container identifiers for storing and supplying ingredients for preparing food by a user following a chef's cooking instructions. In a fifth embodiment, the robotic cooking kitchen includes a robot with an arm and a kitchen apparatus, the robot moving around the kitchen apparatus and preparing food by mimicking the detailed cooking movements of a chef, including possible real-time modifications / adaptations to the preparation process defined in a recipe script.

[0010] The robotic cooking engine includes processes for detection, recording, controlling key parameters such as chef-mimicking cooking movements, temperature, and time, and executing the process using specified utensils, equipment, and tools, thereby reproducing gourmet dishes with the same taste as those prepared by a chef and serving them at a specific, suitable time. In one embodiment, the robotic cooking engine provides a robotic arm for replicating the same movements of a chef using the same ingredients and techniques to produce a dish with the same taste.

[0011] The underlying motivation for this disclosure is the ability to use monitoring sensors, capture sensors, computers, and software to monitor humans using sensors during the natural performance of their activities, and subsequently generate information and commands to replicate human activities using one or more robotic and / or automated systems. While several such activities (e.g., cooking, drawing, playing musical instruments, etc.) can be imagined, one aspect of this disclosure relates to food preparation, specifically to robotic food preparation applications. A step is performed in which a human chef is monitored within an instrumented, application-specific environment (in this case, a standardized robotic kitchen), and this step includes using sensors and computers to observe, monitor, record, and interpret the human chef's motion and movements in order to develop a robotic executable command set that is robust to variations and changes in the environment, enabling the robotic or automated system within the robotic kitchen to prepare dishes that are the same in terms of specifications and quality as those prepared by the human chef.

[0012] The use of multimode sensing systems is a means of collecting the necessary raw data. Sensors capable of collecting and supplying such data include 2D sensors (cameras, etc.) and 3D sensors (lasers, sonar, etc.), human motion capture systems (camera targets worn by humans, instrumented suits / exoskeletons, instrumented gloves, etc.), and environmental and geometric sensors such as instrumented equipment (sensors) and powered equipment (actuators) used during recipe creation and execution (instrumented utensils, cooking equipment, tools, ingredient dispensers, etc.). All of this data is collected by one or more distributed / central computers and processed by various software processes. The algorithms will process and abstract this data until the human and the computer-controlled robotic kitchen can understand the activities, tasks, actions, equipment, ingredients, methods, and processes taken by the human, including the reproduction of key skills of a particular chef. The raw data is processed by one or more software abstraction engines to create a recipe script that is human-readable, and through further processing, machine-understandable and machine-executable, specifying all actions and motions for all stages of a particular recipe that the robot kitchen needs to perform. These commands range in complexity from controlling individual joints in relation to specific stages of the recipe to specific time-series joint motion profiles, and further to abstraction command levels with low-level motion execution commands embedded within. Abstract motion commands (e.g., "crack an egg into the pan," "fry both sides until golden brown") can be generated from raw data and improved and optimized through numerous iterative learning processes conducted in real time and / or offline, enabling the robotic kitchen system to successfully cope with measurement uncertainties, changes in ingredients, etc., and allowing for complex (adaptive) small-scale operational motions using a fingered hand and wrist attached to a robotic arm, based on fairly abstract / high-level commands (e.g., "grab the deep pot with the handle," "pour out the contents," "grab a spoon from the counter and stir the soup," etc.).

[0013] At this point, the ability to create machine-executable command sequences contained within digital files can be shared / transmitted, thereby enabling any robotic kitchen to execute these sequences, opening up the option of performing cooking preparations anywhere, at any time. Thus, the option of buying / selling recipes online is provided, thereby enabling users to access and receive recipes on a per-use or subscription basis.

[0014] The reproduction of dishes prepared by humans is carried out by a robotic kitchen, which is essentially a standardized replica of the instrumented kitchen used by a human chef during cooking, except that human actions are performed by a robotic arm set and handled by computer-controlled appliances, equipment, tools, and dispensers monitored by a computer. Therefore, the fidelity of the dish reproduction is closely tied to the extent to which the robotic kitchen is a replica of the kitchen (and all its elements and ingredients) observed by the human chef during cooking.

[0015] Broadly speaking, the present invention provides a computer-implemented method that operates on a robot device, comprising: an electronic description of one or more food dishes including recipes for a chef to prepare each food dish from ingredients; a step of sensing a sequence of observational movements of a chef using ingredients and kitchen equipment when the chef prepares each food dish using multiple robot sensors; a step of detecting small operations corresponding to the sequence of movements performed at each stage of preparing a particular food dish within the observational sequence; a step of converting the sensed observational sequence into computer-readable instructions for controlling a robot device capable of performing the small operation sequence; a step of storing at least the instruction sequence for each small operation related to each food dish on an electronic medium, with each small operation sequence related to each food dish stored as an electronic record; a step of transmitting each electronic record related to a food dish to a robot device capable of reproducing the stored small operation sequence corresponding to the chef's original movements; and a step of having the robot device execute the instruction sequence for the small operation related to a particular food dish, thereby obtaining a result substantially the same as the original food dish prepared by the chef, wherein the step of executing the instructions includes a step of sensing the properties of the ingredients used to prepare the food dish.

[0016] Advantageously, robotic appliances in a standardized robotic kitchen have the ability to prepare a wide variety of meals from around the world through a global network and database access, compared to chefs who may specialize in only one type of meal. A standardized robotic kitchen can also capture and record a meal for robotic reproduction whenever you want to enjoy one of your favorite dishes, without the repetitive process of struggling to prepare the same dish over and over again.

[0017] The structure and method of the present invention are disclosed in the detailed description below. This abstract is not intended to define the present invention. The present invention is defined by the claims. The above and other embodiments, features, aspects and advantages of the present invention will be better understood by examining the following description, the appended claims and accompanying drawings.

[0018] The present invention will be described below with respect to specific embodiments thereof, with reference to the drawings that follow. [Brief explanation of the drawing]

[0019] [Figure 1] This is a system diagram illustrating an overall robotic food preparation kitchen using the hardware and software according to the present invention. [Figure 2] This is a system diagram illustrating a first embodiment of a food robotic cooking system, which includes a chef studio system and a home robotic kitchen system, according to the present invention. [Figure 3] This is a system diagram illustrating one embodiment of a standardized robotic kitchen for preparing food by replicating the recipe processes, techniques, and movements of a chef according to the present invention. [Figure 4] This is a system diagram illustrating one embodiment of a robotic food preparation engine for use with a computer in a chef studio system and a home robotic kitchen system according to the present invention. [Figure 5A] This is a block diagram illustrating the chef studio recipe creation process according to the present invention. [Figure 5B] This is a block diagram illustrating one embodiment of the standardized teaching / regenerative robot kitchen according to the present invention. [Figure 5C] This is a block diagram illustrating one embodiment of the recipe script generation abstraction engine according to the present invention. [Figure 5D] This is a block diagram illustrating software elements for manipulating objects within a standardized robotic kitchen according to the present invention. [Figure 6]This is a block diagram illustrating the multimode sensing and software engine structure according to the present invention. [Figure 7A] This is a block diagram illustrating a standardized robotic kitchen module used by a chef according to the present invention. [Figure 7B] This is a block diagram illustrating a standardized robotic kitchen module having a pair of robotic arms and hands according to the present invention. [Figure 7C] This is a block diagram illustrating one embodiment of the physical layout of a standardized robotic kitchen module used by a chef according to the present invention. [Figure 7D] This is a block diagram illustrating one embodiment of the physical layout of a standardized robotic kitchen module used by a pair of robotic arms and hands according to the present invention. [Figure 7E] This block diagram illustrates a stepwise flow and method for ensuring that control points or verification points exist during the recipe reproduction process based on a recipe script when performed by a standardized robot kitchen according to the present invention. [Figure 8A] This is a block diagram illustrating one embodiment of a conversion algorithm module for the conversion between the movements of a chef and the movements of a robot that faithfully imitate them, according to the present invention. [Figure 8B] This block diagram illustrates a pair of sensor-equipped gloves worn by Chef 49 to capture and transmit the chef's movements. [Figure 8C] This is a block diagram illustrating the execution of robotic cooking based on sensing data captured from a chef's glove according to the present invention. [Figure 8D] This graph illustrates dynamic stability and dynamic instability curves relative to equilibrium. [Figure 8E] This sequence diagram illustrates a food preparation process that requires a step sequence, referred to as a "stage," according to the present invention. [Figure 8F] This graph illustrates the overall success probability according to the present invention as a function of the number of steps for preparing food dishes. [Figure 8G]This block diagram illustrates recipe execution in multi-stage robotic food preparation using small-scale operations and motion primitives. [Figure 9A] This block diagram shows an example of a robotic hand and robotic wrist having a tactile vibration sensor, a sonar sensor, and a camera sensor for detecting and moving kitchen tools, objects, or kitchen equipment according to the present invention. [Figure 9B] This is a block diagram illustrating a pan-tilt head having a sensor camera coupled to a pair of robotic arms and hands for operation within a standardized robotic kitchen according to the present invention. [Figure 9C] This is a block diagram illustrating a sensor camera on a robot wrist for operation within a standardized robotic kitchen according to the present invention. [Figure 9D] This is a block diagram illustrating an in-hand monocular camera on a robot hand for operation within a standardized robotic kitchen according to the present invention. [Figure 9E] This is a pictorial illustration illustrating an example of a deformable palm within a robot hand according to the present invention. [Figure 9F] This is a pictorial illustration illustrating an example of a deformable palm within a robot hand according to the present invention. [Figure 9G] This is a pictorial illustration illustrating an example of a deformable palm within a robot hand according to the present invention. [Figure 9H] This is a pictorial illustration illustrating an example of a deformable palm within a robot hand according to the present invention. [Figure 9I] This is a pictorial illustration illustrating an example of a deformable palm within a robot hand according to the present invention. [Figure 10A] This block diagram shows an example of a chef recording device worn by a chef in a robotic kitchen environment to record and capture the chef's movements during the food preparation process for a specific recipe. [Figure 10B] This flowchart illustrates one embodiment of a process for evaluating the capture of a chef's motion using the posture, motion, and force of a robot according to the present invention. [Figure 11]This is a block diagram illustrating a side view of an embodiment of a robotic arm for use in a household robotic kitchen system according to the present invention. [Figure 12A] This is a block diagram illustrating one embodiment of a kitchen handle for use with a robotic hand having a palm according to the present invention. [Figure 12B] This is a block diagram illustrating one embodiment of a kitchen handle for use with a robotic hand having a palm according to the present invention. [Figure 12C] This is a block diagram illustrating one embodiment of a kitchen handle for use with a robotic hand having a palm according to the present invention. [Figure 13] This is a pictorial diagram showing an exemplary robot hand having a tactile sensor and a distributed pressure sensor according to the present invention. [Figure 14] This is a pictorial illustration showing an example of a sensing garment to be worn by a chef in a robotic cooking studio according to the present invention. [Figure 15A] This is a pictorial illustration showing one embodiment of a sensor-equipped three-finger tactile glove for food preparation by a chef according to the present invention, and an exemplary sensor-equipped three-finger robotic hand. [Figure 15B] This is a pictorial illustration showing one embodiment of a sensor-equipped three-finger tactile glove for food preparation by a chef according to the present invention, and an exemplary sensor-equipped three-finger robotic hand. [Figure 16] This block diagram illustrates the creation module and execution module for the small-scale operation library database according to the present invention. [Figure 17A] This is a block diagram illustrating a sensing glove used by a chef to perform the standardized operating movements according to the present invention. [Figure 17B] This block diagram illustrates a database of standardized operating movements within a robotic kitchen module according to the present invention. [Figure 18A] This graph illustrates how each robotic hand according to the present invention is covered with an artificial, human-like soft skin glove. [Figure 18B]This block diagram illustrates a robotic hand covered with an artificial human-like skin glove for performing high-level small operations based on a library database having predefined and internally stored small operations according to the present invention. [Figure 18C] This graph illustrates three types of operational action classifications for food preparation according to the present invention. [Figure 18D] This flowchart illustrates one embodiment of the classification of operational actions for food preparation according to the present invention. [Figure 18E] This block diagram shows an example of the mutual function and interaction between a robot arm and a robot hand according to the present invention. [Figure 18F] This is a block diagram illustrating a robotic hand using a standardized kitchen handle that can be attached to a cooking utensil head according to the present invention, and a robotic arm that can be attached to a kitchen utensil. [Figure 19] This is a block diagram illustrating the creation of a small-scale operation that results in cracking an egg with a knife according to the present invention. [Figure 20] This is a block diagram showing an example of recipe execution for small-scale operations involving real-time adjustment according to the present invention. [Figure 21] This flowchart illustrates a software process that captures the movements of a chef preparing food within a standardized kitchen module. [Figure 22] This flowchart illustrates a software process for food preparation by a robotic device within a robot-standardized kitchen module according to the present invention. [Figure 23] This flowchart illustrates one embodiment of a software process for creating, testing, verifying, and storing various parameter combinations for a small-scale operating system according to the present invention. [Figure 24] This flowchart illustrates one embodiment of a software process for creating tasks for a small-scale operating system according to the present invention. [Figure 25]This flowchart illustrates the process of allocating and utilizing a library of standardized kitchen tools, standardized objects, and standardized equipment within a standardized robotic kitchen according to the present invention. [Figure 26] This flowchart illustrates the process of identifying non-standardized objects using the 3D modeling method according to the present invention. [Figure 27] This flowchart illustrates a process for testing and learning small-scale operations according to the present invention. [Figure 28] This flowchart illustrates the processes for quality control and alignment functions of a robot arm according to the present invention. [Figure 29] This table illustrates the database library structure of small manipulative objects for use in a standardized robotic kitchen according to the present invention. [Figure 30] This table illustrates the database library structure of standardized objects for use in a standardized robotic kitchen according to the present invention. [Figure 31] This is a diagram illustrating a robotic hand for performing fish quality checks according to the present invention. [Figure 32] This is a diagram illustrating a robotic sensor head for performing quality checks inside a ball according to the present invention. [Figure 33] This is a pictorial illustration illustrating a detection device or container having a sensor for determining the freshness and quality of food according to the present invention. [Figure 34] This is a system diagram illustrating an online analysis system for determining the freshness and quality of food according to the present invention. [Figure 35] This is a block diagram illustrating a pre-filled container using programmable dispenser control according to the present invention. [Figure 36] This is a block diagram illustrating the structure of a recipe system for use in a standardized robotic kitchen according to the present invention. [Figure 37A] This is a block diagram illustrating a recipe search menu for use in a standardized robotic kitchen according to the present invention. [Figure 37B]This is a block diagram illustrating a recipe search menu for use in a standardized robotic kitchen according to the present invention. [Figure 37C] This is a block diagram illustrating a recipe search menu for use in a standardized robotic kitchen according to the present invention. [Figure 37D] This is a screenshot of a menu with options for creating and submitting a recipe according to the present invention. [Figure 37E] This flowchart illustrates one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal partner page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation according to the present invention. [Figure 37F] This flowchart illustrates one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal partner page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation according to the present invention. [Figure 37G] This flowchart illustrates one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal partner page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation according to the present invention. [Figure 37H] This flowchart illustrates one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal partner page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation according to the present invention. [Figure 37I]This flowchart illustrates one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal partner page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation according to the present invention. [Figure 37J] This flowchart illustrates one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal partner page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation according to the present invention. [Figure 37K] This flowchart illustrates one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal partner page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation according to the present invention. [Figure 37L] This flowchart illustrates one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal partner page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation according to the present invention. [Figure 37M] This flowchart illustrates one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal partner page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation according to the present invention. [Figure 37N] This is a screenshot of one of the various graphical user interfaces and menu options according to the present invention. [Figure 37O]This is a screenshot of one of the various graphical user interfaces and menu options according to the present invention. [Figure 37P] This is a screenshot of one of the various graphical user interfaces and menu options according to the present invention. [Figure 37Q] This is a screenshot of one of the various graphical user interfaces and menu options according to the present invention. [Figure 37R] This is a screenshot of one of the various graphical user interfaces and menu options according to the present invention. [Figure 37S] This is a screenshot of one of the various graphical user interfaces and menu options according to the present invention. [Figure 37T] This is a screenshot of one of the various graphical user interfaces and menu options according to the present invention. [Figure 37U] This is a screenshot of one of the various graphical user interfaces and menu options according to the present invention. [Figure 37V] This is a screenshot of one of the various graphical user interfaces and menu options according to the present invention. [Figure 38] This block diagram illustrates a recipe search menu by selecting a field for use in a standardized robotic kitchen according to the present invention. [Figure 39] This block diagram illustrates a standardized robotic kitchen using sensors extended for 3D tracking and reference data generation. [Figure 40] This is a block diagram illustrating a standardized kitchen module that uses multiple sensors to create a real-time 3D model according to the present invention. [Figure 41A] This is a block diagram illustrating one of the various embodiments and features of the standardized robot kitchen according to the present invention. [Figure 41B]This is a block diagram illustrating one of the various embodiments and features of the standardized robot kitchen according to the present invention. [Figure 41C] This is a block diagram illustrating one of the various embodiments and features of the standardized robot kitchen according to the present invention. [Figure 41D] This is a block diagram illustrating one of the various embodiments and features of the standardized robot kitchen according to the present invention. [Figure 41E] This is a block diagram illustrating one of the various embodiments and features of the standardized robot kitchen according to the present invention. [Figure 41F] This is a block diagram illustrating one of the various embodiments and features of the standardized robot kitchen according to the present invention. [Figure 41G] This is a block diagram illustrating one of the various embodiments and features of the standardized robot kitchen according to the present invention. [Figure 41H] This is a block diagram illustrating one of the various embodiments and features of the standardized robot kitchen according to the present invention. [Figure 41I] This is a block diagram illustrating one of the various embodiments and features of the standardized robot kitchen according to the present invention. [Figure 41J] This is a block diagram illustrating one of the various embodiments and features of the standardized robot kitchen according to the present invention. [Figure 41K] This is a block diagram illustrating one of the various embodiments and features of the standardized robot kitchen according to the present invention. [Figure 41L] This is a block diagram illustrating one of the various embodiments and features of the standardized robot kitchen according to the present invention. [Figure 42A] This is a block diagram illustrating a top view of a standardized robot kitchen according to the present invention. [Figure 42B] This is a block diagram illustrating the layout of a standardized robot kitchen according to the present invention. [Figure 43A]This is a block diagram illustrating a first embodiment of a kitchen module frame having an automatic transparent door within a standardized robotic kitchen according to the present invention. [Figure 43B] This is a block diagram illustrating a first embodiment of a kitchen module frame having an automatic transparent door within a standardized robotic kitchen according to the present invention. [Figure 43C] This is a block diagram illustrating a screenshot of the standardized robot kitchen according to the present invention. [Figure 43D] This is a block diagram illustrating a screenshot of the standardized robot kitchen according to the present invention. [Figure 43E] This is a block diagram illustrating a screenshot of the standardized robot kitchen according to the present invention. [Figure 43F] This is a block diagram illustrating the specifications of a sample kitchen module in the kitchen module according to the present invention. [Figure 44A] This is a block diagram illustrating a second embodiment of a kitchen module frame having an automatic transparent door within a standardized robotic kitchen according to the present invention. [Figure 44B] This is a block diagram illustrating a second embodiment of a kitchen module frame having an automatic transparent door within a standardized robotic kitchen according to the present invention. [Figure 45] This is a block diagram illustrating a standardized robotic kitchen having an extendable actuator according to the present invention. [Figure 46A] This is a block diagram illustrating a front view of a standardized robotic kitchen having a pair of fixed robotic arms without moving rails according to the present invention. [Figure 46B] This block diagram illustrates a perspective view of a standardized robotic kitchen having a pair of fixed robotic arms without moving rails, according to the present invention. [Figure 46C] This is a block diagram illustrating one of the various dimensions of a standardized robotic kitchen having a pair of fixed robotic arms without moving rails. [Figure 46D]This is a block diagram illustrating one of the various dimensions of a standardized robotic kitchen having a pair of fixed robotic arms without moving rails. [Figure 46E] This is a block diagram illustrating one of the various dimensions of a standardized robotic kitchen having a pair of fixed robotic arms without moving rails. [Figure 46F] This is a block diagram illustrating one of the various dimensions of a standardized robotic kitchen having a pair of fixed robotic arms without moving rails. [Figure 46G] This is a block diagram illustrating one of the various dimensions of a standardized robotic kitchen having a pair of fixed robotic arms without moving rails. [Figure 47] This is a block diagram illustrating a programming storage system for use with a standardized robotic kitchen according to the present invention. [Figure 48] This is a block diagram illustrating an elevation view of a programming storage system for use with a standardized robot kitchen according to the present invention. [Figure 49] This is a block diagram illustrating an elevation view of a food access container for use with a standardized robot kitchen according to the present invention. [Figure 50] This is a block diagram illustrating a food quality monitoring dashboard attached to a food access container for use with a standardized robotic kitchen according to the present invention. [Figure 51] This table illustrates a database library of recipe parameters according to the present invention. [Figure 52] This flowchart illustrates a process in one embodiment of the steps for recording a chef's food preparation process according to the present invention. [Figure 53] This flowchart illustrates a process of one embodiment of a robotic apparatus for preparing food dishes according to the present invention. [Figure 54] This flowchart illustrates a process in one embodiment for adjusting quality and function in robotic food preparation according to the present invention, in order to obtain the same or substantially the same results as a chef. [Figure 55] This flowchart illustrates a first embodiment of the process in a robotic kitchen that prepares food by reproducing the movements of a chef from a recorded software file according to the present invention. [Figure 56] This flowchart illustrates the process of receiving and identifying storage containers within a robotic kitchen according to the present invention. [Figure 57] This flowchart illustrates the process of storage retrieval and cooking preparation within a robotic kitchen according to the present invention. [Figure 58] This flowchart illustrates one embodiment of an automated pre-cooking preparation process within a robotic kitchen according to the present invention. [Figure 59] This flowchart illustrates one embodiment of the recipe design and script creation process within a robot kitchen according to the present invention. [Figure 60] This flowchart illustrates a subscription model for users to purchase robot food preparation recipes according to the present invention. [Figure 61A] This flowchart illustrates the process of searching for and purchasing / subscribing to recipes related to a recipe trading platform from a portal according to the present invention. [Figure 61B] This flowchart illustrates the process of searching for and purchasing / subscribing to recipes related to a recipe trading platform from a portal according to the present invention. [Figure 62] This flowchart illustrates the creation of a robot cooking recipe application on the application platform according to the present invention. [Figure 63] This flowchart illustrates the user's search, purchase, and subscription process for cooking recipes according to the present invention. [Figure 64A] This block diagram shows an example of predefined recipe search criteria according to the present invention. [Figure 64B] This block diagram shows an example of predefined recipe search criteria according to the present invention. [Figure 65] This is a block diagram illustrating several predefined containers within the robotic kitchen according to the present invention. [Figure 66] This is a block diagram illustrating a first embodiment of a robotic restaurant kitchen module configured in a rectangular layout using multiple pairs of robotic hands for simultaneous food preparation processing according to the present invention. [Figure 67] This block diagram illustrates a second embodiment of a robotic restaurant kitchen module configured in a U-shape using multiple pairs of robotic hands for simultaneous food preparation processing according to the present invention. [Figure 68] This is a block diagram illustrating a second embodiment of the sensing cooking tool and robotic food preparation system having a sensing curve according to the present invention. [Figure 69] This is a block diagram illustrating some physical elements of a robotic food preparation system in a second embodiment of the present invention. [Figure 70] This is a block diagram illustrating a sensing cooking utensil for a (smart) pan having a real-time temperature sensor for use in a second embodiment of the present invention. [Figure 71] This graph illustrates a temperature curve having multiple data points recorded from different sensors of a sensing cooking device in a chef's studio according to the present invention. [Figure 72] This graph illustrates temperature and humidity curves recorded from sensing cooking equipment in a chef's studio for transmission to the operational control unit according to the present invention. [Figure 73] This is a block diagram illustrating a sensing cooking device for cooking based on data from temperature curves relating to different zones on a flat pan according to the present invention. [Figure 74] This is a block diagram illustrating a sensing cooking device for a (smart) oven that uses a real-time temperature sensor and a humidity sensor for use in a second embodiment of the present invention. [Figure 75] This is a block diagram illustrating a sensing cooking device for a (smart) charcoal grill that uses a real-time temperature sensor for use in a second embodiment of the present invention. [Figure 76]This is a block diagram illustrating a sensing cooking device for a (smart) faucet that uses speed, temperature, and force control for use in a second embodiment of the present invention. [Figure 77] This is a block diagram illustrating a top view of a robotic kitchen having a sensing cooking tool according to a second embodiment of the present invention. [Figure 78] This is a block diagram illustrating a perspective view of a robotic kitchen having a sensing cooking tool according to a second embodiment of the present invention. [Figure 79] This flowchart illustrates a second embodiment of the process in a robotic kitchen according to the present invention, in which a dish is prepared from one or more past parameter curves recorded within the robotic kitchen. [Figure 80] This flowchart illustrates a second embodiment of a robotic food preparation system that incorporates a chef's cooking process using a sensing cooking tool according to the present invention. [Figure 81] This flowchart illustrates a second embodiment of a robotic food preparation system that replicates a chef's cooking process using a sensing cooking tool according to the present invention. [Figure 82] This is a block diagram illustrating a third embodiment of a robotic food preparation kitchen having a cooking operation control module, a command and visual monitoring module according to the present invention. [Figure 83] This is a block diagram illustrating a top view of a third embodiment of a robotic food preparation kitchen using the motion of a robotic arm and hand according to the present invention. [Figure 84] This block diagram illustrates a perspective view of a third embodiment of a robotic food preparation kitchen using the motion of a robotic arm and hand according to the present invention. [Figure 85] This is a block diagram illustrating a plan view of a third embodiment of a robotic food preparation kitchen using a command and visual monitoring device according to the present invention. [Figure 86]This block diagram illustrates a perspective view of a third embodiment of a robotic food preparation kitchen using a command and visual monitoring device according to the present invention. [Figure 87A] This is a block diagram illustrating a fourth embodiment of a robotic food preparation kitchen using a robot according to the present invention. [Figure 87B] This is a block diagram illustrating a top view of a fourth embodiment of a robotic food preparation kitchen using a humanoid robot according to the present invention. [Figure 87C] This is a block diagram illustrating a layout in a fourth embodiment of a robotic food preparation kitchen using a humanoid robot according to the present invention. [Figure 88] This is a block diagram illustrating the robot human emulator electronic intellectual property (IP) library according to the present invention. [Figure 89] This is a block diagram illustrating the robot-human emotion recognition engine according to the present invention. [Figure 90] This is a flowchart illustrating the process of the robot-human emotion engine according to the present invention. [Figure 91A] This flowchart illustrates a process for comparing an individual's emotional profile to a population of emotional profiles having hormones, pheromones, and other parameters according to the present invention. [Figure 91B] This flowchart illustrates a process for comparing an individual's emotional profile to a population of emotional profiles having hormones, pheromones, and other parameters according to the present invention. [Figure 91C] This flowchart illustrates a process for comparing an individual's emotional profile to a population of emotional profiles having hormones, pheromones, and other parameters according to the present invention. [Figure 92A] This block diagram illustrates the detection and analysis of an individual's emotional state by monitoring a hormone set, a pheromone set, and other key parameters according to the present invention. [Figure 92B] This is a block diagram illustrating a robot that evaluates and learns about an individual's emotional behavior according to the present invention. [Figure 93] This is a block diagram illustrating a port device implanted in an individual to detect and record an individual's emotional profile according to the present invention. [Figure 94A] This is a block diagram illustrating the robot-human intelligence engine according to the present invention. [Figure 94B] This is a flowchart illustrating the process of the robot-human intelligence engine according to the present invention. [Figure 95A] This is a block diagram illustrating the robot drawing system according to the present invention. [Figure 95B] This is a block diagram illustrating various components of the robot drawing system according to the present invention. [Figure 95C] This is a block diagram illustrating the robotic human drawing skill reproduction engine according to the present invention. [Figure 96A] This flowchart illustrates the recording process of an artist in a drawing studio according to the present invention. [Figure 96B] This flowchart illustrates the reproduction process using the robot drawing system according to the present invention. [Figure 97A] This is a block diagram illustrating one embodiment of the musician reproduction engine according to the present invention. [Figure 97B] This is a block diagram illustrating the musician reproduction engine process according to the present invention. [Figure 98] This is a block diagram illustrating one embodiment of the nursing simulation engine according to the present invention. [Figure 99A] This is a flowchart illustrating the nursing reproduction engine process according to the present invention. [Figure 99B] This is a flowchart illustrating the nursing reproduction engine process according to the present invention. [Figure 100] This is a block diagram showing an example of a computer device capable of installing and executing computer executable instructions for implementing the robotic methodology described herein. [Modes for carrying out the invention]

[0020] A description of the structural embodiments and methods of the present invention will be presented with reference to Figures 1 to 100. It should be understood that the present invention is not intended to be limited to the specifically disclosed embodiments, and that the present invention can be carried out using other features, elements, methods, and embodiments. Similar elements in various embodiments are commonly referred to by the same reference numerals.

[0021] The following definitions apply to the elements and stages described herein. These terms may be further elaborated upon.

[0022] Abstracted data refers to a practical recipe abstracted for machine execution, which has many other data elements that the machine needs to understand for proper execution and replication. This so-called metadata or additional data, corresponding to a specific stage in the cooking process, is timestamped and used by the robot kitchen to set up, control, and monitor all processes and associated methods and equipment required at every point in time as the robot kitchen progresses through the step sequence in the recipe. Whether it is direct sensor data (such as clock time, water temperature, camera images, tools used, or ingredients used) or data generated by the interpretation or abstraction of a larger dataset (such as a 3D range cloud from a laser used to extract the location and type of objects in an image, overlaid with texture and color maps from camera images), it is used by the robot kitchen to set up, control, and monitor all processes and associated methods and equipment required at every point in time as the robot kitchen progresses through the step sequence in the recipe.

[0023] An abstracted recipe refers to a human-perceived representation of a chef's recipe, expressed as a sequence of processes and methods, and the use of certain ingredients prepared and combined in a specific sequence through the skills of a human chef. Abstracted recipes used by machines for automated execution require various types of classifications and sequences. While the overall execution stages are identical to those of a human chef, an abstracted recipe for a robot kitchen requires additional metadata to be part of every stage in the recipe. Such metadata includes variables such as cooking time, temperature (and its changes over time), oven settings, and tools / equipment used. Essentially, a machine-executable recipe script requires correlating all possible measurable variables important to the cooking process (all measured and stored by a human chef during recipe preparation in the chef's studio) over time as a whole, and also within the scope of each processing stage of the cooking sequence. Thus, an abstracted recipe is a machine-readable representation or a representation of cooking stages mapped to a domain that transfers the required process from the human domain to a machine-understandable and machine-executable domain through a set of logical abstraction stages.

[0024] Acceleration refers to the maximum rate of change in velocity that a robotic arm can achieve by accelerating around an axis or along a short-distance spatial trajectory.

[0025] Accuracy refers to how close a robot can get to a commanded position. Accuracy is determined by the difference between the robot's absolute position and the commanded position. Accuracy can be improved, adjusted, or calibrated by external sensing, such as sensors on the robot hand, or by a real-time 3D model using multiple (multimode) sensors.

[0026] Action primitive – In one embodiment, this refers to an integral robotic action such as moving a robotic device from location X1 to location X2, or sensing the distance from an object toward food preparation, without necessarily obtaining a functional outcome. In another embodiment, this term refers to an integral robotic action within a sequence of one or more such units to achieve a small-scale operation. These are two aspects of the same definition.

[0027] Automated dispensing system - refers to a dispensing container within a standardized kitchen module that releases food compounds of a specific size (such as salt, sugar, pepper, spices, water, oil, extracts, ketchup, and all types of liquids) upon application.

[0028] Automated storage and delivery system - Storage containers that maintain specific temperature and humidity for storing food within a standardized kitchen module, wherein each storage container is assigned a code (e.g., a barcode) for the robotic kitchen to identify and retrieve the location where the food contents stored therein will be delivered.

[0029] A data cloud refers to a collection of numerical measurements (such as 3D laser / acoustic distance measurements, RGB values ​​from camera images, etc.) based on sensors or data from a specific space, collected at regular intervals and aggregated based on numerous relationships such as time and location.

[0030] Degrees of freedom (DOF) refer to the defined modes and / or directions in which a mechanical device or system can move. The number of degrees of freedom is equal to the total number of independent displacement or motion modes. The total number of degrees of freedom is doubled for two robotic arms.

[0031] Edge detection refers to a software-based computer program that can identify the edges of multiple objects, even if they may overlap in a 2D camera image, in order to assist in object identification and planning for grasping and handling.

[0032] Equilibrium value - This refers to the target position of a robotic limb, such as a robotic arm, where the forces acting on the limb are in equilibrium, i.e., there is no net force, and therefore no net movement.

[0033] An execution sequence planner refers to a software-based computer program that can create a sequence of execution scripts or commands for one or more computer-controllable elements or systems, such as arms, dispensers, or devices.

[0034] Food execution fidelity refers to a robotic kitchen designed to replicate recipe scripts generated in a chef's studio by observing, measuring, and understanding the stages, variables, methods, and processes of a human chef, thereby attempting to simulate the chef's techniques and skills. The fidelity of how closely the execution of food preparation resembles that of a human chef is measured by various subjective elements such as concentration, color, and taste, to determine how closely the robot-prepared dish resembles a dish prepared by a human. The concept is that the closer the dish prepared by the robotic kitchen is to that prepared by a human chef, the higher the fidelity of the replication process.

[0035] A food preparation stage (also called a "cooking stage") refers to a combination, either sequentially or in parallel, of one or more small operations, including operational primitives, and computer instructions for controlling various kitchen equipment and appliances within a standardized kitchen module. One or more food preparation stages together represent the entire food preparation process for a particular recipe.

[0036] Geometric inference refers to a software-based computer program that can infer the actual shape and size of a specific spatial region using 2D / 3D surface data and / or volumetric data. Its ability to identify and utilize boundary information also allows for inferences about the starting points and the number of specific geometric elements present (in the image or model).

[0037] Grasping reasoning refers to a software-based computer program that, in order to manipulate an object in three-dimensional space, can plan contact interactions between a robot end effector (grasper, coupler, etc.), or even a tool / instrument held by the end effector, and the object, using geometric and physical reasoning to successfully and stably contact, grasp, and hold the object. This involves multiple contacts (points / areas / spatial regions).

[0038] Hardware automation devices are fixed processing devices that can perform pre-programmed steps in sequence without the ability to modify any of these steps, and such devices are used for repetitive motion that does not require any adjustments.

[0039] Food ingredient management and handling refers to the process of defining each food ingredient in detail (including size, shape, weight, dimensions, characteristics, and properties), making one or more real-time adjustments to variables related to certain food ingredients that may differ from previously stored food ingredient details (such as fish fillet size or egg dimensions), and performing various steps regarding the movements of the food ingredients.

[0040] A kitchen module (or kitchen space area) is a standardized, complete kitchen module having a standardized set of kitchen equipment, a standardized set of kitchen tools, a standardized set of kitchen handles, and a standardized set of kitchen containers, and having predefined internal spaces and dimensions for storing, accessing, and operating each kitchen element. One purpose of the kitchen module is to predefine as many of the kitchen equipment, tools, handles, containers, etc., as possible to provide a relatively fixed kitchen platform for the movements of the robot's arm and hand. Both chefs in chef kitchen studios and individuals in homes with robot kitchens (or in restaurants) use this standardized kitchen module to maximize the predictable performance of the kitchen hardware while minimizing the risks of differentiation, variation, and bias between chef kitchen studios and home robot kitchens. Various embodiments of the kitchen module are possible, including standalone kitchen modules and integrated kitchen modules. Integrated kitchen modules are housed within the conventional kitchen area of ​​a typical house. The kitchen module operates in at least two modes: robot mode and normal (manual) mode.

[0041] Machine learning refers to techniques in which software components or software programs improve their performance based on experience and feedback. One type of machine learning is reinforcement learning, frequently used in robotics, where desirable actions are rewarded and undesirable actions are punished. Another type is case-based learning, where past solutions, such as action sequences performed by a human teacher or the robot itself, are stored along with any constraints or reasons for these solutions, and are then applied or reused in new settings. Furthermore, there are even more types of machine learning, such as inductive and transductive methods.

[0042] A small-scale operation refers to a combination (or sequence) of one or more steps that achieve a basic functional outcome at the highest probability threshold (examples of thresholds are 0.1, 0.001, or within the range of 0.001 for optimal values). Each step can be a basic coded step, a computer program, and other computer programs that can be standalone or function as subroutines, as well as a similar behavioral primitive or another (smaller) small-scale operation. For example, a small-scale operation could be the step of grasping an egg, which consists of motor movements required to extend the robot arm and move the robot finger into the correct configuration, and then to apply the correct subtle force toward grasping, all of which are primitive movements. Another small-scale operation could be the step of striking the egg with a knife, which includes a grasping small-scale operation by one robot hand, followed by a knife grasping small-scale operation by the other hand, and then a primitive movement of striking the egg with the knife using a predetermined force.

[0043] Model elements and classification refer to one or more software-based computer programs that can understand elements within a scene as items used or required in various parts of the work, such as a bowl for mixing and a spoon for stirring. Multiple elements within a scene or world model can be classified into groups, enabling rapid planning and work execution.

[0044] Motion primitives refer to motion actions that define various levels / domains of detailed motion stages. For example, a high-level motion primitive would be grasping a cup, while a low-level motion primitive would be rotating the wrist by only 5 degrees.

[0045] A multimode sensing unit refers to a sensing unit comprised of multiple sensors capable of sensing and detecting in multiple modes or electromagnetic bands or electromagnetic spectra, and in particular capable of capturing three-dimensional position and / or motion information, where the electromagnetic spectrum can range from low to high frequencies and is not limited to those perceptible to humans. Further modes may include, but are not limited to, other physical sensations such as touch and smell.

[0046] Number of axes - Three axes are needed to reach any point in space. Three additional rotational axes (yaw, pitch, and roll) are needed to completely control the orientation of the arm's end (i.e., the wrist).

[0047] A parameter refers to a variable that can take the form of a numerical value or a range of numerical values. Three types of parameters are particularly relevant: parameters in commands to a robotic device (e.g., force or distance in the movement of an arm), user-configurable parameters (e.g., preference for medium versus well-done meat), and chef-defined parameters (e.g., setting the oven temperature to 350°F).

[0048] Parameter adjustment refers to the process of changing parameter values ​​based on input. For example, changing the parameters of a command to a robotic device may be based on, but are not limited to, the properties of the food (e.g., size, shape, orientation), the position / orientation of kitchen tools, equipment, and utensils, or the speed and duration of a small operation.

[0049] Payload or carrying capacity refers to the amount of weight a robotic arm can carry and hold (or even accelerate) against gravity, as a function of the location of the robotic arm's endpoints.

[0050] Physical reasoning refers to a software-based computer program that uses geometrically inferred data as a basis to help the inference engine (program) more accurately model an object and further, to help predict the behavior of this object in the real world, especially when it is grasped and / or manipulated / handled, and can utilize physical information (density, texture, general geometric shape and form).

[0051] Raw data refers to all measured and inferred sensory data and representational information collected as part of the chef studio recipe generation process while observing / monitoring a human chef preparing a dish. Raw data can range from simple data points such as the time on a clock to oven temperature (over time), camera images, 3D laser-generated scene representation data, equipment / appliances used, tools employed, ingredients taken out (type and quantity), and even timing. All information collected by the studio kitchen from its built-in sensors and stored in raw, timestamped form is considered raw data. Raw data is later used by other software processes to convert the raw data into further processed / interpreted data with timestamps to generate a higher level of understanding and a better understanding of the recipe process.

[0052] A robotic device refers to a set of robotic sensors and robotic effectors. An effector includes one or more robotic arms and one or more robotic hands for operation within a standardized robotic kitchen. Sensors include cameras, distance sensors, and force sensors (tactile sensors) that transmit their own information to a processor or set of processors that control the effector.

[0053] Recipe cooking process refers to a robotic script containing summary and detailed instructions for a set of programmable hard automation devices that enable a computer-controllable device to perform a sequence of operations within its environment (e.g., a fully equipped kitchen with ingredients, tools, utensils, and equipment).

[0054] A recipe script refers to a time sequence containing a structure and list of commands and execution primitives (ranging from simple command software to complex command software) that, when executed in a given sequence by robotic kitchen elements (robot arms, automation equipment, appliances, tools, etc.), will result in the proper reproduction and creation of the same dishes prepared by a human chef in a studio kitchen. Such scripts are time sequences, suitable for computer-controlled elements in a robotic kitchen, and thus equivalent to the sequences employed by human chefs to create dishes, although expressed in an understandable way.

[0055] Recipe speed execution refers to managing the timeline in the execution of a recipe when preparing food dishes by replicating the movements of a chef, in which the recipe stage includes standardized food preparation operations (e.g., standardized cooking utensils, standardized equipment, kitchen processors, etc.), smaller operations, and cooking of non-standardized objects.

[0056] Repeatability refers to the allowable preset margin in how accurately and repeatedly a robot arm / hand can return to a programmed position. If the technical specifications in the control memory require the robot hand to move to a certain XYZ position within a range of ±0.1 mm of that position, then repeatability is measured in terms of returning to a range of ±0.1 mm of the taught desired / commanded position.

[0057] A robot recipe script refers to a computer-generated sequence of machine-understandable instructions related to the proper sequence of robot / hard automation execution steps that faithfully mimic the cooking steps in a recipe required to achieve the same final product as if cooked by a chef.

[0058] Robot costume - External instrumentation devices or clothing such as gloves, clothing, etc., with camera-trackable markers, articulated exoskeletons, etc., used to monitor and track the movements and activities of a chef during all aspects of the recipe cooking process in a chef's studio.

[0059] Scene modeling refers to a software-based computer program that can view a scene within the field of view of one or more cameras and detect and identify objects that are important for a particular task. These objects can be pre-taught and / or be part of a computer library with known physical attributes and uses.

[0060] Smart kitchen cooking utensils / equipment refers to items of kitchen cooking utensils (e.g., deep pots or pans) or kitchen equipment (e.g., ovens, grills, or faucets) that have one or more sensors that prepare food dishes based on one or more graph curves (e.g., temperature curves, humidity curves, etc.).

[0061] A software abstraction food engine is defined as a collection of software loops or programs that work together to process input data and create a certain desired output dataset that will be used by other software engines or by end users through any form of text or graphic output interface. An abstraction software engine is a software program focused on acquiring a large amount of input data (such as 3D distance measurements forming a data cloud of 3D measurements captured by one or more sensors) from known sources within a specific domain in order to identify, detect, and classify data indications associated with objects in 3D space (e.g., tabletops, cooking pots, etc.), and then processing this data to arrive at interpretations of the data in different domains (such as detecting and recognizing a table surface within a data cloud based on data having the same vertical data values, etc.). The abstraction process is basically defined as taking a large dataset from a single domain, inferring the structure (geometric shape, etc.) at a higher spatial level (abstracting the data points), then further abstracting the inferred values, identifying objects (deep pot, etc.) from the abstracted dataset, and identifying real-world elements in the image that can later be used by other software engines to make further decisions (such as decisions on handling / operating on the main object). In this application, a synonym for "software abstraction engine" may also be "software interpretation engine," or even "computer software processing and interpretation algorithm."

[0062] Work reasoning refers to a software-based computer program that can analyze a work description to obtain a specific final result defined within the work description, and break it down into a sequence of multiple machine-executable (robot or hard automation system) stages.

[0063] Modeling and understanding of three-dimensional objects refers to software-based computer programs that can detect, identify, and classify objects within the range of all surfaces and spatial regions, and create time-varying three-dimensional models of these surfaces and spatial regions using sensing data, so that the uses and applications of these objects can be understood.

[0064] Torque vector refers to the torsional force applied to a robotic limb, including its direction and magnitude.

[0065] Volumetric Object Estimation (Engine) - This refers to a software-based computer program that can enable the identification of the three-dimensional properties of one or more objects using geometric data, edge information, and other sensory data (color, shape, texture, etc.) to support the object identification and classification processes.

[0066] Figure 1 is a system diagram illustrating a complete robotic food preparation kitchen 10 having robotic hardware 12 and robotic software 14. The complete robotic food preparation kitchen 10 includes robotic food preparation hardware 12 and robotic food preparation software 14 that work together to perform robotic functions for food preparation. The robotic food preparation hardware 12 includes a computer 16 that controls various operations and movements of a standardized kitchen module 18 (typically operating in an instrumented environment with one or more sensors), a multimode three-dimensional sensor 20, a robotic arm 22, a robotic hand 24, and an acquisition glove 26. To obtain the same or substantially the same food preparation results (e.g., the same taste, the same smell, etc.) in which the food dish tastes the same or substantially the same as if it were prepared by a human chef, the robotic food preparation software 14 works together with the robotic food preparation hardware 12 to capture the chef's movements when preparing the food dish and reproduce the chef's movements via the robotic arm and hand.

[0067] The robot food preparation software 14 includes an acquisition module 28, a calibration module 30, a conversion algorithm module 32, a reproduction module 34, a quality check module 36 with a three-dimensional vision system, a result module 38, and a learning module 40. The acquisition module 28 captures the movements of the chef as he prepares the food dish. The calibration module 30 calibrates the robot arm 22 and robot hand 24 before, during, and after the cooking process. The conversion algorithm module 32 is configured to convert data recorded from the chef's movements collected in the chef's studio into modified (or altered) data of the recipe for use in the robot kitchen, where the robot hand reproduces the chef's food preparation. The reproduction module 34 is configured to reproduce the chef's movements in the robot kitchen. The quality check module 36 is configured to perform quality checks on the food dish prepared by the robot kitchen during, before, or after the food preparation process. The same result module 38 is configured to determine whether food dishes prepared by the robotic arm and robotic hand pair in the robotic kitchen will taste the same as, or substantially the same as, those prepared by a chef. The learning module 40 is configured to give the computer 16 that controls the robotic arm and hand the ability to learn. 。

[0068] Figure 2 is a system diagram illustrating a first embodiment of a food robotic cooking system, which includes a chef studio system and a home robotic kitchen system for preparing food by replicating the processes and movements of a chef's recipe. The robotic kitchen cooking system 42 includes a studio kitchen 44 (also called the "chef studio kitchen") that transfers one or more software-recorded recipe files 46 to the robotic kitchen 48 (also called the "home robotic kitchen"). In one embodiment, the level of detail in replicating the food preparation stage is maximized, thereby maximizing the level of detail in replicating the food prepared in the chef kitchen 44 and the robotic kitchen 4 8To reduce variables that may contribute to bias between those prepared by and those prepared by, both the chef kitchen 44 and the robot kitchen 48 use the same standardized robot kitchen module 50 (also called the “robot kitchen module,” “robot kitchen spatial area,” “kitchen module,” or “kitchen spatial area”). 49 The robot wears a robot glove or robot costume with an external sensing device to capture and record its own cooking movements. A standard robot kitchen 50 includes a computer 16 for controlling various computing functions, the computer 16 includes a memory 52 for storing one or more software recipe files from sensors in the glove or costume 54 for capturing the chef's movements, and a robot cooking engine (software) 56. The robot cooking engine 56 includes a motion analysis and recipe abstraction and sequencing module 58. Generally, a robot kitchen 48 operates using a pair of robot arms and robot hands, and an optional user 60 can operate the robot kitchen 4 8 This will be activated or programmed. The computer 16 in the robot kitchen 48 includes a hard automation module 62 for controlling the robot's arms and hands, and a recipe reproduction module 64 for reproducing the chef's movements from a software recipe file (ingredients, sequences, processes, etc.).

[0069] The standardized robotic kitchen 50 is designed to detect, record, and simulate the cooking movements of a chef, and to control the execution of processes in the robotic kitchen using specified appliances, equipment, and tools, as well as important parameters such as temperature over time. The chef's kitchen 44 is designed to control the food preparation for a specific recipe. FuA computing kitchen environment 16 is provided, which has sensor-equipped gloves or sensor-equipped clothing for recording and capturing movements. When the movements and recipe process of the chef 49 for a particular dish are recorded in a software recipe file in memory 52, this software recipe file is transmitted from the chef kitchen 44 to the robot kitchen 48 via a communication network. Ku The software is transferred via a system that allows the user (optional) 60 to purchase one or more software recipe files, receive new software recipe files, or subscribe to the chef kitchen 44 as a member to receive regular updates to existing software recipe files. The home robot kitchen system 48 serves as a robotic computing kitchen environment in residential homes, restaurants, and other locations where a kitchen is built inside for the user 60 to prepare food. The home robot kitchen system 48 includes a robotic cooking engine 56 having one or more robotic arms and hard automation devices to replicate the actions, processes, and movements of a chef's cooking based on software recipe files received from the chef studio system 44.

[0070] The Chef Studio 44 and the Robot Kitchen 48 represent a complexly linked teaching and reproduction system with multiple levels of execution fidelity. The Chef Studio 44 generates high-fidelity process models of how dishes prepared by experts are prepared, while the Robot Kitchen 48 is the execution / reproduction engine / process for recipe scripts created through the chef's work within the Chef Studio. Standardization of the Robot Kitchen module is a means to enhance execution fidelity and success / guarantee.

[0071] The various levels of fidelity in recipe execution depend on the correlation of sensors and equipment between the chef studio 44 and the robot kitchen 48 (and, of course, on the ingredients, in addition to that). Fidelity can be defined as a dish that tastes identical (indistinguishably identical) to one prepared by a human chef at one end of the spectrum (perfect reproduction / execution), while at the opposite end, the dish may have one or more substantial or fatal defects relating to quality (overcooked meat or pasta), taste (slightly burnt), edibility (inappropriate concentration), or even health (undercooked chicken / pork, etc., leading to salmonella exposure).

[0072] A robotic kitchen with the same hardware, sensors, and motion systems that can reproduce movements and processes similar to those recorded by a chef during the cooking process in a chef studio is likely to produce high-fidelity results. This implies that the configuration must be identical, and this configuration has cost and spatial implications. However, a robotic kitchen 48 can still be implemented using more standardized elements that are computer-controlled or computer-monitored (such as sensor-equipped pots and network-connected ovens), thereby requiring a more significantly sensor-based understanding to enable more complex execution monitoring. In this case, the guarantee of having the same results as a chef is undoubtedly lower, due to increased uncertainty regarding key elements (correct ingredient quantities, cooking temperature, etc.) and processes (the use of stirrers / crushers in the case of mixers is not available in a robotic home kitchen).

[0073] The focus in this disclosure is on the concept of a chef's studio 44 combined with a robotic kitchen being a common design idea. The level of the robotic kitchen 48 can vary from a home kitchen equipped with a set of arm sensors and environmental sensors to an exact replica of a studio kitchen where a set of arm and joint motions, tools and appliances, and food ingredient supply can reproduce a chef's recipe in substantially the same manner. The variables in contention are only the final results or the quality level of the dish regarding quality, appearance, taste, edibility, and health.

[0074] A possible way to mathematically describe the above correlation between the recipe outcome and input variables in a robotic kitchen can be optimally described by the following equation. F recipe-outcome =F studio (I,E,P,M,V)+F RobKit (E f ,I,R e ,P mf ) In the formula, F studio = the recipe script fidelity of the chef's studio, F RobKit = the recipe script execution by the robotic kitchen, I = food ingredients, E = equipment, [[ID=三十]]P = process, M = method, V = variables (temperature, time, pressure, etc.), E f = equipment fidelity, R e = reproduction fidelity, P mf = process monitoring fidelity.

[0075] The above formula shows that the outcome of a robot-prepared recipe is what a human chef would prepare and serve (F recipe-outcomeBased on the ingredients used (I) to the extent that they conform to the chef's process (P), the equipment available to perform the chef's process (E), and the method (M) of properly incorporating all key variables (V) during the cooking process, the recipe is properly incorporated and meets the level (F) represented by Chef Studio 44. studio ) is linked to the use of appropriate ingredients (I) and the level of equipment fidelity (E) in the robot kitchen compared to that in the chef studio. f ) and the level at which recipe scripts can be reproduced within the robot kitchen (R e ) and the highest possible process monitoring fidelity (P mf The function (F) is driven by the extent to which there is the ability and need to monitor and perform corrective actions in order to obtain ). RobKit This relates to how well the robot kitchen can represent the process of reproducing / executing the robot recipe script.

[0076] Function (F studio ) and (F RobKit ) can be any combination of a linear or nonlinear functional expression having constants, variables, and any form of algorithmic relations. Examples of such algebraic representations for both functions can be given by the following equations.

[0077] F studio =I(fct.sin(temperature))+E(fct.cooking table 1 * 5) + P(fct. yen (spoon) + V(fct. 0.5 * time)

[0078] The above equation illustrates that the fidelity of the preparation process is related to the sinusoidal change in the food temperature over time within the refrigerator, the speed at which the food can be heated to a specific multiplier on a cooking surface at a particular station, and how well the spoon can be moved in a circular path of a certain amplitude and period. It also states that in order to maintain the fidelity of the preparation process, it is necessary that the process not be carried out at more than half the speed of a human chef.

[0079] F RobKit =Ef,(countertop2,size)+I(1.25 * Size + Linear (Temperature) + R e (Motion transition) + P mf (Compatibility of sensor sets)

[0080] The above equation illustrates that the fidelity of the reproduction process in a robot kitchen is related to the type and layout of utensils and the size of heating elements for a particular cooking area, the size and temperature changes of the food being cooked by surface grilling while maintaining the motion changes of stirring motions and immersion motions at any particular stage, such as surface grilling or mousse stirring, and whether the correspondence between the sensors in the robot kitchen and the sensors in the chef studio is high enough to be trusted that the monitoring sensor data is accurate and detailed enough to give adequate monitoring fidelity of the cooking process in the robot kitchen at all stages in the recipe.

[0081] The outcome of a recipe is not only a function of how faithfully the stages / methods / processes / skills of a human chef are captured by the chef studio, but also a function of how faithfully the robot kitchen can execute these stages / methods / processes / skills, in which case each of these functions has a major factor that affects the performance of its respective subsystem.

[0082] Figure 3 shows one implementation of a standardized robotic kitchen for food preparation, which records the movements of a chef when preparing food and reproduces the food using robotic arms and hands. attitudeThis is an illustrative system diagram. In this context, the term “standardized” (or “standard”) means that the specifications of the components or features are pre-configured as will be described below. Computer 16 is communicatively coupled to several kitchen elements in a standardized robotic kitchen 50, including a three-dimensional vision sensor 66, a retractable safety screen (e.g., glass, plastic, or other type of protective material) 68, a robotic arm 70, a robotic hand 72, standardized cookware / equipment 74, a standardized cookware with sensors 76, a standardized cookware 78, a standardized handle and standardized tools 80, a standardized hard automation dispenser 82 (also called a “robot hard automation module”), a standardized kitchen processor 84, a standardized container 86, and a standardized food storage in a refrigerator 88.

[0083] The standardized hard automation dispenser 82 is a device or set of devices programmable and / or controllable by the cooking computer 16 to dispense or provide pre-packaged (known) quantities or specialized supply quantities of key ingredients, such as spices (salt, pepper, etc.), liquids (water, oil, etc.), or other dry ingredients (flour, sugar, etc.) for the cooking process. The standardized hard automation dispenser 82 can be installed at a specific station or can be accessed and triggered by a robot to perform dispensing according to a recipe sequence. In other embodiments, the robotic hard automation module can be combined with other such modules, robotic arms, or cooking tools, or ordered in series or parallel. In this embodiment, the standardized robotic kitchen 50 includes a robotic arm 70 and a robotic hand 72, which is controlled by a robotic food preparation engine 56 according to a software recipe file stored in memory 52 to replicate the detailed movements of a chef when preparing food and produce food that tastes the same as if the chef had prepared it himself. The 3D vision sensor 66 generates a visual 3D model of kitchen activity and gives the ability to scan the kitchen space area to evaluate the internal dimensions and objects of the standardized robot kitchen 50, enabling 3D modeling of objects. The retractable safety glass 68 provides transparent material on the robot kitchen 50 and, when switched on, extends the safety glass around the robot kitchen to protect people nearby from the movements of the robot arm 70 and robot hand 72, hot water and other liquids, steam, flames, and other hazardous elements. The robot food preparation engine 56 is communicatively coupled to electronic memory 52 to retrieve software recipe files previously sent from the chef studio system 44, and is configured to execute the preparation process as indicated therein, replicating the chef's cooking methods and processes.The combination of the robotic arm 70 and the robotic hand 72 plays a role in replicating the detailed movements of a chef when preparing food so that the resulting food dish tastes identical (or substantially identical) to the same food dish prepared by the chef. Standardized cooking equipment 74 includes, but is not limited to, several types of cooking appliances incorporated as part of the robotic kitchen 50, including stoves / induction cooktops / worktops (electric worktops, gas worktops, induction worktops), ovens, grills, cooking steamers, and microwave ovens. IngredientsIncludes. Standardized cooking utensils and sensors 76 are used as embodiments for recording food preparation stages based on sensors on the cooking utensils and for cooking food dishes based on sensored cooking utensils, including sensored deep pots, sensored pans, sensored ovens, and sensored charcoal grills. Standardized cooking utensils 78 include frying pans, sauté pans, grill pans, multi-pots, roasters, woks, and steamers. Robot arms 70 and robot hands 72 operate standardized handles and standardized tools 80 in the cooking process. In one embodiment, the robot hand 72 is equipped with a standardized handle that can be attached to the tip of a fork, the tip of a knife, and the tip of a spoon for selection as needed. Standardized hard automation dispensers 82 are incorporated into the robot kitchen 50 to provide preferred main ingredients and common / repeated ingredients, which are easily measured / parate / or pre-packaged. Standardized containers 86 are storage locations for storing food at room temperature. The standardized refrigerator container 88 relates to a refrigerator having identified containers for storing fish, meat, vegetables, fruits, milk, and other fresh items. Containers in the standardized container 86 or standardized storage 88 can be coded using container identifiers, from which the robotic food preparation engine 56 can determine the type of food in the container based on this container identifier. The standardized container 86 provides storage space for non-fresh food items such as salt, pepper, sugar, oil, and other spices. Sensor-equipped standardized cooking utensils 76 and cooking utensils 78 can be stored on shelves or in storage for use by the robotic arm 70 for selecting cooking tools to prepare dishes. Generally, raw fish, raw meat, and vegetables can be pre-cut and stored in the identified standardized storage 88. The kitchen counter 90 provides a platform for the robotic arm 70 to perform steps that may or may not include cutting or shredding operations of meat or vegetables as needed. The kitchen faucet 92 provides kitchen sink space for washing or cleaning food in the preparation of dishes.Once the robotic arm 70 has completed the recipe process for preparing the food and the dish is ready for serving, it is placed on the serving counter 90, which allows for further enhancement of the dining environment by adjusting the surrounding settings such as the placement of utensils, wine glasses, and wine selected to complement the food. One embodiment of the equipment within the standardized robotic kitchen module 50 is a commercial set designed to enhance its universal appeal for preparing various types of dishes.

[0084] The standardized robot kitchen module 50 has the objective of standardizing the kitchen module 50 and the various components associated with it, in order to ensure consistency in both the chef kitchen 44 and the robot kitchen 48 so as to maximize the level of detail in recipe reproduction and at the same time minimize the risk of deviation from the detailed reproduction of recipe dishes between the chef kitchen 44 and the robot kitchen 48. One main objective of standardizing the kitchen module 50 is to obtain the same cooking process result (or the same dish) between the first food dish prepared by the chef and the subsequent reproduction of the same recipe process by the robot kitchen. Considering the standardized platform in the standardized robot kitchen module 50 between the chef kitchen 44 and the robot kitchen 48 has several important requirements, such as the same timeline, the same program or mode, and quality checks. The same timeline in the standardized robot kitchen 50, where the chef prepares the food dish in the chef kitchen 44 and the reproduction process by the robot hand is performed in the robot kitchen 48, refers to the same operation sequence, the same start and end times for each operation, and the same speed at which objects are moved between handling operations. The same program or mode in the standardized robotic kitchen 50 relates to the use and operation of standardized equipment between recording and execution stages of each operation. Quality checks relate to three-dimensional visual sensors within the standardized robotic kitchen 50 that monitor and adjust each operational action in real time to correct any biases during the food preparation process and avoid defective results. The adoption of the standardized robotic kitchen module 50 reduces and minimizes the risk that the same results will not be obtained between food dishes prepared by a chef and food dishes prepared by the robotic kitchen using robotic arms and hands.Without standardization of the robot kitchen module and its internal components, a more sophisticated and complex adjustment algorithm will be required between the chef kitchen 44 and the robot kitchen 48 due to the different kitchen modules, different kitchen equipment, different kitchen utensils, different kitchen tools, and different ingredients. This increases the risk that amplified variations between the chef kitchen 44 and the robot kitchen 48 will prevent the same results from being obtained between food dishes prepared by the chef and food dishes prepared by the robot kitchen.

[0085] The standardized robotic kitchen module 50 includes standardization in many aspects. Firstly, the standardized robotic kitchen module 50 includes standardized positioning and orientation (in the XYZ coordinate plane) of all kinds of kitchen utensils, kitchen containers, kitchen tools, and kitchen equipment (along with standardized fixing holes in the kitchen module and device locations). Secondly, the standardized robotic kitchen module 50 includes standardized dimensions and structure of the cooking space area. Thirdly, the standardized robotic kitchen module 50 includes a set of standardized equipment such as ovens, stoves, dishwashers, and faucets. Fourthly, the standardized robotic kitchen module 50 includes standardized kitchen utensils, cooking tools, cooking devices, containers, and food storage in refrigerators with respect to shape, dimensions, structure, material, capacity, etc. Fifthly, in one embodiment, the standardized robotic kitchen module 50 includes a standardized universal handle for handling any kitchen utensils, tools, instruments, containers, and equipment, which enables the robotic hand to hold them in only the correct position while avoiding any improper gripping or incorrect orientation. Sixth, the standard robotic kitchen module 50 includes a standardized robotic arm and a standardized robotic hand for performing a variety of operations. Seventh, the standard robotic kitchen module 50 includes a standardized kitchen processor for standardized food handling. Eighth, the standard robotic kitchen module 50 includes a standardized 3D vision device for creating dynamic 3D visual data, as well as other possible standard sensors for recipe recording, execution tracking, and quality check functions. Ninth, the standard robotic kitchen module 50 includes standardized type, volume, size, and weight for each food item during a specific recipe execution.

[0086] Figure 4 is a system diagram illustrating one embodiment 56 of a robotic cooking engine (also called a "robot food preparation engine") for use in conjunction with the computer 16 in the chef studio system 44 and the home robot kitchen system 48. Other embodiments include the robotic cooking engine in the chef kitchen 44 and the robot kitchen 48. 56The modules within may be modified, added, or changed. The robotic cooking engine 56 includes an input module 50, a calibration module 94, a quality check module 96, a chef motion recording module 98, a cooking utensil sensor data recording module 100, a memory module 102 for storing software recipe files, a recipe abstraction module 104 for generating machine module-specific ordered operation profiles using recorded sensor data, a chef motion reproduction software module 106, a cooking utensil sensing reproduction module 108 using one or more sensing curves, a robotic cooking module 110 (computer-controlled to perform standardized operations, small operations, and non-standardized objects), a real-time adjustment module 112, a learning module 114, a small operation library database module 116, a standardized kitchen operation library database module 117, and an output module 118 to which these modules are communicatively coupled via a bus 120.

[0087] The input module 50 is configured to receive any type of input information, such as software recipe files, sent from another computing device. The calibration module 94 is configured to calibrate itself with the robotic arm 70, the robotic hand 72, and other kitchen utensils and equipment located inside the standardized robotic kitchen module 50. The quality check module 96 is configured to determine the quality and freshness of raw meat, raw vegetables, and milk-related ingredients when raw food is acquired for cooking, and in addition, to check the quality of raw food when these foods are received into the standardized food storage 88. The quality check module 96 can also be configured to perform quality inspections of objects based on senses such as the smell of the food, the color of the food, the taste of the food, and the appearance or look of the food. The chef motion recording module 98 is configured to record the chef's sequence and detailed movements as the chef prepares food dishes. The cooking utensil sensor data recording module 100 is configured to record sensing data from cooking utensils (such as a pan with sensors, a grill with sensors, or an oven with sensors) equipped with sensors located in various zones inside, thereby generating one or more sensory curves. The result is the generation of sensing curves, such as temperature (and / or humidity) curves, that reflect the temperature fluctuations of cooking utensils over time for a particular dish. Memory module 102 is configured as a storage location for either a software recipe file for reproducing the movements of a chef's recipe or other types of software recipe files containing sensing data curves. Recipe abstraction module 104 is configured to generate machine module-specific ordered operating profiles using recorded sensor data. Chef movement reproduction module 106 is configured to reproduce the detailed movements of the chef when preparing a dish based on a software recipe file stored in memory 52. ​​Cooking utensil sensing reproduction module 108 is configured to reproduce food preparation by following the characteristics of one or more recorded sensing curves generated using sensor-equipped standardized cooking utensils 76 when the chef 49 prepares a dish.The robotic cooking module 110 is configured to control and operate standardized kitchen operations, small operations, non-standardized objects, and various kitchen tools and equipment within the standardized robotic kitchen 50. The real-time adjustment module 112 is configured to apply real-time adjustments to variables related to specific kitchen operations or small operations to produce a resulting process that is a detailed reproduction of the chef's movements or a detailed reproduction of the sensing curve. The learning module 114 is configured to give the robotic cooking engine 56 the ability to learn in order to optimize a detailed reproduction of the food dish as if it were prepared by a chef, using methods such as case-based (robot) learning, when the robotic arm 70 and robotic hand 72 prepare the food dish. The small operation library database module 116 is configured to store a first database library of small operations. The standardized kitchen operation library database module 117 is configured to store a second database library of standardized kitchen tools and how to operate these standardized kitchen tools. The output module 118 is configured to send output computer files or control signals outside the robotic cooking engine.

[0088] Figure 5A shows the Chef Studio recipe creation process, illustrating several key functional blocks for creating recipe command scripts for the robot kitchen in response to the use of extended multi-mode sensing. 122This is an illustrative block diagram. Sensor data from numerous sensors, including (but not limited to) a smell sensor 124, a video camera 126, an infrared scanner and rangefinder 128, a stereo (or even tri-lens) camera 130, a tactile glove 132, an articulated laser scanner 134, virtual world goggles 136, a microphone 138, or an exoskeleton motion suit 140, human voice 142, a touch sensor 144, and yet another form of user input 146. These data, including possible human user input (e.g., a chef's screen touch and voice input), are acquired and filtered 150, and then numerous (parallel) software processes generate data that is used to input data into a machine-specific recipe-making process using temporal and spatial data. Sensors are not limited to capturing human position and / or motion and may also capture the position, orientation, and / or motion of other objects within the standardized robotic kitchen 50.

[0089] These individual software modules generate information such as (i) the chef's location and cooking station ID via location and configuration module 152, (ii) the configuration of the arm (via the torso), (iii) the tools used and when and how they are used, (iv) the tools used and their location on the station via hardware and variable abstraction module 154, (v) the processes performed using these tools, (vi) variables that need to be monitored via processing module B156 (temperature, with / without lid, stirring, etc.), (vii) the time (start / finish, type) distribution, (viii) the type of process being applied (stirring, folding, etc.), and (ix) additional ingredients (type, quantity, preparation status, etc.) via cooking sequence and cooking process abstraction module 158 (however, this does not limit the information to these modules alone).

[0090] Subsequently, all of this information is used to create a machine-specific recipe instruction set (not only for the robot arm, but also for ingredient dispensers, tools, and equipment, etc.) through the standalone module 160, which is organized as a script for sequential / parallel overlapping tasks to be executed and monitored. This recipe script (162), along with the entire raw dataset (164), is stored in the data storage module 166 and is made accessible either to a remote robotic cooking station through the robotic kitchen interface module 168 or to a human user 170 through the graphical user interface (GUI) 172.

[0091] Figure 5B shows a standardized chef studio 44 with teaching / reproduction process 176. and This is a block diagram illustrating one embodiment of the robot kitchen 50. The teaching / playback process 176 describes the stage in which the chef's recipe execution process / method / skills 49 are captured within the chef studio 44, where the chef performs a recipe execution 180 while being recorded and monitored, using a set of chef studio standard equipment 74 and ingredients 178 required by the recipe to prepare the dish. Raw sensor data is recorded in 182 (for playback) and further processed to generate information at various levels of abstraction (tools / equipment used, techniques employed, start / end times / temperature, etc.), which is then used to create a recipe script for execution by the robot kitchen 48 184.

[0092] The robotic kitchen 48 engages in a recipe reproduction process 106, which has a profile that depends on whether the kitchen being checked by process 186 is of a standardized or non-standardized type.

[0093] The execution of the robot kitchen depends on the type of kitchen available to the user. If the robot kitchen uses the same / identical (at least functionally) equipment as that used in the chef studio, the recipe reproduction process primarily uses raw data to recreate it as part of the recipe script execution process. However, if the kitchen differs from an (ideal) standard kitchen, the execution engine must rely on abstracted data to generate kitchen-specific execution sequences in an attempt to obtain similar step-by-step results.

[0094] Regardless of whether known studio equipment 196 or mixed / variant non-chef studio equipment 198 is used, the cooking process is continuously monitored by all sensor units in the robot kitchen via the monitoring process 194, so that the system can make modifications as needed, depending on the recipe progress check 200. In one embodiment of the standardized kitchen, the raw data is generally regenerated through the execution module 188 using chef studio type equipment, and a pair is established between the taught dataset and the regenerated dataset. 1 Given the existence of this correspondence, the only expected adjustments are those made to the script execution (repeat certain steps, return to a certain step, slow down execution, etc.). However, in the case of a non-standardized kitchen, the system is very likely to have to modify and adjust the actual recipe itself and its execution by the recipe script modification module 204 to accommodate deviations from the available tools / equipment 192 that differ from those in the chef studio 44 or from the measured recipe script (e.g., meat being cooked too slowly, the heat point in the pot burning the roux, etc.). The progress of the overall recipe script is monitored using similar processes 206 that differ depending on whether chef studio equipment 208 or mixed / kitchen equipment 210 is being used.

[0095] Non-standard kitchens are less likely to produce dishes that closely resemble those prepared by human chefs compared to standardized robotic kitchens that have equipment and capabilities that reflect those used in studio kitchens. The final subjective judgment is, of course, based on the taste test by a human (or chef), which is the quality evaluation 212 that produces the (subjective) quality judgment 214.

[0096] Figure 5C is a block diagram illustrating one embodiment 216 of a recipe script generation and abstraction engine relating to the structure and flow of the recipe script generation process as part of a chef studio recipe rehearsal by a human chef. The first step is to determine whether all available data that can be measured within the chef studio 44 is input by the central computer system, filtered, and then timestamped by the main process 218, whether it is ergonomic data from the chef (arm / hand position and speed, tactile finger data, etc.), status of cooking equipment (oven, refrigerator, dispenser, etc.), specific variables (countertop temperature, food temperature, etc.), equipment or tools being used (deep pot / pan, spatula, etc.), or multispectral sensing equipment (including cameras, lasers, structured light systems, etc.).

[0097] The data processing-mapping algorithm 220 uses simple (generally single-unit) variables to identify where process operations are occurring (e.g., on the countertop and / or in the oven, refrigerator) and assigns usage tags to any items / utensils / equipment being used, whether intermittently or continuously. This algorithm associates cooking stages (e.g., oven baking, grilling, ingredient additions) with specific time intervals and tracks which ingredients were added, when, where, and in what quantities. This (timestamped) information dataset is then made available to the data merging process during the recipe script generation process 222.

[0098] The data extraction and mapping process 224 primarily focuses on two-dimensional information (such as that from a single-lens camera) and extracts important information from it. Several algorithmic processes must be applied to the dataset in order to extract important and more abstract descriptive information from each sequence of images. Such processing steps may include (but are not limited to) edge detection, color, and texture mapping, and then, using the domain knowledge within the images together with object matching information (type and size) extracted from the data organization and abstraction process 226, it becomes possible to identify and locate objects (such as equipment items or food items) also extracted from the data organization and abstraction process 226, and to associate the state (and all related variables describing this state) and items within the images with specific processing stages (frying, boiling, cutting, etc.). Once this data is extracted and associated with a specific image at a specific point in time, it can be passed to the recipe script generation process 222 to formulate the internal sequence and stages of the recipe.

[0099] The data sorting and abstraction engine (software routine set) 226 is designed to reduce large 3D datasets and extract important geometric and association information from them. The first stage is to extract only the specific workspace areas that are important to the recipe at a given point in time from the large 3D data point cloud. Once the dataset is trimmed, important geometric features are identified by a known process as template matching, thereby enabling the identification of items such as the locations of horizontal tables, cylindrical deep and shallow pots, arms and hands. Once typical known (template) geometric entities are determined within the dataset, the object identification and matching process proceeds to the stage of distinguishing all items (e.g., deep pot vs. shallow pot), associating the dimensions (e.g., size of deep or shallow pot) and orientation of these items, and placing them inside the 3D world model being assembled by the computer. All of this abstracted / extracted information is then shared with the data extraction and mapping engine 224, and then all is fed to the recipe script generation engine 222.

[0100] The recipe script generation engine process 222 is responsible for merging (integrating / combining) all available data and sets into a structured sequential cooking script having clear process identifiers (preparation, blanching, frying, washing, plating, etc.) and process-specific stages within each process, which can then be interpreted into a robotic kitchen machine executable command script synchronized based on process completion, total cooking time, and cooking progress. Data merging will require, but will not be limited to, the ability to acquire each (cooking) processing stage and input the data into a step sequence in which appropriately associated elements (ingredients, equipment, etc.) should be executed, the methods and processes to be used between processing stages, and relevant critical control variables (set oven / cooktop temperature / settings) and monitoring variables (water temperature or meat temperature, etc.) to be maintained and checked to verify proper progress and execution. The merged data will then be combined into a structured sequential cooking script, which will resemble a minimal descriptive set (similar to a recipe in a magazine), but will have a considerably larger set of variables associated with each element of the cooking process (equipment, ingredients, process, method, variables, etc.) at any point in the procedure. The final stage will be to take this sequential cooking script and modify it into an equally structured sequential script that can be interpreted by the set of machines / robots / equipment inside the robot kitchen 48. This is the script that the robot kitchen 48 will use to perform the automated recipe execution and monitoring stages.

[0101] All raw (unprocessed) and processed data, as well as associated scripts (both structural sequential cooking sequence scripts and machine-executable cooking sequence scripts), are stored in the data profile storage unit / process 228 and timestamped. From this database, the user can select a desired recipe via the GUI and have it executed by the robot kitchen via the automated execution and monitoring engine 230. To reach a fully plated and served dish, this execution is continuously monitored by the engine's own internal automated cooking process, and any necessary adjustments and modifications to the script are generated by the engine and implemented by the robot kitchen elements.

[0102] Figure 5D is a block diagram illustrating software elements for object manipulation within a standardized robotic kitchen, showing the structure and flow of the object manipulation portion of a robot script for robotic kitchen execution using the concept of motion reproduction combined with a small-scale manipulation phase. For cooking based on an automated robotic arm / hand to be possible, it is insufficient to monitor every single joint in the arm and hand / fingers. Often, only the position and orientation of the hand / wrist are known (and can be further reproduced), in which case the object manipulation phase (identifying location, orientation, posture, grasping location, grasping strategy, and work execution) requires successfully completing the grasping phase / work manipulation phase using local sensing, learned behavior, and strategies for the hand and fingers. These motion profiles (sensor-based / sensor-driven) behaviors and sequences are stored within a small-scale hand manipulation library software repository in the robotic kitchen system. A human chef may wear an arm exoskeleton or an instrumented / target-equipped motion vest, allowing a computer to pinpoint the precise 3D position of the hand and wrist via built-in sensors or camera tracking. Even if all ten joints of the fingers on both hands were instrumented (resulting in more than 30 degrees of freedom for both hands, which would be extremely inconvenient to wear and use, and therefore unlikely to be used), simple playback based on the motion of all joint positions would not guarantee good results in (interactive) object manipulation.

[0103] The small operation library is a command software repository that stores motion behaviors and processes—arm / wrist / finger motions and sequences for successfully completing specific abstract tasks (such as grasping a knife and slicing, grasping a spoon and stirring, grasping a deep pot with one hand and grasping a spatula with the other, placing it under meat and turning it over in a pan)—based on an offline learning process. This repository is constructed to include learned sequences and ordered behaviors of hand / wrist sensor-driven, successful motion profiles (including, where applicable, further arm position corrections) to ensure the successful completion of operations on objects (instruments, equipment, tools) and ingredients described in more abstract language, such as "grasp a knife and slice vegetables," "crack an egg and put it in a bowl," or "turn meat over in a pan." The learning process is iterative, based on multiple trials of chef-taught motion profiles from Chef Studio, and these motion profiles are further executed and iteratively modified by an offline learning algorithm module until an acceptable execution sequence can be demonstrated. The small operational library (command software repository) is intended to be pre-loaded (offline) with all the necessary elements to enable the robotic kitchen system to successfully interact with all equipment (utensils, tools, etc.) and main ingredients that require processing (beyond simply removing them) during the cooking process. While the human chef wore gloves with embedded tactile sensors (proximity, contact, contact location / force) on the fingers and palms, the robotic hand is equipped with similar types of sensors in locations that allow these sensor data to be used to create, modify, and adjust motion profiles to successfully execute desired motion profiles and handling commands.

[0104] The object manipulation portion of the robot kitchen cooking process (a robot recipe script execution software module for interactive manipulation and handling of objects within the kitchen environment) 252 is described in further detail below. Using the robot recipe script database 254 (containing data in raw, abstracted cooking sequence machine-executable script format), the recipe script execution module 256 processes specific recipe execution stages in sequence. The configuration playback module 258 selects configuration commands and passes them to the robot arm system (torso, arm, wrist, and hand) controller 270, which then controls the physical system to simulate the required configuration values ​​(joint position / velocity / torque, etc.).

[0105] The concept of being able to faithfully perform the operation and handling of appropriate environmental interactions is made possible through (i) 3D world modeling and (ii) real-time process verification using small-scale operations. Both the verification and operation phases are carried out through the addition of the robot wrist and robot hand configuration modifier 260. This software module uses data from the 3D world configuration modeler 262 to create a new 3D world model at every sampling stage from sensing data supplied by the multimode sensor unit, in order to ensure that the system and process configuration of the robot kitchen matches what is required by the recipe script (database), and if it does not match, to make corrections to the command system configuration values ​​so that the operation can be completed successfully. Furthermore, the robot wrist and robot hand configuration modifier 260 also uses configuration modification input commands from the small-scale operation motion profile executor 264. The hand / wrist (and possibly arm) configuration modification data supplied to the configuration modifier 260 understands from 258 what the desired configuration reproduction should be, but at the same time, it is based on a small operational motion profile executor 264 that modifies this configuration reproduction based on the 3D object model library 266 and pre-trained (and stored) data from the configuration and ordering library 268 (built on multiple iterative learning stages for handling and processing stages of all major objects).

[0106] The configuration modifier 260 continuously supplies modified command configuration data to the robot arm system controller 270, while relying on the handling / operation verification software module 272 to verify not only that the operation is proceeding correctly but also whether continued operation / handling is required. If continued operation / handling is required (the answer to the decision is "N"), the configuration modifier 260 again requests updates to the configuration modifier (regarding the wrist, hand / fingers, and possibly the arm and possibly the torso) from both the world modeler 262 and the small-scale action profile executor 264. The goal is simple: to verify that a favorable operation / handling stage or sequence has been successfully completed. The handling / operation verification software module 272 performs this check using the knowledge of the recipe script database F2 and the 3D world configuration modeler 262 to verify the proper progress of the cooking stage currently commanded by the recipe script executor 256. If the progress is deemed successful, the recipe script index increment process 274 notifies the recipe script executor 256 to proceed to the next stage in the recipe script execution.

[0107] Figure 6 is a block diagram illustrating the multimode sensing and software engine structure 300 according to the present invention. One of the main features of autonomous cooking, which enables the planning, execution, and monitoring of robotic cooking scripts, requires the use of a multimode sensing input 302 used by multiple software modules to generate the data needed for the following stages, all of which occur in a continuous / iterative closed-loop manner: (i) understanding the world, (ii) modeling the scene and ingredients, (iii) planning the next stage in the robotic cooking sequence, (iv) executing the generated plan, and (v) monitoring the execution to verify proper operation.

[0108] A multimode sensor unit 302, comprising a video camera 304, an IR camera and rangefinder 306, a stereo (or even tri-lens) camera 308, and a multi-dimensional scanning laser 310 (but not limited to these), provides multispectral sensing data to the main software abstraction engine 312 (after being acquired and filtered within the data acquisition and filtering module 314). This data is used in the scene understanding module 316 to carry out several steps, including (but not limited to) constructing high-resolution and low-resolution (laser: high resolution, stereo camera: low resolution) three-dimensional surface space regions of a scene with superimposed video information of color and texture of the visual and IR spectra; enabling edge detection and volumetric object detection algorithms to infer what elements exist in the scene; and enabling the use of shape / color / texture mapping and consistency mapping algorithms based on the processed data to supply processed information to the kitchen cooking process equipment handling module 318. Within module 318, a software-based engine is used to allow the computer to construct and understand a complete scene at a specific point in time, identify and orient kitchen tools and utensils in three dimensions, and identify and tag identifiable food elements (meat, carrots, sauces, liquids, etc.) so that this scene can be used for planning and monitoring the next stage of the process. Engines required to perform such data and information abstraction include, but are not limited to, a grasping reasoning engine, a geometric shape reasoning engine, a physical inference engine, and a work reasoning engine. The output data from both engines 316 and 318 is then used to feed into the scene modeler and content classifier 320, where a 3D world model is created containing all the essential content required to run the robotic cooking script executor.Once the fully data-driven world model is understood, it can be supplied to the motion and handling planner 322, which enables the planning of motion and trajectory for the arm and attached end effectors (graspers, multi-fingered hands). (If gripping and handling of the robot arm is required, the same data can be used to plan the stages of gripping and manipulating them, distinguishing between food items and kitchen items, depending on the required gripping and placement.) The subsequent execution sequence planner 324 creates a proper sequence of commands based on the work for all individual robot kitchen elements / automated kitchen elements, which is then used by the robot kitchen actuation system 326. The entire sequence described above is repeated in a continuous closed loop between robot recipe script execution and monitoring phases.

[0109] Figure 7A depicts a standardized kitchen 50 that serves as a chef's studio where a human chef 49 creates and executes recipes, while being monitored by a multimode sensor system 66 to enable the creation of recipe scripts. Inside the standardized kitchen are several elements necessary for executing recipes, including a main cooking module 350 which includes utensils 360, a work surface 362, a kitchen sink 358, a dishwasher 356, a tabletop mixer and blender (also called a "kitchen mixer") 352, an oven 354, and a refrigerator / freezer combination unit 353.

[0110] Figure 7B depicts a standardized kitchen 50 configured as a standardized robotic kitchen that performs a recipe reproduction process defined in a recipe script. In this case, a twin-arm robotic system having a vertically extendable and freely rotating torso joint 360 and equipped with two arms 70 and two wrist-jointed fingers 72 is configured as a standardized robotic kitchen that performs a recipe reproduction process defined in a recipe script. A multimode sensor system 66 continuously monitors the cooking stages performed by the robot at multiple stages of the recipe reproduction process.

[0111] Figure 7C depicts a system involved in creating a recipe script by monitoring a human chef 49 throughout the recipe execution process. The same standardized kitchen 50 is used in chef studio mode, where the chef can operate the kitchen from either side of the work module. A multi-mode sensor 66 performs the monitoring to collect data. In addition to that, data is also monitored and collected through the gloves 370 worn by the chef and the instrumented cooking utensils 372 and equipment. All the raw data collected is wirelessly transmitted to a processing computer 16 for processing and storage.

[0112] Figure 7D depicts a system included within a standardized kitchen 50 for the reproduction of a recipe script 19 by using a dual-arm system having a telescoping rotating torso 374, two arms 7 0 two robot wrists 71, and two multi-fingered hands 72 having embedded sensing skin and tip sensors. The robot dual-arm system is used on a cooking counter 12 together with instrumented arms and hands and cooking utensils as well as instrumented appliances and cooking utensils (a frying pan in this image) during the execution of specific stages in the recipe reproduction process. During this time, the reproduction process is continuously monitored by a multi-mode sensor unit 66 to ensure that it is carried out as faithfully as possible to what was created by a human chef. To compare and track the recipe reproduction process and follow as faithfully as possible the criteria and stages defined in the recipe script 19 created and stored in a medium 18 in the past, all data from the multi-mode sensor 66, the torso 3 74, the arms 7 0 the wrists 71, and the multi-fingered hands 72, the dual-arm robot system, utensils, cooking utensils, and appliances is wirelessly transmitted to a computer 16, where it is processed by an on-board processing unit 16.

[0113] Figure 7E is a block diagram illustrating a stepwise flow and method 376 for ensuring that control points or verification points exist during the recipe reproduction process based on the recipe script when performed by the standardized robot kitchen 50, ensuring that the cooking results for a particular dish performed by the standardized robot kitchen 50 are as close as possible to those of a dish prepared by a human chef 49. When using a recipe 378 described by a recipe script and executed sequentially in the cooking process 380, the fidelity of the recipe execution by the robot kitchen 50 will largely depend on considering the following key control items. The key control items include the process 381 of selecting and using high-quality pre-processed ingredients in standardized portions and shapes; the use 383 of cooking utensils with standardized tools and implements and standardized handles to ensure proper and secure gripping in known orientations; standardized equipment 385 (ovens, mixers, refrigerators, etc.) within the standardized robot kitchen 50 that are as identical as possible when comparing a chef studio kitchen where a human chef 49 prepares a meal with the standardized robot kitchen 50; the location and arrangement 384 where ingredients should be used in the recipe; and finally, a pair of robotic arms, robotic wrists, and robotic multi-fingered hands within a kitchen module 382 that are continuously monitored by computer-controlled sensors to ensure the successful execution of each stage in all stages of the process of reproducing the recipe script for a particular dish. In short, the ultimate goal for the standardized robot kitchen 50 is to ensure identical results 386.

[0114] Figure 8A is a block diagram illustrating one embodiment of the recipe conversion algorithm module 400 between the chef's movements and the robot's reproduced movements. The recipe algorithm conversion module 404 converts data taken from the chef's movements in the chef studio 44 into a machine-readable and machine-executable language 406 for instructing the robot arm 70 and robot hand 72 to reproduce the food dishes prepared by the chef's movements in the robot kitchen 48. In the chef studio 44, the computer 16 uses a plurality of sensors S0, S1, S2, S3, S4, S5, S6…S located in a vertical column on the table 408. n And the time increments t0, t1, t2, t3, t4, t5, t6…t in the horizontal row end The system captures and records the chef's movements based on sensors on the gloves 26 worn by the chef, as shown by the system. At time t0, the computer 16 uses multiple sensors S0, S1, S2, S3, S4, S5, S6…S n The x, y, and z coordinate positions are recorded from the sensor data received. At time t1, the computer 16 records the x, y, and z coordinate positions from multiple sensors S0, S1, S2, S3, S4, S5, S6…S n The x, y, and z coordinate positions are recorded from the sensor data received. At time t2, the computer 16 records the x, y, and z coordinate positions from multiple sensors S0, S1, S2, S3, S4, S5, S6…S n The x, y, and z coordinate positions are recorded from the sensor data received. This process takes time t for the entire food preparation process. end This continues until it ends. Each time unit t0, t1, t2, t3, t4, t5, t6…t end The elapsed time for each is the same. As a result of the acquired and recorded sensor data, Table 408 shows the sensors S0, S1, S2, S3, S4, S5, S6…S in globe 26. n Table 408 shows the movement in xyz coordinates, which represents the difference between the xyz coordinate position at one specific time and the xyz coordinate position at the next specific time. endIt effectively records how the entire food preparation process changes up to [time]. The example in this embodiment can be extended to two sensored gloves 26 worn by the chef 49 to capture the movements while preparing the food dish. Within the robot kitchen 48, the robotic arm 70 and robotic hand 72 reproduce the recipe recorded from the chef studio 44 and subsequently converted into robotic commands, in which case the robotic arm 70 and robotic hand 72 reproduce the chef 49's food preparation according to the timeline 416. The robotic arm 70 and robotic hand 72 reproduce the food preparation at the same xyz coordinate position from start time t0 to end time t, as shown in the timeline 416. end The process will be carried out at the same speed with the same time increment until the specified time.

[0115] In some embodiments, the chef performs the same food preparation operation multiple times, resulting in sensor readings and parameters that vary somewhat from one time to the next in the corresponding robotic commands. The set of sensor readings for each sensor over multiple repetitions of preparing the same food dish gives a distribution with a mean, standard deviation, minimum, and maximum value. The corresponding changes on the robotic commands (also called effector parameters) over multiple executions of the same food dish by the chef also define a distribution with a mean, standard deviation, minimum, and maximum value. These distributions can be used to determine the fidelity (or precision) of subsequent robotic food preparations.

[0116] In one embodiment, the estimated average accuracy of a robotic food preparation operation is given by the following equation. TIFF0007851769000001.tif15170

[0117] In the above equation, C represents the set of chef parameters (from the 1st to the nth), and R represents the set of robot device parameters (correspondingly from the 1st to the nth). The numerator in the sum represents the difference (i.e., error) between the robot parameters and the chef parameters, and the denominator is the normalization applied to the maximum difference. The sum is the total normalized cumulative error (i.e., The mean error is obtained by giving the data (TIFF0007851769000002.tif1241) and multiplying it by 1 / n. The interpolation of the mean error corresponds to the mean precision.

[0118] Another method of calculating accuracy involves weighting these parameters in terms of importance, in this case each coefficient (α) i ) represents the importance of the i-th parameter, and the normalized cumulative error is, The filename is TIFF0007851769000003.tif1243, and the estimated average precision is given by the following formula. TIFF0007851769000004.tif15170

[0119] Figure 8B is a block diagram illustrating a pair of sensored gloves 26a and 26b worn by chef 49 to capture and transmit the chef's movements. In this particular example, intended to illustrate an example without limiting effects, the right-hand glove 26a has various sensor data points D1, D2, D3, D4, D5, D6, D7, D8, D9, D20 on the glove 26a, which may have optional electronic and mechanical circuits 420. 10 , D 11 , D 12 , D 13 , D 14 , D 15 , D 16 , D 17 , D 18 , D 19 , D 20 , D 21 , D 22 , D 23 , D 24 , and D 25 It includes 25 sensors to capture data. The left glove 26b can have various sensor data points D on the glove 26b, which can have optional electronic and mechanical circuits 422. 26 , D 27 , D 28 , D 29 , D 30 , D 31 , D 32 , D 33 , D 34 , D 35, D 36 , D 37 , D 38 , D 39 , D 40 , D 41 , D 42 , D 43 , D 44 , D 45 , D 46 , D 47 , D 48 , D 49 , D 50 It includes 25 sensors to capture data.

[0120] Figure 8C is a block diagram illustrating a robotic cooking execution stage based on sensory data captured from the chef's gloves 26a and 26b. Inside the chef's studio 44, the chef 49 wears sensored gloves 26a and 26b to capture the food preparation process, and the sensor data is recorded in the table 430. In this example, the chef 49 is cutting carrots with a knife, in this case each slice of carrot is approximately 1 centimeter thick. These motion primitives by the chef 49, recorded by gloves 26a and 26b, can constitute small operations 432 that occur over time slots 1, 2, 3, and 4. The recipe algorithm conversion module 404 is configured to convert the recipe file recorded from the chef's studio 44 into robotic commands to operate the robotic arm 70 and robotic hand 72 in the robotic kitchen 28 according to the software table 434. The robot arm 70 and robot hand 72 prepare food using control signals 436 related to a small operation in which a carrot is cut using a predefined knife in the small operation library 116, and each slice of carrot is approximately 1 centimeter thick. arm The robot hand 70 and the robot hand 72 operate with possible real-time adjustments to the size and shape of a specific carrot by creating a temporary three-dimensional model 440 of the carrot from the real-time adjustment device 112 at the same xyz coordinates 438.

[0121] Those skilled in the art will recognize that operating mechanical robotic mechanisms, such as those described in embodiments of the present invention, requires addressing numerous mechanical and control problems, and that the literature in robotics only describes methods for doing so. Establishing static and / or dynamic stability in robotic systems is a critical requirement. Dynamic stability is particularly desirable in robot operation to prevent accidental breakage or movement beyond what is desired or programmed. Figure 8D illustrates dynamic stability with respect to equilibrium. In this case, the “equilibrium value” is the desired state of the arm (i.e., the arm moves precisely to the location it is programmed to move to, with deviations caused by any number of factors such as inertia, centripetal or centrifugal force, or harmonic oscillations). A dynamically stable system exhibits small and decreasing changes over time, as shown by curve 450. A dynamically unstable system, as depicted by curve 452, is one in which changes may fail to decrease and instead increase over time. A further worst-case scenario is when the arm is statically unstable (for example, unable to hold its weight regardless of what it is gripping) and falls, or fails to recover from any deviation from its programmed position and / or path, as illustrated in curve 454. Additional information on planning (the stages of forming small-scale operation sequences, or the stages of recovery in case something goes wrong) can be found in Garagnani, M. (1999), "Improving the Efficiency of Processed Domain-axioms Planning," PLANSIG-99 Proceedings, Manchester, UK, pp. 190-192, which is incorporated herein by full citation.

[0122] The aforementioned literature refers to conditions for dynamic stability that have been incorporated by reference into this invention to enable the proper functioning of the robot arm. These conditions include the basic principle of the following equation for calculating the torque on the joints of the robot arm. TIFF0007851769000005.tif11170

[0123] In the above equation, T is the torque vector (T has n components, each corresponding to a degree of freedom of the robot arm), M is the system's inertia matrix (M is an n × n positive semi-definite matrix), C is the combination of centripetal and centrifugal forces, also an n × n matrix, G(q) is the gravity vector, and q is the position vector. Furthermore, these conditions include finding the stable point and minimum point, for example, by Lagrange's equations, when the robot position (x') can be described by a twice differentiable function (y'). JPEG0007851769000006.jpg1268 JPEG0007851769000007.jpg538

[0124] For a system consisting of a robotic arm and a hand / grasp to be stable, it is crucial that the system is properly designed and constructed, and that it has an appropriate sensing and control system that operates within the acceptable performance boundaries. This is important because, given the physical system and what its controller is asking of it, the best possible performance (highest speed, along with the best tracking of position / velocity and force / torque, all under stable conditions) must be achieved.

[0125] When proper design is discussed, the concept concerns the proper observability and controllability of a system. Observability means that the system's key variables (joint / finger position and velocity, force, and torque) are measurable by the system, which means that it must have the ability to sense these variables, and further, this means the presence and use (internal or external) of appropriate sensing devices. Controllability means that the entity (in this case, a computer) has the ability to shape or control the main axis of the system based on parameters observed from internal / external sensors, which usually means actuators or direct / indirect control of certain parameters using motors or other computer-controlled actuator systems. The ability to make a system as linear as possible in its response, thereby canceling out harmful nonlinear effects (static friction, reaction, hysteresis, etc.), enables control systems such as PID gain programming and nonlinear controllers such as sliding mode control to guarantee the stability and performance of any high-performance control system, even when taking into account uncertainties in system modeling (errors in mass / inertia estimation, geometric discretization of dimensions, anomalies in sensor / torque discretization, etc.).

[0126] Furthermore, the ability of a system to track rapid motion with a certain maximum frequency component is clearly related to the control bandwidth (closed-loop sampling rate of the computer-controlled system) that the entire system can acquire, and therefore to the frequency response that the system can exhibit (its ability to track motion with a certain speed and motion frequency component). Therefore, the use of a proper computing and sampling system is important.

[0127] All of the above characteristics are important in ensuring that the highly redundant system can actually perform the complex and intricate work that a human chef requires to successfully execute a recipe script, both in a dynamic and a stable manner.

[0128] Machine learning in the context of robot operation related to the present invention can include known methods for parameter adjustment such as reinforcement learning. Another preferred embodiment of the present invention is learning techniques for iterative and complex operations such as the steps of preparing and cooking food through multiple steps over time, that is, case-based learning. Case-based reasoning, also known as analogical reasoning, has been developed over a long period of time.

[0129] As a general overview, case-based reasoning includes the following steps. A. The step of constructing and storing cases. A case is an operation sequence with parameters that is successfully performed to achieve an objective. The parameters include forces, directions, positions, and other physical or electronic measures having values required to successfully perform the work (e.g., cooking operations). First, 1. The step of storing the aspects of the problem just solved together with the following, 2. The method for solving the problem and optionally intermediate steps and their parameter values, and 3. The step of storing the final result (typically). B. The step of applying cases (at a later time) 4. The step of obtaining one or more stored cases having a problem with a strong similarity to the new problem, 5. Optionally, the step of adjusting the parameters from the obtained cases for application to the current case (e.g., an article may be somewhat heavier and thus a somewhat stronger force is required to lift it), 6. The step of using the same method and steps from the case together with the adjusted parameters (if necessary) to at least partially solve the new problem. Therefore, case-based reasoning consists of a stage of storing solutions to past problems and a stage of applying these solutions to very similar new problems with possible parameter modifications. However, something more is needed to apply case-based reasoning to robot manipulation tasks. A change in one parameter of the solution plan will trigger changes in one or more related parameters. This necessitates not only application but also modification of the problem's solution. This new process generalizes the solution to a related, nearby solution (corresponding to small changes in input parameters such as the exact weight, shape, and location of the input ingredients), and is therefore called case-based robot learning. Case-based robot learning works as follows: C. Stage of constructing, storing, and modifying robot operation examples. 1. The stage of storing the characteristics of the problem solved immediately before, along with the following: 2. Parameter values ​​(e.g., inertia matrix from Equation 1, force, etc.), 3. To understand how much the parameter values ​​can change while still obtaining the desired results, perform perturbation analysis by changing domain-related parameters (for example, changing the weight of ingredients or their precise starting position in cooking). 4. Record which other parameter values ​​(e.g., force) will change and by how much they will change through perturbation analysis of the model, 5. Store the modified solution plan (including the dependencies between parameters and the calculation of predicted changes for these values) if the changes are within the operating specifications of the robot device. Stage D. The stage of applying the case (at a later point in time) 6. Retrieve one or more stored instances that have modified exact values ​​(new ranges or calculations for new values ​​that depend on the input parameter values), but include parameter values ​​and value ranges, and have an initial problem that is strongly similar to the new problem, and 7. To solve new problems, at least partially, we will use modified methods and steps derived from case studies. As the chef instructs the robot (which consists of two arms, as well as tactile feedback from the fingers, force feedback from the joints, and sensing devices such as one or more observation cameras), the robot learns not only specific motion sequences and temporal correlations but also small, homogeneous changes around the chef's movements, enabling it to prepare the same dish regardless of minor changes in observable input parameters. This allows the robot to learn generalized and modified plans, which are far more useful than rote memorization. For additional information on case-based reasoning and learning, see the following sources: Leake (1996 book) “Case-Based Reasoning: Experiences, Lessons and Future Directions,” http: / / journals.cambridge.org / action / displayAbstract?fromPage=online&aid=4068324&fileId=S0269888900006585dl.acm.org / citation.cfm?id=524680, which are incorporated herein by full text by citation; and Carbonell (1983) “Learning by Analogy: Formulating and Generalizing Plans from Past Experience,” http: / / link.springer.com / chapter / 10.1007 / 978-3-662-12405-5_5.

[0130] As illustrated in Figure 8E, the cooking process consists of multiple food preparation steps S1, S2, S3…S shown in timeline 456. j ...requires a sequence of steps, which may require strict linear / sequential ordering, or some may be performed in parallel, and in either case, the set of steps {S1, S2, ..., S} must all be successfully completed for overall success. i ,…,S n The success probability for each stage is P(s)i ) and if there are n stages, the overall success probability is estimated by the product of the success probabilities at each stage as follows: TIFF0007851769000008.tif11170

[0131] A person skilled in the art will recognize that even if the success probability of individual steps is relatively high, the overall success probability may be low. For example, assuming 10 steps and a success probability of 90% for each step, the overall success probability is (0.9) 10 = 0.28 or 28%

[0132] A stage in preparing food dishes may include one or more small operations, each involving one or more robotic movements that produce a very distinct intermediate result. For example, the step of slicing vegetables may be a small operation consisting of holding the vegetable in one hand and a knife in the other, and repeatedly applying knife movements until the vegetable is thinly sliced. A stage in preparing a dish may include one or more slicing small operations.

[0133] This success probability formula holds equally well at the stage level and the small-scale operation level, provided that each small-scale operation is relatively independent of other small-scale operations.

[0134] In one embodiment, to mitigate the problem of low success certainty due to potential combined errors, a standardized method is recommended for most or all of the small operations in all stages. The standardized operation can be pre-programmed, pre-tested, and pre-adjusted as needed to select the operation sequence with the highest probability of success. Therefore, if the probability of the standardized method for small operations within a stage is very high, the overall success probability of the food preparation will also be very high due to the work performed before all stages are completed and inspected. For example, returning to the above example, assuming there are 10 stages as before, and each stage utilizes a reliable standardized method with a success probability of 99% (not 90% in the previous example), then the overall success probability is (.99). 10It becomes 90.4%. This is clearly better than the 28% probability of the overall correction result.

[0135] In another embodiment, more than one option method is provided for each stage, and if one option fails, another option is tried. This embodiment requires dynamic monitoring to determine success or failure at each stage and the ability to have a selective plan. The probability of success for a given stage is the complement of the failure probability for all options and is mathematically written as follows. TIFF0007851769000009.tif12170

[0136] In the above expression, s i is the stage, and A(s i ) is the set of options to achieve s i . The failure probability for a given option is the complement of the success probability for that option, i.e., 1 - P(s i |a j ), and the probability of all failing options is the product in the above equation. Therefore, the probability that not all will fail is the complement of this product. Using the option method, the overall success probability can be estimated as the product of each stage with options, i.e., as follows. TIFF0007851769000010.tif11170

[0137] Using this option method, when each of the 10 stages has 4 options and the expected success value for each option for each stage is 90%, the overall success probability is (1 - (1 - (.9)) 4 ) 10 =.99 or 99% compared to only 28% without options. The option method changes the original problem from a series of stages with multiple single points of failure (when any stage fails) to one without single points of failure because all options need to fail for any given stage to fail, providing a more robust result.

[0138] In another embodiment, a standardization stage including standardized minor operations is combined with selective means for a food preparation stage to produce even more robust behavior. In such cases, the corresponding success probability can be very high, even if the choices exist for only some of the stages or minor operations.

[0139] In another embodiment, options in case of failure are provided only for stages with a lower probability of success, such as stages for which a highly reliable standardization method does not exist, or stages with potential variability due to reliance on materials of a special shape, for example. This embodiment reduces the burden of providing options for all stages.

[0140] Figure 8F shows the overall success rate (y-axis) as a function of the number of steps (x-axis) required to cook food in an unstandardized kitchen. of This graph shows the first example curve 458 and the second example curve 459 illustrating a standardized kitchen 50. In this example, it was assumed that the individual success probability for each food preparation stage was 90% in unstandardized operation and 99% in standardized, pre-programmed stages. As shown in curve 458 in comparison to curve 459, the combined error is considerably worse in the case of unstandardized operation.

[0141] Figure 8G is a block diagram illustrating the execution of recipe 460 in multi-stage robotic food preparation using small-scale operation and motion primitives. Each food recipe 460 consists of a first food preparation stage S1470, a second food preparation stage S2, and so on, which are performed by the robot arm 70 and the robot hand 72. nIt can be divided into multiple food preparation stages of 490. The first food preparation stage S1470 includes one or more sub-operations MM1471, MM2472, and MM3473. Each sub-operation includes one or more operation primitives that obtain a functional result. For example, the first sub-operation MM1471 includes a first operation primitive AP1 474, a second operation primitive AP2 475, and a third operation primitive AP3 475, and subsequently obtains a functional result 477. One or more sub-operations MM1471, MM2472, and MM3473 in the first stage S1470 subsequently achieve a stage result 479. One or more food preparation stages S1470, a second food preparation stage S2, and the nth stage food preparation stage S n The combination with 490 produces substantially the same or identical results as those recorded in Chef Studio 44 by replicating Chef 49's food preparation process.

[0142] A set of predefined small operations is available to obtain each functional outcome (e.g., an egg being cracked). Each small operation consists of a set of action primitives that work together to achieve the functional outcome. For example, a robot could start by moving its hand toward an egg, touching the egg to locate its position and verify its size, and then perform the necessary motion and sensing actions to grasp, lift, and place the egg into a known, predetermined configuration.

[0143] Multiple small operations can be grouped together into steps such as making sauces, for convenience when understanding and organizing recipes. The ultimate result of performing all small operations to complete the entire step is that the food dish can be reproduced with consistent results every time.

[0144] Figure 9A is a block diagram showing an example of a robotic hand 72 having five fingers and a wrist with RGB-D sensor capabilities, camera sensor capabilities, and sonar sensor capabilities for detecting and moving kitchen tools, objects, or kitchen equipment items. The palm of the robotic hand 72 includes an RGB-D sensor 500, a camera sensor, or a sonar sensor 504f. Alternatively, the palm of the robotic hand 450 includes both a camera sensor and a sonar sensor. The RGB-D sensor 500 or sonar sensor 504f can detect the location, dimensions, and shape of an object to create a three-dimensional model of the object. For example, the RGB-D sensor 500 uses structured light to capture the shape of an object for three-dimensional mapping and positioning, path planning, navigation, object recognition, and human tracking. The sonar sensor 504f uses acoustic waves to capture the shape of an object. In conjunction with the camera sensor 452 and / or sonar sensor 454, a video camera 66 positioned anywhere in the robot kitchen, such as on a handrail, or on the robot itself, provides a method for capturing, tracking, or guiding the movement of kitchen tools used by the chef 49, as illustrated in Figure 7A. The video camera 66 is positioned at a certain distance and angle from the robot hand 72, thus providing a high-level view of the robot hand 72 grasping objects and whether it has grasped or released an object. A suitable example of an RGB-D (red, green, blue, and depth) sensor is Microsoft's Kinect system, which features an RGB camera, depth sensor, and multi-array microphone operating on software that provides full-body 3D motion capture, facial recognition, and speech recognition capabilities.

[0145] The robot hand 72 has an RGB-D sensor 500 positioned in or near the center of its palm to detect the distance and shape of an object, and to handle kitchen tools. The RGB-D sensor 500 provides guidance to the robot hand 72 to move toward the object and to make the necessary adjustments to grasp the object. A second sonar sensor 502f and / or a touch pressure sensor are positioned near the palm of the robot hand 72 to detect the distance and shape of an object and subsequent contact. The sonar sensor 502f can also guide the robot hand 72 toward the object. Further types of sensors in the hand may include ultrasonic sensors, lasers, radio frequency identification (RFID) sensors, and other appropriate sensors. In addition, the touch pressure sensor acts as a feedback mechanism to determine whether the robot hand 72 should continue to apply further pressure to grasp the object at a point where sufficient pressure exists to safely lift the object. In addition, the sonar sensor 502f in the palm of the robot hand 72 provides tactile sensing capabilities for grasping and handling kitchen tools. For example, when the robot hand 72 grasps a knife to cut beef, when the knife has finished slicing the beef, i.e., when the knife no longer offers resistance, or when it is holding an object, the amount of pressure the robot hand exerts on the knife and applies to the beef can be detected by the tactile sensor. The pressure distributed is not only for holding the object in place, but also to prevent the object (e.g., an egg) from breaking.

[0146] Furthermore, each finger on the robot hand 72 has tactile vibration sensors 502a to 502e and sonar sensors 504a to 504e on its respective fingertip, as shown by the first tactile vibration sensor 502a and first sonar sensor 504a on the fingertip of the thumb, the second tactile vibration sensor 502b and second sonar sensor 504b on the fingertip of the index finger, the third tactile vibration sensor 502c and third sonar sensor 504c on the fingertip of the middle finger, the fourth tactile vibration sensor 502d and fourth sonar sensor 504d on the fingertip of the ring finger, and the fifth tactile vibration sensor 502e and fifth sonar sensor 504e on the fingertip of the little finger. Each of the tactile vibration sensors 502a, 502b, 502c, 502d, and 502e can simulate various surfaces and effects by changing the shape, frequency, amplitude, duration, and direction of the vibration. Each of the sonar sensors 504a, 504b, 504c, 504d, and 504e provides the ability to sense the distance and shape of an object, the ability to sense temperature or humidity, and also provides feedback. Additional sonar sensors 504g and 504h are positioned on the wrist of the robot hand 72.

[0147] Figure 9B is a block diagram illustrating one embodiment of a pan-tilt head 510 having a sensor camera 512 coupled to a pair of robotic arms and hands for operation within a standardized robotic kitchen. The pan-tilt head 510 has an RGB-D sensor 512 for monitoring, capturing, or processing information and 3D images from inside the standardized robotic kitchen 50. The pan-tilt head 510 provides good situational awareness that is independent of the motion of the arms and sensors. The pan-tilt head 510 is coupled to a pair of robotic arms 70 and robotic hands 72 for performing food preparation processes, although the pair of robotic arms 70 and robotic hands 72 may cause obstruction.

[0148] Figure 9C is a block diagram illustrating a sensor camera 514 on a robot wrist 73 for operation within a standardized robot kitchen 50. One embodiment of the sensor camera 514 is an RGB-D sensor that provides a color image and depth perception, mounted on the wrist 73 of each hand 72. Each of the camera sensors 514 on each wrist 73 is subject to limited occlusion by the arm, but is generally not occluded when the robot hand 72 grasps an object. However, the RGB-D sensors 514 may be occluded by each robot hand 72.

[0149] Figure 9D is a block diagram illustrating an in-hand monocular camera 518 on a robot hand 72 for operation within a standardized robotic kitchen 50. Each hand 72 has sensors such as an RGB-D sensor to provide in-hand monocular camera functionality within the standardized robotic kitchen 50. The in-hand monocular camera 518 with an RGB-D sensor in each hand produces high image detail with only limited occlusion by the respective robotic arms 70 and the respective robotic hands 72. However, the robotic hand 72 with the in-hand monocular camera 518 may encounter occlusion when grasping an object.

[0150] Figures 9E to 9G are illustrative diagrams illustrating the configuration of the deformable palm 520 within the robot hand 72. The fingers of the five-fingered hand are denoted as follows: the thumb as the first finger F1 522, the index finger as the second finger F2 524, the middle finger as the third finger F3 526, the ring finger as the fourth finger F4 528, and the little finger as the fifth finger F5 530. The thenar eminence 532 is a convex spatial region of the deformable material on the radial side (first finger F1 522) of the hand. The hypothenar eminence 534 is a convex spatial region of the deformable material on the ulnar side (fifth finger F5 530) of the hand. The metacarpophalangeal pad (MCP pad) 536 is a convex, deformable space region on the palmar side of the metacarpophalangeal joints (finger joints) of the second, third, fourth, and fifth fingers F2 524, F3 526, F4 528, and F5 530. The robotic hand 72 having a deformable palm 520 wears a glove on the outside, which has soft, human-like skin.

[0151] The thenar eminence 532 and hypothenar eminence 534 work together to assist in the application of large forces from the robot arm to an object in the working space, so that these forces apply only minimal external pressure to the robot hand joints (e.g., in the case of a rolling pin). Additional joints within the palm 520 are available for deforming the palm. The palm 520 must be deformed to allow the formation of oblique palmar grooves toward tool gripping in a chef-like manner (typical handle gripping). The palm 520 must be deformed to allow bowl-shaping for conformal gripping of convex objects such as plates and food materials in a chef-like manner, as shown by the bowl-shaped state 542 in Figure 9G.

[0152] The internal joints of the palm 520 that can assist these motions include the thumb-carpometacarpal (CMC) joint located on the radial side of the palm near the wrist, which can have two distinct directions of motion (flexion / extension and abduction / adduction). Further joints required to assist these motions may include the ulnar side joints of the palm near the wrist (CMC joints of the fourth finger F4 528 and the fifth finger F5 530), which allow for oblique flexion to assist in cup-forming motion and palmar groove formation at the hypothenar eminence 534.

[0153] The robotic palm 520 may include additional / different joints necessary to replicate the palm shape observed in human cooking motions, for example, a series of connected flexion joints that assist in forming an arch 540 between the thenar eminence 532 and the hypothenar eminence 534 to deform the palm 520, such as when the thumb F1 522 makes contact with the little finger F5 530, as illustrated in Figure 9F.

[0154] When the palm is in a bowl shape, the thenar eminence 532, the hypothenar eminence 534, and the MCP pad 536 form ridges around the valleys of the palm, allowing the palm to close around a small spherical object (e.g., 2 cm).

[0155] The shape of the deformable palm will be described using the location of feature points relative to a fixed reference frame, as shown in Figures 9H and 9I. Each feature point is represented as a vector of x, y, and z coordinate positions over time. The locations of the feature points are marked on the sensing glove worn by the chef and on the sensing glove worn by the robot. A reference frame is further marked on the glove, as illustrated in Figures 9H and 9I. The feature points are defined on the glove relative to the position of the reference frame.

[0156] Feature points are measured by a calibrated camera mounted in the workspace while the chef performs cooking tasks. The trajectory of the feature points over time is used to match the chef's motion with the robot's motion, including matching the shape of the deformable palm. The trajectory of the feature points from the chef's motion can also be used to inform the robot of the deformable palm design, including the shape and arrangement of the deformable palm surface, as well as the motion range of the robot hand's joints.

[0157] In the embodiment shown in Figure 9H, the feature points are located within the hypothenar eminence 534 and the thenar eminence 532, and the MCP pad 536 has a checkerboard pattern with marks indicating the feature points in each area of ​​the palm. The reference frame in the wrist area has four rectangles that are identifiable as a reference frame. The feature points (or markers) are identified at their respective locations relative to the reference frame. The feature points and reference frame in this embodiment can be implemented under a glove for food safety reasons, but can be seen through the glove for detection.

[0158] Figure 9H shows a robotic hand with a visual pattern that can be used to locate 550 three-dimensional shape feature points. The locations of these shape feature points provide information about the shape of the palm surface as the palmar joints move and as the palm surface deforms in response to applied forces.

[0159] The visual pattern consists of surface marks 552 on a robotic hand or on a glove worn by a chef. These surface marks can be covered by a food safety transparent glove 554, but the surface marks 552 remain visible through the glove.

[0160] If the surface mark 552 is visible in the camera image, a two-dimensional feature point can be identified within this camera image by locating the convex or concave corner within the visual pattern. Each such corner in a single camera image is a two-dimensional feature point.

[0161] If the same feature point is identified in multiple camera images, the three-dimensional location of this point can be determined within a fixed coordinate frame relative to the standardized robot kitchen 50. This calculation is performed based on the two-dimensional location of the point in each image and known camera parameters (position, orientation, field of view, etc.).

[0162] A reference frame 556 fixed to the robot hand 72 can be acquired using the reference frame visual pattern. In one embodiment, the reference frame 556 fixed to the robot hand 72 consists of an origin and three orthogonal coordinate axes. The reference frame is identified by localizing the features of the reference frame's visual pattern in multiple cameras and extracting the origin and coordinate axes using known parameters of the reference frame's visual pattern and known parameters of the cameras.

[0163] Three-dimensional shape feature points represented within the coordinate frame of the food preparation station are transformed into the reference frame of the robot hand when the reference frame of the robot hand is observed.

[0164] The deformable palm shape is entirely composed of vectors of three-dimensional shape feature points represented within a reference coordinate frame fixed to the robot or chef's hand.

[0165] As illustrated in Figure 9I, the feature points 560 in these embodiments are represented by sensors, such as Hall effect sensors, in different areas of the palm (hypothenar eminence 534, thenar eminence 532, and MCP pad 536). The feature points are identifiable at their respective locations relative to a reference frame, which is a magnet in this implementation. The magnet generates a magnetic field that is readable by the sensors. The sensors in this embodiment are embedded under the glove.

[0166] Figure 9I shows a robot hand 72 having embedded sensors and one or more magnets 562 that can be used as a selective mechanism for identifying the location of three-dimensional shape feature points. Each shape feature point is associated with each embedded sensor. The locations of these shape feature points 560 provide information about the shape of the palm surface when the palmar joints move and when the palm surface deforms in response to applied forces.

[0167] The location of the shape feature points is determined based on the sensor signal. The sensor provides an output that enables the calculation of distances within a reference frame magnetically mounted on the robot or chef's hand.

[0168] The three-dimensional location of each shape feature point is calculated based on sensor measurements and known parameters obtained from sensor calibration. The deformable palm shape consists of a vector of three-dimensional shape feature points, all represented within a reference coordinate frame fixed to the robot or chef's hand. For additional information on common contact areas and functions in grasping on a human hand, see Kamakura, Noriko, Michiko Matsuo, Harumi Ishii, Fumiko Mitsuboshi, and Yoriko Miura, “Patterns of static prehension in normal hands,” American Journal of Occupational Therapy 34, No. 7 (1980): pp. 437–445, which is incorporated herein by full citation.

[0169] Figure 10A is a block diagram showing an example of a recording device 550 worn by a chef 49 in a standardized robotic kitchen environment 50 to record and capture the chef's movements during the food preparation process for a specific recipe. The chef recording device 550 includes, but is not limited to, one or more robotic gloves (or robotic clothing) 26, a multimode sensor unit 20, and a pair of robotic glasses 552. Within the chef studio system 44, the chef 49 wears the robotic gloves 26 to cook and record and capture the chef's cooking movements. Alternatively, the chef 49 may wear a robotic clothing that includes robotic gloves, rather than just the robotic gloves 26. In one embodiment, the robotic glove 26 with embedded sensors captures, records, and stores, with timestamped, the position, pressure, and other parameters of the chef's arm, hand, and finger motion in an xyz coordinate system. The robotic glove 26 stores the position and pressure of the chef's arm and finger in a three-dimensional coordinate frame over a duration from start time to end time when preparing a specific food dish. When Chef 49 wears the robotic glove 26, all movements, hand position, gripping motion, and pressure exerted while preparing food in the chef studio system 44 are accurately recorded at regular time intervals, such as every t seconds. The multimode sensor unit 20 includes a video camera, an IR camera and rangefinder 306, a stereo (or even trinocular) camera 308, and a multidimensional scanning laser 310, and supplies multispectral sensing data to the main software abstraction engine 312 (after being acquired and filtered in the data acquisition and filtering module 314). The multimode sensor unit 20 generates three-dimensional surfaces or textures and processes the abstraction model data.This data is used to carry out several steps within the scene understanding module 316, including (but not limited to) the steps of: constructing high-resolution and low-resolution (laser: high resolution, stereo camera: low resolution) three-dimensional surface space regions of the scene with superimposed video information of color and texture of the visual and IR spectra; enabling edge detection and volumetric object detection algorithms to infer what elements exist in the scene; and enabling the use of shape / color / texture mapping and consistency mapping algorithms based on the processed data to supply the processed information to the kitchen cooking process equipment handling module 318. Optionally, in addition to the robotic gloves 76, the chef 49 may wear one pair of robotic glasses 552 having one or more robotic sensors 554 around a frame having robotic earphones 556 and a microphone 558. The robotic glasses 552 provide further visual and capture capabilities, such as a camera, to capture video and record the image the chef 49 sees while cooking food. One or more robotic sensors 554 capture and record the temperature and smell of the food being prepared. The earphone 556 and microphone 558 capture and record sounds heard by the chef 49 during cooking, including human voices and sound characteristics such as frying, grilling, and grinding. The chef 49 can also use the earphone and microphone 82 to record simultaneous voice commands and real-time cooking stages of food preparation. In this regard, the chef robot recorder device 550 records the chef's movement parameters, speed parameters, temperature parameters, and sound parameters during the food preparation process for a specific food dish.

[0170] Figure 10B is a flowchart illustrating one embodiment 560 of the process for evaluating captured chef motion using robot posture, motion, and force. Database 561 stores predefined (or predetermined) grasping postures 562 and predefined hand motions by the robot arm 72 and robot hand 72, which are weighted by importance 564, labeled using contact points 565, and store contact forces 565. In operation 567, the chef motion recording module 98 is configured to capture the chef's motion while preparing food, based in part on the predefined grasping postures 562 and predefined hand motions 563. In operation 568, the robot food preparation engine 56 is configured to acquire posture, motion, and force and evaluate the robot apparatus configuration in terms of its ability to achieve small operations. Subsequently, the robot apparatus configuration undergoes an iterative process 569 in which it evaluates robot design parameters 570, adjusts the design parameters to improve scores and performance 571, and modifies the robot apparatus configuration 572.

[0171] Figure 11 is a block diagram illustrating one embodiment of a side view of a robotic arm 70 for use with a standardized robotic kitchen system 50 in a home robotic kitchen 48. In other embodiments, one or more of the robotic arms 70, such as one arm, two arms, three arms, four arms, or more arms, can be designed for operation within a standardized robotic kitchen. One or more software recipe files 46 from a chef studio system 44 that store the movements of a chef's arms, hands, and fingers during food preparation can be uploaded and converted into robotic commands to control one or more robotic arms 70 and one or more robotic hands 72 to simulate the movements of a chef preparing food dishes. The robotic commands control the robotic apparatus to reproduce the detailed movements of a chef preparing the same food dishes. Each of the robotic arms 70 and each of the robotic hands 72 may also include further features and tools such as knives, forks, spoons, spatulas, and other types of utensils or food preparation instruments to accomplish the food preparation process.

[0172] Figures 12A to 12C are block diagrams illustrating one embodiment of a kitchen handle 580 for use by a robotic hand 72 having a palm 520. The design of the kitchen handle 580 is intended to be universal (or standardized) so that the same kitchen handle 580 can be attached to all kinds of kitchen utensils or tools, such as knives, spatulas, slotted spoons, colanders, backing spoons, etc. Figures 12A to 12B show different perspective views of the kitchen handle 580. The robotic hand 72 grasps the kitchen handle 580 as shown in Figure 12C. Other types of standardized (or universal) kitchen handles can be designed without departing from the ingenuity of the present invention.

[0173] Figure 13 is a pictorial illustration showing an exemplary robotic hand 600 having a tactile sensor 602 and a distributed pressure sensor 604. During the food preparation process, the robotic device uses contact signals generated by sensors in the fingertips and palms of the robotic hand to detect force, temperature, humidity, and toxicity, while the robot replicates stepwise movements and compares the sensed values ​​to a tactile profile of the chef's studio cooking program. Visual sensors help the robot identify its surroundings and take appropriate cooking actions. To ensure that appropriate movements are taken to obtain the same results, the robotic device analyzes images of the immediate environment from the visual sensors and compares them to saved images from the chef's studio cooking program. The robotic device further uses different microphones to improve recognition performance during cooking by comparing the chef's command utterances against background noise from the food preparation process. Optionally, the robot may have an electronic nose (not shown) to detect odor or flavor and ambient temperature. For example, the robotic hand 600 can distinguish real eggs by surface texture signals, temperature signals, and weight signals generated by tactile sensors in its fingers and palm, thereby applying the appropriate force to hold the egg without breaking it. In addition, it can perform quality checks by shaking the egg and listening to the sound of liquid sloshing and cracking, observe the yolk and egg white, and smell it to determine its freshness. The robotic hand 600 can then discard spoiled eggs or select fresh ones. Sensors 602 and 604 on the hand, arm, and head enable the robot to move, touch, see and hear to perform food preparation processes using external feedback, and to obtain results identical to those of a chef's studio cooking in food preparation.

[0174] Figure 14 is a pictorial illustration showing an example of a sensing suit to be worn by a chef 49 in a standardized robotic kitchen 50. During food preparation of food dishes, which are recorded by a software file 46, the chef 49 wears a sensing suit 620 to capture the chef's food preparation movements in real time in a time sequence. The sensing suit 620 is a tactile suit 622 (shown as a single full-length arm and hand suit). and The system may include, but is not limited to, a tactile glove 624, a multimode sensor 626 [no such number], and a head garment 628. The sensored tactile suit 622 can capture data from the chef's movements to record the xyz coordinate position and pressure of the human arm 70 and hand / finger 72 with a timestamp in the XYZ coordinate system, and transmit the captured data to the computer 16. The sensored garment 620 also records the position, velocity, and force / torque of the human arm 70 and hand / finger 72, as well as the endpoint contact behavior, with a system timestamp in the robot coordinate frame and associates it with the relative position within the standardized robot kitchen 50 which has geometric sensors (laser sensors, 3D stereo sensors, or video sensors). The sensored tactile glove 624 is used to capture, record, and store force signals, temperature signals, humidity signals, and toxicity signals detected by the tactile sensors within the glove 624. The headgear 628 includes a feedback device having a visual camera, sonar, laser, or radio frequency identification (RFID), and custom glasses used to sense, capture, and transmit captured data to a computer 16 in order to record and store the image seen by the chef 48 during the food preparation process. In addition, the headgear 628 also includes sensors for detecting ambient temperature and odor signatures within the standardized robotic kitchen 50. Furthermore, the headgear 628 also includes audio sensors for capturing sounds heard by the chef 49, such as sound characteristics of frying, grinding, shredding, etc.

[0175] Figures 15A and 15B are pictorial illustrations showing one embodiment of a sensored three-finger tactile glove 630 for food preparation by a chef 49, and an example of a sensored three-finger robotic hand 640. The embodiment illustrated herein shows a simple robotic hand 640 having fewer than five fingers for food preparation. Accordingly, the complexity of the design of the simple robotic hand 640 is greatly reduced, along with the cost of manufacturing the simple robotic hand 640. Two-fingered grippers or four-fingered robotic hands with or without opposing thumbs are also possible optional implementations. In this embodiment, the chef's hand movements are limited by the functions of three fingers, the thumb, index finger, and middle finger, each having a sensor 632 for sensing data on the chef's movements relating to force, temperature, humidity, toxicity, or tactile sensation. The three-finger tactile glove 630 further includes point sensors or distributed pressure sensors within its palmar area. The movements of a chef wearing a three-finger tactile glove 630 and preparing food using his thumb, index finger, and middle finger are recorded in a software file. Subsequently, a three-finger robotic hand 640 reproduces the chef's movements from the software recipe file, which has been translated into robotic commands to control the thumb, index finger, and middle finger of the robotic hand 640, while monitoring sensors 642b on the fingers of the robotic hand 640 and sensors 644 on the palm. Sensor 644 can be a point sensor or a distributed pressure sensor, while sensor 642 includes a force sensor, temperature sensor, humidity sensor, toxicity sensor, or tactile sensor.

[0176] Figure 16 is a block diagram illustrating the creation module 650 for the small operation library database and the execution module for the small operation library database. The creation module 650 for the small operation database library is a process that creates and examines various possible combinations and selects the best small operation to obtain a specific functional result. One purpose of the creation module 60 is to investigate all possible different combinations for performing a specific small operation and predefine a library of optimal small operations for subsequent execution by the robotic arm 70 and robotic hand 72 when preparing food dishes. The creation module 650 for the small operation library can also be used as a teaching method for the robotic arm 70 and robotic hand 72 to learn about different food preparation functions from the small operation library database. The execution module 660 of the small operation library database is configured to provide a range of small operation functions that the robot device can access and execute from the small operation library database during the process of preparing food dishes. These functions include a first small operation MM1 with a first functional outcome 662, a second small operation MM2 with a second functional outcome 664, a third small operation MM3 with a third functional outcome 666, a fourth small operation MM4 with a fourth functional outcome 668, and a fifth small operation MM5 with a fifth functional outcome 670.

[0177] Figure 17A is a block diagram illustrating a sensing glove 680 used by a chef 49 to sense and capture the chef's movements while preparing food. The sensing glove 680 has multiple sensors 682a, 682b, 682c, 682d, 682e on each of the fingers of the sensing glove 680 and multiple sensors 682f, 682g in the palm area. In another embodiment, at least five pressure sensors 682a, 682b, 682c, 682d, 682e located inside a soft glove are used to capture and analyze the chef's movements during all hand operations. In this embodiment, the multiple sensors 682a, 682b, 682c, 682d, 682e, 682f, and 682g are embedded within the sensing glove 680 but can be seen through the material of the sensing glove 680 for external sensing. The sensing glove 680 may have feature points associated with a plurality of sensors 682a, 682b, 682c, 682d, 682e, 682f, and 682g that reflect the curves (or irregularities) of the hand at various high and low points. The sensing glove 680, which is positioned to cover the robot hand 72, is made of a soft material that mimics the flexibility and shape of human skin. A further description of the robot hand 72 can be found in Figure 9A.

[0178] The robotic hand 72 includes a camera sensor 684, such as an RGB-D sensor, imaging sensor, or visual sensing device, positioned in or near the center of the palm, to detect the distance and shape of an object and to handle kitchen tools. The imaging sensor 682f guides the robotic hand 72 in making the necessary adjustments to move it toward the object and grasp it. In addition, a sonar sensor, such as a tactile pressure sensor, can be positioned near the palm of the robotic hand 72 to detect the distance and shape of an object. The sonar sensor 682f can also guide the robotic hand 72 to move toward the object. Each of the sonar sensors 682a, 682b, 682c, 682d, 682e, 682f, and 682g includes an ultrasonic sensor, laser, radio frequency identification (RFID), and other appropriate sensors. In addition, each of the sonar sensors 682a, 682b, 682c, 682d, 682e, 682f, and 682g serves as a feedback mechanism to determine whether the robot hand 72 should continue to apply further pressure to grasp an object at a point where sufficient pressure exists to grasp and lift the object. Furthermore, the sonar sensor 682f in the palm of the robot hand 72 provides tactile sensing capabilities for handling kitchen tools. For example, when the robot hand 72 grasps a knife and cuts beef, the amount of pressure the robot hand 72 exerts on the knife and applies to the beef allows the tactile sensors to detect when the knife has finished slicing the beef, i.e., when the knife no longer offers resistance. Distributed pressure is not only for securing objects, but also to prevent applying excessive pressure, such as not breaking an egg. Furthermore, each finger on the robot hand 72 has a first sensor 682a on the tip of the thumb, a second sensor 682b on the tip of the index finger, a third sensor 682c on the tip of the middle finger, a fourth sensor 682d on the tip of the ring finger, and a fifth sensor 682 on the tip of the little finger. eAs shown, the sensor is located on the fingertip. Each of the sensors 682a, 682b, 682c, 682d, and 682e provides the ability to sense the distance and shape of an object, the ability to sense temperature or humidity, and the ability to provide tactile feedback.

[0179] In addition to the RGB-D sensor 684 and sonar sensor 682f in the palm, sonar sensors 682a, 682b, 682c, 682d, and 682e in the fingertips of each finger provide a feedback mechanism to the robot hand 72 as a means of grasping non-standardized objects or non-standardized kitchen tools. The robot hand 72 can adjust the pressure to a degree sufficient to grasp non-standardized objects. Figure 17B illustrates a program library 690 that stores grasping function samples 692, 694, and 696 according to specific time intervals, which the robot hand 72 can draw upon to perform specific grasping functions. Figure 17B is a block diagram illustrating a library database 690 of standardized operating movements within a standardized robot kitchen module 50. Standardized operating movements that are predefined and stored in the library database 690 include steps of grasping, positioning, and manipulating kitchen tools or kitchen equipment.

[0180] Figure 18A is a graph illustrating that each of the robot hands 72 is covered with an artificial human-like soft skin glove 700. The artificial human-like soft skin glove 700 is translucent and contains several embedded sensors sufficient for the robot hands 72 to perform high-level small-scale operations. In one embodiment, the soft skin glove 700 includes 10 or more sensors 702 to replicate the movements of a chef's hand.

[0181] Figure 18B is a block diagram illustrating a robotic hand covered with an artificial human-like skin glove to perform high-level miniature operations based on a library database 720 having predefined and internally stored miniature operations. High-level miniature operations relate to a sequence of motion primitives that require substantial amounts of interaction motion and interaction forces, as well as control over them. Three examples of miniature operations stored in the database library 720 are presented. The first example of a miniature operation is step 722, in which a pair of robotic hands 72 are used to knead dough. The second example of a miniature operation is step 724, in which a pair of robotic hands 72 are used to make ravioli. The third example of a miniature operation is step 724, in which a pair of robotic hands 72 are used to make sushi. Each of the three examples of miniature operations has a duration and a speed curve tracked by the computer 16.

[0182] Figure 18C is a graph illustrating three types of operational motion classifications for food preparation, each having a continuous trajectory of motion and force of a robotic arm 70 and robotic hand 72 that produces a desired target state. The robotic arm 70 and robotic hand 72 perform robust gripping and transporting motions 730 to pick up an object by immobile gripping and transport it to a target location without the need for strong force interaction. Examples of robust gripping and transporting include placing a pan on a stove, picking up a salt shaker, sprinkling salt into a dish, dropping ingredients into a bowl, pouring contents from a container, stirring a salad, and flipping a pancake. The robotic arm 70 and robotic hand 72 perform robust gripping 732 by strong force interaction, in which case there is strong force contact between the two surfaces or objects. Examples of robust gripping through strong force interaction include stirring a pot, opening a box, rotating a pan, and sweeping ingredients from a cutting board into a pan. The robotic arm 70 and robotic hand 72 perform strong force interaction 734 through deformation, in which there is strong force contact between two surfaces or objects that causes deformation of one of the two surfaces, such as in the steps of cutting a carrot, cracking an egg, or rolling dough. For additional information on the function of the human hand, the deformation of the human palm, and its function in gripping, see IAKapandji, "The Physiology of the Joints, Volume 1: Upper Limb, 6e," Churchill Livingstone, 6th edition, 2007, which is incorporated herein by full citation.

[0183] Figure 18D is a simplified flowchart illustrating one embodiment of the classification of operational actions for food preparation during the dough kneading stage. The dough kneading stage 740 can be a small operation predefined in the small operation library database in the past. The dough kneading process 740 includes an operation (or short small operation) sequence that includes a stage 742 of grasping the dough, a stage 744 of placing the dough on a surface, and a stage 746 of repeating the kneading operation until the desired shape is obtained.

[0184] Figure 18E is a block diagram illustrating an example of the interfunction and interaction between the robot arm 70 and the robot hand 72. The flexible robot arm 750 offers a small payload, high safety, and gentle motion, but provides low precision. The humanoid robot hand 752 offers high agility, allowing it to handle human tools, is easily retargetable to human hand motions, and is highly flexible, but its design requires high complexity, increased weight, and high production costs. The simple robot hand 754 is lightweight, not very expensive, has low agility, and cannot directly use human tools. Industrial robot arm 756 teeth While highly accurate and possessing a large payload capacity, these robots are generally not considered safe in the presence of humans and, in some cases, could exert a large amount of force, potentially causing harm. One embodiment of the standardized robotic kitchen 50 utilizes a first combination of a flexible arm 750 and a humanoid hand 752. The other three combinations are generally less desirable for implementing the present invention.

[0185] Figure 18F is a block diagram illustrating a standardized kitchen handle 580 to be attached to a custom cooking utensil head and a robotic hand 72 using a robotic arm 70 that can be attached to the kitchen utensil. In one technique for gripping the kitchen utensil, the robotic hand 72 grips the standardized kitchen tool 580 to be attached to one of the custom cooking utensil heads from the illustrated options 760a, 760b, 760c, 760d, 760e and other options. For example, the standardized kitchen handle 580 is attached to a custom spatula head 760e for use in frying ingredients in a pan. In one embodiment, the standardized kitchen handle 580 can be held by the robotic hand 72 in a single position, minimizing potential confusion in various methods for holding the standardized kitchen handle 580. In another technique for gripping kitchen utensils, the robotic arm has one or more holders 762 that can be attached to the kitchen utensil 762, in which case the robotic arm 70 can apply a greater force as needed to hold the kitchen utensil 762 during the robotic hand motion.

[0186] Figure 19 is a block diagram showing an example of a database library structure 770 of small operations that result in "breaking an egg with a knife". The small operation 770 of breaking an egg includes how to hold the egg in the correct position 772, how to hold the knife relative to the egg 774, what is the best angle to strike the egg with the knife 776, and how to open the broken egg 778. Various possible parameters for each of 772, 774, 776, and 778 are tested to find the best method for performing a particular movement. For example, egg of If you hold 772In the first step, various positions, orientations, and techniques for holding the egg are tested to find the optimal technique for holding the egg. Second, the robot hand 72 picks up the knife from a predetermined location. The knife-holding step 774 investigates various positions, orientations, and techniques for holding the knife to find the optimal technique for handling the knife. Third, the knife-hitting step 776 also tests various combinations of striking the egg with the knife to find the best technique for striking the egg with the knife. As a result, the optimal technique for performing the small operation 770 of cracking the egg with a knife is stored in a library database of small operations. The stored egg-cracking small operation 770 will include the best technique for holding the egg 772, the best technique for holding the knife, and the best technique for striking the egg with the knife 776.

[0187] To create a small operation that results in cracking an egg with a knife, multiple parameter combinations must be tested to identify the set of parameters that ensure the desired functional outcome of the egg being cracked is achieved. In this example, parameters are identified to determine how to grasp and hold the egg without crushing it. Through testing, a suitable knife is selected, and the appropriate placement of fingers and palm is found so that it can be held for the striking phase. The striking motion that will successfully crack the egg is identified. The opening motion and / or force that will enable the cracked egg to be successfully opened is identified.

[0188] The teaching / learning process for a robotic device involves multiple iterative tests to identify the parameters necessary to achieve the desired final functional result.

[0189] These tests can be conducted under a variety of scenarios. For example, the eggs may be of different sizes. The location where the eggs should be cracked may vary. The knife may be in a different position. The small-scale operation must be successful in all of these variable situations.

[0190] Once the learning process is complete, the results are stored as a set of behavioral primitives that are known to combine to produce the desired functional result.

[0191] Figure 20 is a block diagram showing an example of recipe execution 800 for a small operation involving real-time adjustment. In recipe execution 800, the robot hand 72 performs a small operation 770 of cracking an egg with a knife, in this case the egg grasp The optimal technique for performing each movement in operation 772, operation 774 (holding the knife), operation 776 (striking the egg with the knife), and operation 778 (opening the cracked egg) is selected from the small operation library database. The process of performing the optimal technique for each of movements 772, 774, 776, and 778 ensures that small operation 770 will obtain the same result (or guarantee thereof) or substantially the same result for this particular small operation. The multimode 3D sensor 20 provides real-time adjustment capability 112 for changes that may occur in one or more ingredients, such as the dimensions and weight of the egg.

[0192] As an example of the operational relationship between the creation of a small operation in Figure 19 and the execution of a small operation in Figure 20, specific variables related to the "crack an egg with a knife" small operation include the initial xyz coordinates of the egg, the initial orientation of the egg, the size of the egg, the shape of the egg, the initial xyz coordinates of the knife, the initial orientation of the knife, the xyz coordinates of the location where the egg is cracked, and the speed and duration of the small operation. These identified "crack an egg with a knife" small operation variables are defined during the creation phase, and in this case, these identifiable variables can be adjusted by the robotic food preparation engine 56 during the execution phase of the related small operation.

[0193] Figure 21 is a flowchart illustrating a software process 810 that captures the movements of a chef preparing food within a standardized kitchen module and generates a software recipe file 46 from the chef studio 44. In step 812, the chef 49 designs various components of a food recipe within the chef studio 44. In step 814, the robotic cooking engine 56 is configured to receive input of names, ID ingredients, and measurements related to the recipe design selected by the chef 49. In step 816, the chef 49 moves the food / ingredients into designated standardized cooking utensils / appliances and to designated locations. For example, the chef 49 picks up two medium-sized shallots and two medium-sized garlic cloves, places eight crimson mushrooms on a cutting board, and moves two thawed 20cm x 30cm puff pastry units from the freezer F02 to the refrigerator (fridge). In step 818, the chef 49 wears an capture glove 26 or tactile garment 622 with sensors that capture the chef's movement data for transmission to the computer 16. In stage 820, chef 49 begins working on a recipe of his choice from stage 122. In stage 822, the chef motion recording module 98 is configured to capture and record in real time the chef's detailed movements, including force, pressure, and XYZ position and orientation of the chef's arms and fingers within the standardized robot kitchen 50. In addition to capturing the chef's movements, pressure, and position, the chef motion recording module 98 is configured to record video (video of cooking, ingredients, process, and interactive operation) and sound (human voice, evaporation sounds during frying, etc.) during the entire food preparation process for a specific recipe. In stage 824, the robot cooking engine 56 is configured to store the captured data from stage 822, including the chef's movements from sensors on the capture glove 26 and the multimode 3D sensor 30. In stage 826, the recipe abstraction software module 104 is configured to generate a recipe script suitable for machine execution.After the recipe data is generated and saved, in step 828, the software recipe file 46 is made available for sale or subscription to users through an app store or marketplace on the user's computer in the home or restaurant, and also incorporates a robot cooking receiving app on mobile devices.

[0194] Figure 22 is a flowchart 830 illustrating a software process for food preparation in a robotic standardized kitchen by a robotic device based on one or more software recipe files 22 received from the Chef Studio System 44. In step 832, the user 24 selects a recipe to purchase or subscribe to from the Chef Studio 44 via the computer 15. In step 834, the robotic food preparation engine 56 in the home robotic kitchen 48 is configured to receive input from the input module 50 regarding the recipe selected to be prepared. In step 836, the robotic food preparation engine 56 in the home robotic kitchen 48 is configured to upload the selected recipe into a memory module 102 containing the software recipe files 46. In step 838, the robotic food preparation engine 56 in the home robotic kitchen 48 is configured to calculate the availability of ingredients to complete the selected recipe and the approximate cooking time required to finish the dish. In step 840, the robotic food preparation engine 56 in the home robotic kitchen 48 is configured to analyze the prerequisites for the selected recipe to determine whether there is a shortage or lack of any of the ingredients, or whether there is insufficient time to serve the food according to the selected recipe and serving schedule. If the prerequisites are not met, in step 842, the robotic food preparation engine 56 in the home robotic kitchen 48 sends an alert indicating that the ingredients must be added to the shopping list, or suggests an alternative recipe or serving schedule. However, if the prerequisites are met, the robotic food preparation engine 56 is configured to confirm the recipe selection in step 844. After the recipe selection is confirmed, in step 846, the user 60, via the computer 16, moves the food / ingredients to specific standardized containers and required locations. After the ingredients are placed in the identified designated containers and locations, in step 848, the robotic food preparation engine 56 in the home robotic kitchen 48 is configured to check whether the start time has been triggered.At this branching point, the home robot food preparation engine 56 performs a second process check to ensure that all prerequisites are met. If the robot food preparation engine 56 in the home robot kitchen 48 is not ready to start the cooking process, the home robot food preparation engine 56 continues to check the prerequisites in stage 850 until the start time is triggered. If the robot food preparation engine 56 is ready to start the cooking process, in stage 852, the raw food quality check module 96 in the robot food preparation engine 56 processes the prerequisites for the selected recipe and inspects each ingredient item against the recipe description (e.g., one center-cut beef tenderloin roast) and conditions (e.g., expiration date / purchase date, odor, color, texture, etc.). In stage 854, the robot food preparation engine 56 sets the time to "0" and uploads the software recipe file 46 to one or more robot arms 70 and robot hands 72 to reproduce the chef's cooking movements and generate the selected dish according to the software recipe file 46. In step 856, one or more robotic arms 72 and robotic hands 74 process the ingredients and perform cooking methods / techniques with the same movements as those of the chef's arms, hands, and fingers, with the same time increments as captured and recorded from the chef's movements, along with precise pressure, precise force, and the same XYZ positions. During this time, one or more robotic arms 70 and robotic hands 72 compare the cooking results against control data (e.g., temperature, weight, loss, etc.) and media data (e.g., color, appearance, smell, portion size, etc.), as illustrated in step 858. After the data has been compared, the robotic apparatus (including the robotic arms 70 and robotic hands 72) aligns and adjusts the results in step 860. In step 862, the robotic food preparation engine 56 is configured to instruct the robotic apparatus to transfer the finished dish to a designated serving plate and place it on the counter.

[0195] Figure 23 is a flowchart illustrating one embodiment of a software process for creating, testing, verifying, and storing various parameter combinations for a small operation library database 870. The small operation library database 870 includes a one-time successful test process 870 (e.g., the egg-holding stage) stored in a temporary library, and a stage 890 (e.g., the entire egg-cracking motion) that tests combinations of one-time test results in the small operation database library. In stage 872, the computer 16 creates a new small operation (e.g., the egg-cracking stage) having multiple action primitives (or multiple discrete recipe actions). In stage 874, the number of objects (e.g., egg and knife) associated with the new small operation is identified. In stage 876, the computer 16 identifies several discrete actions or motions. In stage 878, the computer selects the maximum possible range of key parameters (e.g., object position, object orientation, pressure, and speed) associated with a particular new small operation. In stage 880, for each major parameter, the computer 16 tests and verifies each value of the major parameter using all possible combinations with other major parameters (for example, holding an egg in one position but testing other orientations). In stage 882, the computer 16 determines whether a particular set of major parameters produces a reliable result. Verification of the result can be done by the computer 16 or a human. If the decision is false, the computer 16 proceeds to stage 886 to determine if there are other major parameter combinations that have not yet been tested. In stage 888, the computer 16 increments the major parameter by 1 in order to formulate the next parameter combination for further testing and evaluation of the next parameter combination. If the decision in stage 882 was true, the computer 16 stores the favorable set of major parameter combinations in a temporary storage library. The temporary storage library stores one or more favorable set of major parameter combinations (either having the best test results or the fewest failures).

[0196] In step 892, the computer 16 tests and verifies a particular favorable parameter combination X times (e.g., 100 times). In step 894, the computer 16 calculates the number of failures during the repeated testing of the particular favorable parameter combination. In step 896, the computer 16 selects the next favorable parameter combination from a temporary library and returns the process to step 892 to test this parameter combination X times. If no further favorable parameter combinations remain, in step 898, the computer 16 stores the test results of one or more parameter combination sets that produce reliable (or guaranteed) results. If there are more than one reliable parameter combination sets, in step 899, the computer 16 determines the best or optimal parameter combination set and stores this optimal parameter combination set, relevant to a particular small operation, in a small operation library database for use by robotic devices in the standardized robotic kitchen 50 during the food preparation stage of a recipe.

[0197] Figure 24 is a flowchart illustrating one embodiment 900 of a software process for creating tasks for small-scale operations. In step 902, the computer 16 defines a specific robotic task (e.g., the step of cracking an egg with a knife) to be stored in a database library by a robotic small-scale hand manipulator. In step 904, the computer identifies all possible different object orientations (e.g., the orientation of the egg when holding it) in each small-scale step, and in step 906, it identifies all different position points for holding a kitchen tool relative to an object (e.g., holding a knife relative to an egg). In step 908, the computer empirically identifies all possible techniques for holding the egg with the correct (cutting) motion profile, pressure, and speed to crack it with a knife. In step 910, the computer 16 defines various combinations for properly cracking the egg in terms of holding the egg and positioning the knife relative to it. For example, it finds the optimal combination of parameters such as object orientation, position, pressure, and speed. In step 912, the computer 16 conducts training and testing processes to verify the reliability of various combinations, such as testing all changes and differences, and repeats this process X times until the reliability of each small operation is assured. In step 914, when the chef 49 is performing a certain food preparation task (e.g., cracking an egg with a knife), this task is interpreted as several small manual operations / tasks to be performed as part of it. In step 916, the computer 16 stores various combinations of small operations for the relevant specific task in a database library. In step 918, the computer 16 determines whether there are any further tasks to define and perform for any of the small operations. If there are any further tasks to define, the process returns to step 902. Various embodiments of the kitchen module are possible, including standalone kitchen modules and integrated kitchen modules. The integrated kitchen module is housed within the conventional kitchen area of ​​a typical house. This kitchen module operates in at least two modes: robotic mode and normal (manual) mode. Cracking an egg is an example of a small operation.Furthermore, the small-scale operation library database is applied to various tasks, such as the step of using a fork to grasp a piece of beef by applying the correct pressure in the correct direction, to the appropriate depth relative to its shape, and to the depth of the meat. In step 919, the computer combines a database library of predefined kitchen tasks, each containing one or more small-scale operations.

[0198] Figure 25 is a flowchart illustrating process 920 for assigning and utilizing a library of standardized kitchen tools, standardized objects, and standardized equipment within a standardized robotic kitchen. In step 922, the computer 16 assigns a code (or barcode) to each kitchen tool, object, or piece of equipment / tool ​​that predefines the tool, object, or equipment parameters, such as its three-dimensional position coordinates and orientation. This process standardizes various elements within the standardized robotic kitchen 50, including, but not limited to, standardized kitchen equipment, standardized kitchen tools, standardized knives, standardized forks, standardized containers, standardized pans, standardized utensils, standardized workspaces, standardized fixtures, and other standardized elements. In step 924, when executing a process step in a cooking recipe, and being prompted to access specific kitchen tools, objects, equipment, tools, or utensils according to the food preparation process for a particular recipe, the robotic cooking engine is configured to instruct one or more robotic hands to retrieve the corresponding kitchen tools, objects, equipment, tools, or utensils.

[0199] Figure 26 is a flowchart illustrating a process 926 for identifying non-standard objects by 3D modeling and inference. In step 928, the computer 16 uses sensors to detect non-standard objects, such as food ingredients, which may have different sizes, dimensions, and / or weights. In step 930, the computer 16 identifies the non-standard objects using 3D modeling sensors 66 to capture shape, dimension, orientation, and position information, and the robot hand 72 makes real-time adjustments to perform appropriate food preparation tasks (e.g., cutting or picking up a steak).

[0200] Figure 27 is a flowchart illustrating step 932 for testing and learning small operations. In step 934, the computer performs a food preparation task composition analysis in which each cooking operation (e.g., cracking an egg with a knife) is analyzed, broken down, and constructed into a sequence of motion primitives or small operations. In one embodiment, a small operation relates to a sequence of one or more motion primitives that achieve a basic functional outcome (e.g., an egg is cracked or a vegetable is sliced) that progresses toward a specific result when preparing a food dish. In this embodiment, small operations can be further described as low-level small operations relating to a sequence of motion primitives that requires only minimal interaction forces and relies almost entirely on the use of robotic devices, or high-level small operations relating to a sequence of motion primitives that requires a considerable amount of interaction and interaction forces and their control. The process loop 936 focuses on the small operation and learning stages and consists of tests that are repeated many times (e.g., 100 times) to ensure the reliability of the small operations. In step 938, the robot food preparation engine 56 is configured to evaluate knowledge of all possible food preparation steps or mini-operations, in which case each mini-operation is tested in terms of orientation, position / velocity, angle, force, pressure, and speed with respect to a particular mini-operation. A mini-operation or motion primitive may include the robot hand 72 and a standard object or the robot hand 72 and a non-standard object. In step 940, the robot food preparation engine 56 performs the mini-operation and determines whether the outcome can be considered successful or unsuccessful. In step 942, the computer 16 performs automatic analysis and inference regarding the failure of the mini-operation. For example, a multimode sensor may provide sensory feedback data about the success or failure of the mini-operation. In step 944, the computer 16 is configured to apply real-time adjustments and adjust the parameters of the mini-operation execution process. In step 946, the computer 16 adds new information about the success or failure of parameter adjustments to the mini-operation library as a learning mechanism for the robot food preparation engine 56.

[0201] Figure 28 shows a food preparation illustrating process 950 for quality control and alignment functions for a robotic arm. In step 952, the robotic food preparation engine 56 loads a human chef replica software recipe file 46 via the input module 50. For example, the software recipe file 46 replicates the preparation of food from "Wiener Schnitzel" by Michelin-starred chef Arnd Beuchel. In step 954, the robotic device performs work based on a stored recipe script containing all motion / motion replica data, having the same movements, such as those relating to the torso, hands, and fingers, at the same pace, pressure, force, and xyz position as stored recorded recipe data based on the actions of a human chef preparing the same recipe in a standardized kitchen module with standardized equipment. In step 956, the computer 16 monitors the food preparation process via multimode sensors that generate raw data supplied to abstraction software, in which case the robotic device compares the real-world output to control data based on multimode sensing data (visual, audio, and any other sensing feedback). In step 958, the computer 16 determines whether there is any difference between the control data and the multimode sensing data. In step 960, the computer 16 analyzes whether the multimode sensing data is biased from the control data. If a bias exists, in step 962, the computer 16 makes adjustments to recalibrate the robot arm 70, robot hand 72, or other elements. In step 964, the robot food preparation engine 16 is configured to learn in process 964 by adding the adjustments made to one or more parameter values ​​to the knowledge database. In step 968, the computer 16 stores updated revision information relating to the corrected process, conditions, and parameters in the knowledge database. If no bias difference exists from step 958, process 950 proceeds directly to step 969 and terminates execution.

[0202] Figure 29 is a table illustrating one embodiment 970 of a database library structure for small-scale manipulative objects for use in a standardized robotic kitchen. The database library structure 970 shows several fields for inputting and storing information about a particular small-scale operation, including (1) the name of the small-scale operation, (2) the assignment code of the small-scale operation, (3) the codes of the standardized equipment and standardized tools associated with performing the small-scale operation, (4) the initial position and orientation of the object being operated on (standard or non-standard) (food ingredients and tools), (5) user-defined (or extracted from a recorded recipe during execution) parameters / variables, and (6) the robot hand movement sequence on the timeline (control signals for all servos) and connection feedback parameters for the small-scale operation (from any sensor or video monitoring system). The parameters for a particular small-scale operation may differ depending on the complexity and the object on which the small-scale operation must be performed. In this example, four parameters are identified: the starting XYZ position coordinates, velocity, object size, and object shape within the spatial domain of the standardized kitchen module. Both object size and object shape can be defined or described by non-standard parameters.

[0203] Figure 30 is a table illustrating a database library structure 972 of standardized objects for use in a standardized robotic kitchen. The standardized object database library structure 972 shows several fields for storing information relating to a standardized object, including (1) the name of the object, (2) an image of the object, (3) an assignment code for the object, (4) a virtual 3D model with the full dimensions of the object in a predefined preferred resolution XYZ coordinate matrix, (5) a virtual vector model of the object (if available), (6) definitions and marks of the working elements of the object (elements that can come into contact with the hand and other objects toward operation), and (7) the initial standard orientation of the object toward each specific operation.

[0204] Figure 3 1This depicts the execution of process 1000, used to check the quality of ingredients to be used as part of a recipe reproduction process by a standardized robotic kitchen. A multimode sensor system video sensing element that uses color detection and spectral analysis to detect discoloration indicating possible spoilage can perform the process 1006. Ammonia-sensitive sensor systems can similarly be used to detect further possibilities regarding spoilage, whether embedded in the kitchen or as part of a motion probe handled by a robotic hand. Further tactile sensors in the robotic hand and fingers will enable verification of the freshness of ingredients through a contact sensing process 1004 in which hardness and resistance to contact force are measured (as the amount and velocity of displacement as a function of compression distance). For example, in fish, the color (red) and water content of the gills are indicators of freshness, as are the eyes, which must be clear (not cloudy), and the proper temperature of properly thawed fish flesh should not exceed 40°F. Further contact sensors on the fingertips can perform further quality checks 1002 related to the temperature, texture, and total weight of ingredients through touching, rubbing, and holding / picking motions. All data collected through these tactile sensors and video images can be used within processing algorithms to determine the freshness of the food and decide whether to use or discard it.

[0205] Figure 32 shows a head 20 equipped with a multimode sensor and twin arms having multi-fingered hands 72 that hold ingredients and tools, facing a cooking utensil 1012. storyThe robotic process 1010, which reproduces a recipe script, is depicted. A robotic sensor head 20 with a multimode sensor unit is used to continuously model and monitor the three-dimensional workspace being worked on by both robotic arms, and at the same time to identify tools and implements, utensils and their contents and variables, and also supplies data to a work abstraction module to ensure that execution is proceeding in accordance with the sequence data for the recipe stored in the computer, comparing them to the recipe stages generated by the cooking process sequence. Further sensors within the robotic sensor head 20 are used in the audible domain to hear sounds and smells during key parts of the cooking process. The robotic hand 72 and its tactile sensors are used to properly handle each ingredient, for example, an egg in this case, and sensors in the fingers and palm can detect an egg that is usable, for example, by its surface texture and weight and distribution, and can hold it without breaking it and determine its orientation. The multi-fingered robotic hand 72 can, for example, pick up and handle a certain cooking utensil, which in this case is a ball, and grasp and handle the cooking utensil (in this case, a whisk) with appropriate motion and force application in order to properly process food ingredients as specified in the recipe script (for example, cracking an egg, separating the yolk, and whisking the egg white until a viscous component is obtained).

[0206] Figure 33 illustrates a concept 1020 of a food storage system in which a food storage container 1022, capable of storing any of the necessary cooking ingredients (e.g., meat, fish, poultry, shellfish, vegetables, etc.), is equipped with sensors for measuring and monitoring the freshness of each ingredient. The monitoring sensors embedded in the food storage container 1022 include, but are not limited to, an ammonia sensor 1030, a volatile organic compound sensor 1032, an internal container temperature sensor 1026, and a humidity sensor 1028. In addition, manual probes can be used, whether used by a human chef or a robotic arm and hand, to enable primary measurements (such as temperature) within the spatial area of ​​larger ingredients (e.g., internal temperature of meat).

[0207] Figure 34 illustrates a measurement and analysis process 1040 performed as part of a freshness and quality check of food placed in a food storage container 1042, which includes sensors and detection devices (e.g., temperature probes / needles). The container sends a dataset including temperature data 1046, humidity data 1048, ammonia concentration data 1050, and volatile organic compound data 1052 to a main server via a wireless data network communication stage 1056, using a metadata tag 1044 that specifies the container ID, where the food quality control engine processes the container data. Processing stage 1060 uses the container-specific data 1044 and compares it with acceptable data values ​​and ranges stored in a medium 1058 and retrieved therefrom by a data acquisition and storage process 1054. Subsequently, a set of algorithms makes a decision regarding the suitability of the food, and the real-time food quality analysis results are supplied via a separate communication process 1062 over the data network. The quality analysis results are then used in another process 1064, where they are either sent to a robotic arm for further action, or remotely displayed on a screen (smartphone or other display) so that the user can decide whether the ingredient should be used in a cooking process for later consumption or discarded as spoiled.

[0208] Figure 35 illustrates the function and process steps of the pre-packaged food container 1070 when used in a standardized kitchen, whether it is a standardized robotic kitchen or a chef's studio. The food container 1070 is designed in various sizes 1082 with various uses in mind and is suitable for a suitable storage environment 1080 for storing fresh items using refrigeration, freezing, chilling, etc. to obtain a specific storage temperature range. In addition, the food storage container 1070 is also designed to accommodate various types of food ingredients 1072 using containers that are pre-labeled and filled with solid (salt, flour, rice, etc.), viscous / pasteurized (mustard, mayonnaise, marzipan, jam, etc.), or liquid (water, oil, milk, juice, etc.) ingredients. In this case, the dispensing process 1074 utilizes various additional devices (droppers, chutes, peristaltic dispensing pumps, etc.) depending on the type of food ingredient, and precise computer-controlled dispensing using a dosage control engine 1084 that performs a dosage control process 1076 ensures that the correct amount of food ingredient is dispensed at the correct time. Note that recipe-specified dosages can be adjusted to suit individual taste preferences or diets (e.g., low sodium) through the menu interface or even via a remote telephone application. The dosage determination process 1078 is carried out by the dosage control engine 1084 based on the amount specified in the recipe, and the dispensing stage is performed either by a manual release command or by remote computer control based on the detection of a specific dispensing container at the outlet point of the dispenser.

[0209] Figure 36 is a block diagram illustrating a recipe system structure 1000 for use in a standardized robotic kitchen 50. It shows a food preparation process 1100 divided into multiple stages along a cooking timeline, each stage having one or more raw data blocks for each of the stages 1102, 1004, 1106, and 1108. The data blocks may include elements such as video images, audio recordings, text descriptions, and machine-readable and machine-understandable instruction sets and commands that form part of a control program. The raw data sets are contained within the recipe structure and represent each cooking stage along a timeline divided into many time-ordered stages with varying time interval levels and time sequence levels in the entire process from the start of the recipe reproduction process to the end of the cooking process, or any part of that process.

[0210] Figures 37A to 37C are block diagrams illustrating a recipe search menu for use in a standardized robotic kitchen. As shown in Figure 37A, the recipe search menu 1120 includes the most popular categories such as the type of cuisine (e.g., Italian, French, Chinese), the base of the ingredients in the dish (e.g., fish, pork, beef, pasta), or the cooking time range (e.g., longer than 60 minutes). shortThe system presents criteria and ranges such as cooking time (between 20 and 40 minutes), as well as keyword search options (e.g., ricotta cavatelli, migliaccio cake). Selected personalized recipes can exclude recipes containing allergens that the user can indicate in their personal user profile. In Figure 37B, the user can select search criteria that include requirements such as cooking time less than 44 minutes, serving enough for 7 people, offering vegetarian options, and having a total calorie count of 4521 or less. In Figure 37C, various types of dishes 1122 are shown, in which case the menu 1120 has a hierarchical level where the user can select a category (e.g., type of dish), and then narrow down the selection by expanding this category to the next subcategory (e.g., appetizer, salad, entree…). A screenshot of the implemented recipe creation and submission is illustrated in Figure 37D. Further screenshots of various graphical user interfaces and menu options are illustrated in Figures 37N to 37V.

[0211] Figures 37E to 37M illustrate various functions that the robot food preparation software 14 can perform based on database filtering and present to the user, including a flowchart that functions as a recipe filter, ingredient filter, equipment filter, account and social network access, personal affiliate page, shopping cart page, and information about purchased recipes, registration settings, and recipe creation. As clearly shown in Figure 37E, platform users can access the recipe section and select a desired recipe filter 1130 for automated robot cooking. The most common filter types include type of cuisine (e.g., Chinese, French, Italian), type of cooking (e.g., oven-baked, steamed, fried), vegetarian, and diabetic diets. From the filtered search results, users can view recipe details such as descriptions, photos, ingredients, prices, and ratings. In Figure 37F, users can select a desired ingredient filter 1132 for themselves, such as organic, ingredient type, or ingredient brand. In Figure 37G, the user can apply equipment filters 1134 to the automated robotic kitchen module, such as equipment type, brand, and manufacturer. After making selections, the user can purchase recipes, ingredients, or equipment directly from the relevant vendors through the system portal. This platform allows users to create their own additional filters and parameters, thereby making the entire system customizable and constantly updated. User-added filters and parameters will appear as system filters after approval by the coordinator.

[0212] In Figure 37H, users can connect with other users and sellers by logging into their user account 1136 through the platform's social expert network. The identity of network users is verified, in some cases, through credit card and address details. The account portal also serves as a trading platform for users to share or sell their recipes, as well as advertise to other users. Users can also manage their account resources and equipment through the account portal.

[0213] Figure 37I illustrates an example of collaboration between platform users. One user can provide all the information and details about ingredients, and another user can do the same for their equipment. All information must be filtered through a coordinator before being added to the platform / website database. In Figure 37J, a user can view information about their purchases in their shopping cart 1140. They can also change other options such as delivery and payment methods. Users can also purchase additional ingredients or equipment based on the recipes in their shopping cart.

[0214] Figure 37K shows additional information about purchased recipes that can be accessed from recipe page 1560. Users can read, listen to, and see how to cook, and in addition, they can run automated robot cooking. Communication with the seller or technical support regarding the recipe is also possible from the recipe page.

[0215] Figure 37L is a block diagram illustrating the different platform layers from the "My Account" page 1136 and the settings page 1138. From the "My Account" page, users can read expert cooking news or blogs and write and publish articles. As shown in Figure 37M, there are multiple ways in which users can create their own recipes 1570 through the recipe page under "My Account". Users can create recipes by creating automated robotic cooking scripts, either by incorporating chef cooking movements or by selecting operation sequences from the software library. Users can also create recipes by simply listing ingredients / equipment and then adding audio, video, or images. Users can edit all recipes from the recipe page.

[0216] Figure 38 is a block diagram illustrating a recipe search menu 1150 for use in a standardized robotic kitchen, where fields are selected. By selecting a category based on search criteria or range, the user 60 receives a return page listing various recipe results. The user 60 can sort these results by criteria such as user rating (e.g., high to low), expert rating (e.g., high to low), or food preparation time (e.g., short to long). The computer display may include optional tabs for the complete recipe page, displaying photos / media, title, description, rating, and price information, as well as additional information about the recipe, including a "Read More" button.

[0217] Figure 39 illustrates a possible configuration for the standardized robotic kitchen 50 for the use of the extended sensor system 1854. The extended sensor system 1854 is shown as a single extended sensor system 1854 positioned on a movable computer-controllable linear rail that moves along the length of the kitchen axis with the aim of effectively covering the entire visible three-dimensional workspace of the standardized kitchen.

[0218] Based on the proper placement of the extended sensor system 1854, which is positioned anywhere in the robot kitchen, such as on a computer-controllable rail or on the torso of a robot having an arm and a hand, 3D tracking and raw data generation become possible both during chef monitoring for machine-specific recipe script generation and during monitoring the progress and successful completion of the robot execution phase in the cooking reproduction stage within the standardized robot kitchen 50.

[0219] Figure 39 illustrates a standardized robotic kitchen 50, showing possible configurations for the use of the extended sensor system 20. The standardized robotic kitchen 50 shows a single extended sensor system 20 positioned on a movable computer-controllable linear rail that moves along the length of the kitchen axis, with the aim of effectively covering its entire visible three-dimensional workspace.

[0220] Figure 40 is a block diagram illustrating a standardized kitchen module 50 having multiple camera sensors and / or lasers 20 for real-time three-dimensional modeling 1160 of a food preparation environment. The robotic kitchen cooking system 48 includes a three-dimensional electronic sensor that can supply real-time raw data for a computer to create a three-dimensional model of the kitchen operating environment. One possible implementation of real-time three-dimensional modeling involves the use of three-dimensional laser scanning. Another implementation of real-time three-dimensional modeling is the use of one or more video cameras. A further third method involves the use of projected light patterns observed by the cameras, so-called structural light imaging. The three-dimensional electronic sensor scans the kitchen operating environment in real time to provide a visual representation (shape and dimension data) 1162 of the workspace within the kitchen module. For example, the three-dimensional electronic sensor captures a three-dimensional image in real time of whether or not the robotic arm / hand has picked up meat or fish. Since some objects may have non-standard dimensions, the three-dimensional model of the kitchen also serves as a kind of "human eye" to make adjustments in grasping objects. The computer processing system 16 generates a computer model of the three-dimensional geometric shapes and objects in the workspace and supplies control signals 1164 back to the standardized robot kitchen 50. For example, the three-dimensional modeling of the kitchen can provide a three-dimensional resolution grid with a desired spacing, such as 1 centimeter between grid points.

[0221] The standardized robotic kitchen 50 depicts another configuration that can be taken toward the use of one or more augmented sensor systems 20. The standardized robotic kitchen 50 shows multiple augmented sensor systems 20 positioned in the upper corners of the kitchen work surface along the length of the kitchen axis, with the aim of effectively covering its entire visible three-dimensional workspace.

[0222] Since robotic arms, robotic hands, tools, equipment, and utensils are involved in various stages of multiple sequential steps of cooking reproduction within the standardized robotic kitchen 50, the proper placement of the extended sensor system 20 within the standardized robotic kitchen 50 allows for 3D sensing using video cameras, lasers, sonar, and other 2D and 3D sensor systems, enabling the raw data set to be used to create processed data on the shape, location, orientation, and real-time dynamic model of the robotic arms, robotic hands, tools, equipment, and utensils.

[0223] Raw data is collected at each point in time during stage 1162, enabling processing of the raw data so that the shape, dimensions, location, and orientation of all objects important to various stages in multiple sequential stages of cooking reproduction within the standardized robot kitchen 50 can be extracted. The processed data is further analyzed by a computer system so that the standardized robot kitchen controller can adjust the trajectories and minor operations of the robot's arms and hands by modifying the control signals defined by the robot script. Given the potential variability in many variables (ingredients, temperature, etc.), the execution of the recipe script, and therefore the adjustments to the control signals, are crucial for successfully completing each reproduction stage for a particular dish. The recipe script execution process based on key measurable variables is a critical part of the use of the extended (also expressed as multimode) sensor system 20 during the execution of the reproduction stages for a particular dish within the standardized robot kitchen 50.

[0224] Figure 41A illustrates a robotic kitchen prototype. The prototype kitchen consists of three levels. The upper level includes a rail system for two arms to move along during cooking, and a retractable hood that allows the two robotic arms to be returned to a charging dock and stored when not in use for cooking or when the kitchen is set to manual cooking mode. The middle level includes a sink, a stovetop, a grill, an oven, and a work counter with access to food storage. The middle level also has a computer monitor for operating equipment, selecting recipes, viewing video and text instructions, and listening to audio instructions. The lower level includes an automated container system for storing food / ingredients in their best condition, with the feasibility of automatically delivering the ingredients required by the recipe to the cooking space area. The kitchen prototype also includes an oven, a dishwasher, cooking tools, auxiliary supplies, a cooking utensil organizer, drawers, and a recycling bin.

[0225] Figure 41B illustrates a robotic kitchen prototype having a transparent enclosure that acts as a protective mechanism to prevent potential injury to people in the vicinity while the robotic cooking process is taking place. The transparent enclosure can be made from a variety of transparent materials such as glass, fiberglass, plastic, or any other suitable material. In one example, the transparent enclosure includes one automatic glass door (or more doors). As shown in this embodiment, the automatic glass door is positioned to slide and close from top to bottom or bottom to top (from the bottom section) for safety reasons during the cooking process involving the use of the robotic arm. Variations in the design of the transparent enclosure are possible, such as sliding vertically down, sliding vertically up, horizontally from left to right, horizontally from right to left, or any other way that allows the transparent enclosure to act as a protective mechanism within the kitchen.

[0226] Figure 41C illustrates an embodiment of a standardized robotic kitchen having a horizontal sliding glass door 1190 that can be moved manually or under computer control to the left or right to isolate the robotic arm / hand's workspace from its surroundings, for purposes such as protecting any person standing near the kitchen, limiting contamination in and out of the kitchen work area, or further enabling better climate control within the enclosed space. The automatic sliding glass door slides to the left or right and closes for safety reasons during cooking processes involving the use of the robotic arm.

[0227] Figure 41D illustrates an embodiment of a standardized robotic kitchen in which the countertop or work surface includes an area with sliding doors 1200 that provide access to a food storage space within the storage space area at the bottom of the robotic kitchen counter. These doors can be slid open manually or under computer control to allow access to the food containers therein. Either manually or under computer control, one or more specific containers can be supplied to the countertop level by the food storage and supply unit, thereby allowing manual access (in this depiction, access by a robotic arm / hand) to the containers, their lids, and thus the contents of the containers. The robotic arm / hand can then open the lids, retrieve the food as needed, and place the food in an appropriate location (plate, pan, deep pot, etc.), and then seal the containers and return them to the food storage and supply unit or to the unit itself. The food storage and supply unit then returns the containers to an appropriate location inside the unit for later reuse, cleaning, or restorage. The process of supplying food containers for access by a robotic arm / hand and then stacking them again is an integral and iterative process that forms part of a recipe script, as certain stages within the recipe reproduction process request one or more of a certain type of food based on a stage in the execution of the recipe script in which the standardized robotic kitchen 50 may be involved.

[0228] To access the food storage and supply unit, a portion of the countertop with a sliding door can be opened. In this case, recipe software controls the door, and a robotic arm picks up a designated container, opens the lid, removes the ingredients from the container to a designated location, closes the lid again, and moves the container to an access point where it can be returned to storage. The container is then moved back from the access point to its default location within the storage unit, after which a new / next container item is loaded to the access point from which it is picked up.

[0229] Another embodiment 1210 of the ingredient storage and dispensing unit is depicted in Figure 41E. Specific ingredients or ingredients used repeatedly (salt, sugar, flour, oil, etc.) can be dispensed using a computer-controlled dispensing mechanism, or a specified amount of a specific ingredient can be released by a hand trigger, regardless of whether it is a human or robotic hand or finger. The amount of ingredient to be dispensed can be manually entered by a human or robotic hand on a touch panel, or it can be given by computer control. The dispensed ingredients can then be collected or supplied to kitchen equipment (bowls, pans, pots, etc.) at any point during the recipe reproduction process. This embodiment of the ingredient dispensing and dispensing system can be considered a more cost-effective and space-efficient method, while simultaneously reducing the complexity of container handling and the wasted motion time of the robotic arm / hand.

[0230] In Figure 41F, an embodiment of the standardized robotic kitchen includes a splash guard area fitted with a virtual monitor / display having a touchscreen area to enable a human operating the kitchen in manual mode to interact with the robotic kitchen and its elements. Computer projection images and separate camera monitoring of this projection area can notify the human's hand and fingers based on their location in the projection image when making a particular selection, and the system then acts accordingly. The virtual touchscreen provides access to all control and monitoring functions relating to all aspects of the equipment inside the standardized robotic kitchen 50, including retrieving and storing recipes, reviewing stored videos of complete or partial recipe execution stages by a human chef, and listening to audible playback of the human chef speaking explanations and instructions related to specific stages or operations in a particular recipe.

[0231] Figure 41G illustrates one or more robotic hard automation devices 1230 integrated into a standardized robotic kitchen. These one or more devices are computer-programmable and remotely controllable and are designed to dispense or provide pre-packaged or pre-measured quantities of specialized ingredient elements, such as spices (salt, pepper, etc.), liquids (water, oil, etc.), or other dry ingredients (flour, sugar, baking powder, etc.), as required in the recipe reproduction process. These robotic automation devices 1230 are positioned to be easily accessible to a robotic arm / hand or a human chef's arm / hand, allowing them to set and / or trigger the release of fixed amounts of selected ingredients based on the needs specified in the recipe script.

[0232] Figure 41H illustrates one or more robotic hard automation devices 1340 integrated into a standardized robotic kitchen. These one or more devices are remotely programmable and remotely controllable by a computer and are designed to supply or dispense pre-packaged or pre-measured quantities of commonly used ingredient elements required in the recipe reproduction process, in which case the dosage control engine / system is designed to dispense precisely the correct amount into specific equipment such as bowls, pots, or pans. These robotic automation devices 1340 are positioned to be easily accessible to a robotic arm / hand or a human chef's arm / hand, allowing the robotic arm / hand to set and / or trigger the release of dosage engine-controlled amounts of selected ingredients based on the needs specified in the recipe script. This embodiment of the ingredient supply and dispensing system can be considered a more cost-effective and space-efficient method, while simultaneously reducing the complexity of container handling and wasted motion time by the robotic arm / hand.

[0233] Figure 41I shows a ventilation system 1250 for extracting smoke and steam during an automated cooking process, and for removing harmful smoke and hazardous fumes. flame The image depicts a standard robotic kitchen equipped with both an automatic smoke / flame detection and suppression system 1252 that eliminates either source of smoke or flame, and a sliding door safety glass that surrounds the standard robotic kitchen 50 to contain the affected space.

[0234] Figure 41J illustrates a standardized robotic kitchen 50 having a waste management system 1260 located in a lower storage compartment, which enables easy and rapid disposal of recyclable items (glass, aluminum, etc.) and non-recyclable items (food scraps, etc.) using a set of removable lidded waste containers that include sealing elements (gaskets, O-rings, etc.) that enable airtight sealing to prevent odors from leaking into the standardized robotic kitchen 50.

[0235] Figure 41K illustrates a standardized robotic kitchen 50 having an overloading dishwasher 1270 positioned in a specific location within the kitchen for ease of loading and unloading by a robot. The dishwasher includes a sealed lid that can also be used as a cutting board or workspace with a built-in drain during the execution of the automated recipe reproduction stage.

[0236] Figure 41L depicts a standardized kitchen having an instrumented food quality control system 1280, which consists of an instrumentation panel with sensors and food probes. This area includes sensors on a splatter guard that can detect multiple physical and chemical properties of the food placed inside, including, but not limited to, spoilage (ammonia sensor), temperature (thermocouple), volatile organic compounds (released during biomass decomposition), and moisture / humidity (hygrometer) components. Food probes may also be present using temperature sensor (thermocouple) detection devices used by a robotic arm / hand to examine the internal properties of specific cooked food ingredients or cooking elements (internal temperature of red meat, poultry, etc.).

[0237] Figure 42A illustrates an embodiment of a standardized robotic kitchen in plan view 50, and it should be understood that the elements within it can be arranged in different ways. The standardized robotic kitchen is divided into three levels: upper level 1292-1, counter level 1292-2, and lower level 1292-3.

[0238] The upper level 1292-1 includes multiple storage-type modules with various units to perform specific kitchen functions using built-in fixtures and equipment. In the simplest level, the shelf / storage storage area 1294 contains cooking tools, utensils, and other cooking equipment. BeautyIt includes a storage space area 1296 used for storing and accessing serving utensils (cooking, oven baking, plating, etc.), a storage and ripening storage space area 1298 for specific food items (e.g., fruits, vegetables, etc.), a chilled storage zone 1300 for items such as lettuce and onions, a frozen storage space area 1302 for deep-frozen items, and another pantry zone 1304 for other food items and infrequently used spices, etc.

[0239] The counter level 1292-2 not only accommodates the robotic arm 70 but also includes a serving counter 1306, a counter area 1308 with a sink, another counter area 1310 with a removable work surface (cutting board, etc.), a charcoal-fired grill with slats 1312, and a multi-purpose area 1314 for other cooking equipment including stoves, cookers, steamers, and steaming pots.

[0240] The lower level 1292-3 houses a combination of a convection oven and microwave oven 1316, a dishwasher 1318, and a larger storage area 1320 for maintaining and storing further frequently used cooking and baking utensils, as well as tableware, packaging materials, and knives.

[0241] Figure 42B depicts a perspective view 50 of a standardized robotic kitchen, showing the locations of the upper level 1292-1, counter level 1292-2, and lower level 1294-3 within an xyz coordinate frame having x-axis 1322, y-axis 1324, and z-axis 1326, in order to enable proper geometric reference for positioning the robotic arm 34 within the standardized robotic kitchen.

[0242] Perspective view 50 of the robot kitchen clearly shows one of many possible layouts and locations for equipment on all three levels, including the upper level (storage pantry 1304, standardized cooking tools and utensils 1320, storage aging zone 1298, chilled storage zone 1300, and frozen storage zone 1302), the counter level 1292-2 (robot arm 70, sink 1308, cutting board area 1310, charcoal grill 1312, cooking utensils 1314, and serving counter 1306), and the lower level (dishwasher 1318 and oven and microwave 1316).

[0243] Figure 43A depicts a plan view of one possible physical embodiment of a standardized robotic kitchen layout in which the kitchen is incorporated within a fairly linear and substantially rectangular horizontal layout, depicting a built-in monitor 1328 for the user to operate equipment, select recipes, watch videos, and listen to recorded chef instructions, as well as an automatically computer-controlled left / right movable transparent door 1330 for enclosing the open surface of the standardized robotic cooking space area while the robotic arm is in operation.

[0244] Figure 43B depicts a perspective view of one possible physical embodiment of a standardized robotic kitchen layout in which the kitchen is incorporated within a fairly linear and substantially rectangular horizontal layout, illustrating a built-in monitor 1332 for the user to operate equipment, select recipes, watch videos, and listen to recorded chef instructions, as well as an automatically computer-controlled left / right movable transparent door 1334 for enclosing the open surface of the standardized robotic cooking space area while the robotic arm is in operation. Sample screenshots of the standardized robotic kitchen are shown in Figures 43C to 43E, and Figure 43F illustrates the sample kitchen module specifications.

[0245] Figure 44A depicts a plan view of another possible physical embodiment of a standardized robotic kitchen layout in which the kitchen is incorporated within a fairly linear and substantially rectangular horizontal layout, depicting a built-in monitor 1336 for the user to operate equipment, select recipes, watch videos, and listen to recorded chef instructions, as well as an automatically computer-controlled up / down movable transparent door 1338 for enclosing the open surface of the standardized robotic cooking space area while the robotic arm is in operation.

[0246] Figure 44B depicts a perspective view of one possible physical embodiment of a standardized robotic kitchen layout in which the kitchen is incorporated within a fairly linear and substantially rectangular horizontal layout, illustrating a built-in monitor 1340 for the user to operate equipment, select recipes, watch videos, and listen to recorded chef instructions, as well as an automatically computer-controlled up / down movable transparent door 1342 for enclosing the open surface of the standardized robotic cooking space area while the robotic arm is in operation.

[0247] Figure 45 illustrates a perspective layout of a retractable life 1350 within a standardized robotic kitchen 50, in which a pair of robotic arms, a robotic wrist, and a robotic multi-fingered hand move linearly (in linear, stepped extensions) and retractably as a single unit on a torso, along the vertical y-axis 1352 and the horizontal x-axis 1354, and in addition rotates around the vertical y-axis extending through the centerline of its own torso. Actuators enabling these linear and rotational motions are embedded at a higher level within the torso to allow the robotic arms to move to various locations within the standardized robotic kitchen during all parts of recipe reproduction as specified in the recipe script. These multiple motions are necessary to properly reproduce the motions of a human chef 49 observed in a chef studio kitchen environment during cooking when a human chef is preparing food.

[0248] Figure 46A depicts a plan view of one physical embodiment 1356 of a standardized robotic kitchen layout in which the kitchen is incorporated within a fairly linear and substantially rectangular horizontal layout depicting a set of twin robotic arms having a wrist and a multi-fingered hand, and each arm base is not mounted on a movable rail set or a rotatable body, but instead is immovably mounted on one of the same vertical surfaces of the robotic kitchen, thereby defining and fixing the location and dimensions of the robotic body, while still allowing both robotic arms to work together and reach all areas of the cooking surface and equipment.

[0249] Figure 46B depicts a perspective view of one physical embodiment 1358 of a standardized robotic kitchen layout in which the kitchen is incorporated within a fairly linear and substantially rectangular horizontal layout depicting a set of twin robotic arms having a wrist and a multi-fingered hand, and each arm base is not mounted on a movable rail set or a rotatable body, but instead is immovably mounted on one of the same vertical surfaces of the robotic kitchen, thereby defining and fixing the location and dimensions of the robotic body, but both robotic arms are still able to work together and reach all areas of the cooking surface and equipment (oven on the rear wall, countertops below the robotic arms, and sinks on one side of the robotic arms).

[0250] Figure 46C shows the height along the y-axis and the width along the x-axis of the standardized robot kitchen. 2164mm and 3415mm respectively A dimensioned front view of one possible physical embodiment 1360 of a standardized robotic kitchen is depicted.

[0251] Figure 46D shows the height along the y-axis of the standardized robot kitchen. The total length is 2284mm. A dimensioned side section view is depicted of one possible physical embodiment 1362 of a standardized robotic kitchen, demonstrating that it is a standardized robotic kitchen.

[0252] Figure 46E shows a dimensioned side view of one physical embodiment 1364 of a standard robotic kitchen, showing that the height along the y-axis and the depth along the z-axis of the standard robotic kitchen are 2284 mm and 1504 mm, respectively.

[0253] Figure 46F shows a dimensioned upper section view of one physical embodiment 1366 of a standardized robotic kitchen, including a pair of robotic arms 1368, which show that the overall robotic kitchen module depth along the z-axis is 1504 mm.

[0254] Figure 46G depicts three figures of one physical embodiment of a standardized robotic kitchen, with a cross-sectional view added, showing a total length of 3415 mm along the x-axis, a total height of 2164 mm along the y-axis, a total depth of 1504 mm along the z-axis, and a total height of 2284 mm along the z-axis in the side cross-sectional view.

[0255] Figure 47 is a block diagram illustrating a programmable storage system 88 for use with a standardized robotic kitchen 50. The programmable storage system 88 is structured within the standardized robotic kitchen 50 based on relative xy position coordinates within the storage system 88. In this example, the programmable storage system 88 has 27 storage locations (arranged in a 9x3 matrix) with 9 columns and 3 rows. The programmable storage system 88 can function as a freezer or a refrigerator. In this embodiment, each of the 27 programmable storage locations includes four types of sensors: a pressure sensor 1370, a humidity sensor 1372, a temperature sensor 1374, and an odor (olfactory) sensor 1376. Because each storage location is recognizable by its xy coordinates, the robotic device can access a selected programmable storage location and obtain the food items necessary to prepare a meal at that location. The computer 16 can also monitor each programmable storage location for appropriate temperature profiles, humidity profiles, pressure profiles, and odor profiles, ensuring that optimal storage conditions are monitored and maintained for specific food items or ingredients.

[0256] Figure 48 shows an elevation view of a container storage station 86 in which temperature, humidity, and relative oxygen content (as well as other room conditions) can be monitored and controlled by computer. This storage container unit may include, but is not limited to, a pantry / dry storage area 1304, a wine-critical aging area 1298 with separately controllable temperature and humidity (for fruits / vegetables), a chilled unit 1300 for lower-temperature storage to optimize the effective shelf life of agricultural products / fruits / meat, and a refrigerated unit 1302 for long-term storage of other items (meat, baked goods, seafood, ice cream, etc.).

[0257] Figure 49 shows an elevation view of a food container 1300 that will be accessed by a human chef as well as a robotic arm and a multi-fingered hand. This section of the standardized robotic kitchen includes, but is not limited to, several units, including a food quality monitoring dashboard (display) 1382, a computerized measurement unit 1384 including a barcode scanner, camera, and weighing scale, a separate counter 1386 with automated rack-shelves for food receiving and dispensing, and a recycling unit 1388 for the disposal of recyclable metals (glass, aluminum, metal, etc.) and soft materials suitable for recycling (leftovers and scraps).

[0258] Figure 50 illustrates a food quality monitoring dashboard 1390, a computer-controlled display for use by human chefs. This display allows users to view several items important to the food supply and food quality aspects of human and robotic cooking. These items include a food inventory overview 1392 outlining what is available, selected individual food items, their nutritional content, and relative distribution 1394, quantities and dedicated storage 1396 as a function of storage categories (meat, vegetables, etc.), a schedule 1398 illustrating approaching expiration dates, replenishment / replenishment dates, and items, areas 1400 for all types of alarms (sensed spoilage, abnormal temperature, or malfunction, etc.), and a selection of voice-interpreted command inputs 1402 to enable human users to interact with the computer-processed inventory system through the dashboard 1390.

[0259] Figure 51 is a table illustrating an example of a recipe parameter library database 1410. The recipe parameter library database 1410 includes many categories such as food group profiles 1402, dietary types 1404, media library 1406, recipe data 1408, robot kitchen tools and equipment 1410, ingredient groups 1412, ingredient data 1414, and cooking techniques 1416. Each of these categories provides a list of detailed options available when selecting a recipe. Food group profiles include parameters such as age, gender, weight, allergies, drug use, and lifestyle. Dietary type group profiles 1404 include dietary types by region, culture, or religion, and cooking equipment group type profiles 1410 include items such as pans, grills, or ovens, and cooking time. Recipe data group profiles 1408 include items such as recipe name, version, cooking and preparation time, and required tools and equipment. The ingredient group profile 1412 includes ingredients grouped into categories such as dairy products, fruits and vegetables, grains and other carbohydrates, various types of fluids, and various types of proteins (meat, beans). The ingredient data group profile 1414 includes ingredient descriptor data such as name, description, nutritional information, storage, and handling instructions. The cooking technique group profile 1416 includes information about specific cooking techniques grouped into fields such as mechanical techniques (sauce application, shredding, grating, chopping, etc.) and chemical processing techniques (marinating, pickling, fermentation, smoking, etc.).

[0260] Figure 52 is a flowchart illustrating one embodiment of the process 1420 of one embodiment of the steps for recording the chef's food preparation process. In step 1422, within the chef's studio 44, the multimode 3D sensor 20 scans the kitchen module spatial area to define the xyz coordinate position and orientation of standardized kitchen equipment and all objects within it, whether static or dynamic. Step 1424 toIn this step, the multimode 3D sensor 20 scans the spatial area of ​​the kitchen module to find the xyz coordinate positions of non-standardized objects such as food ingredients. In step 1426, the computer 16 creates 3D models for all non-standardized objects and stores the type and attributes (size, dimensions, usage, etc.) of these objects in the system memory of the computer on the computing device or in the cloud computing environment, defining the shape, size, and type of these non-standardized objects. In step 1428, the chef motion recording module 98 senses and captures the movements of the chef's arm, wrist, and hand (the chef's hand movements are preferably identified and classified according to standard small operations) through the chef's glove over a continuous time interval. In step 1430, the computer 16 stores the sensed and captured data of the chef's movements while preparing food in the computer's memory storage device.

[0261] Figure 53 is a flowchart illustrating one embodiment 1440 of the process of one embodiment of a robotic device for preparing food dishes. In step 1442, a multimode 3D sensor 20 in the robotic kitchen 48 scans the spatial area of ​​the kitchen module to find the xyz position coordinates of non-standardized objects (ingredients, etc.). In step 1444, the multimode 3D sensor 20 in the robotic kitchen 48 creates a 3D model of the non-standardized objects detected in the standardized robotic kitchen 50 and stores the shape, size, and type of these non-standardized objects in the computer's memory. In step 1446, the robotic cooking module 110 starts executing a recipe by replicating the chef's food preparation process at the same pace, with the same movements, and for a similar duration, according to the converted recipe file. In step 1448, the robotic device executes robotic instructions from the converted recipe file having one or more combinations of small operation and motion primitives, thereby resulting in the robotic device in the robotic standardized kitchen preparing a food dish with the same or substantially the same results as if the chef 49 had prepared the food dish himself.

[0262] Figure 54 is a flowchart illustrating a process 1450 of one embodiment in which quality and function adjustments are made to obtain the same or substantially the same results as a chef in robotic food preparation. In step 1452, the quality check module 56 is configured to monitor and verify the recipe reproduction process by the robotic device using one or more multimode sensors, sensors on the robotic device, and to perform a quality check by using abstraction software to compare the output data from the robotic device with control data from a software recipe file created by monitoring and abstracting the cooking process performed while a human chef executes the same recipe in a chef studio version of a standardized robotic kitchen. In step 1454, the robotic food preparation engine 56 is configured to monitor for any differences that would require the robotic device to make adjustments to the food preparation process, for example, differences in at least the size, shape, or orientation of ingredients. If a difference exists, the robotic food preparation engine 56 is configured to correct the food preparation process by adjusting one or more parameters for this particular food preparation stage based on raw sensing input data and processed sensing input data. In step 1454, decisions are made regarding the impact on potential differences between the sensed and abstracted process progress and the stored process variables in the recipe script. If the process result of the cooking process in the standardized robot kitchen is identical to that specified in the recipe script for this process step, the food preparation process continues as described in the recipe script. If modifications or adjustments to the process are required based on the raw sensed input data and the processed sensed input data, adjustment process 1556 is performed by adjusting any parameter required to ensure that the process variables are brought to a state that conforms to those specified in the recipe script for this process step. Following the successful outcome of adjustment process 1456, the food preparation process 1458 resumes as specified in the recipe script sequence.

[0263] Figure 55 is a flowchart illustrating a first embodiment 1460 of the process of a robot kitchen that prepares food by recreating the movements of a chef from a software file recorded within the robot kitchen. In step 1462, the user selects a specific recipe via a computer for the robot device to prepare a food dish. In step 1464, the robot food preparation engine 56 is configured to obtain an abstraction recipe for the selected recipe for food preparation. In step 1468, the robot food preparation engine 56 is configured to upload the selected recipe script into the computer's memory. In step 1470, the robot food preparation engine 56 calculates the availability of ingredients and the required cooking time. In step 1472, the robot food preparation engine 56 is configured to issue an alarm or notification if there are insufficient ingredients or time to prepare the dish according to the selected recipe and serving schedule. In step 1472, the robot food preparation engine 56 sends an alarm to add the missing or insufficient ingredients to the shopping list or to select a different recipe. In step 1474, the user's recipe selection is confirmed. In step 1476, the robotic food preparation engine 1476 is configured to check whether it is time to start preparing the recipe. Process 1460 pauses in step 1476 until the start time arrives. ’ In this process, the robotic device inspects each food item for freshness and condition (e.g., purchase date, expiration date, odor, color). Stage 1462 ’ In this configuration, the robot food preparation engine 56 is configured to send a command to the robot device to move food or ingredients from a standardized container to a food preparation position. Stage 1464 ’ In this configuration, the robot food preparation engine 56 is configured to instruct the robotic device to begin food preparation by reproducing a food dish from a software recipe script file at start time "0". Stage 1466 ’In this scenario, the robotic device within the standardized kitchen 50 replicates food preparation using the same movements, ingredients, pace, and equipment and tools as a chef's arm and fingers. Stage 1468 ’ In this process, the robotic device performs quality checks during the food preparation process and makes any necessary parameter adjustments. (Step 1470) ’ At this point, the robotic device completes the reproduction and preparation of the food dish, and is therefore ready to plate and serve the food dish.

[0264] Figure 56 illustrates the process 1480 for receiving and identifying storage containers. In step 1482, the user selects to receive ingredients using a quality monitoring dashboard. Subsequently, in step 1484, the user examines the ingredient package at the receiving station or receiving counter. In step 1486, using additional data from barcode scanners, weighing scales, cameras, and laser scanners, the robotic cooking engine processes ingredient identification data, maps it to the ingredient and recipe library, and analyzes this data regarding potential allergy implications. If an allergy potential exists based on step 1488, the system notifies the user in step 1490 and decides to discard the ingredient for safety reasons. If the ingredient is deemed acceptable, the system records and verifies it in step 1492. In step 1494, the user unpacks and removes the items (if they have not already been unpacked). In the subsequent step 1496, the items are packaged (foil, vacuum bags, etc.), labeled with computer-printed labels containing all necessary ingredient data, and moved to storage containers and / or storage locations based on the identification results. Then, in step 1498, the robotic cooking engine updates its internal database and displays the available ingredients in the quality monitoring dashboard.

[0265] Figure 57 illustrates the process 1500 for retrieving ingredients from storage and preparing them for cooking. In the first stage 1502, the user selects to retrieve ingredients using a quality monitoring dashboard. In stage 1504, the user selects to retrieve items based on a single item required for one or more recipes. Subsequently, in stage 1506, the computerized kitchen functions to move the specific container containing the selected items from its storage location to the counter area. In stage 1508, once the user has picked up the items, the user processes the items in stage 1510 using one or more of several possible methods (cooking, disposal, recycling, etc.), and in stage 1512, any remaining items are returned to the system, thereby completing the user's interactive operation with the system 1514. When the robotic arm in the standardized robotic kitchen receives the retrieved ingredient items, stage 1516 is performed in which the arm and hand inspect each ingredient item in the container against its identification data (type, etc.) and condition (expiration date, color, odor, etc.). In quality check stage 1518, the robotic cooking engine makes a decision about potential item mismatches or detected quality conditions. If an item is unsuitable, stage 1520 alerts the cooking engine to take appropriate corrective action. If the ingredients are of an acceptable type and quality, in stage 1522 the robotic arm transfers the item to be used in the next cooking process stage.

[0266] Figure 58 illustrates the automated pre-cooking preparation process 1524. In step 1530, the robotic cooking engine calculates the amount of surplus and / or discarded ingredients based on a specific recipe. Subsequently, in step 1532, the robotic cooking engine searches for all possible techniques and methods to execute the recipe using each ingredient. In step 1534, the robotic cooking engine calculates and optimizes ingredient usage in terms of time and energy consumption, especially for dishes requiring parallel multi-task processes. Subsequently, the robotic cooking engine performs multi-level cooking for the planned dish. planIn step 1536, the system creates a request to the robot kitchen system to perform cooking. In the next step 1538, the robot kitchen system moves the ingredients and cooking / baking equipment required for the cooking process from its automated shelving system, and in step 1540, it assembles the tools and equipment to form various work stations.

[0267] Figure 59 illustrates the recipe design and scripting process 1542. In the first stage 1544, the chef selects a specific recipe, and then in stage 1546, the chef inputs or edits recipe data, including but not limited to the name and other metadata (background, techniques, etc.) for this recipe. In stage 1548, the chef inputs or edits the necessary ingredients based on the database and related libraries, and inputs the respective quantities in the weight / volume / units required for the recipe. In stage 1550, the chef selects the necessary techniques to be used in preparing the recipe, based on those available to the chef in the database and related libraries. In stage 1552, the chef makes similar selections, but this time focusing on selecting the cooking methods and preparations required to execute the recipe for this dish. The final stage 1554 then allows the system to create a recipe ID that will be useful for later storage and retrieval in the database.

[0268] Figure 60 illustrates process 1556 of a method that allows a user to select a recipe. The first step 1558 involves the user purchasing a recipe from an online marketplace store or subscribing to a recipe purchase plan via a computer or mobile application, thereby enabling the download of a recipe script that can be reproduced. In step 1560, the user searches an online database and selects a specific recipe from these purchases or available as part of a subscription, based on personal preferences and the availability of ingredients on-site. In the final step 1562, the user enters the date and time they want the meal prepared for serving.

[0269] Figure 61A illustrates process 1570 relating to the process of searching for, purchasing, and / or subscribing to recipes on an online service portal, or so-called recipe trading platform. As an initial step, before users can search for and view these recipes by downloading them via an app on a handheld device or using a TV and / or robotic kitchen module, new users must register with the system in step 1572 (selecting age, gender, dietary preferences, etc., and then selecting their overall preferred cooking or kitchen style). In step 1574, users can choose to perform a search using criteria such as recipe style (including manual cooking recipes) 1576, or based on a specific kitchen or equipment style (woks, steamers, smokers, etc.) 1578. In step 1580, users can choose or configure their search to use predefined criteria and further use a filtering step 1582 to narrow down the search space and subsequent results. In step 1584, users select recipes from the given search results, information, and recommendations. Next, in step 1586, the user can choose to share, collaborate on, or discuss this selection with cooking friends and online communities in the following steps.

[0270] Figure 61B depicts the continuation from Figure 61A. In step 1592, the user is prompted to select a specific recipe based on either a robotic cooking method or a parameter-controlled version of the recipe. In the case of a parameter-controlled recipe, in step 1594, the system provides details of the necessary equipment, including all cooking utensils and tools and robotic arm requirements, and in step 1602, it presents the option of external links to ingredient suppliers and equipment suppliers for detailed ordering instructions. The portal system then performs a recipe type check 1596, where the system either allows direct download and installation 1598 of the recipe program file on a remote device, or in step 1600, requires the user to enter payment information using one of many possible payment methods (PayPal, Bitcoin, credit card, etc.) based on a one-time payment or subscription-based payment.

[0271] Figure 62 illustrates the process 1610 used to create a robot recipe cooking application ("App"). In the first step 1612, the developer must create an account in a place such as the App Store, Google Play, or Windows Mobile, or other such marketplaces including the provision of banking and company information. Subsequently, in step 1614, the user is prompted to obtain and download the specific, up-to-date Application Programming Interface (API) documentation from each app store. Subsequently, in step 1618, the developer must create a recipe program that meets the API documentation requirements according to the specified API requirements. In step 1620, the developer must provide a name and other metadata for the recipe that is suitable for various sites (Apple, Google, Samsung, etc.) and specified by these sites. Step 1622 requires the developer to upload the recipe program and metadata files for approval. In stage 1624, each marketplace site will review, test, and approve the recipe program, and then in stage 1626, each site will list and make available the recipe program for online search, browsing, and purchase through their respective site's purchase interface.

[0272] Figure 63 illustrates the process 1628 for purchasing a specific recipe or subscribing to a recipe delivery plan. In the first stage 1630, the user searches for a specific recipe to order. The user can choose to browse by keyword (stage 1632), narrow down results using preference filters (stage 1634), browse using other predefined criteria (stage 1636), or even browse based on promotional recipes, newly released recipes, or pre-order-based recipes, or even based on live chef cooking events (stage 1638). The search results for recipes are displayed to the user in stage 1640. Subsequently, as part of stage 1642, the user can browse these recipe results and preview each recipe with audio or a short video clip. Then, in stage 1644, the user selects a device and operating system and receives a link to a specific online marketplace application site. If the user chooses to connect to a new provider site in step 1648, the site requires the new user to complete authentication and consent step 1650, and then in step 1652, the site allows the user to download and install site-specific interface software and continue the recipe delivery process. The provider site will ask the user in step 1646 whether to create a robot cooking shopping list, and if the user agrees, the user will select a specific recipe in step 1654, either on a one-off or subscription basis, and choose a specific date and time for the meal to be served. In step 1656, the user is provided and displayed with a shopping list of the necessary ingredients and equipment, including the earliest supplier and its location, availability of ingredients and equipment, and the associated delivery preparation time and price. In step 1658, the user is given the opportunity to examine the description of each item and its default or recommended supplier and brand. Then, in step 1660, the user can view the associated costs for all items on the ingredient and equipment list, including the costs of all associated series items (transportation, taxes, etc.).In step 1662, if the user or buyer wishes to view items other than those suggested on the shopping list, step 1664 is performed to present the user or buyer with alternative suppliers, allowing them to connect to and view alternative purchase and order options. If the user or buyer accepts the suggested shopping list, the system not only saves these selections as personalized options for future purchases and updates the current shopping list (step 1668), but then proceeds to step 1670, where the system selects alternative items from the shopping list based on further criteria such as regional / nearest provider, availability of items based on time and maturity stage, or more effectively, price for equipment from different suppliers that have the same performance but differ significantly in delivery costs to the user or buyer.

[0273] Figures 64A and 64B are block diagrams illustrating example 1672 of predefined recipe search criteria. In this example, the predefined recipe search criteria include categories such as main ingredients, cooking time, dietary style by geographical range and type, chef name search, specialty dishes, and estimated ingredient cost when preparing the food dish. Other possible recipe search fields include food type, special diets, excluded ingredients, dish type and cooking method, occasion and season, reviews and suggestions, and rating ranking.

[0274] Figure 66 is a block diagram illustrating several predefined containers within a robotic standardized kitchen 50. Each container within the standardized robotic kitchen 50 has a container number or barcode associated with the specific contents stored within it. For example, the first container stores large, bulky products such as white cabbage, red cabbage, savoy cabbage, turnips, and cauliflower. The sixth container stores large quantities of solids, including peeled almonds, seeds (sunflower, pumpkin, white), pitted dried apricots, dried papaya, and dried apricots, individually. Figure 66 is a block diagram illustrating a first embodiment of a robotic restaurant kitchen module configured in a rectangular layout with multiple pairs of robotic hands for simultaneous food preparation processing. Another embodiment of the present invention focuses on a multi-tiered configuration relating to multiple sequential or parallel robotic arms and hand stations in a commercial or restaurant kitchen configuration shown in Figure 66. While any geometric arrangement can be used, this embodiment depicts a fairly linear configuration showing multiple robotic arm / hand modules, each focused on creating a specific element, dish, or recipe script stage (for example, six pairs of robotic arms / hands playing different roles in a commercial kitchen, such as sous chef, grill cook, fry / sauté cook, pantry cook, pastry chef, soup and sauce cook, etc.). The robotic kitchen layout is such that access / interaction with any human or between adjacent arm / hand modules lies along a single forward-facing plane. This configuration can be computer-controlled, thereby enabling the entire multi-arm / hand robotic kitchen configuration to perform each recreated cooking task, whether the arm / hand robotic modules execute a single recipe sequentially (the final product from one station is fed to the next station for the next stage in the recipe script) or execute multiple recipes / stages in parallel (e.g., pre-preparing food / ingredients for subsequent use in the dish recreation finishing stage to cope with busy periods).

[0275] Figure 67 is a block diagram illustrating a second embodiment of a robotic restaurant kitchen module configured in a U-shaped layout with multiple pairs of robotic hands for simultaneous food preparation processing. Yet another embodiment of the present invention focuses on another multi-stage configuration relating to multiple sequential or parallel robotic arm and hand stations in a commercial or restaurant kitchen configuration shown in Figure 67. While any geometric arrangement can be used, this embodiment depicts a linear configuration showing multiple robotic arm / hand modules, each focused on creating a specific element, dish, or recipe script stage. This robotic kitchen layout is such that access / interactive operation with any human or between adjacent arm / hand modules lies along both the outward-facing set of U-shaped surfaces and the central portion of the U-shape, thereby enabling arm / hand modules to traverse / reach across opposing work areas and interact with opposing arm / hand modules between recipe reproduction stages. This configuration can be computer-controlled, allowing the entire multi-arm / hand robotic kitchen configuration to perform each recreated cooking task, whether the arm / hand robotic modules execute a single recipe sequentially (the final product from one station is fed to the next station along a U-shaped path to the next step in the recipe script) or execute multiple recipes / steps in parallel (such as pre-prepared food / ingredients for later use in the cooking recreation finishing stage to cope with busy periods, in which case, if possible, the prepared ingredients are stored in containers or utensils (such as refrigerators) housed inside the base of the U-shaped kitchen).

[0276] Figure 68 illustrates a second embodiment 1680 of the robotic food preparation system. A chef studio having a standardized robotic kitchen system 1682 includes a human chef 49 who prepares or executes a recipe, while sensors on the cooking equipment 1682 record important variables (such as temperature) over time and store these in the computer's memory 1684 as sensor curves and parameters that form part of a recipe script raw data file. These sensor curve and parameter data files stored from the chef studio 1682 are delivered to a standardized (remote) robotic kitchen on a purchase or subscription basis. A standardized robotic kitchen 1688 located in a home includes both a computer control system 1690 for operating automated kitchen equipment and / or robotic kitchen equipment based on received raw data corresponding to the measured sensor curve and parameter data files.

[0277] Figure 69 illustrates another embodiment 48 of the standardized robotic kitchen. A computer 16 that operates a robotic cooking (software) engine 56, which includes a cooking operation control module 1692 that processes sensing data recorded, analyzed, and abstracted from recipe scripts, and associated storage media and memory 1694 for storing software files consisting of sensing curves and parameter data, is interfaced with several external devices. These external devices include, but are not limited to, a retractable safety glass 68, a computer-monitored and computer-controllable storage unit 88, several sensors 198 that report on the quality and supply process of raw food, a hard automation module 82 for removing ingredients, a standardized container 86 containing ingredients, and an intelligent cooking utensil 1700 equipped with sensors.

[0278] Figure 71 depicts an intelligent cookware 1700 (a saucepan in this image) that includes a built-in real-time temperature sensor capable of generating and wirelessly transmitting a temperature profile across the bottom surface of the unit, including at least three planar zones, but not limited to, zones 1702, zone 21704, and zone 31706, arranged concentrically across the entire bottom surface of the cookware unit. Each of these three zones can wirelessly transmit data 1708, data 21710, and data 31712, respectively, based on coupled sensors 1716-1, 1716-2, 1716-3, 1716-4, and 1716-5.

[0279] Figure 71 illustrates a typical set of sensing curves 220, each containing recorded temperature profiles for data 1 1720, data 2 1722, and data 3 1724, corresponding to the temperature in each of three zones located on the bottom surface of a specific area of ​​the cooking utensil unit. The unit of measurement for time is minutes, reflecting the cooking time from start to finish (independent variable), while temperature is measured in degrees Celsius (dependent variable).

[0280] Figure 72 illustrates multiple sets of sensing curves 1730 with recorded temperature 1732 and humidity 1734 profiles, with all data from each sensor represented as data 1 1736, data 2 1738 through data N 1740. The raw data stream is sent to the operation control unit 274 for processing. The unit of measurement for time is minutes, reflecting the cooking time from start to finish (independent variable), while the temperature and humidity values ​​are measured in degrees Celsius and relative humidity, respectively (dependent variables).

[0281] Figure 73 illustrates the process configuration for real-time temperature control 1700 using a smart (deep frying) pan. Power supply 1750 uses, but is not limited to, three separate control units, including control unit 1 1752, control unit 2 1754, and control unit 3 1756, to actively heat the induction coil set. The actual control is a function of the temperature values ​​measured within each of the three zones 1758 (Zone 1), 1760 (Zone 2), and 1762 (Zone 3) of the frying pan, in which case temperature sensors 1770 (Sensor 1), 1772 (Sensor 2), and 1774 (Sensor 3) wirelessly supply and return temperature data to the operating control unit 274 via data streams 1776 (Data 1), 1778 (Data 2), and 1780 (Data 3), which in turn directs power supply 1750 to independently control separate zone heating control units 1752, 1754, and 1756. The goal is to obtain and reproduce over time the same desired temperature curve as the sensing curve data recorded during a certain frying stage of a human chef preparing a dish.

[0282] Figure 74 illustrates a smart oven and computer control system coupled to an operating control unit 1790 that enables real-time execution of a temperature profile for the oven appliance 1792 based on previously stored sensing (temperature) curves. The operating control unit 1790 controls the oven door (open / close), tracks the temperature profile given to the oven by the sensing curves, and can self-clean after cooking. The temperature and humidity inside the oven are monitored through a built-in temperature sensor 1794 that generates a data stream 268 (data 1) at various locations, a temperature sensor in the form of a probe inserted into the food to be cooked (meat, poultry, etc.) to monitor the cooking temperature and estimate the degree of doneness, and an additional humidity sensor 1796 that creates a data stream. The operating control unit 1790 takes in all this sensing data and adjusts the oven parameters to enable proper tracking of the sensing curves described in a previously stored and downloaded set of sensing curves for both (dependent) variables.

[0283] Figure 75 illustrates the configuration 1798 of a computer-controlled ignition and control system for a control unit that modulates power 1858 to a charcoal grill so as to properly track sensing curves related to one or more temperature and humidity sensors internally distributed inside the charcoal grill. The heat control unit 1800 starts the grill using an electronic control signal 1802 and adjusts the grill surface distance to the charcoal and the injection of water mist 1808 over the charcoal 1810 using signals 1804 and 1806. The control unit 1800 bases its output signals 1804 and 1806 on a data stream set 1814 (five shown in this figure) relating to humidity measurements 1816, 1818, 1820, 1822, and 1824 from a set of humidity sensors (1 through 5) 1826, 1828, 1830, 1832, and 1834 distributed inside the charcoal grill, and a data stream 1836 relating to temperature measurements 1840, 1842, 1844, 1846, and 1846 from distributed temperature sensors (1 through 5) 1848, 1850, 1852, 1854, and 1856.

[0284] Figure 76 illustrates a computer-controlled faucet 1860 that allows a computer to control the flow rate, temperature, and pressure of water supplied to a sink (or cooking utensil) by the faucet. The faucet is controlled by a control unit 1 that receives separate data streams 1862 (data 1), 1864 (data 2), and 1866 (data 3) corresponding to a water flow sensor 1868 that supplies sensing data 1, a temperature sensor 1870 that supplies sensing data 2, and a water pressure sensor 1872 that supplies sensing data 3. 3 Controlled by 62. In this case, control unit 1 3 62 controls the chilled water supply 1874 with appropriate chilled water temperature and chilled water pressure, which are digitally displayed on the display 1876, in order to obtain the desired pressure, flow rate, and temperature of the water present at the tap, and controls the hot water supply 1878 with appropriate hot water temperature and hot water pressure, which are digitally displayed on the display 1880.

[0285] Figure 77 shows a top view of an embodiment 1882 of a fully instrumented robotic kitchen. This standardized robotic kitchen is divided into three levels, namely the upper level, the counter level, and the lower level, each containing equipment and fixtures, each integrated with sensors 1884 and a computer control unit 1886.

[0286] The upper level includes multiple storage-type modules having various units for performing specific kitchen functions using built-in fixtures and equipment. In the simplest level, the shelf / storage storage area 82 includes a storage space area 1320 used for storing and accessing cooking tools and utensils and other cooking and serving equipment (cooking, baking, plating, etc.), a storage ripening storage space area 1298 for specific food items (e.g., fruits and vegetables), a chilled storage zone 88 for items such as lettuce and onions, a frozen storage space area 1302 for deep-frozen items, and another storage pantry zone 1304 for other food items and rarely used spices, etc. Each module inside the upper level includes a sensor unit 1884 that supplies data either directly to one or more control units 1886 or through one or more central or distributed control computers that enable computer-controlled operation.

[0287] The counter level not only houses the monitoring sensor 1884 and the control unit 1886, but also includes a serving counter 1306, a counter area 1308 with a sink, another counter area 1310 with a removable work surface (cutting board, etc.), a charcoal-fired grill with slats 1312, and a multi-purpose area 1314 for other cooking appliances including stoves, cookers, steamers, and steaming pots. Each module inside the counter level includes a sensor unit 1884 that supplies data either directly to one or more control units 1886, or through one or more central or distributed control computers that enable computer-controlled operation.

[0288] The lower level houses a combination of a convection oven and microwave oven, as well as a steamer, steaming pot, and grill 1316, a dishwasher 1318, a hard-automated food dispenser 82, and a larger storage area 1320 for maintaining and storing additional frequently used cooking and oven-baking utensils, as well as tableware, plates, tools (whisks, knives, etc.), and cutlery. Each module inside the lower level includes a sensor unit 1884 that supplies data either directly to one or more control units 1886, or through one or more central or distributed control computers that enable computer-controlled operation.

[0289] Figure 78 shows a perspective view of one embodiment 1890 of a robotic kitchen cooking system, which has three different levels arranged from top to bottom, each equipped with multiple distributed sensor units 1892, and these sensor units 1892 either directly supply data to one or more control units 1894 or process sensing data and subsequently supply data to one or more central computers that instruct one or more control units 376 to function according to the commands of these computers.

[0290] The upper level includes multiple storage-type modules having various units for performing specific kitchen functions using built-in fixtures and equipment. In the simplest level, the shelf / storage pantry space area 1294 includes a storage space area 1296 used for storing and accessing cooking tools and utensils and other cooking and serving equipment (cooking, baking, plating, etc.), a storage aging storage space area 1298 for specific food items (e.g., fruits and vegetables), a chilled storage zone 88 for items such as lettuce and onions, a frozen storage space area 1302 for deep-frozen items, and another storage pantry zone 1294 for other food items and spices that are rarely used. Each module inside the upper level includes a sensor unit 1892 that supplies data either directly to one or more control units 1894 or through one or more central or distributed control computers that enable computer-controlled operation.

[0291] The counter level includes a counter area 1308 having a sink and an electronically controllable faucet, as well as housing a monitoring sensor 1892 and a control unit 1894; another counter area 1310 having a removable work surface for cutting / shredding on a cutting board; a charcoal-fired grill with slats 1312; and a multi-purpose area 1314 for other cooking appliances including a stove, cooker, steamer, and steamer pot. Each module inside the counter level includes a sensor unit 1892 that supplies data either directly to one or more control units 1894, or through one or more central or distributed control computers that enable computer-controlled operation.

[0292] The lower level houses a combination of a convection oven and microwave oven, as well as a steamer, steaming pot, and grill 1316, a dishwasher 1318, a hard-automated food dispenser 82, and a larger storage area 1310 for maintaining and storing additional frequently used cooking and oven-baking utensils, as well as tableware, plates, tools (whisks, knives, etc.), and cutlery. Each module inside the lower level includes a sensor unit 1892 that supplies data either directly to one or more control units 1896, or through one or more central or distributed control computers that enable computer-controlled operation.

[0293] Figure 79 is a flowchart illustrating a second embodiment 1900 of the process of a robot kitchen that prepares a dish from one or more parameter curves previously re...

Claims

1. It is an integrated robotic kitchen system, A robotic kitchen module having a predetermined cooking environment including a three-dimensional workspace area with one or more pre-configured kitchen elements, A robot comprising one or more robot end effectors and one or more robot arms, wherein each robot arm in the one or more robot arms is coupled to each of the one or more robot end effectors, and the one or more robot arms are coupled to the robot kitchen module, At least one processor, A memory for storing multiple executable instructions, wherein when the multiple executable instructions are executed by the at least one processor, the at least one processor... Receiving a first dataset containing electronic recipes for preparing at least a portion of a food dish, wherein the electronic recipes include one or more cooking operations that include parameters relating to at least one time and parameters relating to at least one ingredient, Executing a second dataset, which includes one or more robotic cooking operations corresponding to the electronic recipe, to operate one or more robotic arms and one or more robotic end effectors in a predetermined cooking environment to prepare at least a portion of the food dish with the same or substantially the same fidelity as a predetermined fidelity, wherein the second dataset, which includes one or more robotic cooking operations, corresponds to the first dataset, which includes one or more cooking operations, the robot produces a predetermined functional result as a functional outcome of the cooking operations related to the one or more cooking operations, each robotic cooking operation in the one or more robotic cooking operations has a predetermined duration, and each robotic cooking operation has been tested in a cooking environment that is the same or substantially the same as the predetermined cooking environment to produce a predetermined functional result with the predetermined fidelity, The system generates a three-dimensional model for recording the surrounding three-dimensional workspace during training or teaching of the one or more robotic arms and the one or more robotic end effectors, and the at least one processor compares multimode sensor data during the execution of the one or more robotic cooking operations with the recorded three-dimensional model. To perform this, memory and An integrated robotic kitchen system, including...

2. The integrated robotic kitchen system according to claim 1, wherein the predetermined cooking environment in which the one or more robotic cooking operations are performed is different from the cooking environment in which the one or more robotic cooking operations are tested.

3. The integrated robotic kitchen system according to claim 1, wherein the memory stores additional executable instructions that, when executed by the at least one processor, cause the at least one processor to access one of the one or more pre-configured kitchen elements in the predetermined cooking environment at each predetermined position and each predetermined orientation as defined in the electronic recipe in order to produce the predetermined functional result.

4. The integrated robotic kitchen system according to claim 1, further comprising a smart appliance for performing sequential operations for the one or more robotic cooking operations, or a smart appliance for performing parallel operations for the one or more robotic cooking operations.

5. When the memory is executed by the at least one processor, the at least one processor will The first sequential cooking operation is performed at a first predetermined start time with a first predetermined duration, After the first sequential cooking operation is completed, the second sequential cooking operation is performed at a second predetermined start time with a second predetermined duration, The integrated robotic kitchen system according to claim 4, which stores additional executable instructions for performing the following actions.

6. The integrated robotic kitchen system according to claim 1, wherein the one or more robotic cooking operations include two or more parallel robotic cooking operations, the two or more parallel robotic cooking operations include a first parallel robotic cooking operation directed toward a first portion of the food dish and a second parallel robotic cooking operation directed toward a second portion of the food dish, the one or more robotic arms coupled to the one or more robotic end effectors include a first robotic arm coupled to a first robotic end effector and a second robotic arm coupled to a second robotic end effector, and the memory stores additional executable instructions, when executed by the at least one processor, that cause the at least one processor to control the first robotic arm coupled to the first robotic end effector to perform the first parallel robotic cooking operation directed toward the first portion of the food dish, and control the second robotic arm coupled to the second robotic end effector to perform the second parallel robotic cooking operation directed toward the second portion of the food dish simultaneously with the first parallel robotic cooking operation.

7. The integrated robotic kitchen system according to claim 1, wherein the one or more pre-configured kitchen elements include one or more pre-configured kitchen utensils, one or more pre-configured kitchen tools, one or more kitchen containers, one or more pre-configured kitchen equipment, or any combination thereof, the one or more kitchen utensils include one or more standardized kitchen utensils, the one or more kitchen containers include one or more standardized kitchen containers, the one or more kitchen tools include standardized kitchen tools, the one or more kitchen equipment includes standardized kitchen equipment, and each of the one or more standardized kitchen utensils, the one or more standardized kitchen containers, the one or more standardized kitchen tools, and the one or more standardized kitchen equipment has a pre-configured shape, pre-configured dimensions, pre-configured structure, pre-configured material, or pre-configured capability, or any combination thereof.

8. The integrated robotic kitchen system according to claim 1, wherein the one or more pre-configured kitchen elements include a universal handle for handling the one or more pre-configured kitchen elements, the universal handle enabling the one or more robotic end effectors to hold the universal handle in a single position when handling the one or more kitchen elements.

9. The integrated robotic kitchen system according to claim 1, wherein the position and orientation of one of the one or more kitchen elements within the predetermined cooking environment are predetermined, and the position and orientation are predetermined based on the coordinate system of the integrated robotic kitchen system and referenced to the origin of the coordinate system.

10. The integrated robotic kitchen system according to claim 1, wherein the position and orientation of one of the one or more kitchen elements within the predetermined cooking environment are predetermined, and the position and orientation are predetermined for the one or more robotic arms or for the one or more robotic end effectors.

11. The integrated robotic kitchen system according to claim 1, wherein the electronic recipe includes one or more parameters for one or more robotic cooking operations, and the one or more robotic cooking operations and the one or more parameters are obtained from a kitchen studio using the same or substantially the same cooking environment as the predetermined cooking environment.

12. The integrated robotic kitchen system according to claim 1, wherein the robotic kitchen module is a standalone kitchen module that functions like a kiosk when preparing at least a portion of the food dishes.

13. The integrated robotic kitchen system according to claim 1, wherein the robotic kitchen module includes a rail system coupled to the one or more robotic arms, and the memory stores additional executable instructions for the at least one processor to operate the rail system to move the one or more robotic arms and the one or more robotic end effectors horizontally, or to move the one or more robotic arms and the one or more robotic end effectors vertically, when executed by the at least one processor.

14. The integrated robotic kitchen system according to claim 1, wherein the robotic kitchen module includes a retractable hood for storing the one or more robotic arms and the one or more robotic end effectors when the one or more robotic arms are not in use or when set to manual cooking mode.

15. The robotic kitchen module includes one or more actuators for moving at least one of the one or more robotic arms and the one or more end effectors vertically, horizontally, and rotationally, wherein a telescopic actuator is coupled to the one or more robotic arms, and the one or more actuators include one or more linear actuators and one or more rotary actuators, the integrated robotic kitchen system according to claim 1.

16. The integrated robotic kitchen system according to claim 1, wherein the specifications or characteristics of the components of the predetermined cooking environment are the same as or substantially the same as the specifications or characteristics of the components of the cooking environment in which the one or more robotic cooking operations were tested.

17. The integrated robotic kitchen system according to claim 1, wherein the one or more robotic cooking operations are predefined or precalculated based on predetermined positions and orientations of the one or more kitchen elements within the predetermined cooking environment.

18. The integrated robotic kitchen system according to claim 1, further comprising a plurality of robotic kitchen modules, each of which is modular and can be used independently of the others of the plurality of robotic kitchen modules.

19. It is a robot kitchen system, A robotic kitchen module having a cooking environment including a three-dimensional workspace area having one or more kitchen elements, A robot comprising one or more robot end effectors and one or more robot arms, wherein each robot arm in the one or more robot arms is coupled to each of the one or more robot end effectors, and the one or more robot arms are coupled to the robot kitchen module, To store multiple robot cooking operations, an electronic library is used, At least one processor, A memory for storing executable instructions, wherein when an executable instruction is executed by the at least one processor, the at least one processor... Receiving a first dataset including cooking operations related to electronic recipes, wherein the cooking operations produce functional results, Receiving a robotic cooking operation to be tested, wherein the robotic cooking operation has one or more parameters, Performing the robot cooking operation in order to operate the one or more robot arms and the one or more robot end effectors, Receiving feedback data regarding the results of the robot cooking operation, Based on the aforementioned feedback data, a specific combination of parameters for the robot cooking operation is determined to achieve a predetermined functional outcome with a predetermined fidelity. Determining a predetermined duration for the robotic cooking operation having the aforementioned specific parameter combination, The electronic library stores a second dataset containing the robot cooking operation having the aforementioned specific parameter combination and the predetermined duration, wherein the second dataset containing the robot cooking operation corresponds to the first dataset containing the cooking operation, and the robot produces a predetermined functional result as a functional outcome of the cooking operation. To perform this, memory and A robotic kitchen system, including...

20. The robotic kitchen system according to claim 19, wherein the robotic cooking operation is associated with a descriptor for identifying the robotic cooking operation in the electronic library.

21. The robotic kitchen system further includes one or more sensors, the memory stores additional executable instructions, and when the additional executable instructions are executed by the at least one processor, the processor receives The one or more sensors sense a sequence of observations corresponding to the chef's movements when the chef prepares at least a portion of a food dish in the cooking environment using ingredients and the one or more kitchen elements. Converting the perceived observation sequence into a robotic cooking operation that can be executed by the one or more robotic arms and the one or more robotic end effectors, A robot kitchen system according to claim 19, which performs the following.

22. The robot kitchen system according to claim 19, further comprising one or more sensors, the memory storing additional executable instructions, and the additional executable instructions, when executed by the at least one processor, cause the processor to receive sensor data from the one or more sensors when performing the robot cooking operation as feedback data.

23. The robotic kitchen system according to claim 19, wherein the memory stores additional executable instructions, and when the additional executable instructions are executed by the at least one processor, the processor receives input data as feedback data via a user interface when performing or after performing the one or more robotic cooking operations, the input data including timing, color, smell, temperature, image, humidity, texture, taste, weight loss, or portion size.

24. The memory stores additional executable instructions, and when the additional executable instructions are executed by the at least one processor, the processor receives the following information: To create a new robotic cooking operation having one or more motion primitives, Selecting the maximum possible range of key parameters associated with the new robotic cooking operation, For each of the aforementioned major parameters within its maximum possible range, the values ​​of each major parameter are tested and verified using all possible combinations with other major parameters. Determining whether a specific set of key parameter combinations produces a predetermined functional outcome, If the above determination is true, store a specific set of the above-mentioned combinations of major parameters, A robot kitchen system according to claim 19, which performs the following.

25. The memory stores additional executable instructions, and when the additional executable instructions are executed by the at least one processor, the processor receives the following information: This involves repeatedly testing and verifying a specific set of key parameter combinations, Calculate the number of failed results during repeated testing of a specific set of the aforementioned key parameter combinations, The test results of a specific set of the aforementioned main parameter combinations are stored, Determining the best or optimal set of parameter combinations, For use in the aforementioned electronic library, the best or optimal set of parameter combinations associated with a new robotic cooking operation is stored. A robot kitchen system according to claim 19, which performs the following.

Images