Robot kitchen system, robot food preparation system, method for preparing cooked food, and method for generating small-scale library

Abstract

To provide a method, computer program product, and computer system for a robot device that uses robot instructions to reproduce cooked food with substantially the same result as a case where a chef prepares cooked food.SOLUTION: Primitives are defined by motions / actions of articulated degrees of freedom that range in complexity from simple to complex, and which can be combined in any form in serial / parallel fashion. These motion-primitives are termed to be mini-manipulations and each has a clear time-indexed command input-structure and output behavior / performance profile that is intended to achieve a certain function. The mini-manipulations comprise a new way of creating a general programmable-by-example platform for humanoid robots.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation-in-part of co-pending U.S. patent application Ser. No. 14 / 627,900, filed Feb. 20, 2015, entitled "Methods and Systems for Food Preparation in a Robotic Cooking Kitchen."

[0002] This continuation-in-part application is a continuation-in-part application of U.S. Provisional Patent Application No. 62 / 202,030, filed August 6, 2015, entitled "Robotic Manipulation Methods and Systems Based on Electronic Mini-Manipulation Libraries," U.S. Provisional Patent Application No. 62 / 189,670, filed July 7, 2015, entitled "Robotic Manipulation Methods and Systems Based on Electronic Mini-Manipulation Libraries," U.S. Provisional Patent Application No. 62 / 166,879, filed May 27, 2015, entitled "Robotic Manipulation Methods and Systems Based on Electronic Mini-Manipulation Libraries," U.S. Provisional Patent Application No. 62 / 166,879, filed May 13 ...27, 2015, entitled "Robotic Manipulation Methods and Systems Based on Electronic Mini-Manipulation Libraries," U.S. Provisional Patent Application No. 62 / 166,879, filed May 13, 2015, entitled "Robotic Manipulation Methods and Systems Based on Electronic Mini-Manipulation Libraries," U.S. Provisional Patent Application No. 62 / 166,879 U.S. Provisional Patent Application No. 62 / 161,125, entitled "Robotic Manipulation Methods and Systems Based on Electronic Mini-Manipulation Libraries," filed April 12, 2015; U.S. Provisional Patent Application No. 62 / 146,367, entitled "Robotic Manipulation Methods and Systems Based on Electronic Mini-Manipulation Libraries," filed February 16, 2015; U.S. Provisional Patent Application No. 62 / 116, entitled "Method and System for Food Preparation in a Robotic Cooking Kitchen," filed February 16, 2015;No. 563, U.S. Provisional Patent Application No. 62 / 113,516, filed February 8, 2015, entitled "Method and System for Food Preparation in a Robotic Cooking Kitchen," U.S. Provisional Patent Application No. 62 / 109,051, filed January 28, 2015, entitled "Method and System for Food Preparation in a Robotic Cooking Kitchen," U.S. Provisional Patent Application No. 62 / 104,680, filed January 16, 2015, entitled "Method and System for Robotic Cooking Kitchen," U.S. Provisional Patent Application No. 62 / 104,680, filed December 10, 2014, entitled "Method and System for Robotic Cooking Kitchen," U.S. Provisional Patent Application No. 62 / 113,516, filed February 8, 2015, entitled "Method and System for Food Preparation in a Robotic Cooking Kitchen," U.S. Provisional Patent Application No. 62 / 113,516, filed January 8, 2015, entitled "Method and System for Food Preparation in a Robotic Cooking Kitchen," U.S. Provisional Patent Application No. 62 / 104,680, filed December 10, 2014, entitled "Method and System for ... No. 62 / 090,310 entitled "Method and System for Robotic Cooking Kitchen," filed on November 22, 2014; U.S. Provisional Patent Application No. 62 / 083,195 entitled "Method and System for Robotic Cooking Kitchen," filed on October 31, 2014; U.S. Provisional Patent Application No. 62 / 073,846 entitled "Method and System for Robotic Cooking Kitchen," filed on September 26, 2014; U.S. Provisional Patent Application No. 62 / 055,799 entitled "Method and System for Robotic Cooking Kitchen," filed on September 2, 2014; This application claims priority to U.S. Provisional Patent Application No. 62 / 044,677, entitled "Kitchen."

[0003] U.S. Patent Application No. 14 / 627,900 is a subsidiary of U.S. Provisional Patent Application No. 62 / 116,563, filed February 16, 2015, entitled "Method and System for Food Preparation in a Robotic Cooking Kitchen," U.S. Provisional Patent Application No. 62 / 113,516, filed February 8, 2015, entitled "Method and System for Food Preparation in a Robotic Cooking Kitchen," U.S. Provisional Patent Application No. 62 / 109,051, filed January 28, 2015, entitled "Method and System for Food Preparation in a Robotic Cooking Kitchen," U.S. Provisional Patent Application No. 62 / 109,051, filed January 16, 2015, entitled "Method and System for Food Preparation in a Robotic Cooking Kitchen," and U.S. Provisional Patent Application No. 62 / 109,051, filed January 28, 2015, entitled "Method and System for Food Preparation in a Robotic Cooking Kitchen." U.S. Provisional Patent Application No. 62 / 104,680 entitled "Method and System for Robotic Cooking Kitchen," filed December 10, 2014; U.S. Provisional Patent Application No. 62 / 090,310 entitled "Method and System for Robotic Cooking Kitchen," filed November 22, 2014; U.S. Provisional Patent Application No. 62 / 083,195 entitled "Method and System for Robotic Cooking Kitchen," filed October 31, 2014; U.S. Provisional Patent Application No. 62 / 073 entitled "Method and System for Robotic Cooking Kitchen," filed October 31, 2014;No. 846, U.S. Provisional Patent Application No. 62 / 055,799, entitled "Method and System for Robotic Cooking Kitchen," filed September 26, 2014; U.S. Provisional Patent Application No. 62 / 044,677, entitled "Method and System for Robotic Cooking Kitchen," filed September 2, 2014; U.S. Provisional Patent Application No. 62 / 024,948, entitled "Method and System for Robotic Cooking Kitchen," filed July 15, 2014; U.S. Provisional Patent Application No. 62 / 013,691, entitled "Method and System for Robotic Cooking Kitchen," filed June 18, 2014; U.S. Provisional Patent Application No. 62 / 013,691, entitled "Method and System for Robotic Cooking Kitchen," filed June 17, 2014; U.S. Provisional Patent Application No. 62 / 013,502 entitled "Method and System for Robotic Cooking Kitchen," filed June 17, 2014; U.S. Provisional Patent Application No. 62 / 013,190 entitled "Method and System for Robotic Cooking Kitchen," filed May 8, 2014; U.S. Provisional Patent Application No. 61 / 990,431 entitled "Method and System for Robotic Cooking Kitchen," filed May 1, 2014; U.S. Provisional Patent Application No. 61 / 987 entitled "Method and System for Robotic Cooking Kitchen," filed May 1, 2014;This application claims priority to U.S. Provisional Patent Application No. 61 / 953,930, entitled "Method and System for Robotic Cooking Kitchen," filed March 16, 2014, and U.S. Provisional Patent Application No. 61 / 942,559, entitled "Method and System for Robotic Cooking Kitchen," filed February 20, 2014.

[0004] The subject matter of all of the foregoing disclosures is incorporated herein by reference in its entirety.

[0005] The present disclosure relates generally to the interdisciplinary fields of robotics and artificial intelligence (AI), and more specifically to computer-based robotic systems that use electronic libraries of miniature operations with transformative robotic instructions to replicate movements, processes, and techniques with real-time electronic adjustments. [Background technology]

[0006] Research and development in robotics has been ongoing for decades, but advances have been largely in heavy industrial applications such as automotive manufacturing automation or military applications. While simple robotic systems have been designed for the consumer market, widespread application in the home consumer robotics space has not yet been seen. With technological advancements coupled with a high-income population, this market is considered ripe for technological advancements to create opportunities for improving people's lives. Robotics has continued to improve automation techniques through advanced artificial intelligence and many forms of mimicking human skills and tasks to operate robotic devices or humanoid robots. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] U.S. Non-Provisional Patent Application No. 14 / 627,900 [Non-patent literature]

[0008] [Non-Patent Document 1] Garagnani, M. (1999) "Improving the Efficiency of Processed Domain-axioms Planning", PLANSIG-99 Proceedings, Manchester, UK, pp. 190-192. [Non-patent document 2] Leake (1996 book) "Case-Based Reasoning: Experiences, Lessons and Future Directions," http: / / journals.cambridge.org / action / displayAbstract?fromPage=online&aid=4068324&fileId=S0269888900006585dl.acm.org / citation.cfm?id=524680 [Non-patent document 3] Carbonell (1983) "Learning by Analogy: Formulating and Generalizing Plans from Past Experience," http: / / link.springer.com / chapter / 10.1007 / 978-3-662-12405-5_5 [Non-patent document 4] Kamakura, Noriko, Michiko Matsuo, Harumi Ishii, Fumiko Mitsuboshi, and Yoriko Miura, "Patterns of static prehension in normal hands," American Journal of Occupational Therapy 34, No. 7 (1980): pp. 437-445 [Non-patent document 5] A. Kapandji, The Physiology of the Joints, Volume 1: Upper Limb, 6e, Churchill Livingstone, 6th edition, 2007 Summary of the Invention [Problem to be solved by the invention]

[0009] Since robots were first developed in the 1970s, the concept of robots replacing humans in certain fields and performing tasks typically performed by humans has been a continuously evolving concept. Manufacturing sectors have long used robots in a teach-and-play mode, where robots are taught which movements to continuously replicate without modification or deviation through the generation and downloading of resting or offline fixed trajectories. Companies have also incorporated pre-programmed trajectory execution of computer-taught trajectories and robot movement playback in application domains such as beverage mixing, welding, or automotive painting. However, all of these traditional applications have focused solely on having the robot faithfully execute movement commands, using a 1:1 computer-to-robot or teach-and-play principle, where the robot typically follows a taught / pre-calculated trajectory without deviation. [Means for solving the problem]

[0010] Disclosed embodiments of the present invention relate to methods, computer program products, and computer systems for robotic devices that use robotic instructions to replicate a food dish with substantially the same results as if a chef had prepared the food dish. In a first embodiment, a robotic device in a standardized robotic kitchen includes two robotic arms and hands that replicate the exact movements of a chef in the same sequence (or substantially the same sequence). The two robotic arms and hands replicate the movements to prepare the food dish with the same timing (or substantially the same timing) based on a software file (recipe script) that previously recorded the exact movements of the chef in preparing the same food dish. In a second embodiment, a computer-controlled cooking appliance prepares the same food dish based on sensory curves, such as temperature over time, recorded in a software file from previous chef-prepared food dishes using a cooking appliance with sensors, where a computer records sensor values ​​over time as the chef prepares the food dish on the sensor-equipped cooking appliance. In a third embodiment, a kitchen appliance includes the robotic arm of the first embodiment and the cooking appliance with sensors of the second embodiment to prepare a dish, whereby both the robotic arm and one or more sensing curves are combined, and the robotic arm can perform quality checks on characteristics such as taste, smell, and appearance during the cooking process, allowing any cooking adjustments to the food dish preparation stage. In a fourth embodiment, the kitchen appliance includes a food storage system using computer-controlled containers and container identifiers to store and supply ingredients for a user to prepare a food dish by following a chef's cooking instructions. In a fifth embodiment, a robotic cooking kitchen includes a robot with an arm and kitchen appliances, where the robot moves around the kitchen appliance and prepares a food dish by mimicking the chef's detailed cooking movements, including possible real-time modifications / adaptations to the preparation process defined in a recipe script.

[0011] The robotic cooking engine includes the processing to detect, record, chef-imitate cooking movements, control critical parameters such as temperature and time, and execute using specified appliances, equipment, and tools, thereby replicating a gourmet meal with the same taste as the same dish prepared by the chef and served at a specific preferred time. In one embodiment, the robotic cooking engine provides a robotic arm to replicate the chef's identical movements using the same ingredients and techniques to produce the identical tasting dish.

[0012] The underlying motivation for the present disclosure revolves around the ability to monitor humans using sensors while they naturally perform their activities, and then use monitoring sensors, capturing sensors, computers, and software to generate information and instructions to replicate the human activity using one or more robotic and / or automated systems. While multiple such activities (e.g., cooking, drawing, playing a musical instrument, etc.) can be envisioned, one aspect of the present disclosure relates to preparing food, essentially a robotic food preparation application. Monitoring a human chef within an instrumented, application-specific environment (in this case, a standardized robotic kitchen) involves using sensors and computers to observe, monitor, record, and interpret the movements and actions of the human chef to develop a robot-executable instruction set that is robust to variations and changes in the environment that can enable the robotic or automated system in the robotic kitchen to prepare dishes that are identical in terms of standard and quality to those prepared by the human chef.

[0013] The use of multimodal sensing systems is a means of collecting the necessary raw data. Sensors that can collect and provide such data include environmental and geometric sensors, such as 2D sensors (e.g., cameras) and 3D sensors (e.g., lasers, sonar), human motion capture systems (e.g., human-worn camera-targets, instrumented suits / exoskeletons, instrumented gloves), and instrumented and powered equipment (sensors) and actuators (e.g., instrumented appliances, cooking equipment, tools, ingredient dispensers) used during recipe creation and execution. All this data is collected by one or more distributed / central computers and processed by various software processes. Algorithms will process and abstract this data to the point where the human and the computer-controlled robotic kitchen can understand the activities, tasks, motions, equipment, ingredients, methods, and processes employed by the human, including replicating the key skills of a particular chef. The raw data is processed by one or more software abstraction engines to create a recipe script that is human-readable, and through further processing, machine-understandable and machine-executable, specifying all the actions and movements for all the steps of a particular recipe that the robotic kitchen needs to execute. These instructions range in complexity from controlling individual joints in relation to a particular step in a recipe, to specific joint movement profiles over time, all the way down to the abstract command level with lower-level movement execution instructions embedded within. Abstract motion commands (e.g., "crack an egg into a pan," "fry the surface until golden brown on both sides," etc.) are generated from raw data and can be refined and optimized through multiple iterative learning processes performed in real time and / or offline, thereby enabling the robotic kitchen system to successfully deal with measurement uncertainties, variations in ingredients, etc., and enabling complex (adaptive) small-scale manipulation movements using fingered hands and wrists attached to robotic arms based on highly abstract / high-level commands (e.g., "grab the pot by the handle," "pour out the contents," "grab the spoon from the counter and stir the soup," etc.).

[0014] The ability to create machine-executable instruction sequences, now contained within digital files, can be shared / transmitted, thereby enabling any robotic kitchen to execute these sequences, opening up the option to perform food preparation anywhere, at any time. This, in turn, gives the option to buy / sell recipes online, allowing users to access and distribute recipes on a pay-per-use or subscription basis.

[0015] The reproduction of a human-prepared dish is performed by a robotic kitchen that is essentially a standardized replica of the instrumented kitchen used by a human chef during the creation of the dish, except that the human actions are now performed by a set of robotic arms and hands, computer-monitored and computer-controllable appliances, equipment, tools, dispensers, etc. Thus, dish reproduction fidelity is closely tied to the extent to which the robotic kitchen is a replica of the kitchen (and all its elements and ingredients) observed during the human chef's preparation of the dish.

[0016] Broadly stated, a humanoid robot having a robotic computer controller operated with robot instructions by a Robot Operating System (ROS) includes a database having a plurality of electronic miniature manipulation libraries, each containing a plurality of miniature manipulation elements, the plurality of electronic miniature manipulation libraries being combinable to generate one or more machine-executable application-specific instruction sets, the plurality of miniature manipulation elements in the electronic miniature manipulation libraries being combinable to generate one or more machine-executable application-specific instruction sets, a robotic structure having an upper body and a lower body connected to a head through an articulated neck, the upper body including a torso, shoulders, arms, and hands, and a control system communicatively coupled to the database, a sensing system, a sensor data interpretation system, a motion planner, and actuators and associated controllers, for executing the application-specific instruction sets to operate the robotic structure.

[0017] Additionally, embodiments of the present disclosure relate to robotic device methods, computer program products, and computer systems for executing robotic instructions from one or more mini-manipulation libraries. Two types of parameters affect the operation of a mini-manipulation: element parameters and application parameters. During the creation phase of a mini-manipulation, element parameters provide variables for testing different combinations, parameters, and degrees of freedom to create a successful mini-manipulation. During the execution phase of a mini-manipulation, application parameters are programmable or customizable to adapt one or more mini-manipulation libraries to specific applications, such as food preparation, making sushi, playing the piano, drawing, picking out books, and other types of applications.

[0018] Mini-manipulation involves a new approach to creating a generic, example-programmable platform for humanoid robots. Current technology largely requires explicit development of control software by specialized programmers for each and every step of a robot's motion or motion sequence. An exception to the above concerns highly repetitive, low-level tasks, such as factory assembly, where the fundamental principle of learning by imitation exists. Mini-manipulation libraries provide a large set of higher-level sense-and-act sequences that are common building blocks for complex tasks, such as cooking, caring for the infirm, or other tasks to be performed by next-generation humanoid robots. More specifically, unlike the prior art, the present disclosure provides the following unique features: First, a potentially very large library of predefined / pre-trained sense-and-act sequences, called mini-manipulations. Second, encoding the pre-conditions required for the sense-and-act sequences so that each mini-manipulation successfully produces the desired functional outcome (i.e., post-condition) with a clearly defined probability of success (e.g., 100% or 97%, depending on the complexity and difficulty of the mini-manipulation). Third, each mini-operation references a set of variables whose values ​​can be set a priori or by sensing operation before performing that operation. Fourth, each mini-operation modifies the values ​​of a set of variables that represent the functional outcome (post-condition) of performing that sequence of operations. Fifth, mini-operations can be derived from repeated observation by a human instructor (e.g., a master chef) to determine the sensing and acting sequence and the range of acceptable values ​​for the variables. Sixth, mini-operations can be composed into larger units that perform complete tasks such as preparing a meal or cleaning a room. These larger units are multi-stage applications of mini-operations in either strict sequence, parallel, or observing a partial order in which some steps must occur before others, but are not in a fully ordered sequence (e.g., to prepare a given dish, three ingredients must be combined and mixed in precise amounts in a mixing bowl; the order in which each ingredient is placed in the bowl is not limited, but all must be placed before the mixing).Seventh, the assembly of mini-manipulations into full-scale tasks is performed by a robot plan that takes into account the preconditions and postconditions of the component mini-manipulations. Eighth, case-based reasoning can use observations of past experience of humans performing full-scale tasks, or other robots performing them, or the same robot to derive a library of reusable robot plans to generate examples (particular instances performing full-scale tasks) that are both successful to replicate and unsuccessful to learn what to avoid.

[0019] In a first aspect of the present disclosure, a robotic device performs a task by replicating the operation of a human skill, such as food preparation, piano playing, or drawing, by accessing one or more mini-manipulation libraries. The replicating process of the robotic device simulates the transfer of a human intelligence or skill set using a pair of hands, such as how a chef uses a pair of hands to prepare a particular dish, or how a virtuoso piano player uses a pair of hands (and possibly also foot and body movements) to play a piano masterpiece. In a second aspect of the present disclosure, a robotic device provides a programmable or customizable humanoid robot for domestic use designed to provide a psychologically, emotionally, and / or functionally comfortable robot, thereby providing enjoyment to the user. In a third aspect of the present disclosure, one or more mini-manipulation libraries are created and implemented first as one or more basic mini-manipulation libraries and second as one or more application-specific mini-manipulation libraries. The one or more basic mini-manipulation libraries are created based on the element parameters and degrees of freedom of the humanoid robot or robotic device. The humanoid robot or robotic device is programmable such that one or more basic mini-manipulation libraries can be programmed or customized to become one or more application-specific mini-manipulation libraries specifically tailored to the user's needs in the operational capabilities of the humanoid robot or robotic device.

[0020] Some embodiments of the present disclosure relate to technical features related to the ability to create complex humanoid robot locomotion, motion, and interaction with tools and the environment by automatically constructing locomotion for a humanoid robot with movements and behaviors based on a set of computer-coded robotic locomotion and movement primitives. The primitives are defined by multi-degree-of-freedom movements / actions that range in complexity from simple to complex and can be combined in any number of serial / parallel configurations. These movement primitives are called Miniature Manipulations (MMs), and each MM has a well-defined time-indexed command input structure and output behavior / outcome profile designed to achieve a certain function. MMs can range from the simple (“rotate a single knuckle one degree”) to the more intricate (“grasp a tool”), such as “pick up a tool and cut bread”), to the quite abstract (“play the first bar of Schubert's Piano Concerto No. 1”).

[0021] Thus, MMs are software-based and are represented by discrete program-like input and output data sets, underlying processing algorithms, and outcome descriptors, with input / output data files and subroutines contained within discrete runtime source code that, when compiled, produces object code that can be aggregated and collected into a variety of different software libraries called collections of mini-manipulation libraries (MMLs). MMLs can be categorized into groups based on whether they are associated with (i) specific hardware elements (fingers / hands, wrists, arms, torsos, feet, legs, etc.), (ii) behavioral elements (touching, grasping, handling, etc.), or even (iii) application domains (cooking, drawing, playing musical instruments, etc.). Furthermore, within each of these groups, MMLs can be arranged based on levels (from simple to complex) related to the complexity of the desired behavior.

[0022] Therefore, it should be understood that the design concept of Miniature Manipulation (MM) (defining and relating, measured variables, controlled variables, and combinations thereof, and using and modifying values, etc.) and implementing MM through the use of multiple MMLs in nearly infinite combinations relates to the definition and control of basic behaviors (movements and interactions) at levels ranging from a single joint (such as a finger joint) to a combination of joints (such as a finger and hand, arm), up to one or more degrees of freedom (joints that move under the control of an actuator) to even higher degree of freedom systems (such as a torso, upper body) in sequences and combinations to obtain desired and successful movement sequences in free space so that the robotic system can perform a desired function or output on and with the surrounding world using tools, implements, and other items, and to obtain a desired degree of interaction with the real world.

[0023] Examples of the above definition can range from (i) a simple command sequence for a finger to flick a marble along a table, to (ii) stirring a liquid in a pot with a utensil, and (iii) playing a musical piece on a musical instrument (violin, piano, harp, etc.). The basic concept is that MM is represented at multiple levels by sets of MM commands executed in sequence and in parallel at successive times, working together to create movements and actions / interactions with the outside world that arrive at a desired function (mixing liquid, bowing a violin, etc.) to obtain a desired result (cooking pasta sauce, playing a Bach concerto, etc.).

[0024] The basic elements of any MM sequence, from low to high level, include movements for each subsystem, the combination of which is described as a set of commanded positions / velocities and forces / torques executed by one or more joints that articulate in the sequence as needed under actuator power. Fidelity of each articulated joint execution is ensured through closed-loop behavior described within each MM sequence and enforced by local and global control algorithms inherent in each joint controller and higher level behavior controllers.

[0025] The implementation of the above movements (described by joint positions and velocities) and environment interactions (described by joint / joint torques and forces) is achieved by having a computer play the desired values ​​for all required variables (positions / velocities and forces / torques) and feeding these values ​​to a controller system which faithfully implements these values ​​on each joint as a function of time at each time step. These variables and their sequences and feedback loops (hence not only data files but also controlgrams) are all described in data files which are combined into multi-level MML to ensure fidelity of the commanded movements / interactions; these data files can be accessed and combined in multiple ways to enable the humanoid robot to perform multiple actions such as cooking a meal, playing classical music on the piano, lifting an infirm person in and out of bed, etc. There are MMLs that describe simple elementary movements / interactions, and these MMLs are used as building blocks for higher level MMLs that describe everything from higher level small scale operations like "grasp", "lift", "cut" etc. to higher level primitives like "stir liquid in a pot" / "pluck a harp string in g flat" or even higher level actions like "make a vinaigrette" / "paint a summer scene in Brittany" / "play Bach's Piano Concerto No. 1" etc. The higher level instructions are simply the combination of serial / parallel sequences of lower and mid level MM primitives executed along a common timed stepped sequence overseen by a set of planners that execute sequence / path / interaction profiles, in combination with feedback controllers to ensure the required execution fidelity (defined in the output data contained within each MM sequence).

[0026] The values ​​for the desired positions / velocities and forces / torques and the playback sequences can be obtained in several ways. One possible approach is by observing and extracting the actions and movements of a human performing the same task, extracting the necessary variables and their values ​​as a function of time from the observation data (video, sensors, modeling software, etc.), and associating these variables and values ​​with different small-scale operations at different levels by using specialized software algorithms to extract the required MM data (variables, sequences, etc.) into various types of MMLs from low to high levels. This approach will allow a computer program to automatically generate the MMLs and automatically define all the sequences and associations without any human intervention.

[0027] Another approach would be to learn from online data (videos, photographs, sound recordings, etc.) how to construct the required sequences of operable sequences using existing low-level MML (again using an automated computer-controlled process using specialized algorithms) to construct the correct sequences and combinations to generate task-specific MML.

[0028] Yet another approach, although arguably less efficient (time wise) and less cost effective, would be for a human programmer to assemble a set of low level MM primitives to generate even higher level sets of actions / sequences within the higher level MML, again to obtain more complex task sequences composed of existing low level MML.

[0029] Modifications and improvements to individual variables (i.e., joint positions / velocities and torques / forces at each piecewise time interval, along with their associated gains and combination algorithms) and movement / interaction sequences are also possible and can be made in many different ways. To achieve higher levels of execution fidelity at various levels of MML ranging from low to high, the learning algorithm can monitor each and every movement / interaction sequence and perform simple variable perturbations to confirm the results and determine whether, how, when, and which variables and sequences should be modified. Such a process can be fully automatic, allowing updated datasets to be exchanged across multiple interconnected platforms, thereby enabling massively parallel cloud-based learning via cloud computing.

[0030] Advantageously, the robotic devices in the standardized robotic kitchen have the ability to prepare a wide variety of meal styles from around the world through global network and database access, compared to a chef who may specialize in one type of meal style. The standardized robotic kitchen can also capture and record the food dish for reproduction by the robotic devices at the time the food dish is desired to be enjoyed, without the laborious and repetitive process of repeatedly preparing the same dish.

[0031] The structures and methods of the present disclosure are disclosed in detail in the following description. This summary is not intended to define the present disclosure. The present disclosure is defined by the claims. These and other embodiments, features, aspects, and advantages of the present disclosure will become more clearly understood by reviewing the following description, the appended claims, and the accompanying drawings.

[0032] The present disclosure will now be described with respect to specific embodiments thereof and will refer to the following drawings. [Brief explanation of the drawings]

[0033] [Figure 1] FIG. 1 is a system diagram illustrating an overall robotic food preparation kitchen using hardware and software in accordance with the present invention. [Figure 2] FIG. 1 is a system diagram illustrating a first embodiment of a food robotic cooking system including a chef studio system and a home robotic kitchen system in accordance with the present invention. [Figure 3] FIG. 1 is a system diagram illustrating one embodiment of a standardized robotic kitchen for preparing dishes by replicating a chef's recipe processes, techniques, and movements in accordance with the present invention. [Figure 4] FIG. 1 is a system diagram illustrating one embodiment of a robotic food preparation engine for use with a computer in a chef studio system and home robotic kitchen system in accordance with the present invention. [Figure 5A] FIG. 1 is a block diagram illustrating a chef studio recipe creation process in accordance with the present invention. [Figure 5B] FIG. 1 is a block diagram illustrating one embodiment of the standardized teach / play robotic kitchen in accordance with the present invention. [Figure 5C] FIG. 1 is a block diagram illustrating one embodiment of a recipe-script generation abstraction engine in accordance with the present invention. [Figure 5D] FIG. 10 is a block diagram illustrating software elements for object manipulation in the standardized robotic kitchen in accordance with the present invention. [Figure 6] FIG. 1 is a block diagram illustrating a multi-modal sensing and software engine architecture in accordance with the present invention. [Figure 7A] FIG. 10 is a block diagram illustrating the standardized robotic kitchen module used by a chef in accordance with the present invention. [Figure 7B] FIG. 10 is a block diagram illustrating an example standardized robotic kitchen module with a pair of robotic arms and hands in accordance with the present invention. [Figure 7C]FIG. 10 is a block diagram illustrating one embodiment of the physical layout of the standardized robotic kitchen module used by a chef in accordance with the present invention. [Figure 7D] FIG. 1 is a block diagram illustrating one embodiment of the physical layout of a standardized robotic kitchen module used by a pair of robotic arms and hands in accordance with the present invention. [Figure 7E] FIG. 10 is a block diagram depicting the step-by-step flow and method to ensure that control or verification points exist during the recipe replication process based on a recipe-script when executed by the standardized robotic kitchen in accordance with the present invention. [Figure 7F] FIG. 10 is a block diagram of cloud-based recipe software for facilitating interaction between the chef studio, the robotic kitchen, and other sources. [Figure 8A] FIG. 10 is a block diagram illustrating one embodiment of a transformation algorithm module between chef movements and robotic mimicking movements in accordance with the present invention. [Figure 8B] FIG. 1 is a block diagram illustrating a pair of sensor-equipped gloves worn by a chef to capture and transmit the chef's movements. [Figure 8C] FIG. 10 is a block diagram illustrating robotic cooking execution based on sensory data captured from a chef's glove in accordance with the present invention. [Figure 8D] FIG. 1 is a graph illustrating dynamic stability and instability curves for equilibrium. [Figure 8E] FIG. 1 is a sequence diagram illustrating a food preparation process that requires a sequence of steps, referred to as stages, in accordance with the present disclosure. [Figure 8F] FIG. 10 is a graph illustrating the overall probability of success as a function of the number of stages for preparing a food dish in accordance with the present invention. [Figure 8G] FIG. 10 is a block diagram illustrating recipe execution in multi-stage robotic food preparation using mini-manipulation and action primitives. [Figure 9A]FIG. 1 is a block diagram illustrating an example of a robotic hand and wrist with tactile vibration sensors, sonar sensors, and camera sensors for detecting and moving kitchen tools, objects, or kitchen equipment in accordance with the present invention. [Figure 9B] FIG. 10 is a block diagram illustrating a pan-tilt head with a sensor camera coupled to a pair of robotic arms and hands for operation in the standardized robotic kitchen in accordance with the present invention. [Figure 9C] FIG. 10 is a block diagram illustrating a sensor camera on a robot wrist for operation in the standardized robotic kitchen in accordance with the present invention. [Figure 9D] FIG. 10 is a block diagram illustrating an in-hand monocular camera on a robotic hand for operation in the standardized robotic kitchen in accordance with the present invention. [Figure 9E] FIG. 1 is a pictorial diagram illustrating an embodiment of a deformable palm in a robotic hand in accordance with the present invention. [Figure 9F] FIG. 1 is a pictorial diagram illustrating an embodiment of a deformable palm in a robotic hand in accordance with the present invention. [Figure 9G] FIG. 1 is a pictorial diagram illustrating an embodiment of a deformable palm in a robotic hand in accordance with the present invention. [Figure 9H] FIG. 1 is a pictorial diagram illustrating an embodiment of a deformable palm in a robotic hand in accordance with the present invention. [Figure 9I] FIG. 1 is a pictorial diagram illustrating an embodiment of a deformable palm in a robotic hand in accordance with the present invention. [Figure 10A] FIG. 10 is a block diagram illustrating an example of a chef recording device worn by a chef in a robotic kitchen environment to record and capture the chef's movements during the food preparation process for a particular recipe. [Figure 10B] 1 is a flow diagram illustrating one embodiment of a process for evaluating a chef's captured movements with robotic poses, movements, and forces in accordance with the present invention. [Figure 11] FIG. 10 is a block diagram illustrating a side view of a robotic arm embodiment for use in a home robotic kitchen system in accordance with the present invention. [Figure 12A] FIG. 1 is a block diagram illustrating one embodiment of a kitchen handle for use with a robotic hand having a palm in accordance with the present invention. [Figure 12B] FIG. 1 is a block diagram illustrating one embodiment of a kitchen handle for use with a robotic hand having a palm in accordance with the present invention. [Figure 12C] FIG. 1 is a block diagram illustrating one embodiment of a kitchen handle for use with a robotic hand having a palm in accordance with the present invention. [Figure 13] FIG. 1 is a pictorial diagram illustrating an exemplary robotic hand having tactile sensors and distributed pressure sensors in accordance with the present invention. [Figure 14] 1 is a pictorial diagram illustrating an example of a sensing outfit for a chef to wear in a robotic cooking studio in accordance with the present invention; [Figure 15A] 1 is a pictorial diagram illustrating one embodiment of a sensored three-fingered tactile glove and an exemplary sensored three-fingered robotic hand for chef food preparation in accordance with the present invention; [Figure 15B] 1 is a pictorial diagram illustrating one embodiment of a sensored three-fingered tactile glove and an exemplary sensored three-fingered robotic hand for chef food preparation in accordance with the present invention; [Figure 15C] FIG. 1 is a block diagram illustrating an example of the inter-functions and interactions between a robotic arm and a robotic hand in accordance with the present invention. [Figure 15D] FIG. 10 is a block diagram illustrating a robotic hand using a standardized kitchen handle that can be attached to a cookware head and a robotic arm that can be attached to a kitchen utensil in accordance with the present invention. [Figure 16] FIG. 10 is a block diagram illustrating a mini-manipulation database library creation module and a mini-manipulation database library execution module in accordance with the present invention. [Figure 17A] FIG. 10 is a block diagram illustrating a sensory glove used by a chef in performing standardized actuation movements in accordance with the present invention. [Figure 17B]FIG. 10 is a block diagram illustrating a database of standardized actuation moves in a robotic kitchen module in accordance with the present invention. [Figure 18A] FIG. 10 is a graph illustrating each of the robotic hands being covered with an artificial human-like soft-skin glove in accordance with the present invention. [Figure 18B] FIG. 1 is a block diagram illustrating a robotic hand covered with an artificial human-like skin glove for performing high-level mini-manipulations based on a library database having predefined and internally stored mini-manipulations in accordance with the present invention. [Figure 18C] FIG. 1 is a graphical diagram illustrating three types of classification of manipulation actions for food preparation according to the present invention. [Figure 18D] 1 is a flow chart illustrating one embodiment of a classification of control actions for food preparation according to the present invention. [Figure 19] FIG. 10 is a block diagram illustrating the creation of a mini-manipulation that results in cracking an egg with a knife in accordance with the present invention. [Figure 20] FIG. 1 is a block diagram illustrating an example of a recipe execution for a mini-operation with real-time adjustment in accordance with the present invention. [Figure 21] 1 is a flow diagram illustrating a software process for capturing chef food preparation movements within a standardized kitchen module in accordance with the present invention. [Figure 22] 1 is a flow diagram illustrating a software process for robotic food preparation in a robotic standardized kitchen module in accordance with the present invention. [Figure 23] 1 is a flow diagram illustrating one embodiment of a software process for creating, testing, validating, and storing various parameter combinations for a mini-manipulation system in accordance with the present invention. [Figure 24] 1 is a flow diagram illustrating one embodiment of a software process for creating tasks for a miniature manipulation system in accordance with the present invention. [Figure 25]10 is a flow diagram illustrating a process for allocating and utilizing a library of standardized kitchen tools, objects, and equipment in a standardized robotic kitchen in accordance with the present invention. [Figure 26] 1 is a flow diagram illustrating a process for identifying non-standardized objects using three-dimensional modeling in accordance with the present invention. [Figure 27] 1 is a flow diagram illustrating a process for testing and learning mini-manipulations in accordance with the present invention. [Figure 28] 1 is a flow diagram illustrating a process for quality control of a robotic arm and an alignment function process in accordance with the present invention. [Figure 29] 13 is a table illustrating a database library structure of small manipulation objects for use in the standardized robotic kitchen in accordance with the present invention. [Figure 30] 13 is a table illustrating a database library structure of standardized objects for use in the standardized robotic kitchen in accordance with the present invention. [Figure 31] FIG. 1 is a pictorial diagram illustrating a robotic hand for performing fish quality checks in accordance with the present invention. [Figure 32] 1 is a pictorial diagram illustrating a robotic sensor head for performing quality checks within a ball in accordance with the present disclosure; [Figure 33] 1 is a pictorial diagram illustrating a detection device or container having a sensor for determining the freshness and quality of food in accordance with the present disclosure. [Figure 34] FIG. 1 is a system diagram illustrating an online analytical system for determining food freshness and quality in accordance with the present invention. [Figure 35] FIG. 1 is a block diagram illustrating a pre-filled container using a programmable dispenser control in accordance with the present disclosure. [Figure 36] FIG. 10 is a block diagram illustrating an example recipe structure and process for food preparation in the standardized robotic kitchen in accordance with the present invention. [Figure 37A]FIG. 10 is a block diagram illustrating a recipe search menu for use in the standardized robotic kitchen in accordance with the present invention. [Figure 37B] FIG. 10 is a block diagram illustrating a recipe search menu for use in the standardized robotic kitchen in accordance with the present invention. [Figure 37C] FIG. 10 is a block diagram illustrating a recipe search menu for use in the standardized robotic kitchen in accordance with the present invention. [Figure 37D] 1 is a screenshot of a menu with options for creating and submitting a recipe in accordance with the present disclosure. [Figure 37E] 10 is a screenshot depicting ingredient types. [Figure 37F] 1 is a flow diagram illustrating one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal associate page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation in accordance with the present invention. [Figure 37G] 1 is a flow diagram illustrating one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal associate page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation in accordance with the present invention. [Figure 37H] 1 is a flow diagram illustrating one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal associate page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation in accordance with the present invention. [Figure 37I]1 is a flow diagram illustrating one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal associate page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation in accordance with the present invention. [Figure 37J] 1 is a flow diagram illustrating one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal associate page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation in accordance with the present invention. [Figure 37K] 1 is a flow diagram illustrating one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal associate page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation in accordance with the present invention. [Figure 37L] 1 is a flow diagram illustrating one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal associate page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation in accordance with the present invention. [Figure 37M] 1 is a flow diagram illustrating one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal associate page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation in accordance with the present invention. [Figure 37N]1 is a flow diagram illustrating one embodiment of a food preparation user interface having functional capabilities including a recipe filter, an ingredient filter, an equipment filter, account and social network access, a personal associate page, a shopping cart page, and information about purchased recipes, registration settings, and recipe creation in accordance with the present invention. [Figure 38] FIG. 10 is a block diagram illustrating a recipe search menu by selecting fields for use in the standardized robotic kitchen in accordance with the present invention. [Figure 39] FIG. 10 is a block diagram illustrating a standardized robotic kitchen with augmented sensors for 3D tracking and reference data generation in accordance with the present invention. [Figure 40] FIG. 1 is a block diagram illustrating a standardized kitchen module using multiple sensors to create a real-time three-dimensional model in accordance with the present invention. [Figure 41A] FIG. 1 is a block diagram illustrating various embodiments and features of the standardized robotic kitchen in accordance with the present invention. [Figure 41B] FIG. 1 is a block diagram illustrating various embodiments and features of the standardized robotic kitchen in accordance with the present invention. [Figure 41C] FIG. 1 is a block diagram illustrating various embodiments and features of the standardized robotic kitchen in accordance with the present invention. [Figure 41D] FIG. 1 is a block diagram illustrating various embodiments and features of the standardized robotic kitchen in accordance with the present invention. [Figure 41E] FIG. 1 is a block diagram illustrating various embodiments and features of the standardized robotic kitchen in accordance with the present invention. [Figure 41F] FIG. 1 is a block diagram illustrating various embodiments and features of the standardized robotic kitchen in accordance with the present invention. [Figure 41G] FIG. 1 is a block diagram illustrating various embodiments and features of the standardized robotic kitchen in accordance with the present invention. [Figure 41H]FIG. 1 is a block diagram illustrating various embodiments and features of the standardized robotic kitchen in accordance with the present invention. [Figure 41I] FIG. 1 is a block diagram illustrating various embodiments and features of the standardized robotic kitchen in accordance with the present invention. [Figure 41J] FIG. 1 is a block diagram illustrating various embodiments and features of the standardized robotic kitchen in accordance with the present invention. [Figure 41K] FIG. 1 is a block diagram illustrating various embodiments and features of the standardized robotic kitchen in accordance with the present invention. [Figure 41L] FIG. 1 is a block diagram illustrating various embodiments and features of the standardized robotic kitchen in accordance with the present invention. [Figure 42A] FIG. 10 is a block diagram illustrating a top view of the standardized robotic kitchen in accordance with the present invention. [Figure 42B] FIG. 10 is a block diagram illustrating a perspective view of the standardized robotic kitchen in accordance with the present invention. [Figure 43A] FIG. 10 is a block diagram illustrating a first embodiment of a kitchen module frame with automatic transparent doors in the standardized robotic kitchen in accordance with the present invention. [Figure 43B] FIG. 10 is a block diagram illustrating a first embodiment of a kitchen module frame with automatic transparent doors in the standardized robotic kitchen in accordance with the present invention. [Figure 44A] FIG. 10 is a block diagram illustrating a second embodiment of a kitchen module frame with automatic transparent doors in the standardized robotic kitchen in accordance with the present invention. [Figure 44B] FIG. 10 is a block diagram illustrating a second embodiment of a kitchen module frame with automatic transparent doors in the standardized robotic kitchen in accordance with the present invention. [Figure 45] FIG. 10 is a block diagram illustrating a standardized robotic kitchen with telescopic actuators in accordance with the present invention. [Figure 46A]FIG. 10 is a block diagram illustrating a front view of the standardized robotic kitchen with a pair of fixed robotic arms without moving rails in accordance with the present invention. [Figure 46B] FIG. 10 is a block diagram illustrating a perspective view of the standardized robotic kitchen with a pair of fixed robotic arms without moving rails in accordance with the present invention. [Figure 46C] FIG. 10 is a block diagram illustrating various dimensions of the standardized robotic kitchen with a pair of fixed robotic arms without moving rails. [Figure 46D] FIG. 10 is a block diagram illustrating various dimensions of the standardized robotic kitchen with a pair of fixed robotic arms without moving rails. [Figure 46E] FIG. 10 is a block diagram illustrating various dimensions of the standardized robotic kitchen with a pair of fixed robotic arms without moving rails. [Figure 46F] FIG. 10 is a block diagram illustrating various dimensions of the standardized robotic kitchen with a pair of fixed robotic arms without moving rails. [Figure 46G] FIG. 10 is a block diagram illustrating various dimensions of the standardized robotic kitchen with a pair of fixed robotic arms without moving rails. [Figure 47] FIG. 10 is a block diagram illustrating an exemplary program storage system for use with the standardized robotic kitchen in accordance with the present invention. [Figure 48] FIG. 10 is a block diagram illustrating an elevation view of a program storage system for use with the standardized robotic kitchen in accordance with the present invention. [Figure 49] FIG. 10 is a block diagram illustrating an elevation view of an ingredient access container for use with the standardized robotic kitchen in accordance with the present invention. [Figure 50] FIG. 10 is a block diagram illustrating an ingredient quality monitoring dashboard associated with ingredient access containers for use with the standardized robotic kitchen in accordance with the present invention. [Figure 51]1 is a table illustrating a database library of recipe parameters in accordance with the present invention. [Figure 52] 1 is a flow chart illustrating the process of one embodiment of recording a chef's food preparation process in accordance with the present invention. [Figure 53] 1 is a flow chart illustrating the process of one embodiment of a robotic device for preparing a food dish in accordance with the present invention. [Figure 54] 1 is a flow diagram illustrating the process of one embodiment of adjusting quality and functionality in achieving the same (or substantially the same) result as a chef in robotic food dish preparation according to the present invention. [Figure 55] FIG. 1 is a flow diagram illustrating a first embodiment of a robotic kitchen process for preparing a dish by replicating a chef's movements within the robotic kitchen from a recorded software file in accordance with the present invention. [Figure 56] 14 is a flow diagram illustrating the process of inventorying and identifying storage containers in the robotic kitchen in accordance with the present invention. [Figure 57] 14 is a flow diagram illustrating the process of storage retrieval and food preparation in the robotic kitchen in accordance with the present invention. [Figure 58] 1 is a flow diagram illustrating one embodiment of an automated pre-cooking preparation process in the robotic kitchen in accordance with the present invention. [Figure 59] 1 is a flow diagram illustrating one embodiment of a recipe design and scripting process in the robotic kitchen in accordance with the present invention. [Figure 60] 1 is a flow diagram illustrating a subscription model for users to purchase robotic food preparation recipes in accordance with the present invention. [Figure 61A] 1 is a flow diagram illustrating the process of searching for and purchasing a recipe subscription on a recipe commerce platform from a portal in accordance with the present invention. [Figure 61B] 1 is a flow diagram illustrating the process of searching for and purchasing a recipe subscription on a recipe commerce platform from a portal in accordance with the present invention. [Figure 62] 1 is a flow diagram illustrating the creation of a robotic cooking recipe app on an app platform in accordance with the present invention. [Figure 63] 1 is a flow chart illustrating a user's process of searching, purchasing, and subscribing to cooking recipes in accordance with the present invention. [Figure 64A] FIG. 1 is a block diagram illustrating an example of predefined recipe search criteria in accordance with the present invention. [Figure 64B] FIG. 1 is a block diagram illustrating an example of predefined recipe search criteria in accordance with the present invention. [Figure 65] FIG. 10 is a block diagram illustrating some predefined containers in the robotic kitchen in accordance with the present invention. [Figure 66] FIG. 12 is a block diagram illustrating a first embodiment of a robotic restaurant kitchen module configured in a rectangular layout using multiple pairs of robotic hands for simultaneous food preparation processing in accordance with the present invention. [Figure 67] FIG. 10 is a block diagram illustrating a second embodiment of a robotic restaurant kitchen module configured in a U-shape layout using multiple pairs of robotic hands for simultaneous food preparation processing in accordance with the present invention. [Figure 68] FIG. 10 is a block diagram illustrating a second embodiment of a robotic food preparation system with sensory utensils and sensory curves in accordance with the present invention. [Figure 69] FIG. 10 is a block diagram illustrating some physical elements of a robotic food preparation system in a second embodiment according to the present invention. [Figure 70] FIG. 10 is a block diagram illustrating a sensory cookware for a (smart) pan with real-time temperature sensors for use in a second embodiment according to the present invention. [Figure 71] FIG. 10 is a graph illustrating a temperature curve with multiple data points recorded from different sensors of a sensory cookware in a chef studio in accordance with the present invention. [Figure 72]10A-10C are graphs illustrating temperature and humidity curves recorded from sensory cookware in a chef studio for transmission to an operational control unit in accordance with the present invention. [Figure 73] FIG. 10 is a block diagram illustrating a sensory cookware for cooking based on data from temperature curves for different zones on a pan in accordance with the present invention. [Figure 74] FIG. 10 is a block diagram illustrating a (smart) oven sensory cookware using real-time temperature and humidity sensors for use in a second embodiment according to the present invention. [Figure 75] FIG. 10 is a block diagram illustrating a sensory cooking utensil for a (smart) charcoal grill using real-time temperature sensors for use in a second embodiment according to the present invention. [Figure 76] FIG. 10 is a block diagram illustrating a sensory cookware for a (smart) faucet with speed, temperature, and power control for use in a second embodiment according to the present invention. [Figure 77] FIG. 10 is a block diagram illustrating a top view of the robotic kitchen with sensory cookware in a second embodiment in accordance with the present invention. [Figure 78] FIG. 10 is a block diagram illustrating a perspective view of the robotic kitchen with sensory cookware in a second embodiment in accordance with the present invention. [Figure 79] FIG. 10 is a flow diagram illustrating a second embodiment of a robotic kitchen process for preparing a dish from one or more previously recorded parameter curves in the standardized robotic kitchen in accordance with the present invention. [Figure 80] FIG. 1 depicts one embodiment of a sensory data capture process within a chef studio in accordance with the present invention. [Figure 81] FIG. 1 depicts the process and flow of the domestic robotic cooking process, the first step of which involves a user selecting a recipe and obtaining the recipe in digital form according to the present invention. [Figure 82]FIG. 10 is a block diagram illustrating a third embodiment of the robotic food preparation kitchen with a cooking operation control module and a command and visual monitoring module in accordance with the present invention. [Figure 83] FIG. 10 is a block diagram illustrating a top view of a third embodiment of the robotic food preparation kitchen with robotic arm and hand movements in accordance with the present invention. [Figure 84] FIG. 10 is a block diagram illustrating a top view of a third embodiment of the robotic food preparation kitchen using robotic arm and hand movements in accordance with the present invention. [Figure 85] FIG. 10 is a block diagram illustrating a floor plan of a third embodiment of a robotic food preparation kitchen with a command and visual monitoring device in accordance with the present invention. [Figure 86] FIG. 10 is a block diagram illustrating a perspective view of a third embodiment of the robotic food preparation kitchen with a command and visual monitoring device in accordance with the present invention. [Figure 87A] FIG. 10 is a block diagram illustrating a fourth embodiment of a robotic food preparation kitchen using a robot in accordance with the present invention. [Figure 87B] FIG. 10 is a block diagram illustrating a top view of a fourth embodiment of the robotic food preparation kitchen with a humanoid robot in accordance with the present invention. [Figure 87C] FIG. 10 is a block diagram illustrating a perspective view of a fourth embodiment of the robotic food preparation kitchen using a humanoid robot in accordance with the present invention. [Figure 88] FIG. 1 is a block diagram illustrating a robotic human-emulator electronic intellectual property (IP) library in accordance with the present invention. [Figure 89] FIG. 1 is a block diagram illustrating a robot human-emotion recognition engine according to the present invention. [Figure 90] 1 is a flow diagram illustrating the process of a robot human-emotion engine according to the present invention. [Figure 91A]1 is a flow diagram illustrating a process for comparing an individual's emotional profile against a population of emotional profiles having hormones, pheromones, and other parameters in accordance with the present invention. [Figure 91B] 1 is a flow diagram illustrating a process for comparing an individual's emotional profile against a population of emotional profiles having hormones, pheromones, and other parameters in accordance with the present invention. [Figure 91C] 1 is a flow diagram illustrating a process for comparing an individual's emotional profile against a population of emotional profiles having hormones, pheromones, and other parameters in accordance with the present invention. [Figure 92A] FIG. 1 is a block diagram illustrating emotion detection and analysis of an individual's emotional state by monitoring a set of hormones, a set of pheromones, and other key parameters in accordance with the present invention. [Figure 92B] FIG. 1 is a block diagram illustrating a robot for assessing and learning about the emotional behavior of an individual according to the present invention. [Figure 93] FIG. 1 is a block diagram illustrating a port device implanted within an individual for detecting and recording the individual's emotional profile in accordance with the present invention. [Figure 94A] FIG. 1 is a block diagram illustrating a robotic human-intelligence engine in accordance with the present invention. [Figure 94B] 1 is a flow chart illustrating the process of a robotic human-intelligence engine in accordance with the present invention. [Figure 95A] FIG. 1 is a block diagram illustrating a robotic drawing system in accordance with the present invention. [Figure 95B] FIG. 1 is a block diagram illustrating various components of a robotic drawing system in accordance with the present invention. [Figure 95C] FIG. 1 is a block diagram illustrating a robotic human-drawing-skill-reproduction engine according to the present invention. [Figure 96A] 1 is a flow chart illustrating an artist's recording process in an art studio in accordance with the present invention; [Figure 96B]1 is a flow diagram illustrating a reproduction process by a robotic drawing system in accordance with the present invention. [Figure 97A] FIG. 1 is a block diagram illustrating one embodiment of a musician reproduction engine in accordance with the present invention. [Figure 97B] FIG. 10 is a block diagram illustrating a musician reproduction engine process in accordance with the present invention. [Figure 98] FIG. 1 is a block diagram illustrating one embodiment of a nursing replication engine in accordance with the present invention. [Figure 99A] 1 is a flow diagram illustrating a nursing replication engine process in accordance with the present invention. [Figure 99B] 1 is a flow diagram illustrating a nursing replication engine process in accordance with the present invention. [Figure 100] FIG. 1 is a block diagram illustrating the general applicability (or versatility) of a robotic human skill replication system having an author record system and a commercial robotic system according to the present disclosure. [Figure 101] FIG. 1 is a software system diagram illustrating a robotic human skill replication engine having various modules in accordance with the present invention. [Figure 102] FIG. 1 is a block diagram illustrating one embodiment of a robotic human skill replication system in accordance with the present invention. [Figure 103] FIG. 1 is a block diagram illustrating a humanoid robot according to the present invention with standardized actuation tools, standardized positions and orientations, and control points for a skill performance or replication process using standardized equipment. [Figure 104] FIG. 1 is a schematic block diagram illustrating a humanoid robot reproduction program for reproducing the process of recorded human skill movement by tracking the activity of glove sensors at periodic time intervals according to the present invention; [Figure 105] FIG. 1 is a block diagram illustrating the recording of creator movements and humanoid robot reproduction according to the present invention. [Figure 106] FIG. 1 depicts the overall robotic control platform for a generic humanoid robot as a high-level description of the functionality of the present disclosure. [Figure 107] FIG. 1 is a block diagram illustrating a system diagram for the creation, transfer, implementation, and use of mini-manipulation libraries as part of a humanoid robot application-task replication process in accordance with the present invention. [Figure 108] FIG. 1 is a block diagram illustrating categories and types of studio and robot-based sensory data input in accordance with the present invention. [Figure 109] FIG. 1 is a block diagram illustrating a physics / systems-based mini-manipulation library motion-based dual-arm and fuselage topology in accordance with the present invention; [Figure 110] FIG. 10 is a block diagram illustrating mini-manipulation library operation phase combinations and transitions for task-specific operation sequences in accordance with the present invention. [Figure 111] FIG. 1 is a block diagram illustrating the process of building one or more mini-manipulation libraries (generic and task-specific) from studio data in accordance with the present invention. [Figure 112] FIG. 1 is a block diagram illustrating robotic task execution with one or more mini-manipulation library data sets in accordance with the present invention. [Figure 113] FIG. 1 is a block diagram illustrating a system diagram for an automated mini-manipulation parameter set construction engine in accordance with the present invention. [Figure 114A] FIG. 1 is a block diagram illustrating a data-centric view of a robotic system in accordance with the present invention. [Figure 114B] 1A-1C are block diagrams illustrating examples of various mini-manipulation data formats for organizing, linking, and converting mini-manipulation robot behavior data in accordance with the present invention. [Figure 115] FIG. 1 is a block diagram illustrating various bidirectional abstraction levels between a robot hardware engineering design concept, a robot software engineering design concept, a robot business design concept, and mathematical algorithms to support the robot engineering design concept. [Figure 116] FIG. 1 is a block diagram showing a humanoid robotic arm and each hand having five fingers in accordance with the present invention. [Figure 117A]FIG. 1 is a block diagram illustrating an embodiment of a humanoid robot according to the present disclosure. [Figure 117B] FIG. 1 is a block diagram illustrating an embodiment showing a gyroscope-equipped humanoid robot and graphic data in accordance with the present disclosure. [Figure 117C] 1 is a graphical diagram illustrating an author recording device mounted on a humanoid robot including a sensing bodysuit, arm exoskeleton, headgear, and sensing gloves in accordance with the present invention; [Figure 118] FIG. 1 is a block diagram illustrating a humanoid robot human skill expert mini-manipulation library in accordance with the present invention. [Figure 119] FIG. 1 is a block diagram illustrating the process of creating an electronic library of generic mini-manipulations to replace human hand skill movements in accordance with the present invention. [Figure 120] FIG. 1 is a block diagram illustrating a robotic task with multi-stage execution according to a general mini-manipulation in accordance with the present invention. [Figure 121] FIG. 10 is a block diagram illustrating real-time parameter adjustment during the execution phase of a mini-manipulation in accordance with the present invention. [Figure 122] FIG. 1 is a block diagram illustrating a set of mini-operations for making sushi in accordance with the present invention. [Figure 123] FIG. 1 is a block diagram illustrating a first mini-operation for filleting fish in a set of mini-operations for making sushi in accordance with the present invention. [Figure 124] FIG. 10 is a block diagram illustrating a second mini-operation of removing rice from a container in a set of mini-operations for making sushi in accordance with the present invention. [Figure 125] FIG. 10 is a block diagram illustrating a third mini-operation for filleting fish in the set of mini-operations for making sushi in accordance with the present invention. [Figure 126] FIG. 10 is a block diagram illustrating a fourth mini-operation for firming rice and fish fillets into a desired shape within a set of mini-operations for making sushi in accordance with the present invention. [Figure 127]FIG. 10 is a block diagram illustrating a fifth mini-operation of holding rice with fish fillets in the set of mini-operations for making sushi in accordance with the present invention. [Figure 128] FIG. 1 is a block diagram illustrating a set of mini-operations for playing the piano that may be performed in any sequence or combination according to the present disclosure. [Figure 129] FIG. 1 is a block diagram showing a first right-hand mini-operation and a second left-hand mini-operation performed in parallel with playing the piano from a set of mini-operations for playing the piano according to the present invention. [Figure 130] FIG. 10 is a block diagram showing a third mini-operation for the right foot and a fourth mini-operation for the left foot of a set of mini-operations performed in parallel from a set of mini-operations for playing piano in accordance with the present invention. [Figure 131] FIG. 10 is a block diagram illustrating a fifth mini-operation for body movements performed in parallel with one or more mini-operations from a set of mini-operations for playing piano in accordance with the present invention. [Figure 132] FIG. 1 is a block diagram illustrating a set of mini-operations for humanoid robot walking performed in any sequence and in any combination according to the present disclosure. [Figure 133] FIG. 1 is a block diagram showing a first small-scale manipulation of walking in a step posture of the right leg of the small-scale manipulation set for walking of a humanoid robot according to the present invention. [Figure 134] FIG. 10 is a block diagram showing a second small-scale operation of walking with a right leg squash posture of the set of small-scale operations for walking of a humanoid robot in accordance with the present invention. [Figure 135] FIG. 10 is a block diagram showing a third small-scale manipulation of walking with a passing posture of the right leg of the small-scale manipulation set for walking of a humanoid robot according to the present invention. [Figure 136] FIG. 10 is a block diagram showing a fourth small-scale operation of walking with the right leg extended in the small-scale operation set for walking of a humanoid robot according to the present invention. [Figure 137]FIG. 10 is a block diagram showing a fifth small-scale manipulation of the set of small-scale manipulations for walking of a humanoid robot in accordance with the present invention, for walking in a step posture of the left leg. [Figure 138] FIG. 1 is a block diagram illustrating a robotic nursing care module with a three-dimensional vision system in accordance with the present invention. [Figure 139] FIG. 1 is a block diagram illustrating a robotic nursing care module with a standardized cabinet in accordance with the present invention. [Figure 140] FIG. 1 is a block diagram illustrating a robotic nursing care module having one or more standardized storages, a standardized screen, and a standardized wardrobe in accordance with the present invention. [Figure 141] FIG. 1 is a block diagram illustrating a robotic nursing care module having a telescoping body with a pair of robotic arms and a pair of robotic hands in accordance with the present invention. [Figure 142] FIG. 1 is a block diagram showing a first example of implementing a robotic nursing care module with various locomotions to assist elderly people according to the present invention. [Figure 143] FIG. 10 is a block diagram illustrating a second example of implementing the wheelchair loading and unloading robotic nursing care module in accordance with the present invention. [Figure 144] FIG. 1 is a pictorial diagram showing a humanoid robot acting as an intermediary between two humans according to the present invention. [Figure 145] 1 is a pictorial diagram showing a first embodiment in which a humanoid robot according to the present invention cares for person B as a therapist while directly observing person A. [Figure 146] FIG. 1 is a block diagram showing a first embodiment of motor placement for a humanoid robotic hand and arm requiring high torque for hand and arm operation in accordance with the present invention. [Figure 147] FIG. 1 is a block diagram showing a first embodiment of motor placement for a humanoid robotic hand and arm requiring low torque for hand and arm operation in accordance with the present invention. [Figure 148A]FIG. 10 is a pictorial diagram showing a front view of the robotic arm extending from an overhead mount of the robotic kitchen with oven in accordance with the present invention. [Figure 148B] FIG. 10 is a pictorial diagram showing a top view of the robotic arm extending from an overhead mount of the robotic kitchen with an oven. [Figure 149A] FIG. 10 is a pictorial diagram showing a front view of the robotic arm extending from the overhead mount of the robotic kitchen with additional spacing in accordance with the present invention. [Figure 149B] FIG. 10 is a pictorial diagram showing a top view of the robotic arm extending from the overhead mount of the robotic kitchen with additional spacing in accordance with the present invention. [Figure 150A] FIG. 10 is a pictorial diagram showing a front view of the robotic arm extending from an overhead mount of the robotic kitchen with sliding storage in accordance with the present invention. [Figure 150B] FIG. 10 is a pictorial diagram showing a top view of the robotic arm extending from an overhead mount of the robotic kitchen with sliding storage in accordance with the present invention. [Figure 151A] FIG. 10 is a pictorial diagram showing a front view of the robotic arm extending from an overhead mount of the robotic kitchen with sliding storage with shelves in accordance with the present invention. [Figure 151B] FIG. 10 is a pictorial diagram showing a top view of the robotic arm extending from an overhead mount of the robotic kitchen with sliding storage with shelves in accordance with the present invention. [Figure 152] 1A-1C are pictorial diagrams illustrating various embodiments of robotic gripping options in accordance with the present disclosure. [Figure 153] 1A-1C are pictorial diagrams illustrating various embodiments of robotic gripping options in accordance with the present disclosure. [Fig. 154] 1A-1C are pictorial diagrams illustrating various embodiments of robotic gripping options in accordance with the present disclosure. [Figure 155] 1A-1C are pictorial diagrams illustrating various embodiments of robotic gripping options in accordance with the present disclosure. [Figure 156]1A-1C are pictorial diagrams illustrating various embodiments of robotic gripping options in accordance with the present disclosure. [Figure 157] 1A-1C are pictorial diagrams illustrating various embodiments of robotic gripping options in accordance with the present disclosure. [Figure 158] 1A-1C are pictorial diagrams illustrating various embodiments of robotic gripping options in accordance with the present disclosure. [Figure 159] 1A-1C are pictorial diagrams illustrating various embodiments of robotic gripping options in accordance with the present disclosure. [Figure 160] 1A-1C are pictorial diagrams illustrating various embodiments of robotic gripping options in accordance with the present disclosure. [Figure 161] 1A-1C are pictorial diagrams illustrating various embodiments of robotic gripping options in accordance with the present disclosure. [Figure 162A] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162B] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162C] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162D] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162E] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162F] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162G] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162H]FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162I] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162J] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162K] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162L] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162M] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162N] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162O] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162P] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162Q] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162R] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 162S] FIG. 1 is a pictorial diagram showing a cookware handle suitable for attaching a robotic hand to cookware and appliances in accordance with the present disclosure. [Figure 163] FIG. 10 is a pictorial diagram showing a blender portion for use in the robotic kitchen in accordance with the present invention. [Figure 164A] FIG. 10 is a pictorial diagram showing various kitchen holders for use in the robotic kitchen in accordance with the present invention. [Figure 164B] FIG. 10 is a pictorial diagram showing various kitchen holders for use in the robotic kitchen in accordance with the present invention. [Figure 164C] FIG. 10 is a pictorial diagram showing various kitchen holders for use in the robotic kitchen in accordance with the present invention. [Figure 165A] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165B] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165C] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165D] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165E] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165F] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165G] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165H] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165I] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165J] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165K] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165L] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165M] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165N] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165O] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165P] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165Q] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165R] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165S] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165T] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165U] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 165V] FIG. 1 is a block diagram illustrating an example of a mini-manipulation that does not limit the disclosure of the present invention. [Figure 166A] FIG. 1 illustrates sample types of kitchen utensils listed in Table A according to the present disclosure. [Figure 166B] FIG. 1 illustrates sample types of kitchen utensils listed in Table A according to the present disclosure. [Figure 166C] FIG. 1 illustrates sample types of kitchen utensils listed in Table A according to the present disclosure. [Figure 166D] FIG. 1 illustrates sample types of kitchen utensils listed in Table A according to the present disclosure. [Figure 166E] FIG. 1 illustrates sample types of kitchen utensils listed in Table A according to the present disclosure. [Figure 166F] FIG. 1 illustrates sample types of kitchen utensils listed in Table A according to the present disclosure. [Figure 166G] FIG. 1 illustrates sample types of kitchen utensils listed in Table A according to the present disclosure. [Figure 166H] FIG. 1 illustrates sample types of kitchen utensils listed in Table A according to the present disclosure. [Figure 166I] FIG. 1 illustrates sample types of kitchen utensils listed in Table A according to the present disclosure. [Figure 166J] FIG. 1 illustrates sample types of kitchen utensils listed in Table A according to the present disclosure. [Figure 166K] FIG. 1 illustrates sample types of kitchen utensils listed in Table A according to the present disclosure. [Figure 166L] FIG. 1 illustrates sample types of kitchen utensils listed in Table A according to the present disclosure. [Figure 167A] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167B] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167C] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167D] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167E] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167F] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167G] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167H] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167I] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167J] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167K] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167L] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167M] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167N] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167O] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167P] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167Q] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167R] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167S] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167T] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167U] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 167V] FIG. 1 illustrates sample types of food ingredients listed in Table B according to the present disclosure. [Figure 168A] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168B] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168C]FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168D] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168E] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168F] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168G] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168H] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168I] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168J] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168K] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168L] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168M] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168N] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168O] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168P] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168Q] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168R] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168S] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168T] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168U] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168V] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168W] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168X] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168Y] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 168Z] FIG. 1 illustrates a sample list of food preparations, methods, utensils, and recipes in Table C according to the present disclosure. [Figure 169A] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169B] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169C] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169D] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169E]FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169F] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169G] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169H] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169I] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169J] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169K] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169L] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169M] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169N] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169O] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169P] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Q] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169R] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169S] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169T] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169U] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169V]FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169W] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169X] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Y] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z1] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z2] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z3] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z4] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z5] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z6] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z7] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z8] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z9] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z10] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z11] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z12] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z13]FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z14] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 169Z15] FIG. 10 illustrates various sample bases in Table C according to the present disclosure. [Figure 170A] FIG. 1 illustrates the recipes and food sample types described in Table D according to the present disclosure. [Figure 170B] FIG. 1 illustrates the recipes and food sample types described in Table D according to the present disclosure. [Figure 170C] FIG. 1 illustrates the recipes and food sample types described in Table D according to the present disclosure. [Figure 171A] FIG. 1 illustrates one embodiment of a robotic food preparation system as described in Table E according to the present disclosure. [Figure 171B] FIG. 1 illustrates one embodiment of a robotic food preparation system as described in Table E according to the present disclosure. [Figure 171C] FIG. 1 illustrates one embodiment of a robotic food preparation system as described in Table E according to the present disclosure. [Figure 171D] FIG. 1 illustrates one embodiment of a robotic food preparation system as described in Table E according to the present disclosure. [Figure 171E] FIG. 1 illustrates one embodiment of a robotic food preparation system as described in Table E according to the present disclosure. [Figure 172A] FIG. 1 illustrates sample mini-manipulations for a robot making sushi, playing the piano, moving a robot from a first position to a second position, turning from one position to the next, a humanoid robot retrieving a book from a bookshelf, a humanoid robot moving a trunk from one position to the next, a robot unscrewing a bottle, and a robot putting cat food in a bowl in accordance with the present disclosure. [Figure 172B]FIG. 1 illustrates sample mini-manipulations for a robot making sushi, playing the piano, moving a robot from a first position to a second position, turning from one position to the next, a humanoid robot retrieving a book from a bookshelf, a humanoid robot moving a trunk from one position to the next, a robot unscrewing a bottle, and a robot putting cat food in a bowl in accordance with the present disclosure. [Figure 172C] FIG. 1 illustrates sample mini-manipulations for a robot making sushi, playing the piano, moving a robot from a first position to a second position, turning from one position to the next, a humanoid robot retrieving a book from a bookshelf, a humanoid robot moving a trunk from one position to the next, a robot unscrewing a bottle, and a robot putting cat food in a bowl in accordance with the present disclosure. [Figure 173A] FIG. 10 illustrates an example of sample multi-level miniature operations for a robot to perform oxygen supplementation, thermoregulation, catheter insertion, physical therapy, feeding, analytical sample collection, fistula and catheter monitoring treatment, wound treatment, drug administration, and sanitary safety operations according to the present disclosure. [Figure 173B] FIG. 10 illustrates an example of sample multi-level miniature operations for a robot to perform oxygen supplementation, thermoregulation, catheter insertion, physical therapy, feeding, analytical sample collection, fistula and catheter monitoring treatment, wound treatment, drug administration, and sanitary safety operations according to the present disclosure. [Figure 173C] FIG. 10 illustrates an example of sample multi-level miniature operations for a robot to perform oxygen supplementation, thermoregulation, catheter insertion, physical therapy, feeding, analytical sample collection, fistula and catheter monitoring treatment, wound treatment, drug administration, and sanitary safety operations according to the present disclosure. [Figure 173D] FIG. 10 illustrates an example of sample multi-level miniature operations for a robot to perform oxygen supplementation, thermoregulation, catheter insertion, physical therapy, feeding, analytical sample collection, fistula and catheter monitoring treatment, wound treatment, drug administration, and sanitary safety operations according to the present disclosure. [Figure 173E]FIG. 10 illustrates an example of sample multi-level miniature operations for a robot to perform oxygen supplementation, thermoregulation, catheter insertion, physical therapy, feeding, analytical sample collection, fistula and catheter monitoring treatment, wound treatment, drug administration, and sanitary safety operations according to the present disclosure. [Figure 173F] FIG. 10 illustrates an example of sample multi-level miniature operations for a robot to perform oxygen supplementation, thermoregulation, catheter insertion, physical therapy, feeding, analytical sample collection, fistula and catheter monitoring treatment, wound treatment, drug administration, and sanitary safety operations according to the present disclosure. [Figure 173G] FIG. 10 illustrates an example of sample multi-level miniature operations for a robot to perform oxygen supplementation, thermoregulation, catheter insertion, physical therapy, feeding, analytical sample collection, fistula and catheter monitoring treatment, wound treatment, drug administration, and sanitary safety operations according to the present disclosure. [Figure 173H] FIG. 10 illustrates an example of sample multi-level miniature operations for a robot to perform oxygen supplementation, thermoregulation, catheter insertion, physical therapy, feeding, analytical sample collection, fistula and catheter monitoring treatment, wound treatment, drug administration, and sanitary safety operations according to the present disclosure. [Figure 173I] FIG. 10 illustrates an example of sample multi-level miniature operations for a robot to perform oxygen supplementation, thermoregulation, catheter insertion, physical therapy, feeding, analytical sample collection, fistula and catheter monitoring treatment, wound treatment, drug administration, and sanitary safety operations according to the present disclosure. [Fig. 174] FIG. 10 illustrates an example of sample multi-level miniature operations for a robot to perform oxygen supplementation, thermoregulation, catheter insertion, physical therapy, feeding, analytical sample collection, fistula and catheter monitoring treatment, wound treatment, drug administration, and sanitary safety operations according to the present disclosure. [Figure 175] 1 shows a sample medical equipment list and a medical device list according to the present disclosure. [Figure 176A] FIG. 1 illustrates a sample nursing care service using mini-operations in accordance with the present invention. [Figure 176B] FIG. 1 illustrates a sample nursing care service using mini-operations in accordance with the present invention. [Figure 177] FIG. 10 illustrates another device list in accordance with the present disclosure. [Figure 178] FIG. 1 is a block diagram illustrating an example of a computing device on which computer-executable instructions can be installed and executed, the instructions implementing the methodologies described herein. DETAILED DESCRIPTION OF THE INVENTION

[0034] A description of structural embodiments and methods of the present disclosure is provided with reference to Figures 1-178. It is not intended that the present disclosure be limited to the specifically disclosed embodiments, and it is understood that the present disclosure may be practiced using other features, elements, methods, and embodiments. Similar elements in various embodiments are commonly designated with similar reference numerals.

[0035] The following definitions apply to the elements and steps described herein. In some cases, these terms are further elaborated upon.

[0036] Abstracted Data—refers to a working abstract recipe for machine execution with many other data elements that the machine needs to know for proper execution and replication. This data is so-called metadata or additional data corresponding to specific steps in the cooking process, whether it is direct sensor data (such as clock time, water temperature, camera images, utensils or ingredients used) or data generated by interpretation or abstraction of larger datasets (such as a 3D range cloud from a laser used to extract the location and type of objects in the image, overlaid with texture and color maps from a camera image). This metadata is time-stamped and used by the robotic kitchen to configure, control, and monitor all processes and associated methods and equipment needed at every point as the robotic kitchen moves through the sequence of steps in the recipe.

[0037] Abstracted Recipe—refers to a representation of a chef's recipe as known to humans, expressed as a sequence of processes and methods and the use of certain ingredients prepared and combined in a certain sequence through the skill of a human chef. Abstracted recipes used by machines for automated execution require various types of classification and sequencing. While the overall execution steps are identical to those of a human chef, an abstracted working recipe for a robotic kitchen requires additional metadata to be part of every step in the recipe. Such metadata includes cooking time and variables such as temperature (and its change over time), oven settings, tools / equipment used, etc. Essentially, a machine-executable recipe script requires all possible measurement variables important to the cooking process (all measured and stored by a human chef during the preparation of the recipe in the chef studio) to be correlated with time as a whole, as well as within each processing step of the cooking sequence. An abstracted recipe is thus a machine-readable representation or domain-mapped representation of cooking steps that moves the necessary process from the human domain through a set of logical abstraction steps to a machine-understandable and machine-executable domain.

[0038] Acceleration - refers to the maximum rate of change of velocity that a robotic arm can accelerate around an axis or along a spatial trajectory over a short distance.

[0039] Accuracy—refers to how close the robot can get to a commanded position. Accuracy is determined by the difference between the absolute position of the robot compared to the commanded position. Accuracy can be improved, adjusted, or calibrated by external sensing, such as sensors on the robot hand, or by real-time 3D models using multiple (multi-modal) sensors.

[0040] Action Primitive—In one embodiment, refers to an indivisible robotic action, such as moving a robotic device from location X1 to location X2, or sensing distance from an object toward food preparation without necessarily achieving a functional outcome. In another embodiment, the term refers to an indivisible robotic action in a sequence of one or more such units to accomplish a mini-manipulation. These are two aspects of the same definition.

[0041] Automatic Dosing System - refers to a dosing container in a standardized kitchen module that, upon application, releases a specific size of food compound (salt, sugar, pepper, spices, etc., water, oil, extracts, any type of liquid, e.g. ketchup, etc.).

[0042] Automated Storage and Delivery System - refers to storage containers that maintain a specific temperature and humidity for storing food within standardized kitchen modules, each assigned a code (e.g., barcode) that allows the robotic kitchen to identify and determine where to deliver the food contents stored therein.

[0043] Data Cloud - refers to a collection of sensor or data-based numerical measurements (e.g., 3D laser / acoustic distance measurements, RGB values ​​from a camera image, etc.) from a specific space collected at regular intervals and aggregated based on a number of relationships such as time, location, etc.

[0044] Degrees of Freedom ("DOF") - Refers to the defined modes and / or directions in which a mechanical device or system can move. The number of degrees of freedom is equal to the total number of independent displacements or modes of motion. The total number of degrees of freedom is doubled for two robotic arms.

[0045] Edge detection - refers to a software-based computer program that can identify the edges of multiple objects, even those that may overlap in a two-dimensional camera image, to aid in object identification and planning for grasping and handling.

[0046] Equilibrium Value—refers to the target position of a robotic appendage, such as a robotic arm, where the forces acting on the appendage are in equilibrium, i.e., there are no net forces and therefore no net movement.

[0047] Execution Sequence Planner - refers to a software-based computer program capable of creating execution scripts or sequences of instructions for one or more computer-controllable elements or systems such as arms, dispensers, tools, etc.

[0048] Food Execution Fidelity - refers to a robotic kitchen designed to replicate the recipe script generated in Chef Studio by observing, measuring, and understanding the steps, variables, methods, and processes of a human chef, thereby attempting to mimic the techniques and skills of this chef. Fidelity, or how closely the execution of food preparation comes to that of a human chef, is measured by measuring how closely the food prepared by the robot resembles that prepared by a human through various subjective factors such as consistency, color, and taste. The concept is that the closer the food prepared by the robotic kitchen is to that prepared by a human chef, the higher the fidelity of the replication process.

[0049] Food Preparation Stage (also called "Cooking Stage") - refers to the combination, either serially or in parallel, of one or more mini-operations, including action primitives, and computer instructions for controlling various kitchen equipment and appliances in a standardized kitchen module. One or more food preparation stages together represent the entire food preparation process for a particular recipe.

[0050] Geometric Reasoning - refers to a software-based computer program that can use two-dimensional (2D) and three-dimensional (3D) surface and / or volumetric data to reason about the actual shape and size of a particular spatial region. The ability to determine and use boundary information also allows for inference about the beginning and end of particular geometric elements and how many are present in an image or model.

[0051] Grasping Reasoning—refers to a software-based computer program that relies on geometric and physical reasoning to plan multi-contact (point / area / region of space) contact interactions between a robotic end effector (grasper, coupler, etc.) or even a tool / instrument held by the end effector, in order to successfully contact, grasp, and hold an object in order to manipulate it in 3D space.

[0052] Hardware automation device - a fixed processing device that can execute pre-programmed steps in succession without the ability to modify any of these steps; such devices are used for repetitive movements that do not require any adjustments.

[0053] Ingredient Management and Manipulation - refers to the detailed definition of each ingredient (including size, shape, weight, dimensions, characteristics, and properties), the real-time adjustment of one or more variables associated with a given ingredient that may differ from previously stored ingredient details (fish fillet size, egg dimensions, etc.), and the process of performing different stages of operational movement on an ingredient.

[0054] Kitchen module (or kitchen space area)—A standardized, complete kitchen module with a standardized set of kitchen equipment, a standardized set of kitchen tools, a standardized set of kitchen handles, and a standardized set of kitchen containers, with predefined internal space and dimensions for storing, accessing, and manipulating each kitchen element. One purpose of the kitchen module is to predefine as much of the kitchen equipment, tools, handles, containers, etc. as possible to provide a relatively fixed kitchen platform for the robotic arm and hand movement. This standardized kitchen module is used by both the chef in the chef kitchen studio and the individual in the home (or individual in the restaurant) with the robotic kitchen to maximize the predicted performance of the kitchen hardware while minimizing the risk of differentiation, variation, and deviation between the chef kitchen studio and the home robotic kitchen. Various embodiments of the kitchen module are possible, including standalone kitchen modules and integrated kitchen modules. The integrated kitchen module fits within the conventional kitchen area of ​​a typical home. The kitchen module operates in at least two modes: robotic mode and normal (manual) mode.

[0055] Machine learning—refers to techniques by which software components or programs improve their performance based on experience and feedback. One type of machine learning frequently used in robotics is reinforcement learning, in which desirable actions are rewarded and undesirable actions are punished. Another type is example-based learning, in which past solutions, e.g., action sequences by a human teacher or the robot itself, along with any constraints or reasons for these solutions, are stored and then applied or reused in new settings. Still other types of machine learning exist, such as inductive and transductive methods.

[0056] Miniature Manipulation (MM) - Generally, MM refers to the performance of one or more behaviors or tasks, at any number or combination and at various levels of descriptive abstraction, by a robotic device executing commanded movement sequences under sensor-driven computer control, acting through one or more hardware-based elements and guided by one or more software controllers, to obtain the task performance level required to reach near-optimal levels of performance within an acceptable execution fidelity threshold. Acceptable execution fidelity thresholds are task-dependent and therefore defined for each task (also called "domain-specific application"). In the absence of a task-specific threshold, a typical threshold is 0.001 (0.1%) of optimal performance. In one embodiment from a robotics perspective, the term MM refers to a set of well-defined, pre-programmed actuator action sequences and sensory feedback in a robot's task execution behavior defined by performance and execution parameters (such as variables, constants, controller types, and controller behaviors) used in one or more low-level to high-level control loops to achieve a desired movement / interaction behavior for one or more actuators ranging from individual actuations to serial and / or parallel sequences of coordinated multi-actuator movement (position and velocity) / interaction (force and torque) to achieve a specific task with desired performance physics metrics. MMs can be combined in various ways by combining low-level MM behaviors in series and / or parallel to achieve very complex application-specific task behaviors at a very high level of abstraction (task description). In another embodiment from a software / mathematics perspective, the term MM refers to a combination (or sequence) of one or more steps that achieve a basic functional outcome within an optimal outcome threshold (example thresholds are within optimal values ​​of 0.1, 0.01, 0.001, or 0.0001, with 0.001 being a preferred default). Each step can be a sensing actuation, actuator movement, or behavior primitive corresponding to another (smaller) MM, similar to a computer program composed of basic coding steps and other computer programs that can stand alone or serve as subroutines. For example, a MM could be an egg grasping step composed of the motor actions needed to sense the location and orientation of an egg, then extend the robotic arm, move the robotic fingers into the correct configuration, and apply the correct subtle amount of force for grasping, all of which are primitive actions. Another MM can be cracking an egg with a knife, which includes a grasping MM with one robotic hand, followed by a MM with the other hand grasping a knife, followed by a primitive action of hitting the egg with the knife with a predefined force and a predefined location. High-Level, Task-Specific Behavior—Refers to behavior that can be described in natural, human-understandable language and is readily recognizable by humans as a clear, necessary step in accomplishing or achieving a high-level goal. It should be understood that to successfully achieve a high-level task-specific goal, many other lower-level behaviors and actions / movements must occur, sometimes in series, sometimes in parallel, and sometimes even in an iterative manner, with many degrees of freedom individually actuated and controlled. Thus, to achieve more complex task-specific behavior, higher-level behaviors are composed of multiple levels of lower-level MM. For example, a command to play the first note of the first measure of a particular piece of music on a harp assumes that this note is known (i.e., g-flat), but requires the execution of low-level MMs involving the curling of specific fingers, the movement of the whole hand or palm so that the fingers contact the correct string, and the subsequent strumming / plucking of the chord at the proper speed and movement to obtain the correct note. Because all these individual MMs of the isolated fingers and / or hand / palm are not aware of the overall goal (producing a specific note on a specific instrument), all of these MMs can be thought of as various lower-level MMs. The task-specific action of playing a specific note to obtain a desired sound is clearly a higher-level, application-specific task, but this action is aware of the overall goal and requires interactions between behaviors / movements to control all of the lower-level MMs required for successful completion. One can even define the steps of playing a specific musical score as a lower-level MM for the behaviors or instructions of the overall higher-level, application-specific task detailing the performance of an entire piano concerto, in which case the steps of playing individual notes can each be thought of as a lower-level MM structured by the composer's intended score. Low-Level Small-Scale Manipulation Behaviors—Basic elements, movements required as basic building blocks to achieve higher-level task-specific movements / movements or behaviors. Low-level behavior blocks or elements can be combined in one or more serial or parallel fashions to achieve complex mid-level or high-level behaviors. For example, curling a single finger at all knuckles is a low-level behavior because it can be combined with curling all other fingers on the same hand in a specific sequence, triggered to start / stop based on contact / force thresholds, to achieve the high-level behavior of grasping, regardless of whether it is a tool or implement. Thus, the high-level task-specific behavior of grasping is composed of serial / parallel combinations of sensory-driven low-level behaviors by each of the five fingers on the hand. Thus, all behaviors can be decomposed into basic low-level movements / movements that, when combined in a specific fashion, achieve the high-level task behavior. The classification or boundary between low-level and high-level behavior can be somewhat arbitrary, but one way to conceive it is that movements, actions, or behaviors that humans tend to perform as part of a more human-language task (such as "grasping a tool") without much cognitive thought can and should be considered low-level (such as making contact and curling fingers around the tool / implement until sufficient contact force is obtained). In terms of machine-language implementation languages, all actuator-specific instructions without awareness of the high-level task are certainly considered low-level behavior.

[0057] Model Elements and Classification - refers to one or more software-based computer programs that can understand elements in a scene as items used or needed for different parts of a task, such as bowls for mixing and spoons for stirring. Multiple elements in a scene or world model can be classified into groups, allowing for rapid planning and task execution.

[0058] Motion Primitives - refers to motion actions that define different levels / domains of detailed motion stages, for example a high level motion primitive would be grasping a cup, while a low level motion primitive would be rotating the wrist by 5 degrees.

[0059] Multimodal Sensing Unit—refers to a sensing unit comprised of multiple sensors capable of sensing and detecting multiple modes or electromagnetic bands or spectrums, particularly capable of capturing three-dimensional position and / or movement information. The electromagnetic spectrum can range from low to high frequencies and need not be limited to those perceived by humans. Further modes can include, but are not limited to, other physical senses such as touch, smell, etc.

[0060] Number of axes - Three number of axes are required to reach any point in space. Three additional rotational axes (yaw, pitch, and roll) are required to fully control the orientation of the end of the arm (i.e., the wrist).

[0061] Parameter—refers to a variable that can take on a number or a range of numbers. Three types of parameters are particularly relevant: parameters in instructions to a robotic device (e.g., force or distance in arm movement), user-configurable parameters (e.g., preference for medium vs. well-done meat), and chef-defined parameters (e.g., setting oven temperature to 350°F).

[0062] Parameter Adjustment—refers to the process of changing parameter values ​​based on input. For example, changing parameters of instructions to a robotic device can be based on, but not limited to, the properties of ingredients (e.g., size, shape, orientation), the position / orientation of kitchen tools, equipment, appliances, and the speed and duration of a mini-manipulation.

[0063] Payload or carrying capacity - refers to how much weight a robotic arm can carry and hold (or even accelerate) against gravity as a function of the location of its endpoint.

[0064] Physical Reasoning - refers to a software-based computer program that relies on geometrically inferred data and can use physical information (density, texture, general geometry and shape) to help the inference engine (program) better model an object and further predict how this object will behave in the real world, especially when grasped and / or manipulated / handled.

[0065] Raw Data—refers to all measured and inferred sensory data and representational information collected as part of the Chef Studio recipe generation process while watching / monitoring a human chef preparing a dish. Raw data can range from simple data points such as clock time to oven temperatures (over time), camera images, 3D laser-generated scene representation data, appliances / equipment used, tools employed, ingredients dispensed (type and amount), and even timing. All information that the Studio Kitchen collects from its built-in sensors and stores in raw, time-stamped form is considered raw data. Raw data is later used by other software processes to generate higher levels of understanding and recipe process understanding, transforming the raw data into further processed / interpreted data with time stamps.

[0066] Robotic Device—refers to a set of robotic sensors and robotic effectors. The effectors include one or more robotic arms and one or more robotic hands for operation within the standardized robotic kitchen. The sensors include cameras, distance sensors, and force sensors (tactile sensors) that send their information to a processor or set of processors that control the effectors.

[0067] Recipe Cooking Process—refers to a robotic script that contains abstract and detailed level instructions for a collection of programmable hard automation devices to enable the computer-controllable devices to perform sequenced operations within their environment (e.g., a kitchen complete with ingredients, tools, utensils, and appliances).

[0068] Recipe Script—refers to a recipe script as a timed sequence containing a structure and list of commands and execution primitives (from simple to complex command software) that, when executed in a given sequence by the robotic kitchen elements (robot arms, automated equipment, appliances, tools, etc.), will result in the proper reproduction and creation of the same dish prepared by a human chef in a studio kitchen. Such a script is a timed sequence and is equivalent to the sequence employed by a human chef to create the dish, albeit in a representation appropriate to and understandable by the computer-controlled elements in the robotic kitchen.

[0069] Recipe Speed ​​Execution—refers to managing the timeline of recipe step execution in preparing a food dish by replicating chef movements, where recipe steps include standardized food preparation operations (e.g., standardized cookware, standardized equipment, kitchen processors, etc.), MM, and cooking of non-standardized objects.

[0070] Repeatability - refers to the preset margin of tolerance in how accurately a robot arm / hand can repeatedly return to a programmed position. If the technical specifications in the control memory call for the robot hand to move to a certain XYZ position within ±0.1 mm of this position, then repeatability is measured in terms of returning to within ±0.1 mm of the desired / commanded taught position.

[0071] Robotic Recipe Script - refers to a computer generated sequence of machine understandable instructions relating to the proper sequence of robotic / hard automation execution of the cooking steps in a recipe that closely mimics the steps required to arrive at the same end product as if cooked by a chef.

[0072] Robotic Costume - An external instrumented device or garment such as a glove, garment with camera-friendly markers, articulated exoskeleton, etc. that is used to monitor and track the chef's movements and activities during all aspects of the recipe cooking process within the chef studio.

[0073] Scene Modeling—refers to a software-based computer program that can view a scene within one or more camera fields of view and detect and identify objects important to a particular task. These objects can be pre-taught and / or can be part of a computer library with known physical attributes and usage applications.

[0074] Smart Kitchen Cookware / Equipment—refers to an item of kitchen cookware (e.g., a pot or pan) or an item of kitchen equipment (e.g., an oven, grill, or faucet) that has one or more sensors that prepare a food dish based on one or more graphical curves (e.g., a temperature curve, a humidity curve, etc.).

[0075] Software Abstraction Food Engine - refers to a software engine, defined as a collection of software loops or programs that work together to process input data and generate a certain desired output data set to be consumed by other software engines or by an end user through any form of textual or graphical output interface. An abstraction software engine is a software program focused on taking large amounts of input data from known sources within a particular domain (e.g., 3D distance measurements that form a data cloud of 3D measurements captured by one or more sensors) to identify, detect, and classify data readings associated with objects in 3D space (e.g., tabletops, cooking pots, etc.), and then processing these data to arrive at an interpretation of the data in a different domain (e.g., detecting and recognizing tabletops within a data cloud based on data having the same vertical data value, etc.). Essentially, the abstraction process is defined as taking a large data set from one domain, inferring structure (such as geometry) at a higher spatial level (abstraction of data points), then abstracting the inferences even further, identifying objects (such as pots) from the abstracted data set, and identifying real-world elements in the image that can then be used by other software engines to make further decisions (such as handling / manipulation decisions for the primary object). A synonym for "software abstraction engine" in this application could be "software interpretation engine" or even "computer software processing and interpretation algorithms."

[0076] Task Reasoning - refers to a software-based computer program that can analyze a task description and break it down into a sequence of multiple machine-executable (robot or hard automation system) steps to achieve a specific end result defined within the task description.

[0077] 3D World Object Modeling and Understanding - refers to a software-based computer program that can use sensory data to create time-varying 3D models of all surfaces and spatial regions so that it can detect, identify, and classify objects within those surfaces and regions and understand the use and applications of those objects.

[0078] Torque Vector - refers to the twisting force on the robotic appendage, including its direction and magnitude.

[0079] Volumetric Object Inference (Engine) - refers to a software-based computer program that can enable the identification of three-dimensional characteristics of one or more objects using geometric data and edge information, as well as other sensory data (color, shape, texture, etc.) to aid in the object identification and classification process.

[0080] For further information regarding robotic devices and MM library replication, please see co-pending U.S. Non-Provisional Patent Application No. 14 / 627,900, entitled "Methods and Systems for Food Preparation in a Robotic Cooking Kitchen."

[0081] FIG. 1 is a system diagram illustrating an overall robotic food preparation kitchen 10 having robotic hardware 12 and robotic software 14. The overall robotic food preparation kitchen 10 includes robotic food preparation hardware 12 and robotic food preparation software 14 that work together to perform robotic functions for food preparation. The robotic food preparation hardware 12 includes a computer 16 that controls the various operations and movements of a standardized kitchen module 18 (which generally operates within an instrumented environment having one or more sensors), a multi-modal three-dimensional sensor 20, a robotic arm 22, a robotic hand 24, and a capture glove 26. The robotic food preparation software 14 works with the robotic food preparation hardware 12 to capture the chef's movements and replicate the chef's movements through the robotic arms and hands in preparing the food dish to obtain the same or substantially the same food dish result (e.g., same taste, same smell, etc.) that will taste the same or substantially the same as if the food dish were prepared by a human chef.

[0082] The robotic food preparation software 14 includes a multi-modal 3D sensor 20, a capture module 28, a calibration module 30, a transformation algorithm module 32, a replication module 34, a quality check module 36 with a 3D vision system, a result module 38, and a learning module 40. The capture module 28 captures the chef's movements as he prepares a food dish. The calibration module 30 calibrates the robotic arm 22 and the robotic hand 24 before, during, and after the cooking process. The transformation algorithm module 32 is configured to convert recorded data from the chef's movements collected in the chef studio into recipe correction data (or modified data) for use in the robotic kitchen, where the robotic hand replicates the food preparation of the chef's dish. The replication module 34 is configured to replicate the chef's movements in the robotic kitchen. The quality check module 36 is configured to perform quality check functions on food dishes prepared by the robotic kitchen before, during, or after the food preparation process. The outcome module 38 is configured to determine whether a food dish prepared by a robotic arm and hand pair in the robotic kitchen will taste the same or substantially the same as if prepared by a chef. The learning module 40 is configured to provide learning capabilities to the computer 16 that controls the robotic arms and hands.

[0083] 2 is a system diagram illustrating a first embodiment of a food robotic cooking system including a chef studio system and a home robotic kitchen system for preparing dishes by replicating the processes and movements of a chef's recipe. The robotic kitchen cooking system 42 includes a studio kitchen 44 (also referred to as a "chef studio kitchen") that transfers one or more software-recorded recipe files 46 to a robotic kitchen 48 (also referred to as a "home robotic kitchen"). In one embodiment, both the chef kitchen 44 and the robotic kitchen 48 use the same standardized robotic kitchen module 50 (also referred to as a "robotic kitchen module," "robotic kitchen domain," "kitchen module," or "kitchen domain") to maximize the detail of the replication of the steps for preparing a food dish and thereby reduce variables that may contribute to deviations between a food dish prepared in the chef kitchen 44 and one prepared by the robotic kitchen 48. The chef 49 wears a robotic glove or robotic costume with external sensing devices to capture and record his or her cooking movements. The standardized robotic kitchen 50 includes a computer 16 for controlling various computing functions, which includes a memory 52 for storing one or more software recipe files from sensors on gloves or costumes 54 to capture chef movements, and a robotic cooking engine (software) 56. The robotic cooking engine 56 includes a movement analysis and recipe abstraction and sequencing module 58. Typically, the robotic kitchen 48 operates autonomously using a pair of robotic arms and hands, and an optional user 60 activates or programs the robotic kitchen 48. The computer 16 in the robotic kitchen 48 includes a hard automation module 62 for steering the robotic arms and hands, and a recipe replication module 64 for replicating the chef movements from a software recipe (ingredients, sequence, process, etc.) file.

[0084] The standardized robotic kitchen 50 is designed to detect, record, and simulate a chef's cooking movements and control critical parameters such as temperature over time, as well as process execution in the robotic kitchen using specified appliances, equipment, and tools. The chef kitchen 44 provides a computing kitchen environment 16 with sensor-equipped gloves or sensor-equipped costumes to record and capture the chef's 49 movements in food preparation for a specific recipe. Once the chef's 49 movements and recipe process for a specific dish are recorded in a software recipe file in memory 52, the software recipe file is transferred from the chef kitchen 44 to the robotic kitchen 48 over a communication network 46, whereby a user (optional) 60 can purchase one or more software recipe files or subscribe to the chef kitchen 44 as a member to receive new software recipe files or periodic updates of existing software recipe files. The domestic robotic kitchen system 48 serves as a robotic computing kitchen environment in residential homes, restaurants, and other locations where a kitchen is built in for a user 60 to prepare food. The home robotic kitchen system 48 includes a robotic cooking engine 56 having one or more robotic arms and hard automation devices to replicate the chef's cooking actions, processes, and movements based on the software recipe file received from the chef studio system 44.

[0085] The Chef Studio 44 and the robotic kitchen 48 represent a complex linked teach-play system with multiple levels of execution fidelity. The Chef Studio 44 generates a high-fidelity process model of how to prepare a professionally cooked dish, while the robotic kitchen 48 is the execution / replication engine / process for the recipe script created through the chef working within the Chef Studio. Standardization of the robotic kitchen modules is a means to increase execution fidelity and success / assurance.

[0086] Different levels of fidelity for recipe execution depend on the correlation of sensors and equipment between those in the chef studio 44 and those in the robotic kitchen 48 (in addition to, of course, being ingredient dependent). Fidelity can be defined as a dish that tastes identical (indistinguishably identical) to that prepared by a human chef at one end of the spectrum (perfect replication / execution), whereas at the opposite end, the dish can have one or more substantial or fatal defects with quality (overcooked meat or pasta), taste (burnt taste), edibility (improper consistency), or even health implications (such as undercooked meat such as chicken / pork with Salmonella exposure).

[0087] A robotic kitchen with identical hardware, sensors, and movement systems that can replicate movements and processes similar to those from the chef recorded during the chef-studio cooking process is likely to produce a high-fidelity outcome. The implication in this case is that the configuration needs to be identical, which has cost and spatial implications. However, the robotic kitchen 48 can still be implemented with more standardized computer-controlled or computer-monitored elements (such as sensor-equipped pots, networked ovens, and other appliances), thereby requiring significantly more sensor-based understanding to enable more complex execution monitoring. In this case, there is arguably less guarantee of having an identical outcome to that from the chef, due to increased uncertainty regarding key elements (such as correct ingredient amounts, cooking temperatures, etc.) and processes (such as the use of a stirrer / masher in the case of a blender, which is not available in the robotic home kitchen).

[0088] The emphasis in this disclosure is that the concept of a chef studio 44 combined with a robotic kitchen is a general design concept. The level of the robotic kitchen 48 can vary from a domestic kitchen equipped with a set of arm and environmental sensors to an identical replica of a studio kitchen with a set of arms and joint movements, tools and appliances, and ingredient supplies that can replicate a chef's recipe in a nearly identical fashion. The only competing variable is the quality of the end result or dish in terms of quality, presentation, taste, edibility, and healthiness.

[0089] A possible way to mathematically describe the above correlation between recipe outcome and input variables in the robotic kitchen can best be described by the following equation: F recipe-outcome =F studio (I,E,P,M,V)+F RobKit (E f ,I,R e ,P mf ) In the formula, F studio = Chef Studio recipe script fidelity, F RobKit = Recipe script execution by the robot kitchen, I = ingredients, E = Equipment, P = process, M = Method V = variable (temperature, time, pressure, etc.) E f = equipment fidelity, R e = reproduction fidelity, P mf = Process monitoring fidelity.

[0090] The above formula is the result of a robot-prepared recipe that will be prepared and served by a human chef (F recipe-outcome) based on the ingredients (I) used, the equipment (E) available to carry out the chef's process (P), and the method (M) by properly capturing all the key variables (V) during the cooking process, the recipe is properly captured and the level (F) expressed by Chef Studio 44. studio ) and furthermore, mainly the use of appropriate ingredients (I) and the level of equipment fidelity in the robotic kitchen compared to that in the chef studio (E f ) and the level at which recipe scripts can be reproduced in the robot kitchen (R e ) and the highest possible process monitoring fidelity (P mf ) and the extent to which there is an ability and need to monitor and take corrective action to obtain RobKit ) relates to the extent to which the robotic kitchen can represent the process of replicating / executing the robotic recipe script.

[0091] Function (F studio ) and (F RobKit ) can be any combination of linear or non-linear functional expressions with constants, variables, and any form of algorithmic relationship. Examples of such algebraic expressions for both functions can be the following expressions:

[0092] F studio = I(fct.sin(temperature)) + E(fct.cooking table 15) + P(fct.circle(spoon)) + V(fct.0.5 time)

[0093] The above equation illustrates that the fidelity of the preparation process is related to the temperature of ingredients, which varies sinusoidally over time in a refrigerator, how quickly ingredients can be heated at a specific scale on the countertop at a specific station, and how well one can move a spoon in a circular path of a certain amplitude and frequency; and that to maintain the fidelity of the preparation process, it is necessary to perform this process no slower than half the speed of a human chef.

[0094] FRobKit =Ef,(Countertop 2, Size) + I(1.25 Size + Linear (Temperature)) + R e (Movement transition)+P mf (Sensor set compatibility)

[0095] The above equation depicts that the fidelity of the replication process in the robotic kitchen is related to the type and layout of appliances and size of heating elements for a particular cooking zone, the size and temperature progression of the food item being seared and cooked while also maintaining the motion progression of any stirring and immersion motions for a particular step, such as searing or stirring a mousse, and whether the correspondence between the sensors in the robotic kitchen and the sensors in the chef studio is high enough to trust that the monitoring sensor data is accurate and detailed enough to give adequate monitoring fidelity of the cooking process in the robotic kitchen during all steps in the recipe.

[0096] The outcome of a recipe is not only a function of the fidelity with which the human chef's cooking steps / methods / processes / skills were captured by Chef Studio, but also the fidelity with which the robotic kitchen can execute these steps / methods / processes / skills, with each of these functions having key components that influence the respective subsystem performance.

[0097] 3 is a system diagram illustrating one embodiment of a standardized robotic kitchen 50 for food preparation by recording a chef's movements in preparing a food dish and replicating the food dish with robotic arms and hands. In this context, the term "standardized" (or "standard") means that the specifications of a component or feature are pre-set, as will be explained below. The computer 16 is communicatively coupled to multiple kitchen elements in the standardized robotic kitchen 50, including a three-dimensional vision sensor 66, a retractable safety screen 68 (e.g., glass, plastic, or other type of protective material), a robotic arm 70, a robotic hand 72, standardized cookware / equipment 74, standardized cookware with sensors 76, standardized handles or standardized cookware 78, standardized handles and standardized tools 80, a standardized hard automation dispenser 82 (also referred to as a "robotic hard automation module"), a standardized kitchen processor 84, standardized containers 86, and standardized food storage 88 in a refrigerator.

[0098] The standardized (hard) automation dispenser 82 is a device or series of devices programmable and / or controllable by the cooking computer 16 to dispense or apply pre-packaged (known) or specialized amounts of key ingredients, such as spices (salt, pepper, etc.), liquids (water, oil, etc.), or other dry ingredients (flour, sugar, etc.), to the cooking process. The standardized hard automation dispenser 82 can be installed at a specific station or can be accessed and triggered by a robot to dispense according to a recipe sequence. In other embodiments, the robotic hard automation module can be combined with other modules, robotic arms, or cooking tools or sequenced in series or parallel. In this embodiment, the standardized robotic kitchen 50 includes a robotic arm 70 and a robotic hand 72 controlled by the robotic food preparation engine 56 according to a software recipe file stored in memory 52 to replicate the chef's detailed movements in preparing a dish and produce a dish that tastes the same as if the chef had prepared it himself. The 3D vision sensor 66 provides the capability to generate a visual 3D model of kitchen activity and enable 3D modeling of objects by scanning the kitchen volume to assess dimensions and objects within the standardized robotic kitchen 50. The retractable safety glass 68 comprises a transparent material on the robotic kitchen 50 that, when turned on, extends the safety glass around the perimeter of the robotic kitchen to protect nearby personnel from the movement of the robotic arms 70 and robotic hands 72, hot water and other liquids, steam, flames, and other dangerous influences. The robotic food preparation engine 56 is communicatively coupled to the electronic memory 52 to obtain a software recipe file previously sent from the chef studio system 44, for which the robotic food preparation engine 56 is configured to execute the preparation process as indicated therein and replicate the chef's cooking method and process.The combination of the robotic arm 70 and the robotic hand 72 serves to replicate the chef's detailed movements in preparing a dish so that the resulting food dish tastes identical (or substantially identical) to the same food dish prepared by the chef. The standardized cooking equipment 74 includes several types of cooking implements 46 incorporated as part of the robotic kitchen 50, including, but not limited to, stoves / induction cooktops / cooktops (electric cooktops, gas cooktops, induction cooktops), ovens, grills, cooking steamers, and microwave ovens. The standardized cookware and sensors 76 are used as an embodiment for recording food preparation steps based on sensors on the cookware and cooking steps of the food dish based on sensory cookware, including a sensory pot, a sensory pan, a sensory oven, and a sensory charcoal grill. The standardized cookware 78 includes a frying pan, a sauté pan, a grill pan, a multi-pot, a roaster, a wok, and a braiser. The robotic arm 70 and the robotic hand 72 manipulate standardized handles and standardized implements 80 in the cooking process. In one embodiment, the robotic hand 72 is equipped with standardized handles that attach to the fork tines, knife tines, and spoon tines for selection as needed. Standardized hard automated dispensers 82 are integrated into the robotic kitchen 50 to dispense preferred staples and common / recurring ingredients, which are easily measured / dosed / or pre-packaged. Standardized containers 86 are storage locations for keeping food at room temperature. Standardized refrigerator containers 88 refer to, but are not limited to, refrigerators with identified containers for storing fish, meat, vegetables, fruits, milk, and other perishable items. Containers in the standardized containers 86 or standardized storage 88 can be coded with a container identifier, from which the robotic food preparation engine 56 can determine the type of food in the container based on the container identifier. Standardized containers 86 provide storage space for non-perishable food items, such as salt, pepper, sugar, oil, and other spices. The sensor-equipped standardized cookware 76 and cookware 78 can be stored on shelves or in storage for use by the robotic arm 70 to select cooking tools to prepare a dish.Typically, raw fish, raw meat, and vegetables can be pre-cut and stored in the identified standardized storage 88. The kitchen countertop 90 provides a platform for the robotic arm 70 to perform any necessary handling steps for the meat or vegetables, which may or may not include cutting or shredding. The kitchen faucet 92 provides a kitchen sink space for washing or cleaning food in preparation for cooking. Once the robotic arm 70 has completed the recipe process for preparing a dish and the dish is ready for serving, the dish is placed on the serving counter 90, which further allows the robotic arm 70 to enhance the dining environment by adjusting surrounding settings such as utensils, wine glasses, and placement of wine selected to complement the food. One embodiment of the equipment in the standardized robotic kitchen module 50 is a commercial set to enhance universal appeal in preparing various types of dishes.

[0099] The standardized robotic kitchen module 50 has as one objective the standardization of the kitchen module 50 and various components associated therewith to ensure consistency in both the chef kitchen 44 and the robotic kitchen 48 so as to maximize the level of detail in the recipe replication and, at the same time, minimize the risk of deviation from the detailed replication of the recipe dish between the chef kitchen 44 and the robotic kitchen 48. One main objective of standardizing the kitchen module 50 is to obtain the same cooking process result (or the same dish) between the first food dish prepared by the chef and the subsequent replication of the same recipe process by the robotic kitchen. Considering a standardized platform in the standardized robotic kitchen module 50 between the chef kitchen 44 and the robotic kitchen 48 has some important requirements, such as the same timeline, the same program or mode, and quality checks. The same timeline in the standardized robotic kitchen 50, where the chef prepares the food dish in the chef kitchen 44 and the replication process by the robotic hand is carried out in the robotic kitchen 48, refers to the same operation sequence, the same start and end time of each operation, and the same speed at which the object is moved during the handling operation. The same program or mode in the standardized robotic kitchen 50 involves the use and operation of standardized equipment during the recording and execution stages of each operation. The quality check involves a 3D vision sensor in the standardized robotic kitchen 50 that monitors and adjusts each operation action in real time to correct any deviations during the food preparation process and avoid defective results. The adoption of the standardized robotic kitchen module 50 reduces and minimizes the risk of not achieving the same results between a food dish prepared by a chef and one prepared by the robotic kitchen using robotic arms and hands.Without standardization of the robotic kitchen modules and components within the robotic kitchen modules, the amplified variability between the chef kitchen 44 and the robotic kitchen 48 increases the risk of not being able to achieve the same results between the food dishes prepared by the chef and the food dishes prepared by the robotic kitchen, as more sophisticated and complex adjustment algorithms would be required with different kitchen modules, different kitchen equipment, different kitchen implements, different kitchen tools, and different ingredients between the chef kitchen 44 and the robotic kitchen 48.

[0100] The standardized robotic kitchen module 50 includes standardization in many aspects. First, the standardized robotic kitchen module 50 includes standardized positions and orientations (in the XYZ coordinate plane) of all types of kitchen utensils, kitchen containers, kitchen tools, and kitchen equipment (along with standardized fixation holes at the kitchen module and device locations). Second, the standardized robotic kitchen module 50 includes standardized dimensions and structures of the cooking space area. Third, the standardized robotic kitchen module 50 includes a standardized set of equipment, such as an oven, stove, dishwasher, and faucet. Fourth, the standardized robotic kitchen module 50 includes kitchen utensils, cooking tools, cooking devices, containers, and refrigerator food storage that are standardized in terms of shape, size, structure, material, capacity, etc. Fifth, in one embodiment, the standardized robotic kitchen module 50 includes a standardized universal handle for handling all kitchen utensils, tools, appliances, containers, and equipment, allowing the robotic hand to hold them in a unique, correct position while avoiding any improper grip or incorrect orientation. Sixth, the standard robotic kitchen module 50 includes a standardized robotic arm and a standardized robotic hand for versatile manipulations. Seventh, the standard robotic kitchen module 50 includes a standardized kitchen processor for standardized ingredient manipulations. Eighth, the standard robotic kitchen module 50 includes a standardized 3D vision device for creating dynamic 3D visual data, as well as other possible standard sensors for recipe recording, execution tracking, and quality check functions. Ninth, the standard robotic kitchen module 50 includes standardized type, volume, size, and weight for each ingredient during a specific recipe execution.

[0101] 4 is a system diagram illustrating one embodiment of a robotic cooking engine (also referred to as a "robotic food preparation engine") 56 for use with computer 16 in chef studio system 44 and home robotic kitchen system 48. Other embodiments may have modifications, additions, or variations of modules in the robotic cooking engine 56 in chef kitchen 44 and robotic kitchen 48. The robotic cooking engine 56 includes an input module 51, a calibration module 94, a quality check module 96, a chef movement recording module 98, a cookware sensor data recording module 100, a memory module 102 for storing software recipe files, a recipe abstraction module 104 for generating machine module-specific sequenced operation profiles using recorded sensor data, a chef movement replication software module 106, a cookware sensory replication module 108 using one or more sensory curves, a robotic cooking module 110 (computer-controlled to perform standardized operations, mini-manipulations, and non-standardized objects), a real-time adjustment module 112, a learning module 114, a mini-manipulation library database module 116, a standardized kitchen operation library database module 118, and an output module 120. These modules are communicatively coupled via a bus 122.

[0102] The input module 51 is configured to receive any type of input information, such as a software recipe file sent from another computing device. The calibration module 94 is configured to calibrate itself with the robotic arm 70, the robotic hand 72, and other kitchen utensils and equipment within the standardized robotic kitchen module 50. The quality check module 96 is configured to determine the quality and freshness of raw meat, raw vegetables, and dairy-related ingredients when the raw foods are retrieved for cooking, as well as check the quality of the raw foods when they are accepted into the standardized food storage 88. The quality check module 96 may also be configured to perform sensory-based quality inspections of objects, such as food smell, food color, food taste, and food appearance or presentation. The chef movement recording module 98 is configured to record the sequence and detailed movements of a chef as he or she prepares a food dish. The cookware sensor data recording module 100 is configured to record sensory data from cookware equipped with sensors located within various zones therein (such as a sensor pan, sensor grill, or sensor oven), thereby generating one or more sensory curves. The result is the generation of a sensory curve, such as a temperature (and / or humidity) curve, that reflects the temperature fluctuations of the cookware over time for a particular dish. Memory module 102 is configured as a repository for storing either software recipe files for chef recipe movement replication or other types of software recipe files containing sensory data curves. Recipe abstraction module 104 is configured to generate machine module-specific sequenced operating profiles using the recorded sensor data. Chef movement replication module 106 is configured to replicate the chef's detailed movements in preparing a dish based on the software recipe files stored in memory 52. ​​Utensil sensory replication module 108 is configured to replicate a food dish preparation by tracing the characteristics of one or more recorded sensory curves generated using sensor-equipped standardized cookware 76 when chef 49 prepared the dish.The robotic cooking module 110 is configured to autonomously control and manipulate standardized kitchen operations, mini-operations, non-standardized objects, and various kitchen tools and equipment in the standardized robotic kitchen 50. The real-time adjustment module 112 is configured to make real-time adjustments to variables associated with a particular kitchen operation or mini-operation to generate a resulting process that is a detailed replica of a chef's movements or a detailed replica of a sensory curve. The learning module 114 is configured to provide learning capabilities to the robotic cooking engine 56 to optimize detailed replication of a food dish as if it were prepared by a chef, using methods such as case-based (robotic) learning, when preparing a food dish with the robotic arm 70 and robotic hand 72. The mini-operation library database module 116 is configured to store a first database library of mini-operations. The standardized kitchen operation library database module 118 is configured to store a second database library of standardized kitchen utensils and information on how to manipulate the standardized kitchen utensils. The output module 120 is configured to send output computer files or control signals external to the robotic cooking engine.

[0103] 5A is a block diagram illustrating the Chef Studio recipe creation process 124 showing some major functional blocks that support the use of extensive multimodal sensing to create recipe instruction scripts for the robotic kitchen. Sensor data from multiple sensors, such as (but not limited to) smell 126, video camera 128, infrared scanner and rangefinder 130, stereo (or even trinocular) camera 132, haptic glove 134, articulated laser scanner 136, virtual world goggles 138, microphone 140, or exoskeleton motion suit 142, human voice 144, contact sensor 146, and yet other forms of user input 148, are used to collect data through a sensor interface module 150. This data, including possible human user input 148 (e.g., chef screen touch and voice input), is acquired and filtered 152, after which multiple (parallel) software processes utilize the temporal and spatial data to generate data that is used to populate the machine-specific recipe creation process. The sensors may not be limited to capturing the position and / or movement of humans, but may also capture the position, orientation, and / or movement of other objects within the standardized robotic kitchen 50.

[0104] These individual software modules generate information such as (i) chef location and cooking station ID through Location and Configuration module 154, (ii) arm configuration (through the torso), (iii) when and how tools are handled, (iv) utensils used and location on the station through Hardware and Variable Abstraction module 156, (v) process performed with these utensils, (vi) variables that need to be monitored (temperature, lid on / off, stirring, etc.) through Processing module 158, (vii) time (start / done, type) distribution, (viii) type of process being applied (stirring, folding, etc.), and (ix) ingredients added (type, amount, readiness, etc.) through Cooking Sequence and Cooking Process Abstraction module 160 (but are not thereby limited to only these modules).

[0105] All this information is then used to create a machine-specific recipe instruction set (not only for the robotic arm, but also for the ingredient dispensers, tools, and implements, etc.) organized as a script of sequential / parallel overlapping tasks to be executed and monitored through a stand-alone module 162. This recipe script 164, along with the entire raw data set 166, is stored in a data storage module 168 and made accessible either to a remote robotic cooking station through a robotic kitchen interface module 170 or to a human user 172 through a graphical user interface (GUI) 174.

[0106] 5B is a block diagram illustrating one embodiment of the standardized chef studio 44 robotic kitchen 50 with the teach / playback process 176. The teach / playback process 176 describes the steps to capture the recipe execution process / method / skills of a chef 49 within the chef studio 44 where the chef uses a set of chef studio standardized equipment 74 and ingredients 178 required by the recipe to create a dish and performs a recipe execution 180 while being recorded and monitored. Raw sensor data is recorded 182 (for playback) and further processed to generate information at various levels of abstraction (tools / equipment used, techniques employed, start / end times / temperatures, etc.), which is then used to create a recipe script 184 for execution by the robotic kitchen 48.

[0107] The robotic kitchen 48 engages in a recipe replication process 106 with a profile that depends on whether the kitchen is of standardized or non-standardized type, as checked by process 186. The execution of the robotic kitchen depends on the type of kitchen available to the user. If the robotic kitchen uses the same / identical (at least functionally) equipment as used in the chef studio, the recipe replication process primarily uses raw data and reproduces it as part of the recipe-script execution process. However, if the kitchen differs from the (ideal) standardized kitchen, the execution engine will have to rely on abstracted data to generate kitchen-specific execution sequences in an attempt to achieve similar step-by-step results.

[0108] Whether known studio equipment 196 or mixed / atypical non-chef studio equipment 198 is used, the cooking process is continuously monitored by all sensor units in the robotic kitchen through a monitoring process 194, so the system can make corrections as needed, relying on a recipe progress check 200. In one embodiment of a standardized kitchen, the raw data is typically played back through the execution module 188 using chef studio-type equipment, and since there is a one-to-one correspondence between the teaching data set and the playback data set, the only adjustments expected are adjustments 202 in the script execution (repeat a certain step, go back to a certain step, slow down the execution, etc.). However, in the case of a non-standardized kitchen, the system will most likely have to modify and adjust the actual recipe itself and its execution through a recipe-script modification module 204 to accommodate available tools / appliances 192 that are different from those in the chef studio 44 or deviations from the measured recipe-script (meat cooking too slowly, hot spots in the pot burning the roux, etc.). The progress of the overall recipe script is monitored using a similar process 206 that differs depending on whether chef studio equipment 208 or mixed / kitchen equipment 210 is being used.

[0109] A non-standardized kitchen is less likely to produce food that approaches that prepared by a human chef than a standardized robotic kitchen with equipment and capabilities that mirror those used in a studio kitchen. The final subjective judgment is, of course, the human (or chef) tasting or quality assessment 212 that results in a (subjective) quality judgment 214.

[0110] 5C is a block diagram illustrating one embodiment of a recipe-script generation and abstraction engine 216 related to the structure and flow of the recipe-script generation process as part of a chef-studio recipe rehearsal by a human chef. The first step is for all available data that can be measured within the chef-studio 44, whether it be ergonomic data from the chef (arm / hand position and velocity, tactile finger data, etc.), status of cooking equipment (oven, refrigerator, dispenser, etc.), specific variables (cooktop temperature, ingredient temperature, etc.), equipment or tools being used (pots / pans, spatulas, etc.), or multi-spectral sensing equipment (including cameras, lasers, structured light systems, etc.), to be input, filtered, and time-stamped by the central computer system and then processed by a main process 218.

[0111] The data processing-mapping algorithm 220 uses simple (typically single unit) variables to determine where process actions are occurring (countertop and / or oven, refrigerator, etc.) and assigns a usage tag to every item / utensil / equipment that is being used, whether intermittently or continuously. The algorithm associates cooking steps (baking, grilling, adding ingredients, etc.) with specific time intervals and tracks which ingredients were added, where, when, and in what quantities. This (time-stamped) information dataset is then made available to the data merging process during the recipe-script generation process 222.

[0112] The Data Extraction and Mapping process 224 focuses primarily on two-dimensional information (such as from a single-lens camera) and extracts meaningful information from this information. To extract meaningful, more abstract, and descriptive information from each successive image, some algorithmic processing must be applied to the data set. Such processing steps may include, but are not limited to, edge detection, color, and texture mapping. Domain knowledge within the image, combined with object matching information (type and size) extracted from the Data Reduction and Abstraction process 226, is then used to identify and locate objects (such as equipment or food items) also extracted from the Data Reduction and Abstraction process 226, enabling association of the state (and all associated variables describing this state) and items with specific processing steps (such as frying, boiling, cutting, etc.) within the image. Once this data is extracted and associated with a particular image at a particular time, it can be passed to the Recipe Script Generation process 222 to develop the internal sequence and steps of the recipe.

[0113] The data reduction and abstraction engine (a set of software routines) 226 is responsible for reducing large 3D data sets and extracting important geometric and association information from them. The first step is to extract only the specific task-space area important to the recipe at that particular time from the large 3D data point cloud. Once the data set is trimmed, important geometric features are identified through a process known as template matching, allowing for the identification of items such as horizontal tabletops, cylindrical pots and pans, and arm and hand locations. Once typical known (template) geometric entities are determined within the data set, the object identification and matching process continues to distinguish all items (e.g., pots vs. pans), associate their dimensions (e.g., pot or pan size) and orientation, and place them within the 3D world model being constructed by the computer. All of this abstracted / extracted information is then shared with the data extraction and mapping engine 224, which then all feeds into the recipe-script generation engine 222.

[0114] The recipe-script generation engine process 222 is responsible for merging (combining / combining) all available data and sets into a structured sequential cooking script with clear process identifiers (prep, blanch, fry, wash, plate, etc.) and process-specific steps within each process, which can then be interpreted into a robotic kitchen machine-executable command script that is synchronized based on process completion, total cooking time, and cooking progress. Data merging will require, at a minimum, but not be limited to, the ability to take each (cooking) processing step and populate the step sequence to be performed with the properly associated elements (ingredients, equipment, etc.), the method and process to be used during the processing step, and associated critical control variables (set oven / cooktop temperature / settings) and monitoring variables (water temperature or meat temperature, etc.) to be maintained and checked to verify proper progression and execution. The merged data is then combined into a structured sequential cooking script that resembles a set of minimally descriptive steps (similar to a recipe found in a magazine), but with a much larger set of variables associated with each element of the cooking process (equipment, ingredients, process, method, variables, etc.) at any one point in the procedure. The final step is to take this sequential cooking script and transform it into an equally structured sequential script that can be interpreted by the set of machines / robots / equipment inside the robotic kitchen 48. It is this script that the robotic kitchen 48 uses to perform the automated recipe execution and monitoring steps.

[0115] All raw (unprocessed) and processed data, as well as the associated scripts (both structural sequential cooking sequence scripts and machine-executable cooking sequence scripts), are stored and time-stamped in the data and profile storage unit / process 228. It is from this database that the user can select a desired recipe through a GUI and have the robotic kitchen execute it through the automation execution and monitoring engine 230; this execution is continuously monitored by the engine's own internal automated cooking process to arrive at a perfectly plated and served dish, and any necessary adjustments and modifications to the script are generated by the engine and implemented by the robotic kitchen elements.

[0116] Figure 5D is a block diagram illustrating software elements for object manipulation (or object handling) in a standardized robotic kitchen, showing the structure and flow 250 of the object manipulation portion of the robotic kitchen's execution of a robot script using the concept of motion replication coupled with / using a mini-manipulation stage. For automated robotic arm / hand-based cooking to be feasible, monitoring every single joint in the arm and hand / finger is insufficient. Often, only the position and orientation of the hand / wrist are known (and can be replicated), in which case the object manipulation stage (identifying location, orientation, pose, grasp location, grasp technique, and task execution) requires local sensing, learned behavior, and hand and finger strategies to successfully complete the grasping stage / task manipulation stage. These motion profiles (sensor-based / sensor-driven) behaviors and sequences are stored in a mini-hand manipulation library software repository within the robotic kitchen system. A human chef can wear an arm exoskeleton or an instrumented / target-mounted motion vest, allowing a computer to determine the precise 3D position of the hand and wrist through built-in sensors or camera tracking. Even if all joints of the ten fingers on both hands were instrumented (more than 30 DoF (degrees of freedom) on both hands, which would be very inconvenient to wear and use and therefore unlikely to be used), simple reproduction based on the movement of all joint positions would not guarantee successful (interactive) object manipulation.

[0117] The Mini-Manipulation Library is a command software repository in which movement behaviors and processes—the arm / wrist / finger movements and sequences—to successfully complete specific abstract tasks (e.g., grasping a knife to slice, grasping a spoon to stir, grasping a pot with one hand and a spatula with the other to place under meat and flip it in the pan) are stored based on an offline learning process. This repository is constructed to contain learned sequences of successful sensor-driven movement profiles and ordered behaviors for the hand / wrist (possibly including corrections to arm position) to ensure successful completion of manipulations of objects (utensils, equipment, tools) and ingredients described in more abstract language, such as "grab a knife to slice vegetables," "crack an egg into a bowl," or "flip meat in a pan." The learning process is iterative and based on multiple trials of chef-taught movement profiles from the Chef Studio, which are then iteratively modified by an offline learning algorithm module executed until an acceptable execution sequence can be demonstrated. The mini-manipulation library (command software repository) is intended to be pre-populated (offline in advance) with all the elements necessary to enable the robotic kitchen system to successfully interact with all the equipment (utensils, tools, etc.) and main ingredients that require handling (steps beyond just dispensing) during the cooking process. Just as a human chef wears a glove with embedded tactile sensors (proximity, contact, contact location / force) on the fingers and palm, the robotic hand will be equipped with similar types of sensors in locations that allow it to use these sensor data to create, modify, and adjust movement profiles to successfully execute desired movement profiles and handling commands.

[0118] The object manipulation portion of the robotic kitchen cooking process (robotic recipe script execution software module for interactive manipulation and handling of objects within the kitchen environment) 252 is described in further detail below. Using the robotic recipe script database 254 (containing data in the form of raw abstracted cooking sequence machine-executable scripts), the recipe script execution module 256 steps through the specific recipe execution steps. The configuration playback module 258 selects configuration commands and passes them to the robotic arm system (torso, arm, wrist, and hand) controller 270, which in turn controls the physical systems to simulate the required configuration (joint position / velocity / torque, etc.) values.

[0119] The concept of being able to faithfully perform manipulation and handling tasks with proper environment interaction is made possible through (i) 3D world modeling and (ii) real-time process validation using mini-manipulations. Both the validation and manipulation phases are implemented through the addition of a robot wrist and robot hand configuration modifier 260. This software module uses data from a 3D world configuration modeler 262, which creates a new 3D world model at every sampling stage from sensory data provided by the multi-modal sensor unit, to verify that the robotic kitchen's system and process configurations match those required by the recipe script (database) and, if not, implement modifications to command system configuration values ​​to ensure the task is completed successfully. Additionally, the robot wrist and robot hand configuration modifier 260 also uses configuration modification input commands from a mini-manipulation motion profile executor 264. The hand / wrist (and possibly also arm) configuration correction data provided to the configuration corrector 260 is based on a small-scale manipulation motion profile executor 264 that knows from 258 what the desired configuration reproduction should be, but at the same time modifies this configuration reproduction based on pre-trained (and stored) data from a 3D object model library 266 and a configuration and ordering library 268 (built based on multiple iterative learning steps for all major object handling and processing steps).

[0120] The configuration modifier 260 continuously supplies modified command configuration data to the robotic arm system controller 270 while relying on the handling / manipulation verification software module 272 to verify not only that the operation is proceeding properly but also whether continued manipulation / handling is required. If continued manipulation / handling is required (answer to the question is "N"), the configuration modifier 260 re-requests updated configuration modifications (for the wrist, hand / fingers, and possibly the arm and possibly also the torso) from both the world modeler 262 and the mini-manipulation profile executor 264. The goal is simple: to verify that a successful manipulation / handling step or sequence has been successfully completed. The handling / manipulation verification software module 272 performs this check by using its knowledge of the recipe-script database F2 and the 3D world configuration modeler 262 to verify proper progression of the cooking step currently commanded by the recipe-script executor 256. If the progress is deemed successful, the increment recipe-script index process 274 notifies the recipe-script executor 256 to proceed to the next step in the recipe-script execution.

[0121] 6 is a block diagram illustrating a multi-modal sensing and software engine architecture 300 according to the present disclosure. One of the key features of autonomous cooking, enabling planning, execution, and monitoring of robotic cooking scripts, requires the use of multi-modal sensory inputs 302 that are used by multiple software modules to generate the data required for (i) understanding the world, (ii) modeling the scene and ingredients, (iii) planning the next step in the robotic cooking sequence, (iv) executing the generated plan, and (v) monitoring execution to verify proper operation, all of which occur in a continuous / iterative closed-loop manner.

[0122] The multi-mode sensor unit 302, which includes but is not limited to a video camera 304, an IR camera and rangefinder 306, a stereo (or even trinocular) camera 308, and a multi-dimensional scanning laser 310, provides multi-spectral sensing data (after being acquired and filtered in a data acquisition and filtering module 314) to a main software abstraction engine 312. This data is used in a scene understanding module 316 to perform several steps, including but not limited to constructing high-resolution and low-resolution (laser: high resolution, stereo camera: low resolution) three-dimensional surface volume regions of the scene with overlaid color and texture video information in the visual and IR spectrum, enabling edge detection and volumetric object detection algorithms to infer what elements are present in the scene, and enabling the use of shape / color / texture mapping and consistency mapping algorithms to proceed based on the processed data and provide the processed information to a kitchen cooking process equipment handling module 318. Within module 318, software-based engines are used to identify and 3D locate the positions and orientations of kitchen tools and utensils, and identify and tag identifiable food elements (meat, carrots, sauces, liquids, etc.) so that the computer can construct and understand the complete scene at a particular point in time and use this scene for next-stage planning and process monitoring. Engines required to perform such data and information abstraction include, but are not limited to, a grasp inference engine, a robotic motion and geometry inference engine, a physics inference engine, and a task inference engine. The output data from both engines 316 and 318 is then used to feed a scene modeler and content classifier 320, which creates a 3D world model with all the important content required to run the robotic cooking script executor.Once a fully populated world model is understood, it can be used to feed a motion and handling planner 322 to enable planning of movements and trajectories for the arm and attached end effectors (grasper, multi-fingered hand). (If robotic arm grasping and handling is required, the same data can be used to plan the steps to grasp and manipulate food items versus kitchen items, depending on the required grasp and positioning.) A subsequent execution sequence planner 324 creates the appropriate sequence of task-based commands for all individual robotic kitchen elements / automated kitchen elements, which is then used by the robotic kitchen actuation system 326. During the robotic recipe-script execution and monitoring phase, the entire sequence described above is repeated in a continuous closed loop.

[0123] 7A depicts a standardized kitchen 50, in this case acting as a chef studio where a human chef 49 creates and executes recipes while being monitored by a multi-modal sensor system 66 to enable the creation of recipe scripts. Included within the standardized kitchen are multiple elements required to execute recipes, including a main cooking module 350 that includes utensils 360, a countertop 362, a kitchen sink 358, a dishwasher 356, a tabletop mixer and blender (also referred to as a "kitchen blender") 352, an oven 354, and a refrigerator / freezer combination unit 364.

[0124] 7B depicts a standardized kitchen 50 configured as a standardized robotic kitchen in which a dual-arm robotic system, in this case having a vertical telescopic rotating torso joint 366 and equipped with two arms 70 and two wrist-jointed fingered hands 72, performs a recipe replication process defined in a recipe script. A multi-modal sensor system 66 continuously monitors the cooking steps performed by the robot during multiple stages of the recipe replication process.

[0125] 7C depicts the system involved in creating a recipe script by monitoring a human chef 49 during the entire recipe execution process. The same standardized kitchen 50 is used in chef studio mode, where the chef can operate the kitchen from either side of the task module. Multi-mode sensors 66 monitor and collect data; additional data is monitored and collected through gloves 370 worn by the chef and instrumented utensils 372 and equipment; all collected raw data is wirelessly transmitted to processing computer 16 for processing and storage.

[0126] 7D depicts a system contained within standardized kitchen 50 for replicating recipe script 19 using a dual-arm system with a telescoping rotating torso 374 and consisting of two arms 70, two robotic wrists 71, and two multi-fingered hands 72 with embedded sensing skin and tip sensors. The robotic dual-arm system uses instrumented arms and hands with cooking implements and instrumented utensils and tools (in this image, pans) on countertop 12 while performing specific steps in the recipe replication process, all the while being continuously monitored by multi-modal sensor unit 66 to ensure that the replication process is performed as faithfully as possible to that created by a human chef. All data from the multi-modal sensor 66, the dual-arm robotic system consisting of the torso 74, arm 70, wrist 71 and multi-fingered hand 72, tools, utensils and appliances are wirelessly transmitted to the computer 16 where it is processed by the on-board processing unit 16, in order to compare and track the recipe reproduction process and follow as closely as possible the criteria and steps defined in the recipe script 19 previously created and stored on the medium 18.

[0127] Some suitable robotic hands that can be modified for use with the robotic kitchen 48 include the Shadow Dexterous Hand and Hand-Lite designed by Shadow Robot Company of London, UK, the Servo Electric Five-Fingered Grasping Hand SVH designed by SCHUNK GmbH & Co. KG of Laufen / Neckar, Germany, and the DLR HIT HAND II designed by DLR Robotics and Mechatronics of Cologne, Germany.

[0128] Some robotic devices 72 are suitable for modification to work with the robotic kitchen 48, including the UR3 Robot and UR5 Robot by Universal Robots A / S of Odense, Denmark, Industrial Robots with various payloads designed by KUKA Robotics of Augsburg, Bavaria, Germany, and Industrial Robot Arm Models designed by Yaskawa Motoman of Kitakyushu, Japan.

[0129] 7E is a block diagram depicting a step-by-step flow and method 376 for ensuring that there are control or verification points during the recipe replication process based on a recipe-script when executed by the standardized robotic kitchen 50 that ensure the cooking result is as close to identical as possible for a particular dish executed by the standardized robotic kitchen 50 when compared to a dish prepared by a human chef 49. With a recipe 378 described by a recipe-script and executed in sequential steps in the cooking process 380, the fidelity of the execution of the recipe by the robotic kitchen 50 will be highly dependent on consideration of the following key control items: Key control items include the process of selecting and utilizing high-quality prepared ingredients in standardized portion sizes and shapes 382; the use of standardized tools and implements, utensils with standardized handles to ensure proper and secure gripping in known orientations 384; standardized equipment 386 (oven, blender, refrigerator, refrigerator, etc.) in the standardized kitchen that is as identical as possible when comparing the chef studio kitchen where the human chefs 49 prepare the dishes and the standardized robotic kitchen 50; the location and arrangement 388 where ingredients should be used in the recipe; and finally, a pair of robotic arms, wrists, and multi-fingered hands in the kitchen module 50 that are continuously monitored by sensors 390 with computer-controlled movements to ensure successful execution of each step at every stage of the process of replicating the recipe script for a particular dish. Thus, the task of ensuring identical results 392 is the ultimate goal for the standardized robotic kitchen 50.

[0130] FIG. 7F depicts a block diagram of cloud-based recipe software for facilitating communication between the chef studio, the robotic kitchen, and other sources. Various types of data are communicated, modified, and stored on cloud computing 395 between chef kitchen 44, which operates standardized robotic kitchen 50, and robotic kitchen 48, which operates standardized robotic kitchen 50. Cloud computing 395 provides a central location for storing software files, including those for robotic food preparation 56 operations, from which the software files can be conveniently retrieved and uploaded over the network between chef kitchen 44 and robotic kitchen 48. Chef kitchen 44 is communicatively coupled to cloud computing 395 through wired or wireless network 396 via the internet, wireless protocols, and short-range communication protocols such as Bluetooth. Robotic kitchen 48 is communicatively coupled to cloud computing 395 through wired or wireless network 397 via the internet, wireless protocols, and short-range communication protocols such as Bluetooth. Cloud computing 395 includes computer storage locations for storing task library 398a with actions, recipes, and mini-operations, user profile / data 398b with login information, IDs, and subscriptions, recipe metadata 398c with text, audio media, etc., object recognition module 398d with standardized images, non-standardized images, dimensions, weight, and orientation, environment / instrumentation map 398e for object position, location, and navigation of the operating environment, and control software files 398f for storing robot command instructions, high-level software files, and low-level software files. In another embodiment, Internet of Things (IoT) devices can be incorporated to operate with chef kitchen 44, cloud computing 396, and robotic kitchen 48.

[0131] 8A is a block diagram illustrating one embodiment of a recipe translation algorithm module 400 between chef movements and robotic replica movements. The recipe algorithm translation module 404 converts captured data from the chef movements within the chef studio 44 into machine-readable and machine-executable language 406 for instructing the robotic arms 70 and robotic hands 72 to replicate the food dish prepared by the chef movements within the robotic kitchen 48. Within the chef studio 44, the computer 16 senses multiple sensors S0, S1, S2, S3, S4, S5, S6...S arranged in a vertical row on a table 408. n and the time intervals in the horizontal rows are t0, t1, t2, t3, t4, t5, t6...t end At time t0, the computer 16 captures and records the chef's movements based on sensors on a glove 26 worn by the chef, represented by S0, S1, S2, S3, S4, S5, S6...S n At time t1, the computer 16 records x, y, and z coordinate positions from the sensor data received from the plurality of sensors S0, S1, S2, S3, S4, S5, S6...S n At time t2, the computer 16 records the x, y, and z coordinate positions from the sensor data received from the plurality of sensors S0, S1, S2, S3, S4, S5, S6...S n This process records the xyz coordinate positions from the sensor data received from the food preparation unit. end It continues until it ends at each time unit t0, t1, t2, t3, t4, t5, t6...t end As a result of the captured and recorded sensor data, the table 408 displays the time elapsed for sensors S0, S1, S2, S3, S4, S5, S6...S in the glove 26. n , which indicates the difference between the xyz coordinate position at one particular time and the xyz coordinate position at the next particular time. Table 408 shows the chef's movements from start time t0 to end time t endThe robotic arm 70 and the robotic hand 72 replicate the recipe recorded from the chef studio 44 and subsequently converted into robotic instructions, where the robotic arm 70 and the robotic hand 72 replicate the food preparation of the chef 49 according to a timeline 416. The robotic arm 70 and the robotic hand 72 replicate the food preparation at the same x, y, and z coordinate locations from a start time t0 to an end time t1, as shown in the timeline 416. end The test is carried out at the same speed in the same time intervals.

[0132] In some embodiments, a chef performs the same food preparation operation multiple times, resulting in sensor readings and parameters in the corresponding robotic instructions that vary somewhat from one time to the next. The set of sensor readings for each sensor across multiple iterations of preparing the same food dish provides a distribution with a mean, standard deviation, and minimum and maximum values. The corresponding changes in the robotic instructions (also called effector parameters) across multiple executions of the same food dish by the chef also define distributions with a mean, standard deviation, minimum, and maximum values. These distributions can be used to determine the fidelity (or accuracy) of subsequent robotic food preparations.

[0133] In one embodiment, the estimated average accuracy of a robotic food preparation operation is given by: JPEG2025072400000002.jpg1665

[0134] where C represents the set of chef parameters (1st through nth) and R represents the set of robotic device parameters (1st through nth correspondingly). The numerator in the summation represents the difference (i.e., error) between the robot parameters and the chef parameters, and the denominator normalizes to the maximum difference. The summation is the total normalized accumulated error (i.e., Given JPEG2025072400000003.jpg1137, multiplying it by 1 / n gives the average error. The complement of the average error corresponds to the average precision.

[0135] Another form of accuracy calculation weights these parameters in terms of importance, where each coefficient (each α) represents the importance of the i-th parameter, and the normalized cumulative error is JPEG2025072400000004.jpg1135, and the estimated average accuracy is given by JPEG2025072400000005.jpg1684

[0136] 8B is a block diagram illustrating a pair of sensor-equipped gloves 26a and 26b worn by a chef 49 to capture and transmit the chef's movements. In this specific example, which is intended to illustrate by way of example and not by way of limitation, the right-hand glove 26a includes various sensor data points D1, D2, D3, D4, D5, D6, D7, D8, D9, D10, D11, D12, D13, D14, D15, D16, D17, D18, D19, D20, D21, D22, D23, D24, D25, D26, D27, D28, D29, D30, D31, D32, D33, D34, D35, D36, D37, D38, D40, D41, D42, D43, D44, D45, D46, D47, D48, D49, D50, D51, D52, D53, D54, D55, D56, D57, D58, D59, D60, D61, D62, D63, D64, D65, D66, D67, D68, D69, D70, D71, D72, D73, D74, D75, D76, D77, D78, D79, D80, D81, D82, D83, D84, D85, D86, D87, D88, D89, D90, D91, D92, D93, D94, D95, D96, D97, D98, D99, D99, D100, D101, D102, D103, D104, D105, D106, D107, D108, D 10 , D 11 , D 12 , D 13 , D 14 , D 15 , D 16 , D 17 , D 18 , D 19 , D 20 , D 21 , D 22 , D 23 , D 24 , and D 25 The left hand glove 26b includes 25 sensors to capture various sensor data points D on the glove 26b, which may have optional electronic and mechanical circuitry 422. 26 , D 27 , D 28 , D 29 , D 30 , D 31 , D 32 , D 33 , D 34 , D 35 , D 36, D 37 , D 38 , D 39 , D 40 , D 41 , D 42 , D 43 , D 44 , D 45 , D 46 , D 47 , D 48 , D 49 , D 50 It includes 25 sensors to capture

[0137] 8C is a block diagram illustrating the robotic cooking execution steps based on sensory data captured from the chef's sensory capture gloves 26a and 26b. Within the chef studio 44, the chef 49 wears the sensor-equipped gloves 26a and 26b to capture the food preparation process, and the sensor data is recorded in a table 430. In this example, the chef 49 is cutting a carrot with a knife, where each slice of the carrot is approximately one centimeter thick. These action primitives by the chef 49 recorded by the gloves 26a, 26b can constitute mini-manipulations 432 occurring over time slots 1, 2, 3, and 4. The recipe algorithm translation module 404 is configured to translate the recorded recipe file from the chef studio 44 into robotic instructions for operating the robotic arm 70 and robotic hand 72 according to the software table 434 within the robotic kitchen 28. The robotic arm 70 and robotic hand 72 prepare the food dish using control signals 436 for the mini-manipulation of cutting a carrot with a knife, where each slice of carrot is approximately one centimeter thick, predefined in the mini-manipulation library 116. The robotic arm 70 and robotic hand 72 operate autonomously at the same xyz coordinates 438 with possible real-time adjustment to the size and shape of the particular carrot by creating a temporary three-dimensional model 440 of the carrot from the real-time adjustment device 112.

[0138] Those skilled in the art will recognize that many mechanical and control issues must be addressed to autonomously operate a mechanical robotic mechanism, such as those described in the embodiments of the present disclosure, and the robotics literature describes methods for doing just that. Establishing static and / or dynamic stability in a robotic system is a key requirement. Dynamic stability is a highly desirable property, particularly in robotic manipulation, to prevent accidental damage or movement beyond what is desired or programmed. FIG. 8D illustrates dynamic stability relative to equilibrium. In this case, the "equilibrium value" is the desired state of the arm (i.e., the arm moves exactly where it is programmed to move, with deviations caused by any number of factors, e.g., inertia, centripetal or centrifugal forces, harmonic vibrations, etc.). A dynamically stable system is one in which changes are small and decay over time, as represented by curve 450. A dynamically unstable system is one in which changes fail to decay and may decay over time, as depicted by curve 452. Additionally, the worst case scenario is when the arm is statically unstable (e.g., unable to hold the weight of whatever it is gripping) and drops or fails to recover from any deviation from the programmed position and / or path, as illustrated by curve 454. Additional information on planning (forming mini-manipulation sequences or recovering if something goes wrong) is found in Garagnani, M. (1999), "Improving the Efficiency of Processed Domain-axioms Planning," PLANSIG-99 Proceedings, Manchester, UK, pp. 190-192, which is incorporated herein by reference in its entirety.

[0139] The above references refer to conditions for dynamic stability which are incorporated by reference into the present disclosure to enable proper functioning of the robot arm. These conditions include the following basic principles for calculating torques on the joints of the robot arm: JPEG2025072400000006.jpg1160

[0140] where T is the torque vector (T has n components, each corresponding to a degree of freedom of the robot arm), M is the inertia matrix of the system (M is a positive semidefinite nxn matrix), C is the combination of centripetal and centrifugal forces, also an nxn matrix, G(q) is the gravity vector, and q is the position vector. Additionally, these conditions include finding stable and minimum points, for example, by Lagrange's equation, when the robot position (x') can be described by a twice differentiable function (y'). TIFF2025072400000007.tif1268TIFF2025072400000008.tif639

[0141] For a system consisting of a robot arm and hand / grasper to be stable, it must be properly designed and constructed, and have the appropriate sensing and control systems to operate within acceptable performance boundaries. Given the physical system and what its controller wants it to do, you want the best possible performance (highest speed, best tracking of position / velocity and force / torque, all under stable conditions).

[0142] When proper design is discussed, the concept is related to proper observability and controllability of the system. Observability means that the important variables of the system (joint / finger positions and velocities, forces, and torques) are measurable by the system, which means that it must have the ability to sense these variables, which in turn implies the presence and use of proper sensing devices (internal or external). Controllability means that the subject (computer in this case) has the ability to shape or control the system's axes based on parameters observed from internal / external sensors, which usually means actuators or direct / indirect control of certain parameters using motors or other computer-controlled actuation systems. The ability to make a system as linear as possible in its response, thereby canceling out deleterious nonlinear effects (stiction, recoil, hysteresis, etc.), allows control schemes such as PID gain programming and nonlinear controllers such as sliding mode control to guarantee system stability and performance, even given system modeling uncertainties (errors in mass / inertia estimation, geometric discretization of dimensions, sensor / torque discretization anomalies, etc.), which are necessarily present in any high performance control system.

[0143] Furthermore, the use of a proper computing and sampling system is important because the ability of the system to follow fast movements with a certain maximum frequency content is clearly related to the control bandwidth (closed loop sampling rate of the computer control system) that the overall system can achieve, and therefore the frequency response (ability to track movements of a certain speed and movement frequency content) that the system can exhibit.

[0144] All of the above properties are important with regard to ensuring that a highly redundant system can actually perform the complex and skilled tasks required by a human chef for successful recipe script execution in both a dynamic and stable manner.

[0145] Machine learning in the context of robotics relevant to the present disclosure can include known methods for parameter adjustment, such as reinforcement learning. Another preferred embodiment for the present disclosure is case-based learning, a learning technique for repetitive and complex operations, such as preparing and cooking food through multiple steps over time. Case-based reasoning, also known as analogical reasoning, has been developed over time.

[0146] As a general overview, case-based reasoning involves the following steps: A. Building and storing cases. A case is a sequence of actions with parameters that are successfully performed to accomplish a goal. Parameters include force, direction, position, and other physical or electronic measures whose values ​​are required to successfully perform a task (e.g., a cooking operation). First, 1. storing aspects of the most recently solved problem together with: 2. A method for solving the problem and, optionally, intermediate steps and their parameter values; and 3. The final result is (typically) stored. B. Applying the Case (at a later point in time) 4. Obtaining one or more stored cases having a problem that bears a strong similarity to the new problem; 5. Optionally, adjusting parameters from the acquired case to apply to the current case (e.g., an item may be somewhat heavier and therefore require somewhat more force to lift it); 6. Using the same methods and steps from the example, with adjusted parameters (if necessary), to at least partially solve the new problem. Case-based reasoning thus involves storing solutions to past problems and applying these solutions with possible parameter modifications to new problems that are very similar. However, something more is needed to apply case-based reasoning to robotic manipulation problems: a change in one parameter of the solution plan will result in a change in one or more related parameters, thereby requiring modification of the problem solution rather than just adaptation. This new process generalizes a solution to related neighboring solutions (those that correspond to small changes in input parameters, such as the exact weight, shape, and location of the input ingredients), and we call this process case-based robot learning. Case-based robot learning works as follows: C. Building, storing, and modifying robot operation cases 1. storing aspects of the most recently solved problem together with: 2. Values ​​of parameters (e.g., inertia matrix from Eq. 1, forces, etc.), 3. Performing a perturbation analysis by varying the parameters relevant to the domain (e.g., changing the weight of an ingredient or its exact starting position in a cooking) to see how much the parameter value can be varied and still obtain the desired result; 4. Record which other parameter values ​​(e.g., forces) are changed through perturbation analysis of the model, and by how much; and 5. Store the modified solution plan (with calculation of dependencies between parameters and expected changes in these values) if the changes are within the operating specifications of the robotic device. D. Applying the Case (at a later point in time) 6. Retrieving one or more stored cases with an initial problem that has a strong similarity to the new problem, including parameter values ​​and value ranges, but with modified exact values ​​(new ranges or calculations for the new values ​​depending on the values ​​of the input parameters); and 7. Use modified methods and steps from the case to at least partially solve a new problem. As the chef teaches the robot (two arms and sensing devices such as tactile feedback from the fingers, force feedback from the joints, and one or more observation cameras), the robot learns not only the specific movement sequences and time correlations but also the small variations around the chef's movements so that it can prepare the same dish despite minor changes in observable input parameters, thus learning a generalized and adapted plan, giving it much greater utility than rote memorization. For additional information on case-based reasoning and learning, see Leake (1996 book), "Case-Based Reasoning: Experiences, Lessons and Future Directions," http: / / journals.cambridge.org / action / displayAbstract?fromPage=online&aid=4068324&fileId=S269888900006585dl.acm.org / citation.cfm?id=524680, and Carbonell (1983), "Learning by Analogy: Formulating and Generalizing Plans from Past Experience," http: / / link.springer.com / chapter / 10.1007 / 978-3-662-12405-5_5, which are incorporated herein by reference in their entirety.

[0147] As depicted in FIG. 8E, the cooking process proceeds through multiple stages S1, S2, S3...S4 of food preparation shown in timeline 456. j .... These steps may require a strict linear / sequential ordering, or some may be performed in parallel, either way resulting in a set of stages {S1, S2, ..., S} that must all be successfully completed to achieve overall success. i ,…,S n}. The success probability for each stage is P(si ), and if there are n stages, the overall success probability is estimated by the product of the success probabilities at each stage as: TIFF2025072400000009.tif1628

[0148] Those skilled in the art will recognize that even if the individual stage success probabilities are relatively high, the overall success probability may be low. For example, assuming 10 stages and each stage success probability of 90%, the overall success probability is (.9) = .28 or 28%.

[0149] A stage in preparing a food dish may involve one or more mini-operations, each involving one or more robotic actions that produce a well-defined intermediate result. For example, slicing a vegetable may be a mini-operation that involves holding the vegetable in one hand and a knife in the other and applying repeated knife movements until the vegetable is thinly sliced. A stage in preparing a dish may involve one or more slicing mini-operations.

[0150] This success probability formula applies equally to the stage level and the mini-operation level, as long as each mini-operation is relatively independent of other mini-operations.

[0151] In one embodiment, to mitigate the problem of low success probability due to potential compounding errors, a standardized method is recommended for most or all of the mini-operations in all of the stages. The standardized operation can be pre-programmed, pre-tested, and pre-adjusted as necessary to select the operation sequence with the highest success probability. Thus, if the standardized method of mini-operations within a stage has a very high probability, the overall success probability of food preparation will also be very high due to the previous tasks until all of the stages are completed and tested. For example, returning to the example above, assuming there are still 10 stages, if each stage utilizes a reliable standardized method and its success probability is 99% (rather than 90% as in the previous example), then the overall success probability will be (0.99). 10 = 90.4%, which is significantly better than the 28% probability of the overall corrected outcome.

[0152] In another embodiment, more than one alternative is provided for each stage, and if one alternative fails, another alternative is attempted. This embodiment requires dynamic monitoring to determine the success or failure of each stage and the ability to have alternative plans. The probability of success for that stage is the complement of the probability of failure for all of the alternatives, and is written mathematically as follows: TIFF2025072400000010.tif1683

[0153] In the above expression, s i is the stage, and A(s i ) is s i The probability of failure for a given alternative is the complement of the probability of success for that alternative, i.e., 1-P(s i |a j ), and the probability of all alternatives failing is the product in the above equation. Therefore, the probability that not all will fail is the complement of this product. Using the substitution method, the overall probability of success can be estimated as the product of each stage with its alternatives, i.e., TIFF2025072400000011.tif1649

[0154] Using this substitution method, if each of the 10 stages had four alternatives, and each alternative for each stage had a 90% expected success rate, the overall probability of success was (1-(1-(.9)) compared to only 28% with no alternatives. 4 ) 10 = 0.99 or 99%. The alternatives method modifies the original problem from a series of stages with multiple single points of failure (if every stage fails) to one with no single point of failure, giving a more robust outcome, since all alternatives would have to fail for any given stage to fail.

[0155] In another embodiment, a standardized stage consisting of standardized mini-operations is combined with selective measures in the food dish preparation stage to produce even more robust behavior. In such cases, even if alternatives exist for only some of the stages or mini-operations, the corresponding probability of success can be very high.

[0156] In another embodiment, alternatives in case of failure are provided only for stages with a lower probability of success, such as those for which a highly reliable standardization method does not exist, or for which there is potential variability depending on, for example, a particular type of material. This embodiment reduces the burden of providing alternatives for all stages.

[0157] 8F is a graph illustrating the overall probability of success (y-axis) as a function of the number of stages required to prepare a food dish (x-axis) for a first curve 458 illustrating a non-standardized kitchen 458 and a second curve 459 illustrating a standardized kitchen 50. In this example, it was assumed that the individual probability of success for each food preparation stage was 90% for the non-standardized operation and 99% for the standardized pre-programmed stages. As shown in curve 458 compared to curve 459, the composite error is significantly worse for the non-standardized operation.

[0158] 8G is a block diagram illustrating the execution of recipes 460 in a multi-stage robotic food preparation using mini-manipulation and action primitives. Each food recipe 460 includes a first food preparation stage S1470, a second food preparation stage S2, ... an nth stage food preparation stage S3, all performed by the robotic arm 70 and robotic hand 72. n The food preparation process can be divided into multiple food preparation stages S1470, S1471, S1472, S1473, S1474, S1475, S1476, S1477, S1478, S1479, S1480, S1481, S1482, S1483, S1484, S1485, S1486, S1487, S1488, S1489, S1490, S1491, S1492, S1493, S1494, S1495, S1496, S1497, S1498, S1499, S1500, S1501, S1502, S1503, S1504, S1505, S1506, S1507, S1508, S1509, S1510, S1511, S1512, S1513, S1514, S1515, S1516, S1517, S1518, S1519, S1520, S1521, S1522, S1523, S1524, S1525, S1526, S1527, S1528, S1529, S1530, S1531, S1532, S1533, S1534, S1535, S1536, S1537, S1538, S1540, S1541, S1542, S1543, S1544, S1545, S1546, S1547, S1548, S1549, S1550, S1551, S1552, S1553, S n 490 produces substantially the same or identical results as those recorded within chef studio 44 by replicating the food preparation process of chef 49.

[0159] To achieve each functional result (e.g., an egg is cracked), a predefined mini-manipulation is available. Each mini-manipulation is composed of a set of action primitives that act together to achieve the functional result. For example, a robot can start by moving its hand towards an egg, touching it to locate its position and verify its size, and performing the movement and sensing actions necessary to grasp, lift, and place the egg in a known, predefined configuration.

[0160] Multiple mini-operations can be grouped into stages, such as making a sauce, for convenience in understanding and organizing a recipe. The end result of performing all of the mini-operations to complete all of the stages is that the food dish is reproduced with consistent results every time.

[0161] FIG. 9A is a block diagram illustrating an example of a robotic hand 72 having five fingers and a wrist with RGB-D sensor capabilities, camera sensor capabilities, and sonar sensor capabilities for detecting and manipulating kitchen tools, objects, or kitchen equipment items. The palm of the robotic hand 72 includes an RGB-D sensor 500, a camera sensor, or a sonar sensor 504f. Alternatively, the palm of the robotic hand 450 includes both a camera sensor and a sonar sensor. The RGB-D sensor 500 or sonar sensor 504f can detect the location, size, and shape of an object to create a three-dimensional model of the object. For example, the RGB-D sensor 500 uses structured light to capture the shape of an object for three-dimensional mapping and positioning, path planning, navigation, object recognition, and human tracking. The sonar sensor 504f uses acoustic waves to capture the shape of an object. Video cameras 66 placed anywhere in the robotic kitchen, such as on the handrails, or on the robot, in conjunction with camera sensors 452 and / or sonar sensors 454, provide a way to capture, follow, or guide the movements of kitchen tools used by chef 49, as illustrated in FIG. 7A. Video cameras 66 are positioned at an angle and some distance from robotic hand 72, thus providing a high-level view of the object grasp of robotic hand 72 and whether it has grasped or dropped / released the object. A suitable example of an RGB-D (red light beam, green light beam, blue light beam, and depth) sensor is the Kinect system by Microsoft, which features an RGB camera, depth sensor, and multiple array microphones running software that provides full-body 3D motion capture, face recognition, and voice recognition capabilities.

[0162] The robotic hand 72 has an RGB-D sensor 500 located at or near the center of the palm to detect the distance and shape of an object, as well as to handle kitchen tools. The RGB-D sensor 500 provides the robotic hand 72 with guidance to move the robotic hand 72 toward an object and make adjustments necessary to grasp the object. A second sonar sensor 502f and / or a tactile sensor is located near the palm of the robotic hand 72 to detect the distance and shape of the object and subsequent contact. The sonar sensor 502f can also guide the robotic hand 72 toward an object. Additional types of sensors in the hand can include ultrasonic sensors, lasers, radio frequency identification (RFID) sensors, and other suitable sensors. Additionally, the tactile sensor serves as a feedback mechanism to determine whether the robotic hand 72 should continue to apply additional pressure to grasp the object, at which point sufficient pressure exists to safely lift the object. Additionally, sonar sensors 502f in the palm of the robotic hand 72 provide tactile sensing capabilities for grasping and handling kitchen tools. For example, when the robotic hand 72 grasps a knife to cut beef, the tactile sensors can detect the amount of pressure the robotic hand exerts against the knife and applies to the beef when the knife finishes slicing the beef, i.e., when the knife has no resistance, or when it grasps an object. The pressure dispensed is not only to secure the object, but also to prevent the object (e.g., an egg) from breaking.

[0163] Furthermore, each finger on the robotic hand 72 has a tactile vibration sensor 502a-502e and a sonar sensor 504a-504e on its respective fingertip, as shown by a first tactile vibration sensor 502a and a first sonar sensor 504a on the tip of the thumb, a second tactile vibration sensor 502b and a second sonar sensor 504b on the tip of the index finger, a third tactile vibration sensor 502c and a third sonar sensor 504c on the tip of the middle finger, a fourth tactile vibration sensor 502d and a fourth sonar sensor 504d on the tip of the ring finger, and a fifth tactile vibration sensor 502e and a fifth sonar sensor 504e on the tip of the little finger. Each of the tactile vibration sensors 502a, 502b, 502c, 502d, and 502e can simulate various surfaces and effects by changing the shape, frequency, amplitude, duration, and direction of vibration. Each of the sonar sensors 504a, 504b, 504c, 504d, and 504e provides sensing capabilities for object distance and shape, temperature or moisture sensing, and feedback capabilities. Additional sonar sensors 504g and 504h are located on the wrist of the robotic hand 72.

[0164] 9B is a block diagram illustrating one embodiment of a pan-tilt head 510 with a sensor camera 512 coupled to a pair of robotic arms and hands for operation in a standardized robotic kitchen. The pan-tilt head 510 has an RGB-D sensor 512 to monitor, capture, or process information and three-dimensional images of the interior of the standardized robotic kitchen 50. The pan-tilt head 510 provides good situational awareness that is not dependent on arm and sensor movement. The pan-tilt head 510 is coupled to a pair of robotic arms 70 and robotic hands 72 to perform food preparation processes, where the pair of robotic arms 70 and robotic hands 72 may cause occlusions. In one embodiment, the robotic device comprises one or more robotic devices 70 and one or more robotic hands (or robotic graspers) 72.

[0165] 9C is a block diagram illustrating the sensor cameras 514 on the robot wrists 73 for operation in the standardized robotic kitchen 50. One embodiment of the sensor cameras 514 is an RGB-D sensor that provides color images and depth perception, mounted on the wrist 73 of each hand 72. Each of the camera sensors 514 on each wrist 73 experiences limited occlusion by the arm, but is generally not occluded when the robotic hand 72 grasps an object. However, the RGB-D sensors 514 may be occluded by each robotic hand 72.

[0166] 9D is a block diagram illustrating an in-hand monocular camera 518 on a robotic hand 72 for operation in the standardized robotic kitchen 50. Each hand 72 has a sensor, such as an RGB-D sensor, to provide in-hand monocular camera functionality by the robotic hand 72 in the standardized robotic kitchen 50. The in-hand monocular camera 518 with RGB-D sensor in each hand produces high image detail with limited occlusion by the respective robotic arm 70 and respective robotic hand 72. However, the robotic hand 72 with the in-hand monocular camera 518 may encounter occlusion when grasping an object.

[0167] 9E-9G are pictorial diagrams illustrating an embodiment of a deformable palm 520 within a robotic hand 72. The fingers of the five-fingered hand are labeled as the thumb (first finger F1 522), the index finger (second finger F2 524), the middle finger (third finger F3 526), ​​the ring finger (fourth finger F4 528), and the pinky finger (fifth finger F5 530). Thenar eminence 532 is a convex spatial region of the deformable material on the radial (first finger F1 522) side of the hand. Thenar eminence 534 is a convex spatial region of the deformable material on the ulna (fifth finger F5 530) side of the hand. The metacarpophalangeal pad (MCP pad) 536 is a convex deformable spatial area on the palmar (palmar) side of the metacarpophalangeal joints (knuckles) of the second, third, fourth, and fifth fingers F2 524, F3 526, F4 528, and F5 530. The robotic hand 72 with the deformable palm 520 wears an outer glove with soft, human-like skin.

[0168] The thenar eminence 532 and hypothenar eminence 534 combine to aid in the application of large forces from the robotic arm to objects within the task space while ensuring that these force applications exert only minimal external pressure on the robotic hand joints (e.g., a rolling pin situation). Additional joints within the palm 520 are available for palm deformation. The palm 520 must deform to allow the formation of an oblique palmar groove for chef-like tool grasping (a typical handle grasp). The palm 520 must deform to allow cup-shape formation for conformal grasping of convex objects such as plates and food ingredients in a chef-like manner, as shown by the cup-shaped state 542 in FIG. 9G.

[0169] Joints within the palm 520 that can facilitate these movements include the thumb carpometacarpal joint (CMC), located on the radial side of the palm near the wrist, which can have two distinct directions of movement (flexion / extension and abduction / adduction). Additional joints required to facilitate these movements include joints on the ulnar side of the palm near the wrist (CMC joints of the fourth finger F4 528 and the fifth finger F5 530) that allow flexion at an oblique angle that assists in the cupping movement of the hypothenar eminence 534 and the formation of the palmar groove.

[0170] The robotic palm 520 may include additional / different joints required to replicate palm shapes observed in human cooking movements, for example, a series of coupled flexion joints that assist in forming an arch 540 between the thenar eminence 532 and the hypothenar eminence 534 to deform the palm 520, such as when the thumb F1 522 contacts the little finger F5 530, as illustrated in FIG. 9F.

[0171] When the palm is in the cupped position, the thenar eminence 532, hypothenar eminence 534, and MCP pad 536 form a ridge around the palmar valley that allows the palm to close around a small spherical object (e.g., 2 cm).

[0172] The shape of the deformable palm will be described using the locations of feature points relative to a fixed coordinate system, as shown in Figures 9H and 9I. Each feature point is represented as a vector of x, y, and z coordinate positions over time. The locations of the feature points are marked on the sensing glove worn by the chef and on the sensing glove worn by the robot. A coordinate system is also marked on the glove, as illustrated in Figures 9H and 9I. The feature points are defined relative to the coordinate system's location on the glove.

[0173] Feature points are measured by a calibrated camera mounted in the task space as the chef performs the cooking task. The trajectories of the feature points over time are used to match the chef's movements to the robotic movements, including matching the shape of the deformable palm. The feature point trajectories from the chef's movements can also be used to inform the robot of the deformable palm design, including the shape and placement of the deformable palm surface and the range of motion of the joints of the robotic hand.

[0174] In the embodiment depicted in Figure 9H, the feature points are in the hypothenar eminence 534, thenar eminence 532, and the MCP pad 536 is a checkerboard pattern with marks indicating the feature points in each area of ​​the palm. The coordinate system in the wrist area has four rectangles identifiable as the coordinate system. The feature points (or markers) are identified at their respective locations relative to the coordinate system. The feature points and coordinate system in this embodiment can be implemented under the glove for food safety reasons, but are visible through the glove for detection.

[0175] 9H shows a robotic hand with a visual pattern that can be used to determine the locations of three-dimensional shape features 550. The locations of these shape features provide information about the shape of the palmar surface as the palmar joints move and as the palmar surface deforms in response to applied forces.

[0176] The visual pattern includes surface marks 552 on the robotic hand or on gloves worn by the chef. These surface marks can be covered by food-safe transparent gloves 554, but the surface marks 552 remain visible through the gloves.

[0177] If the surface mark 552 is visible in a camera image, two-dimensional feature points can be identified within this camera image by locating convex or concave corners within the visual pattern. Each such corner within a single camera image is a two-dimensional feature point.

[0178] If the same feature point is identified in multiple camera images, the three-dimensional location of this point can be determined in a coordinate system fixed relative to the standardized robotic kitchen 50. This calculation is performed based on the two-dimensional location of the point in each image and the known camera parameters (position, orientation, field of view, etc.).

[0179] A coordinate system 556 fixed to the robotic hand 72 can be obtained using the coordinate system visual pattern. In one embodiment, the coordinate system 556 fixed to the robotic hand 72 consists of an origin and three Cartesian coordinate axes. The coordinate system is identified by locating features of the coordinate system visual pattern in multiple cameras and extracting the origin and coordinate axes using known parameters of the coordinate system visual pattern and known parameters of the cameras.

[0180] The 3D shape features expressed in the coordinate system of the food preparation station are transformed into the coordinate system of the robotic hand when the coordinate system of the robotic hand is observed.

[0181] The shape of a deformable palm consists of a vector of 3D shape features, all expressed within a reference coordinate system fixed to the robot's or chef's hand.

[0182] As illustrated in FIG. 9I, the feature points 560 in these embodiments are represented by sensors, such as Hall Effect sensors, in different regions of the palm (hypothenar eminence 534, thenar eminence 532, and MCP pad 536). The feature points are identifiable at their respective locations relative to a coordinate system, which in this implementation is a magnet. The magnet generates a magnetic field that is readable by the sensor. The sensors in this embodiment are embedded under the glove.

[0183] Figure 9I shows a robotic hand 72 with embedded sensors and one or more magnets that can be used as an optional mechanism to determine the location of three-dimensional geometric features. One geometric feature is associated with each embedded sensor. The locations of these geometric features 560 provide information about the shape of the palmar surface as the palmar joints move and as the palmar surface deforms in response to applied forces.

[0184] The location of the shape feature is determined based on the sensor signal, which provides an output that allows calculation of the distance within a coordinate system attached to a magnet attached to the robot or chef's hand.

[0185] The three-dimensional location of each feature point is calculated based on sensor measurements and known parameters obtained from sensor calibration. The deformable palm shape is composed of vectors of three-dimensional feature points, all expressed within a reference coordinate system fixed to the robot's or chef's hand. For additional information about common contact areas on the human hand and their function in grasping, see Kamakura, Noriko, Michiko Matsuo, Harumi Ishii, Fumiko Mitsuboshi, and Yoriko Miura, "Patterns of static prehension in normal hands," American Journal of Occupational Therapy 34, No. 7 (1980): 437-445, which is incorporated herein by reference in its entirety.

[0186] FIG. 10A is a block diagram illustrating an example of a recording device 550A worn by a chef 49 in a standardized robotic kitchen environment 50 to record and capture the chef's movements during the food preparation process for a particular recipe. The chef recording device 550A includes, but is not limited to, one or more robotic gloves (or robotic garments) 26, a multi-mode sensor unit 20, and a pair of robotic glasses 552A. Within the chef studio system 44, the chef 49 wears the robotic gloves 26 to cook and record and capture the chef's cooking movements. Alternatively, the chef 49 can wear not only the robotic gloves 26 but also a robotic garment with the robotic gloves. In one embodiment, the robotic gloves 26 with embedded sensors capture, record, and store time-stamped positions, pressures, and other parameters of the chef's arm, hand, and finger movements in an x-y-z coordinate system. The robotic gloves 26 store the positions and pressures of the chef's arms and fingers in a three-dimensional coordinate system over a duration from a start time to an end time when preparing a particular food dish. When the chef 49 wears the robotic glove 26, all of his movements, hand positions, gripping movements, and amount of pressure exerted while preparing a food dish within the chef studio system 44 are precisely recorded at regular time intervals, such as every t seconds. The multi-mode sensor unit 20 includes a video camera, an IR camera and rangefinder 306, a stereo (or even trinocular) camera 308, and a multi-dimensional scanning laser 310, and provides multi-spectral sensing data to a main software abstraction engine 312 (after being acquired and filtered in a data acquisition and filtering module 314). The multi-mode sensor unit 20 generates three-dimensional surfaces or textures and processes the abstracted model data.This data is used within the Scene Understanding Module 316 to perform several steps, including (but not limited to) constructing high- and low-resolution (laser: high-resolution, stereo camera: low-resolution) three-dimensional surface volumetric regions of the scene with overlaid color and texture video information in the visual and IR spectrum; enabling edge detection and volumetric object detection algorithms to infer what elements are present in the scene; and enabling the use of shape / color / texture mapping and consistency mapping algorithms to proceed based on the processed data and provide the processed information to the Kitchen Cooking Process Equipment Handling Module 318. Optionally, in addition to the robotic gloves 76, the chef 49 can wear a pair of robotic glasses 552A with one or more robotic sensors 554A around a frame that includes robotic earphones 556A and a microphone 558. The robotic glasses 552A provide further vision and capture capabilities, such as a camera for capturing video and recording what the chef 49 sees while preparing food. The one or more robotic sensors 554A capture and record the temperature and smell of the food being prepared. The earphones 556 and microphone 558 capture and record the sounds that the chef 49 hears while cooking, which may include sound characteristics of a human voice, frying, grilling, grinding, etc. The chef 49 can also use the earphones and microphone 82 to record simultaneous voice commands and real-time cooking stages of food preparation. In this regard, the chef robot recorder device 550 records the chef's movement parameters, speed parameters, temperature parameters, and sound parameters during the food preparation process for a particular food dish.

[0187] 10B is a flow diagram illustrating one embodiment 560A of a process for evaluating captured chef movements using robotic poses, movements, and forces. Database 561 stores predefined (or pre-defined) grasping poses 562A and predefined hand movements by robotic arms 72 and robotic hands 72, which are weighted by importance 564, labeled with contact points 565, and stored with contact forces 565. In operation 567, chef movement recording module 98 is configured to capture chef movements in preparing a food dish based in part on predefined grasping poses 562A and predefined hand movements 563. In operation 568, robotic food preparation engine 56 is configured to obtain the poses, movements, and forces and evaluate the robotic device configuration for its ability to perform the mini-manipulation. The robotic device configuration then undergoes an iterative process 569 of evaluating robot design parameters 570, adjusting design parameters 571 to improve score and performance 571, and modifying the robotic device configuration 572.

[0188] FIG. 11 is a block diagram illustrating one embodiment of a side view of a robotic arm 70 for use with the standardized robotic kitchen system 50 in the home robotic kitchen 48. In other embodiments, one or more of the robotic arms 70, such as one arm, two arms, three arms, four arms, or more arms, can be designed for operation in the standardized robotic kitchen. One or more software recipe files 46 from the chef studio system 44, which store the movements of a chef's arms, hands, and fingers during food preparation, can be uploaded and converted into robotic instructions for controlling one or more robotic arms 70 and one or more robotic hands 72 to mimic the chef's movements to prepare the chef-prepared food dish. The robotic instructions control the robotic device 75 to replicate the chef's detailed movements to prepare the same food dish. Each of the robotic arms 70 and each of the robotic hands 72 can also include additional features and tools, such as knives, forks, spoons, spatulas, or other types of utensils or food preparation equipment, to accomplish the food preparation process.

[0189] 12A-12C are block diagrams illustrating one embodiment of a kitchen handle 580 for use by a robotic hand 72 having a palm 520. The design of the kitchen handle 580 is intended to be universal (or standardized) such that the same kitchen handle 580 can be attached to any type of kitchen implement or tool, such as a knife, spatula, strainer, colander, turner, etc. FIGS. 12A-12B show different perspective views of the kitchen handle 580. The robotic hand 72 grasps the kitchen handle 580 as shown in FIG. 12C. Other types of standardized (or universal) kitchen handles can be designed without departing from the inventive spirit of the present disclosure.

[0190] FIG. 13 is a pictorial diagram illustrating an exemplary robotic hand 600 having tactile sensors 602 and distributed pressure sensors 604. During the food preparation process, the robotic device 75 uses contact signals generated by sensors in the fingertips and palm of the robotic hand to detect force, temperature, humidity, and toxicity as the robot replicates step-by-step movements and compares the sensed values ​​to the tactile profile of the chef's studio cooking program. Vision sensors help the robot identify its surroundings and take appropriate cooking actions. The robotic device 75 analyzes images of the immediate environment from the vision sensors and compares them to stored images of the chef's studio cooking program so that appropriate movements are taken to achieve identical results. The robotic device 75 also uses a separate microphone to compare the chef's vocal commands against background noise from the food preparation process to improve recognition performance during cooking. Optionally, the robot can have an electronic nose (not shown) to detect odors or flavors as well as ambient temperature. For example, the robotic hand 600 can distinguish real eggs from real ones by surface texture, temperature, and weight signals generated by tactile sensors in the fingers and palm, allowing it to apply the appropriate amount of force to hold the egg without breaking it. Additionally, it can check the quality of the egg by shaking it and listening to the liquid sloshing and cracking sounds, and observe and smell the yolk and white to determine freshness. The robotic hand 600 can then discard damaged eggs or select fresh eggs. The sensors 602 and 604 on the hand, arm, and head enable the robot to move, touch, see, and hear to execute food preparation processes with external feedback and achieve results in food preparation identical to those of a chef's studio cooking.

[0191] 14 is a pictorial diagram illustrating an example of a sensing costume for wear by a chef 49 in a standardized robotic kitchen 50. During food preparation of a food dish, as recorded by software file 46, chef 49 wears sensing costume 620 to capture a time sequence of the chef's food preparation movements in real time. Sensing costume 620 can include, but is not limited to, a haptic suit 622 (shown with one full-length arm and hand costume), haptic gloves 624, multi-mode sensors 626, and a head costume 628. Sensored haptic suit 622 can capture data from the chef's movements and transmit the captured data to computer 16 to record the xyz coordinate positions and pressures of the human arm 70 and hand / fingers 72 in an XYZ coordinate system with time stamps. The sensing costume 620 also records and associates the position, velocity, and force / torque, as well as end-point contact behavior of the human arm 70 and hand / fingers 72 with a system timestamp in the robot coordinate system for correlation with relative positions within the standardized robotic kitchen 50 with geometric sensors (laser sensors, 3D stereo sensors, or video sensors). The sensored tactile gloves 624 are used to capture, record, and store force, temperature, humidity, and toxicity signals detected by the tactile sensors in the gloves 624. The head costume 628 includes feedback devices with visual cameras, sonar, lasers, or radio frequency identification (RFID), as well as custom glasses used to sense, capture, and transmit the captured data to the computer 16 for recording and storing images seen by the chef 48 during the food preparation process. In addition, the head costume 628 also includes sensors for detecting ambient temperature and odor signatures within the standardized robotic kitchen 50. Additionally, head suit 628 also includes audio sensors for capturing sounds heard by chef 49, such as sound characteristics of frying, grinding, chopping, etc.

[0192] 15A-15B are pictorial diagrams illustrating one embodiment of a sensored three-fingered tactile glove 630 for food preparation by a chef 49 and an example sensored three-fingered robotic hand 640. The embodiment illustrated herein shows a simplified robotic hand 640 with fewer than five fingers for food preparation. Correspondingly, the design complexity of the simplified robotic hand 640, along with the cost to manufacture the simplified robotic hand 640, is significantly reduced. A two-fingered grasper or a four-fingered robotic hand with or without an opposable thumb is also a possible implementation option. In this embodiment, the chef's hand movement is limited by the functionality of three fingers: the thumb, index finger, and middle finger, each of which has sensors 632 for sensing data of the chef's movement related to force, temperature, humidity, toxicity, or tactile sensation. The three-fingered tactile glove 630 further includes point sensors or distributed pressure sensors within its palm area. The movements of a chef wearing the three-fingered tactile glove 630 and using the thumb, index finger, and middle finger to prepare a food dish are recorded in a software file. The three-fingered robotic hand 640 then replicates the chef's movements from the software recipe file, which are converted into robotic instructions for controlling the thumb, index finger, and middle finger of the robotic hand 640, while monitoring sensors 642b on the fingers and sensors 644 on the palm of the robotic hand 640. Sensors 644 can be implemented using point sensors or distributed pressure sensors, while sensors 642 include force sensors, temperature sensors, humidity sensors, toxicity sensors, or tactile sensors.

[0193] FIG. 15C is a block diagram illustrating an example of the inter-functions and interactions between the robotic arm 70 and the robotic hand 72. The malleable robotic arm 750 provides smaller payloads, greater safety, and quieter operation, but does not provide as much precision. The humanoid robotic hand 752 provides greater dexterity to handle human tools, is easier to retarget human hand movements, and is more malleable, but requires more complex designs, adds weight, and increases product costs. The simple robotic hand 754 is lighter, less expensive, has less dexterity, and cannot directly use human tools. The industrial robotic arm 756 provides greater precision and a higher payload capacity, but is generally not considered safe around humans and can potentially exert greater forces that could cause harm. One embodiment of the standardized robotic kitchen 50 utilizes the first combination of the malleable arm 750 and the humanoid hand 752. The other three combinations are generally less desirable for implementing the present disclosure.

[0194] 15D is a block diagram illustrating a robotic hand 72 using a standardized kitchen handle 580 attached to a custom cookware and a robotic arm 70 attachable to the kitchen utensil. In one technique for grasping the kitchen utensil, the robotic hand 72 grasps the standardized kitchen tool 580 for attachment to any one of the illustrated custom cookware heads 760a, 760b, 760c, 760d, 760e, and other options. For example, the standardized kitchen handle 580 is attached to a custom spatula head 760e for use in stir-frying ingredients in a pan. In one embodiment, the standardized kitchen handle 580 can be held by the robotic hand 72 in a single position, minimizing potential confusion over various techniques for holding the standardized kitchen handle 580. In another technique for grasping the utensil, the robotic arm 70 has one or more holders 762 that can be attached to the utensil 762, in which case the robotic arm 70 can exert more force as needed to hold down the utensil 762 during the robotic hand movement.

[0195] 16 is a block diagram illustrating the mini-manipulation library database creation module 650 and the mini-manipulation library database execution module. The mini-manipulation database library creation module 60 is a process that creates and tests various possible combinations to select the best mini-manipulations to achieve a particular functional result. One purpose of the creation module 60 is to explore all the different combinations possible for performing a particular mini-manipulation and predefine a library of best mini-manipulations for subsequent execution by the robotic arm 70 and robotic hand 72 in preparing a food dish. The mini-manipulation library creation module 650 can also be used as a teaching method for the robotic arm 70 and robotic hand 72 to learn different food preparation functions from the mini-manipulation library database. The mini-manipulation library database execution module 660 is configured to provide the robotic device with a range of mini-manipulation functions that it can access and execute from a mini-manipulation library database including a first mini-manipulation MM with a first function outcome 662, a second mini-manipulation MM with a second function outcome 664, a third mini-manipulation MM with a third function outcome 666, a fourth mini-manipulation MM with a fourth function outcome 668, and a fifth mini-manipulation MM with a fifth function outcome 670 during the process of preparing a food dish.

[0196] Generalized Mini-Operations: A generalized mini-operation involves a well-defined sequence of sensing and actuator actions with an expected functional outcome. Associated with each mini-operation is a set of preconditions and a set of postconditions. Preconditions assert what must be true in the world state to allow the mini-operation to occur. Postconditions are changes to the world state caused by the mini-operation.

[0197] For example, a miniature manipulation to grasp a small object would involve observing the location and orientation of the object, moving the robotic hand (grasper) to align it with the object's location, applying the necessary force based on the object's weight and stiffness, and moving the arm upward.

[0198] In this example, the preconditions include that a graspable object is located within the reach of the robotic hand and that its weight is within the lifting capacity of the arm, and the postconditions are that the object is no longer resting on a surface where it was previously found and that the object is currently being held by the robotic hand.

[0199] More generally, the generalized small-scale operation M has three parts:<PRE、ACT、POST> where PRE={s1,s2,…,s n} is the action ACT=[a1,a2,…,a k ] occurs, and POST={p1,p2,…,p m} is the set of items in the world state that must be true before the set of changes to the world state, represented by [brackets], can occur. Note that [brackets] denote a sequence, and {curly brackets} denote an unordered set. Each postcondition can also have a probability where the outcome is not certain. For example, a mini-manipulation to grasp an egg may have a 0.99 probability that the egg will fall into the robot's hand (the remaining .01 probability may correspond to accidentally breaking the egg while trying to grasp it, or some other undesirable outcome).

[0200] Even more generally, a mini-manipulation may contain other (smaller) mini-manipulations within its own motion sequence instead of simply being a sequence of atomic or elementary robotic sensing or actuations. In such cases, a mini-manipulation may be defined as a sequence ACT = [a1, m2, m3, ..., a k], where a primitive operation denoted by "a" is interspersed with a mini-operation denoted by "m". In such a case, the postcondition set will be satisfied by the union of the preconditions for that primitive operation and the union of the preconditions of all of its sub-mini-operations.

[0201] TIFF2025072400000012.tif1658

[0202] The postconditions will be those of the generalized small-scale operations and will be determined in a similar manner as follows:

[0203] TIFF2025072400000013.tif1664

[0204] In particular, note that the pre- and post-conditions are not simply mathematical notations, but relate to specific aspects of the physical world (location, orientation, weight, shape, etc.) In other words, the software and algorithms that implement the selection and assembly of mini-manipulations have a direct effect on the robotic machinery, which in turn has a direct effect on the physical world.

[0205] In one embodiment, if a threshold performance for a mini-manipulation is specified, whether generalized or basic, measurements are taken against the post-conditions and the actual results are compared against the optimal results. For example, if a part is positioned within 1% of its desired orientation and location in an assembly task, and the performance threshold is 2%, the mini-manipulation is successful. Similarly, if the threshold in the above example was 0.5%, the mini-manipulation is unsuccessful.

[0206] In another embodiment, instead of specifying a threshold outcome for a mini-operation, tolerance ranges are defined for the post-condition parameters, and the mini-operation is successful if the resulting parameter value after performing this mini-operation falls within the specified range. These ranges are task dependent and specified for each task. For example, in an assembly task, the location of a part can be specified within a statistical data analysis range (or tolerance range) between 0 and 2 millimeters of another part, and the mini-operation is successful if the final location of this part falls within this range.

[0207] In a third embodiment, a mini-operation is successful if its post-condition matches the pre-condition of the next mini-operation in the robot task. For example, if the post-condition of one mini-operation's assembly task is to place a new part one millimeter from a previously placed part, and the next mini-operation (e.g., welding) has a pre-condition specifying that the parts be within two millimeters, then the first mini-operation is successful.

[0208] In general, the preferred embodiments for all mini-manipulations, both basic and generalized, stored in the mini-manipulation library have been designed, programmed, and tested to ensure that these mini-manipulations will perform successfully in the situations for which they are expected.

[0209] Tasks composed of mini-operations: A robotic task is composed of one or (more commonly) several mini-operations. These mini-operations can be performed sequentially, in parallel, or in a partial order. "Sequential" means that each step is completed before the next one begins. "Parallel" means that the robotic device can perform multiple steps simultaneously or in any order. "Partial order" means that some steps must be performed in the sequence specified in the partial order, and others can be performed before, after, or during the steps specified in the partial order. A partial order is defined in the standard mathematical sense as a set of steps S and a set of steps s i →sj i.e., step i must be performed before step j. These steps can be mini-operations or combinations of mini-operations. For example, in Robot Chef, if two ingredients must be mixed in a bowl, there is an ordering constraint that each ingredient must be placed in the bowl before being mixed, but there is no ordering constraint on which ingredient is placed in the mixing bowl first.

[0210] 17A is a block diagram illustrating a sensing glove 680 used by chef 49 to sense and capture the chef's movements while preparing a food dish. Sensing glove 680 has multiple sensors 682a, 682b, 682c, 682d, 682e on each of the fingers of sensing glove 680 and multiple sensors 682f, 682g in the palm area. In another embodiment, at least five pressure sensors 682a, 682b, 682c, 682d, 682e inside a soft glove are used to capture and analyze the chef's movements during all hand manipulations. In this embodiment, multiple sensors 682a, 682b, 682c, 682d, 682e, 682f, and 682g are embedded within sensing glove 680 but are visible through the material of sensing glove 680 for external sensing. The sensing glove 680 can have feature points associated with multiple sensors 682a, 682b, 682c, 682d, 682e, 682f, 682g that reflect the curvature (or unevenness) of the hand at various high and low points therein. The sensing glove 680, which is placed over the robotic hand 72, is made of a soft material that mimics the flexibility and shape of human skin. Further details about the robotic hand 72 can be found in FIG. 9A.

[0211] The robotic hand 72 includes a camera sensor 684, such as an RGB-D sensor, an imaging sensor, or a vision sensing device, located at or near the center of the palm to detect the distance and shape of an object, as well as the distance of the object and to handle kitchen tools. The imaging sensor 682f guides the robotic hand 72 in moving the robotic hand 72 toward an object and making the adjustments necessary to grasp the object. In addition, a sonar sensor, such as a tactile sensor, can be located near the palm of the robotic hand 72 to detect the distance and shape of an object. The sonar sensor 682f can also guide the robotic hand 72 to move toward an object. Each of the sonar sensors 682a, 682b, 682c, 682d, 682e, 682f, and 682g includes an ultrasonic sensor, a laser, a radio frequency identification (RFID), or other suitable sensor. Additionally, each of sonar sensors 682a, 682b, 682c, 682d, 682e, 682f, and 682g serves as a feedback mechanism for determining whether the robotic hand 72 should continue to apply additional pressure to grasp an object, at which point sufficient pressure exists to grasp and lift the object. Additionally, sonar sensor 682f in the palm of the robotic hand 72 provides tactile sensing for handling kitchen tools. For example, if the robotic hand 72 grasps a knife to cut beef, the amount of pressure the robotic hand 72 exerts on the knife and applies to the beef allows the tactile sensor to detect when the knife has finished slicing the beef, i.e., when the knife is no longer offering resistance. The distributed pressure is not only intended to secure the object but also to prevent excessive pressure from being applied, for example, to avoid cracking an egg. Additionally, each finger on the robotic hand 72 has a sensor on its fingertip, as shown by a first sensor 682a on the thumb tip, a second sensor 682b on the index finger tip, a third sensor 682c on the middle finger tip, a fourth sensor 682d on the ring finger tip, and a fifth sensor 682f on the pinky finger tip. Each of the sensors 682a, 682b, 682c, 682d, 682e provides sensing capabilities for object distance and shape, temperature or moisture sensing, and tactile feedback capabilities.

[0212] In addition to the RGB-D sensor 684 and sonar sensor 682f in the palm, sonar sensors 682a, 682b, 682c, 682d, and 682e in the fingertips of each finger provide a feedback mechanism for the robotic hand 72 as a means of grasping non-standardized objects or kitchen tools. The robotic hand 72 can adjust pressure sufficiently to grasp non-standardized objects. FIG. 17B illustrates a program library 690 that stores grasp function samples 692, 694, and 696 according to specific time intervals for the robotic hand 72 to draw upon to perform a specific grasp function. FIG. 17B is a block diagram illustrating a library database 690 of standardized actuation moves within the standardized robotic kitchen module 50. The standardized actuation moves predefined and stored in the library database 690 include grasping, positioning, and manipulating kitchen tools or kitchen equipment with a movement / interaction time profile 698.

[0213] 18A is a graph illustrating that each of the robotic hands 72 is covered with an artificial human-like soft-skin glove 700. The artificial human-like soft-skin glove 700 is see-through and contains multiple embedded sensors sufficient to allow the robotic hands 72 to perform high-level miniature manipulations. In one embodiment, the soft-skin glove 700 contains 10 or more sensors to replicate the hand movements of a chef.

[0214] FIG. 18B is a block diagram illustrating a robotic hand covered with an artificial human-like skin glove for performing high-level mini-manipulations based on a library database 720 having predefined and internally stored mini-manipulations. High-level mini-manipulations refer to motion primitive sequences that require a substantial amount of interactive movement and interaction forces and control over them. Three example mini-manipulations stored in the database library 720 are presented. The first example mini-manipulation is kneading dough 722 using a pair of robotic hands 72. The second example mini-manipulation is making ravioli 724 using a pair of robotic hands 72. The third example mini-manipulation is making sushi 726 using a pair of robotic hands 72. Each of the three example mini-manipulations has a movement / interaction time profile 728 that is tracked by the computer 16.

[0215] 18C is a graph illustrating three types of manipulation motion classifications for food preparation with continuous trajectories of robotic arm 70 and robotic hand 72 motion and force that result in desired goal states. The robotic arm 70 and robotic hand 72 perform robust grasp and transport movements 730 to pick up an object with a rigid grasp and transport it to a target location without the need for strong force interactions. Examples of robust grasp and transport include placing a pan on a stove, picking up a salt shaker, sprinkling salt on a dish, dropping ingredients into a bowl, pouring contents from a container, stirring a salad, and flipping a pancake. The robotic arm 70 and robotic hand 72 perform robust grasp 732 with strong force interactions, where there is strong force contact between two surfaces or objects. Examples of robust grasping with strong force interactions include stirring a pot, opening a box, turning a pan, and sweeping material from a cutting board into a pan. The robotic arm 70 and robotic hand 72 perform deformation strong force interactions 734, where there is strong force contact between two surfaces or objects that results in deformation of one of the two surfaces, such as cutting a carrot, cracking an egg, or rolling dough. For additional information about the function of the human hand, the deformation of the human palm, and its function in grasping, see IAKapandji, "The Physiology of the Joints, Volume 1: Upper Limb, 6e," Churchill Livingstone, 6th Edition, 2007, which is incorporated herein by reference in its entirety.

[0216] 18D is a simple flow diagram illustrating one embodiment of a classification of manipulation actions for food preparation during the kneading of dough stage. The kneading of dough stage 740 can be a mini-manipulation previously predefined in a mini-manipulation library database. The kneading of dough process 740 includes a sequence of actions (or short mini-manipulations) including grasping the dough stage 742, placing the dough on a surface stage 744, and repeating the kneading action until the desired shape is achieved stage 746.

[0217] FIG. 19 is a block diagram illustrating an example of a database library structure 770 of mini-operations that result in "cracking an egg with a knife." The mini-operation 770 for cracking an egg includes how to hold the egg in the correct position 772, how to hold the knife relative to the egg 774, what is the best angle to hit the egg with the knife 776, and how to open the cracked egg 778. Various possible parameters for each of 772, 774, 776, and 778 are tested to find the best way to perform a particular movement. For example, when holding an egg 772, various positions, orientations, and techniques for holding the egg are tested to find the optimal technique for holding the egg. Second, the robotic hand 72 picks up a knife from a predefined location. The step 774 for holding the knife explores various positions, orientations, and techniques for holding the knife to find the optimal technique for handling the knife. Third, the step of hitting the egg with a knife 776 is also tested with various combinations of hitting the knife on the egg to find the best way to hit the egg with the knife. The resulting best ways to perform the mini-operation 770 of cracking an egg with a knife are stored in a library database of mini-operations. The stored mini-operation 770 of cracking an egg with a knife will include the best way to hold the egg 772, the best way to hold the knife, and the best way to hit the knife on the egg 776.

[0218] To generate a mini-manipulation that results in cracking an egg with a knife, multiple parameter combinations must be tested to identify a parameter set that ensures the desired functional result of the egg being cracked. In this example, parameters are identified that determine how to grasp and hold the egg without crushing it. Through testing, an appropriate knife is selected and the proper placement of the fingers and palm to hold it for the smacking phase is found. A smacking motion that successfully cracks the egg is identified. An opening motion and / or force that allows the cracked egg to be successfully opened is identified.

[0219] The teaching / learning process for the robotic device 75 involves multiple iterative tests to identify the parameters necessary to achieve the desired end functional result.

[0220] These tests can be performed across a variety of scenarios. For example, the eggs can be different sizes. The eggs can be cracked in different places. The knife can be in different locations. The small-scale operation must be successful in all of these variables.

[0221] Once the learning process is complete, the results are stored as a collection of behavioral primitives that are known to work together to achieve a desired functional outcome.

[0222] 20 is a block diagram illustrating an example recipe execution 800 for a mini-manipulation with real-time adjustments through 3D modeling of a non-standard object 112. In recipe execution 780, the robotic hand 72 executes a mini-manipulation 770 to crack an egg with a knife, where an optimal technique for executing each of the movements in the knife-cracking operation 772, the knife-holding operation 774, the knife-striking operation 776, and the cracked egg opening operation 778 is selected from a mini-manipulation library database. The process of executing the optimal technique for performing each of the movements 772, 774, 776, and 778 ensures that the mini-manipulation 770 will achieve the same result (or a guarantee thereof) or substantially the same outcome for this particular mini-manipulation. The multi-modal three-dimensional sensor 20 provides real-time adjustment capability 112 for possible variations in one or more ingredients, such as the size and weight of the egg.

[0223] As an example of the operational relationship between the creation of mini-operations in Figure 19 and the execution of mini-operations in Figure 20, the specific variables associated with the "crack an egg with a knife" mini-operation include the initial x, y, and z coordinates of the egg, the initial orientation of the egg, the size of the egg, the shape of the egg, the initial x, y, and z coordinates of the knife, the initial orientation of the knife, the x, y, and z coordinates of where to crack the egg, and the speed and duration of the mini-operation. The "crack an egg with a knife" mini-operation variables thus identified are defined during the creation phase, where these identifiable variables can be adjusted by the robotic food preparation engine 56 during the execution phase of the associated mini-operation.

[0224] FIG. 21 is a flow diagram illustrating a software process 782 that captures a chef's food preparation movements within the standardized kitchen module to generate a software recipe file 46 from the chef studio 44. In step 784, within the chef studio 44, the chef 49 designs various components of a food recipe. In step 786, the robotic cooking engine 56 is configured to receive input of names, ID ingredients, and measurements for the recipe design selected by the chef 49. In step 788, the chef 49 moves food / ingredients into designated standardized cookware / utensils and to designated locations. For example, the chef 49 picks up two medium shallots and two medium garlic cloves, places eight crimini mushrooms on a cutting board, and moves two thawed 20 cm x 30 cm puff pastry units from freezer F02 to the refrigerator (fridge). In step 790, the chef 49 dons a capture glove 26 or haptic garment 622 having sensors that capture the chef's movement data for transmission to the computer 16. In step 792, the chef 49 begins working on his / her selected recipe from step 784. In step 794, the chef movement recording module 98 is configured to capture and record detailed chef movements in real time within the standardized robotic kitchen 50, including the force, pressure, and XYZ position and orientation of the chef's arms and fingers. In addition to capturing the chef's movements, pressure, and position, the chef movement recording module 98 is configured to record video (footage of cooking, ingredients, processes, and interactions) and sound (human voice, evaporation sounds during frying, etc.) during the entire food preparation process for the specific recipe. In step 796, the robotic cooking engine 56 is configured to store the captured data from step 794, including the chef's movements from the sensors on the capture glove 26 and the multi-mode three-dimensional sensor 30. In step 798, the recipe abstraction software module 104 is configured to generate a recipe script suitable for machine implementation.After the recipe data is generated and stored, in step 799 the software recipe file 46 is made available for sale or subscription to users through an app store or marketplace for users' computers in the home or restaurant, and also incorporating a robotic cooking receiving app on mobile devices.

[0225] 22 is a flow diagram 800 illustrating a software process for food preparation in a robotic standardized kitchen by a robotic device 75 based on one or more software recipe files 22 received from the chef studio system 44. In step 802, a user 24, via computer 15, selects a recipe to purchase or subscribe to from the chef studio 44. In step 804, the robotic food preparation engine 56 in the home robotic kitchen 48 is configured to receive input from the input module 50 regarding the recipe selected to prepare. In step 806, the robotic food preparation engine 56 in the home robotic kitchen 48 is configured to upload the selected recipe into the memory module 102 having the software recipe files 46. In step 808, the robotic food preparation engine 56 in the home robotic kitchen 48 is configured to calculate ingredient availability to complete the selected recipe and the approximate cooking time required to complete the dish. In step 810, the robotic food preparation engine 56 in the home robotic kitchen 48 is configured to analyze the prerequisites for the selected recipe to determine whether there are any shortages or missing ingredients, or whether there is insufficient time to serve the food according to the selected recipe and serving schedule. If the prerequisites are not met, in step 812, the robotic food preparation engine 56 in the home robotic kitchen 48 sends an alert indicating that the ingredients should be added to the shopping list or suggests an alternative recipe or serving schedule. However, if the prerequisites are met, the robotic food preparation engine 56 is configured to confirm the recipe selection in step 814. After the recipe selection is confirmed, in step 816, the user 60, via the computer 16, moves the food / ingredients into specific standardized containers and required locations. After the ingredients are placed in the identified designated containers and locations, in step 818, the robotic food preparation engine 56 in the home robotic kitchen 48 is configured to check whether the start time has been triggered.At this juncture, the home robotic food preparation engine 56 performs a second process check to ensure all prerequisites are met. If the robotic food preparation engine 56 in the home robotic kitchen 48 is not ready to begin the cooking process, the home robotic food preparation engine 56 continues checking the prerequisites in step 820 until the start time is triggered. If the robotic food preparation engine 56 is ready to begin the cooking process, in step 822, the raw food quality check module 96 in the robotic food preparation engine 56 processes the prerequisites for the selected recipe and inspects each ingredient item against the recipe description (e.g., one seared center-cut beef tenderloin) and conditions (e.g., expiration / purchase date, odor, color, texture, etc.). In step 824, the robotic food preparation engine 56 sets the time to the "0" stage and uploads the software recipe file 46 to one or more robotic arms 70 and robotic hands 72 to replicate the chef's cooking movements and produce the selected dish according to the software recipe file 46. In step 826, one or more robotic arms 72 and robotic hands 74 handle the ingredients and execute the cooking method / technique using movements identical to those of the chef's 49 arms, hands, and fingers with precise pressure, precise force, and the same XYZ positions at the same time intervals as captured and recorded from the chef's movements. During this time, the one or more robotic arms 70 and robotic hands 72 compare the cooking results against control data (e.g., temperature, weight, loss, etc.) and media data (e.g., color, appearance, smell, portion size, etc.), as illustrated in step 828. After the data is compared, the robotic device 75 (including the robotic arms 70 and robotic hands 72) aligns and adjusts the results in step 830. In step 832, the robotic food preparation engine 56 is configured to command the robotic device 75 to move the completed dish to a designated serving tray and place it on a counter.

[0226] 23 is a flow diagram illustrating one embodiment of a software process for creating, testing, validating, and storing various parameter combinations in relation to a mini-manipulation library database 840. The mini-manipulation library database 840 includes a one-time success test process 840 (e.g., the step of holding an egg) that is stored in a temporary library, and a step 860 that tests the one-time result combination in the mini-manipulation database library (e.g., the entire move to crack an egg). In step 842, the computer 16 creates a new mini-manipulation (e.g., the step of cracking an egg) that has multiple action primitives (or multiple discrete recipe actions). In step 844, the number of objects (e.g., eggs and knives) associated with the new mini-manipulation is identified. In step 846, the computer 16 identifies some discrete actions or moves. In step 848, the computer selects the maximum possible ranges for key parameters (such as object position, object orientation, pressure, and speed) associated with the particular new mini-manipulation. In step 850, for each primary parameter, the computer 16 tests and verifies each value of the primary parameter with all possible combinations with other primary parameters (e.g., holding the egg in one position but testing other orientations). In step 852, the computer 16 is configured to determine whether a particular primary parameter set combination produces reliable results. Verification of the results can be performed by the computer 16 or a human. If the determination is false, the computer 16 proceeds to step 856 to determine whether there are other primary parameter combinations that have not yet been tested. In step 858, the computer 16 increments the primary parameter by 1 in formulating the next parameter combination for further testing and evaluation of the next parameter combination. If the determination in step 852 is true, the computer 16 stores the successful primary parameter combination set in a temporary storage library in step 854. The temporary storage library stores one or more successful primary parameter combination sets (either those with the most successful or optimal tests or those with the fewest failed results).

[0227] In step 862, the computer 16 tests and verifies the particular successful parameter combination X times (e.g., 100 times). In step 864, the computer 16 calculates the number of failed results during the repeated testing of the particular successful parameter combination. In step 866, the computer 16 selects the next single successful parameter combination from the temporary library and returns the process to step 862 to test this parameter combination X times. If no more single successful parameter combinations remain, in step 868, the computer 16 stores the test results of one or more parameter combination sets that produce reliable (or guaranteed) results. If more than one reliable parameter combination set exists, in step 870, the computer 16 determines the best or optimal parameter combination set associated with the particular mini-manipulation and stores this optimal parameter combination set associated with the particular mini-manipulation in the mini-manipulation library database for use by the robotic device 75 in the standardized robotic kitchen 50 during the food preparation stage of the recipe.

[0228] Figure 24 is a flow diagram illustrating one embodiment 880 of a software process for creating tasks for mini-manipulations. In step 882, the computer 16 defines a specific robotic task (e.g., cracking an egg with a knife) for a robotic mini-hand manipulator to store in a database library. In step 884, the computer identifies all different object orientations (e.g., egg orientation when holding an egg) possible for each mini-step, and in step 886, identifies all different positions for holding a kitchen tool relative to an object (e.g., holding a knife relative to an egg). In step 888, the computer empirically identifies all possible ways to hold an egg and crack it with a knife using the correct (cutting) movement profile, pressure, and speed. In step 890, the computer 16 defines various combinations of holding the egg and positioning the knife relative to the egg to properly crack the egg (e.g., finding the optimal parameter combination of object orientation, position, pressure, speed, etc.). In step 892, the computer 16 performs a training and testing process to verify the reliability of various combinations of all variations, steps testing differences, etc., and repeats this process X times until reliability is assured for each mini-manipulation. In step 894, when the chef 49 is performing a certain food preparation task (e.g., cracking an egg with a knife), this task is interpreted into steps / tasks of small hand-operated tasks to be performed as part of the task. In step 896, the computer 16 stores the various combinations of mini-manipulations for that particular task in a database library. In step 818, the computer 16 determines whether there are further tasks to be defined and performed for every mini-manipulation. If there are further tasks to be defined, the process returns to step 882. Various embodiments of the kitchen module are possible, including a standalone kitchen module and an integrated robotic kitchen module. The integrated robotic kitchen module is housed within the conventional kitchen area of ​​a typical home. The robotic kitchen module operates in at least two modes: a robotic mode and a normal (manual) mode. Cracking an egg is an example of a mini-manipulation.Additionally, the mini-manipulation library database is applied to various tasks such as using a fork to grasp a piece of beef by applying the correct pressure in the correct direction and to the correct depth for the shape and depth of the meat. In step 900, the computer assembles a database library of predefined kitchen tasks, each containing one or more mini-manipulations.

[0229] 25 is a flow diagram illustrating a process 920 for allocating and utilizing a library of standardized kitchen tools, standardized objects, and standardized equipment in the standardized robotic kitchen. In step 922, the computer 16 assigns each kitchen tool, object, or equipment / utensil a code (or barcode) that predefines the tool, object, or equipment parameters, such as its three-dimensional position coordinates and orientation. This process standardizes various elements in the standardized robotic kitchen 50, including, but not limited to, standardized kitchen equipment, standardized kitchen tools, standardized knives, standardized forks, standardized containers, standardized pans, standardized utensils, standardized task spaces, standardized fixtures, and other standardized elements. In step 924, when performing a process-step in a cooking recipe, the robotic cooking engine is configured to direct one or more robotic hands to obtain the corresponding kitchen tool, object, equipment, utensil, or utensil when prompted to access a particular kitchen tool, object, equipment, utensil, or utensil according to the food preparation process for the particular recipe.

[0230] 26 is a flow chart illustrating a process 926 for identifying non-standard objects through three-dimensional modeling and inference. In step 928, the computer 16 uses sensors to detect non-standard objects, such as food ingredients, which may have different sizes, dimensions, and / or weights. In step 930, the computer 16 identifies the non-standard object using the three-dimensional modeling sensors 66 to capture shape, size, orientation, and position information, and the robotic hand 72 makes real-time adjustments to perform the appropriate food preparation task (e.g., cutting or picking up a steak).

[0231] 27 is a flow diagram illustrating a process 932 for mini-manipulation testing and learning. In step 934, the computer performs a food preparation task composition analysis in which each cooking operation (e.g., cracking an egg with a knife) is analyzed, decomposed, and constructed into a sequence of action primitives or mini-manipulations. In one embodiment, a mini-manipulation refers to a sequence of one or more action primitives that achieves a basic functional outcome (e.g., an egg is cracked or a vegetable is sliced) that progresses toward a particular result in preparing a food dish. In this embodiment, the mini-manipulations can be further described as low-level mini-manipulations, which refer to sequences of action primitives that require minimal interaction forces and rely almost exclusively on the use of the robotic device 75, or high-level mini-manipulations, which refer to sequences of action primitives that require a significant amount of interaction and interaction forces and control thereof. Process loop 936 focuses on the mini-manipulation and learning phase and consists of repeated testing (e.g., 100 times) to ensure the reliability of the mini-manipulations. In step 938, the robotic food preparation engine 56 is configured to evaluate knowledge of all possibilities for performing a food preparation stage or mini-manipulation, where each mini-manipulation is tested for orientation, position / velocity, angle, force, pressure, and speed with respect to the particular mini-manipulation. The mini-manipulation or motion primitive may include the robotic hand 72 and a standard object or the robotic hand 72 and a non-standard object. In step 940, the robotic food preparation engine 56 executes the mini-manipulation and determines whether the outcome can be considered successful or unsuccessful. In step 942, the computer 16 performs automated analysis and inference on the failure of the mini-manipulation. For example, a multi-modal sensor can provide sensory feedback data on the success or failure of the mini-manipulation. In step 944, the computer 16 is configured to make real-time adjustments to adjust parameters of the mini-manipulation execution process. In step 946, the computer 16 adds new information about the success or failure of parameter adjustments to the mini-manipulation library as a learning mechanism for the robotic food preparation engine 56.

[0232] FIG. 28 is a food recipe illustrating a process 950 for quality control and alignment functions for a robotic arm. In step 952, the robotic food preparation engine 56 loads a human chef replication software recipe file 46 through the input module 50. For example, the software recipe file 46 replicates a food preparation from Michelin-starred chef Arnd Beuchel's "Wiener Schnitzel." In step 954, the robotic device 75 executes a task based on a stored recipe script containing all movement / motion replication data, including the same movements of the torso, hands, fingers, etc., at the same pace, with the same pressure, force, and xyz positions as the recorded recipe data stored based on the actions of a human chef preparing the same recipe in a standardized kitchen module with standardized equipment. In step 956, the computer 16 monitors the food preparation process through multimodal sensors that generate raw data that is fed to abstraction software, where the robotic device 75 compares real-world output to control data based on multimodal sensory data (visual, audio, and any other sensory feedback). In step 958, the computer 16 determines whether any differences exist between the control data and the multi-modal sensory data. In step 960, the computer 16 analyzes whether the multi-modal sensory data deviates from the control data. If deviations exist, in step 962, the computer 16 makes adjustments to recalibrate the robotic arm 70, robotic hand 72, or other element. In step 964, the robotic food preparation engine 16 is configured to learn in process 964 by adding the adjustments made to one or more parameter values ​​to the knowledge database. In step 964, the robotic food preparation engine 16 is configured to learn in process 964 by adding the adjustments made to one or more parameter values ​​to the knowledge database. In step 968, the computer 16 stores updated revision information related to the corrected process, conditions, and parameters in the knowledge database.If there are no deviation differences from step 958, process 950 proceeds directly to step 970 and terminates execution.

[0233] Figure 29 is a table illustrating one embodiment of a database library structure 972 for mini-manipulation objects for use in the standardized robotic kitchen. The database library structure 972 shows some fields for inputting and storing information about a specific mini-manipulation, including (1) the name of the mini-manipulation, (2) the assignment code of the mini-manipulation, (3) the codes of the standardized equipment and standardized tools associated with performing the mini-manipulation, (4) the initial position and orientation of the manipulated (standard or non-standard) objects (ingredients and tools), (5) parameters / variables defined by the user (or extracted from a recorded recipe during execution), and (6) the robotic hand movement sequence over time (control signals for all servos) and the mini-manipulation's connected feedback parameters (from any sensors or video surveillance system). The parameters for a specific mini-manipulation can vary depending on the complexity and the object on which the mini-manipulation needs to be performed. In this example, four parameters are identified: starting XYZ position coordinates within the spatial domain of the standardized kitchen module, velocity, object size, and object shape. Both object size and object shape can be defined or described by non-standard parameters.

[0234] 30 is a table illustrating a standard object database library structure 974 for use in the standardized robotic kitchen 50, including three-dimensional models of the standard objects. The standard object database library structure 974 shows some fields for storing information related to the standard objects, including: (1) the name of the object; (2) an image of the object; (3) an assigned code for the object; (4) a virtual 3D model with the object's full dimensions in an XYZ coordinate matrix at a predefined preferred resolution; (5) a virtual vector model of the object (if available); (6) a definition and marking of the object's task elements (elements that can be contacted by hands and other objects for manipulation); and (7) the object's initial, standard orientation for each specific manipulation. The sample database structure 974 of the electronic library includes three-dimensional models of all standard objects (i.e., all kitchen equipment, kitchen tools, kitchen utensils, and containers) that are part of the overall standardized kitchen module 50. The three-dimensional models of the standard objects can be visually captured by a three-dimensional camera and stored in the database library structure 974 for subsequent use.

[0235] FIG. 31 illustrates the implementation of process 980 by using a robotic hand 640 with one or more sensors 642 to check the quality of ingredients as part of the recipe replication process by the standardized robotic kitchen. A multi-modal sensor system video sensing element using color detection and spectral analysis to detect discoloration indicative of possible spoilage can implement process 982. An ammonia-sensitive sensor system, whether embedded in the kitchen or part of a mobile probe handled by the robotic hand, can similarly be used to detect further possible spoilage. Further tactile sensors in the robotic hand and fingers would enable verifying the freshness of ingredients through contact-sensing process 984, where firmness and resistance to contact force are measured (amount and speed of deflection as a function of compression distance). For example, in fish, gill color (red) and moisture content are indicators of freshness, as are eyes that should be clear (unclouded), and the ideal temperature for properly thawed fish flesh should not exceed 40°F. Further contact sensors on the fingertips can perform further quality checks 986 related to the temperature, texture, and total weight of the ingredient through touch, rubbing, and holding / picking movements. All of the data collected through these tactile sensors and video images can be used within the processing algorithms to make decisions about the freshness of the ingredient and whether to use or discard it.

[0236] FIG. 32 illustrates a robotic recipe script replication process 988 in which a multi-modal sensor-equipped head 20 and a dual arm with a multi-fingered hand 72 holding ingredients and tools interact with a cooking implement 990. The robot sensor head 20 with its multi-modal sensor unit is used to continuously model and monitor the three-dimensional task space being worked on by both robot arms, while also providing data to the task abstraction module to identify tools, implements, and their contents and variables, compare them to the recipe steps generated by the cooking process sequence, and ensure that execution is proceeding according to the sequence data for the computer-stored recipe. Additional sensors in the robot sensor head 20 are used in the audio domain to hear and smell during critical parts of the cooking process. The robot hand 72 and its tactile sensors are used to properly handle each ingredient, in this case an egg; sensors in the fingers and palm detect available eggs, for example, by their surface texture and weight and distribution, and can hold and orient them without breaking them. The multi-fingered robotic hand 72 can also pick up and handle certain cooking implements, such as a bowl in this case, and grasps and manipulates the implement (a whisk in this case) with the appropriate movements and force application to properly process the food ingredients as specified in the recipe script (e.g., cracking eggs to separate the yolks and beating the egg whites until a sticky consistency is achieved).

[0237] 33 depicts an ingredient storage system concept 1000 in which a food storage container 1002 capable of storing any of the required cooking ingredients (e.g., meat, fish, poultry, shellfish, vegetables, etc.) is equipped with sensors to measure and monitor the freshness of each ingredient. The monitoring sensors embedded within the food storage container 1002 include, but are not limited to, an ammonia sensor 1004, a volatile organic compound sensor 1006, an internal container temperature sensor 1008, and a humidity sensor 1010. Additionally, a manual probe (or detection device) 1012 having one or more sensors can be used, whether used by a human chef or a robotic arm and hand, to enable primary measurements (such as temperature) within the spatial region of a larger ingredient (e.g., the internal temperature of meat).

[0238] 34 depicts a measurement and analysis process 1040 performed as part of a freshness and quality check on food ingredients placed in a food storage container 1042 that includes sensors and detection devices (e.g., temperature probes / probes) for online analysis of food freshness via cloud computing or computer-based analysis over the Internet or a computer network. The container uses a metadata tag 1044 specifying the container ID to send a data set including temperature data 1046, humidity data 1048, ammonia concentration data 1050, and volatile organic compound data 1052 to a main server via a wireless data network communication step 1056, where a food quality control engine processes the container data. A processing step 1060 uses the container-specific data 1044 and compares it with values ​​and ranges of data considered acceptable, stored in and retrieved from a medium 1058 by a data acquisition and storage process 1054. A set of algorithms then makes a decision regarding the suitability of the food ingredients and provides real-time food quality analysis results via a separate data network communication process 1062. The quality analysis results are then used in another process 1064 where they are sent to a robotic arm for further action or can be displayed remotely on a screen (smart phone or other display) for the user to decide whether the ingredient should be used in a cooking process for later consumption or discarded as spoilage.

[0239] 35 depicts the functions and process-steps of a pre-filled ingredient container 1070 with one or more programmable dispenser controls for use in the standardized robotic kitchen 50, whether in the standardized robotic kitchen or in a chef studio. The ingredient containers 1070 are designed in various sizes 1082 and for various uses to accommodate appropriate storage environments 1080 containing perishable items, using refrigeration, freezing, chilling, etc. to achieve specific storage temperature ranges. Additionally, the pre-filled ingredient storage containers 1070 can also be designed to accommodate various types of ingredients 1072, with the containers being pre-labeled and pre-filled with solid (e.g., salt, flour, rice), viscous / paste (e.g., mustard, mayonnaise, marzipan, jam), or liquid (e.g., water, oil, milk, juice) ingredients, where the dispensing process 1074 utilizes a variety of different attachment devices (e.g., droppers, chutes, peristaltic dosing pumps) depending on the type of ingredient, and precise computer-controllable dispensing using a dosage control engine 1084 running a dosage control process 1076 ensures that the correct amount of ingredient is dispensed at the correct time. Note that recipe-specified dosages can be adjusted to suit individual taste preferences or diets (e.g., low sodium) through the menu interface or even through a remote phone application. The dosage determination process 1078 is performed by the dosage control engine 1084 based on the amounts specified in the recipe, and the dispensing step is performed either by manual release command or by remote computer control based on the detection of a certain dispensing container at the outlet point of the dispenser.

[0240] FIG. 36 is a block diagram illustrating a recipe structure and process 1090 for food preparation in the standardized robotic kitchen 50. The food preparation process 1090 is shown divided into multiple stages along a cooking timeline, with each stage having one or more raw data blocks for each stage 1092, 1094, 1096, and 1098. The data blocks can include elements such as video images, audio recordings, text descriptions, and machine-readable and machine-understandable instruction sets and commands that form part of a control program. The raw data sets are contained within the recipe structure and represent each cooking stage along the timeline, divided into many time-ordered stages with various time interval and time sequence levels, from the beginning of the recipe replication process to the end of the cooking process, or any subprocess therein.

[0241] 37A-37C are block diagrams illustrating example recipe search menus for use in the standardized robotic kitchen. As shown in FIG. 37A, the recipe search menu 1110 presents criteria and ranges such as most popular categories, such as type of diet (e.g., Italian, French, Chinese), dish ingredient base (e.g., fish, pork, beef, pasta), or cooking time range (e.g., less than 60 minutes, between 20 and 40 minutes), as well as keyword search implementations (e.g., ricotta cavatelli, migliaccio cake). The selected personalized recipe can exclude recipes with allergenic ingredients, which the user can indicate in their personal user profile, where the user can define, by the user or from another source, which allergenic ingredients they can abstain from. In FIG. 37B, the user can select search criteria including requirements consisting of cooking time less than 44 minutes, serving portions sufficient for seven people, presenting vegetarian options, and having a total calorie count of 4521 or less, as shown in this figure. In Figure 37C, various types of cuisines 1112 are shown, where the menu 1110 has hierarchical levels where the user can select a category (e.g., type of cuisine) 1112, which then expands to the next subcategory (e.g., appetizers, salads, entrees...) to further refine the selection. An example screenshot of recipe creation and submission is shown in Figure 37D. Another screenshot depicting ingredient types is shown in Figure 37E.

[0242] One embodiment of a flow diagram for recipe filters, ingredient filters, equipment filters, account and social network access, personal associate pages, shopping cart pages, and information about purchased recipes, registration settings, and recipe creation is illustrated in FIGS. 37F through 37O, illustrating various functions the robotic food preparation software 14 can implement based on filtering the database and present information to the user. As shown in FIG. 37F, a platform user can access the recipe section and select a desired recipe filter 1130 for automated robotic cooking. The most common filter types include dietary type (e.g., Chinese, French, Italian), cooking type (e.g., baked, steamed, fried), vegetarian, and diabetic. From the filtered search results, the user can view recipe details such as descriptions, photos, ingredients, prices, and ratings. In FIG. 37G, the user can select a desired ingredient filter 1132 for themselves, such as organic, ingredient type, or ingredient brand. In FIG. 37G, a user can apply equipment filters 1134, such as equipment type, brand, and manufacturer, to the automated robotic kitchen module. After selection, the user will be able to purchase recipes, ingredients, or equipment directly through the system portal from the relevant sellers. The platform allows users to create their own additional filters and parameters, making the entire system customizable and constantly updated. User-added filters and parameters will appear as system filters after approval by the moderator.

[0243] In Figure 37H, users can connect to other users and sellers through the platform's social professional network by logging into their user account 1140. Network users' identities are optionally verified through credit card and address details. The account portal also serves as a trading platform for users to share or sell their recipes and advertise them to other users. Users can also manage their account funds and equipment through the account portal.

[0244] An example of collaboration between users of the platform is demonstrated in Figure 37J. One user can provide all the information and details about an ingredient, and another user can do the same for their equipment. All information must be filtered through a moderator before being added to the platform / website database. In Figure 37K, a user can review information about their purchases in the shopping cart 1142. They can also change other options such as delivery and payment method. A user can also purchase additional ingredients or equipment based on the recipe in their shopping cart.

[0245] 37L shows additional information about purchased recipes that can be accessed from the recipe page 1144. The user can read, listen, or see how to cook the recipe, as well as perform automated robotic cooking. Communication with the seller or technical support regarding the recipe is also possible from the recipe page.

[0246] FIG. 37M is a block diagram illustrating the different platform layers from the "My Account" page 1136 and from the Settings page 1138. From the "My Account" page, users will be able to read professional cooking news or blogs and can write and publish articles. As shown in FIG. 37N, through the Recipes page under "My Account," there are multiple ways users can create their own recipes 1146. Users can create recipes by creating automated robotic cooking scripts by either capturing chef cooking moves or selecting operation sequ...

Claims

1. one or more robotic arms; one or more robot end effectors coupled to each of the robot arms; a mini-manipulation library including one or more mini-manipulations; and at least one processor communicatively coupled to the one or more robotic arms, The at least one processor receiving a software recipe file including one or more cooking operations; retrieving the one or more mini-operations from the mini-operations library corresponding to the one or more cooking operations related to the software recipe file; and and executing the one or more mini-manipulations from the mini-manipulation library by controlling the robotic arm and the one or more robotic end effectors to repeat the one or more cooking actions associated with the software recipe file to prepare a food dish or portion thereof; each mini-operation of the one or more mini-operations includes one or more parameters; each mini-manipulation of said one or more mini-manipulations is obtainable from said mini-manipulation library based on said one or more parameters; Each mini-manipulation includes one or more motion primitives or at least one smaller mini-manipulation, and Each mini-manipulation is pre-tested multiple times by the one or more robotic arms and the one or more robotic end effectors using multiple parameter combinations until the one or more robotic arms and the one or more robotic end effectors are able to execute a particular mini-manipulation that is within a threshold that achieves a predefined functional outcome.

2. one or more robotic arms; one or more robot end effectors coupled to each of the robot arms; a mini-operation library including a plurality of mini-operations; and at least one processor communicatively coupled to the one or more robotic arms, The at least one processor receiving a software recipe file including one or more cooking operations; retrieving the one or more mini-operations from the mini-operations library corresponding to the one or more cooking operations related to the software recipe file; and and executing the one or more mini-manipulations from the mini-manipulation library by controlling the robotic arm and the one or more robotic end effectors to repeat the one or more cooking actions associated with a software recipe file to prepare a food dish or portion thereof; each mini-operation of said one or more mini-operations comprises one or more parameters, and each mini-operation of said one or more mini-operations is obtainable from said mini-operation library based on said one or more parameters; Each mini-manipulation includes one or more motion primitives or at least one smaller mini-manipulation; Each mini-manipulation has been pre-tested multiple times by the at least one robotic arm and the at least one robotic end effector using multiple parameter combinations until the at least one robotic arm and the at least one robotic end effector are able to perform a particular mini-manipulation that achieves a predefined functional outcome within a predefined threshold range.

3. one or more robotic arms; one or more robot end effectors coupled to each of the robot arms; and at least one processor communicatively coupled to the one or more robotic arms, The at least one processor receiving a software recipe file including one or more cooking operations; Retrieving the one or more mini-operations from a mini-operation library corresponding to the one or more cooking operations related to the software recipe file; and performing said one or more mini-manipulations from said mini-manipulation library by controlling said robotic arm and said one or more robotic end effectors associated with said software recipe file to prepare a food dish or portion thereof; each mini-operation of said one or more mini-operations comprising one or more parameters, each mini-operation of said one or more mini-operations being obtainable from said mini-operation library based on said one or more parameters; 13. The robotic kitchen system of claim 12, wherein each mini-manipulation includes one or more movement primitives or at least one smaller mini-manipulation that are pre-tested multiple times with multiple parameter combinations by the one or more robotic arms and the one or more robotic end effectors until the one or more robotic arms and the one or more robotic end effectors are able to execute a particular mini-manipulation that is within a threshold that achieves a pre-defined functional outcome in light of one or more parameter changes.

4. one or more robotic arms; one or more robot end effectors coupled to each of the robot arms; a mini-operation library including a plurality of mini-operations; and at least one processor communicatively coupled to the one or more robotic arms, The at least one processor Receiving one or more user commands to generate a first data set including one or more cooking actions; operative to access the mini-manipulation library to obtain a second data set based on the first data set; In this case, the second data set comprises one or more mini-manipulations corresponding to the one or more cooking actions in the first data set, each mini-manipulation of the one or more mini-manipulations being obtainable from the mini-manipulation library by the one or more parameters, each mini-manipulation comprising one or more movement primitives or at least one smaller mini-manipulation; each mini-manipulation having been pre-tested one or more times by the one or more robotic arms and the one or more robotic end effectors using a combination of multiple parameters until the one or more robotic arms and the one or more robotic end effectors perform a particular mini-manipulation of the one or more mini-manipulations to achieve a pre-defined functional outcome; and The at least one processor further comprises: a robotic food preparation system configured to execute the second data set including the one or more mini-manipulations and actuate the one or more robotic arms coupled to the one or more robotic end effectors within an instrumented environment to prepare a food dish.

5. 1. A method of preparing a food dish by a robotic device of a robotic kitchen system, the robotic device having one or more robotic arms and one or more robotic end effectors respectively coupled to each of the robotic arms, comprising: receiving, by at least one processor, one or more instructions to prepare at least a portion of the food dish, the one or more instructions including one or more cooking actions; accessing, by said at least one processor, a mini-manipulation library to obtain one or more mini-manipulations from said mini-manipulation library comprising a plurality of mini-manipulations corresponding to said one or more cooking actions, each mini-manipulation of said plurality of mini-manipulations being obtainable from said mini-manipulation library by said one or more parameters, each mini-manipulation comprising one or more movement primitives or at least one smaller mini-manipulation, to verify a particular mini-manipulation of said mini-manipulation library; accessing said mini-manipulation library, each mini-manipulation having been pre-tested multiple times by said one or more robotic arms and said one or more robotic end effectors using multiple respective parameter combinations until said one or more robotic arms and said one or more robotic end effectors execute a particular mini-manipulation of said plurality of mini-manipulations to achieve a predefined functional outcome; and performing, by the at least one processor, the one or more mini-manipulations to actuate the one or more robotic arms coupled to the one or more robotic end effectors within an instrumented environment to prepare at least a portion of the food dish.

6. 1. A method for generating a mini-manipulation library for a robotic device of a robotic kitchen system, the robotic device having one or more robotic arms and one or more robotic end effectors coupled to each robotic arm, comprising: receiving, by at least one processor, one or more cooking operations; generating, by said at least one processor, a mini-manipulation library comprising one or more mini-manipulations, said one or more mini-manipulations corresponding to said one or more cooking actions, each mini-manipulation of said one or more mini-manipulations being obtainable from said mini-manipulation library by one or more parameters, each mini-manipulation comprising a movement, one or more primitives or at least one smaller mini-manipulation; wherein each of the one or more mini-operations is pre-tested multiple times with multiple parameter combinations to determine a specific parameter combination; generating said mini-manipulation library, each mini-manipulation being pre-tested a number of times with a number of parameter combinations by said one or more robotic arms and said one or more robotic end effectors until said one or more robotic arms and said one or more robotic end effectors are able to execute a particular mini-manipulation of said one or more mini-manipulations to obtain a predefined functional outcome; and and storing the one or more mini-manipulations in the mini-manipulation library for subsequent use during run-time of the robotic kitchen.

7. one or more robotic arms; one or more robot end effectors coupled to each of the robot arms; a mini-manipulation library including one or more mini-manipulations; and at least one processor communicatively coupled to the one or more robotic arms, The at least one processor receiving a software recipe file including one or more cooking operations; Retrieving the one or more mini-operations from the mini-operation library corresponding to the one or more cooking operations related to the software recipe file; and performing said one or more mini-manipulations from said mini-manipulation library by controlling said robotic arm and said one or more robotic end effectors associated with said software recipe file to prepare a food dish or portion thereof; each mini-operation of said one or more mini-operations comprising one or more parameters, each mini-operation of said one or more mini-operations being obtainable from said mini-operation library based on said one or more parameters; A robotic kitchen system in which each mini-manipulation includes at least one movement primitive or at least one smaller mini-manipulation that is designed and tested within a threshold to achieve a predefined functional outcome.

8. one or more robot end effectors; one or more robotic arms, each robotic arm of the one or more robotic arms coupled to a respective one of the one or more robotic end effectors; a software library for storing a plurality of robotic cooking actions, where at least one robotic cooking action of the plurality of robotic cooking actions has one or more parameters and has been tested a plurality of times in a testing environment for cooking, and where the at least one robotic cooking action of the plurality of robotic cooking actions has been tested using a plurality of different parameter combinations to determine a particular parameter combination for achieving a predetermined functional outcome with a predetermined probability of success; at least one processor communicatively coupled to the software library; and a memory for storing a plurality of instructions, The instructions, when executed by the at least one processor, cause the at least one processor to: receiving a software recipe for cooking at least a portion of a food dish in an instrumented cooking environment of the robotic kitchen system, the software recipe including one or more recipe steps; acquiring one or more robotic cooking actions among the plurality of robotic cooking actions, where the one or more robotic cooking actions correspond to the one or more recipe steps in the software recipe; and operating the one or more robotic end effectors and the one or more robotic arms within the instrumented cooking environment of the robotic kitchen system to perform the one or more robotic cooking actions to cook at least a portion of the food dish, where each robotic cooking action of the one or more robotic cooking actions is performed with the specified parameter combination to achieve the predetermined functional outcome. A robot kitchen system that makes this possible.

9. 10. The robotic kitchen system of claim 8, wherein the at least one robotic cooking action of the plurality of robotic cooking actions defines a pre-planned movement of the one or more robotic arms and the one or more robotic end effectors.

10. The inspection environment is the same as the instrumented cooking environment of the robotic kitchen system, 10. The robotic kitchen system of claim 8, wherein the inspection environment is separate and remote from the instrumented cooking environment.

11. the one or more parameters of the at least one robotic cooking action coordinate a movement of the one or more robotic arms and the one or more robotic end effectors when the at least one robotic cooking action is performed within the instrumented cooking environment; 10. The robotic kitchen system of claim 8, wherein the one or more robotic cooking actions are retrieved from the software library based on the one or more parameters.

12. the at least one robotic cooking action of the plurality of robotic cooking actions is associated with one or more timing parameters, the one or more timing parameters including a start time, a duration, an end time, or a combination thereof; 10. The robotic kitchen system of claim 8, wherein the memory stores additional instructions that, when executed by the at least one processor, cause the at least one processor to implement the at least one robotic cooking action in accordance with the one or more timing parameters.

13. the at least one robotic cooking action of the plurality of robotic cooking actions includes one or more movement primitives that cooperate to achieve the predetermined functional outcome; 10. The robotic kitchen system of claim 8, wherein the one or more parameters include one or more sensor data parameters, one or more object data parameters, one or more object-related data parameters, one or more timing parameters, or any combination thereof.

14. execution of the at least one robotic cooking action results in a transition of an object in the instrumented cooking environment of the robotic kitchen system from a first state to a second state; the first state is different from the second state; 10. The robotic kitchen system of claim 8, wherein the second state corresponds to a desired change in the instrumented cooking environment defined as a predetermined functional outcome.

15. At least one of the one or more recipe steps in the software recipe is associated with control data; The memory stores additional instructions that, when executed by the at least one processor, cause the at least one processor to: comparing the control data associated with the at least one recipe step to an outcome achieved by execution of the one or more robotic cooking actions corresponding to the at least one recipe step; determining whether the one or more robotic arms and the one or more robotic end effectors successfully performed the one or more robotic cooking actions based on the comparison; The robotic kitchen system of claim 8 .

16. the at least one robotic cooking action is a mini-manipulation; 10. The robotic kitchen system of claim 8, wherein the mini-manipulation comprises a collection or sequence of one or more movement primitives that cooperate to achieve the predetermined functional outcome.

17. a motion primitive in the one or more motion primitives is an indivisible robot motion; 10. The robotic kitchen system of claim 8, wherein the set or sequence of one or more movement primitives includes sensing and actuator actions.

18. 10. The robotic kitchen system of claim 8, wherein the one or more parameters include one or more parameters defining ingredients used in the robotic cooking action, or one or more ingredient-related data parameters.

19. one or more robot end effectors; one or more robotic arms, each robotic arm of the one or more robotic arms coupled to a respective one of the one or more robotic end effectors; Inspection environment for cooking, A software library for storing multiple robotic cooking actions; At least one processor; and a memory for storing instructions, The instructions, when executed by the at least one processor, cause the at least one processor to: generating a robotic cooking action based on a plurality of movement primitives, the robotic cooking action including one or more parameters; testing the robotic cooking action by performing the robotic cooking action using a plurality of different parameter combinations for the one or more parameters of the robotic cooking action in the testing environment; determining a particular parameter among said plurality of different parameter combinations that achieves a predetermined functional outcome; testing the robotic cooking operations multiple times with the particular parameter combinations to ensure that a predetermined functional outcome is achieved with a predefined probability of success; if the predefined probability of success exceeds a threshold, storing the robotic cooking action at the particular parameter combination in the software library for later use within an instrumented cooking environment; A robot kitchen system that makes this possible.

20. The memory stores additional instructions that, when executed by the at least one processor, cause the at least one processor to: receiving, via a user interface, user input that the robotic cooking operation has achieved the predetermined functional outcome; or determining whether an outcome achieved by execution of the robotic cooking action is within a certain range of values ​​based on sensor data collected as feedback by one or more sensors of the robotic kitchen system; 20. The robotic kitchen system of claim 19,

21. The kitchen environment and one or more robotic arms; one or more robotic end effectors, each robotic arm coupled to a respective robotic end effector; at least one processor, The at least one processor receiving a software recipe file including one or more cooking operations; retrieving one or more mini-manipulations from a mini-manipulation library corresponding to the one or more cooking actions associated with the software recipe file, where each robotic action within the one or more mini-manipulations includes one or more movement primitives or at least one small robot action, and each robotic action has been tested multiple times using multiple parameter combinations to determine a particular parameter combination in achieving a predetermined fidelity threshold of a predetermined functional outcome; performing the one or more mini-manipulations through movement of the one or more robotic arms and the one or more robotic end effectors to replicate the one or more cooking actions associated with the software recipe file within the kitchen environment to prepare a food dish; A robotic kitchen system configured to:

22. Each robotic action of the one or more mini-manipulations comprises one or more parameters, and each robotic action of the one or more mini-manipulations is retrievable from the mini-manipulation library based on the one or more parameters, the one or more parameters comprising one or more environmental parameters; 22. The robotic kitchen system of claim 21, wherein the at least one processor is operable to execute the one or more mini-manipulations to operate the one or more robotic arms coupled to the one or more robotic end effectors within an instrumented environment to prepare the food dish in accordance with the one or more environmental parameters.

23. 22. The robotic kitchen system of claim 21, wherein the at least one processor compares functional outcomes of control data associated with the one or more cooking operations from the software recipe file with functional outcomes of the one or more robotic arms and the one or more robotic end effectors performing the one or more cooking operations to verify that the one or more robotic arms and the one or more robotic end effectors successfully performed the one or more mini-manipulations, where the control data includes timing, color, smell, temperature, image, humidity, texture, taste, weight loss, or portion size, or ambience of the kitchen environment.

24. 22. The robotic kitchen system of claim 21, wherein the at least one processor is further operable to command the one or more robotic arms and the one or more robotic end effectors to perform the one or more robotic movements to sequentially cook multiple food dishes.

25. 22. The robotic kitchen system of claim 21, wherein the predetermined fidelity threshold is comprised within a threshold of an optimum in achieving a predetermined functional outcome, the optimum threshold being task specific and defaulting to 1% of the optimum if not specified otherwise for each given domain specific application.

26. The software recipe file includes a plurality of stages S1, S2, S3 ... Sj ... Sn, each stage of the plurality of stages includes one or more mini-operations, and the overall probability of success of the software recipe file is equal to (0.99) multiplied by the number of stages S1, S2, S3 ... Sj ... Sn. n This is expressed by the following formula:

22. The robotic kitchen system of claim 21, where the term P(Si) represents the probability of success of each stage for a plurality of n stages.

27. 22. The robotic kitchen system of claim 21, wherein the at least one processor is further operable to command the one or more robotic arms and the one or more robotic end effectors to perform the one or more mini-manipulations to cook multiple food dishes in series or in parallel.

28. the combination of parameters for each tested robot motion includes one or more object data parameters, one or more object-related data parameters, and / or one or more timing parameters; 22. The robotic kitchen system of claim 21, wherein the one or more object data parameters include one or more kitchen cookware, one or more smart appliances, or one or more ingredients, and the one or more object-related data parameters include one or more ingredient quantities associated with an ingredient, one or more ingredient forms associated with the ingredient, or one or more ingredient shapes associated with the ingredient.

29. 22. The robotic kitchen system of claim 21, wherein the one or more cooking operations comprise a multi-step process file including a first food preparation stage and a second food preparation stage, the first food preparation stage having one or more first mini-operations, and the second food preparation stage having one or more second mini-operations.

30. one or more robotic arms; one or more robotic end effectors, each robotic arm coupled to a respective robotic end effector; at least one processor, The at least one processor receiving a software recipe file including one or more cooking operations; retrieving one or more mini-operations from a robotic operation library corresponding to the one or more cooking operations associated with the software recipe file, where each robotic operation within the one or more mini-operations includes one or more movement primitives or at least one small robot operation, each robotic operation having one or more parameters, and which has been tested multiple times using multiple parameter combinations to determine a particular parameter combination that achieves a predetermined threshold of functional performance; performing the one or more mini-manipulations through movement of the one or more robotic arms and the one or more robotic end effectors to replicate the one or more cooking actions associated with the software recipe file within a kitchen environment to prepare a food dish; A robotic kitchen system configured to:

31. a robot having one or more robotic arms and one or more robotic end effectors; a software library for storing a plurality of robotic cooking actions, Each robotic cooking action includes one or more movements for the robot to achieve a predetermined outcome related to preparing a food dish, each robotic cooking action having one or more parameters for operating the robot; each robotic cooking action is tested multiple times in a testing environment by performing the robotic cooking action using multiple different combinations of parameters of the robotic cooking action to determine a particular parameter combination that achieves a predetermined outcome associated with cooking the food dish with a predefined probability of success; at least one processor communicatively coupled to the software library; and a memory for storing a plurality of instructions, The instructions, when executed by the at least one processor, cause the at least one processor to: receiving a software recipe for cooking a food dish, the software recipe including one or more recipe steps; retrieving one or more robotic cooking actions from the software library corresponding to the one or more recipe steps; executing, within an instrumented environment, the one or more movements of the one or more robotic arms and robotic end effectors associated with each captured robotic cooking action to prepare the food dish using the particular parameter combination of each robotic cooking action to achieve a predetermined outcome of each robotic cooking action with at least a predetermined probability of success; A robot kitchen system that makes this possible.

32. 32. The robotic kitchen system of claim 31, wherein the particular parameter combination that achieves a predetermined outcome of a robotic cooking action associated with preparing the food dish with a predetermined probability of success comprises a parameter combination that performs the robotic cooking action multiple times within a predetermined threshold of an optimum value or combination thereof.

33. the test environment in which each robotic cooking action was tested is substantially structurally identical to the instrumented environment in which the robotic kitchen system executes the robotic cooking actions to prepare a food dish; 32. The robotic kitchen system of claim 31, wherein the inspection environment is separate and remote from the instrumentation environment.

34. 32. The robotic kitchen system of claim 31, wherein at least one of the parameters of a robotic cooking action defines a movement of the one or more robotic arms and the one or more robotic end effectors when the at least one robotic cooking action is performed, the movement being defined by at least the initial position, the end position, and speed parameters for performing the movement from an initial position to an end position.

35. Each robotic cooking action is associated with one or more timing parameters, the timing parameters including at least one of a start time, a duration, and an end time; 32. The robotic kitchen system of claim 31, wherein the memory stores additional instructions that, when executed by the at least one processor, cause the at least one processor to perform the at least one robotic cooking action in accordance with the one or more timing parameters.

36. The movements of the robotic cooking actions include one or more movement primitives that cooperate to achieve a predetermined outcome; 32. The robotic kitchen system of claim 31, wherein the one or more parameters include one or more of a sensor data parameter, an object data parameter, a timing parameter, or an environmental parameter.

37. At least one of the one or more recipe steps in the software recipe is associated with quality check data; the quality check data includes at least one of a temperature, a weight, a shape of the food dish or of an item within the food dish; The memory stores additional instructions that, when executed by the at least one processor, cause the at least one processor to: comparing the quality check data associated with the at least one recipe step with outcomes achieved by execution of the one or more robotic cooking actions corresponding to the at least one recipe step; determining whether the one or more robotic arms and the one or more robotic end effectors successfully performed the one or more robotic cooking actions based on the comparison; 32. The robotic kitchen system of claim 31 .

38. 32. The robotic kitchen system of claim 31, wherein the robot includes one or more actuators for repositioning the one or more robot arms or the one or more end effectors within the instrumented environment to extend the reachability of the robot.

39. Cooking inspection environment and a robot having one or more robot arms and one or more robot end effectors disposed in the inspection environment; a software library for storing a plurality of robotic cooking actions associated with steps of cooking a food dish; At least one processor; and a memory storing instructions, the memory storing instructions comprising: The instructions, when executed by the at least one processor, cause the at least one processor to: generating robotic cooking actions based on a plurality of motion primitives, where each motion primitive defines a motion of the robot, and each robotic cooking action has one or more parameters that cause the robot to move; iteratively testing the robotic cooking actions by executing the robotic cooking actions using multiple different parameter combinations for the movement primitives of the robotic cooking actions; determining, for each robotic cooking action, a particular combination of the parameters of the movement primitives of the robotic cooking action that achieves a predetermined outcome of the robotic cooking action; testing the robotic cooking action multiple times with the particular parameter combinations of the movement primitives of the robotic cooking action to determine a success probability of the robotic cooking action achieving a predetermined functional outcome using the particular parameter combinations; storing the robotic cooking action at the particular parameter combination in the software library for later use within an instrumented environment only if the determined probability of success exceeds a threshold; A robot kitchen system that makes this possible.

40. 40. The robotic kitchen system of claim 39, wherein the memory stores additional instructions that, when executed by the at least one processor, enable the at least one processor to automatically determine, based on sensor data collected as feedback data by one or more sensors of the robotic kitchen system, whether a performance achieved by performance of the robotic cooking action is within a threshold of an optimal performance of the robotic cooking action.

41. 40. The robotic kitchen system of claim 39, wherein the robot includes one or more actuators for repositioning the one or more robot arms or the one or more end effectors within the instrumented environment to extend the reachability of the robot.

42. The kitchen environment and a robot having one or more robotic arms and one or more robotic end effectors; at least one processor, The at least one processor receiving a software recipe file associated with at least a portion of a food dish, the software recipe file including one or more cooking operations; obtaining one or more mini-manipulations corresponding to said cooking action from a mini-manipulation library, where each mini-operation includes a sequence of movements of the robot to achieve an outcome associated with the food dish; each mini-manipulation is tested multiple times in a testing environment using multiple different parameter combinations of said mini-manipulation to determine a particular parameter combination that achieves a predetermined outcome with a predetermined level of fidelity for the outcome; executing the one or more mini-operations by operating the robot using the specific parameter combination for each mini-operation to replicate the one or more cooking actions associated with the software recipe file within the kitchen environment to cook at least a portion of the food dish with at least the predetermined level of fidelity; A robotic kitchen system configured to:

43. each mini-manipulation is associated with one or more parameters of said mini-manipulation, said one or more parameters including one or more environmental parameters; 43. The robotic kitchen system of claim 42, wherein the at least one processor is operable to execute mini-manipulations to operate one or more robotic arms or robotic end effectors within an instrumented environment to prepare the food dish in accordance with the one or more environmental parameters.

44. The at least one processor further comprises: comparing predefined functional outcomes encoded in quality check data associated with the cooking operation from the software recipe file with actual outcomes of the one or more robotic arms and the one or more robotic end effectors performing the mini-manipulation to verify that the one or more robotic arms and the one or more robotic end effectors successfully performed the mini-manipulation, where the quality check data includes timing of the mini-manipulation, temperature of a food item, an image of a food item, a size of a food item, or environmental conditions of the kitchen environment; 43. The robotic kitchen system of claim 42, configured to:

45. 43. The robotic kitchen system of claim 42, wherein the pre-defined fidelity level is within a threshold of an optimal value in achieving the pre-defined outcome, the threshold being specific to the mini-manipulation task and defaulting to 1% of the optimal value if not specified otherwise for each given domain-specific application.

46. the software recipe file includes n stages S1, S2, S3 ... Sj ... Sn, each stage including one or more mini-operations; The overall success probability of the software recipe file is calculated from the joint product of the success probabilities P(Si) of each stage S1, S2, S3 ... Sj ... Sn, and is expressed as follows:

43. The robotic kitchen system of claim 42, wherein an overall probability of success of the software recipe file is at least 0.

99.

47. the plurality of different parameter combinations for each tested mini-manipulation include one or more object data parameters and one or more timing parameters; 43. The robotic kitchen system of claim 42, wherein the one or more object data parameters include one or more kitchen cookware, one or more smart appliances, or one or more ingredients.

48. the software recipe file includes a plurality of stages, each stage including one or more mini-operations; 43. The robotic kitchen system of claim 42, wherein the processor is further configured to execute the plurality of mini-manipulations by executing the stages sequentially, in parallel, or a combination of sequential and parallel execution.

49. 43. The robotic kitchen system of claim 42, wherein the robot includes one or more actuators for repositioning the one or more robot arms or the one or more end effectors within an instrumented environment to extend the reachability of the robot.

50. a robot having one or more robotic arms and one or more robotic end effectors; at least one processor, The at least one processor receiving a software recipe file for cooking at least a portion of a food dish, the software recipe file including one or more cooking operations; retrieving one or more mini-operations from a robotic operation library, where the mini-operations correspond to the one or more cooking operations associated with the software recipe file, where: Each mini-operation corresponds to one or more robotic actions to achieve a functional outcome associated with cooking at least a portion of the food dish, each robotic action defining a movement of a robot; each mini-manipulation having one or more parameters for executing the one or more robotic actions and a post-condition defining a functional state associated with the robotic actions, at least a portion of the food dish, or an environment of the robotic kitchen system; each mini-operation is pre-tested multiple times in a testing environment using multiple parameter combinations to determine a particular parameter combination that achieves the functional outcome within a predefined threshold of the functional outcome based on a comparison of the outcome after execution of the mini-operation to the post-conditions contained in the parameters; performing the mini-operations by operating the robot using the particular combination of parameters of each mini-operation to replicate the one or more cooking actions associated with the software recipe file within a kitchen environment to cook at least a portion of the food dish; A robotic kitchen system configured to:

51. 51. The robotic kitchen system of claim 50, wherein the parameters of each mini-manipulation further include a set of preconditions that define conditions that must be met before the robotic movement in one or more parameters is executed.

52. 51. The robotic kitchen system of claim 50, wherein each mini-manipulation implements a robotic action that replicates one or more movements performed by a human chef to prepare the food dish.

53. 51. The robotic kitchen system of claim 50, wherein the robot includes one or more actuators for repositioning the one or more robot arms or the one or more end effectors within an instrumented environment to extend the reachability of the robot.

54. the software recipe file includes a plurality of stages S1, S2, S3 ... Sj ... Sn, each stage of the plurality of stages including one or more mini-operations; The overall probability of success of the software recipe file is the sum of the success rates of the stages S1, S2, S3 ... Sj ... Sn multiplied together (0.99) n This is expressed by the following formula:

32. The robotic kitchen system of claim 31, where the term P(Si) represents the probability of success of each stage for a plurality of n stages.

Images