Cooking robot, cooking robot control device, control method

Abstract

This technology pertains to a cooking robot, a cooking robot control device, and a control method that allow reproducibility to be improved when using a cooking robot to reproduce a dish made by a cook. A cooking robot in one aspect of this technology controls cooking actions that are performed by a cooking arm by way of using recipe data that includes a data set that links: cooking operation data describing information pertaining to dish ingredients and information pertaining to the actions of a cook in a cooking process using the ingredients; and sensory data indicating the sensations of the cook measured in conjunction with the progress of the cooking process. This technology can be applied to a cooking robot that cooks by operating an arm.

Description

Technical Field

[0001] The present technology relates to a cooking robot, a cooking robot control device, and a control method, and particularly relates to a cooking robot, a cooking robot control device, and a control method that can enhance the reproducibility when reproducing the same dish as the one made by a cook in the cooking robot.

Background Art

[0002] There has been a study on a technology for sensing the movements of a cook during cooking and storing and transmitting data of the sensing results, so as to reproduce the dish made by the cook on the cooking robot side. The cooking operation by the cooking robot is performed, for example, by realizing the same movement as the movement of the cook's hand based on the sensing result.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional cooking method using a cooking robot, even when the cooking process is advanced according to the recipe, it is practically difficult to reproduce the dish as intended by the cook.

[0005] This is because, in addition to the fact that the senses such as taste and smell are different among cooks and people who eat the dish, the types, sizes, textures, origins, etc. of the ingredients are different between the cook side and the reproduction side, the types, capabilities, etc. of the cooking utensils are different, and the cooking environments such as temperature and humidity are different.

[0006] The present technology has been made in view of such a situation, and aims to improve the reproducibility when a cooking robot reproduces the same dish as the one made by a cook.

Means for Solving the Problem

[0007] A cooking robot according to one aspect of the present technology includes a cooking arm that performs a cooking operation for cooking a dish, cooking operation data in which information on ingredients of the dish and information on the actions of a cook in a cooking process using the ingredients are described, and data indicating the sensations of the cook measured in conjunction with the progress of the cooking process, the data being data indicating at least any one of the flavor of the ingredients before cooking, the flavor of the ingredients after cooking in the cooking process, and the flavor of the dish completed through all the cooking processes, a control unit that controls the cooking operation performed by the cooking arm using recipe data including a data set linking the sensation data, and a flavor measurement unit that acquires at least any one of the flavor of the ingredients cooked by the cooking operation performed by the cooking arm and the flavor of the dish completed by the cooking operation performed by the cooking arm. The control unit corrects the recipe data based on the specifications of a first sensor that measures at least any one of the flavor of the ingredients cooked by the cooking operation performed by the cooking arm and the flavor of the dish completed by the cooking operation performed by the cooking arm. Then, using the cooking operation data and the sensory data related to the flavor adjustment performed by the cook after cooking and taste testing, the cooking operation performed by the cooking arm is controlled 。

[0008] In one aspect of the present technology, using recipe data including a data set linking a cooking arm that performs a cooking operation for cooking a dish, cooking operation data in which information on ingredients of the dish and information on the actions of a cook in a cooking process using the ingredients are described, and data indicating the sensations of the cook measured in conjunction with the progress of the cooking process, the data being data indicating at least any one of the flavor of the ingredients before cooking, the flavor of the ingredients after cooking in the cooking process, and the flavor of the dish completed through all the cooking processes, the cooking operation performed by the cooking arm is Controland at least one of the flavor of the food material cooked by the cooking operation performed by the cooking arm and the flavor of the dish completed by the cooking operation performed by the cooking arm is acquired is performed. Also Based on the specifications of the first sensor that measures at least one of the flavor of the food material cooked by the cooking operation performed by the cooking arm and the flavor of the dish completed by the cooking operation performed by the cooking arm, the recipe data is corrected , using the cooking operation data and the sensory data related to the flavor adjustment performed by the cook after cooking and taste testing, the cooking operation performed by the cooking arm is controlled .

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

DETAILED DESCRIPTION OF THE INVENTION

[0010] <SUMMARY OF THE PRESENT TECHNOLOGY> The present technology focuses on the difference (difference) between the feeling when a cook makes a dish and the feeling when cooking based on a recipe created by the cook, and links the feeling data obtained by digitizing the feeling of the cook when making a dish to the data describing the ingredients and cooking process, and manages it as recipe data.

[0011] In addition, based on the feeling of the cook represented by the feeling data, the present technology adjusts the cooking operation of the cooking robot so that the cooking robot can reproduce a dish with the flavor as intended by the cook.

[0012] Furthermore, in addition to the feeling data, the present technology also utilizes the data sensed during the cooking operation at the time of reproduction to adjust the ingredients and cooking operation, thereby realizing flexible cooking according to the characteristics (attributes, states, etc.) of the person who eats the dish.

[0013] Hereinafter, the embodiments for implementing the present technology will be described. The description will be carried out in the following order. 1. Generation of recipe data in the cooking system and reproduction of cooking 2. Regarding recipe data 3. Example of the flow of generating recipe data and reproducing cooking 4. Example of the configuration of the cooking system 5. Operation of the cooking system 6. Variation

[0014] <Generation of recipe data in the cooking system and reproduction of cooking> FIG. 1 is a diagram showing an example of the overall processing in a cooking system according to an embodiment of the present technology.

[0015] As shown in FIG. 1, the cooking system is composed of a chef-side configuration for cooking and a reproduction-side configuration for reproducing the dishes made by the chef.

[0016] The chef-side configuration is, for example, a configuration provided in a certain restaurant, and the reproduction-side configuration is, for example, a configuration provided in an ordinary household. As the reproduction-side configuration, a cooking robot 1 is prepared.

[0017] The cooking system in FIG. 1 is a system that reproduces the same dishes as those made by the chef in the cooking robot 1 as the reproduction-side configuration. The cooking robot 1 has a driving system device such as a cooking arm and various sensors, and is a robot equipped with a function for cooking.

[0018] Recipe data is provided from the chef-side configuration to the reproduction-side configuration including the cooking robot 1 as indicated by the arrow. As will be described in detail later, the recipe data describes information about the dishes made by the chef, including the ingredients of the dishes.

[0019] In the configuration on the reproduction side, the cooking of the dish is reproduced by controlling the cooking operation of the cooking robot 1 based on the recipe data. For example, the cooking is reproduced by causing the cooking robot 1 to perform a cooking operation to achieve the same process as that of the chef's cooking process.

[0020] Although a chef is shown as the person who cooks, regardless of the name such as a sushi chef, a cook, etc., or the role in the kitchen, as long as it is a person who cooks, the cooking system in FIG. 1 is applicable to any person who cooks.

[0021] Also, in FIG. 1, only the configuration on the chef's side of one person is shown, but the cooking system includes configurations on the chef's side provided in a plurality of restaurants, etc. respectively. For the configuration on the reproduction side, for example, recipe data for a predetermined dish made by a predetermined chef selected by the person who eats the dish reproduced by the cooking robot 1 is provided.

[0022] Note that "dish" means the finished product obtained through cooking. "Cooking" means the process of making a dish or the act (operation) of making a dish.

[0023] FIG. 2 is a diagram for explaining the difference in ingredients used on each of the chef's side and the reproduction side.

[0024] For example, when carrots are used in the chef's cooking, the recipe data describes information indicating that carrots are used as an ingredient. Also, information regarding the cooking process using carrots is described.

[0025] Similarly, on the reproduction side, a cooking operation using carrots is performed based on the recipe data.

[0026] Here, even for ingredients classified as the same "carrot", there are differences in taste, aroma, and texture between the carrots prepared by the chef and those prepared on the reproduction side due to differences in type, origin, harvest time, growth conditions, post-harvest environment, etc. There are no completely identical natural ingredients.

[0027] Therefore, even if the cooking robot 1 is made to perform exactly the same cooking operations as the chef, the flavors of the dishes made with carrots will be different. The details of the flavors will be described later.

[0028] To complete one dish, multiple cooking steps are required. Even when looking at the intermediate dishes produced after going through one cooking step using carrots, the flavors will be different between the chef's side and the reproduction side.

[0029] Similarly, due to differences in seasonings used in a certain cooking step, cooking tools such as knives and pans used for cooking, equipment such as heat intensity, etc., the flavors of the completed dishes and intermediate dishes will be different between the chef's side and the reproduction side.

[0030] Therefore, in the cooking system of FIG. 1, the flavor that the chef obtains as a sensation when cooking is measured, for example, every time one cooking step is performed. In the recipe data provided to the reproduction side, the sensory data obtained by the chef by digitizing the flavor is described, for example, linked to the information on ingredients and operations related to one cooking step.

[0031] <Regarding Recipe Data> FIG. 3 is a diagram showing an example of the description content of recipe data.

[0032] As shown in FIG. 3, one piece of recipe data is composed of a plurality of cooking step data sets. In the example of FIG. 3, a cooking step data set related to cooking step #1, a cooking step data set related to cooking step #2, ···, a cooking step data set related to cooking step #N are included.

[0033] Thus, in the recipe data, information regarding one cooking process is described as one cooking process dataset.

[0034] FIG. 4 is a diagram showing an example of information included in the cooking process dataset.

[0035] As shown in the balloon in FIG. 4, the cooking process dataset is composed of cooking operation information, which is information regarding the cooking operations for realizing the cooking process, and flavor information, which is information regarding the flavor of the food materials that have undergone the cooking process.

[0036] 1. Cooking operation information The cooking operation information is composed of food material information and operation information.

[0037] 1-1. Food material information The food material information is information regarding the food materials used by the chef in the cooking process. The information regarding the food materials includes information representing the type of the food materials, the amount of the food materials, the size of the food materials, etc.

[0038] For example, when the chef performs cooking using carrots in a certain cooking process, the information indicating that carrots are used is included in the food material information. Information representing various foods used by the chef as cooking ingredients such as water and seasonings is also included in the food material information. Food refers to various substances that can be eaten by people.

[0039] Note that the food materials include not only the food materials that have not been cooked at all but also the cooked (pre-processed) food materials obtained by performing a certain cooking. The food material information included in the cooking operation information of a certain cooking process includes the information of the food materials that have undergone the previous cooking process.

[0040] The food materials used by the chef are recognized, for example, by analyzing an image obtained by photographing the chef who is performing cooking with a camera. The food material information is generated based on the recognition result of the food materials. The image photographed by the camera may be a moving image or a still image.

[0041] When generating recipe data, ingredient information may be registered by a chef or other person such as a staff member who supports the chef.

[0042] 1-2. Action information Action information is information regarding the movements of a chef in the cooking process. Information regarding the movements of a chef includes information representing the type of cooking tool used by the chef, the movements of the chef's body at each time including hand movements, and the standing position of the chef at each time.

[0043] For example, when a chef cuts a certain ingredient using a kitchen knife, the action information includes information indicating that a kitchen knife was used as the cooking tool, information representing the cutting position, the number of cuts, the cutting force adjustment, the angle, the speed, etc.

[0044] Also, when a chef stirs a pot containing a liquid as an ingredient using a ladle, the action information includes information indicating that a ladle was used as the cooking tool, information representing the stirring force adjustment, the angle, the speed, the time, etc.

[0045] When a chef bakes a certain ingredient using an oven, the action information includes information indicating that an oven was used as the cooking tool, information representing the oven's heat intensity, the baking time, etc.

[0046] When a chef plates the food, the action information includes information on the plating method representing the tableware used for plating, the way of arranging the ingredients, the color and taste of the ingredients, etc.

[0047] The movements of a chef are recognized, for example, by analyzing an image of the chef cooking taken by a camera or by analyzing sensor data measured by a sensor worn by the chef. Action information is generated based on the recognition result of the chef's movements.

[0048] 2. Flavor information As shown in FIG. 4, the flavor information is composed of flavor sensor information and flavor subjective information. Flavor is something obtained as a sensation. The flavor information included in the cooking process dataset corresponds to the sensory data that digitizes the chef's sensations.

[0049] FIG. 5 is a diagram showing an example of the components of flavor.

[0050] As shown in FIG. 5, the deliciousness that a person feels in the brain, that is, "flavor", is mainly composed of the taste obtained by a person's taste buds, the aroma obtained by a person's sense of smell, and the texture obtained by a person's sense of touch.

[0051] Since the way of feeling deliciousness also changes depending on the perceived temperature and the color of the food ingredients, the perceived temperature and color are also included in the flavor.

[0052] The components of flavor (flavor) will be described.

[0053] (1) Taste Taste includes five types of tastes (saltiness, sourness, bitterness, sweetness, umami) that can be felt by taste receptor cells in the tongue and oral cavity. Saltiness, sourness, bitterness, sweetness, and umami are called the basic five tastes.

[0054] In addition to the basic five tastes, taste also includes pungency that can be felt by vanilloid receptors belonging to the TRP (Transient Receptor Potential) channel family, which is not only in the oral cavity but also the pain sensation throughout the body. Depending on the concentration, it becomes a taste overlapping with bitterness, and astringency is also a type of taste.

[0055] Each taste will be described.

[0056] · Saltiness Substances that give a salty taste include minerals (such as Na, K, Fe, Mg, Ca, Cu, Mn, Al, Zn, etc.) that form salts by ionic bonds.

[0057] · Sourness Substances that impart a sour taste include acids such as citric acid and acetic acid. Generally, a sour taste is felt depending on a decrease in pH (for example, around pH 3).

[0058] ·Sweetness Substances that impart a sweet taste include saccharides such as sucrose and glucose, lipids, amino acids such as glycine, and artificial sweeteners.

[0059] ·Umami Substances that impart an umami taste include amino acids such as glutamic acid and aspartic acid, nucleic acid derivatives such as inosinic acid, guanylic acid, and xanthylic acid, organic acids such as succinic acid, and salts.

[0060] ·Bitterness Substances that impart a bitter taste include alkaloids such as caffeine, fumarones such as theobromine, nicotine, catechin, and terpenoid, limonin, cucurbitacin, naringin of flavanone glycoside, bitter amino acids, bitter peptides, bile acids, calcium salts of inorganic salts, and magnesium salts.

[0061] ·Astringency Substances that impart an astringent taste include polyphenols, tannins, catechins, polyvalent ions (Al, Zn, Cr), ethanol, and acetone. Astringency is recognized or measured as part of bitterness.

[0062] ·Spiciness Substances that impart a spicy taste include capsinoids. As a biological function, capsaicin, which is a component of hot chili peppers and various spices that feel hot, and menthol, which is a component of peppermint that feels cold, are recognized not as taste but as pain by the warm receptor of the TRP channel family.

[0063] (2) Aroma Aroma is perceived by volatile low-molecular-weight organic compounds with a molecular weight of 300 or less, which are recognized (bound) by olfactory receptors expressed in the nasal cavity and the nasopharynx.

[0064] (3) Texture Texture is an index called so-called mouthfeel, and is represented by hardness, stickiness, viscosity, cohesiveness, polymer content, moisture content, greasiness, and the like.

[0065] (4) Apparent temperature Apparent temperature is the temperature felt by human skin. Apparent temperature includes not only the temperature of the food itself, but also temperature sensations felt by the surface layer of the skin, such as feeling a sense of coolness from foods containing volatile substances like mint, or feeling a sense of warmth from foods containing pungent components like chili peppers.

[0066] (5) Color The color of food reflects the pigments contained in the food and the components of bitterness and astringency. For example, plant-derived foods contain pigments formed by photosynthesis and components related to the bitterness and astringency of polyphenols. By optical measurement methods, it is possible to estimate the components contained in food from the color of the food.

[0067] 2-1. Flavor sensor information The flavor sensor information that constitutes flavor information is sensor data obtained by measuring the flavor of ingredients with a sensor. Sensor data obtained by measuring the flavor of ingredients that have not been cooked at all with a sensor may be included in the flavor information as flavor sensor information.

[0068] Since flavor is composed of taste, aroma, texture, apparent temperature, and color, the flavor sensor information includes sensor data related to taste, sensor data related to aroma, sensor data related to texture, sensor data related to apparent temperature, and sensor data related to color. All the sensor data may be included in the flavor sensor information, or some of the sensor data may not be included in the flavor sensor information.

[0069] Each sensor data that constitutes flavor sensor information is referred to as taste sensor data, olfactory sensor data, texture sensor data, body temperature sensor data, and color sensor data.

[0070] Taste sensor data is sensor data measured by a taste sensor. The taste sensor data is composed of at least one of the parameters of a saltiness sensor value, sourness sensor value, bitterness sensor value, sweetness sensor value, umami sensor value, spiciness sensor value, and astringency sensor value.

[0071] Examples of taste sensors include artificial lipid membrane type taste sensors that use an artificial lipid membrane in the sensor part. The artificial lipid membrane type taste sensor is a sensor that detects a change in membrane potential caused by an electrostatic interaction or hydrophobic interaction of a lipid membrane with a flavor substance, which is a substance that causes a taste sensation, and outputs it as a sensor value.

[0072] As long as it is a device that can digitize and output each element of saltiness, sourness, bitterness, sweetness, umami, spiciness, and astringency that make up the taste of food, instead of an artificial lipid membrane type taste sensor, various devices such as taste sensors using a polymer membrane can be used as taste sensors.

[0073] Olfactory sensor data is sensor data measured by an olfactory sensor. The olfactory sensor data is composed of values for each element expressing scents such as a spicy scent, a fruity scent, a green smell, a moldy smell (cheesy), a citrus scent, and a rose scent.

[0074] Examples of olfactory sensors include sensors with innumerable sensors such as crystal oscillators. A crystal oscillator is used instead of a receptor in a human nose. The olfactory sensor using a crystal oscillator detects a change in the vibration frequency of the crystal oscillator when a scent component hits the crystal oscillator, and outputs a value expressing the scent described above based on the pattern of the change in the vibration frequency.

[0075] Instead of a sensor using a crystal oscillator, any device that can output a value representing a scent can use various sensors made of various materials such as carbon as an olfactory sensor instead of the receptors in the human nose.

[0076] Texture sensor data is sensor data identified by analyzing an image captured by a camera or sensor data measured by various sensors. The texture sensor data is composed of at least one parameter among information representing hardness (rigidity), stickiness, viscosity (stress), cohesiveness, polymer content, moisture content, oil content, etc.

[0077] Hardness, stickiness, viscosity, and cohesiveness are recognized, for example, by analyzing an image of the food being cooked by a chef taken with a camera. For example, by analyzing an image of the soup being stirred by a chef, it becomes possible to recognize values such as hardness, stickiness, viscosity, and cohesiveness. These values may also be recognized by measuring the stress when the chef cuts the food with a knife.

[0078] Polymer content, moisture content, and oil content are measured by a sensor that measures those values, for example, by irradiating the food with light of a predetermined wavelength and analyzing the reflected light.

[0079] A database associating each food with each parameter of texture may be prepared, and the texture sensor data of each food may be recognized by referring to the database.

[0080] Somatic sensation temperature sensor data is sensor data obtained by measuring the temperature of the food with a temperature sensor.

[0081] Color sensor data is data identified by analyzing the color of the food from an image captured by a camera.

[0082] 2-2. Flavor subjective information Flavor subjective information is information representing the subjective perception of flavor by a person such as a chef who is cooking. The flavor subjective information is calculated based on flavor sensor information.

[0083] Since flavor is composed of taste, aroma, texture, perceived temperature, and color, the flavor subjective information includes subjective information related to taste, subjective information related to aroma, subjective information related to texture, subjective information related to perceived temperature, and subjective information related to color. All of the subjective information related to taste, subjective information related to aroma, subjective information related to texture, subjective information related to perceived temperature, and subjective information related to color may be included in the flavor subjective information, or any of the subjective information may not be included in the flavor subjective information.

[0084] Each of the subjective information constituting the flavor subjective information is referred to as taste subjective information, olfactory subjective information, texture subjective information, perceived temperature subjective information, and color subjective information.

[0085] FIG. 6 is a diagram showing an example of calculating taste subjective information.

[0086] As shown in FIG. 6, the taste subjective information is calculated using a taste subjective information generation model, which is a neural network model generated by deep learning or the like. The taste subjective information generation model is pre-generated, for example, by performing learning using taste sensor data of a certain food ingredient and information (numerical values) representing the way a chef who ate the food ingredient perceives the taste.

[0087] For example, as shown in FIG. 6, when each of the saltiness sensor value, sourness sensor value, bitterness sensor value, sweetness sensor value, umami sensor value, spiciness sensor value, and astringency sensor value, which are taste sensor data of a certain food ingredient, is input, the taste subjective information generation model outputs each of the saltiness subjective value, sourness subjective value, bitterness subjective value, sweetness subjective value, umami subjective value, spiciness subjective value, and astringency subjective value.

[0088] The saltiness subjective value is a value representing how a chef perceives saltiness. The sourness subjective value is a value representing how a chef perceives sourness. Similarly, the bitterness subjective value, sweetness subjective value, umami subjective value, spiciness subjective value, and astringency subjective value are values representing how a chef perceives bitterness, sweetness, umami, spiciness, and astringency, respectively.

[0089] As shown in Figure 7, the taste subjective information of a certain ingredient is represented as a chart by the respective values of the saltiness subjective value, sourness subjective value, bitterness subjective value, sweetness subjective value, umami subjective value, spiciness subjective value, and astringency subjective value. Ingredients with similar shapes of the taste subjective information charts are ingredients with similar tastes for the chef when only paying attention to the taste among the flavors.

[0090] Similarly, the other subjective information constituting the flavor subjective information is calculated using the respective models for generating subjective information.

[0091] That is, the olfactory subjective information is calculated by inputting olfactory sensor data into the olfactory subjective information generation model, the texture subjective information is calculated by inputting texture sensor data into the texture subjective information generation model. The somatosensory temperature subjective information is calculated by inputting somatosensory temperature subjective sensor data into the somatosensory temperature subjective information generation model, and the color subjective information is calculated by inputting color sensor data into the color subjective information generation model.

[0092] Rather than using a neural network model, the taste subjective information may be calculated based on table information associating the taste sensor data of a certain ingredient with the information representing how the chef who ate the ingredient perceives the taste. Various methods can be adopted for the method of calculating the flavor subjective information using the flavor sensor information.

[0093] As described above, the recipe data is constituted by linking (associating) the cooking operation information, which is information about the cooking operations for realizing the cooking process, with the flavor information, which is information about the flavors of the ingredients and dishes measured in conjunction with the progress of the cooking process.

[0094] Recipe data containing the above information is prepared for each dish as shown in FIG. 8. Which recipe data to use for reproducing the dish is selected, for example, by the person at the location where the cooking robot 1 is installed.

[0095] <Example of the flow of recipe data generation and dish reproduction> FIG. 9 is a diagram showing an example of the flow of recipe data generation.

[0096] As shown in FIG. 9, usually, cooking by a chef is performed by repeating, for each cooking step, cooking using ingredients, tasting the cooked ingredients, and adjusting the flavor.

[0097] For flavor adjustment, for example, regarding taste, when the saltiness is insufficient, salt is added, and when the sourness is insufficient, lemon juice is squeezed. Regarding aroma, for example, herbs are minced and added, or the ingredients are passed over fire. Regarding texture, for example, when the ingredients are hard, they are pounded to make them soft, or the cooking time is increased.

[0098] The cooking operation information constituting the cooking process dataset is generated based on the sensing of the actions of the chef who cooks using ingredients and the actions of the chef who adjusts the flavor, based on the sensing results.

[0099] Also, the flavor information is generated based on the sensing of the flavor of the cooked ingredients, based on the sensing results.

[0100] In the example of FIG. 9, as shown by arrows A1 and A2, based on the sensing results of the cooking actions performed by the chef as cooking step #1 and the actions of the chef who adjusts the flavor, the cooking operation information constituting the cooking process dataset for cooking step #1 is generated.

[0101] Also, as shown by arrow A3, flavor information that constitutes the cooking process dataset of cooking process #1 is generated based on the sensing result of the flavor of the food material after cooking by cooking process #1.

[0102] After cooking process #1 is completed, cooking process #2, which is the next cooking process, is performed.

[0103] Similarly, as shown by arrows A11 and A12, cooking operation information that constitutes the cooking process dataset of cooking process #2 is generated based on the sensing result of the cooking operation of the chef as cooking process #2 and the operation of the chef to adjust the flavor.

[0104] Also, as shown by arrow A13, flavor information that constitutes the cooking process dataset of cooking process #2 is generated based on the sensing result of the flavor of the food material after cooking by cooking process #2.

[0105] One dish is completed through such a plurality of cooking processes. Also, when the dish is completed, recipe data describing the cooking process dataset of each cooking process is generated.

[0106] Hereinafter, mainly, the case where one cooking process is composed of three cooking operations of cooking, tasting, and adjustment will be described, but the unit of the cooking operation included in one cooking process can be arbitrarily set. One cooking process may be composed of cooking operations without tasting or adjusting the flavor after tasting, or may be composed only of adjusting the flavor. Similarly in this case, flavor sensing is performed for each cooking process, and the flavor information obtained based on the sensing result is included in the cooking process dataset.

[0107] Flavor sensing is not performed every time one cooking process ends, and the timing of flavor sensing can also be arbitrarily set. For example, flavor sensing may be repeatedly performed during one cooking process. In this case, the cooking process dataset will include time-series data of flavor information.

[0108] Rather than all cooking process datasets containing flavor information, each time flavor measurement is performed at an arbitrary timing, the flavor information may be included in the cooking process dataset together with the information on the cooking operation being performed at that timing.

[0109] FIG. 10 is a diagram showing an example of the flow of reproducing a dish based on recipe data.

[0110] As shown in FIG. 10, the reproduction of the dish by the cooking robot 1 is performed by repeating, for each cooking process, cooking based on the cooking operation information included in the cooking process dataset described in the recipe data, measuring the flavor of the cooked ingredients, and adjusting the flavor.

[0111] The flavor adjustment is performed, for example, by adding operations so that the flavor measured by a sensor prepared on the cooking robot 1 side approaches the flavor represented by the flavor information. Details of the flavor adjustment by the cooking robot 1 will be described later.

[0112] The measurement and adjustment of the flavor may be repeated a plurality of times, for example, in one cooking process. That is, each time an adjustment is made, the flavor of the adjusted ingredients is measured, and the flavor is adjusted based on the measurement result.

[0113] In the example of FIG. 10, as shown by arrow A21, the cooking operation of the cooking robot 1 is controlled based on the cooking operation information constituting the cooking process dataset of cooking process #1, and the same operation as the operation of the chef's cooking process #1 is performed by the cooking robot 1.

[0114] After the same operation as the operation of the chef's cooking process #1 is performed by the cooking robot 1, the flavor of the cooked ingredients is measured, and as shown by arrow A22, the flavor adjustment of the cooking robot 1 is controlled based on the flavor information constituting the cooking process dataset of cooking process #1.

[0115] When the flavor measured by the sensor prepared on the side of the cooking robot 1 matches the flavor represented by the flavor information, the adjustment of the flavor is completed and the cooking process #1 also ends. For example, not only when they completely match, but also when the flavor measured by the sensor prepared on the side of the cooking robot 1 and the flavor represented by the flavor information are similar to or above the threshold value, they are determined to match.

[0116] After the cooking process #1 ends, the cooking process #2, which is the next cooking process, is performed.

[0117] Similarly, as shown by the arrow A31, based on the cooking operation information that constitutes the cooking process dataset of the cooking process #2, the cooking operation of the cooking robot 1 is controlled, and the same operation as the operation of the chef's cooking process #2 is performed by the cooking robot 1.

[0118] After the cooking robot 1 performs the same operation as the operation of the chef's cooking process #2, the flavor of the cooked food is measured, and as shown by the arrow A32, based on the flavor information that constitutes the cooking process dataset of the cooking process #2, the adjustment of the flavor of the cooking robot 1 is controlled.

[0119] When the flavor measured by the sensor prepared on the side of the cooking robot 1 matches the flavor represented by the flavor information, the adjustment of the flavor is completed and the cooking process #2 also ends.

[0120] Through such a plurality of cooking processes, the dish made by the chef is reproduced by the cooking robot 1.

[0121] FIG. 11 is a diagram showing the flow on the chef's side and the flow on the reproduction side together.

[0122] As shown on the left side of FIG. 11, when one dish is completed through a plurality of cooking processes #1 to #N, recipe data describing the cooking process datasets of each cooking process is generated.

[0123] On the other hand, on the reproduction side, based on the recipe data generated by the chef's cooking, through a plurality of cooking steps #1 to #N, which are the same as the cooking steps performed on the chef's side, one dish is reproduced.

[0124] Since the cooking by the cooking robot 1 is performed so as to adjust the flavor for each cooking step, the finally completed dish will be the same as the dish made by the chef or a dish with a similar flavor. In this way, a dish with the same flavor as the dish made by the chef is reproduced based on the recipe data in a highly reproducible manner.

[0125] The chef can provide a dish with the same flavor as the dish he / she made to people who, for example, cannot visit the restaurant he / she operates. Also, the chef can leave the dish he / she makes in a form that can be reproduced as recipe data.

[0126] On the other hand, a person who eats the dish reproduced by the cooking robot 1 can eat a dish with the same flavor as the dish made by the chef.

[0127] Figure 12 is a diagram showing an example of other description contents of the recipe data.

[0128] As shown in Figure 12, flavor information regarding the flavor of the completed dish may be included in the recipe data. In this case, the flavor information regarding the flavor of the completed dish is linked to the overall cooking operation information.

[0129] In this way, the association relationship between the cooking operation information and the flavor information does not have to be one-to-one.

[0130] <Configuration example of the cooking system> (1) Overall configuration Figure 13 is a diagram showing a configuration example of a cooking system according to an embodiment of the present technology.

[0131] As shown in Fig. 13, the cooking system is configured by connecting a data processing device 11 provided as a chef-side configuration and a control device 12 provided as a reproduction-side configuration via a network 13 such as the Internet. As described above, a plurality of such chef-side configurations and reproduction-side configurations are provided in the cooking system.

[0132] The data processing device 11 is a device that generates the above-described recipe data. The data processing device 11 is composed of a computer or the like. The data processing device 11 transmits, for example, the recipe data of the dish selected by the person who eats the reproduced dish to the control device 12 via the network 13.

[0133] The control device 12 is a device that controls the cooking robot 1. The control device 12 is also composed of a computer or the like. The control device 12 receives the recipe data provided from the data processing device 11 and controls the cooking operation of the cooking robot 1 by outputting command commands based on the description of the recipe data.

[0134] The cooking robot 1 drives each part such as a cooking arm according to the command commands supplied from the control device 12 and performs the cooking operations of each cooking process. The command commands include information for controlling the torque, driving direction, and driving amount of the motor provided on the cooking arm.

[0135] Until the dish is completed, command commands are sequentially output from the control device 12 to the cooking robot 1. By the cooking robot 1 performing operations according to the command commands, finally, the dish is completed.

[0136] Fig. 14 is a diagram showing another configuration example of the cooking system.

[0137] As shown in Fig. 14, the provision of recipe data from the chef side to the reproduction side may be performed via a server on the network.

[0138] The recipe data management server 21 shown in FIG. 14 receives the recipe data transmitted from each data processing device 11 and manages it by storing it in a database or the like. The recipe data management server 21 transmits predetermined recipe data to the control device 12 in response to a request from the control device 12 made via the network 13.

[0139] The recipe data management server 21 has a function of centrally managing the recipe data of the dishes made by the chefs of various restaurants and distributing the recipe data in response to requests from the reproduction side.

[0140] FIG. 15 is a diagram showing an arrangement example of the control device 12.

[0141] As shown in A of FIG. 15, the control device 12 is provided, for example, as a device outside the cooking robot 1. In the example of A in FIG. 15, the control device 12 and the cooking robot 1 are connected via the network 13.

[0142] The command commands transmitted from the control device 12 are received by the cooking robot 1 via the network 13. Various data such as images taken by the camera of the cooking robot 1 and sensor data measured by sensors provided in the cooking robot 1 are transmitted from the cooking robot 1 to the control device 12 via the network 13.

[0143] Rather than connecting one cooking robot 1 to one control device 12, a plurality of cooking robots 1 may be connected to one control device 12.

[0144] As shown in B of FIG. 15, the control device 12 may be provided inside the housing of the cooking robot 1. In this case, the operations of each part of the cooking robot 1 are controlled according to the command commands generated by the control device 12.

[0145] Hereinafter, mainly, the case where the control device 12 is provided as a device outside the cooking robot 1 will be described.

[0146] (2) Chef-side configuration (2-1) Configuration around the kitchen FIG. 16 is a diagram showing an example of the configuration around the kitchen where the chef cooks.

[0147] Around the kitchen 31 where the chef cooks, various devices for measuring information used for analyzing the chef's actions and the flavor of the ingredients are provided. Some of those devices are attached to the chef's body.

[0148] The devices provided around the kitchen 31 are each connected to the data processing device 11 via wired or wireless communication. Each device provided around the kitchen 31 may be connected to the data processing device 11 via a network.

[0149] As shown in FIG. 16, cameras 41-1 and 41-2 are provided above the kitchen 31. The cameras 41-1 and 41-2 photograph the state of the chef who is cooking and the state on the top plate of the kitchen 31, and transmit the images obtained by the photographing to the data processing device 11.

[0150] A small camera 41-3 is attached to the head of the chef. The shooting range of the camera 41-3 is switched according to the direction of the chef's line of sight. The camera 41-3 photographs the state of the chef's hands while cooking, the state of the ingredients to be cooked, and the state on the top plate of the kitchen 31, and transmits the images obtained by the photographing to the data processing device 11.

[0151] In this way, a plurality of cameras are provided around the kitchen 31. When there is no need to distinguish between the cameras 41-1 to 41-3, they are collectively referred to as the camera 41 as appropriate.

[0152] An olfactory sensor 42 is attached to the upper body of the chef. The olfactory sensor 42 measures the smell of the ingredients and transmits the olfactory sensor data to the data processing device 11.

[0153] A taste sensor 43 is provided on the top plate of the kitchen 31. The taste sensor 43 measures the taste of the food ingredients and transmits taste sensor data to the data processing device 11.

[0154] As shown in FIG. 17, the taste sensor 43 is used by bringing the sensor unit 43A provided at the tip of the cable into contact with food ingredients to be cooked. When the taste sensor 43 is the above-described artificial lipid membrane type taste sensor, the lipid membrane is provided on the sensor unit 43A.

[0155] Not only taste sensor data but also texture sensor data and body sensation temperature sensor data among the sensor data constituting the flavor sensor information may be measured by the taste sensor 43 and transmitted to the data processing device 11. In this case, the taste sensor 43 is provided with functions as a texture sensor and a body sensation temperature sensor. For example, texture sensor data such as polymer content, moisture content, and oil content is measured by the taste sensor 43.

[0156] Various devices other than the devices shown in FIG. 16 are provided around the kitchen 31.

[0157] FIG. 18 is a block diagram showing a configuration example on the chef side.

[0158] In the configuration shown in FIG. 18, the same components as the above-described configuration are denoted by the same reference numerals. Redundant descriptions will be omitted as appropriate.

[0159] As shown in FIG. 18, a camera 41, an olfactory sensor 42, a taste sensor 43, an infrared sensor 51, a texture sensor 52, and an environment sensor 53 are connected to the data processing device 11. The same components as the above-described configuration are denoted by the same reference numerals. Redundant descriptions will be omitted as appropriate.

[0160] The infrared sensor 51 outputs IR light and generates an IR image. The IR image generated by the infrared sensor 51 is output to the data processing device 11. Various analyses such as the actions of the chef and the ingredients may be performed based on the IR image captured by the infrared sensor 51, rather than the image (RGB image) captured by the camera 41.

[0161] The texture sensor 52 is composed of various sensors that output sensor data used for texture analysis, such as a hardness sensor, a stress sensor, a moisture content sensor, and a temperature sensor. The hardness sensor, stress sensor, moisture content sensor, and temperature sensor may be provided on cooking tools such as knives, frying pans, and ovens. The sensor data measured by the texture sensor 52 is output to the data processing device 11.

[0162] The environment sensor 53 is a sensor that measures the cooking environment, which is the environment of a space such as a kitchen where the chef cooks. In the example of FIG. 18, the environment sensor 53 is composed of a camera 61, a temperature / humidity sensor 62, and an illuminance sensor 63.

[0163] The camera 61 outputs an image of the cooking space to the data processing device 11. By analyzing the image of the cooking space, for example, the color (brightness, hue, saturation) of the cooking space is measured.

[0164] The temperature / humidity sensor 62 measures the temperature and humidity of the space on the chef's side and outputs information representing the measurement results to the data processing device 11.

[0165] The illuminance sensor 63 measures the brightness of the space on the chef's side and outputs information representing the measurement results to the data processing device 11.

[0166] The color, temperature, and brightness of the space where the food is eaten affect the way people perceive the flavor. For example, when considering the seasoning of the same dish, the lighter the taste is preferred as the temperature is higher, and the stronger the taste is preferred as the temperature is lower.

[0167] Regarding the cooking environment that may affect the way people perceive the flavor, it may be measured during cooking and included in the recipe data as environmental information.

[0168] On the reproduction side, the environment such as the color, temperature, and brightness of the room where the person eating the dish is located is adjusted to be the same as the cooking environment represented by the environmental information included in the recipe data.

[0169] This makes it possible to make the way of perceiving the flavor when eating the reproduced dish closer to the way the chef felt during cooking.

[0170] Various information that may affect the way of perceiving the flavor, such as the air pressure and noise in the chef's space, the season and time zone during cooking, may be measured by the environmental sensor 53 and included in the recipe data as environmental information.

[0171] (2-2) Configuration of the data processing device 11 FIG. 19 is a block diagram showing a configuration example of the hardware of the data processing device 11.

[0172] As shown in FIG. 19, the data processing device 11 is composed of a computer. The CPU (Central Processing Unit) 201, ROM (Read Only Memory) 202, and RAM (Random Access Memory) 203 are interconnected by a bus 204.

[0173] An input / output interface 205 is further connected to the bus 204. An input unit 206 composed of a keyboard, a mouse, etc., and an output unit 207 composed of a display, a speaker, etc. are connected to the input / output interface 205.

[0174] In addition, a storage unit 208 composed of a hard disk, a non-volatile memory, etc., a communication unit 209 composed of a network interface, etc., and a drive 210 for driving a removable medium 211 are connected to the input / output interface 205.

[0175] In the computer configured as described above, various processes are performed by the CPU 201 loading and executing, via the input / output interface 205 and the bus 204, a program stored in the storage unit 208 into the RAM 203, for example.

[0176] FIG. 20 is a block diagram showing a functional configuration example of the data processing apparatus 11.

[0177] At least a part of the functional units shown in FIG. 20 is realized by executing a predetermined program by the CPU 201 of FIG. 19.

[0178] As shown in FIG. 20, a data processing unit 221 is realized in the data processing apparatus 11. The data processing unit 221 is composed of a cooking operation information generation unit 231, a flavor information generation unit 232, a recipe data generation unit 233, an environment information generation unit 234, an attribute information generation unit 235, and a recipe data output unit 236.

[0179] The cooking operation information generation unit 231 is composed of a food ingredient recognition unit 251, a tool recognition unit 252, and an action recognition unit 253.

[0180] The food ingredient recognition unit 251 analyzes an image captured by the camera 41 and recognizes the types of food ingredients used by the chef for cooking. Recognition information for recognizing various types of food ingredients, such as feature information, is provided to the food ingredient recognition unit 251.

[0181] The tool recognition unit 252 analyzes an image captured by the camera 41 and recognizes the types of cooking tools used by the chef for cooking. Recognition information for recognizing various types of cooking tools is provided to the tool recognition unit 252.

[0182] The action recognition unit 253 analyzes an image captured by the camera 41, sensor data representing the measurement results of sensors attached to the chef's body, etc., and recognizes the actions of the chef performing the cooking.

[0183] The information representing the recognition results by each part of the cooking operation information generation unit 231 is supplied to the recipe data generation unit 233.

[0184] The flavor information generation unit 232 is composed of a taste measurement unit 261, a smell measurement unit 262, a texture measurement unit 263, a perceived temperature measurement unit 264, a color measurement unit 265, and a subjective information generation unit 266.

[0185] The taste measurement unit 261 measures the taste of the food material by controlling the taste sensor 43 and obtains taste sensor data. The food materials to be measured include all foods handled by the chef, such as the food materials before cooking, the food materials after cooking, and the completed dishes.

[0186] The smell measurement unit 262 measures the smell of the food material by controlling the smell sensor 42 and obtains the smell sensor data of the food material.

[0187] The texture measurement unit 263 measures the texture of the food material by analyzing the image taken by the camera 41 or the measurement result by the texture sensor 52 and obtains the texture sensor data of the food material.

[0188] The perceived temperature measurement unit 264 obtains the perceived temperature sensor data representing the perceived temperature of the food material measured by the temperature sensor.

[0189] The color measurement unit 265 recognizes the color of the food material by analyzing the image taken by the camera 41 and obtains the color sensor data representing the recognition result. When the object of color recognition is the dish completed by plating the food material, the colors of each part in the whole dish are recognized.

[0190] The subjective information generation unit 266 generates subjective information based on the sensor data obtained by each of the taste measurement unit 261 to the color measurement unit 265. The process of converting the objective data related to the flavor represented by the sensor data into the subjective data representing the chef's feeling of the flavor is performed in the subjective information generation unit 266.

[0191] The subjective information generation unit 266 is provided with information used for generating subjective information, such as a neural network described with reference to FIG. 6.

[0192] For example, the subjective information generation unit 266 inputs the taste sensor data acquired by the taste measurement unit 261 into a taste subjective information generation model, and generates taste subjective information of the food material.

[0193] Similarly, the subjective information generation unit 266 inputs the olfactory sensor data acquired by the aroma measurement unit 262 into an olfactory subjective information generation model, and generates olfactory subjective information of the food material. The subjective information generation unit 266 inputs the texture sensor data acquired by the texture measurement unit 263 into a texture subjective information generation model, and generates texture subjective information of the food material.

[0194] The subjective information generation unit 266 inputs the somatosensory temperature sensor data acquired by the somatosensory temperature measurement unit 264 into a somatosensory temperature subjective information generation model, and generates somatosensory temperature subjective information of the food material. The subjective information generation unit 266 inputs the color sensor data acquired by the color measurement unit 265 into a color subjective information generation model, and generates color subjective information of the food material.

[0195] The sensor data acquired by each of the taste measurement unit 261 to the color measurement unit 265 and the respective subjective information generated by the subjective information generation unit 266 are supplied to the recipe data generation unit 233.

[0196] The recipe data generation unit 233 generates cooking operation information based on the information supplied from each part of the cooking operation information generation unit 231. That is, the recipe data generation unit 233 generates food material information based on the recognition result by the food material recognition unit 251, and generates operation information based on the recognition results by the tool recognition unit 252 and the operation recognition unit 253. The recipe data generation unit 233 generates cooking operation information including the food material information and the operation information.

[0197] In addition, the recipe data generation unit 233 generates flavor information based on the information supplied from each part of the flavor information generation unit 232. That is, the recipe data generation unit 233 generates flavor sensor information based on the sensor data acquired by the taste measurement unit 261 to the color measurement unit 265, and generates flavor subjective information based on the subjective information generated by the subjective information generation unit 266. The recipe data generation unit 233 generates flavor information including the flavor sensor information and the flavor subjective information.

[0198] The recipe data generation unit 233 generates a cooking process data set by associating the cooking operation information and the flavor information, for example, for each cooking process of the chef. The recipe data generation unit 233 generates recipe data describing a plurality of cooking process data sets by summarizing the cooking process data sets for each cooking process from the first cooking process to the last cooking process of a certain dish.

[0199] The recipe data generation unit 233 outputs the recipe data generated in this way to the recipe data output unit 236. The recipe data output by the recipe data generation unit 233 appropriately includes the environment information generated by the environment information generation unit 234 and the attribute information generated by the attribute information generation unit 235.

[0200] The environment information generation unit 234 generates environment information representing the cooking environment based on the measurement results by the environment sensor 53. The environment information generated by the environment information generation unit 234 is output to the recipe data generation unit 233.

[0201] The attribute information generation unit 235 generates attribute information representing the attributes of the chef. The attributes of the chef include, for example, the chef's age, gender, nationality, and living area. Information representing the chef's physical condition and the like may be included in the attribute information.

[0202] The chef's age, gender, nationality, and living area affect the way of feeling the flavor. That is, it is considered that the flavor subjective information included in the recipe data is affected by the chef's age, gender, nationality, living area, etc.

[0203] On the reproduction side, when performing processing using the flavor subjective information included in the recipe data, correction of the flavor subjective information is appropriately performed according to the difference between the attributes of the chef represented by the attribute information and the attributes of the person eating the reproduced dish, and processing is performed using the corrected flavor subjective information.

[0204] For example, assume that the chef is French and the person eating the reproduced dish is Japanese. In this case, the way the chef feels the flavor represented by the flavor subjective information included in the recipe data is the French way of feeling, which is different from the Japanese way of feeling.

[0205] The flavor subjective information included in the recipe data is corrected based on information representing the Japanese way of feeling corresponding to the French way of feeling so that the Japanese can obtain the same way of feeling the flavor even when eating. The information used for correcting the flavor subjective information is information associating the French way of feeling and the Japanese way of feeling for each flavor, and is, for example, generated statistically and prepared in advance on the reproduction side.

[0206] Attributes such as the category of the dish made by the chef, such as French cuisine, Japanese cuisine, Italian cuisine, Spanish cuisine, etc., may be included in the attribute information.

[0207] Also, the attributes of the ingredients and seasonings used in cooking may be included in the attribute information. The attributes of the ingredients include the place of origin, variety, etc. The attributes of the seasonings also include the place of origin, variety, etc.

[0208] In this way, the cook attribute information that is attribute information representing the attributes of the chef, the food attribute information that is attribute information representing the attributes of the dish and the ingredients, and the seasoning attribute information that is attribute information representing the attributes of the seasonings among the ingredients may be included in the recipe data.

[0209] The recipe data output unit 236 controls the communication unit 209 (FIG. 19) and outputs the recipe data generated by the recipe data generation unit 233. The recipe data output from the recipe data output unit 236 is supplied to the control device 12 or the recipe data management server 21 via the network 13.

[0210] (3) Configuration on the reproduction side (3-1) Configuration of the cooking robot 1 · Appearance of the cooking robot 1 FIG. 21 is a perspective view showing the appearance of the cooking robot 1.

[0211] As shown in FIG. 21, the cooking robot 1 is a kitchen-type robot having a horizontally long rectangular parallelepiped housing 311. Various configurations are provided inside the housing 311 that forms the main body of the cooking robot 1.

[0212] On the back side of the housing 311, a cooking assistance system 312 is provided standing upright from the upper surface of the housing 311. Each space formed in the cooking assistance system 312 by partitioning with a thin plate-like member has functions for assisting cooking by the cooking arms 321-1 to 321-4, such as a refrigerator, an oven range, and storage.

[0213] Rails are provided in the longitudinal direction on the top plate 311A, and the cooking arms 321-1 to 321-4 are provided on the rails. The cooking arms 321-1 to 321-4 can change their positions along the rails as a moving mechanism.

[0214] The cooking arms 321-1 to 321-4 are robot arms configured by connecting cylindrical members at joint parts. Various operations related to cooking are performed by the cooking arms 321-1 to 321-4.

[0215] The space above the top plate 311A becomes a cooking space where the cooking arms 321-1 to 321-4 perform cooking.

[0216] In Fig. 21, four cooking arms are shown, but the number of cooking arms is not limited to four. Hereinafter, when it is not necessary to distinguish each of the cooking arms 321-1 to 321-4 as appropriate, they are collectively referred to as the cooking arm 321.

[0217] Fig. 22 is a diagram showing an enlarged view of the state of the cooking arm 321.

[0218] As shown in Fig. 22, attachments having various cooking functions are attached to the tips of the cooking arms 321. As attachments for the cooking arm 321, various attachments such as an attachment having a manipulator function (hand function) for grasping foodstuffs and tableware, and an attachment having a knife function for cutting foodstuffs are prepared.

[0219] In the example of Fig. 22, a knife attachment 331-1, which is an attachment having a knife function, is attached to the cooking arm 321-1. Using the knife attachment 331-1, a lump of meat placed on the top plate 311A is being cut.

[0220] An attachment spindle attachment 331-2, which is used to fix or rotate foodstuffs, is attached to the cooking arm 321-2.

[0221] A peeler attachment 331-3, which is an attachment having a function of peeling the skin of foodstuffs, is attached to the cooking arm 321-3.

[0222] The skin of a potato lifted by the cooking arm 321-2 using the spindle attachment 331-2 is being peeled by the cooking arm 321-3 using the peeler attachment 331-3. Thus, it is also possible for a plurality of cooking arms 321 to cooperate to perform one operation.

[0223] The cooking arm 321-4 has a manipulator attachment 331-4, which is an attachment with manipulator functions, attached thereto. Using the manipulator attachment 331-4, a frying pan with chicken on it is carried into the space of the cooking assistance system 312 having an oven function.

[0224] Cooking by such a cooking arm 321 can be advanced by appropriately replacing the attachment according to the content of the work. The replacement of the attachment is automatically performed by, for example, the cooking robot 1.

[0225] It is also possible to attach the same attachment to a plurality of cooking arms 321, such as attaching the manipulator attachment 331-4 to each of the four cooking arms 321.

[0226] Cooking by the cooking robot 1 is performed not only using the above-described attachments prepared as tools for the cooking arm, but also appropriately using the same tools as those used by humans for cooking. For example, a knife used by a human is grasped by the manipulator attachment 331-4, and cooking such as cutting ingredients is performed using the knife.

[0227] · Configuration of the cooking arm FIG. 23 is a diagram showing the appearance of the cooking arm 321.

[0228] As shown in FIG. 23, the cooking arm 321 is generally configured by connecting thin cylindrical members with hinge portions serving as joint portions. Each hinge portion is provided with a motor or the like that generates a force for driving each member.

[0229] As the cylindrical members, a detachable member 351, a relay member 353, and a base member 355 are provided in order from the tip. The detachable member 351 is a member having a length of about 1 / 5 of the length of the relay member 353. The combined length of the length of the detachable member 351 and the length of the relay member 353 is substantially the same as the length of the base member 355.

[0230] The detachable member 351 and the relay member 353 are connected by a hinge portion 352, and the relay member 353 and the base member 355 are connected by a hinge portion 354. Hinge portions 352 and 354 are provided at both ends of the relay member 353.

[0231] In this example, the cooking arm 321 is constituted by three cylindrical members, but it may be constituted by four or more cylindrical members. In this case, a plurality of relay members 353 are provided.

[0232] At the tip of the detachable member 351, a detachable portion 351A to which an attachment is detachably attached is provided. The detachable member 351 has a detachable portion 351A to which various attachments are detachably attached, and functions as a cooking function arm portion that performs cooking by operating the attachment.

[0233] At the rear end of the base member 355, a detachable portion 356 attached to a rail is provided. The base member 355 functions as a moving function arm portion that realizes the movement of the cooking arm 321.

[0234] FIG. 24 is a diagram showing an example of the movable range of each part of the cooking arm 321.

[0235] As shown by being surrounded by ellipse #1, the detachable member 351 is rotatable about the central axis of the circular cross section. The flat small circle shown at the center of ellipse #1 indicates the direction of the rotation axis of the dashed line.

[0236] As shown by being surrounded by circle #2, the detachable member 351 is rotatable about an axis passing through the fitting portion 351B with the hinge portion 352. Further, the relay member 353 is rotatable about an axis passing through the fitting portion 353A with the hinge portion 352.

[0237] The two small circles shown inside circle #2 indicate the directions of their respective rotation axes (perpendicular to the plane of the paper). The range of movement of the detachable member 351 centered on the axis passing through the fitting portion 351B and the range of movement of the relay member 353 centered on the axis passing through the fitting portion 353A are, for example, within a range of 90 degrees each.

[0238] The relay member 353 is configured to be separated into a member 353-1 on the tip side and a member 353-2 on the rear end side. As shown surrounded by ellipse #3, the relay member 353 is rotatable about the central axis of the circular cross-section at the connection portion 353B between the member 353-1 and the member 353-2.

[0239] Other movable parts basically have a similar range of movement.

[0240] That is, as shown surrounded by circle #4, the relay member 353 is rotatable about the axis passing through the fitting portion 353C with the hinge portion 354. Also, the base member 355 is rotatable about the axis passing through the fitting portion 355A with the hinge portion 354.

[0241] The base member 355 is configured to be separated into a member 355-1 on the tip side and a member 355-2 on the rear end side. As shown surrounded by ellipse #5, the base member 355 is rotatable about the central axis of the circular cross-section at the connection portion 355B between the member 355-1 and the member 355-2.

[0242] As shown surrounded by circle #6, the base member 355 is rotatable about the axis passing through the fitting portion 355C with the detachable portion 356.

[0243] As shown surrounded by ellipse #7, the detachable portion 356 is attached to the rail so as to be rotatable about the central axis of the circular cross-section.

[0244] In this way, the detachable member 351 having the detachable portion 351A at the tip, the relay member 353 connecting the detachable member 351 and the base member 355, and the base member 355 to which the detachable portion 356 is connected at the rear end are each rotatably connected by a hinge portion. The movement of each movable part is controlled according to a command command by a controller in the cooking robot 1.

[0245] FIG. 25 is a diagram showing an example of the connection between the cooking arm and the controller.

[0246] As shown in FIG. 25, the cooking arm 321 and the controller 361 are connected via wiring in a space 311B formed inside the housing 311. In the example of FIG. 25, the cooking arms 321-1 to 321-4 and the controller 361 are connected via wirings 362-1 to 362-4, respectively. The flexible wirings 362-1 to 362-4 will bend appropriately according to the positions of the cooking arms 321-1 to 321-4.

[0247] In this way, the cooking robot 1 is a robot capable of performing various operations related to cooking by driving the cooking arm 321.

[0248] · Configuration around the cooking robot 1 FIG. 26 is a block diagram showing an example of the configuration of the cooking robot 1 and its surroundings.

[0249] The cooking robot 1 is configured by connecting each part to the controller 361. Among the configurations shown in FIG. 26, the same configurations as those described above are labeled with the same reference numerals. Redundant explanations will be omitted as appropriate.

[0250] Connected to the controller 361 are, in addition to the cooking arm 321, a camera 401, an olfactory sensor 402, a taste sensor 403, an infrared sensor 404, a texture sensor 405, an environment sensor 406, and a communication unit 407.

[0251] Although not shown in FIG. 21 and the like, the same sensors as those provided on the chef side are provided on the cooking robot 1 itself or at predetermined positions around the cooking robot 1. The camera 401, the olfactory sensor 402, the taste sensor 403, the infrared sensor 404, the texture sensor 405, and the environment sensor 406 each have the same functions as the camera 41, the olfactory sensor 42, the taste sensor 43, the infrared sensor 51, the texture sensor 52, and the environment sensor 53 on the chef side.

[0252] The controller 361 is composed of a computer having a CPU, a ROM, a RAM, a flash memory, etc. The controller 361 executes a predetermined program by the CPU and controls the overall operation of the cooking robot 1.

[0253] In the controller 361, a command command acquisition unit 421 and an arm control unit 422 are realized by executing a predetermined program.

[0254] The command command acquisition unit 421 acquires the command command transmitted from the control device 12 and received by the communication unit 407. The command command acquired by the command command acquisition unit 421 is supplied to the arm control unit 422.

[0255] The arm control unit 422 controls the operation of the cooking arm 321 according to the command command acquired by the command command acquisition unit 421.

[0256] The camera 401 photographs the state of the cooking arm 321 performing the cooking operation, the state of the food to be cooked, and the state on the top plate 311A of the cooking robot 1, and outputs the image obtained by the photographing to the controller 361. The camera 401 is provided at various positions such as the front of the cooking assistance system 312 and the tip of the cooking arm 321.

[0257] The olfactory sensor 402 measures the smell of the food and transmits the olfactory sensor data to the controller 361. The olfactory sensor 402 is provided at various positions such as the front of the cooking assistance system 312 and the tip of the cooking arm 321.

[0258] The taste sensor 403 measures the taste of the food material and transmits taste sensor data to the controller 361. Also on the reproduction side, a taste sensor 403 such as an artificial lipid membrane type taste sensor is provided.

[0259] An attachment having functions as the olfactory sensor 402 and the taste sensor 403 may be prepared and used by attaching it to the cooking arm 321 during measurement.

[0260] The infrared sensor 404 outputs IR light and generates an IR image. The IR image generated by the infrared sensor 404 is output to the controller 361. Based on the IR image captured by the infrared sensor 404 instead of the image (RGB image) captured by the camera 401, various analyses such as the operation of the cooking robot 1 and the food material may be performed.

[0261] The texture sensor 405 is composed of various sensors used for texture analysis, such as a hardness sensor, a stress sensor, a moisture content sensor, and a temperature sensor, which output sensor data. The hardness sensor, the stress sensor, the moisture content sensor, and the temperature sensor may be provided on an attachment attached to the cooking arm 321 or on cooking tools such as a kitchen knife, a frying pan, and an oven. The sensor data measured by the texture sensor 405 is output to the controller 361.

[0262] The environment sensor 406 is a sensor that measures the dining environment, which is the environment of a space such as a dining room where the meal of the dish reproduced by the cooking robot 1 is taken. In the example of FIG. 26, the environment sensor 406 is composed of a camera 441, a temperature / humidity sensor 442, and an illuminance sensor 443. The environment of the reproduction space where the cooking robot 1 performs cooking may be measured by the environment sensor 406.

[0263] The camera 441 outputs an image of the dining space to the controller 361. By analyzing the image of the dining space, for example, the color (brightness, hue, saturation) of the dining space is measured.

[0264] The temperature and humidity sensor 442 measures the temperature and humidity in the dining space and outputs information representing the measurement results to the controller 361.

[0265] The illuminance sensor 443 measures the brightness of the dining space and outputs information representing the measurement results to the controller 361.

[0266] The communication unit 407 is a wireless communication module such as a wireless LAN module and a mobile communication module compatible with LTE (Long Term Evolution). The communication unit 407 communicates with external devices such as the control device 12 and the recipe data management server 21 on the Internet.

[0267] In addition, the communication unit 407 communicates with mobile terminals such as smartphones and tablet terminals used by the user. The user is a person who eats the dishes reproduced by the cooking robot 1. User operations on the cooking robot 1, such as dish selection, may be input by operations on the mobile terminal.

[0268] As shown in FIG. 26, a motor 431 and a sensor 432 are provided on the cooking arm 321.

[0269] The motor 431 is provided at each joint of the cooking arm 321. The motor 431 performs a rotational motion around the axis according to the control by the arm control unit 422. An encoder for measuring the rotation amount of the motor 431 and a driver for adaptively controlling the rotation of the motor 431 based on the measurement results by the encoder are also provided at each joint.

[0270] The sensor 432 is composed of, for example, a gyro sensor, an acceleration sensor, a touch sensor, etc. The sensor 432 measures the angular velocity, acceleration, etc. of each joint during the operation of the cooking arm 321 and outputs information indicating the measurement results to the controller 361. Sensor data indicating the measurement results of the sensor 432 is also transmitted from the cooking robot 1 to the control device 12 as appropriate.

[0271] Information regarding the specifications of the cooking robot 1, such as the number of cooking arms 321, is provided from the cooking robot 1 to the control device 12 at a predetermined timing. In the control device 12, operation planning is performed according to the specifications of the cooking robot 1. The instruction commands generated in the control device 12 are in accordance with the specifications of the cooking robot 1.

[0272] (3-2) Configuration of the control device 12 The control device 12 that controls the operation of the cooking robot 1 is configured by a computer as shown in FIG. 19, similar to the data processing device 11. Hereinafter, the configuration of the data processing device 11 shown in FIG. 19 will be appropriately cited and described as the configuration of the control device 12.

[0273] FIG. 27 is a block diagram showing an example of the functional configuration of the control device 12.

[0274] At least a part of the functional units shown in FIG. 27 is realized by a predetermined program being executed by the CPU 201 (FIG. 19) of the control device 12.

[0275] As shown in FIG. 27, a command generation unit 501 is realized in the control device 12. The command generation unit 501 is composed of a recipe data acquisition unit 511, a recipe data analysis unit 512, a robot state estimation unit 513, a flavor information processing unit 514, a control unit 515, and a command output unit 516.

[0276] The recipe data acquisition unit 511 controls the communication unit 209 and acquires recipe data by receiving the recipe data transmitted from the data processing device 11 or by communicating with the recipe data management server 21, etc. The recipe data acquired by the recipe data acquisition unit 511 is, for example, the recipe data of a dish selected by the user.

[0277] A database of recipe data may be provided in the storage unit 208. In this case, recipe data is acquired from the database provided in the storage unit 208. The recipe data acquired by the recipe data acquisition unit 511 is supplied to the recipe data analysis unit 512.

[0278] The recipe data analysis unit 512 analyzes the recipe data acquired by the recipe data acquisition unit 511. When it is time to perform a certain cooking process, the recipe data analysis unit 512 analyzes the cooking process data set related to that cooking process and extracts cooking operation information and flavor information. The cooking operation information extracted from the cooking process data set is supplied to the control unit 515, and the flavor information is supplied to the flavor information processing unit 514.

[0279] When the recipe data includes attribute information and environmental information, those pieces of information are also extracted by the recipe data analysis unit 512 and supplied to the flavor information processing unit 514.

[0280] The robot state estimation unit 513 controls the communication unit 209 and receives the image and sensor data transmitted from the cooking robot 1. From the cooking robot 1, an image captured by the camera of the cooking robot 1 and sensor data measured by a sensor provided at a predetermined position of the cooking robot 1 are transmitted at a predetermined cycle. The image captured by the camera of the cooking robot 1 shows the state around the cooking robot 1.

[0281] The robot state estimation unit 513 estimates the state around the cooking robot 1, such as the state of the cooking arm 321 and the state of the food ingredients, by analyzing the image and sensor data transmitted from the cooking robot 1. Information indicating the state around the cooking robot 1 estimated by the robot state estimation unit 513 is supplied to the control unit 515.

[0282] The flavor information processing unit 514 cooperates with the control unit 515 and controls the operation of the cooking robot 1 based on the flavor information supplied from the recipe data analysis unit 512. The operation of the cooking robot 1 controlled by the flavor information processing unit 514 is, for example, an operation related to adjusting the flavor of the food ingredients.

[0283] For example, the flavor information processing unit 514 controls the operation of the cooking robot 1 so that the flavor of the food ingredient being cooked by the cooking robot 1 becomes the same as the flavor represented by the flavor sensor information. Details of the control by the flavor information processing unit 514 will be described with reference to FIG. 28.

[0284] The control unit 515 generates command commands and controls the operation of the cooking robot 1 by causing them to be transmitted from the command output unit 516. The control of the operation of the cooking robot 1 by the control unit 515 is performed based on the cooking operation information supplied from the recipe data analysis unit 512 or based on a request from the flavor information processing unit 514.

[0285] For example, the control unit 515 identifies the food ingredients to be used in the cooking process that is the execution target based on the food ingredient information included in the cooking operation information. Further, the control unit 515 identifies the cooking tools to be used in the cooking process and the operations to be executed on the cooking arm 321 based on the operation information included in the cooking operation information.

[0286] The control unit 515 sets the state where the preparation of the food ingredients is completed as the goal state and sets the operation sequence from the current state, which is the current state of the cooking robot 1, to the goal state. The control unit 515 generates command commands for causing each operation constituting the operation sequence to be performed and outputs them to the command output unit 516.

[0287] In the cooking robot 1, the cooking arm 321 is controlled according to the command commands generated by the control unit 515, and the preparation of the food ingredients is performed. Information representing the state of the cooking robot 1 at each timing, including the state of the cooking arm 321, is transmitted from the cooking robot 1 to the control device 12.

[0288] In addition, when the preparation of the food ingredients is completed, the control unit 515 sets the state where the cooking using the prepared food ingredients (cooking of one cooking process to be executed) is finished as the goal state, and sets the operation sequence from the current state to the goal state. The control unit 515 generates instruction commands for causing each operation constituting the operation sequence to be performed, and outputs them to the command output unit 516.

[0289] In the cooking robot 1, the cooking arm 321 is controlled according to the instruction commands generated by the control unit 515, and cooking using the food ingredients is performed.

[0290] When the cooking using the food ingredients is finished, the control unit 515 generates an instruction command for causing the measurement of the flavor to be performed, and outputs it to the command output unit 516.

[0291] In the cooking robot 1, the cooking arm 321 is controlled according to the instruction commands generated by the control unit 515, and the measurement of the flavor of the food ingredients is appropriately performed using the camera 401, the olfactory sensor 402, the taste sensor 403, the infrared sensor 404, and the texture sensor 405. Information representing the measurement result of the flavor is transmitted from the cooking robot 1 to the control device 12.

[0292] In the flavor information processing unit 514, how to adjust the flavor and the like are planned, and it is required from the flavor information processing unit 514 to the control unit 515 to perform an operation for adjusting the flavor.

[0293] When it is required to perform an operation for adjusting the flavor, the control unit 515 sets the state where the operation is finished as the goal state, and sets the operation sequence from the current state to the goal state. The control unit 515 outputs instruction commands for causing each operation constituting the operation sequence to be performed to the command output unit 516.

[0294] In the cooking robot 1, the cooking arm 321 is controlled according to the instruction commands generated by the control unit 515, and an operation for adjusting the flavor is executed.

[0295] The control of the operation of the cooking robot 1 by the control unit 515 is performed using, for example, the above command commands. The control unit 515 has a function as a generation unit that generates command commands.

[0296] Note that the command command generated by the control unit 515 may be a command that commands the execution of the entire action for causing a certain state transition, or may be a command that commands the execution of a part of the action. That is, one action may be executed according to one command command, or may be executed according to a plurality of command commands.

[0297] The command output unit 516 controls the communication unit 209 and transmits the command command generated by the control unit 515 to the cooking robot 1.

[0298] FIG. 28 is a block diagram showing a configuration example of the flavor information processing unit 514.

[0299] As shown in FIG. 28, the flavor information processing unit 514 is composed of a flavor measurement unit 521, a flavor adjustment unit 522, a subjective information analysis unit 523, an attribute information analysis unit 524, and an environment information analysis unit 525.

[0300] The flavor measurement unit 521 is composed of a taste measurement unit 541, a fragrance measurement unit 542, a texture measurement unit 543, a body sensation temperature measurement unit 544, and a color measurement unit 545.

[0301] The taste measurement unit 541 acquires taste sensor data transmitted from the cooking robot 1 in response to the measurement of the flavor. The taste sensor data acquired by the taste measurement unit 541 is measured by the taste sensor 403 (FIG. 26). In the cooking robot 1, the flavor of the food is measured at a predetermined timing such as the timing when the cooking operation of a certain cooking process is completed.

[0302] The aroma measurement unit 542 acquires the olfactory sensor data transmitted from the cooking robot 1 in response to the measurement of the flavor. The olfactory sensor data acquired by the aroma measurement unit 542 is the data measured by the olfactory sensor 402.

[0303] The texture measurement unit 543 acquires the texture sensor data transmitted from the cooking robot 1 in response to the measurement of the flavor. The texture sensor data acquired by the texture measurement unit 543 is the data measured by the texture sensor 405.

[0304] The body sensation temperature measurement unit 544 acquires the body sensation temperature sensor data transmitted from the cooking robot 1 in response to the measurement of the flavor. The body sensation temperature sensor data acquired by the body sensation temperature measurement unit 544 is the data measured by a temperature sensor provided at a predetermined position of the cooking robot 1, such as inside the taste sensor 403.

[0305] The color measurement unit 545 acquires the color sensor data transmitted from the cooking robot 1 in response to the measurement of the flavor. The color sensor data acquired by the color measurement unit 545 is the data recognized by analyzing the image captured by the camera 401 of the cooking robot 1.

[0306] The sensor data acquired by each part of the flavor measurement unit 521 is supplied to the flavor adjustment unit 522.

[0307] The flavor adjustment unit 522 is composed of a taste adjustment unit 551, an aroma adjustment unit 552, a texture adjustment unit 553, a body sensation temperature adjustment unit 554, and a color adjustment unit 555. The flavor information supplied from the recipe data analysis unit 512 is input to the flavor adjustment unit 522.

[0308] The taste adjustment unit 551 compares the taste sensor data that constitutes the flavor sensor information included in the recipe data with the taste sensor data acquired by the taste measurement unit 541, and determines whether or not they match. Here, when the same operation as the chef's cooking operation is performed by the cooking robot 1, it is determined whether or not the taste of the food material obtained by the cooking operation of the cooking robot 1 matches the taste of the food material obtained by the chef's cooking operation.

[0309] When it is determined that the taste sensor data that constitutes the flavor sensor information included in the recipe data matches the taste sensor data acquired by the taste measurement unit 541, the taste adjustment unit 551 determines that no adjustment is required for the taste.

[0310] On the other hand, when it is determined that the taste sensor data that constitutes the flavor sensor information included in the recipe data does not match the taste sensor data acquired by the taste measurement unit 541, the taste adjustment unit 551 plans how to adjust the taste and requests the control unit 515 to perform an operation for adjusting the taste.

[0311] The control unit 515 is requested to perform operations such as adding salt when the saltiness is insufficient and squeezing lemon juice when the sourness is insufficient.

[0312] Similarly, in other processing units of the flavor adjustment unit 522, it is determined whether or not the flavor of the food material obtained by the cooking operation of the cooking robot 1 matches the flavor of the food material obtained by the chef's cooking operation, and the flavor is adjusted as appropriate.

[0313] That is, the aroma adjustment unit 552 compares the olfactory sensor data that constitutes the flavor sensor information included in the recipe data with the olfactory sensor data acquired by the aroma measurement unit 542, and determines whether or not they match. Here, it is determined whether or not the aroma of the food material obtained by the cooking operation of the cooking robot 1 matches the aroma of the food material obtained by the chef's cooking operation.

[0314] When it is determined that the olfactory sensor data constituting the flavor sensor information included in the recipe data matches the olfactory sensor data acquired by the fragrance measurement unit 542, the fragrance adjustment unit 552 determines that no adjustment is required for the fragrance.

[0315] On the other hand, when it is determined that the olfactory sensor data constituting the flavor sensor information included in the recipe data does not match the olfactory sensor data acquired by the fragrance measurement unit 542, the fragrance adjustment unit 552 plans how to adjust the fragrance and requests the control unit 515 to perform operations for adjusting the fragrance.

[0316] The control unit 515 is requested to perform operations such as squeezing lemon juice when it has a fishy smell or adding minced herbs when the citrus fragrance is weak.

[0317] The texture adjustment unit 553 compares the texture sensor data constituting the flavor sensor information included in the recipe data with the texture sensor data acquired by the texture measurement unit 543, and determines whether or not they match. Here, it is determined whether or not the texture of the food material obtained by the cooking operation of the cooking robot 1 matches the texture of the food material obtained by the cooking operation of the chef.

[0318] When it is determined that the texture sensor data constituting the flavor sensor information included in the recipe data matches the texture sensor data acquired by the texture measurement unit 543, the texture adjustment unit 553 determines that no adjustment is required for the texture.

[0319] On the other hand, when it is determined that the texture sensor data constituting the flavor sensor information included in the recipe data does not match the texture sensor data acquired by the texture measurement unit 543, the texture adjustment unit 553 plans how to adjust the texture and requests the control unit 515 to perform operations for adjusting the texture.

[0320] The control unit 515 is requested to perform operations such as hitting to soften the food material when it is hard or increasing the cooking time.

[0321] The sensed temperature adjustment unit 554 compares the sensed temperature sensor data that constitutes the flavor sensor information included in the recipe data with the sensed temperature sensor data acquired by the sensed temperature measurement unit 544, and determines whether or not they match. Here, it is determined whether or not the sensed temperature of the food material obtained by the cooking operation of the cooking robot 1 matches the sensed temperature of the food material obtained by the cooking operation of the chef.

[0322] When it is determined that the sensed temperature sensor data that constitutes the flavor sensor information included in the recipe data matches the sensed temperature sensor data acquired by the sensed temperature measurement unit 544, the sensed temperature adjustment unit 554 determines that no adjustment is required for the sensed temperature.

[0323] On the other hand, when it is determined that the sensed temperature sensor data that constitutes the flavor sensor information included in the recipe data does not match the sensed temperature sensor data acquired by the sensed temperature measurement unit 544, the sensed temperature adjustment unit 554 plans how to adjust the sensed temperature and requests the control unit 515 to perform an operation for adjusting the sensed temperature.

[0324] When the sensed temperature of the food material is low, the control unit 515 is requested to perform an operation such as heating using an oven, and when the sensed temperature of the food material is high, to cool it.

[0325] The color adjustment unit 555 compares the color sensor data that constitutes the flavor sensor information included in the recipe data with the color sensor data acquired by the color measurement unit 545, and determines whether or not they match. Here, it is determined whether or not the color of the food material obtained by the cooking operation of the cooking robot 1 matches the color of the food material obtained by the cooking operation of the chef.

[0326] When it is determined that the color sensor data that constitutes the flavor sensor information included in the recipe data matches the color sensor data acquired by the color measurement unit 545, the color adjustment unit 555 determines that no adjustment is required for the color.

[0327] On the other hand, when it is determined that the color sensor data constituting the flavor sensor information included in the recipe data does not match the color sensor data acquired by the color measurement unit 545, the color adjustment unit 555 plans how to adjust the color and requests the control unit 515 to perform an operation for color adjustment.

[0328] When the cooked food is served, if the way of serving by the cooking robot 1 is different from the way of serving by the chef, the control unit 515 is requested to perform an operation such as moving the position of the food so as to approach the way of serving by the chef.

[0329] The subjective information analysis unit 523 analyzes the flavor subjective information included in the flavor information, and reflects the way the chef feels the flavor represented by the flavor subjective information in the flavor adjustment performed by the flavor adjustment unit 522.

[0330] The attribute information analysis unit 524 analyzes the attribute information included in the recipe data, and reflects the chef's attributes in the flavor adjustment performed by the flavor adjustment unit 522.

[0331] The environmental information analysis unit 525 analyzes the environmental information included in the recipe data, and reflects the difference between the cooking environment and the dining environment measured by the environment sensor 406 in the flavor adjustment performed by the flavor adjustment unit 522.

[0332] <Operation of the cooking system> Here, the operation of the cooking system having the above configuration will be described.

[0333] (1) Operations on the chef side First, with reference to the flowchart of FIG. 29, the recipe data generation process of the data processing device 11 will be described.

[0334] The process of FIG. 29 starts when the preparation of food ingredients and cooking tools is completed and the chef begins cooking. Photographing by the camera 41, generation of IR images by the infrared sensor 51, sensing by sensors attached to the chef's body, etc. also start.

[0335] In step S1, the food ingredient recognition unit 251 in FIG. 20 analyzes the image captured by the camera 41 and recognizes the food ingredients used by the chef.

[0336] In step S2, the motion recognition unit 253 analyzes the image captured by the camera 41, sensor data representing the measurement results of sensors attached to the chef's body, etc., and recognizes the cooking actions of the chef.

[0337] In step S3, the recipe data generation unit 233 generates cooking action information based on the food ingredient information generated based on the recognition result by the food ingredient recognition unit 251 and the motion information generated based on the recognition result by the motion recognition unit 253.

[0338] In step S4, the recipe data generation unit 233 determines whether one cooking process has ended. If it is determined that one cooking process has not ended yet, it returns to step S1 and repeats the above-described process.

[0339] If it is determined in step S4 that one cooking process has ended, the process proceeds to step S5.

[0340] In step S5, flavor information generation processing is performed. Flavor information is generated by the flavor information generation processing. Details of the flavor information generation processing will be described later with reference to the flowchart of FIG. 30.

[0341] In step S6, the recipe data generation unit 233 generates a cooking process data set by associating the cooking action information and the flavor information.

[0342] In step S7, the recipe data generation unit 233 determines whether or not all cooking steps have been completed. If it is determined that not all cooking steps have been completed yet, the process returns to step S1 and the above-described processing is repeated. The same processing is repeated for the next cooking step.

[0343] If it is determined in step S7 that all cooking steps have been completed, the process proceeds to step S8.

[0344] In step S8, the recipe data generation unit 233 generates recipe data including all cooking step data sets.

[0345] Next, with reference to the flowchart of FIG. 30, the flavor information generation process performed in step S5 of FIG. 29 will be described.

[0346] In step S11, the taste measurement unit 261 measures the taste of the food material by controlling the taste sensor 43.

[0347] In step S12, the aroma measurement unit 262 measures the aroma of the food material by controlling the olfactory sensor 42.

[0348] In step S13, the texture measurement unit 263 measures the texture of the food material based on an image captured by the camera 41, measurement results by the texture sensor 52, and the like.

[0349] In step S14, the somatic sensation temperature measurement unit 264 measures the somatic sensation temperature of the food material measured by the temperature sensor.

[0350] In step S15, the color measurement unit 265 measures the color of the food material based on an image captured by the camera 41.

[0351] In step S16, the subjective information generation unit 266 generates flavor subjective information based on the sensor data acquired by each of the taste measurement unit 261 to the color measurement unit 265.

[0352] In step S17, the recipe data generation unit 233 generates flavor information based on the flavor sensor information composed of the sensor data measured by the taste measurement unit 261 to the color measurement unit 265 and the flavor subjective information generated by the subjective information generation unit 266.

[0353] After the flavor information is generated, the process returns to step S5 in FIG. 29, and the subsequent processing is performed.

[0354] (2) Operation on the reproduction side With reference to the flowchart in FIG. 31, the cooking reproduction process of the control device 12 will be described.

[0355] In step S31, the recipe data acquisition unit 511 in FIG. 27 acquires the recipe data transmitted from the data processing device 11. The recipe data acquired by the recipe data acquisition unit 511 is analyzed by the recipe data analysis unit 512, and cooking operation information and flavor information are extracted. The cooking operation information is supplied to the control unit 515, and the flavor information is supplied to the flavor information processing unit 514.

[0356] In step S32, the control unit 515 selects one cooking process as the execution target. It is selected as the execution target in order from the cooking process data set related to the first cooking process.

[0357] In step S33, the control unit 515 determines whether the cooking process to be executed is a cooking process for plating the cooked ingredients. If it is determined in step S33 that it is not a cooking process for plating the cooked ingredients, the process proceeds to step S34.

[0358] In step S34, the control unit 515 prepares the ingredients to be used in the cooking process to be executed based on the description of the ingredient information included in the cooking operation information.

[0359] In step S35, the control unit 515 generates a command based on the description of the operation information included in the cooking operation information, and transmits it to the cooking robot 1 to cause the cooking arm 321 to execute the cooking operation.

[0360] In step S36, flavor measurement processing is performed. By the flavor measurement processing, the flavor of the cooked food cooked by the cooking robot 1 is measured. Details of the flavor measurement processing will be described later with reference to the flowchart of FIG. 32.

[0361] In step S37, the flavor adjustment unit 522 determines whether the flavor of the cooked food matches the flavor represented by the flavor sensor information included in the recipe data. Here, for all of the taste, aroma, texture, body temperature, and color, which are the components of the flavor, when the flavor of the cooked food matches the flavor represented by the flavor sensor information, it is determined that the flavors match.

[0362] If it is determined in step S37 that the flavors do not match because any of the components do not match, flavor adjustment processing is performed in step S38. By the flavor adjustment processing, the flavor of the cooked food is adjusted. Details of the flavor adjustment processing will be described later with reference to the flowchart of FIG. 33.

[0363] After the flavor adjustment processing is performed in step S38, the process returns to step S36, and the above-described processing is repeatedly executed until it is determined that the flavors match.

[0364] On the other hand, if it is determined in step S33 that the cooking process to be executed is a cooking process of plating the cooked food, the process proceeds to step S39.

[0365] In step S39, the control unit 515 generates a command based on the description of the cooking operation information, and transmits it to the cooking robot 1 to cause the cooking arm 321 to perform plating.

[0366] When the plating of the food ingredients is completed, or when it is determined in step S37 that the flavor represented by the flavor of the cooked food ingredients matches the flavor sensor information included in the recipe data, the process proceeds to step S40.

[0367] In step S40, the control unit 515 determines whether or not all the cooking steps have been completed. If it is determined that not all the cooking steps have been completed, the process returns to step S32 and the above-described process is repeated. The same process is repeated for the next cooking step.

[0368] On the other hand, if it is determined in step S40 that all the cooking steps have been completed, the dish is completed and the dish reproduction process is terminated.

[0369] Next, with reference to the flowchart of FIG. 32, the flavor measurement process performed in step S36 of FIG. 31 will be described.

[0370] In step S51, the taste measurement unit 541 of FIG. 28 causes the cooking robot 1 to measure the taste of the cooked food ingredients and acquires taste sensor data.

[0371] In step S52, the aroma measurement unit 542 causes the cooking robot 1 to measure the aroma of the cooked food ingredients and acquires olfactory sensor data.

[0372] In step S53, the texture measurement unit 543 causes the cooking robot 1 to measure the texture of the cooked food ingredients and acquires texture sensor data.

[0373] In step S54, the perceived temperature measurement unit 544 causes the cooking robot 1 to measure the perceived temperature of the cooked food ingredients and acquires perceived temperature sensor data.

[0374] In step S55, the color measurement unit 545 causes the cooking robot 1 to measure the color of the cooked food ingredients and acquires color sensor data.

[0375] Through the above processing, the flavor of the cooked food ingredients is measured and can be used for the flavor adjustment processing described later. Then, the process returns to step S36 in FIG. 31, and the subsequent processing is performed.

[0376] Next, with reference to the flowchart of FIG. 33, the flavor adjustment processing performed in step S38 of FIG. 31 will be described.

[0377] In step S61, the taste adjustment unit 551 performs taste adjustment processing. The taste adjustment processing is performed when the taste of the cooked food ingredients does not match the taste represented by the taste sensor data included in the flavor sensor information. The details of the taste adjustment processing will be described later with reference to the flowchart of FIG. 34.

[0378] In step S62, the aroma adjustment unit 552 performs aroma adjustment processing. The aroma adjustment processing is performed when the aroma of the cooked food ingredients does not match the aroma represented by the olfactory sensor data included in the flavor sensor information.

[0379] In step S63, the texture adjustment unit 553 performs texture adjustment processing. The texture adjustment processing is performed when the texture of the cooked food ingredients does not match the texture represented by the texture sensor data included in the flavor sensor information.

[0380] In step S64, the perceived temperature adjustment unit 554 performs perceived temperature adjustment processing. The perceived temperature adjustment processing is performed when the perceived temperature of the cooked food ingredients does not match the perceived temperature represented by the perceived temperature sensor data included in the flavor sensor information.

[0381] In step S65, the color adjustment unit 555 performs color adjustment processing. The color adjustment processing is performed when the color of the cooked food ingredients does not match the color represented by the color sensor data included in the flavor sensor information.

[0382] For example, when an operation of pouring lemon juice over food ingredients is performed as a taste adjustment process to increase the sourness, the aroma of the food ingredients may change as a result, and it may be necessary to adjust the aroma as well. In this case, an aroma adjustment process is performed together with the taste adjustment process.

[0383] As described above, the adjustment of any one element of the flavor may affect other elements, and in practice, the adjustment of multiple elements is performed together.

[0384] Next, with reference to the flowchart of FIG. 34, the taste adjustment process performed in step S61 of FIG. 33 will be described.

[0385] In step S71, the taste adjustment unit 551 identifies the current value of the taste of the cooked food ingredients in the taste space based on the taste sensor data acquired by the taste measurement unit 541.

[0386] In step S72, the taste adjustment unit 551 sets a target value for the taste based on the description of the flavor sensor information included in the recipe data. The taste of the food ingredients obtained by the cooking operation performed by the chef, which is represented by the taste sensor data included in the flavor sensor information, is set as the target value.

[0387] In step S73, the taste adjustment unit 551 plans the adjustment content for transitioning the taste of the food ingredients from the current value to the target value.

[0388] FIG. 35 is a diagram showing an example of the planning.

[0389] The vertical axis shown in FIG. 35 represents any one of the seven types of tastes, and the horizontal axis represents another one of the tastes. For the sake of convenience of explanation, in FIG. 35, the taste space is represented as a two-dimensional space. However, when the taste is composed of the seven types of salty, sour, bitter, sweet, umami, spicy, and astringent tastes as described above, the taste space becomes a seven-dimensional space.

[0390] The taste of the cooked food ingredients is represented as the current value based on the taste sensor data measured in the cooking robot 1.

[0391] Also, based on the taste sensor data included in the flavor sensor information, the target taste is set. The target taste is the taste of the ingredients cooked by the chef.

[0392] Since there is no seasoning or ingredient that changes only one type of taste among saltiness, sourness, bitterness, sweetness, umami, spiciness, and astringency, there may be cases where the taste of the ingredients cannot be directly changed from the current taste to the target taste. In this case, as shown by the white arrow, planning of the cooking operation to achieve the target taste through multiple tastes is performed.

[0393] Returning to the explanation of FIG. 34, in step S74, the taste adjustment unit 551 causes the control unit 515 to perform an operation for adjusting the taste according to the plan.

[0394] Thereafter, it returns to step S61 in FIG. 33, and the subsequent processing is performed.

[0395] The aroma adjustment process (step S62), the texture adjustment process (step S63), the perceived temperature adjustment process (step S64), and the color adjustment process (step S65) are also performed in the same manner as the taste adjustment process in FIG. 34. That is, with the flavor of the cooked ingredients as the current value and the flavor represented by the flavor sensor information in the recipe data as the target value, a cooking operation for transitioning the taste of the ingredients from the current value to the target value is performed.

[0396] Through the above series of processes, a dish with the same flavor as the dish made by the chef is reproduced by the cooking robot 1. The user can eat a dish with the same flavor as the dish made by the chef.

[0397] Also, the chef can provide dishes with the same flavor as the dishes they made to various people. Also, the chef can leave the dishes they make in a form that can be reproduced as recipe data.

[0398] <Modification Example> · Example of updating the cooking process on the reproduction side In some cases, the reproduction side may not be able to prepare the same ingredients as those described in the recipe data (ingredient information) for cooking. In this case, the control unit 515 (Fig. 27) may perform a process of partially updating the recipe data.

[0399] For example, when there is a shortage of a certain ingredient, the control unit 515 refers to the substitute ingredient database and selects a substitute ingredient from among the ingredients that can be prepared on the reproduction side. A substitute ingredient is an ingredient that is used in place of the ingredient described in the recipe data for cooking. The ingredients that can be prepared on the reproduction side are specified, for example, by recognizing the situation around the cooking robot 1.

[0400] The substitute ingredient database referred to by the control unit 515 describes information about substitute ingredients determined in advance by, for example, the food pairing method.

[0401] For example, when the control unit 515 cannot prepare the ingredient "sea urchin" described in the recipe data, it refers to the substitute ingredient database and selects the combined ingredient of "pudding" and "soy sauce" as the substitute ingredient. It is well known that the flavor of "sea urchin" can be reproduced by combining "pudding" and "soy sauce".

[0402] The control unit 515 updates the cooking operation information describing the cooking process using "sea urchin" to the cooking operation information describing the operation of combining "pudding" and "soy sauce" and the cooking process using the substitute ingredient. The control unit 515 controls the cooking operation of the cooking robot 1 based on the updated cooking operation information.

[0403] The flavor of the substitute ingredient prepared in this way may be measured, and the flavor may be adjusted as appropriate.

[0404] Fig. 36 is a flowchart for explaining the process of the control device 12 for adjusting the flavor of the substitute ingredient.

[0405] The process in FIG. 36 is performed after the alternative ingredients are prepared.

[0406] In step S111, the flavor measurement unit 521 of the flavor information processing unit 514 measures the flavor of the prepared alternative ingredients and obtains sensor data representing the flavor of the alternative ingredients.

[0407] In step S112, the flavor adjustment unit 522 determines whether the flavor of the alternative ingredients matches the flavor of the ingredients before substitution. In the case of the example described above, it is determined whether the flavor of the alternative ingredients combined with "pudding" and "soy sauce" matches the flavor of "sea urchin". The flavor of "sea urchin" is specified by the flavor sensor information included in the recipe data.

[0408] If it is determined in step S112 that the flavor of the alternative ingredients does not match the flavor of the ingredients before substitution because the sensor data representing the flavor of the alternative ingredients does not match the flavor sensor information included in the recipe data, the process proceeds to step S113.

[0409] In step S113, the flavor adjustment unit 522 adjusts the flavor of the alternative ingredients. The adjustment of the flavor of the alternative ingredients is performed in the same manner as the process of adjusting the flavor of the cooked ingredients described above.

[0410] When the flavor of the alternative ingredients is adjusted, or if it is determined in step S112 that the flavor of the alternative ingredients matches the flavor of the ingredients before substitution, the process of adjusting the flavor of the alternative ingredients ends. Thereafter, using the alternative ingredients, processing according to the updated cooking process is performed.

[0411] Thereby, even when it is not possible to prepare the same ingredients as those used by the chef on the reproduction side, it is possible to proceed with cooking using alternative ingredients. Since the flavor of the alternative ingredients is the same as the flavor of the ingredients before substitution, the finally completed dish will be the same as or have a similar flavor to the dish made by the chef.

[0412] Instead of having the ingredient database prepared in the control device 12, it may be prepared in a predetermined server such as the recipe data management server 21. The update of the cooking operation information may be performed in the control device 12, or may be performed in the data processing device 11.

[0413] · Examples of using flavor subjective information There may be cases where the specifications of the sensors on both sides are different, such as when the sensors provided on the chef side have higher measurement accuracy than the sensors provided on the reproduction side. When the specifications of both sides are different, the measurement results will be different when measuring the flavor of the same ingredient with each sensor.

[0414] In order to be able to determine the flavor of the cooked ingredients by the cooking robot 1 and the flavor of the ingredients cooked by the chef even when the specifications of the sensors provided on both the chef side and the reproduction side are different, flavor subjective information is used.

[0415] FIG. 37 is a diagram showing an example of flavor determination.

[0416] In the example described above, as shown on the left side of FIG. 37, when a cooked ingredient is obtained by cooking in a certain cooking process on the reproduction side, the flavor is measured and sensor data representing the flavor of the cooked ingredient is obtained.

[0417] Also, as shown on the right side of FIG. 37, flavor sensor information is extracted from the recipe data, and as shown by arrow A101, by comparing the sensor data representing the flavor of the cooked ingredient with the flavor sensor information, a flavor determination (determination of whether the flavors match) is made.

[0418] FIG. 38 is a diagram showing an example of flavor determination using flavor subjective information.

[0419] When determining flavor using flavor subjective information, on the reproduction side, as shown on the left side of FIG. 38, flavor subjective information is calculated based on sensor data representing the flavor of the cooked ingredients. For calculating the flavor subjective information, a model generated based on the chef's way of perceiving taste, as described with reference to FIG. 6, is used.

[0420] The subjective information analysis unit 523 (FIG. 28) of the flavor information processing unit 514 has the same model as the model for generating taste subjective information prepared on the chef side.

[0421] As indicated by arrow A102, the subjective information analysis unit 523 determines the flavor by comparing the flavor subjective information calculated based on the sensor data representing the flavor of the cooked ingredients with the flavor subjective information extracted from the recipe data. When the flavor subjective information of both matches, it is determined that the flavors match, and the processing of the next cooking step is performed.

[0422] This makes it possible to reproduce ingredients and dishes with the same flavor as the flavor felt by the chef, even when the specifications of the sensors provided on both the chef side and the reproduction side are different.

[0423] In this way, for the mode of determining flavor, a mode based on sensor data and a mode based on flavor subjective information are prepared.

[0424] FIG. 39 is a diagram showing an example of a model for generating sensor data.

[0425] As shown in FIG. 39, a model capable of calculating sensor data under the specifications of the sensors provided on the reproduction side based on the flavor subjective information included in the recipe data may be prepared in the subjective information analysis unit 523.

[0426] The model for generating taste sensor information shown in Fig. 39 is a model such as a neural network generated by performing deep learning or the like based on sensor data related to taste measured by a sensor prepared on the reproduction side and subjective values representing the chef's taste perception. For example, a model corresponding to the specifications of various sensors is prepared by an administrator who manages recipe data and provided to the reproduction side.

[0427] In this case, the subjective information analysis unit 523 calculates the corresponding sensor data by inputting the flavor subjective information into the model. The subjective information analysis unit 523 determines the flavor by comparing the sensor data obtained by measuring the flavor of the cooked food by the cooking robot 1 with the sensor data calculated using the model.

[0428] ·Usage example of attribute information The recipe data includes attribute information representing the attributes of the chef, etc. Since age, gender, nationality, living area, etc. affect the way of perceiving flavor, the flavor of the reproduced ingredients may be adjusted according to the differences between the attributes of the chef and the attributes of the person who eats the dish reproduced by the cooking robot 1.

[0429] The cook attribute information, which is the attribute information extracted from the recipe data, is supplied to the attribute information analysis unit 524 and used for controlling the flavor adjustment performed by the flavor adjustment unit 522. The diner attribute information representing the attributes of the diner, which is input by the person who eats the dish reproduced by the cooking robot 1, is also supplied to the attribute information analysis unit 524.

[0430] The attribute information analysis unit 524 identifies the attributes of the chef based on the cook attribute information and the attributes of the diner based on the diner attribute information.

[0431] For example, when the attribute information analysis unit 524 identifies that the age of the diner is much higher than the age of the chef and the diner is an elderly person, it adjusts the texture of the ingredients to be softer.

[0432] In addition, when the nationalities of the diner and the chef are different, as described above, the flavor adjustment unit 522 adjusts the flavor of the food ingredients based on the pre-prepared information according to the difference in nationality. Similarly, when other attributes of the diner and the chef, such as gender and living area, are different, the attribute information analysis unit 524 controls the flavor of the food ingredients adjusted by the flavor adjustment unit 522 according to the difference in their attributes.

[0433] As a result, basically, although the flavor is the same as that felt by the chef, a dish with fine-tuned flavor according to the diner's preference is reproduced.

[0434] In addition, the attribute information analysis unit 524 identifies the attributes of the food ingredients based on the food attribute information and also identifies the attributes of the food ingredients prepared on the reproduction side.

[0435] When the attributes of the food ingredients used on the chef side and the food ingredients prepared on the reproduction side are different, the attribute information analysis unit 524 controls the flavor of the food ingredients adjusted by the flavor adjustment unit 522 according to the difference in attributes.

[0436] In this way, the flavor of the food ingredients may be adjusted on the reproduction side based on the differences in various attributes between the chef side and the reproduction side.

[0437] ·Example of using environmental information (1) Adjustment of dining environment The recipe data includes environmental information representing the cooking environment, which is the environment of the space where the chef cooks. Since the color, temperature, brightness, etc. of the space affect the way of feeling the flavor, the dining environment such as the dining room where the meal of the dish reproduced by the cooking robot 1 is taken may be adjusted to be close to the cooking environment. The environmental information extracted from the recipe data is supplied to the environmental information analysis unit 525 and used for adjusting the dining environment.

[0438] For example, the environmental information analysis unit 525 controls the lighting equipment in the dining room so as to approximate the color of the dining environment measured by analyzing the image captured by the camera 441 (Fig. 26) to the color of the cooking environment represented by the environmental information. The environmental information analysis unit 525 has a function as an environment control unit that adjusts the dining environment by controlling external devices.

[0439] Also, the environmental information analysis unit 525 controls the air conditioning equipment in the dining room so as to approximate the temperature and humidity of the dining environment measured by the temperature and humidity sensor 442 to the temperature and humidity of the cooking environment represented by the environmental information.

[0440] The environmental information analysis unit 525 controls the lighting equipment in the dining room so as to approximate the brightness of the dining environment measured by the illuminance sensor 443 to the brightness of the cooking environment represented by the environmental information.

[0441] Thereby, the dining environment can be approximated to the cooking environment, and it becomes possible to approximate the way a person eating the dish reproduced by the cooking robot 1 feels the flavor to the way a chef feels the flavor.

[0442] (2) Correction of flavor sensor information Information regarding the specifications of the sensors provided on the chef side may be included in the environmental information and provided to the reproduction side. On the reproduction side, the flavor sensor information included in the recipe data is corrected based on the difference between the sensors provided on the chef side and the sensors provided on the reproduction side.

[0443] Fig. 40 is a flowchart for explaining the processing of the control device 12 that corrects the flavor sensor information.

[0444] In step S121, the environmental information analysis unit 525 acquires the specifications of the sensors provided on the chef side based on the environmental information included in the recipe data.

[0445] In step S122, the environmental information analysis unit 525 acquires the specifications of the sensors provided around the cooking robot 1 on the reproduction side.

[0446] In step S123, based on the difference between the specifications of the sensors provided on the chef side and the specifications of the sensors provided around the cooking robot 1, the environmental information analysis unit 525 corrects the flavor sensor information included in the recipe data, which is the sensor data measured on the chef side. Information representing the correspondence between the measurement results of the sensors provided on the chef side and the measurement results of the sensors provided on the reproduction side is prepared as correction information for the environmental information analysis unit 525.

[0447] The flavor sensor information corrected in this way is used for determining the flavor. As a result, it becomes possible to absorb the difference in the environment and perform the determination of the flavor.

[0448] <Others> ·Configuration modification examples Although the cooking robot 1 that reproduces a dish based on recipe data is installed in the home, the dish may be reproduced in cooking robots provided in various locations. For example, the above-described technology is applicable even when reproducing a dish in a cooking robot provided in a factory or a cooking robot provided in a restaurant.

[0449] Also, although the cooking robot that reproduces a dish based on recipe data is the cooking robot 1 that operates a cooking arm to perform cooking, the dish may be reproduced in various cooking robots that can cook ingredients by a configuration other than the cooking arm.

[0450] In the above, the control of the cooking robot 1 is assumed to be performed by the control device 12, but it may be directly performed by the data processing device 11 that generates the recipe data. In this case, the data processing device 11 is provided with each configuration of the command generation unit 501 described with reference to FIG. 27.

[0451] Further, each component of the command generation unit 501 may be provided in the recipe data management server 21.

[0452] The server function of the recipe data management server 21 that manages recipe data and provides it to other devices may be provided in the data processing device 11 that generates recipe data.

[0453] FIG. 41 is a diagram showing another configuration example of the cooking system.

[0454] The recipe data management unit 11A included in the data processing device 11 has a server function of managing recipe data and providing it to other devices. The recipe data managed by the recipe data management unit 11A is provided to a plurality of cooking robots and a control device that controls the cooking robots.

[0455] · Management of data The above-described recipe data, cooking process data set (cooking operation information, flavor information), etc. can be regarded as works that creatively express the ideas and feelings about the cooking process, and thus can also be considered as copyrighted works.

[0456] For example, a chef who cooks (such as a chef who manages a famous restaurant) repeats trials such as ingredient selection and taste testing in the cooking process to complete a delicious and creative dish. In this case, the recipe data and the cooking process data set (cooking operation information, flavor information) have value as data, and it is also conceivable that a consideration is required when others use them.

[0457] Therefore, it is also conceivable to apply copyright management to recipe data, cooking process data sets (cooking operation information, flavor information), etc. in the same way as music.

[0458] That is, in the present disclosure, it is also possible to protect individual recipe data and cooking process data sets by using copyright protection technologies such as copy prevention and encryption that provide a protection function for individual data.

[0459] In this case, for example, the recipe data management server 21 in FIG. 14 (data processing device 11 in FIG. 41) manages the chef and the recipe data (or cooking process data set) in a state of copyright management in an associated manner.

[0460] Next, when the user wants to have the cooking robot 1 perform cooking using the recipe data, by the user paying the usage fee for the recipe data, for example, the recipe data downloaded to the control device 12 can be used for cooking in the cooking robot 1. Note that the usage fee is returned to the chef who is the creator of the recipe data, the data administrator who manages the recipe data, and the like.

[0461] Also, in the present disclosure, it is also possible to protect individual recipe data and cooking process data sets by using blockchain technology in which the transaction history of data is distributed and managed by a server as a ledger.

[0462] In this case, for example, the recipe data management server 21 in FIG. 14 (data processing device 11 in FIG. 41) manages the chef and the recipe data (or cooking process data set) in an associated manner by using blockchain technology in which the transaction history of data is distributed and managed by a server (such as a cloud server or an edge server) as a ledger.

[0463] Next, when the user wants to have the cooking robot 1 perform cooking using the recipe data, by the user paying the usage fee for the recipe data, for example, the recipe data downloaded to the control device 12 can be used for cooking in the cooking robot 1. Note that the usage fee is returned to the chef who is the creator of the recipe data, the data administrator who manages the recipe data, and the like.

[0464] In this way, it is possible to efficiently manage the recipe data (or cooking process data set) as a work expressed in a creative form in consideration of the respective relationships among the chef, the user, and the usage fee.

[0465] Characterization of foodstuffs using temperature changes in absorption spectra The flavor of foodstuffs is represented by sensor data such as taste, aroma, and texture, but it may also be represented by other indices. As an index for expressing the flavor of foodstuffs, it is possible to use the temperature change in the absorption spectrum.

[0466] Principle Using a spectrophotometer, the absorption spectrum of a specimen (foodstuff) is measured. The absorption spectrum changes according to the temperature of the specimen. As a background for the change in the absorption spectrum with an increase in temperature, the following reactions are conceivable.

[0467] (1) Dissociation from association The associated state of the components contained in the specimen (a state in which two or more molecules move like one molecule due to weak intermolecular bonds) changes with temperature. When the temperature decreases, association or aggregation becomes easier, and conversely, when the temperature increases, molecular vibration becomes intense, so the molecules tend to deviate from the association. Therefore, the peak value of the absorption wavelength derived from the association decreases, and the peak value of the absorption wavelength derived from the dissociated single molecules increases.

[0468] (2) Decomposition of molecules by thermal energy By absorbing heat, the weak part of the binding force is released and the molecule is broken.

[0469] (3) Decomposition of molecules by enzyme activity The molecule is broken through a degrading enzyme.

[0470] (4) Oxidation-reduction As the temperature rises, the pH of water decreases (the H+ concentration increases). In the case of oils and fats, the oxidation rate increases.

[0471] Here, from the viewpoints of the taste and aroma of natural products such as foodstuffs, among the components contained in natural products, the taste substances are components in the liquid phase, and the aroma substances are volatile components in the gas phase.

[0472] Molecules in an associated state are less likely to enter the gas phase, while single molecules dissociated from the associated state are more likely to transfer to the gas phase.

[0473] Furthermore, for example, terpenes, which are deeply related to fragrance, exist in the form of glycosides with sugars in plants. However, through thermal decomposition or enzymatic decomposition, they become aglycones without sugars, making them more volatile.

[0474] Therefore, as the temperature increases, the number of volatile molecules increases, the peak value of the absorption wavelength of the fragrance substance just before volatilization rises, and the peak value of the absorption wavelength related to the molecular group to which the fragrance substance was associated until then decreases.

[0475] From this property, it can be considered that the temperature change of the absorption spectrum reflects the phase transition from the liquid phase related to "taste" to the gas phase related to "fragrance".

[0476] Therefore, it is possible to keep the target sample at at least two or more different temperatures, measure the absorption spectra of the samples in each heat-retaining state, and use the dataset as information characterizing the taste and fragrance of the sample. It becomes possible to identify the sample from the characteristics (patterns) of the dataset of the absorption spectrum.

[0477] This can be said to take into account the high probability of phase transition from the liquid phase to the gas phase as a result of the dissociation of molecules from association or the decomposition of molecules by thermal decomposition or enzymatic decomposition, and the temperature change of the absorption spectrum. This method can be said to be a method of characterizing a sample by an absorption spectrum of three-dimensional data by adding the dimension of temperature to the absorption spectrum represented as two-dimensional data of wavelength and absorbance.

[0478] ·Regarding the program The above-described series of processes can be executed either by hardware or by software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware or a general-purpose personal computer or the like.

[0479] The installed program is provided by being recorded on a removable medium 211 shown in FIG. 19, which includes an optical disk (such as a CD-ROM (Compact Disc-Read Only Memory) or a DVD (Digital Versatile Disc)) or a semiconductor memory. Further, it may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be installed in advance in the ROM 202 or the storage unit 208.

[0480] The program executed by the computer may be a program in which processing is performed in time series in accordance with the order described in this specification, or a program in which processing is performed in parallel or at a necessary timing such as when a call is made.

[0481] In this specification, a system means a collection of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a single device in which a plurality of modules are housed in one housing are both systems.

[0482] The effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0483] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

[0484] For example, the present technology can adopt a cloud computing configuration in which one function is shared and jointly processed by a plurality of devices via a network.

[0485] In addition, each step described in the above flowchart can be executed by one device or can be shared and executed by a plurality of devices.

[0486] Furthermore, when a plurality of processes are included in one step, the plurality of processes included in that one step can be executed by one device or can be shared and executed by a plurality of devices.

Description of Reference Numerals

[0487] 1 Cooking robot, 11 Data processing device, 12 Control device, 21 Recipe data management server, 41 Camera, 42 Olfactory sensor, 43 Taste sensor, 51 Infrared sensor, 52 Texture sensor, 53 Environment sensor, 221 Data processing unit, 231 Cooking operation information generation unit, 232 Flavor information generation unit, 233 Recipe data generation unit, 234 Environment information generation unit, 235 Attribute information generation unit, 236 Recipe data output unit, 321 Cooking arm, 361 Controller, 401 Camera, 402 Olfactory sensor, 403 Taste sensor, 404 Infrared sensor, 405 Texture sensor, 406 Environment sensor, 407 Communication unit, 501 Information processing unit, 511 Recipe data acquisition unit, 512 Recipe data analysis unit, 513 Robot state estimation unit, 514 Flavor information processing unit, 515 Control unit, 516 Command output unit

Claims

1. A cooking arm that performs a cooking operation for cooking, Recipe data including a data set in which cooking operation data describing information on ingredients of the dish and information on the actions of the cook in the cooking process using the ingredients is linked with sensation data indicating the sensations of the cook measured in conjunction with the progress of the cooking process, the sensation data being data indicating at least any one of the flavor of the ingredients before cooking, the flavor of the ingredients after cooking cooked in the cooking process, and the flavor of the dish completed through all the cooking processes, and a control unit that controls the cooking operation performed by the cooking arm using the recipe data, A flavor measurement unit that acquires at least any one of the flavor of the ingredients cooked by the cooking operation performed by the cooking arm and the flavor of the dish completed by the cooking operation performed by the cooking arm Comprising: The control unit: Based on the specifications of the first sensor that measures at least any one of the flavor of the ingredients cooked by the cooking operation performed by the cooking arm and the flavor of the dish completed by the cooking operation performed by the cooking arm, correct the recipe data, Using the cooking operation data and the sensation data regarding the flavor adjustment performed by the cook after cooking and tasting, control the cooking operation performed by the cooking arm A cooking robot.

2. The sensation data includes taste information indicating at least any one of sweetness, sourness, saltiness, bitterness, umami, spiciness, and astringency The cooking robot according to claim 1.

3. The sensation data includes other information regarding taste obtained by using the taste information as an input to a model generated by performing deep learning, The cooking robot according to claim 2.

4. By using other information regarding taste included in the sensation data as an input to a model generated by performing deep learning based on the sensor data of the first sensor, correct the recipe data The cooking robot according to claim 3.

5. The recipe data includes information indicating the specifications of a second sensor that measures flavor in conjunction with the progress of the cooking process, The control unit corrects the recipe data based on the specifications of the first sensor and the specifications of the second sensor. The cooking robot according to claim 1.

6. The flavor measurement unit measures at least one of the taste that forms the flavor of the food material and the taste that forms the flavor of the dish. The control unit causes the cooking arm to perform a cooking operation for adjusting the taste so that the taste acquired by the flavor measurement unit matches the taste represented by the sensory data. The cooking robot according to claim 1.

7. The sensory data includes texture information indicating at least one of the texture of the food material and the texture of the dish. The flavor measurement unit measures at least one of the texture that forms the flavor of the food material and the texture that forms the flavor of the dish. The control unit causes the cooking arm to perform a cooking operation for adjusting the texture so that the texture acquired by the flavor measurement unit matches the texture represented by the sensory data. The cooking robot according to claim 1.

8. The texture information is information indicating at least one of stress, hardness, and moisture content measured by the first sensor. The cooking robot according to claim 7.

9. The recipe data includes cooking environment data indicating the environment of the cooking space measured in conjunction with the progress of the cooking process. The cooking robot further includes an environment control unit that controls the environment of the dining space so that the environment of the dining space where the meal of the dish completed by the cooking operation performed by the cooking arm is taken matches the environment of the cooking space represented by the cooking environment data. The cooking robot according to claim 1.

10. The cooking environment data is data indicating at least one of the temperature, humidity, air pressure, brightness, hue, and saturation of the cooking space. The cooking robot according to claim 9.

11. The recipe data includes food attribute information indicating at least one of the attributes of the food material before cooking, the attributes of the food material after cooking cooked in the cooking process, and the attributes of the dish completed after going through all the cooking processes. The cooking robot according to claim 1.

12. The cooking operation data describes the types and amounts of seasonings used in the cooking process. The recipe data includes seasoning attribute information indicating the attributes of the seasonings. The cooking robot according to claim 1.

13. The recipe data includes cook attribute information indicating the attributes of the cook. The cooking robot according to claim 1.

14. The control unit updates the cooking process according to the difference between the attributes of the cook indicated by the cook attribute information and the attributes of the user who eats the dish completed by the cooking operation performed by the cooking arm. The cooking robot according to claim 13.

15. The cooking robot further includes a situation recognition unit that recognizes the situation when the cooking arm performs a cooking operation. The control unit updates the cooking process according to the recognized situation. The cooking robot according to claim 1.

16. The control unit updates the cooking process according to the situation of the ingredients. The cooking robot according to claim 15.

17. The control unit controls the cooking arm according to an instruction command for instructing a cooking operation, which is generated based on the recipe data. The cooking robot according to claim 1.

18. The control unit causes the cooking arm to perform a cooking operation by coordinating a plurality of the cooking arms according to the instruction command. The cooking robot according to claim 17.

19. A cooking robot including a cooking arm that performs a cooking operation for cooking, using recipe data including a data set in which cooking operation data describing information on the ingredients of the dish and information on the actions of the cook in the cooking process using the ingredients is linked with sensation data indicating the sensation of the cook measured in conjunction with the progress of the cooking process, the sensation data being data indicating at least any one of the flavor of the ingredients before cooking, the flavor of the ingredients after cooking in the cooking process, and the flavor of the dish completed after all the cooking processes, to control the cooking operation performed by the cooking arm; acquiring at least any one of the flavor of the ingredients cooked by the cooking operation performed by the cooking arm and the flavor of the dish completed by the cooking operation performed by the cooking arm; including; Controlling the cooking operation performed by the cooking arm using the recipe data includes: correcting the recipe data based on the specifications of a first sensor that measures at least any one of the flavor of the ingredients cooked by the cooking operation performed by the cooking arm and the flavor of the dish completed by the cooking operation performed by the cooking arm; controlling the cooking operation performed by the cooking arm using the cooking operation data and the sensation data regarding the flavor adjustment performed by the cook after cooking and taste testing; including a control method.

20. Recipe data including a data set in which cooking operation data describing information on cooking ingredients and information on the actions of a cook in a cooking process using the ingredients is linked with sensory data indicating the sensations of the cook measured in conjunction with the progress of the cooking process, the sensory data being data indicating at least any one of the flavor of the ingredients before cooking, the flavor of the ingredients after cooking cooked in the cooking process, and the flavor of the dish completed after all the cooking processes, and a control unit that controls the cooking operations performed by a cooking arm provided in a cooking robot a flavor measurement unit that acquires at least any one of the flavor of the ingredients cooked by the cooking operations performed by the cooking arm and the flavor of the dish completed by the cooking operations performed by the cooking arm comprising the control unit corrects the recipe data based on the specifications of a first sensor that measures at least any one of the flavor of the ingredients cooked by the cooking operations performed by the cooking arm and the flavor of the dish completed by the cooking operations performed by the cooking arm controls the cooking operations performed by the cooking arm using the cooking operation data and the sensory data regarding the flavor adjustment performed by the cook after cooking and taste testing Cooking robot control device

21. A cooking robot control device using recipe data including a data set in which cooking operation data describing information on cooking ingredients and information on the actions of a cook in a cooking process using the ingredients is linked with sensory data indicating the sensations of the cook measured in conjunction with the progress of the cooking process, the sensory data being data indicating at least any one of the flavor of the ingredients before cooking, the flavor of the ingredients after cooking cooked in the cooking process, and the flavor of the dish completed after all the cooking processes, to control the cooking operations performed by a cooking arm provided in a cooking robot; and acquiring at least any one of the flavor of the ingredients cooked by the cooking operations performed by the cooking arm and the flavor of the dish completed by the cooking operations performed by the cooking arm including Controlling the cooking operations performed by the cooking arm using the recipe data includes Correcting the recipe data based on the specifications of a first sensor that measures at least one of the flavor of the food material cooked by the cooking operation performed by the cooking arm and the flavor of the dish completed by the cooking operation performed by the cooking arm; Controlling the cooking operation performed by the cooking arm by using the cooking operation data and the sensory data related to the flavor adjustment performed by the cook after cooking and tasting; A control method including the above.

Images