Information processing device, information processing method, cooking system, and cooking method

The information processing device and system generate new food plating arrangements by learning from chef's actions and subjective evaluations, enabling innovative presentations that can be executed by cooking robots, addressing the limitations of traditional plating methods.

JP7593318B2Active Publication Date: 2024-12-03SONY GROUP CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2021537230
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-08-08
Filing Date
2020-07-27
Publication Date
2024-12-03
Estimated Expiration
2040-07-27

AI Technical Summary

Technical Problem

The challenge in creating new food presentations is due to fixed concepts based on culinary common sense and culture, making it difficult for chefs to innovate beyond traditional plating methods.

Method used

An information processing device and system that generates new food plating arrangements by learning from time-series data of chef's plating actions, incorporating subjective evaluations, and generating new plating information using a predictive model, which can be executed by a cooking robot to reproduce the new presentations.

Benefits of technology

Enables chefs to create unique and innovative food presentations by providing new plating ideas and allows cooking robots to execute these presentations accurately, breaking away from conventional plating norms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593318000001
    Figure 0007593318000001
  • Figure 0007593318000002
    Figure 0007593318000002
  • Figure 0007593318000003
    Figure 0007593318000003
Patent Text Reader

Abstract

The present technology relates to an information processing device, an information processing method, a cooking robot, a cooking method, and a cooking instrument which make it possible to create a new plating for a meal. An information processing device of one aspect of the present technology generates new plating information, which is information pertaining to a new plating, on the basis of plating information including: ingredient information, which is information pertaining to the ingredients used in the plating for a meal; plating action information, which is information pertaining to the plating actions of a chef; and cooking tools information, which is information pertaining to the cooking tools used in the plating. The present technology can be applied to a computer provided in a kitchen.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present technology relates in particular to an information processing device and an information processing method that are capable of generating new presentations of food, Cooking systems, and How to cook By law Regarding. [Background technology]

[0002] In recent years, there has been an increasing demand for food presentation in order to improve appearance, for example, in services that share videos of cooking processes or photos of plated food.

[0003] In cooking, presentation is a way to express taste, appearance, story, etc. For example, restaurant chefs are constantly required to create new presentations. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2017-506169 [Patent Document 2] Special Publication No. 2017-536247 Summary of the Invention [Problem to be solved by the invention]

[0005] It is not easy to continue creating new presentations because of preconceived notions based on common sense, culture, or experience in the culinary world.

[0006] The present technology has been made in light of these circumstances, and makes it possible to generate new presentations of food. [Means for solving the problem]

[0007] The information processing device according to the first aspect of the present technology detects and processes food ingredients used in plating a dish, plating action information related to a chef's plating action, and cooking tool information related to a cooking tool used in plating the dish. Time series data The apparatus includes a presentation generating section that generates new presentation information, which is information regarding new presentation, based on the presentation information. The presentation generation unit generates the new presentation information including information regarding the presentation action at each time based on a model generated by learning based on the time series data of the presentation information, subjective information representing a person's subjective evaluation of the chef's presentation, and an image representing the chef's presentation results.

[0008] The second aspect of this technology Cooking System teeth, The system includes an information processing unit that generates new plating information, which is information regarding new plating, based on time-series data of plating information including ingredient information, which is information regarding ingredients used in plating the food, plating action information, which is information regarding the plating action of the chef, and cooking tool information, which is information regarding the cooking tools used in plating, and a cooking robot that performs the new plating action based on the new plating information. The information processing unit generates the new plating information, which includes information regarding the plating action at each time, based on a model generated by learning based on the time-series data of the plating information, subjective information representing a person's subjective evaluation of the chef's plating, and an image showing the plating result of the chef. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing an example of a Plating being presented by an information processing device according to an embodiment of the present technology; [Diagram 2] FIG. 13 is a diagram showing an example of a screen used to specify a plating subjective evaluation. [Diagram 3] FIG. 13 is a diagram showing an example of a screen used to specify a plating image. [Figure 4] FIG. 13 is a diagram showing an example of a sample image. [Diagram 5] FIG. 2 is a diagram illustrating an example of input / output to / from an information processing device. [Figure 6] FIG. 13 is a diagram showing how learning data is collected. [Figure 7] FIG. 11 is a diagram showing an example of a flow of generating cooking action information and plating action information. [Figure 8] 11 is a diagram showing an example of information constituting plating operation information. FIG. [Figure 9] FIG. 13 is a diagram showing a specific example of a plating operation. [Figure 10] FIG. 10 is a diagram showing a specific example of the Plating operation following FIG. 9. [Figure 11] FIG. 11 is a diagram showing a specific example of the Plating operation following FIG. 10. [Figure 12] FIG. 12 is a diagram showing a specific example of the Plating operation following FIG. [Figure 13]FIG. 13 is a diagram showing a specific example of the Plating operation following FIG. 12. [Figure 14] FIG. 14 is a diagram showing a specific example of the Plating operation following FIG. 13. [Figure 15] FIG. 13 is a diagram illustrating an example of a recipe DB. [Figure 16] FIG. 13 is a diagram showing an example of information used in learning a plating generation model. [Figure 17] FIG. 13 is a diagram illustrating an example of learning. [Figure 18] FIG. 13 is a diagram showing the state of the Plating generation model during inference. [Figure 19] FIG. 13 is a diagram showing an example of prediction using RNNPB. [Figure 20] FIG. 13 is a diagram showing an example of Plating action information S(t) output by the Plating generation model. [Figure 21] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Figure 22] 2 is a block diagram showing an example of a functional configuration of an information processing device; [Diagram 23] 11 is a flowchart illustrating processing of an information processing device. [Figure 24] FIG. 1 illustrates an example of the configuration of a network system. [Diagram 25] FIG. 1 is a diagram illustrating an example of the configuration of a control system. [Figure 26] FIG. 4 is a diagram showing an example of the description of recipe data; [Figure 27] FIG. 11 is a diagram showing an example of a flow of recreating a dish based on recipe data. [Figure 28] FIG. 1 is a diagram illustrating an example of an arrangement of data processing devices. [Figure 29] FIG. 2 is a perspective view showing the appearance of the cooking robot. [Diagram 30] FIG. 2 is an enlarged view of the cooking arm. [Diagram 31] FIG. 2 is a diagram showing the appearance of the cooking arm. [Diagram 32] 11A to 11C are diagrams showing examples of ranges of motion of each part of a cooking arm. [Diagram 33]FIG. 13 is a diagram showing an example of a connection between a cooking arm and a controller. [Diagram 34] FIG. 2 is a block diagram showing a configuration example of a cooking robot. [Diagram 35] 2 is a block diagram showing an example of a functional configuration of a data processing device. FIG. [Diagram 36] 11 is a flowchart illustrating processing of the data processing device. [Figure 37] FIG. 13 is a diagram illustrating another example of the configuration of the control system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <Overview of this technology> This technology generates a new plating of food and presents it to a cook, known as a chef, itamae, cook, etc.

[0011] Plating (Food Plating) refers to the presentation of food. A dish is completed by plating cooked ingredients on it. Here, a new plating means a plating that is different from the plating prepared on the plating generation side. The new plating includes platings that differ from the plating prepared on the generation side in at least one of the elements that make up the plating of food, such as platings with different ingredients used for plating, platings with different plating positions, platings with different colors, platings with different tableware used for plating, and platings with a different plating order.

[0012] Ingredients used for plating include not only meat and fish that have been cooked or heated, but also vegetables cut with a knife, herbs, tomatoes, and other ingredients themselves. Seasonings such as salt and pepper, and liquids such as sauces made by mixing multiple seasonings are also included in the ingredients used for plating.

[0013] That is, the food ingredients used for plating include all the elements that make up the final dish.

[0014] Note that a dish means the finished product obtained through cooking. Cooking means the process of making a dish or the act (operation) of making a dish.

[0015] In addition, this technology generates a new Plating for a dish and reproduces it in a cooking robot. The cooking robot cooks according to the description of the recipe data prepared for each dish, and completes the dish by plating the food ingredients produced by the cooking.

[0016] Hereinafter, the embodiments for implementing this technology will be described. The description will be carried out in the following order. 1. Presentation of Plating 2. Learning of the Plating generation model 3. Configuration and operation of each device 4. Example of control of the cooking robot 5. Other examples

[0017] <Presentation of Plating> FIG. 1 is a diagram showing an example of the presentation of Plating by an information processing apparatus according to an embodiment of this technology.

[0018] The situation shown in the upper part of FIG. 1 is a situation where a chef who is cooking is considering the Plating of a dish. In the example of FIG. 1, the information processing apparatus 1, which is a tablet terminal, is placed next to the chef who is cooking. The information processing apparatus 1 has a function of presenting a new Plating in response to a request from the chef.

[0019] For example, as shown in the speech bubble #1, assume that the chef desires a Plating with a "bright and gentle image". In this example, the presentation of the Plating is performed in response to specifying a Plating subjective evaluation, which is a subjective evaluation of the appearance of the Plating such as "bright and gentle image".

[0020] FIG. 2 is a diagram showing an example of a screen used to specify a Plating subjective evaluation.

[0021] The Plating subjective evaluation designation screen shown in Fig. 2 is displayed on the display of the information processing device 1 when the chef performs a predetermined operation. In the example of Fig. 2, a radar chart 11 is displayed approximately in the center of the designation screen. The radar chart 11 is a radar chart used to designate the Plating subjective evaluation.

[0022] The radar chart 11 is a radar chart with nine elements, namely brightness, gentleness, roughness, simplicity, vividness, size, temperature, weather, and season, as its axes. In this example, the Plating subjective evaluation is expressed by nine elements, such as brightness, gentleness, and roughness. For example, the chef specifies the value of each element by directly touching the position of each element on the radar chart 11 with his / her finger.

[0023] The Plating subjective evaluation may be designated by voice instead of by using a radar chart as shown in Fig. 2. When the Plating subjective evaluation is designated by voice, the information processing device 1 performs voice recognition, language analysis, etc., to identify the meaning of the chef's remarks.

[0024] The subjective evaluation of plating may not be specified by specifying a value on a radar chart, but the value of each element may be directly input by operating a keyboard or the like.

[0025] Along with the specification of such a plating subjective evaluation, for example, a plating image, which is an image of plating that the chef imagines, is specified.

[0026] FIG. 3 is a diagram showing an example of a screen used to specify a Plating image.

[0027] As shown in Fig. 3, a plurality of sample images serving as Plating examples are displayed on the Plating image designation screen. In the example of Fig. 3, three sample images P1 to P3 are displayed. More sample images can be displayed on the designation screen by scrolling or the like.

[0028] By selecting a preferred sample image, the chef can specify a plating image that represents a plating close to what he or she has in mind.

[0029] FIG. 4 is a diagram showing an example of a sample image.

[0030] As shown in the speech bubble in Figure 4, attribute information is set for each sample image. The attribute information includes keywords that describe the characteristics of the plating and subjective evaluation information for the plating. The subjective evaluation included in the attribute information is the subjective evaluation of the plating by the chef himself and many third parties when they viewed the plating in the sample image.

[0031] In this way, when a plating is presented, the chef selects a subjective evaluation of the plating and an image of the plating.

[0032] Returning to the explanation of FIG. 1, when the plating desired by the chef is specified as described above, the information processing device 1 generates a plating according to the chef's request and presents it to the chef as shown in speech bubbles #2 and #3.

[0033] In the example of Figure 1, the plating is presented by displaying an image of the plating. Also, the way to realize the plating is presented to the chef, such as "Pour strawberry sauce into the sauce dispenser..." In this way, the plating is presented to the chef using voice and screen display.

[0034] FIG. 5 is a diagram showing an example of input / output to the information processing apparatus 1.

[0035] As shown in FIG. 5, a Plating generation model is prepared in the information processing apparatus 1. The Plating generation model is a prediction model that takes a Plating subjective evaluation and a Plating image as inputs and outputs a new Plating. The Plating generation model is generated, for example, by machine learning such as deep learning and is prepared in the information processing apparatus 1.

[0036] The chef can receive the presentation of the Plating generated using such a Plating generation model, perform Plating as presented, or obtain hints therefrom and consider a new Plating. The information processing apparatus 1 can be called a Plating generator that presents the new Plating itself or information serving as a hint for the new Plating to the chef.

[0037] Note that the Plating generation model as shown in FIG. 5 may not be prepared in the information processing apparatus 1 but may be prepared in a server on the Internet. In this case, a new Plating is generated in the server on the Internet, and the information of the generated Plating is presented to the chef by the information processing apparatus 1.

[0038] <Learning of the Plating Generation Model> ·Collection of Learning Data FIG. 6 is a diagram showing the state of collection of learning data.

[0039] As shown in FIG. 6, various devices for measuring the actions of the chef are provided around the kitchen where the chef cooks. In the example of FIG. 6, cameras 41-1 and 41-2 facing the angle of view of the chef who is cooking are installed. Based on the images captured by the cameras 41-1 and 41-2, the actions of the chef are analyzed. The images captured by the cameras 41-1 and 41-2 may be moving images or still images.

[0040] Various sensors are attached to the chef's body, such as an acceleration sensor, a position sensor, a temperature sensor, a distance sensor, etc. The analysis of the chef's movements is also based on the sensor data measured by these sensors.

[0041] The chef's actions are analyzed by a learning server 51 that is connected to the chef's device via the Internet. The learning server 51 receives information transmitted from each sensor, including the cameras 41-1 and 41-2, and analyzes the chef's actions.

[0042] The chefs whose actions are analyzed include not only the chefs who receive the Plating, but also many other chefs. The Plating generation model is trained based on the actions of various chefs.

[0043] The measurement of the chef's movements continues until the dish is completed. A single dish may go through multiple cooking steps, and the movements of each step are analyzed.

[0044] FIG. 7 is a diagram showing an example of a flow of generating cooking action information and plating action information.

[0045] In the example of FIG. 7, cooking steps #1 to #N are performed, and the ingredients prepared by these steps are plated in a plating step to complete a certain dish.

[0046] In this case, each action performed in the cooking steps #1 to #N is measured (sensed), and cooking action information #1 to #N, which is information about the actions in each cooking step, is generated based on the measurement results.

[0047] Furthermore, the operation performed in the plating process is measured, and plating operation information, which is information on the operation in the plating process, is generated based on the measurement results.

[0048] Here, the cooking operation information includes, for example, ingredient information and operation information. The ingredient information is information about ingredients used by the chef in the cooking process. The information about ingredients includes information about the type of ingredient, the amount of ingredient, the size of ingredient, etc.

[0049] For example, if a chef uses carrots in a certain cooking step, the ingredient information includes information indicating that carrots were used. Information indicating various foods, such as water and seasonings, that the chef used as ingredients in the dish is also included in the ingredient information. Foods are various things that people can eat.

[0050] The ingredients used by the chefs are recognized, for example, by analyzing images of the chefs cooking captured by cameras 41-1 and 41-2. Ingredient information is generated based on the results of ingredient recognition.

[0051] The motion information is information about the movements of the chef in the cooking process. The information about the movements of the chef includes information about the types of cooking tools used by the chef, the body movements of the chef at each time including hand movements, and the standing position of the chef at each time.

[0052] For example, when a chef uses a knife to cut an ingredient, the action information includes information indicating that the knife was used as a cooking tool, the cutting position, the number of cuts, the force, angle, speed, etc.

[0053] In addition, when a chef uses a ladle to stir a pot containing liquid as an ingredient, the action information includes information indicating that the ladle was used as a cooking tool, as well as information indicating the amount of force, angle, speed, and time used for stirring.

[0054] Plating operation information FIG. 8 is a diagram showing an example of information constituting the Plating operation information.

[0055] As shown in FIG. 8, the plating action information includes information describing the ingredients used in the plating, information regarding the chef's movements during the plating process, information regarding the tools used in the plating, information regarding the placement of the ingredients, and information regarding the tableware used in the plating.

[0056] The plating operation information is made up of the above information at each time in the plating process. The plating operation information is time-series data that represents ingredients, movements, tools, placement, and tableware.

[0057] Specific examples of the Plating operation information will be described with reference to FIGS.

[0058] FIG. 9 is a diagram showing an example of the operation at the start time of the plating process.

[0059] Assume that the operation shown in the left side of Fig. 9 is performed at time t0, which is the start time of the plating process. In this example, the sauce dispenser T1 filled with strawberry sauce is held in the right hand, the tip of the dispenser is pressed against a specific position on the plate, and a small amount of sauce is pushed out in this state. This operation is photographed by cameras 41-1, 41-2, etc.

[0060] In this case, as indicated by the tip of the white arrow, it is specified that strawberry sauce was used as an ingredient in the plating.

[0061] Further, as the chef's actions, actions such as holding the sauce dispenser T1 with the right hand and moving the right hand holding the sauce dispenser T1 to a reference position on the plate are identified.

[0062] It is identified that the tool used for plating was a source dispenser T1.

[0063] The arrangement of the plating is determined by the coordinates representing the two-dimensional position of the tableware and the layer representing the three-dimensional position.

[0064] It is identified that the dish used for plating was a circular flat dish with a specific ID.

[0065] FIG. 10 is a diagram showing an example of the operation at time t1.

[0066] Assume that at time t1, a predetermined time after time t0, the operation shown on the left side of Fig. 10 is performed. In this example, the strawberry sauce is extruded in dots and arranged in a ring around the center of the plate.

[0067] At time t1 when the fourth dot is arranged, as shown by the tip of the white arrow, the ingredients, tools, and tableware remain unchanged, but the chef's actions include shifting the position of his right hand, etc. Also, the position of the fourth dot is specified as the arrangement of Plating.

[0068] FIG. 11 is a diagram showing an example of the operation at time t2.

[0069] At time t2, a predetermined time after time t1, the operation shown on the left side of Fig. 11 is performed. In this example, the operation of arranging dots of strawberry sauce is completed, and then the tool is changed to brush T2 and pressed against a certain dot.

[0070] In this case, as shown by the white arrow, the ingredients and tableware remain the same, but the chef's actions are identified as grasping the brush and moving the right hand holding the brush to the position of the dot at the reference position on the plate. It is also identified that brush T2 was used as the tool used for plating. The position of the first dot is identified as the placement of plating.

[0071] FIG. 12 is a diagram showing an example of the operation at time t3.

[0072] At time t3, a predetermined time after time t2, the action shown on the left side of Fig. 12 is performed. In this example, the action of spreading each dot of strawberry sauce toward the center of the plate with brush T2 is performed.

[0073] At time t3 when the first strawberry sauce dot is stretched, as shown by the white arrow, the chef's action is to slide his right hand toward the center of the plate while the ingredients, tools, and tableware remain unchanged. In addition, the position of the Plating after the slide is specified.

[0074] FIG. 13 is a diagram showing an example of the operation at time t4.

[0075] At time t4, a predetermined time after time t3, the action shown on the left side of Fig. 13 is performed. In this example, the action of spreading the strawberry sauce is completed, and then the action of switching the tool to the turner T3 and placing the cooked steak meat in the center of the plate is performed.

[0076] In this case, as shown by the white arrow, it is specified that steak meat was used as the ingredient used in Plating. In addition, the chef's actions are specified as moving the right hand holding the turner T3 with the steak meat on it to the center of the plate, lowering the turner T3, etc.

[0077] In addition, it is identified that Turner T3 was used as the tool used for Plating. The position of Plating is identified as the center of the plate. The position where the steak meat was placed is identified as Layer 2 because it is the layer above Layer 1 where the strawberry sauce was placed.

[0078] FIG. 14 is a diagram showing an example of the operation at time t5.

[0079] At time t5, a predetermined time after time t4, the operation shown on the left side of Fig. 14 is performed. In this example, the operation of placing herbs next to the steak meat is performed, and the plating process is then completed.

[0080] In this case, as shown by the white arrow, it is specified that herbs were used as an ingredient in the plating, and the action of placing the herbs next to the steak is specified as the action of the chef.

[0081] In addition, the tool used for Plating is identified as not using a tool, and the position of the steak meat placed in the center of the plate is identified as the placement of the Plating.

[0082] The plating action information, which is made up of time-series data representing the ingredients, movements, tools, arrangement, and tableware related to plating as described above, is generated in the learning server 51 based on images captured during the plating process.

[0083] In addition, the plating action information is linked to cooking action information generated based on the cooking step before the plating step, and cooking data for each dish is generated.

[0084] FIG. 15 is a diagram illustrating an example of a recipe DB.

[0085] 15, information linking cooking action information and plating action information is accumulated as cooking data for each dish in the dish DB managed by the learning server 51. The plating action information included in the cooking data for each dish is used for learning the plating generation model.

[0086] Since new platings are generated based on a plating generation model generated by learning using such plating motion information, the new platings are generated based on plating motion information that includes information on ingredients, motions, tools, placement, tableware, etc.

[0087] About learning FIG. 16 is a diagram showing an example of information used for learning the plating generation model.

[0088] As shown in FIG. 16, plating action information S(t), subjective evaluation information E(T), and plating result information V(T) are used for training the plating generation model.

[0089] The Plating action information S(t) is time-series data of the Plating action information described above.

[0090] The subjective evaluation information E(T) is information that represents the subjective evaluation of the plating by a person who has seen the finished dish after the plating process is completed. Time T represents the time when the plating process is completed.

[0091] The plating result information V(T) is an image of the finished dish after the plating process is completed.

[0092] In the learning server 51, the plating action information S(t), the subjective evaluation information E(T), and the plating result information V(T) of each dish are linked and managed. As described above, the plating generation model is a prediction model that takes the plating subjective evaluation and the plating image as input and outputs a new plating.

[0093] Since the new Plating is the result of the Plating action information S(t), it is sufficient that the relationship between the subjective evaluation information E(T) and the Plating result information V(T) and the Plating action information S(t) has been learned.

[0094] FIG. 17 is a diagram illustrating an example of learning.

[0095] 17, Plating action information S(t), subjective assessment information E(T), and Plating result information V(T) are input to a learning device 61 that learns a Plating generation model. The learning device 61 is provided in the learning server 51.

[0096] Since the Plating action represented by the Plating action information S(t) results in the Plating represented by the Plating result information V(T), the learning device 61 learns parameters constituting a neural network that outputs the Plating action information S(t) when the Plating result information V(T) is input.

[0097] Furthermore, since the subjective evaluation information E(T) is a subjective evaluation of the Plating represented by the Plating result information V(T), the relationship between the subjective evaluation information E(T) and the Plating result information V(T) is learned.

[0098] When subjective evaluation information E(T) and plating result information V(T) of a certain dish are input, a neural network that outputs plating action information S(t) for that dish, as shown in Figure 18, is trained as a plating generation model.

[0099] Here, when the subjective evaluation information E(T) is changed and the changed subjective evaluation information E(T) and the Plating result information V(T) are input, the Plating generation model outputs Plating action information S(t) that has been at least partially changed.

[0100] The Plating realized by the Plating operation information S(t) with at least a part changed is a Plating different from the Plating represented by the Plating result information V(T), that is, a new Plating.

[0101] The Plating subjective evaluation designated by the chef using the screen as described with reference to Figure 2 corresponds to the changed subjective evaluation information E(T) referred to here. Also, the Plating image designated by the chef using the screen as shown in Figure 3 corresponds to the Plating result information V(T).

[0102] When the plating subjective evaluation and plating image specified by the chef are input, the plating generation model outputs plating action information S(t) for realizing a plating different from the plating represented by the plating image.

[0103] For example, RNNPB (Recurrent Neural Network with Parametric Bias) is used as a plating generation model. RNNPB is a neural network that can learn / predict nonlinear time series data by having recurrent connections. In addition, by inputting a PB value corresponding to a certain pattern, it is possible to output a desired pattern.

[0104] FIG. 19 is a diagram showing an example of prediction using the RNNPB.

[0105] As shown in Figure 19, the RNNPB that constitutes the Plating generation model is composed of a lower hierarchy that outputs Plating movement information at time t+1 when Plating movement information at time t is input, and an upper hierarchy that outputs a value to be used as the PB value (PBt) of the lower hierarchy when a subjective evaluation and a PB value (PB(t')) corresponding to the Plating result information V(T) are input.

[0106] In the example of Fig. 19, brightness 5, gentleness 3, roughness 1, simplicity 2, ... are input to RNNPB as the Plating subjective evaluation specified by the chef. In addition, the PB value corresponding to a certain Plating image is input to RNNPB.

[0107] Here, the Plating subjective evaluations specified by the chef, which are brightness 5, gentleness 3, roughness 1, simplicity 2, ..., are different from the subjective evaluations (Fig. 4) set in the sample image selected as the Plating image. As a result, the Plating operation information S(t) output from the RNNPB becomes Plating operation information S(t) for realizing a Plating different from the Plating represented by the Plating image.

[0108] The neural network that constitutes the Plating generation model is not limited to RNNPB. The Plating generation model can be constructed by various neural networks that are generated by learning the relationship between the subjective evaluation information E(T) and the Plating result information V(T) and the Plating action information S(t).

[0109] FIG. 20 is a diagram showing an example of the Plating action information S(t) outputted from the Plating generation model.

[0110] As shown in Figure 20, the plating motion information S(t) corresponds to the plating motion information in Figure 8, which serves as learning data, and is time-series data on which ingredients to plate, what motion to use to plate them, what tool to use to plate them, where to place them, and what kind of dish to serve them on.

[0111] Presenting this information output by the plating generation model will result in presenting a new plating.

[0112] <Configuration and operation of each device> Configuration of information processing device 1 FIG. 21 is a block diagram showing an example of the hardware configuration of the information processing device 1. As shown in FIG.

[0113] 21, the information processing device 1 is configured with a computer such as a tablet terminal, etc. A CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103 are connected to each other by a bus 104.

[0114] An input / output interface 105 is further connected to the bus 104. An input unit 106 including a keyboard, a mouse, etc., and an output unit 107 including a display, a speaker, etc. are connected to the input / output interface 105.

[0115] Furthermore, to the input / output interface 105, a storage unit 108 formed of a hard disk or non-volatile memory, a communication unit 109 formed of a network interface, and a drive 110 for driving a removable medium 111 are connected.

[0116] The CPU 101 loads a program stored in the storage unit 108 into the RAM 103 via the input / output interface 105 and the bus 104 and executes the program, thereby performing various processes such as generating a new plating.

[0117] FIG. 22 is a block diagram showing an example of a functional configuration of the information processing device 1.

[0118] At least some of the functional units shown in FIG. 22 are realized by the CPU 101 in FIG. 21 executing a predetermined program.

[0119] 22, an information processing unit 151 is realized in the information processing device 1. The information processing unit 151 includes an acquisition unit 161, a Plating generation unit 162, and a presentation unit 163.

[0120] The acquisition unit 161 acquires the plating subjective evaluation and plating image designated by the chef. The information acquired by the acquisition unit 161 is supplied to the plating generation unit 162.

[0121] The plating generation unit 162 has a plating generation model. The plating generation unit 162 inputs the plating subjective evaluation and the plating image supplied from the acquisition unit 161 to the plating generation model, and outputs plating action information S(t).

[0122] The Plating generation unit 162 outputs information representing the content of the Plating action information S(t) as new Plating information to the presentation unit 163. An image representing an image of Plating realized based on the Plating action information S(t) output from the Plating generation model may be generated in the Plating generation unit 162 and supplied to the presentation unit 163.

[0123] The presentation unit 163 presents the new plating information supplied from the plating generation unit 162. The plating may be presented by voice from a speaker or by screen display on a display. For example, the presentation unit 163 presents to the chef an explanation of each operation for realizing the new plating in order.

[0124] Operation of the information processing device 1 The process of the information processing device 1 for generating a new Plating will be described with reference to the flowchart of FIG.

[0125] In step S1, the acquisition unit 161 acquires a subjective evaluation of the plating imagined by the chef.

[0126] In step S2, the acquisition unit 161 acquires a plating image that represents an image of the chef's plating.

[0127] In step S3, the plating generation unit 162 inputs the plating subjective evaluation and the plating image designated by the chef to the plating generation model, and outputs plating action information S(t).

[0128] In step S4, the presentation unit 163 presents the new plating information supplied from the plating generation unit 162 to the chef.

[0129] Through the above process, the information processing device 1 can generate a new plating and present it to the chef. The information processing device 1 presents a plating not by selecting a plating that meets the conditions from a plurality of platings prepared in advance, but by generating a plating according to the conditions specified by the chef each time.

[0130] Chefs can receive suggestions for plating that match their image, and then either try plating according to the suggestions or use them as inspiration to create a new plating.

[0131] In the above, the chef specifies the plating subjective evaluation and the plating image, but instead of the plating image, a keyword related to the plating may be specified.

[0132] When a keyword is specified, a sample image that includes the keyword in its attribute information is selected and used as the Plating image. As described with reference to FIG. 4, the keyword is set in the attribute information of each sample image.

[0133] Furthermore, when the similarity between sample images is set, a sample image similar to the sample image selected by the chef may be used as the Plating image.

[0134] This may result in a Plating being generated and presented that was not intended by the chef.

[0135] The chef may also be allowed to modify the sample image. The modification of the sample image is performed, for example, by clicking on the part to be modified and performing an on-screen operation to change the shape or color. When the sample image is modified, the modified sample image is used as the Plating image, and Plating is generated.

[0136] This allows a chef to specify a plating image that matches his / her image, even if there is no sample image that expresses his / her image, and have plating generated.

[0137] Instead of looking at the sample images displayed on the screen and specifying the sample image to be used as the Plating image, the Plating image may be specified by drawing an illustration by hand, etc. When an illustration is drawn by hand, a sample image that is similar to the illustration is searched for, and the sample image is used as the Plating image.

[0138] Once a Plating is presented, it may be modified according to the chef's utterance. For example, if the chef utters something like "brighter," "more spring-like," etc., the Plating subjective evaluation is modified according to the content of the chef's utterance, and the Plating is generated using the modified Plating subjective evaluation.

[0139] In addition, it may be possible to specify that the ingredients of a plating that has already been presented be changed. For example, when a plating using vanilla sauce is presented and a request is made to change it to a plating using strawberry sauce, a sample image of a plating using strawberry sauce that is similar to the previously specified sample image is searched for, and the sample image is used as the plating image.

[0140] In this way, various methods can be used for the subjective evaluation of Plating and the designation of Plating images.

[0141] Network system configuration FIG. 24 is a diagram illustrating an example of the configuration of a network system.

[0142] 24 shows a configuration in which a new plating is generated in a plating generation server 171 on the Internet. The plating generation server 171 is provided with the same configuration as the information processing unit 151 shown in FIG.

[0143] Communication is performed via the Internet between the plating generation server 171 and the information processing device 1 provided on the chef's side. Information representing the plating subjective evaluation and plating image designated by the chef is transmitted from the information processing device 1 to the plating generation server 171.

[0144] The acquisition unit 161 of the plating generation server 171 receives the plating subjective evaluation and the plating image transmitted from the information processing device 1.

[0145] The plating generation unit 162 generates a plating based on the plating subjective evaluation and the plating image transmitted from the information processing device 1 as described above.

[0146] The presentation unit 163 transmits the plating information generated by the plating generation unit 162 to the information processing device 1, and causes it to be presented to the chef.

[0147] In this way, it is possible for a new plating to be generated in the plating generation server 171 on the Internet.

[0148] <Example of controlling a cooking robot> Control system configuration Although the above describes the generation of platings for human chefs, platings for cooking robots may be generated. In this case, plating actions according to the newly generated plating are performed by the cooking robot.

[0149] FIG. 25 is a diagram illustrating an example of the configuration of a control system.

[0150] As shown in Fig. 25, the control system is composed of a data processing device 301 and a cooking robot 302. The cooking robot 302 is a robot equipped with a driving system such as a cooking arm, various sensors, and a function for cooking. The cooking robot 302 is installed, for example, in a home.

[0151] The data processing device 301 is a device that controls the cooking robot 302. The data processing device 301 is configured with a computer or the like.

[0152] As shown on the left side of FIG. 25, the data processing device 301 controls the cooking robot 302 based on recipe data prepared for each dish. The recipe data describes information on each cooking step and plating step. The recipe data supplied to the data processing device 301 is generated in the plating generation unit 162 (FIG. 22) by, for example, linking information on cooking steps input from outside with information on plating steps generated by the data processing device 301. The plating generation unit 162 also functions as a recipe data generation unit that generates recipe data used to control the cooking robot 302.

[0153] The data processing device 301 controls the cooking robot 302 based on the recipe data to make the cooking robot 302 cook the food. Recipe data including plating information generated by the information processing unit 151 in FIG. 22 is supplied to the data processing device 301 and used to control the cooking robot 302.

[0154] For example, when recipe data is input as indicated by an arrow A1, the data processing device 301 controls the cooking operation of the cooking robot 302 by outputting an instruction command based on the description of the recipe data as indicated by an arrow A2.

[0155] The cooking robot 302 drives each part, such as the cooking arm, according to the command supplied from the data processing device 301, and performs the operation of each cooking process. The cooking robot 302 also drives each part, such as the cooking arm, according to the command supplied from the data processing device 301, and performs the operation of the plating process. The command includes information for controlling the torque, driving direction, and driving amount of the motor provided in the cooking arm.

[0156] Until the food is cooked, the data processing device 301 sequentially outputs commands to the cooking robot 302. The cooking robot 302 performs actions according to the commands, and the food is finally cooked.

[0157] FIG. 26 is a diagram showing an example of the description contents of recipe data.

[0158] As shown in Fig. 26, one recipe data is composed of multiple cooking step data sets. In the example of Fig. 26, a cooking step data set for cooking step #1, a cooking step data set for cooking step #2, ..., a cooking step data set for cooking step #N are included.

[0159] Each cooking step data set includes cooking action information, which is information about cooking actions for implementing a cooking step. For example, one cooking step data set is composed of time-series data of cooking action information for implementing one cooking step.

[0160] The cooking operation information includes ingredient information and operation information.

[0161] The ingredient information is information about ingredients used in the cooking process, including information about the type, amount, and size of the ingredient.

[0162] Ingredients include not only ingredients that have not been cooked at all, but also cooked (prepared) ingredients obtained by a certain cooking process. The ingredient information included in the cooking operation information of a certain cooking process includes information on ingredients that have been through a previous cooking process.

[0163] The operation information is information about the movement of the cooking arm and the like in the cooking process. The information about the movement includes information about the type of cooking tool used in cooking.

[0164] For example, the action information for a cooking process of cutting an ingredient includes information indicating the use of a knife as a cooking tool, the cutting position, the number of cuts, the strength of the cut, the angle, the speed, and the like.

[0165] In addition, the action information for the cooking process of stirring a pot containing liquid as an ingredient includes information indicating the use of a ladle as a cooking tool, and information indicating the strength, angle, speed, time, etc. of stirring.

[0166] Operation information for a cooking process of baking certain ingredients using an oven includes information indicating that the oven is used as a cooking tool, the heat of the oven, baking time, and the like.

[0167] 26, one recipe data includes a plating process data set. The plating process data set includes plating operation information for realizing the plating process. For example, the plating process data set is configured by time series data of the plating operation information for realizing the plating process.

[0168] FIG. 27 is a diagram showing an example of a flow of recreating a dish based on recipe data.

[0169] 27, the cooking robot 302 reproduces a dish by repeating cooking for each cooking step based on cooking action information for each time included in the cooking step data set described in the recipe data. Through multiple cooking steps, cooking steps #1 to #N, the ingredients to be used for plating are prepared.

[0170] After cooking process #N is finished, the plating operation is performed based on the plating operation information at each time included in the plating process data set, thereby completing the dish.

[0171] In the example of FIG. 27, the plating step is performed after all the cooking steps have been completed, but the plating step may be performed before other cooking steps as appropriate.

[0172] FIG. 28 is a diagram showing an example of the arrangement of the data processing device 301. As shown in FIG.

[0173] As shown in A of Fig. 28, the data processing device 301 is provided as, for example, an external device of the cooking robot 302. In the example of A of Fig. 28, the data processing device 301 and the cooking robot 302 are connected via a network such as the Internet.

[0174] The command transmitted from the data processing device 301 is received by the cooking robot 302 via the network. Various data, such as images captured by a camera of the cooking robot 302 and sensor data measured by a sensor provided in the cooking robot 302, are transmitted from the cooking robot 302 to the data processing device 301 via the network.

[0175] 28B, the data processing device 301 may be provided inside the housing of the cooking robot 302. In this case, the operation of each part of the cooking robot 302 is controlled according to the instruction commands generated by the data processing device 301.

[0176] The following description will be mainly based on the assumption that the data processing device 301 is provided as an external device to the cooking robot 302.

[0177] -External appearance of cooking robot FIG. 29 is a perspective view showing the appearance of the cooking robot 302.

[0178] 29, the cooking robot 302 is a kitchen-type robot having a horizontally elongated rectangular parallelepiped housing 311. Inside the housing 311, which is the main body of the cooking robot 302, various components are provided.

[0179] A cooking assistance system 312 is provided on the rear side of the housing 311. Each space formed in the cooking assistance system 312 by dividing it with thin plate-like members has a function to assist cooking by the cooking arms 321-1 to 321-4, such as a refrigerator, an oven range, and storage.

[0180] A rail is provided in the longitudinal direction of the top plate 311A, and cooking arms 321-1 to 321-4 are provided on the rail. The cooking arms 321-1 to 321-4 are capable of changing their positions along the rail, which serves as a moving mechanism.

[0181] The cooking arms 321-1 to 321-4 are robot arms formed by connecting cylindrical members at joints. Various tasks related to cooking are performed by the cooking arms 321-1 to 321-4.

[0182] The space above the top plate 311A ​​is a cooking space where cooking is performed by the cooking arms 321-1 to 321-4.

[0183] 29 shows four cooking arms, the number of cooking arms is not limited to four. Hereinafter, when there is no need to distinguish between the cooking arms 321-1 to 321-4, they will be collectively referred to as cooking arm 321.

[0184] FIG. 30 is an enlarged view of the cooking arm 321.

[0185] 30, attachments with various cooking functions are attached to the tip of the cooking arm 321. As attachments for the cooking arm 321, various attachments are prepared, such as an attachment with a manipulator function (hand function) for grasping ingredients or tableware, and an attachment with a knife function for cutting ingredients.

[0186] In the example of Fig. 30, knife attachment 331-1, which is an attachment having a knife function, is attached to cooking arm 321-1. A chunk of meat placed on top plate 311A ​​is cut using knife attachment 331-1.

[0187] A spindle attachment 331-2, which is an attachment used to fix and rotate ingredients, is attached to the cooking arm 321-2.

[0188] A peeler attachment 331-3 that functions as a peeler for peeling food ingredients is attached to the cooking arm 321-3.

[0189] The skin of the potato, which is lifted by the cooking arm 321-2 using the spindle attachment 331-2, is peeled by the cooking arm 321-3 using the peeler attachment 331-3. In this way, it is possible for multiple cooking arms 321 to work together to perform one task.

[0190] A manipulator attachment 331-4 having a manipulator function is attached to the cooking arm 321-4. Using the manipulator attachment 331-4, the frying pan with the chicken on it is carried to the space of the cooking assistance system 312 having an oven function.

[0191] The cooking and plating operations using the cooking arm 321 can be performed by appropriately changing the attachments depending on the contents of the work. It is also possible to attach the same attachment to multiple cooking arms 321, such as attaching the manipulator attachment 331-4 to each of the four cooking arms 321.

[0192] The cooking and plating operations of the cooking robot 302 are performed not only using the above-mentioned attachments prepared as tools for the cooking arm, but also using the same tools as those used by humans for cooking, as appropriate. For example, a knife used by a human is grasped by the manipulator attachment 331-4, and cooking such as cutting ingredients is performed using the knife.

[0193] Cooking arm configuration FIG. 31 is a diagram showing the appearance of the cooking arm 321.

[0194] As shown in Fig. 31, the cooking arm 321 is generally made up of thin cylindrical members connected by hinges that serve as joints. Each hinge is provided with a motor or the like that generates a force to drive each member.

[0195] The cylindrical member includes, in order from the tip, a detachable member 351, a relay member 353, and a base member 355.

[0196] The detachable member 351 and the relay member 353 are connected by a hinge portion 352 , and the relay member 353 and the base member 355 are connected by a hinge portion 354 .

[0197] A detachable portion 351A to which an attachment is attached and detached is provided at the tip of the detachable member 351. The detachable member 351 has the detachable portion 351A to which various attachments are attached and detached, and functions as a cooking function arm portion that performs cooking by operating the attachment.

[0198] A detachable part 356 that is attached to a rail is provided at the rear end of the base member 355. The base member 355 functions as a moving function arm part that realizes the movement of the cooking arm 321.

[0199] FIG. 32 is a diagram showing an example of the range of motion of each part of the cooking arm 321.

[0200] As shown by the ellipse #1, the detachable member 351 can rotate about the central axis of the circular cross section. The small flat circle shown in the center of the ellipse #1 indicates the direction of the axis of rotation of the dashed line.

[0201] As shown by the circle #2, the detachable member 351 is rotatable about an axis passing through a fitting portion 351B with the hinge portion 352. The relay member 353 is rotatable about an axis passing through a fitting portion 353A with the hinge portion 352.

[0202] The two small circles shown inside circle #2 indicate the directions of the respective rotation axes (perpendicular to the paper surface). The movable range of detachable member 351 about the axis passing through fitting portion 351B and the movable range of relay member 353 about the axis passing through fitting portion 353A are each, for example, a range of 90 degrees.

[0203] The relay member 353 is composed of a leading end member 353-1 and a trailing end member 353-2, which are separated from each other. As shown by the ellipse #3, the relay member 353 is rotatable about the central axis of the circular cross section at the connecting portion 353B between the members 353-1 and 353-2. The other movable parts basically have the same range of motion.

[0204] In this manner, the detachable member 351 having the detachable part 351A at its tip, the relay member 353 connecting the detachable member 351 and the base member 355, and the base member 355 having the detachable part 356 connected at its rear end are rotatably connected by hinges. The movement of each movable part is controlled by a controller in the cooking robot 302 according to an instruction command.

[0205] FIG. 33 is a diagram showing an example of the connection between the cooking arm and the controller.

[0206] As shown in Fig. 33, the cooking arm 321 and the controller 361 are connected via wires in a space 311B formed inside the housing 311. In the example of Fig. 33, the cooking arms 321-1 to 321-4 and the controller 361 are connected via wires 362-1 to 362-4, respectively. The flexible wires 362-1 to 362-4 bend appropriately depending on the positions of the cooking arms 321-1 to 321-4.

[0207] Cooking Robot 302 Configuration FIG. 34 is a block diagram showing an example of the configuration of the cooking robot 302.

[0208] The cooking robot 302 is configured by connecting each unit to a controller 361 (FIG. 33) that serves as a control device for controlling the operation of the cooking robot 302. Among the components shown in FIG. 34, the same components as those described above are denoted by the same reference numerals. Duplicate descriptions will be omitted as appropriate.

[0209] In addition to the cooking arm 321, a camera 401, a sensor 402, and a communication unit 403 are connected to the controller 361.

[0210] The controller 361 is configured by a computer having a CPU, a ROM, a RAM, a flash memory, etc. The controller 361 executes a predetermined program by the CPU and controls the overall operation of the cooking robot 302. The controller 361 may configure the data processing device 301.

[0211] For example, the controller 361 controls the communication unit 403 to transmit an image captured by the camera 401 and sensor data measured by the sensor 402 to the data processing device 301 .

[0212] In the controller 361, a command acquisition unit 411 and an arm control unit 412 are realized by executing a predetermined program.

[0213] The command acquisition unit 411 acquires a command transmitted from the data processing device 301 and received by the communication unit 403. The command acquired by the command acquisition unit 411 is supplied to an arm control unit 412.

[0214] The arm control unit 412 controls the operation of the cooking arm 321 according to the command acquired by the command acquisition unit 411 .

[0215] The camera 401 captures the surroundings of the cooking robot 302 and outputs the captured images to the controller 361. The camera 401 is provided at various positions, such as in front of the cooking assistance system 312 and at the tip of the cooking arm 321.

[0216] The sensor 402 is composed of various sensors such as a temperature and humidity sensor, a pressure sensor, a light sensor, a distance sensor, a human sensor, a positioning sensor, a vibration sensor, etc. Measurement by the sensor 402 is performed at a predetermined cycle. Sensor data indicating the measurement results by the sensor 402 is supplied to the controller 361.

[0217] The camera 401 and the sensor 402 may be provided at a position away from the housing 311 of the cooking robot 302.

[0218] The communication unit 403 is a wireless communication module such as a wireless LAN module, a mobile communication module compatible with LTE (Long Term Evolution), etc. The communication unit 403 communicates with the data processing device 301 and external devices such as a server on the Internet.

[0219] As shown in FIG. 34, the cooking arm 321 is provided with a motor 421 and a sensor 422.

[0220] The motor 421 is provided at each joint of the cooking arm 321. The motor 421 rotates around its axis under the control of the arm control unit 412. An encoder that measures the amount of rotation of the motor 421, a driver that adaptively controls the rotation of the motor 421 based on the measurement result by the encoder, and the like are also provided at each joint.

[0221] The sensor 422 is composed of, for example, a gyro sensor, an acceleration sensor, a touch sensor, etc. The sensor 422 measures the angular velocity, acceleration, etc. of each joint during the operation of the cooking arm 321, and outputs information indicating the measurement results to the controller 361. Sensor data indicating the measurement results of the sensor 422 are also transmitted from the cooking robot 302 to the data processing device 301 as appropriate.

[0222] Configuration of data processing device 301 FIG. 35 is a block diagram showing an example of the functional configuration of the data processing device 301.

[0223] At least some of the functional units shown in FIG. 35 are realized by the CPU of the computer constituting the data processing device 301 executing a predetermined program.

[0224] 35, a command generation unit 431 is realized in the data processing device 301. The command generation unit 431 is composed of a recipe data acquisition unit 451, a robot state estimation unit 452, a control unit 453, and a command output unit 454.

[0225] The recipe data acquisition unit 451 acquires recipe data newly generated in the information processing device 1 or the like, and outputs it to the control unit 453. The recipe data acquisition unit 451 may include an information processing unit 151 (FIG. 22) having a function of generating the entire recipe including plating.

[0226] The robot state estimation unit 452 receives images and sensor data transmitted from the cooking robot 302. The cooking robot 302 transmits images taken by its own camera and sensor data measured by sensors provided at predetermined positions of the cooking robot 302 at a predetermined cycle. The images taken by the cooking robot 302's camera show the surroundings of the cooking robot 302.

[0227] The robot state estimation unit 452 estimates the state of the surroundings of the cooking robot 302, such as the state of the cooking arm 321 and the state of ingredients, and the state of the cooking process by analyzing the images and sensor data transmitted from the cooking robot 302. Information indicating the state of the surroundings of the cooking robot 302 estimated by the robot state estimation unit 452 is supplied to the control unit 453.

[0228] The control unit 453 generates an instruction command for controlling the cooking robot 302 based on the cooking process data set and the plating process data set described in the recipe data supplied from the recipe data acquisition unit 451. For example, an instruction command for causing the cooking arm 321 to perform an operation represented by the cooking operation information included in the cooking process data set is generated.

[0229] The command is generated based on the surrounding state of the cooking robot 302 estimated by the robot state estimation unit 452. The command generated by the control unit 453 is supplied to the command output unit 454.

[0230] The command output unit 454 transmits the instruction command generated by the control unit 453 to the cooking robot 302.

[0231] Operation of the data processing device 301 The processing of the data processing device 301 that controls the operation of the cooking robot 302 will be described with reference to the flowchart in FIG.

[0232] In step S101, the recipe data acquisition unit 451 acquires recipe data representing a recipe generated by the information processing device 1 or the like.

[0233] In step S102, the control unit 453 selects a cooking operation based on the cooking process data set described in the recipe data, and generates an instruction command for performing the selected cooking operation. For example, the cooking process data set is selected in the order of cooking operations, and the cooking operations included in the selected cooking operations are selected in the order of execution.

[0234] In step S103, the command output unit 454 transmits an instruction command to the cooking robot 302 to cause it to perform a cooking operation.

[0235] In step S104, the robot state estimation unit 452 estimates the state of the cooking robot 302.

[0236] In step S105, the control unit 453 determines whether or not all cooking operations have been completed. If it is determined in step S105 that all cooking operations have not been completed, the process returns to step S102, the next cooking operation is selected, and the above processing is repeated.

[0237] If it is determined in step S105 that all cooking operations have been completed, in step S106, the control unit 453 generates an instruction command for performing the plating operation based on the plating process data set described in the recipe data.

[0238] In step S107, the command output unit 454 transmits an instruction command to the cooking robot 302 to cause it to perform the plating action.

[0239] In step S108, the robot state estimation unit 452 estimates the state of the cooking robot 302.

[0240] In step S109, the control unit 453 determines whether or not the plating operation has been completed. If it is determined in step S109 that the plating operation has not been completed, the process returns to step S106, and the above processes are repeated.

[0241] If it is determined in step S109 that the plating operation has ended, the process ends. At this point, the dish is completed based on new recipe data generated by the information processing device 1 or the like.

[0242] In this way, it is possible for the information processing device 1 to generate recipe data for controlling a robot that uses a cooking arm to cook.

[0243] FIG. 37 is a diagram illustrating another example of the configuration of the control system.

[0244] In the control system shown in Fig. 37, an electronic cooking appliance 303 such as a microwave oven is provided instead of the cooking robot 302. The electronic cooking appliance 303 performs cooking and plating operations according to instructions supplied from the data processing device 301 to cook food. The electronic cooking appliance 303 with a heating function performs a plating operation, for example, by heating ingredients such as chocolate to melt it on the food.

[0245] In this way, recipe data can be used to control various devices that automatically perform cooking and plating operations.

[0246] <Other examples> Although both the cooking process and the plating process are performed by the cooking robot 302 in the above embodiment, the cooking process may be performed by a chef and only the plating process may be performed by the cooking robot 302. In this case, only information related to the plating process is described in the recipe data.

[0247] Computer configuration example The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0248] The program to be installed is provided by being recorded on removable medium 111 shown in FIG. 21, which is made up of an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or a semiconductor memory. Alternatively, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be installed in advance in ROM 102 or storage unit 108.

[0249] The program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0250] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0251] The effects described herein are merely examples and are not limiting, and other effects may also be present.

[0252] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0253] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0254] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0255] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices. [Explanation of symbols]

[0256] 1 Information processing device, 51 Learning server, 61 Learning device, 151 Information processing unit, 161 Acquisition unit, 162 Plating generation unit, 163 Presentation unit, 171 Plating generation server, 301 Data processing device, 302 Cooking robot

Claims

1. a presentation generation unit that generates new presentation information, which is information regarding new presentation, based on time-series data of presentation information including food ingredient information, which is information regarding food ingredients used in presentation of a dish, presentation action information, which is information regarding the presentation action of a chef, and cooking tool information, which is information regarding cooking tools used in presentation of a dish; The plating generation unit generates the new plating information including information about plating actions at each time based on a model generated by performing learning based on the time-series data of the plating information, subjective information representing a person's subjective evaluation of the plating by the chef, and an image representing the plating result by the chef. Information processing device.

2. The arrangement generating unit generates the new arrangement information based on the arrangement information further including arrangement information that is information regarding the arrangement of the plated ingredients. The information processing device according to claim 1 .

3. The arrangement generating unit generates the new arrangement information based on the arrangement information further including tableware information that is information regarding tableware used for arrangement. The information processing device according to claim 1 .

4. The method further includes a recipe generating unit that generates recipe data by linking cooking process information, which is information about a cooking process for preparing ingredients to be used in the new presentation, with the new presentation information. The information processing device according to claim 1 .

5. The cooking robot further includes a command generating unit that generates a command for causing the cooking robot to perform the cooking process and the new plating process based on the recipe data of a predetermined dish. The information processing device according to claim 4.

6. At least one of the ingredient information, the plating action information, and the cooking tool information is information obtained by analyzing an image obtained by photographing the chef plating the food, or by analyzing sensor data that measures the chef's plating action. The information processing device according to claim 1 .

7. The model is a neural network that receives as input a person's subjective evaluation of the presentation and an image showing a predetermined presentation result, and outputs time-series data regarding the new presentation. The information processing device according to claim 1 .

8. The apparatus further includes an acquisition unit that acquires the subjective evaluation and an image representing the predetermined arrangement result, which are inputs to the model, based on an input by a user requesting a new arrangement. The information processing device according to claim 7.

9. an information processing device that generates new presentation information, which is information regarding new presentation, based on time-series data of presentation information including food ingredient information, which is information regarding food ingredients used in presentation of a dish, presentation action information, which is information regarding a chef's presentation action, and cooking tool information, which is information regarding a cooking tool used in presentation; and generating the new plating information, which includes information regarding plating actions at each time, based on a model generated by performing learning based on the time-series data of the plating information, subjective information representing a person's subjective evaluation of the plating by the chef, and an image representing the plating result by the chef. Information processing methods.

10. An information processing unit that generates new presentation information, which is information regarding new presentation, based on time-series data of presentation information including ingredient information, which is information regarding ingredients used in presentation of a dish, presentation action information, which is information regarding the chef's presentation action, and cooking tool information, which is information regarding the cooking tools used in presentation; a cooking robot that performs the new plating operation based on the new plating information; and Equipped with The information processing unit generates the new plating information including information about plating actions at each time based on a model generated by performing learning based on the time-series data of the plating information, subjective information representing a person's subjective evaluation of the plating by the chef, and an image representing the plating result by the chef. Cooking system.

11. The information processing unit generates the new presentation information based on the presentation information further including arrangement information that is information regarding the arrangement of the plated ingredients. The cooking system of claim 10.

12. The information processing unit generates the new presentation information based on the presentation information further including tableware information that is information regarding tableware used for presentation. The cooking system of claim 10.

13. The information processing unit includes a recipe generating unit that generates recipe data by linking cooking process information, which is information about a cooking process when preparing ingredients to be used in the new presentation, with the new presentation information; The cooking robot controls the cooking process operation and the new plating operation based on the recipe data. The cooking system of claim 10.

14. The cooking robot includes an instruction command generation unit that generates an instruction command for causing the cooking robot to perform the cooking process and the new plating process based on the recipe data. The cooking system of claim 13.

15. An information processing device that generates new presentation information, which is information regarding new presentation, based on time-series data of presentation information including food ingredient information, which is information regarding food ingredients used in presentation of a dish, presentation action information, which is information regarding the chef's presentation action, and cooking tool information, which is information regarding the cooking tools used in presentation; Cooking robot and A cooking system comprising: generating the new plating information including information about plating actions at each time point based on a model generated by learning based on the time-series data of the plating information, subjective information representing a person's subjective evaluation of the chef's plating, and an image representing the chef's plating result; performing the new plating operation based on the new plating information; Cooking methods including.

Citation Information

Patent Citations

  • Method and system for food preparation in a robotic cooking kitchen

    JP2017506169A

  • Robot Manipulation Method and System for Executing Domain Specific Applications in Instrumented Environments Using Electronic Small Manipulation Libraries

    JP2017536247A

  • Food cooking device

    JP2019120485A

  • Analytics-based design and planning of creative menus

    US20150161912A1

  • Techniques for modifying recipes to reduce preparation times and / or incorporate preferred ingredients

    US20160179935A1