Data processing device and data processing method
A data processing device uses image and text data from recipes to autonomously control a cooking robot, addressing the inconvenience of manual programming and enhancing cooking efficiency.
Patent Information
- Application Number
- CN201980071475.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-11-05
- Filing Date
- 2019-10-24
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing cooking robots require manual programming to execute cooking tasks, which is inconvenient and inefficient.
A data processing device generates instruction commands for a robot based on image and text data from recipe information, allowing the robot to autonomously perform cooking tasks without manual programming.
Enables the robot to follow recipe instructions accurately, reducing the need for human intervention and facilitating the preparation of dishes using existing recipe data.
Smart Images

Figure CN113168604B_ABST
Abstract
Description
Technical Field
[0001] The present technology relates to a data processing device and a data processing method, and more particularly to a data processing device and a data processing method for enabling control of a robot based on instruction data including image data and text data.
[0002] CROSS-REFERENCE TO RELATED APPLICATIONS
[0003] This application claims the benefit of Japanese Priority Patent Application JP 2018-208192 filed on Nov. 5, 2018, the entire contents of which are incorporated herein by reference. Background Art
[0004] There is a service that displays instruction data including recipes of dishes to the public. Users can search for favorite recipes by searching for recipes by category or entering the name of an ingredient as a keyword. In the recipe, various types of information are described, such as a photo of the finished dish, ingredients, and cooking process.
[0005] Of course, the instructions provided by such a service are ready for people to see. Even in the case of instructions that only describe a rough process, people can estimate the detailed process based on past experience and knowledge of other dishes, etc., and can cook.
[0006] By the way, with the development of artificial intelligence (AI) and sensor technology, cooking robots that automatically perform cooking processes have attracted attention. Some cooking robots use two arms to perform cooking processes in the same way as humans.
[0007] Reference List
[0008] Patent Literature
[0009] Patent Document 1: JP 2017-506169A Summary of the invention
[0010] Technical issues
[0011] It would be convenient if the robot could be driven based on instructions prepared for human viewing. There is no need for a human to prepare a program for driving an arm or the like as described in instruction data such as recipe data.
[0012] The present technology has been made in view of the foregoing, and enables control of a robot based on instruction data including image data and text data.
[0013] Solution to the problem
[0014] According to one aspect of the present technology, a data processing device includes: a command generation unit configured to generate an instruction command, the instruction command being used to give instructions for one or more operations to be performed during a process performed by a robot provided with at least one arm, wherein the instruction command is generated based on instruction data, the instruction data including: image data, the image data being obtained by taking one or more images of a situation during or after the process, and text data, the text data indicating at least one of an object to be used in the process or an operation to be performed during the process.
[0015] According to another aspect of the present technology, a data processing method includes: generating an instruction command by a data processing device, the instruction command being used to give instructions for one or more operations to be performed by a robot provided with at least one arm, wherein the instruction command is generated based on instruction data, the instruction data including: image data, the image data being obtained by taking one or more images of the situation during or after the processing, and text data, the text data indicating at least one of an object to be used in the processing or the operations to be performed during the processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] [ Figure 1 ] Figure 1 is a diagram showing a configuration example of a control system according to an embodiment of the present technology.
[0017] [ Figure 2 ] Figure 2 is a diagram showing an arrangement example of a data processing device.
[0018] [ Figure 3 ] Figure 3 is a perspective view showing the appearance of the cooking robot.
[0019] [ Figure 4 ] Figure 4 is a view showing an enlarged state of a cooking arm.
[0020] [ Figure 5 ] Figure 5 is a view showing the appearance of a cooking arm.
[0021] [ Figure 6 ] Figure 6 are views showing examples of movable ranges of respective parts of a cooking arm.
[0022] [ Figure 7 ] Figure 7 is a view showing an example of connection between a cooking arm and a controller.
[0023] [ Figure 8 ] Figure 8 is a diagram showing an example of recipe data.
[0024] [ Fig. 9 ] Fig. 9 is a diagram showing an example of recipe data.
[0025] [ Fig.10 ] Fig.10 It is a diagram showing the flow of the entire processing executed by the data processing device.
[0026] [ Fig.11 ] Fig.11 is a diagram showing an example of description of a state.
[0027] [ Fig.12 ] Fig.12 is a diagram showing a specific example of the description of the target state.
[0028] [ Fig.13 ] Fig.13 is a diagram showing an example of planning of a state sequence.
[0029] [ Fig.14 ] Fig.14 is a diagram showing an example of an entire cooking sequence.
[0030] [ Fig.15 ] Fig.15 is a diagram showing an example of state transition.
[0031] [ Fig.16 ] Fig.16 is a diagram showing an example of a database.
[0032] [ Fig.17 ] Fig.17 is a diagram showing an example of information to be stored in the raw material DB.
[0033] [ Fig.18 ] Fig.18 is a diagram showing a configuration example of a cooking method DB.
[0034] [ Fig.19 ] Fig.19 is a diagram showing an example of information to be stored in the state transition DB.
[0035] [ Fig. 20 ] Fig. 20 is a diagram showing an example of information to be stored in an action DB.
[0036] [ Fig.21 ] Fig.21 is a diagram showing an example of information to be stored in the provision method DB.
[0037] [ Fig. 22 ] Fig. 22 is a diagram showing an expression example of an action.
[0038] [ Fig.23 ] Fig.23 is a diagram showing state transitions.
[0039] [ Fig.24 ] Fig.24 It is a diagram showing the state of the operation.
[0040] [ Fig.25 ] Fig.25 is a block diagram showing a configuration example of hardware of a data processing device.
[0041] [ Fig.26 ] Fig.26 is a block diagram showing a functional configuration example of a data processing device.
[0042] [ Fig. 27 ] Fig. 27 is a block diagram showing a configuration example of a cooking robot.
[0043] [ Fig.28 ] Fig.28 is a flowchart for describing a process of controlling a data processing apparatus.
[0044] [ Fig.29 ] Fig.29 Yes Fig.28 A flowchart for describing the process of controlling a data processing device.
[0045] [ Fig.30 ] Fig.30 is a diagram showing an example of selection of an action.
[0046] [ Fig.31 ] Fig.31 : is a diagram showing an example of selection of action candidates.
[0047] [ Fig.32 ] Fig.32 is a diagram showing an example of evaluation of action candidates.
[0048] [ Fig.33 ] Fig.33 is a diagram showing an example of generation of an instruction command.
[0049] [ Fig.34 ] Fig.34 is a flowchart for describing a cooking process of a cooking robot.
[0050] [ Fig.35 ] Fig.35 is a diagram showing an example of processing of recipe data. DETAILED DESCRIPTION
[0051] Hereinafter, embodiments for implementing the present technology will be described. The description will be given in the following order.
[0052] 1. Cooking robot control system
[0053] 2. Planning and Actions of State Sequences
[0054] 3. Device Configuration and Operation
[0055] 4. Modifications
[0056] <<Control system of cooking robot>>
[0057] <System Configuration>
[0058] Figure 1 is a diagram showing a configuration example of a control system according to an embodiment of the present technology.
[0059] like Figure 1 As shown, the control system is configured by a data processing device 1 and a cooking robot 2. The cooking robot 2 is a robot including a driving system device such as a cooking arm and various sensors and having a cooking function. The cooking robot 2 is installed in a home, for example.
[0060] The data processing device 1 is a device that controls the cooking robot 2. The data processing device 1 is configured by a computer or the like.
[0061] like Figure 1 As shown on the left end of , the control of the cooking robot 2 by the data processing device 1 is performed based on the recipe data prepared for each dish. In the recipe data, a dish image (which is an image obtained by taking an image of a finished dish), the name of the dish, the name of the raw material, a sentence explaining the cooking process, etc. are described. The recipe data is also used by a cooking person by viewing a recipe displayed by the recipe data.
[0062] Note that a dish refers to the finished product after cooking. Cooking refers to the process of making a dish or the action (work) of making a dish.
[0063] The data processing device 1 controls the cooking robot 2 to prepare a dish based on recipe data, which is also used for cooking by a person.
[0064] As shown by arrow A1 , for example, when recipe data of a dish X is input, the data processing device 1 executes planning based on the description of the recipe data until the dish X is completed.
[0065] For example, planning is performed by setting a state transition from a starting state to a target state. The state of the uncooked raw material is set as the starting state, and the state of the finished dish X is set as the target state. When the state of the raw material changes due to the operation of the cooking robot 2, etc., the state transition occurs.
[0066] The data processing device 1 generates an instruction command for causing the cooking robot 2 to perform an operation for causing a transition from a current state to a next state, and outputs the instruction command to the cooking robot 2 as indicated by an arrow A2.
[0067] The cooking robot 2 drives each part such as a cooking arm to perform cooking-related operations according to instruction commands provided from the data processing device 1. The instruction commands include torque of a motor provided in the cooking arm, a driving direction, information for controlling a driving amount, and the like.
[0068] The data processing device 1 sequentially outputs instruction commands to the cooking robot 2 until the state reaches the target state. As the cooking robot 2 operates according to the instruction commands, the dish X is finally completed.
[0069] As described above, the data processing device 1 is used as a device that converts recipe data (which can be said to be incomplete information for the cooking robot 2) into instruction commands and controls the operation of the cooking robot 2. Since the recipe data is information prepared for a person who checks the cooking process, etc., the recipe data can be said to be incomplete information as information for controlling the cooking robot 2.
[0070] Since the cooking robot 2 can be controlled based on the recipe data for the cook, the administrator of the control system, for example, does not need to prepare a program for controlling the cooking robot 2 for each dish.
[0071] By being prepared as a service for presenting recipes to the public, recipe data for cooking personnel exists in large quantities. The user can make the cooking robot 2 prepare a selected dish from among many dishes. For example, according to the user's selection, the recipe data is acquired from a server providing a service for presenting recipes to the public, and the recipe data is input to the data processing device 1.
[0072] Figure 2 is a diagram showing an arrangement example of the data processing device 1 .
[0073] like Figure 2 As shown in FIG. 1A , the data processing device 1 is provided as a device external to the cooking robot 2, for example. Figure 2 In the example of A, a data processing device 1 and a cooking robot 2 are connected via a network 3 such as the Internet.
[0074] The cooking robot 2 receives instruction commands transmitted from the data processing device 1 via the network 3. Images captured by the camera of the cooking robot 2 and various data such as sensor data measured by sensors provided in the cooking robot 2 are transmitted from the cooking robot 2 to the data processing device 1 via the network 3.
[0075] A plurality of cooking robots 2 may be connected to one data processing device 1 instead of one cooking robot 2 being connected to one data processing device 1 .
[0076] like Figure 2 As shown in FIG. 1B , the data processing device 1 may be provided in the housing of the cooking robot 2 . In this case, the operation of each part of the cooking robot 2 is controlled according to the instruction command generated by the data processing device 1 .
[0077] Hereinafter, a description will be given on the assumption that the data processing device 1 is provided as a device external to the cooking robot 2 .
[0078] <Structure of the cooking robot>
[0079] Figure 3 : is a perspective view showing the appearance of the cooking robot 2 .
[0080] like Figure 3 As shown, the cooking robot 2 is a kitchen type robot having a rectangular parallelepiped housing 11. Various configurations are provided inside the housing 11 as a main body of the cooking robot 2.
[0081] The cooking assistance system 12 is provided at the rear side of the housing 11 so as to stand upright from the upper surface of the housing 11. The space divided by the thin plate-like member in the cooking assistance system 12 has a function for assisting the cooking arms 21-1 to 21-4 in cooking, such as a refrigerator, an oven, and a storage room.
[0082] A rail is provided on the top plate 11A along the length direction, and cooking arms 21-1 to 21-4 are provided on the rail. The cooking arms 21-1 to 21-4 can change positions along the rail as a moving mechanism.
[0083] The cooking arms 21-1 to 21-4 are robot arms constructed by connecting cylindrical members with joints. Various operations related to cooking are performed by the cooking arms 21-1 to 21-4.
[0084] The space above the top plate 11A is a cooking space where the cooking arms 21-1 to 21-4 perform cooking. The cooking space includes not only a space that is completely identical to the space where the cooking arms 21-1 to 21-4 perform cooking, but also a partial space included in such a space.
[0085] Although in Figure 3 Four cooking arms are shown in FIG. 1 , but the number of cooking arms is not limited to four. Hereinafter, in the case where the cooking arms 21 - 1 to 21 - 4 do not need to be distinguished, the cooking arms 21 - 1 to 21 - 4 are collectively referred to as cooking arms 21 .
[0086] Figure 4 is a view showing an enlarged state of the cooking arm 21 .
[0087] like Figure 4 As shown, accessories having various cooking functions are attached to the distal end of the cooking arm 21. As accessories of the cooking arm 21, various accessories are prepared, such as an accessory having a manipulator function (hand function) for gripping raw materials, tableware, etc., and an accessory having a knife function for cutting raw materials.
[0088] exist Figure 4 In the example of FIG. 1 , a knife attachment 31 - 1 as an attachment having a knife function is attached to the cooking arm 21 - 1 . The knife attachment 31 - 1 is used to cut a piece of meat placed on the top plate 11A.
[0089] A spindle attachment 31 - 2 , which is an attachment for fixing a material or rotating a material, is attached to the cooking arm 21 - 2 .
[0090] A peeler attachment 31 - 3 , which is an attachment having a peeler function to peel ingredients, is attached to the cooking arm 21 - 3 .
[0091] Potatoes lifted by cooking arm 21-2 using spindle attachment 31-2 are peeled by cooking arm 21-3 using peeler attachment 31-3. As described above, a plurality of cooking arms 21 can perform one job in cooperation with each other.
[0092] A robot attachment 31-4 as an attachment having a robot function is attached to the cooking arm 21-4. The frying pan with the chicken is carried to the space of the cooking assistance system 12 having an oven function using the robot attachment 31-4.
[0093] Such cooking by the cooking arm 21 is performed by appropriately replacing the attachment according to the content of the work. The replacement of the attachment is automatically performed by the cooking robot 2, for example.
[0094] It is also possible to attach the same accessory to multiple cooking arms 21 , such as attaching the robot accessory 31 - 4 to each of the four cooking arms 21 .
[0095] The cooking robot 2 not only uses the above-prepared attachments as tools of the cooking arm to perform cooking, but also uses the same tools as those used by the cooking personnel to properly perform cooking. For example, a knife used by a person is clamped by the manipulator attachment 31-4, and the knife is used to perform cooking, such as cutting raw materials.
[0096] <Structure of cooking arm>
[0097] Figure 5 2 is a view showing the appearance of the cooking arm 21 .
[0098] like Figure 5 As shown, the cooking arm 21 is generally constructed by connecting thin cylindrical members with hinge portions serving as joints. Each hinge portion is provided with a motor or the like that generates a force for driving each member.
[0099] As cylindrical members, an attaching / detaching member 51, a relay member 53, and a base member 55 are provided in order from the distal end. The attaching / detaching member 51 is a member having a length of about 1 / 5 of the length of the relay member 53. The combined length of the length of the attaching / detaching member 51 and the length of the relay member 53 is substantially the same as the length of the base member 55.
[0100] The attaching / detaching member 51 and the relay member 53 are connected by a hinge portion 52, and the relay member 53 and the base member 55 are connected by a hinge portion 54. The hinge portion 52 and the hinge portion 54 are provided at both ends of the relay member 53.
[0101] In this example, the cooking arm 21 is configured by three cylindrical members. However, the cooking arm 21 may be configured by four or more cylindrical members. In this case, a plurality of relay members 53 are provided.
[0102] An attaching / detaching portion 51A to attach or detach an accessory is provided at a distal end of the attaching / detaching member 51. The attaching / detaching member 51 includes the attaching / detaching portion 51A to attach or detach various accessories, and serves as a cooking function arm unit for cooking by operating the accessories.
[0103] An attaching / detaching portion 56 to be mounted to a rail is provided at a rear end of the base member 55. The base member 55 serves as a moving functional arm unit that enables movement of the cooking arm 21.
[0104] Figure 6 2 is a view showing an example of a movable range of each part of the cooking arm 21.
[0105] As shown in oval #1, the attaching / detaching member 51 is rotatable about the central axis of the circular cross section. The small flat circle shown in the center of oval #1 indicates the direction of the rotation axis of the alternate long and short dashed lines.
[0106] As shown by circle #2, the attaching / detaching member 51 is rotatable about an axis passing through the fitting portion 51B with the hinge portion 52. Furthermore, the relay member 53 is rotatable about an axis passing through the fitting portion 53A with the hinge portion 52.
[0107] The two small circles shown in circle #2 indicate the directions of the respective rotation axes (in the direction perpendicular to the paper surface). The movable range of the attachment / detachment member 51 centered on the axis passing through the fitting portion 51B and the movable range of the relay member 53 centered on the axis passing through the fitting portion 53A are, for example, a range of 90 degrees.
[0108] The relay member 53 is configured to be divided into a member 53-1 on the distal end side and a member 53-2 on the rear end side. As shown by an ellipse #3, the relay member 53 is rotatable about the central axis of the circular cross section at a connection portion 53B between the members 53-1 and 53-2.
[0109] The other movable parts basically have similar movable ranges.
[0110] In other words, as shown by circle #4, the relay member 53 is rotatable about an axis passing through the fitting portion 53C with the hinge portion 54. Furthermore, the base member 55 is rotatable about an axis passing through the fitting portion 55A with the hinge portion 54.
[0111] The base member 55 is configured to be divided by a member 55-1 on the distal end side and a member 55-2 on the rear end side. As shown by an ellipse #5, the base member 55 is rotatable about the central axis of the circular cross section at a connection portion 55B between the members 55-1 and 55-2.
[0112] As indicated by circle #6, the base member 55 is rotatable about an axis passing through the fitting portion 55C with the attaching / detaching portion 56 .
[0113] As shown by oval #7, the attaching / detaching portion 56 is mounted to the rail so as to be rotatable about the central axis of the circular cross section.
[0114] Therefore, the attachment / detachment member 51 having the attachment / detachment portion 51A at the distal end, the relay component 53 connecting the attachment / detachment member 51 and the base member 55, and the base member 55 having the attachment / detachment portion 56 connected at the rear end are respectively connected to be rotatable through the hinge portion. The movement of each movable portion is controlled by the controller in the cooking robot 2 according to the instruction command.
[0115] Figure 7 is a view showing an example of connection between a cooking arm and a controller.
[0116] like Figure 7 As shown, the cooking arm 21 and the controller 61 are connected via a wire in a space 11B formed inside the housing 11. Figure 7 In the example of FIG. 1 , the cooking arms 21 - 1 to 21 - 4 and the controller 61 are connected via wires 62 - 1 to 62 - 4 , respectively. The wires 62 - 1 to 62 - 4 having flexibility are appropriately bent according to the positions of the cooking arms 21 - 1 to 21 - 4 .
[0117] As described above, the cooking robot 2 is a robot capable of performing various works related to cooking by driving the cooking arm 21 .
[0118] <Recipe Data>
[0119] Figure 8 and Fig. 9 is a diagram showing an example of recipe data.
[0120] Figure 8 and Fig. 9 The recipe data is the recipe data of "delicious salad". Figure 8 As shown, the recipe data of “delicious salad” includes a dish image P1 (which is a dish image obtained by taking an image of a finished “delicious salad”) and text data indicating ingredients.
[0121] exist Figure 8 In the example of , as text data indicating ingredients for 3 to 4 people, text data "1 / 2 piece of lettuce, 1 piece of onion, 1 piece of avocado, etc." is included. The recipe data of "delicious salad" also includes text data "delicious salad" indicating the name of the dish.
[0122] Recipe data usually includes Fig. 9 Shown are text data explaining a rough cooking process, and an image obtained by capturing an image of the cooking situation.
[0123] exist Fig. 9 In the example of , as information about the first step of work, text data "Cut tomatoes and lettuce into bite-sized pieces by hand. Open a can of tuna." and image P11 are included. Image P11 is an image obtained by capturing an image of the situation after the first step of work.
[0124] Furthermore, as information on the second step work, text data "Slice the cucumber and cut the onion into rings. Cut the boiled egg into bite-sized pieces" and image P12 are included. Image P12 is an image obtained by capturing an image of the situation after the second step work.
[0125] As information about the third step work, text data "Put all the ingredients on the plate and add dressing to the salad to your liking" and image P13 are included. Image P13 is an image obtained by taking an image of the completed "delicious salad" as the situation after the third step work. Image P13 and dish image P1 are the same image.
[0126] As described above, the recipe data used by the data processing device 1 to control the operation of the cooking robot 2 includes at least image data and text data. The image included in the recipe data may be a still image or a moving image.
[0127] Fig. 9 The instructions for the cooking process may or may not be included in the recipe data. Fig. 9 In the case of cooking instructions, based on Figure 8 The illustrated dish image P1 and text data indicating ingredients control the operation of the cooking robot 2 .
[0128] Hereinafter, a description will be given of a process based on recipe data of "delicious salad" etc. Processes based on recipe data of other dishes are also similarly performed.
[0129] <<Planning and Actions of State Sequences>>
[0130] <Planning based on recipe data>
[0131] Fig.10 2 is a diagram showing the flow of the entire process executed by the data processing device 1 .
[0132] In the case where recipe data of “delicious salad” is input, as indicated by the points of arrows A11 and A12 , the data processing device 1 sets a target state based on the dish image and text data included in the recipe data.
[0133] For example, the ingredients required for "delicious salad" are estimated based on the text data. In addition, the state of each ingredient such as lettuce, onion, and avocado is estimated by analyzing the dish image P1 of "delicious salad". The state of each ingredient includes a cooking state, such as what kind of cooking is performed on the ingredient, and an arrangement state, such as how the cooked ingredient is served. The state of "delicious salad" appearing in the dish image P1 is set as a target state.
[0134] Fig.11 is a diagram showing an example of a description of a state including a target state.
[0135] like Fig.11 As shown, each state set by the data processing device 1 is represented as a discrete state by a combination of an ID as an identifier of the state and a parameter set.
[0136] Each state is appropriately represented by a combination of an ID and vector data, or a combination of an ID, a parameter set, and vector data.
[0137] Fig.12 is a diagram showing a specific example of the description of the target state.
[0138] A predetermined ID is assigned to the target state S which is the target state of "delicious salad" G .
[0139] like Fig.12As shown, indicating the target state S G The parameter set includes parameters about tableware for supply. The parameters about tableware include parameters indicating the type of tableware and parameters indicating the number of tableware pieces.
[0140] The type and number of tableware are estimated by analyzing, for example, the dish image P1. For example, a parameter indicating "platter" is set as the type of tableware and "1" as the number of tableware.
[0141] Indicates the target state S G The parameter set includes parameters about tomatoes as raw materials. The parameters about tomatoes include parameters indicating how to cut tomatoes, parameters indicating the number of tomato pieces after cutting, and parameters indicating the direction / arrangement of the cut tomatoes.
[0142] It is estimated that the ingredients include tomatoes based on, for example, the text data of the ingredients. It is estimated how to cut the tomatoes, the number of tomato pieces after cutting, and the direction / arrangement of the cut tomatoes based on the text data explaining the cooking process or by analyzing the dish image P1. Fig.12 In the example, the parameter indicating the number of tomato pieces after cutting is set to "6".
[0143] Indicates the target state S G The parameter set includes parameters about eggs as raw materials. The parameters about eggs include parameters indicating the state of eggs after cooking, parameters indicating how to cut eggs, parameters indicating the number of egg pieces after cutting, and parameters indicating the direction / arrangement of eggs after cutting.
[0144] The inclusion of eggs in the ingredients is estimated based on, for example, text data of the ingredients. The state of the eggs after cooking, how to cut the eggs, the number of egg pieces after cutting, and the direction / arrangement of the eggs after cutting are estimated based on text data explaining the cooking process or by analyzing the dish image P1. Fig.12 In the example, set the parameters that represent the state of the egg after cooking as "hard-boiled" and how to cut the egg as "half-cut".
[0145] Similarly, indicating the target state S G The parameter set includes parameters for toppings, parameters for seasoning, and parameters for temperature.
[0146] The goal state S of "delicious salad" G It is represented by a set of such parameters estimated from the recipe data. Each state from the start state to the goal state is also configured by a combination of an ID and a set of parameters.
[0147] Back to Fig.10 description, and sets the target state S represented by this description GThereafter, as indicated by the point of arrow A13, planning of a state sequence from the start state to the target state is performed.
[0148] As indicated by arrow A14, planning of the state sequence is performed based on the text data included in the recipe data. To plan the state sequence, the image data of the dish image P1 and the image included in the explanation of the cooking process may be used appropriately.
[0149] In the case where the planning of the state sequence is completed, as indicated by the point of arrow A15, control of the cooking robot 2 is performed. The description of the recipe data is also appropriately used to control the cooking robot 2, as indicated by arrow A16.
[0150] The control for causing the cooking robot 2 to execute an action for causing transitions between states constituting the state sequence is repeated, and finally, the state reaches the state set as the target state S G The state of "delicious salad" is shown as the arrow A17. As shown by the arrow A18, the change of the transition sequence is appropriately performed until the state reaches the target state S G .
[0151] Fig.13 is a diagram showing an example of planning of a state sequence from a start state to a target state.
[0152] The state S0 shown in the upper right is the starting state. Fig.13 In the example of , the parameter set of state S0 is configured by a parameter representing "a whole" of "tomato", a parameter representing "three leaves" of "lettuce", a parameter representing "a can" of "tuna", and the like.
[0153] The parameter set indicating the state S0 is set based on, for example, the text data of the raw material. The preparation of the raw material may be performed based on the text data of the raw material, and the state of the raw material indicating the preparation may be set as the starting state.
[0154] State S1 shown at the point of arrow A21 is the next state of state S0. The parameter set of state S1 is configured by parameters indicating the state of "six pieces (wedge-shaped)" of "six" "tomatoes" arranged in "plate 1", parameters indicating an opened "can of tuna", and the like.
[0155] The parameter set representing the state S1 is set based on information on the first step work and text data of ingredients included in the description of the cooking process as indicated by the arrow A31, for example. Fig.13 The cooking process shown on the left is referenced Fig. 9 The cooking process is the same as described.
[0156] As described above, the recipe data of “delicious salad” includes, as information on the first step work, text data “Cut tomatoes and lettuce into bite-sized pieces by hand. Open a can of tuna.” and an image P11 showing the situation after the work.
[0157] By analyzing this information, it is possible to estimate how to cut tomatoes, how to store canned tuna, etc., and set the parameter set of state S1. For example, the state where six tomatoes obtained by cutting tomatoes into six pieces are arranged on a small plate is captured in image P11. In addition, the state where a can of tuna is opened is captured in image P11.
[0158] State S2 shown at the point of arrow A22 is the next state of state S1. The parameter set of state S2 is configured by parameters indicating the state of "four pieces", "half-cut", "half-cooked", "eggs" arranged on "plate 3", parameters indicating the state of "cucumbers" cut into "3 mm" thickness arranged on "plate 4", etc.
[0159] The parameter set indicating the state S2 is set based on information on the first step work and the second step work and text data of the ingredients included in the description of the cooking process, for example, as indicated by arrow A32.
[0160] As described above, the recipe data of “delicious salad” includes, as information on the second step work, text data “slice the cucumber, and cut the onion into rings. Cut the boiled egg into bite-sized pieces” and an image P12 showing the situation after the work.
[0161] By analyzing this information, it is possible to estimate how to cut a cucumber, how to preserve an egg, etc., and the parameter set for state S2 can be set.
[0162] The next state of state S2 is the target state S G The target state S is set based on the dish image P1 of "delicious salad" and the text data of the ingredients included in the recipe data and all the information about the cooking process shown by the arrow A33. G Parameter set.
[0163] As described above, planning of a state sequence from a starting state to a target state is performed by, for example, converting the description of each step of the cooking process described in the recipe data into a state expression and connecting each state in sequence. With state S0 as the starting state, the state can reach the target state S as the target state through state transitions via states S1 and S2. G of "delicious salad".
[0164] Hereinafter, the entire state sequence from the starting state to the target state appropriately set based on the recipe data is referred to as the entire cooking sequence.
[0165] Fig.14 is a diagram showing an example of an entire cooking sequence.
[0166] like Fig.14 As shown in the figure, the entire cooking sequence set on the recipe data of "delicious salad" consists of state S0, state S1, state S2, target state S G and edge configurations of connection states.
[0167] The state transition corresponding to each edge is represented by a separate sequence, which is the connection state S0, state S1, state S2 and the target state S G In order to change the state from one state to the next state that constitutes the entire cooking sequence, it is usually necessary to change the state of multiple ingredients. The cooking for changing the state of each ingredient is represented by a separate sequence.
[0168] Between state S0 and state S1, between state S1 and state S2, and between state S2 and target state S G Each individual sequence between is represented by individual sequence #0 to #2, such as Fig.14 Each of the individual sequences #0 to #2 is also configured by a plurality of states and edges connecting these states.
[0169] For example, when the current state is state S0, as the state transition is performed according to separate sequence #0, the state reaches state S1. Also, when the current state is state S1, as the state transition is performed according to separate sequence #1, the state reaches state S2. When the current state is state S2, as the state transition is performed according to separate sequence #2, the state reaches the target state S G .
[0170] While the entire cooking sequence is set based on recipe data, the individual sequence is basically set based on information prepared in a database (DB). For example, information about an individual sequence, which is a sequence for connecting states to configure the entire cooking sequence, is prepared in advance in the data processing device 1.
[0171] Note that in cases where the entire sequence of states from the starting state to the target state can be fully planned based on the recipe data alone, no separate sequence is required.
[0172] Fig.15 is a diagram showing an example of state transition.
[0173] like Fig.15 As shown, transitions between states configuring a separate sequence are caused by actions of the cooking robot 2. The actions of the cooking robot 2 for causing state transitions are selected by referring to a database of the data processing device 1.
[0174] exist Fig.15 In the example of , “cut” is selected as the action for causing a transition from the state of “whole tomato” (the state before the transition) to the state of “cut tomato” (the state after the transition).
[0175] The database of the data processing device 1 stores information in which two states, a state before transition and a state after transition, are associated with an action for causing transition between the two states. The database of the data processing device 1 will be described below.
[0176] Note that for ease of description, Fig.15 The actions and states shown in . In practice, more detailed actions are selected which specify specifications such as the way of cutting and the way of applying force.
[0177] like Fig.15 As shown, a single sequence is configured by connecting multiple state transitions of one ingredient, and such a single sequence connection constitutes the states of the entire cooking sequence.
[0178] <Database Example>
[0179] Fig.16 is a diagram showing an example of a database for setting individual sequences and executing actions.
[0180] like Fig.16 As shown, a raw material DB 101, a cooking method DB 102, and a serving method DB 103 are prepared. Each database is provided in the data processing device 1, for example.
[0181] Each database may be provided in the data processing device 1 or the cooking robot 2, or each database may be provided in a server accessible from both the data processing device 1 and the cooking robot 2. The databases may be provided in different devices, for example, the ingredient DB 101 is provided in the cooking robot 2, and the cooking method DB 102 and the supply method DB 103 are provided in the data processing device 1.
[0182] The preparation of raw materials is performed according to the recipe data, as shown by the point of arrow A51. The preparation of raw materials is performed with reference to the raw material DB101.
[0183] The raw material DB 101 is used to identify a target raw material, for example, in the case of performing an action on a predetermined raw material. In order to perform an action of raw material preparation, it is necessary to prepare information for raw material identification. The details of the information stored in the database will be described below.
[0184] After the ingredients are prepared, cooking is performed according to the state sequence, such as cutting the ingredients, as shown by arrow A52. Referring to the cooking method DB 102, cooking of the ingredients is performed multiple times. The cooking method DB 102 is used not only for the case of performing actions related to cooking of ingredients, but also for the case of setting the above-mentioned separate sequence.
[0185] After all the ingredients have been cooked, as shown by arrow A53, supply of the cooked ingredients is performed. Referring to the supply method DB 103, supply of the cooked ingredients is performed multiple times.
[0186] -Raw Materials DB 101
[0187] Fig.17 is a diagram showing an example of information to be stored in the raw material DB 101 .
[0188] like Fig.17 As shown, the material DB 101 stores image data of material appearances in association with material names.
[0189] exist Fig.17 In the example of , image data of raw materials "cucumber", "tomato", "avocado", and "green pepper" are shown. As the image data of the raw materials, data indicating the characteristics of the raw materials is prepared, which is extracted by analyzing an image obtained by photographing an image of the raw materials.
[0190] exist Fig.17 In the present invention, one image data is associated with one raw material. However, in reality, a plurality of image data obtained by taking images of raw materials different in size, shape, color, softness, orientation, etc. are associated with one raw material.
[0191] For example, planning of a state sequence requires estimating the current state, such as what raw materials have been prepared. Information stored in the raw material DB 101 is used when estimating the current state.
[0192] The data processing device 1 analyzes the image captured by the camera of the cooking robot 2 and extracts the characteristics of each raw material around the cooking robot 2. The data processing device 1 matches the characteristics of the raw materials around the cooking robot 2 with the characteristics of each raw material indicated by the information stored in the raw material DB 101 to identify the raw materials around the cooking robot 2 and estimate the current state.
[0193] When performing an action, the information stored in the ingredient DB 101 is also used. For example, in the case of preparing "cucumber", the data processing device 1 reads out the image data of "cucumber" by referring to the ingredient DB 101. The data processing device 1 includes the read image data into the instruction command and sends the instruction command to the cooking robot 2.
[0194] The cooking robot 2 analyzes the image captured by the camera device and extracts the characteristics of each raw material around the cooking robot 2. The cooking robot 2 matches the characteristics of the raw materials around the cooking robot 2 with the characteristics represented by the image data of the “cucumber” included in the instruction command and sent from the data processing device 1 to recognize the position and direction of the “cucumber” around the cooking robot 2.
[0195] Thereby, the cooking robot 2 can hold the “cucumber” or move the “cucumber” to a predetermined position. The preparation of other raw materials is performed similarly.
[0196] The information stored in the raw material DB 101 is referred to not only when preparing raw materials but also when performing various actions that require identification of raw materials.
[0197] - Cooking Method DB 102
[0198] Fig.18 is a diagram showing a configuration example of the cooking method DB 102 .
[0199] like Fig.18 As shown, the cooking method DB 102 includes a state transition DB 111 and an action DB 112 .
[0200] The state transition DB 111 is a database storing information about the state transition from the state before cooking to the state after cooking. In the state transition DB 111, information about such state transition is prepared according to the combination of various states such as each raw material, each cooking method, etc. The above-mentioned separate sequence is set based on the information stored in the state transition DB 111.
[0201] Meanwhile, the action DB 112 is a database storing information on specifications of actions for causing state transition of ingredients. Based on the information stored in the action DB 112, instruction commands for executing actions related to cooking of ingredients are generated.
[0202] In the cooking method DB 102, information on state transitions of ingredients and information on specifications of actions for causing respective state transitions are managed in association with each other.
[0203] Fig.19 is a diagram showing an example of information to be stored in the state transition DB 111.
[0204] like Fig.19 As shown, in the state transition DB 111, the information of the raw material name, how to cook, and the state transition from the state before cooking to the state after cooking are stored in association with each other. As described above, each state configuring the state transition is represented by, for example, an ID and a parameter set.
[0205] exist Fig.19 In the example of , a cutting method (how to cut) and a state transition from a state before cutting to a state after cutting are associated with each raw material of "cucumber", "tomato", and "avocado". As ways of cutting "cucumber", "slicing", "shredding", and "dicing" are set.
[0206] For example, in the case of setting a separate sequence including a state in which a "cucumber" is "sliced", the data processing device 1 acquires information about a state transition from a state before "slicing" to a state after "slicing" from the state transition DB 111. The data processing device 1 sets the separate sequence to include the state transition indicated by the acquired information.
[0207] The description of the cooking process of the recipe data may include a description such as "slice the cucumber..." as described above. A separate sequence including a state in which the "cucumber" has been "sliced" may be set based on such a description.
[0208] Regarding other raw materials, information on state transitions of how each raw material is cooked is similarly stored in the state transition DB 111. In addition to the cutting method, various types of information on how to cook such as the methods of washing, peeling, boiling, and roasting are stored.
[0209] As described above, information about state transition from a state before cooking to a state after cooking when cooking raw materials by various methods is stored in the state transition DB 111. The information stored in the state transition DB 111 is generated by representing the state of the raw materials when cooking by various methods using the parameters set as described above.
[0210] The generation of the information stored in the state transition DB 111 may be performed by a predetermined device by analyzing a moving image obtained by capturing an image of the situation during cooking, or may be performed manually by an administrator of the data processing device 1 .
[0211] Fig. 20 is a diagram showing an example of information to be stored in the action DB 112 .
[0212] like Fig. 20 As shown, the action DB 112 is configured by a material dynamics DB as a database of material specification information, a cooking tool dynamics DB as a database of cooking tool specification information, and a robot operation dynamics DB as a database of robot operation specification information.
[0213] The ingredient specification information, the cooking tool specification information, and the robot operation specification information are stored in association with each action for causing the state transition indicated by the information stored in the state transition DB 111 .
[0214] The material specification information is information on the specifications of the material to be the target of the action. The material specifications include various states of the material, such as size, shape, and softness.
[0215] For example, material specification information indicating the state of the material at each time during the execution of the action is stored in the material dynamics DB.
[0216] The cooking tool specification information is information about the specifications of the tool (cooking tool) used for the action. The tool specifications include specifications, functions, etc. of various tools such as accessories used when the cooking robot 2 performs cooking. In the case where the state of the tool changes when performing the action, information indicating the change is included in the cooking tool specification information.
[0217] For example, cooking tool specification information is stored in the cooking tool dynamic DB, in which the specifications, functions, and changed states of the tools are associated with each other at each time during the execution of an action.
[0218] The robot operation specification information is information related to the specifications of the operation when the cooking robot 2 performs an action. The specifications of the operation of the cooking robot 2 include the position of the cooking arm 21 performing the action on the track, the speed, rotation amount, and torque of each motor configuring the cooking arm 21, etc. The specifications of the operation of the cooking robot 2 also include sensor data measured as feedback during the execution of the action.
[0219] As robot operation specification information, information expressing the position of the cooking arm 21 on the track, the speed, rotation amount, and torque of each motor configuring the cooking arm 21, and sensor data used as feedback using vector expressions is prepared.
[0220] In the case where the action is performed using hardware other than the cooking arm 21, information on the specifications of the operation of the hardware is also prepared as the robot operation specification information.
[0221] For example, robot operation specification information indicating the specification of the operation of the cooking arm 21 at each time during the execution of the action is stored in the robot operation dynamics DB.
[0222] For example, when performing an action using a predetermined tool, the driving amount of the motor configuring the cooking arm 21 performing the action differs according to the size of the target material. Information on the operation of the cooking robot 2 when cooking materials in various states using various tools is stored in the action DB 112.
[0223] In other words, information indicating how to operate the cooking arm 21 to perform an action using which function of which tool for which ingredient is stored in the action DB 112 .
[0224] For example, the information stored in the action DB 112 is generated by cooking ingredients in various states using various tools while the manager of the data processing device 1 adjusts the operation of the cooking arm 21 .
[0225] Information generated by a predetermined device by analyzing a moving image obtained by capturing images of situations when ingredients in various states are cooked using various tools may be stored in the action DB 112 .
[0226] Each of the ingredient specification information, the cooking tool specification information, and the robot operation specification information may be generated by appropriately estimating from other information.
[0227] For example, in the case of considering the action of cutting meat in a certain state into two with a knife, if the raw material specification information and the cooking tool specification information are specified, the approximate robot operation specification information can be estimated. Therefore, the information estimated based on other information can be prepared in the action DB 112.
[0228] As described above, in the cooking method DB 102, a state transition DB 111 is prepared, which is a data set that associates the cooking method indicated by the state transition with the raw material. In addition, in the cooking method DB 102, an action DB 112 is prepared in association with the cooking method, which is a data set of specifications of the raw material, specifications of tools used when performing cooking (action) according to the cooking method, and specifications of the operation of the cooking robot 2.
[0229] - Supply Method DB 103
[0230] Fig.21 is a diagram showing an example of information to be stored in the provision method DB 103.
[0231] like Fig.21 As shown, in the supply method DB 103, ingredient IDs, which are identifiers of cooked ingredients necessary to complete the dish, are stored in association with the dish.
[0232] Supply position information, which is information about supply positions, is set for the cooked ingredients. The supply position information indicates the position and layer (level) on the plane of each ingredient in the tableware to be supplied.
[0233] Fig.21 The example in shows information about each of "gourmet salad", "Caesar salad", and "hamburger steak".
[0234] For “delicious salad”, as identifiers of cooked ingredients, ID1 indicating cut lettuce, ID2 indicating cut tomatoes, and ID3 indicating cut cucumbers are associated.
[0235] Furthermore, "x1, y1, L0" is set as information indicating the position of the cut lettuce. "x1, y1" indicates the position of the cut lettuce on the plane, and "L0" indicates that the cut lettuce is placed on the first layer as the lowermost layer.
[0236] In addition, "x2, y2, L3" is set as information indicating the position of the cut tomatoes. "x2, y2" indicates the position of the cut tomatoes on the plane, and "L3" indicates that the cut tomatoes are placed on the fourth layer.
[0237] For other raw materials including cut cucumbers, information indicating the supply location is similarly set.
[0238] For example, when the cutting of "lettuce", "tomato", "cucumber" and the like required for "delicious salad" is completed, the data processing device 1 sets a state sequence regarding the supply of the cooked ingredients. The data processing device 1 generates an instruction command for placing each ingredient at the position indicated by the supply position information according to the state sequence, and outputs the command to the cooking robot 2.
[0239] As described above, a data set indicating required ingredients and supply locations is stored for each dish in the supply method DB 103. For example, the information stored in the supply method DB 103 is generated by analyzing a dish image included in the recipe data.
[0240] The raw materials not photographed in the image and their supply positions are appropriately generated based on information prepared in advance as knowledge of general supply methods. The information stored in the supply method DB 103 may be manually set by the administrator of the data processing device 1 .
[0241] Information indicating a combination of a position on a plane and a hierarchical position in the tableware has been prepared as the supply position information. However, information indicating only a position on a plane in the tableware or information indicating only a hierarchical position may be prepared as the supply position information.
[0242] Furthermore, information indicating a plane arrangement pattern in which raw materials are arranged in the same hierarchy in a predetermined pattern or information indicating a hierarchical arrangement pattern in which raw materials are arranged in different hierarchies in a predetermined pattern may be prepared as the supply position information.
[0243] The information stored in each database may be appropriately changed according to the cooking robot 2 that actually performs an action.
[0244] <Action description example>
[0245] Although the operation of the cooking robot 2 is represented by a state sequence, the operation of the cooking robot 2 may be represented by a tree structure.
[0246] Fig. 22 is a diagram showing an expression example of an action.
[0247] like Fig. 22 As shown, one action can be represented by a tree structure. Information on various actions represented by a tree structure is prepared in a predetermined database of the data processing device 1. Fig. 22 The tree structure shown shows "cut method 1" as an action.
[0248] The "cutting method 1" branch is as state S 11 The "Select Pick Tool" and as state S 12 Select Cutting Tool. 11 is the state of the cooking arm 21-1 performing the action of "cutting method 1". 12 This is the state of cooking arm 21 - 2 performing the action of “cutting method 1 ” in cooperation with cooking arm 21 - 1 .
[0249] "Selecting a picking tool" is a state performed using a manipulator attachment, which is, for example, a tool for gripping a raw material. In addition, "selecting a cutting tool" is a state performed using a knife attachment, which is, for example, a tool for cutting a raw material.
[0250] The state S is configured by a sequence of multiple states that serve as leaves 11 and state S 12 Each state in .
[0251] exist Fig. 22 In the example, the state S 11-1 The "search target" of 11-2 The "pickup target" of 11-3 of "move it to position", and as state S 11-4 The sequence configuration state S of the "fixed target" 11 .
[0252] "Find Target" is the state of searching for the target raw material, while "Pick Up Target" is the state of gripping the raw material with the robot attachment. "Move It to Position" is the state of moving the raw material to the predetermined position, while "Fix Target" is the state of fixing the raw material with the robot attachment.
[0253] At the same time, as the state S 12-1 The "search target" of the state S 12-2 The "Placement Tool" of the 12-3 The sequence configuration state S of the "Move Tool" 12 .
[0254] "Find Target" is a state of searching for a material to be cut, and "Place Tool" is a state of moving the tool attachment close to the material. "Move Tool" is a state of cutting the material by moving the tool attachment close to the material.
[0255] Each state used as a leaf is represented by Fig.23 The state transition from the start state to the goal state is shown.
[0256] Fig.24 It is a diagram showing the state of the operation.
[0257] exist Fig.24 In the example of FIG. 1 , a piece of meat as a raw material is fixed by the cooking arm 21 - 1 to which the robot attachment is attached, and the meat is cut by the cooking arm 21 - 2 to which the knife attachment is attached.
[0258] according to Fig. 22 , the operation of the cooking arm 21-1 for fixing the piece of meat by the robot attachment is realized by executing "selecting a picking tool". Fig. 22 As described in , the operation of the cooking arm 21-2 for cutting meat by the knife attachment is achieved by executing "Select Cutting Tool".
[0259] As described above, the actions of the cooking robot 2 can be controlled by combining the actions represented by the tree structure.
[0260] The action of the expression with a tree structure can be selected according to the state before conversion and the state after conversion, and the action can be set as the edge between the states constituting a separate sequence. In this case, the operation of the cooking robot 2 is controlled using a combination of the state sequence and the action of the expression with a tree structure.
[0261] As described above, the control of the cooking robot 2 is performed using state transition or a tree structure. The control of the cooking robot 2 may be performed using a neural network (NN) such as a deep neural network (DNN) or a recurrent neural network (RNN).
[0262] <<Device Configuration and Operation>>
[0263] <Device Configuration>
[0264] - Configuration of the data processing device 1
[0265] Fig.25 2 is a block diagram showing a configuration example of hardware of the data processing device 1 .
[0266] like Fig.25As shown, the data processing apparatus 1 is configured by a computer. A central processing unit (CPU) 201 , a read only memory (ROM) 202 , and a random access memory (RAM) 203 are connected to each other via a bus 204 .
[0267] Furthermore, an input / output interface 205 is connected to the bus 204. To the input / output interface 205, an input unit 206 including a keyboard, a mouse, and the like, and an output unit 207 including a display, a speaker, and the like are connected.
[0268] Furthermore, a storage unit 208 including a hard disk, a nonvolatile memory, and the like, a communication unit 209 including a network interface and the like, and a drive 210 for driving a removable medium 211 are connected to the input / output interface 205 .
[0269] In the computer configured as described above, the CPU 201 loads a program stored in the storage unit 208 into the RAM 203 via the input / output interface 205 and the bus 204 , for example, and executes the program, thereby performing a series of processes.
[0270] Fig.26 is a block diagram showing a functional configuration example of the data processing device 1 .
[0271] Fig.26 At least a portion of the functional units shown are composed of Fig.25 The CPU 201 in the system executes a predetermined program.
[0272] like Fig.26 As shown, in the data processing device 1, a command generation unit 221 is implemented. The command generation unit 221 includes: a recipe data acquisition unit 231, a recipe data analysis unit 232, a robot state estimation unit 233, a state control unit 234, a database management unit 235, and a command output unit 236.
[0273] The recipe data collection unit 231 controls the communication unit 209 to acquire the recipe data by communicating with the server that manages the recipe data. The recipe data acquired by the recipe data collection unit 231 is, for example, the recipe data of a dish selected by the user.
[0274] A database of recipe data may be provided in the storage unit 208. In this case, the recipe data is acquired from the database provided in the storage unit 208. The recipe data acquired by the recipe data acquisition unit 231 is provided to the recipe data analysis unit 232.
[0275] The recipe data analysis unit 232 analyzes the image and text data included in the recipe data acquired by the recipe data acquisition unit 231. The analysis result of the recipe data analysis unit 232 is provided to the state control unit 234.
[0276] The robot state estimation unit 233 controls the communication unit 209 to receive images and sensor data transmitted from the cooking robot 2. From the cooking robot 2, images captured by the camera of the cooking robot 2 and sensor data measured by sensors provided at predetermined positions of the cooking robot 2 are transmitted at a predetermined cycle. In the images captured by the camera of the cooking robot 2, the situation around the cooking robot 2 is captured.
[0277] The robot state estimation unit 233 estimates the state around the cooking robot 2, such as the state of the cooking arm 21 and the state of the ingredients, by analyzing the image and sensor data transmitted from the cooking robot 2. Information indicating the state around the cooking robot 2 estimated by the robot state estimation unit 233 is provided to the state control unit 234.
[0278] The state control unit 234 sets a target state based on the analysis result of the recipe data by the recipe data analysis unit 232. The state control unit 234 performs planning of the entire cooking sequence from the current state to the target state based on the analysis result of the recipe data, using the state estimated by the robot state estimation unit 233 as the current state.
[0279] Furthermore, the state control unit 234 refers to the cooking method DB 102 (state transition DB 111) managed by the database management unit 235, and sets a separate sequence as a state sequence for connecting states configuring the entire cooking sequence.
[0280] The state control unit 234 selects an action according to the state before the conversion and the state after the conversion by referring to the cooking method DB 102. The state control unit 234 reads out robot operation specification information related to the selected action from the action DB 112, and generates an instruction command.
[0281] The instruction command generated by the state control unit 234 is a command for causing the cooking robot 2 to perform an action according to the state before the conversion and the state after the conversion according to the specifications indicated by the robot operation specification information. Therefore, the state control unit 234 has a function of generating an instruction command. The instruction command generated by the state control unit 234 is provided to the command output unit 236.
[0282] Note that the instruction command generated by the state control unit 234 may be a command for giving an instruction for the execution of the entire action for causing a certain state transition, or may be a command for giving an instruction for the execution of a part of the action. In other words, an action may be executed according to one instruction command, or may be executed according to multiple instruction commands.
[0283] The database management unit 235 manages the ingredient DB 101, the cooking method DB 102, and the supply method DB 103. The ingredient DB 101, the cooking method DB 102, and the supply method DB 103 are constructed in the storage unit 208, for example.
[0284] The command output unit 236 controls the communication unit 209 to transmit the instruction command generated by the state control unit 234 to the cooking robot 2 .
[0285] -Configuration of Cooking Robot 2
[0286] Fig. 27 is a block diagram showing a configuration example of the cooking robot 2 .
[0287] By connecting each part to the controller 61 ( Figure 7 ) to configure the cooking robot 2. Fig. 27 In the configuration shown, the same configuration as the above configuration is denoted by the same symbol, and overlapping descriptions will be omitted as appropriate.
[0288] In addition to the cooking arm 21 , a camera 251 , a sensor 252 , and a communication unit 253 are connected to the controller 61 .
[0289] The controller 61 is configured by a computer having a CPU, ROM, RAM, flash memory, etc. The controller 61 executes a predetermined program through the CPU to control the overall operation of the cooking robot 2. As described above, the data processing apparatus 1 can be configured by the controller 61.
[0290] For example, the controller 61 controls the communication unit 253 to transmit an image captured by the camera 251 and sensor data measured by the sensor 252 to the data processing device 1 .
[0291] In the controller 61 , a predetermined program is executed to realize the instruction command acquisition unit 261 and the arm control unit 262 .
[0292] The instruction command acquisition unit 261 acquires the instruction command transmitted from the data processing device 1 and received by the communication unit 253. The instruction command acquired by the instruction command acquisition unit 261 is supplied to the arm control unit 262.
[0293] The arm control unit 262 controls the operation of the cooking arm 21 according to the instruction command acquired by the instruction command acquisition unit 261 .
[0294] The camera 251 captures images of the situation around the cooking robot 2 and outputs the images obtained by the capturing to the controller 61. The camera 251 is provided at various locations such as the front of the cooking assistance system 12 and the distal end of the cooking arm 21.
[0295] The sensor 252 is configured by various sensors such as a temperature and humidity sensor, a pressure sensor, a light sensor, a distance sensor, a motion sensor, a position sensor, and a vibration sensor. The measurement of the sensor 252 is performed at a predetermined cycle. Sensor data indicating the measurement result of the sensor 252 is provided to the controller 61.
[0296] The camera 251 and the sensor 252 may be disposed at a location away from the housing 11 of the cooking robot 2 .
[0297] The communication unit 253 is a wireless communication module such as a wireless LAN module or a portable communication module compatible with Long Term Evolution (LTE). The communication unit 253 communicates with the data processing device 1 or an external device such as a server on the Internet.
[0298] In addition, the communication unit 253 communicates with a portable terminal such as a smartphone or a tablet terminal used by the user. The user's operation on the cooking robot 2, such as the selection of a dish, can be input through the operation on the portable terminal.
[0299] like Fig. 27 As shown, the cooking arm 21 is provided with a motor 271 and a sensor 272 .
[0300] The motor 271 is provided at each joint of the cooking arm 21. The motor 271 performs a rotation operation around an axis under the control of the arm control unit 262. An encoder for measuring the rotation amount of the motor 271, a driver for adaptively controlling the rotation of the motor 271 based on the measurement result of the encoder, and the like are also provided at each joint.
[0301] The sensor 272 is configured, for example, by a gyro sensor, an acceleration sensor, a touch sensor, etc. The sensor 272 measures the angular velocity, acceleration, etc. of each joint during the operation of the cooking arm 21, and outputs information indicating the measurement result to the controller 61. Sensor data indicating the measurement result of the sensor 272 is also appropriately transmitted from the cooking robot 2 to the data processing device 1.
[0302] At a predetermined timing, information related to the specifications regarding the capabilities of the cooking robot 2, such as the number of cooking arms 21, is provided from the cooking robot 2 to the data processing device 1. In the data processing device 1, the above-mentioned planning is performed by referring to the information in the database according to the specifications regarding the capabilities of the cooking robot 2. The instruction commands generated in the data processing device 1 also correspond to the specifications regarding the capabilities of the cooking robot 2.
[0303] <Device Operation>
[0304] Here, the operations of the data processing device 1 and the cooking robot 2 having the above-described configuration will be described.
[0305] - Operation of the data processing device 1
[0306] First, refer to Fig.28 and Fig.29 The flowchart describes the processing of the data processing device 1 for controlling the operation of the cooking robot 2.
[0307] For example, when a user chooses to cook a predetermined dish, Fig.28 and Fig.29 The processing in starts.
[0308] In step S1, the recipe data collection unit 231 acquires recipe data.
[0309] In step S2 , the recipe data analyzing unit 232 analyzes the recipe data acquired by the recipe data collecting unit 231 .
[0310] In step S3, the state control unit 234 sets the state of the dish appearing in the dish image included in the recipe data as a target state based on the analysis result of the recipe data analysis unit 232. The target state is represented by, for example, an ID and a parameter set as described above.
[0311] In step S4 , the robot state estimation unit 233 estimates the current state based on the image and sensor data transmitted from the cooking robot 2 .
[0312] In step S5 , the state control unit 234 determines whether the target state needs to be corrected.
[0313] In a case where the state control unit 234 determines in step S5 that the target state needs to be corrected, the state control unit 234 corrects the target state in step S6 .
[0314] For example, in the case where "tomato" is not around the cooking robot 2 even though "tomato" is used as a raw material of "delicious salad" set as the target state, it is determined that the target state needs to be corrected. The absence of "tomato" around the cooking robot 2 is specified by the estimation result of the current state by the robot state estimation unit 233.
[0315] In this case, the state control unit 234 corrects the target state, for example, by deleting the parameter about "tomato" from the parameter set. In the case where an unreachable state is set as the target state, for example, the state of the raw material available for cooking does not satisfy the state of the raw material set as the target state, the target state is corrected to an attainable state. The target state may be corrected to change the cooking method, rather than correcting the target state to change the raw material. In addition, the target state may be corrected to change both the raw material and the cooking method.
[0316] In the case where the state of the raw materials available for cooking does not satisfy the target state, a dish that can be cooked with the raw materials available for cooking may be reselected, and the process for cooking a different dish may be continued.
[0317] In the case where it is determined in step S5 that the target state does not need to be corrected, the processing in step S6 is skipped.
[0318] In step S7, the state control unit 234 performs planning of the entire cooking sequence from the current state to the target state based on the analysis result of the recipe data.
[0319] exist Fig.29 In step S8, the state control unit 234 selects a sub-target state. For example, in the state of configuring the entire cooking sequence, the next state is selected as the sub-target state with reference to the current state.
[0320] In step S9 , the state control unit 234 refers to the state transition DB 111 managed by the database management unit 235 , and sets a separate sequence for connecting the current state and the sub-goal state.
[0321] As reference Fig.19 As described above, the state transition DB 111 stores information about state transitions when each ingredient is cooked by various methods. For example, a separate sequence is set by combining state transitions indicated by the information stored in the state transition DB 111.
[0322] When setting an individual sequence, information stored in the action DB 112 may be referred to.
[0323] For example, suppose that no "state transition T" is found that connects the state s before the transition and the state s' after the transition. In this case, the state control unit 234 sets the state set S of the transition source S and the state set S of the transition destination D . State set S S Including state s, and the state set S D Including state s'.
[0324] The state control unit 234 searches for the action with the state set S from the actions having the information stored in the action DB 112. S The states included in i The state control unit 234 applies the actions found by the search to the state s i (actually causing a positive state transition), and adding the state of the transition destination to the state set S S .
[0325] When in the state set SD The states included in exist and are added to the state set S S When the state is the same as the state, the state control unit 234 selects the connection state s i and added to the state set S S The state transition of the state is referred to as "state transition T".
[0326] In addition, when in the state set S D The states included in S do not exist in the state set S and are added to it. S In the case of a state in which the state is the same as the state of the state set S, the state control unit 234 searches for an action having information stored in the action DB 112. D The states included in j The state control unit 234 applies the actions found by the search to the state s j (actually causing a reverse state transition), and adding the state of the transition destination to the state set S D .
[0327] When in the state set S S The states included in exist and are added to the state set S D When the state is the same as the state, the state control unit 234 selects the connection state s j and added to the state set S D The state transition of the state is referred to as "state transition T".
[0328] In step S10, the state control unit 234 refers to the cooking method DB 102, selects an action for causing a state transition from a current state to a next state, and generates an instruction command. Details of the selection of the action will be described below.
[0329] In step S11 , the instruction output unit 236 sends an instruction command to the cooking robot 2 , and causes the cooking robot 2 to perform the action selected by the state control unit 234 .
[0330] In step S12 , the robot state estimation unit 233 estimates the current state based on the image and sensor data transmitted from the cooking robot 2 .
[0331] In step S13 , the state control unit 234 determines whether state transition to the next state has been completed based on the current state estimated by the robot state estimation unit 233 .
[0332] When the state control unit 234 determines that the state transition has not been completed in step S13 , the state control unit 234 determines whether the state transition is in a difficult state in step S14 .
[0333] In a case where the state control unit 234 determines in step S14 that the state transition is not in a difficult state, the process returns to step S10 , and the subsequent processes are repeated.
[0334] On the other hand, in the case where the state control unit 234 determines in step S14 that the state transition is in a difficult state, the process returns to step S8 and the sub-target state is reselected.
[0335] For example, suppose that the robot state estimation unit 233 estimates that the raw material to be cut falls from the top plate 11A due to the execution of the cutting action and the cutting of the raw material becomes impossible. In this case, it is determined that the state transition is in a difficult state, and thereafter, the state of not using the raw material to be cut is selected as a new sub-target state.
[0336] In a case where the state control unit 234 determines in step S13 that the state transition to the next state has been completed, the state control unit 234 determines in step S15 whether the state has transitioned to the sub-target state.
[0337] In a case where it is determined in step S15 that the state has not yet transitioned to the sub-target state, the process returns to step S10 , and the above-described process is repeated to perform state transition according to a separate sequence.
[0338] When the state control unit 234 determines that the state has reached the sub-target state in step S15 , the state control unit 234 determines whether the state has reached the target state in step S16 .
[0339] When it is determined in step S16 that the state transition has not yet reached the target state, the process returns to step S5 and the subsequent process is repeated. In other words, each time the state reaches a sub-target state, it is determined whether the target state needs to be corrected, and the target state is appropriately corrected and the process continues.
[0340] The various states up to the target state configuring the entire cooking sequence may be corrected in conjunction with the correction of the target state.
[0341] When it is determined in step S16 that the state has reached the target state, the process ends. At this time, the current state is a state in which the dish set as the target state is completed.
[0342] As described above, the control of the cooking robot 2 is performed while autonomously correcting (changing) the target state every time a state transition occurs. Since correction is performed to set an attainable state as the target state, the process is not terminated when the dish is not completed.
[0343] As a method of controlling the robot, for example, there is a method of giving up reaching a target state and terminating processing when the state becomes an unreachable state for a set target state.
[0344] If the process is terminated in response to the state of the cooking robot 2 becoming an unreachable state with respect to the target state, the raw materials used so far are wasted.
[0345] By advancing the process while appropriately correcting the target state and controlling the cooking robot 2 to ensure that the target state is achieved, it is possible to prevent the occurrence of ingredient waste.
[0346] - Action selection
[0347] Fig.30 It means to Fig.29 FIG. 5 is a diagram of an example of selection of actions performed in step S10 .
[0348] The selection of the action to be performed by the cooking robot 2 is performed by referring to the cooking method DB 102. As described above, in the cooking method DB 102 managed by the database management unit 235, the specifications of the state transition and the action constituting a separate sequence are managed in association with each other.
[0349] like Fig.30 As shown on the left side of FIG. , the state control unit 234 uses a pair of IDs of the state before the conversion and the IDs of the state after the conversion (<from_state_id,to_state_id> ) as a keyword to search the cooking method DB 102. By performing the search, actions managed in association with the paired ID as the keyword are acquired as action candidates.
[0350] For example, an action candidate corresponding to a paired ID is obtained, wherein the current state is a state before transition, and the next state of the current state in a separate sequence is a state after transition. The action candidate may also be obtained based on a parameter set indicating a state.
[0351] In the case where a plurality of action candidates are acquired, the state control unit 234 evaluates the state transition in the case where the action according to each action candidate is performed. An evaluator for evaluating the state transition is prepared in the form of being linked to each action candidate.
[0352] As shown in bubble #11, the evaluation results of the action candidates are represented by the success rate, variance value, time taken to execute the action, load applied to cleanup after executing the action, etc.
[0353] The state control unit 234 selects, for example, an action candidate causing state transition having the best evaluation result as an action to be performed by the cooking robot 2. For example, an action candidate with a high success rate has a higher evaluation result than an action candidate with a low success rate. In order to evaluate state transition, constraints, priorities, etc. according to the current state are also appropriately used.
[0354] Fig.31 : is a diagram showing an example of selection of action candidates.
[0355] like Fig.31 As shown, when the state before conversion is "whole tomato" and the state after conversion is "1 / 8 tomato wedge", the action candidate "cut with a knife" is acquired by searching with the pair ID of the above states as a key.
[0356] As other action candidates, as shown in bubble #21, "cut with scissors", "cut with a slicer", and "tear with hands" are obtained.
[0357] Fig.32 is a diagram showing an example of evaluation of action candidates.
[0358] By using the evaluator associated with the action candidate “cut with a knife”, such as Fig.32 As shown in , the evaluation result of the action candidate "cutting with a knife" is output. Fig.32 In the example of , the success rate of the action candidate "cutting with a knife" is 99%, and the time taken to perform the action is 20 seconds. In addition, the load applied to cleanup after performing the action is 0.5.
[0359] Each element constituting the evaluation result may be weighted to evaluate the action candidates.
[0360] exist Fig.29 In step S10 , for example, an action is selected based on such an evaluation.
[0361] -Generation of command
[0362] After the motion to be performed by the cooking robot 2 is selected, the instruction command has been generated based on the information stored in the motion DB 112. However, the instruction command may be generated using a neural network (NN).
[0363] Fig.33 is a diagram showing an example of generation of an instruction command.
[0364] like Fig.33 As shown, a NN for generating instruction commands is prepared for each action. The NN for generating instruction commands is, for example, an NN for outputting instruction commands, which takes parameter sets indicating the state before and after the conversion, information indicating the current state, etc. as input.
[0365] The current state is represented by, for example, vector data obtained by converting an image captured by the cooking robot 2, vector data obtained by converting sensor data measured by the cooking robot 2, etc. The NN of each action is generated in advance by learning using various learning data such as a time series of vector data, and is managed by, for example, the database management unit 235.
[0366] Fig.33 The NN in is the NN for the action of "cutting with a knife". When the action of "cutting with a knife" is selected, as shown by arrow A101, the image P photographed by the cooking robot 2 is input to the NN for the action of "cutting with a knife" together with other information. The image P is an image obtained by photographing an image of a "knife" placed on a "tomato".
[0367] As indicated by the point of arrow A102 , when a predetermined instruction command output from the NN for the action of “cutting with a knife” is sent to the cooking robot 2 and executed, an action for transitioning to the next state is performed.
[0368] -Operation of Cooking Robot 2
[0369] Next, we will refer to Fig.34 The flowchart of FIG. 1 describes a cooking process of the cooking robot 2 .
[0370] Fig.34 The processing shown is started, for example, when the communication unit 253 receives an instruction command transmitted from the data processing device 1 .
[0371] In step S31 , the instruction command acquisition unit 261 acquires the instruction command sent from the data processing device 1 .
[0372] In step S32, the arm control unit 262 controls the operation of the cooking arm 21 according to the instruction command. Thereafter, the process returns to step S31, and the above-described process is repeated every time an instruction command is transmitted from the data processing device 1.
[0373] The data processing device 1 can estimate the ingredients to be used for a dish, the method of cooking each ingredient, and the method of supplying the cooked ingredients based on the image and text data included in the recipe data through the above-mentioned series of processing.
[0374] Since information on cooking methods according to various states of ingredients is prepared in the database, the data processing device 1 can cause the cooking robot 2 to cook in a manner corresponding to the various states. The information prepared in the database can be said to be information corresponding to the knowledge of cooking that people have.
[0375] As described above, images corresponding to visual information of ingredients that people have are prepared in the ingredient DB 101. In addition, information corresponding to knowledge about cooking methods such as types of cooking and how to cook is prepared in the cooking method DB 102. In addition, information corresponding to knowledge about how to supply is prepared in the supply method DB 103.
[0376] The data processing device 1 supplements the contents not expressed in the recipe data with information corresponding to the knowledge, thereby enabling the cooking robot 2 to perform cooking in a form corresponding to various states.
[0377] Meanwhile, the cooking robot 2 can complete the dish by cooking the ingredients and supplying the cooked ingredients based on the estimation result of the data processing device 1. Since the planning of the state transition is repeatedly performed, the cooking robot 2 can complete the dish regardless of the state.
[0378] As mentioned above, Figure 1 The illustrated control system enables the cooking robot 2 to perform cooking based on recipe data, which includes image and text data and may be incomplete information for controlling the robot.
[0379] <<Edit>>
[0380] Although cooking planning has been performed based on the recipe data and information prepared in the database, planning can be performed taking into account the user's preferences. For example, planning can be performed by adding ingredients that the user likes or excluding ingredients that the user does not like.
[0381] Planning may be performed by changing ingredients or changing the amount of ingredients according to limitations such as the type and quantity of ingredients available for cooking.
[0382] Cooking with entertainment properties can be performed. For example, the cooking robot 2 can perform a process of explaining how to cut ingredients with sound before cutting or reproducing music when the dish is finished, thereby giving the cooking robot 2 cooking with entertainment properties.
[0383] The case of cooking one dish based on one recipe data has been described. However, a plurality of recipe data may be input to the data processing device 1, and planning of dishes etc. may be performed based on the plurality of recipe data. In the case of inputting a plurality of recipe data, the data processing device 1 processes the recipe data segments and combines them into one recipe data.
[0384] Fig.35 is a diagram showing an example of processing of recipe data.
[0385] In the case of preparing a plurality of types of dishes, cooking for preparing the plurality of types of dishes in parallel is implemented instead of preparing one type of dishes at a time in sequence.
[0386] For example, in the case where the preparation of dish A and dish B has been selected, the state control unit 234 combines the recipe data of dish A and the recipe data of dish B to generate recipe data for preparing dish A and dish B, such as Fig.35 shown.
[0387] In the processed recipe data, the work of making dish A and the work of making dish B are combined.
[0388] Depending on the number of cooking arms 21 or the types of available accessories, another work for making dish B can be performed during the work for making dish A.
[0389] By changing the description content of the recipe data, when making a plurality of types of dishes, the work for making the respective dishes can be appropriately performed in parallel, and the cooking time can be shortened.
[0390] <Computer Configuration Example>
[0391] The above-described series of processes can be executed by hardware or software. In the case where the series of processes are executed by software, a program constituting the software is installed from a program recording medium into a computer incorporated into dedicated hardware, a general-purpose personal computer, or the like.
[0392] The above-described series of processes can be executed by hardware or software. In the case where the series of processes are executed by software, a program constituting the software is installed in a computer incorporated into dedicated hardware, a general-purpose personal computer, or the like.
[0393] The programs to be installed are recorded in Fig.25 The program is provided on and by the removable medium 211 shown, which includes an optical disk (compact disk-read only memory (CD-ROM), digital versatile disk (DVD), etc.), a semiconductor memory, etc. In addition, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be pre-installed in ROM 202 or storage unit 208.
[0394] The program executed by the computer may be a program that is processed time-sequentially according to the sequence described in this specification, or may be a program that is executed in parallel or at necessary timing such as when a call is made.
[0395] Note that in this specification, the term "system" refers to a group of multiple configuration elements (devices, modules (components), etc.), and it does not matter whether all the configuration elements are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network and one device housing multiple modules in one housing are both systems.
[0396] The effects described in this specification are merely examples and are not limited, and other effects may be exhibited.
[0397] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the gist of the present technology.
[0398] For example, in the present technology, a configuration of cloud computing may be adopted in which one function is shared and cooperatively processed by a plurality of devices via a network.
[0399] Furthermore, the steps described in the above flowcharts may be performed by one device, or may be shared and performed by a plurality of devices.
[0400] Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in one step may be executed by one device, or may be shared and executed by a plurality of devices.
[0401] Those skilled in the art should understand that various modifications, combinations, sub-combinations and alterations may be made depending on design requirements and other factors as long as the modifications, combinations, sub-combinations and alterations are within the scope of the appended claims or the equivalents thereof.
[0402] It should be noted that the present technology can adopt the following configurations.
[0403] (1) A data processing device comprising:
[0404] a command generation unit configured to generate an instruction command for giving instructions for one or more operations to be performed during processing by a robot provided with at least one arm,
[0405] Wherein, the instruction command is generated based on instruction data, and the instruction data includes:
[0406] image data obtained by taking one or more images of the situation during or after the processing, and
[0407] text data indicating at least one of an object to be used in the process or an operation to be performed during the process, and
[0408] Wherein, the command generation unit is implemented via at least one processor.
[0409] (2) A data processing device according to (1), wherein the one or more operations include one or more cooking operations to be performed during a cooking process, the instruction data include recipe data of a dish, and at least one object used in the process includes at least one ingredient used in the cooking process, and the command generation unit generates the instruction command by referring to a cooking data set, the cooking data set including information associating the at least one ingredient with the cooking operation to be performed using the at least one ingredient.
[0410] (3) The data processing device according to (1) or (2), wherein the cooking data set is information in which the cooking operation and a cooking tool used when performing the cooking operation are associated with each other.
[0411] (4) The data processing device according to any one of (1) to (3), wherein the cooking data set is information in which the cooking operation, cooking tools used when performing the cooking operation, and specifications of the cooking tools are associated with each other.
[0412] (5) The data processing device according to any one of (1) to (4), wherein the command generation unit generates the instruction command by referring to specifications related to the capability of the robot and the cooking data set.
[0413] (6) The data processing device according to any one of (1) to (5), wherein the command generation unit generates the instruction command by referring to a supply data set, the supply data set including information related to a method of supplying at least one ingredient in the dish.
[0414] (7) A data processing device according to any one of (1) to (6), wherein the supply data set is information in which an identifier of the at least one raw material and position data indicating a position of the at least one raw material in the tableware for supply are associated with each other.
[0415] (8) The data processing device according to any one of (1) to (7), wherein the position data is information indicating at least one of a planar position of the at least one raw material or a hierarchical position of the at least one raw material.
[0416] (9) A data processing device according to any one of (1) to (8), wherein the position data is information indicating a plane layout pattern and indicating a hierarchical layout pattern, wherein the plane layout pattern arranges the raw materials in a predetermined pattern at the same level as the plane position of the at least one raw material, and the hierarchical layout pattern arranges the raw materials in a predetermined pattern at a level different from the hierarchical position of the raw materials.
[0417] (10) A data processing device according to any one of (1) to (9), wherein the image data includes data of one or more still images obtained by capturing the situation during or after the processing, or data of moving images obtained by capturing one or more operations performed during the processing.
[0418] (11) The data processing device according to any one of (1) to (10), wherein the instruction data further includes at least one of text data indicating a name of the object or text data indicating the processing as the text data.
[0419] (12) The data processing device according to any one of (1) to (11), wherein the command generation unit generates the instruction command according to a state transition in which a situation occurring in the image data is set as a target state.
[0420] (13) A data processing device according to any one of (1) to (12), wherein, when the state of the object to be used in the processing does not satisfy the state of the object described in the instruction data, the command generation unit sets the state transition according to the state of the object to be used in the processing and generates the instruction command.
[0421] (14) The data processing device according to any one of (1) to (13), wherein the command generation unit sets the state transition according to a user's preference and generates the instruction command.
[0422] (15) The data processing device according to any one of (1) to (14), wherein the command generation unit autonomously changes the state transition according to the state of one or more operations to be performed during the processing and generates the instruction command.
[0423] (16) A data processing device according to any one of (1) to (15), wherein the command generation unit sets the state transition, in which at least one of the object or the processing is changed according to the state of the one or more operations, and the command generation unit generates the instruction command.
[0424] (17) A data processing device according to any one of (1) to (16), wherein the command generation unit sets the state transition while referring to a data set, wherein the data set includes information that associates an object to be used in the cooking process and an operation to be performed using the object with each other multiple times.
[0425] (18) A data processing device according to any one of (1) to (17), wherein the command generation unit sets the state transition while referring to a supply data set, and the supply data set includes information related to a method of supplying the object multiple times after the object is used in the processing.
[0426] (19) The data processing device according to any one of (1) to (18), wherein the command generation unit generates the instruction command based on combined instruction data obtained by combining a plurality of instruction data.
[0427] (20) A data processing method comprising:
[0428] generating, by a data processing device, instruction commands for giving instructions for one or more operations to be performed during processing by a robot provided with at least one arm,
[0429] Wherein, the instruction command is generated based on instruction data, and the instruction data includes:
[0430] image data obtained by taking one or more images of the situation during or after the processing, and
[0431] Text data indicating at least one of an object to be used in the process or an operation to be performed during the process.
[0432] Reference numerals list
[0433] 1 Data processing device
[0434] 2 Cooking Robot
[0435] 21-1 to 21-4 Cooking arms
[0436] 61 Controller
[0437] 221 Command Generation Unit
[0438] 231 Recipe Data Collection Unit
[0439] 232 Recipe Data Analysis Unit
[0440] 233 Robot State Estimation Unit
[0441] 234 Status Control Unit
[0442] 235 Database Management Unit
[0443] 236 Command output unit
[0444] 251 Camera
[0445] 252 Sensors
[0446] 253 Communication Unit
[0447] 261 Command acquisition unit
[0448] 262 Arm Control Unit
Claims
1. A data processing device, comprising: a command generating unit configured to generate an instruction command for giving instructions for one or more operations to be performed during processing by a cooking robot provided with at least one arm, Wherein, the instruction command is generated based on instruction data, and the instruction data includes: image data obtained by taking one or more images of the situation during or after the processing, and text data indicating at least one of an object to be used in the process or an operation to be performed during the process, wherein the command generation unit generates the instruction command according to the state of the object appearing in the image data set as the target state, and wherein, in a case where the current state of the object used in the processing does not satisfy the target state of the object appearing in the image data, the command generation unit generates the instruction command according to the current state of the object used in the processing, and Wherein, the command generation unit is implemented via at least one processor.
2. The data processing device according to claim 1, wherein: The one or more operations include one or more cooking operations to be performed during a cooking process, the instruction data includes recipe data for a dish, and the at least one object used in the process includes at least one ingredient used in the cooking process, and The command generation unit generates the instruction command by referring to a cooking data set including information that associates the at least one ingredient and a cooking operation to be performed using the at least one ingredient with each other.
3. The data processing device according to claim 2, wherein: The cooking data set is information in which the cooking operation and cooking tools used when performing the cooking operation are associated with each other.
4. The data processing device according to claim 2, wherein: The cooking data set is information in which the cooking operation, cooking tools used when performing the cooking operation, and specifications of the cooking tools are associated with each other.
5. The data processing device according to claim 2, wherein: The command generation unit generates the instruction command by referring to specifications related to the capability of the cooking robot and the cooking data set.
6. The data processing device according to claim 2, wherein: The command generation unit generates the instruction command by referring to a supply data set including information on a method of supplying at least one ingredient in the dish.
7. The data processing device according to claim 6, wherein: The supply data set is information in which an identifier of the at least one ingredient and position data indicating a position of the at least one ingredient in tableware for supply are associated with each other.
8. The data processing device according to claim 7, wherein: The position data is information indicating at least one of a plane position of the at least one raw material or a hierarchical position of the at least one raw material.
9. The data processing device according to claim 8, wherein: The position data is information indicating a plane arrangement pattern and indicating a hierarchical arrangement pattern, wherein the plane arrangement pattern arranges the raw materials in a predetermined pattern in a hierarchy that is the same hierarchy as the plane position of the at least one raw material, and the hierarchical arrangement pattern arranges the raw materials in a predetermined pattern in a hierarchy that is different from the hierarchical position of the raw materials.
10. The data processing device according to claim 1, wherein: The image data includes data of one or more still images obtained by capturing a situation during or after the process, or data of a moving image obtained by capturing one or more operations performed during the process.
11. The data processing device according to claim 1, wherein: The instruction data further includes at least one of text data indicating a name of the object or text data indicating the process as the text data.
12. The data processing device according to claim 1, wherein: The command generation unit sets the state transition according to a user's preference and generates the instruction command.
13. The data processing device according to claim 1, wherein: The command generation unit autonomously changes the state transition according to the state of one or more operations to be performed during the process, and generates the instruction command.
14. The data processing apparatus according to claim 13, wherein: The command generation unit sets the state transition in which at least one of the object or the process is changed according to the state of the one or more operations, and generates the instruction command.
15. The data processing apparatus according to claim 1, wherein: The command generation unit sets the state transition while referring to a data set including information that an object to be used in a cooking process and an operation to be performed using the object are associated with each other a plurality of times.
16. The data processing apparatus according to claim 1, wherein: The command generation unit sets the state transition while referring to a supply data set including information on a method of supplying the object a plurality of times after the object is used in the process.
17. The data processing apparatus according to claim 1, wherein: The command generation unit generates the instruction command based on combined instruction data obtained by combining a plurality of instruction data.
18. A data processing method, comprising: generating, by a data processing device, an instruction command for giving instructions for one or more operations to be performed during processing by a cooking robot provided with at least one arm, Wherein, the instruction command is generated based on instruction data, and the instruction data includes: image data obtained by taking one or more images of the situation during or after the processing, and text data indicating at least one of an object to be used in the process or an operation to be performed during the process, wherein the instruction command is generated according to the state of the object appearing in the image data set as the target state, Wherein, in a case where a current state of the object used in the processing does not satisfy the target state of the object appearing in the image data, the instruction command is generated based on the current state of the object used in the processing.
