Cooking device and trained model generation method

Abstract

To provide a cooker and a learned model generation method that can reduce labor of a user operating the cooker.SOLUTION: A cooker 1 comprises a heating unit 30, an imaging unit 40, an estimating unit 11, and a heating control unit 12. The heating unit 30 heats an object to be heated. The imaging unit 40 images the object when the object is heated. The estimating unit 11 estimates the name of the object during heating of the object on the basis of an imaging result of the imaging unit 40. The heating control unit 12 controls the heating unit 30 depending on an estimating result of the estimating unit 11.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a cooking device and a trained model generating method. [Background technology]

[0002] The cooking apparatus described in Patent Document 1 includes an operation input device as an input means, a heating means, an imaging device, and a central information processing device. A user uses the operation input device to set a desired heating menu from menus presented in advance. The imaging device images the food before heating. If the heating menu selected by the operation input device is grill-type heating, the central information processing device processes the captured image of the food before heating into an image of the food after it has been finished so as to add a browned color to the surface of the food. Then, the heating means heats the food (object to be heated). The imaging device further captures an image of the food while it is being heated. The central information processing device determines whether the image of the processed food matches the image of the food captured while it is being heated, and if it determines that they match, stops the heating output. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 9-229373 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the cooking device described in Patent Document 1, the user needs to use the operation input device to set a desired heating menu from among menus presented in advance. Therefore, the effort of setting the heating menu is a burden for the user.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a cooking device and a learning model generation method that can reduce the effort required by a user to operate the cooking device. [Means for solving the problem]

[0006] According to one aspect of the present application, a cooking device includes a heating unit, an imaging unit, an estimation unit, and a heating control unit. The heating unit heats an object to be heated. The imaging unit images the object to be heated while the object is being heated. The estimation unit estimates a name of the object to be heated based on an imaging result of the imaging unit while the object is being heated. The heating control unit controls the heating unit according to an estimation result of the estimation unit.

[0007] According to another aspect of the present application, a trained model generating method includes the steps of acquiring training data and generating a trained model for estimating a name of an object to be heated by machine learning the training data, the training data including a training target dish image representing a dish and name information representing a name of the dish. Effect of the Invention

[0008] According to the present invention, the effort required by a user to operate a cooking appliance can be reduced. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a cooking device according to a first embodiment of the present invention. [Diagram 2] 1 is a block diagram showing an outline of the operation of the cooking device according to the first embodiment. [Diagram 3] 1 is a block diagram showing a learning device according to a first embodiment. [Figure 4A] FIG. 4 is a diagram showing an example of learning data according to the first embodiment. [Figure 4B] FIG. 11 is a diagram showing another example of learning data according to the first embodiment. [Figure 4C] FIG. 11 is a diagram showing another example of learning data according to the first embodiment. [Figure 4D] FIG. 11 is a diagram showing another example of learning data according to the first embodiment. [Figure 4E]FIG. 11 is a diagram showing another example of learning data according to the first embodiment. [Figure 4F] FIG. 11 is a diagram showing another example of learning data according to the first embodiment. [Diagram 5] FIG. 2 is a block diagram showing input information and output information of a trained model in the first embodiment. [Figure 6] 1 is a flowchart showing a trained model generation method according to embodiment 1. [Figure 7] 4A to 4C are diagrams showing an example of an imaging result of an imaging section according to the first embodiment. [Figure 8] FIG. 4 is a diagram showing a table stored in a storage unit according to the first embodiment. [Figure 9] 5A to 5C are diagrams illustrating an example of the operation of a heating unit according to the first embodiment. [Figure 10] 3 is a flowchart showing a process executed by the cooking device according to the first embodiment. [Figure 11] FIG. 4 is a block diagram showing a cooking device according to a second embodiment of the present invention. [Figure 12A] FIG. 11 is a diagram showing an example of a food image according to the second embodiment. [Figure 12B] FIG. 11 is a diagram showing another example of a food image according to the second embodiment. [Figure 12C] FIG. 11 is a diagram showing another example of a food image according to the second embodiment. [Figure 12D] FIG. 11 is a diagram showing another example of a food image according to the second embodiment. [Figure 12E] FIG. 11 is a diagram showing another example of a food image according to the second embodiment. [Figure 12F] FIG. 11 is a diagram showing another example of a food image according to the second embodiment. [Figure 13] 13A and 13B are diagrams showing an imaging result of an imaging unit according to the second embodiment and an imaging result after processing. [Figure 14] 10 is a flowchart showing a process executed by a cooking device according to a second embodiment. [Figure 15] FIG. 13 is a diagram showing learning data according to a modified example of the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference characters and description thereof will not be repeated.

[0011] First, the cooking device 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the cooking device 1.

[0012] The cooking device 1 is, for example, a microwave oven, an oven, a rice cooker, or an electric pot. Hereinafter, the present invention will be described using an example in which the cooking device 1 is a microwave oven. As shown in Fig. 1, in the first embodiment, the cooking device 1 includes a control unit 10, a storage unit 20, a heating unit 30, an imaging unit 40, an operation unit 50, and a main body unit (not shown).

[0013] The main body includes a heating chamber in which an object to be heated is placed. The object to be heated is, for example, food material.

[0014] The heating unit 30 heats the object to be heated placed in the heating chamber. In the first embodiment, the heating unit 30 has, for example, a microwave heating function, a hot air circulation heating function, and a grill heating function. The microwave heating function is, for example, a function of cooking the object to be heated by radiating microwaves into the heating chamber. The hot air circulation heating function is, for example, a function of cooking the object to be heated by circulating hot air in the heating chamber to make the temperature in the heating chamber uniform, or by blowing hot air directly onto the top surface of the object to be heated. The grill heating function is a function of cooking the object to be heated mainly by exposing the object to radiant heat. That is, the heating unit 30 includes, for example, a microwave supply unit and a heater.

[0015] The imaging unit 40 is, for example, a camera. Specifically, the imaging unit 40 is, for example, a camera composed of a CCD (Charge Coupled Device) digital television camera or the like. The imaging unit 40 is, for example, installed outside the heating chamber so as to face an opening formed on a wall surface constituting the heating chamber. The imaging unit 40 images the heated object, for example, when the heated object is heated by the heating unit 30. In the first embodiment, the imaging unit 40 images the heated object, for example, every predetermined time. Specifically, the imaging unit 40 images the heated object every time a predetermined time elapses while the heated object is heated by the heating unit 30. For example, the predetermined time is 2 seconds.

[0016] The storage unit 20 includes a storage device and stores computer programs such as software and various data. Specifically, the storage unit 20 includes a main storage device (e.g., a semiconductor memory) such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and an auxiliary storage device such as a solid state drive and / or a hard disk drive. The storage unit 20 may include removable media. The storage unit 20 is an example of a storage medium (e.g., a non-transitory computer-readable storage medium).

[0017] Furthermore, the storage unit 20 stores, for example, a trained model LM and a table Tb.

[0018] In the first embodiment, the trained model LM is constructed by machine learning training data. Hereinafter, in this specification, the training data may be referred to as training data TD. The training data TD will be described later.

[0019] In the first embodiment, in the table Tb, each of the names of a plurality of dishes (pizza, toast, sponge cake, etc.) is associated with a heating time and a heating intensity corresponding to the dish. That is, the storage unit 20 stores the name of a dish in association with a heating time and a heating intensity corresponding to the dish. The heating intensity is, for example, the temperature or wattage when heating an object to be heated.

[0020] The operation unit 50 includes, for example, operation keys provided on the cooking appliance 1, or a touch panel provided on a display unit such as a liquid crystal display. Note that the cooking appliance 1 does not have to include a display unit. The operation keys also include input devices such as a mouse and / or a keyboard. The operation unit 50 is operated by a user to receive, for example, an instruction to start heating by the heating unit 30. The operation unit 50 is an example of a "reception unit."

[0021] The control unit 10 includes a processor such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The storage unit 20 and the control unit 10 may be configured with an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). The processor is an example of a "computer."

[0022] The processor of the control unit 10 executes a computer program stored in the storage device of the storage unit 20 to control the storage unit 20, the heating unit 30, the imaging unit 40, and the operation unit 50. The processor of the control unit 10 also executes a computer program stored in the storage device of the storage unit 20 to function as an estimation unit 11, a heating control unit 12, and a measurement unit 13. In other words, the control unit 10 includes the estimation unit 11, the heating control unit 12, and the measurement unit 13.

[0023] The estimation unit 11 estimates the name of the object to be heated based on the imaging result of the imaging unit 40 while the object to be heated is being heated.

[0024] The heating control unit 12 controls the heating unit 30 according to the estimation result of the estimation unit 11. For example, the heating control unit 12 stops the heating of the object by the heating unit 30 after a certain time has elapsed according to the estimation result of the estimation unit 11. More specifically, the heating control unit 12 controls the heating unit 30 so that the object is heated for a heating time according to the name of the object estimated by the estimation unit 11.

[0025] The measuring unit 13 measures the time during which the object to be heated is heated. For example, the measuring unit 13 measures the time during which the object to be heated is heated from when the heating unit 30 starts heating the object to when the estimation unit 11 estimates the name of the object to be heated.

[0026] Next, an outline of the operation of the cooking device 1 will be described with reference to Figures 1 and 2. Figure 2 is a block diagram showing an outline of the operation of the cooking device 1.

[0027] As shown in FIG. 2, first, the measurement unit 13 starts measuring the time for which the object to be heated is heated from the time when the operation unit 50 accepts an instruction to start heating by the heating unit 30.

[0028] Meanwhile, the estimation unit 11 inputs the food image HG, which is the image capturing result of the imaging unit 40, to the trained model LM while the heated object is being heated. Specifically, the estimation unit 11 inputs the food image HG to the trained model LM while the heated object is being heated. The food image HG shows the heated object captured by the imaging unit 40.

[0029] The trained model LM outputs name information MN indicating the name of the object to be heated shown in the input food image HG based on the input food image HG. Specifically, the trained model LM outputs name information MN of the object to be heated shown in the food image HG while the object to be heated is being heated.

[0030] Then, the estimation unit 11 acquires the name information MN output by the trained model LM while the heated object is being heated. That is, the estimation unit 11 estimates the name of the heated object based on the cooking image HG while the heated object is being heated. Therefore, the estimation unit 11 can accurately estimate the name of the heated object. Then, the estimation unit 11 inputs the name information MN to the heating control unit 12.

[0031] The heating control unit 12 controls the heating unit 30 according to the name information MN. That is, the heating control unit 12 controls the heating unit 30 according to the estimation result of the estimation unit 11. Therefore, the object to be heated can be heated according to the name of the object to be heated without the user having to operate the operation unit 50 in advance to select a heating method (menu) for the object to be heated. As a result, the user's effort in operating the heating cooker 1 can be reduced.

[0032] Specifically, the heating control unit 12 determines the heating time and heating intensity according to the object to be heated according to the estimation result of the estimation unit 11. Therefore, the object to be heated can be heated according to the name of the object to be heated without the user having to operate the operation unit 50 in advance to select the heating time and heating intensity for the object to be heated. As a result, the object to be heated can be heated with the heating time and heating intensity according to the name of the object to be heated while reducing the effort required for the user to operate the heating cooker 1. In other words, the object to be heated can be appropriately heated according to the name of the object to be heated.

[0033] Specifically, for example, when the object to be heated is toast, the estimation unit 11 estimates that the object to be heated is toast. Meanwhile, for example, in the table Tb stored in the storage unit 20, toast is associated with a heating time indicating 11 minutes and a heating intensity indicating 180°C. Therefore, the heating control unit 12 determines the time for heating the object to be 11 minutes and the heating intensity for heating the object to be 180°C. Therefore, the heating control unit 12 controls the heating unit 30, for example, so that the object to be heated is heated at 180°C for 11 minutes. As a result, the toast is properly toasted.

[0034] More specifically, in the first embodiment, the heating control unit 12 determines the remaining heating time for the object to be heated based on the determined heating time and the measurement result of the measurement unit 13. Therefore, the heating control unit 12 controls the heating unit 30 so as to heat the object to be heated for the determined remaining heating time. As a result, it is possible to prevent the object to be heated from being excessively heated by the heating unit 30. In other words, it is possible to appropriately heat the object to be heated.

[0035] Next, a learning device 100 that generates a trained model LM will be described with reference to FIG. 3. FIG. 3 is a block diagram showing the learning device 100. The learning device 100 is, for example, a computer. The learning device 100 generates the trained model LM by machine learning the training data TD. Then, the learning device 100 outputs the trained model LM. As described above, the trained model LM estimates the name of the heated object indicated by the input cooking image HG. The training data TD is the subject of machine learning by the learning device 100.

[0036] As shown in FIG. 3, the learning device 100 includes a processing unit 110 and a storage unit 120.

[0037] The processing unit 110 includes a processor such as a CPU and a GPU. The storage unit 120 includes a storage device and stores data and computer programs. The processor of the processing unit 110 executes the computer programs stored in the storage device of the storage unit 120 to perform various processes. For example, the storage unit 120 may include a main storage device and an auxiliary storage device, and may also include a removable medium, similar to the storage unit 20 of the cooking appliance 1. The storage unit 120 is, for example, a non-transitory computer-readable storage medium.

[0038] The storage unit 120 stores a learning program LP. The learning program LP is a program for executing a machine learning algorithm for finding certain rules from multiple pieces of learning data TD and generating a learned model LM that expresses the found rules.

[0039] The machine learning algorithm is not particularly limited as long as it is supervised learning, and may be, for example, a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. Therefore, the trained model LM includes a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. In the machine learning for generating the trained model LM, an error backpropagation method may be used.

[0040] For example, the neural network includes an input layer, one or more intermediate layers, and an output layer. Specifically, the neural network is a deep neural network (DNN), a recurrent neural network (RNN), or a convolutional neural network (CNN), and performs deep learning. For example, the deep neural network includes an input layer, multiple intermediate layers, and an output layer.

[0041] The processing unit 110 performs machine learning on a plurality of pieces of training data TD based on the training program LP. As a result, certain rules are found from the plurality of pieces of training data TD, and a trained model LM is generated. In other words, the trained model LM is constructed by machine learning the training data TD. The storage unit 120 stores the trained model LM.

[0042] Specifically, the processing unit 110 finds certain rules between the training target dish images LG and the name information MN contained in the training data TD, and generates a trained model LM. The training target dish images LG are explanatory variables, and the name information MN is a target variable. In the first embodiment, the training target dish images LG indicate dishes. The name information MN indicates the name of the dish shown in the training target dish images LG.

[0043] More specifically, the processing unit 110 performs machine learning on a plurality of pieces of training data TD based on the training program LP to calculate a plurality of trained parameters and generate a trained model LM including one or more functions to which the plurality of trained parameters are applied. The trained parameters are parameters (coefficients) obtained based on the results of machine learning using the plurality of pieces of training data TD.

[0044] The trained model LM causes the computer to function so as to input input information and output output information. In other words, the trained model LM inputs input information and outputs output information. Specifically, the trained model LM causes the computer to function so as to estimate the name of the dish indicated by the input dish image HG. In other words, the trained model LM estimates the name of the dish indicated by the input dish image HG.

[0045] Next, the learning data TD will be described with reference to Fig. 4A to Fig. 4F. Fig. 4A shows an example of the learning data TD. Fig. 4B to Fig. 4F each show another example of the learning data TD. Hereinafter, the learning data TD shown in Fig. 4A may be referred to as learning data TD1, the learning data TD shown in Fig. 4B may be referred to as learning data TD2, the learning data TD shown in Fig. 4C may be referred to as learning data TD3, the learning data TD shown in Fig. 4D may be referred to as learning data TD4, the learning data TD shown in Fig. 4E may be referred to as learning data TD5, and the learning data TD shown in Fig. 4F may be referred to as learning data TD6.

[0046] In the first embodiment, the learning data TD includes learning object dish images LG and name information MN. In the first embodiment, the name of the dish shown in the learning object dish images LG is determined, for example, by a user. That is, the learning object dish images LG are labeled with name information MN indicating the name of the dish determined by the user. To facilitate understanding, the browning of the dish is indicated by hatching. Specifically, the wider the hatching interval, the lighter the browning, and the narrower the hatching interval, the darker the browning.

[0047] As shown in Figures 4A to 4F, the multiple pieces of learning data TD correspond to multiple dishes (e.g., pizza and toast, etc.). Specifically, the multiple pieces of learning data TD shown in Figures 4A to 4C correspond to pizza. Moreover, the multiple pieces of learning data TD shown in Figures 4D to 4F correspond to toast. Moreover, each of the learning object dish images LG of the multiple pieces of learning data TD shows a dish at a different time in the process of heating the same dish.

[0048] For example, each of the learning data TD1 to TD3 shown in FIGS. 4A to 4C represents a pizza at a different time in the process of heating the pizza.

[0049] As shown in Fig. 4A, the pizza shown in the training object dish image LG of training data TD1 is an earlier stage pizza than the pizza shown in the training object dish image LG of training data TD2 and the training object dish image LG of training data TD3. In embodiment 1, the training object dish image LG of training data TD1 is labeled with name information MN indicating "pizza". In this specification, the name information MN indicating "pizza" may be referred to as name information MN1.

[0050] 4B, the pizza shown in the training object dish image LG of training data TD2 is at an intermediate stage between the pizza shown in the training object dish image LG of training data TD1 and the pizza shown in the training object dish image LG of training data TD3. In embodiment 1, the training object dish image LG of training data TD2 is labeled with name information MN1 indicating "pizza."

[0051] 4C, the pizza shown in the training object dish image LG of training data TD3 is at a later stage than the pizza shown in the training object dish image LG of training data TD1 and the pizza shown in the training object dish image LG of training data TD2. In addition, in embodiment 1, the training object dish image LG of training data TD3 is labeled with name information MN1 indicating "pizza."

[0052] Furthermore, for example, each of the learning data TD4 to TD6 shown in FIGS. 4D to 4F indicates toast at a different time in the process of heating the toast.

[0053] As shown in Fig. 4D, the toast shown in the training object dish image LG of the training data TD4 is an earlier stage of toast than the toast shown in the training object dish image LG of the training data TD5 and the toast shown in the training object dish image LG of the training data TD6. In addition, in the first embodiment, the training object dish image LG of the training data TD4 is labeled with name information MN indicating "toast". In this specification, the name information MN indicating "toast" may be referred to as name information MN2.

[0054] 4E, the toast shown in the training object dish image LG of the training data TD5 is at a stage between the toast shown in the training object dish image LG of the training data TD4 and the toast shown in the training object dish image LG of the training data TD6. In the first embodiment, the training object dish image LG of the training data TD5 is labeled with name information MN2 indicating "toast".

[0055] 4F, the toast shown in the training object dish image LG of the training data TD6 is at a later stage than the toast shown in the training object dish image LG of the training data TD4 and the toast shown in the training object dish image LG of the training data TD5. In the first embodiment, the training object dish image LG of the training data TD6 is labeled with name information MN2 indicating "toast".

[0056] The name information MN may be expressed as a character string such as "pizza" or "toast", or may be indicated by identification information (ID) corresponding to each of "pizza" and "toast".

[0057] Next, the input information and output information of the trained model LM will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the input information and output information of the trained model LM.

[0058] As shown in Fig. 5, the input information of the trained model LM is a food image HG, which is the imaging result of the imaging unit 40. The trained model LM outputs name information MN of the heated object shown in the food image HG based on the input food image HG. In the example shown in Fig. 5, the trained model LM estimates that the name of the heated object shown in the food image HG is "toast". Therefore, the trained model LM outputs name information MN indicating "toast".

[0059] Next, a learning method executed by the learning device 100 will be described with reference to Fig. 3 and Fig. 6. Fig. 6 is a flowchart showing a trained model generating method according to embodiment 1. As shown in Fig. 6, the trained model generating method includes steps S41 to S44.

[0060] 3 and 6, in step S41, the processing unit 110 of the learning device 100 acquires learning data TD. For example, the processing unit 110 acquires learning data TD input by a user.

[0061] Next, in step S42, the processing unit 110 performs machine learning on the learning data TD based on the learning program LP.

[0062] Next, in step S43, the processing unit 110 determines whether or not a learning end condition is satisfied. The learning end condition is a condition that is determined in advance for ending the machine learning. For example, the learning end condition is that the number of iterations reaches a specified number.

[0063] If a negative determination (No) is made in step S43, the process proceeds to step S41. As a result, the machine learning is repeated.

[0064] On the other hand, if an affirmative determination (Yes) is made in step S43, the process proceeds to step S44.

[0065] In step S44, the processing unit 110 outputs a model (one or more functions) to which the latest parameters (coefficients), that is, the learned parameters (coefficients), are applied as the learned model LM. Then, the storage unit 120 stores the learned model LM.

[0066] As described above, the learning device 100 executes steps S41 to S44, thereby generating the trained model LM.

[0067] Next, the operation of the cooking device 1 will be described with reference to Figs. 7 to 9. First, with an object to be heated placed in the heating chamber, the operation unit 50 receives an instruction to start heating by the heating unit 30. Then, the heating control unit 12 controls the heating unit 30 so as to heat the object to be heated. As a result, the heating unit 30 starts heating the object to be heated. At this time, the heating unit 30 heats the object to be heated with, for example, a predetermined heating intensity. In addition, as the heating unit 30 starts heating the object to be heated, the measurement unit 13 starts measuring the time for which the object to be heated is heated.

[0068] Fig. 7 is a diagram showing a food image HG that is the result of imaging by the imaging unit 40. As shown in Fig. 7, the estimation unit 11 inputs the food image HG that is the result of imaging by the imaging unit 40 to the trained model LM. The trained model LM outputs name information MN indicating "pizza" based on the input food image HG. Then, the estimation unit 11 acquires the name information MN output by the trained model LM.

[0069] FIG. 8 is a diagram showing a table Tb stored in the storage unit 20. As shown in FIG. 8, in the table Tb, a heating time Ht and a heating intensity Hs are associated with name information MN. The heating control unit 12 determines the heating time Ht and the heating intensity Hs for the object to be heated according to the estimation result of the estimation unit 11, that is, the name information MN acquired by the estimation unit 11. For example, since the name information MN of the object to be heated shown in the cooking image HG shown in FIG. 7 is estimated to be "pizza", the heating control unit 12 determines the heating time Ht to be "25 minutes" and the heating intensity Hs to be "250°C".

[0070] FIG. 9 is a diagram showing an example of the operation of the heating unit 30. As shown in FIG. 9, the measurement result of the measuring unit 13 when the estimation unit 11 acquires the name information MN, that is, the heating time Ct of the heated object is 4 minutes. In other words, when 4 minutes have passed since the heating unit 30 started heating the heated object, the name information MN is acquired based on the food image HG, which is the imaging result. Then, the heating control unit 12 determines the heating time Ht to be "25 minutes" and the heating intensity Hs to be "250°C". Therefore, the heating control unit 12 determines the remaining heating time Rt for the heated object based on the measurement result (heating time Ct) of the measuring unit 13 and the determined heating time Ht. Then, the heating control unit 12 controls the heating unit 30 so that the heating of the heated object is terminated in response to the lapse of the remaining heating time Rt after the estimation unit 11 acquires the name information MN.

[0071] Next, a process executed by the cooking device 1 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing a process executed by the cooking device 1 according to embodiment 1. The process executed by the cooking device 1 includes steps S1 to S10.

[0072] 10, in step S1, the operation unit 50 receives an instruction to start heating by the heating unit 30. Then, the heating control unit 12 controls the heating unit 30 to heat the object to be heated. As a result, the heating unit 30 heats the object to be heated. In addition, as the heating unit 30 starts heating the object to be heated, the measurement unit 13 starts measuring the time for which the object to be heated is heated.

[0073] In step S2, the control unit 10 controls the imaging unit 40 to capture an image of the object to be heated. As a result, the imaging unit 40 captures an image of the object to be heated.

[0074] In step S3, the estimation unit 11 inputs the food image HG, which is the imaging result, into the trained model LM.

[0075] In step S4, the estimation unit 11 determines whether or not the trained model LM has output name information MN based on the food image HG input in step S3. If a negative determination (No) is made in step S4, i.e., if the trained model LM has not output name information MN, or if the trained model LM has output "unknown", processing proceeds to step S2. On the other hand, if a positive determination (Yes) is made in step S4, i.e., if the trained model LM has output name information MN, processing proceeds to step S5. The cases where the trained model LM has not output name information MN and where the trained model LM has output "unknown" will be described later.

[0076] In step S5, the estimation unit 11 acquires the name information MN output by the trained model LM. Then, the estimation unit 11 inputs the acquired state information SB to the heating control unit 12.

[0077] In step S6, the heating control unit 12 determines the heating time Ht and the heating intensity Hs for the object to be heated, in accordance with the estimation result of the estimation unit 11.

[0078] In step S7, the heating control unit 12 determines the remaining heating time Rt for the object to be heated based on the measurement result of the measurement unit 13 and the heating time Ht determined in step S6.

[0079] In step S8, the heating control unit 12 adjusts the heating intensity of the heating unit 30 based on the heating intensity Hs determined in step S6.

[0080] In step S9, the heating control unit 12 determines whether the remaining heating time Rt determined in step S7 has elapsed since the remaining heating time Rt was determined in step S7. If a positive determination (Yes) is made in step S9, the process proceeds to step S10. On the other hand, if a negative determination (No) is made in step S9, the process returns to step S9.

[0081] In step S10, the heating control unit 12 controls the heating unit 30 to stop heating the object to be heated. As a result, the heating unit 30 stops heating.

[0082] (Embodiment 2) Next, a cooking device 1a according to a second embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing a cooking device 1a according to a second embodiment. The second embodiment is mainly different from the first embodiment in that the cooking device 1a includes a turntable 60 and that the estimation unit 11a estimates the name of the object to be heated by pattern matching. The following mainly describes the differences between the second embodiment and the first embodiment.

[0083] The cooking appliance 1a includes a control unit 10a, a storage unit 20a, a heating unit 30, an imaging unit 40, an operation unit 50, and a turntable 60.

[0084] An object to be heated is placed on the turntable 60. In the second embodiment, while the heating unit 30 heats the object to be heated, the turntable 60 rotates, for example, at a substantially constant angular velocity. That is, the turntable 60 rotates with the object to be heated placed thereon.

[0085] The control unit 10a includes a processor such as a CPU and an MPU. The processor of the control unit 10 executes a computer program stored in a storage device of the storage unit 20a to control the storage unit 20a, the heating unit 30, the imaging unit 40, the operation unit 50, and a motor (not shown) for rotating the turntable 60.

[0086] Furthermore, the processor of the control unit 10 executes a computer program stored in the storage device of the memory unit 20a to function as an estimation unit 11a, a heating control unit 12, a measurement unit 13, and a correction unit 14. That is, the control unit 10 includes an estimation unit 11a, a heating control unit 12, a measurement unit 13, and a correction unit 14. The estimation unit 11a and the correction unit 14 will be described later.

[0087] The storage unit 20a stores pattern matching data PD and a table Tb. The pattern matching data PD includes a plurality of food images MG and name information MN.

[0088] Next, the pattern matching data PD stored in the storage unit 20a will be described with reference to Figs. 12A to 12F. Fig. 12A shows an example of the pattern matching data PD. Each of Figs. 12B to 12F shows another example of the pattern matching data PD. Hereinafter, the pattern matching data PD shown in Fig. 12A may be described as pattern matching data PD1, the pattern matching data PD shown in Fig. 12B may be described as pattern matching data PD2, the pattern matching data PD shown in Fig. 12C may be described as pattern matching data PD3, the pattern matching data PD shown in Fig. 12D may be described as pattern matching data PD4, the pattern matching data PD shown in Fig. 12E may be described as pattern matching data PD5, and the pattern matching data PD shown in Fig. 12F may be described as pattern matching data PD6.

[0089] 12A to 12F, the food images MG of the plurality of pattern matching data PD correspond to a plurality of dishes (e.g., pizza, toast, etc.) respectively. Furthermore, the plurality of food images MG each show a dish at a different time during the process of heating the same dish.

[0090] For example, each of the dish images MG of the pattern matching data PD1 to PD3 shown in FIGS. 12A to 12C shows a pizza at a different time in the process of heating the pizza.

[0091] 12A, the pizza shown in the dish image MG in the pattern matching data PD1 is an earlier stage of pizza than the pizza shown in the dish image MG in the pattern matching data PD2 and the pizza shown in the dish image MG in the pattern matching data PD3. In the second embodiment, the dish image MG in the pattern matching data PD1 is labeled with name information MN1 indicating "pizza."

[0092] 12B, the pizza shown in the dish image MG of the pattern matching data PD2 is at a stage between the pizza shown in the dish image MG of the pattern matching data PD1 and the pizza shown in the dish image MG of the pattern matching data PD3. In the second embodiment, the dish image MG of the pattern matching data PD2 is labeled with name information MN1 indicating "pizza."

[0093] 12C, the pizza shown in the dish image MG in the pattern matching data PD3 is at a later stage than the pizza shown in the dish image MG in the pattern matching data PD1 and the pizza shown in the dish image MG in the pattern matching data PD2. In the second embodiment, the dish image MG in the pattern matching data PD3 is labeled with name information MN1 indicating "pizza."

[0094] Also, for example, each of the pattern matching data PD4 to PD6 shown in FIGS. 12D to 12F indicates toast at a different time in the process of heating the toast.

[0095] 12D, the toast shown in the dish image MG in the pattern matching data PD4 is an earlier stage of toast than the toast shown in the dish image MG in the pattern matching data PD5 and the toast shown in the dish image MG in the pattern matching data PD6. In the second embodiment, the dish image MG4 is labeled with name information MN2 indicating "toast."

[0096] 12E, the toast shown in the dish image MG of the pattern matching data PD5 represents a stage between the toast shown in the dish image MG of the pattern matching data PD5 and the toast shown in the dish image MG of the pattern matching data PD6. In the second embodiment, the dish image MG of the pattern matching data PD5 is labeled with name information MN2 indicating "toast."

[0097] 12F, the toast shown in the dish image MG in the pattern matching data PD6 is at a later stage than the toast shown in the dish image MG4 in the pattern matching data PD4 and the toast shown in the dish image MG5 in the pattern matching data PD5. In the second embodiment, the dish image MG6 in the pattern matching data PD6 is labeled with name information MN2 indicating "toast."

[0098] Next, correction of the orientation of the imaging result will be described with reference to Fig. 13. Fig. 13 is a diagram showing a food image HG that is the imaging result of the imaging unit 40, and a food image HGA that is the imaging result after correction. As shown in Fig. 13, a case where the object to be heated is pizza Pz will be described as an example. That is, in the example shown in Fig. 13, pizza Pz is placed on turntable 60.

[0099] The correction unit 14 corrects the orientation of the food image HG based on the angular velocity ω when the turntable 60 rotates. The food image HGA is obtained by correcting the orientation of the food image HG by the correction unit 14 based on the angular velocity ω and the measurement result of the measurement unit 13 when the heated object is heated. The measurement result of the measurement unit 13 is the time t when the heated object is heated by the heating unit 30. Specifically, the angle θ for correcting the orientation of the food image HG is calculated by θ=ω×t. The correction unit 14 corrects the food image HG by rotating the food image HG by the angle θ in the direction opposite to the direction in which the turntable 60 rotates.

[0100] The estimation unit 11a compares the corrected food image HGA with the food image MG stored in the storage unit 20a, and estimates the name of the heated object based on the comparison result. Therefore, the name of the heated object can be estimated more accurately than when the name of the heated object is estimated without correcting the orientation of the food image HG.

[0101] In this embodiment, the estimation unit 11a judges whether the matching rate indicated by the comparison result is equal to or higher than a threshold. The matching rate indicates the matching ratio between the heated object indicated by the food image HGA and the food image MG stored in the storage unit 20a. The threshold is, for example, 90%. In other words, when the matching rate between the food image HGA and the food image MG is 90% or higher, the estimation unit 11a estimates that the name of the heated object is the name information MN associated with the food image MG. The heating control unit 12 then controls the heating unit 30 according to the estimation result of the estimation unit 11a.

[0102] Next, a process executed by the cooking device 1a of the second embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the process executed by the cooking device 1a.

[0103] 14, in step S21, the operation unit 50 receives an instruction to start heating by the heating unit 30. Then, the heating control unit 12 controls the heating unit 30 to heat the object to be heated. As a result, the heating unit 30 heats the object to be heated. In addition, as the heating unit 30 starts heating the object to be heated, the measurement unit 13 starts measuring the time for which the object to be heated is heated.

[0104] In step S22, the control unit 10 controls the imaging unit 40 to capture an image of the object to be heated. As a result, the imaging unit 40 captures an image of the object to be heated.

[0105] In step S23, the correction unit 14 determines whether or not the turntable 60 is rotating. If an affirmative determination (Yes) is made in step S23, that is, if the correction unit 14 determines that the turntable 60 is rotating, the process proceeds to step S24. On the other hand, if a negative determination (No) is made in step S23, that is, if the correction unit 14 determines that the turntable 60 is not rotating, the process proceeds to step S25. Here, the correction unit 14 may determine whether or not the turntable 60 is rotating by determining whether or not a motor that rotates the turntable 60 is being driven.

[0106] In step S24, the correction unit 14 corrects the food image HG, which is the image capture result of the imaging unit 40 in step S22, based on the angular velocity ω when the turntable 60 rotates and the measurement result of the measurement unit 13.

[0107] In step S25, the estimation unit 11 compares the food image HG, which is the imaging result, with the food image MG. If the food image HG, which is the imaging result, is corrected by the correction unit 14 in step S24, the estimation unit 11 compares the food image HGA, which is the imaging result after the correction, with the food image MG.

[0108] In step S26, the estimation unit 11 determines whether the matching rate indicated by the comparison result in step S25 is equal to or greater than a threshold. If the determination in step S26 is negative (No), i.e., if the matching rate between the imaging result and the food image MG is less than the threshold, the process proceeds to step S22. On the other hand, if the determination in step S26 is positive (Yes), i.e., if the matching rate between the imaging result and the food image MG is equal to or greater than the threshold, the process proceeds to step S27.

[0109] In step S27, the estimation unit 11 estimates the name of the object to be heated. Specifically, the estimation unit 11 obtains name information MN associated with the food image MG whose matching rate is equal to or higher than a threshold value.

[0110] In step S28, the heating control unit 12 determines the heating time Ht and the heating intensity Hs for the object to be heated, in accordance with the estimation result of the estimation unit 11.

[0111] In step S29, the heating control unit 12 determines the remaining heating time Rt for the object to be heated based on the measurement result of the measurement unit 13 and the heating time Ht determined in step S6.

[0112] In step S30, the heating control unit 12 adjusts the heating intensity of the heating unit 30 based on the heating intensity Hs determined in step S29.

[0113] In step S31, the heating control unit 12 determines whether or not the heating time Rt has elapsed since the remaining heating time Rt was determined in step S29. If a positive determination (Yes) is made in step S31, the process proceeds to step S32. On the other hand, if a negative determination (No) is made in step S31, the process returns to step S31.

[0114] In step S32, the heating control unit 12 controls the heating unit 30 to stop heating the object to be heated. As a result, the heating unit 30 stops heating.

[0115] (Modification) Next, a cooking device 1 according to a modified example of the first embodiment will be described with reference to Fig. 15. Fig. 15 is a diagram showing learning data Tda obtained by machine learning of a trained model LM of the cooking device 1 according to a modified example of the first embodiment. In the modified example, the main difference between the modified example and the first embodiment is that one learning data Tda includes a plurality of learning target dish images LD. Below, the differences between the modified example and the first embodiment will be mainly described.

[0116] As shown in Fig. 15, one learning data set Tda includes multiple learning object dish images LD and name information MN. Specifically, the learning data set Tda includes multiple learning object dish images LD1 to LDNX. To make it easier to understand, the browning of the dish is indicated by hatching. Specifically, the wider the hatching interval, the lighter the browning, and the narrower the hatching interval, the darker the browning.

[0117] The multiple training object dish images LD each show a dish at a different time during the process of heating the dish. Specifically, the multiple training object dish images LD each show a dish captured at a predetermined time interval during the process of heating the dish. In the modified example, the predetermined time is, for example, two seconds. In other words, the training object dish images LD that the trained model LM according to the modified example learns from are cooking videos.

[0118] The imaging unit 40 according to the modified example captures an image of the object to be heated at predetermined time intervals, for example. Then, the estimation unit 11 inputs cooking videos, which are multiple imaging results captured at predetermined time intervals, to the trained model LM. That is, in the modified example, the input information of the trained model LM is the video captured by the imaging unit 40. The trained model LM outputs name information MN of the object to be heated indicated by the cooking video based on the input cooking video.

[0119] The embodiments and modifications of the present invention have been described above with reference to the drawings. However, the present invention is not limited to the above embodiments, and can be implemented in various aspects without departing from the gist of the present invention. In addition, various inventions can be formed by appropriately combining multiple components disclosed in the above embodiments. For example, some components may be deleted from all components shown in the embodiments. The drawings are mainly shown schematically for ease of understanding, and the number of each component shown may differ from the actual number due to the convenience of drawing. In addition, each component shown in the above embodiments is an example, is not particularly limited, and various modifications are possible within a range that does not substantially deviate from the effects of the present invention.

[0120] (1) As described with reference to FIG. 7, the trained model LM outputs name information MN indicating the name of the object to be heated indicated by the food image HG based on the food image HG input by the estimation unit 11. However, if the trained model LM cannot estimate the name of the object to be heated based on the food image HG input by the estimation unit 11, the trained model LM may not output the name information MN, or may output the name information MN indicating "unknown". If the trained model LM does not output the name information MN, or if the trained model LM outputs the name information MN indicating "unknown", the estimation unit 11 controls the imaging unit 40 so that the imaging unit 40 images the object to be heated again. Specifically, the estimation unit 11 controls the imaging unit 40 so that the imaging unit 40 images the object to be heated a predetermined time after the imaging unit 40 images the object to be heated last time. That is, the imaging unit 40 images the object to be heated at predetermined time intervals until the estimation unit 11 estimates the name of the object to be heated. Therefore, it is possible to prevent the object to be heated from being heated with a heating intensity and heating time that are not appropriate for the object to be heated.

[0121] (2) As described with reference to Fig. 9, the heating control unit 12 determined the remaining heating time Rt for the object to be heated based on the measurement result (heating time Ct) of the measurement unit 13 and the determined heating time Ht. Then, the heating control unit 12 controlled the heating unit 30 so that the heating of the object to be heated is terminated when the remaining heating time Rt has elapsed since the estimation unit 11 acquired the name information MN. However, the heating control unit 12 may also control the heating unit 30 so that the heating of the object to be heated is terminated after a certain time has elapsed since the remaining heating time Rt has elapsed.

[0122] (3) The order of steps S1 to S10 in FIG. 10 may be changed as appropriate.

[0123] (4) The order of step S21 in FIG. 14 to step S32 in FIG. 15 may be changed as appropriate.

[0124] (5) As described with reference to Figures 4A to 4C, three training object dish images LG for one dish were input to the trained model LM. However, as long as the trained model LM can estimate the name of the heated object, it may train one or two training object dish images LG for one dish, or train four or more training object dish images LG.

[0125] (6) As described with reference to Fig. 12 and Fig. 13, the cooking device 1a according to the second embodiment includes the turntable 60. However, as long as the cooking device 1a heats an object to be heated, the cooking device 1a does not need to include the turntable 60. In this case, the correction unit 14 does not correct the imaging result.

[0126] (7) As described with reference to Fig. 12 and Fig. 13, the estimation unit 11a of the cooking device 1a according to the second embodiment compares the food image HG, which is the imaging result, with the food image MG stored in the storage unit 20a, and estimates the name of the heated object based on the comparison result. However, as long as the estimation unit 11a can estimate the name of the heated object, the estimation unit 11a may compare the difference between the food images HG, which are the imaging results, with the difference between the food images MG stored in the storage unit 20a, and estimate the name of the heated object based on the comparison result. In this case, the difference between the food images MG stored in the storage unit 20a refers to the difference between the food images MG showing a food at different times during the heating process of one food. [Industrial Applicability]

[0127] The present invention can be used in the fields of cooking appliances and trained model generation methods. [Explanation of symbols]

[0128] 1 Cooker 11 Estimation part 12 Heating control section 13. Measurement section 14 Correction section 20 Memory section 30 Heating section 40 Imaging unit 50 Control section 60 Turntable HG Food Images Ht heating time Hs heating intensity LM pre-trained model LG Learning Food Images MG Finishing Image ST Status Information TD training data

Claims

1. A heating unit that heats the object to be heated; An imaging unit that images the object to be heated while the object to be heated is being heated; An estimation unit that estimates a name of the object to be heated based on an imaging result of the imaging unit while the object to be heated is being heated; A memory unit that stores a trained model constructed by machine learning of training data; a heating control unit that controls the heating unit in response to an estimation result of the estimation unit; Equipped with The estimation unit inputs the imaging result into the trained model, The trained model outputs name information indicating a name of the heated object based on the input imaging result, The learning data includes a plurality of imaging results of one object to be heated at different times during a process in which the object to be heated is heated, The storage unit stores a table in which a name of a dish is associated with a heating time and a heating intensity corresponding to the dish, The heating control unit is When receiving an instruction to start heating, the heating unit starts heating the object to be heated at a predetermined heating intensity regardless of the object to be heated; After starting heating of the object to be heated, a heating time and a heating intensity associated with the name of the dish in the table are determined based on the name of the dish corresponding to the name of the object to be heated estimated by the estimation unit; Based on the determined heating time and the elapsed time from the start of heating of the object to the time when the name of the object is estimated, adjust the remaining heating time for the object from the time when the name of the object is estimated to the time when heating of the object is terminated; adjusting the heating intensity of the heating unit based on the determined heating intensity; Heating cooker.

2. The learning data includes a learning target dish image showing a dish and name information showing a name of the dish, The estimation unit inputs a food image that is the imaging result into the trained model, The trained model outputs name information indicating the name of the heated object based on the input cooking image, The cooking device according to claim 1 , wherein the estimation unit acquires the name information output by the trained model.

3. The heating cooker according to claim 2 , wherein each of the learning object dish images of the plurality of learning data shows one of the dishes at different times during the process of heating the dish.

4. A storage unit is further provided for storing a plurality of dish images and names of the dishes for each dish, each of the plurality of food images shows the food at a different time during the process of the food being heated; The cooking device according to claim 1 , wherein the estimation unit compares the imaging result with the food image and estimates the name of the object to be heated based on a comparison result.

5. A turntable on which the object to be heated is placed and rotated; A measuring unit that measures the time during which the object to be heated is heated; a correction unit that corrects a direction of the imaging result based on an angular velocity when the turntable rotates and a measurement result of the measurement unit; Further equipped with The cooking device according to claim 4 , wherein the estimation unit estimates a name of the object to be heated based on the corrected imaging result.

Images