Cooking assistant system

Abstract

To provide a heat cooking assistance system that can achieve improvement of a user's cooking skills while inhibiting the user from feeling burdensome.SOLUTION: In a heat cooking assistance system 100, a cooking menu identification unit 105 identifies a cooking menu for a user to cook an ingredient F1 by heating with heating means 10 on the basis of input information. An analysis unit 101 includes a first learning unit 110, a second learning unit 120 and a third learning unit 130. The analysis unit 101 is configured to receive a cooking menu being executed, the input information and cooking state information, estimate a progress state, cooking assistance information and a notification timing by executing arithmetic processing with the first to third learning units 110 to 130, and cause notification means 53D, 54 to provide the cooling assistance information corresponding to the progress state on the basis of the notification timing.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to a cooking assistance system. [Background technology]

[0002] A cooking system that is an example of a conventional cooking assistance system is disclosed in Patent Document 1. This cooking system includes a cooking container, a heating means, a pre-heating camera, a cooking menu estimation unit, and a heat amount control unit.

[0003] The cooking vessel is a pot or the like and contains the food to be cooked. The heating means is a gas stove or the like and heats the cooking vessel. The pre-heating camera photographs the state of the food to be cooked, the cooking vessel, and the heating means before heating, and obtains a pre-heating object image.

[0004] The cooking menu estimation unit uses a trained learning device that has performed machine learning to estimate a cooking menu based on pre-heated object images to estimate a cooking menu for a user to heat and cook an object to be cooked using a heating means. The cooking menu estimation unit inputs pre-heated object images acquired from a pre-heating camera into the learning device and estimates a cooking menu by executing arithmetic processing by the trained learning device. The heat amount control unit controls the heat amount of the heating means based on the cooking menu estimated by the cooking menu estimation unit.

[0005] This allows the cooking system to perform cooking with an amount of heat appropriate for the cooking menu. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-181871 Summary of the Invention [Problem to be solved by the invention]

[0007] However, conventional cooking assistance systems such as the above-mentioned cooking system leave little room for users to think about and execute cooking menus on their own, making it difficult for users who are beginners at cooking to improve their cooking skills.

[0008] In this regard, it is possible to improve the user's cooking skills by informing the user of information related to the amount of heat, procedures, precautions, tips, know-how, tacit knowledge, etc. for each step in the cooking menu and encouraging the user to act based on that information.

[0009] However, in this case, there is a problem that the notification is likely to be an annoyance to the user. Also, even if the system is configured to allow the user to input prerequisite information to limit the information notified to only the information the user wants to know, there is a problem that the effort required to input the prerequisite information is likely to be an annoyance to the user.

[0010] The present invention has been made in consideration of the above-mentioned conventional situation, and aims to solve the problem of providing a heating and cooking support system that can improve a user's cooking skills while minimizing the inconvenience to the user. [Means for solving the problem]

[0011] The cooking assistance system of the present invention comprises: a cooking container for accommodating food to be cooked; a cooking utensil used together with the cooking container to cook the food; a heating means for heating the cooking vessel; an input interface that receives input from a user and acquires input information; an information acquisition means for acquiring cooking status information of at least one of the food to be cooked, the cooking container, the cooking utensil, and the heating means; a notification means for notifying the cooking status information; a cooking menu specification unit that specifies a cooking menu for the user to heat and cook the food using the heating means based on the input information; an analysis unit, The analysis unit a first learning unit that has performed machine learning to estimate the progress of the steps of the cooking menu currently being executed by the user based on the input information, the cooking status information, and the cooking results when the cooking menu was executed in the past; a second learning unit that has performed machine learning to estimate cooking assistance information related to the cooking menu currently being executed by the user based on the input information, the cooking status information, and the cooking results when the cooking menu was executed in the past; a third learning unit that has performed machine learning to estimate an appropriate notification timing for the user to be notified of the cooking assistance information corresponding to the cooking progress, based on the input information, the cooking status information, and the cooking results when the cooking menu was previously executed; The analysis unit The cooking menu currently being executed, the input information, and the cooking status information are input; The first learning unit, the second learning unit, and the third learning unit execute calculation processing to estimate the progress status, the cooking assistance information, and the notification timing; The cooking assistance information corresponding to the progress status is notified to the notifying means based on the notification timing.

[0012] In the cooking assistance system of the present invention, the analysis unit can accurately estimate the progress of the steps of the cooking menu being executed, cooking assistance information related to the cooking menu being executed, and notification timing suitable for the user to be notified of the cooking assistance information corresponding to the progress, through calculation processing by the first to third learning units. Then, the analysis unit causes the notification means to notify the cooking assistance information corresponding to the progress based on the notification timing.

[0013] For example, when the analysis unit receives input information requesting the next step while a cooking menu is being executed, the analysis unit can accurately estimate cooking support information regarding the next step and accurately estimate the notification timing that will result in a quick response.

[0014] Furthermore, when the analysis unit receives input information during the execution of a cooking menu requesting advice on how to cut ingredients before heating, the cooking utensils to use, the amount and ratio of seasonings, etc., it can accurately estimate cooking support information related to the advice and accurately estimate the timing of notification to ensure a quick response.

[0015] Furthermore, if the analysis unit acquires cooking status information indicating that the amount of heat applied by the heating means is excessive during the execution of a cooking menu, the cooking results of the cooking menu are likely to deteriorate, so the analysis unit can accurately estimate cooking support information regarding operations to reduce the amount of heat to an appropriate level and accurately estimate the timing of notification to ensure a quick response.

[0016] Furthermore, when the analysis unit acquires cooking status information indicating that the temperature of the cooking container is too low while a cooking menu is being executed, there is sufficient time before the cooking results of the cooking menu deteriorate, so the analysis unit can accurately estimate cooking assistance information regarding operations to raise the temperature to an appropriate level and accurately estimate the notification timing that will provide a response that allows the user sufficient time to think about and respond on their own.

[0017] On the other hand, the analysis unit can suppress notification of cooking assistance information to the user for the cooking menu being executed while the delay in progress or deterioration of the cooking results is not a problem.

[0018] This allows the user more room to think about and execute cooking menus on their own, making it easier to adjust the cooking menu to suit their preferences. It also reduces the effort required for the user to input prerequisite information to limit the cooking assistance information notified to only the information they want to know.

[0019] Therefore, the cooking assistance system of the present invention can improve the cooking skills of the user while minimizing the inconvenience felt by the user.

[0020] The cooking assistance information preferably includes information regarding tasks that the user should perform to prevent delays in progress or deterioration of cooking results.

[0021] In this case, the effectiveness of the cooking assistance information can be improved, which further reduces the user's inconvenience and further improves the user's cooking skills.

[0022] The information acquisition means preferably includes a photographing device that photographs the food, the cooking vessel, and the heating means to acquire cooking image information, and the cooking status information preferably includes the cooking image information.

[0023] In this case, the cooking image information appropriately reflects the progress of the cooking menu steps, allowing for a larger amount of cooking status information. This allows the analysis unit to more accurately estimate the progress of the cooking menu steps, cooking assistance information, and notification timing. As a result, this cooking assistance system can further improve the user's cooking skills while further reducing user inconvenience.

[0024] It is desirable that the photographing device also photographs the user who is performing the cooking menu to acquire user image information, and the cooking image information preferably includes the user image information.

[0025] In this case, the cooking image information not only appropriately reflects the progress of the cooking menu steps but also appropriately reflects the user's cooking status, thereby further increasing the amount of cooking status information. This allows the analysis unit to more accurately estimate the progress of the cooking menu steps, cooking assistance information, and notification timing. For example, if the analysis unit infers from the user image information that the user is behaving in a confused manner while performing a cooking menu because they do not know the next step, it can accurately estimate cooking assistance information related to the next step. If the cooking result of the cooking menu is likely to deteriorate, it can accurately estimate the notification timing that will allow the user to respond quickly. If there is sufficient time before the cooking result of the cooking menu deteriorates, it can accurately estimate the notification timing that will allow the user sufficient time to respond by themselves. As a result, this cooking assistance system can further reduce user annoyance while further improving the user's cooking skills.

[0026] It is desirable that the analysis unit has a user database in which information relating to multiple users is pre-registered, identifies the user currently performing a cooking menu based on user image information, and adjusts the cooking assistance information according to the identified user by referring to the user database.

[0027] In this case, the analysis unit can easily optimize the notification of cooking support information for each user by registering the age, gender, cooking skills, etc. of multiple users in advance.

[0028] The cooking menu specification unit preferably has already performed machine learning to estimate a cooking menu based on cooking image information captured in the preparation process when a cooking menu was previously executed, and preferably specifies a cooking menu based on the cooking image information of the preparation process as well.

[0029] In this case, even when the user does not input the cooking menu to be performed into the input interface, the cooking menu specification unit specifies the cooking menu based on the cooking image information of the preparation process, thereby further reducing the inconvenience felt by the user.

[0030] The input interface preferably includes a voice interface for receiving voice input from a user to obtain input information.

[0031] In this case, the user can be saved the trouble of operating a touch panel, keyboard, etc. to input information, which further reduces the inconvenience to the user.

[0032] The information acquisition means preferably has a first status sensor that detects the status of the cooking vessel, and the cooking status information preferably includes the detection result of the first status sensor.

[0033] In this case, the detection result of the first status sensor preferably reflects the progress of the cooking menu steps, so the amount of cooking status information can be increased. This allows the analysis unit to more accurately estimate the progress of the cooking menu steps, cooking assistance information, and notification timing. As a result, this cooking assistance system can further improve the user's cooking skills while further reducing the user's inconvenience.

[0034] The information acquisition means preferably includes a second status sensor provided in the cooking tool for detecting the status of the food to be cooked and / or the status of any additional ingredients to be added to the food, and the cooking status information preferably includes the detection result of the second status sensor.

[0035] In this case, the detection result of the second status sensor preferably reflects the progress of the cooking menu steps, thereby increasing the amount of cooking status information. This allows the analysis unit to more accurately estimate the progress of the cooking menu steps, cooking assistance information, and notification timing. As a result, this cooking assistance system can further improve the user's cooking skills while further reducing the user's inconvenience. [Effects of the Invention]

[0036] According to the cooking assistance system of the present invention, it is possible to improve the cooking skills of a user while minimizing the inconvenience felt by the user. [Brief explanation of the drawings]

[0037] [Figure 1] FIG. 1 is a perspective view of a cooking assistance system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram of the cooking assistance system according to the embodiment. [Figure 3] FIG. 3 is a front view of the portable information terminal. [Figure 4] FIG. 4 is a block diagram of the external server. [Figure 5] FIG. 5 is a diagram showing the software configuration of the cooking menu learning unit. [Figure 6] FIG. 6 is a diagram showing the software configuration of the cooking menu specification unit. [Figure 7] FIG. 7 is a diagram showing the software configuration of the first learning unit. [Figure 8] FIG. 8 is a diagram showing the software configuration of the second learning unit. [Figure 9] FIG. 9 is a diagram showing the software configuration of the third learning unit. [Figure 10] FIG. 10 is a diagram showing the software configuration of the analysis unit. [Figure 11] Figure 11 is a flowchart of the cooking assistant app. DETAILED DESCRIPTION OF THE INVENTION

[0038] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, specific embodiments of the present invention will be described with reference to the drawings.

[0039] (Example) 1 and 2, a cooking assistance system 100 according to the embodiment is an example of a specific aspect of the cooking assistance system of the present invention. The cooking assistance system 100 includes a cooking appliance 10, a cooking container 20, a ladle 30A, a cooked state detection rod 30B, a photographing device 40, and a mobile information terminal 50.

[0040] The cooking appliance 10 is an example of a "heating means" of the present invention. The ladle 30A and the cookedness detection rod 30B are each an example of a "cooking utensil" of the present invention. The photographing device 40 is an example of an "information acquisition means" of the present invention.

[0041] The cooking appliance 10 is a gas stove installed in the system kitchen of the kitchen of the house H1. The photographing device 40 is suspended from the ceiling or the frame of the range hood of the kitchen of the house H1 and is located above the cooking appliance 10. The photographing device 40 may also be built into the range hood.

[0042] <Mobile information terminal> The mobile information terminal 50 is a smartphone, a mobile tablet terminal, or the like carried by a user residing in the house H1. As shown in FIG. 2 , the mobile information terminal 50 has a control unit 51, a touch panel 53, a speaker 54, a microphone 55, and a terminal communication unit 57.

[0043] The control unit 51 is an electronic circuit unit including a CPU (not shown), a storage unit 51M including storage elements such as ROM and RAM, an interface circuit, etc. The control unit 51 executes control processing related to the operation of the mobile information terminal 50.

[0044] The storage unit 51M stores various programs and setting information for operating the mobile information terminal 50. The storage unit 51M also stores various information acquired by the control unit 51 as appropriate. The programs stored in the storage unit 51M include an application program for operating the cooking assistance system 100 (hereinafter referred to as a "cooking assistant app").

[0045] The control unit 51 has a cooking control unit 52. The cooking control unit 52 functions when the cooking assistant app is executed on the mobile information terminal 50. The cooking control unit 52 will be described in detail later.

[0046] The touch panel 53 has a display unit 53D that displays various information such as text and images, and an input unit 53A that covers the display unit 53D in a manner that allows the user to see it and receives various inputs by touching it with the user's fingertips.

[0047] The speaker 54 outputs the voice of the other party when using the telephone, and can also output various types of information as voice when various application programs are executed.

[0048] The microphone 55 picks up the user's voice when using the telephone, and can also receive voice input from the user when various application programs are executed.

[0049] The input unit 53A when the cooking assistant app is executed on the mobile information terminal 50 is an example of the "input interface" of the present invention.

[0050] The microphone 55 used when the cooking assistant app is executed on the mobile information terminal 50 is an example of the "input interface" and "voice interface" of the present invention.

[0051] The display unit 53D and the speaker 54 when the cooking assistant app is executed on the mobile information terminal 50 are an example of the "notification means" of the present invention.

[0052] The terminal communication unit 57 is capable of making calls using a frequency band for mobile phones, and also has built-in electronic circuits that perform short-range wireless communication using Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), etc.

[0053] The terminal communication unit 57 is connected to an external network NW1 via a wireless router 7 installed in the house H1. The terminal communication unit 57 can perform network communication with an external server 9, a plurality of information processing terminals 8, etc. via the wireless router 7 and the network NW1.

[0054] The cooking assistance system 100 includes an external server 9 as an information processing terminal that cooperates with the mobile information terminal 50. In the cooking assistance system 100, the external server 9 performs complex calculations involving large amounts of information on behalf of the cooking control unit 52 of the mobile information terminal 50, thereby reducing the calculation load on the cooking control unit 52.

[0055] The external server 9 also has a storage unit 9M. The storage unit 9M stores various information for supporting the execution of the cooking assistance system 100. The control unit 51 can download a cooking assistant app via network communication with the external server 9 and store it in the storage unit 51M.

[0056] As shown in Figure 1, when a user places food F1 to be cooked, such as ingredients for a stew, water for the broth, various seasonings, etc., in a cooking container 20 and cooks the food using a cooking appliance 10, the user executes a cooking assistant app on a mobile information terminal 50 to receive assistance from the cooking assistance system 100.

[0057] Then, as shown in Figure 3, the display unit 53D of the touch panel 53 is controlled by the cooking control unit 52 to display cooking assistance information to assist in the cooking of the food F1, such as an image of a graph showing the container temperature changing over time.

[0058] The display section 53D also displays a detection button 53B1 related to the ladle 30A and a detection button 53B2 related to the cookedness detection rod 30B.

[0059] The speaker 54 is also controlled by the cooking control unit 52 as necessary to notify cooking assistance information by voice.

[0060] The cooking assistance information notified by the display unit 53D and the speaker 54 includes various information that is considered to be useful for improving the completeness of the cooking menu, such as information regarding the cooking menu that the user performs to cook the food F1 using the heating cooking appliance 10, and information regarding the tasks that the user should perform to prevent delays in progress or deterioration of the cooking results.

[0061] The input unit 53A of the touch panel 53 receives input from the user who is cooking the food F1, acquires input information, and transmits the input information to the cooking control unit 52.

[0062] The microphone 55 is also controlled by the cooking control unit 52 as necessary, receives voice input from the user cooking the food F1, acquires input information, and transmits the input information to the cooking control unit 52.

[0063] The input information acquired by the input unit 53A and microphone 55 of the touch panel 53 includes various information that is considered useful for the heating cooking assistance system 100 in providing cooking assistance, such as questions about the cooking menu to be performed by the user, the name of the cooking menu, the ingredients actually prepared for the cooking menu and their weight, and the next steps to be taken while the cooking menu is being performed.

[0064] <Heating cooking equipment> As shown in Fig. 2, the cooking appliance 10 has a heating unit 11, a heating control unit 12, a container temperature sensor 16, a display operation unit 13, and a first communication unit 17. The heating control unit 12 is an example of the "information acquisition means" of the present invention. The container temperature sensor 16 is an example of the "information acquisition means" and the "first status sensor" of the present invention.

[0065] As shown in Fig. 1, the heating unit 11 is a gas burner that burns fuel gas, such as city gas, supplied from a fuel gas supply source. Although not shown, the heating unit 11 includes an ignition device and a fuel gas control valve that controls the supply of fuel gas, adjusting the amount of supply, and stopping the supply. The heating unit 11 is located in the center of a trivet located on the top surface of the cooking appliance 10. When activated, the heating unit 11 heats a cooking container 20 placed on the trivet.

[0066] In addition, the heating cooking appliance 10 has another heating section similar to the heating section 11 on the top surface of the heating cooking appliance 10, and also has a grill chamber inside the heating cooking appliance 10, but the description of this will be omitted in this embodiment.

[0067] The container temperature sensor 16 is located in the center of the burner top of the heating unit 11. The container temperature sensor 16 is a well-known temperature sensor that uses a thermistor or the like and is housed in a metal cylindrical body that can come into contact with the bottom of the cooking container 20. The container temperature sensor 16 comes into contact with the bottom of the cooking container 20 placed on the trivet and detects the temperature information of the cooking container 20.

[0068] The temperature information of the cooking container 20 may be real-time raw data of the container temperature of the cooking container 20, or may be, for example, historical data that combines a time axis with changes in the container temperature, or processed data obtained by extracting a portion of the data. Furthermore, the heating control unit 12 or the cooking control unit 52 that acquires the temperature information of the cooking container 20 may process the temperature information. In this embodiment, the temperature information of the cooking container 20, which is the detection result of the container temperature sensor 16, is real-time raw data of the container temperature of the cooking container 20.

[0069] The container temperature sensor 16 acquires temperature information of the cooking container 20 as cooking state information of the cooking container 20. The temperature information of the cooking container 20 detected by the container temperature sensor 16 is an example of the "state of the cooking container" in the present invention.

[0070] The display operation unit 13 has an operation knob, a 7-segment LED, a push button, etc., which are arranged on the front surface of the cooking appliance 10.

[0071] As shown in FIG. 2, the heating control unit 12 receives operation inputs made by the user to the operation knobs and the like of the display operation unit 13, and controls the fuel gas control valve and the ignition device, thereby controlling the gas combustion by the heating unit 11.

[0072] The display operation unit 13 is used to set the timer and display the remaining time, and is also used to set the cooking mode and display the current mode.

[0073] The heating control unit 12 acquires cooking status information of the heating cooking appliance 10. The cooking status information of the heating cooking appliance 10 is, for example, information on the start of heating by the heating unit 11, heating output, heating end, combustion abnormality, operation input received by the display operation unit 13, etc.

[0074] As shown in Fig. 1, the first communication unit 17 is built into and disposed on the front side of the cooking appliance 10. As shown in Fig. 2, the first communication unit 17 performs short-range wireless communication. In this embodiment, the first communication unit 17 performs short-range wireless communication using Bluetooth (registered trademark), which consumes very little power. The first communication unit 17 is paired with a terminal communication unit 57 of the mobile information terminal 50, and performs short-range wireless communication with the terminal communication unit 57.

[0075] When the cooking assistant app is executed, the cooking status information of the heating cooking appliance 10 acquired by the heating control unit 12 and the cooking status information of the cooking container 20 acquired by the container temperature sensor 16 are transmitted to the cooking control unit 52 via the first communication unit 17 and the terminal communication unit 57, and further transmitted to the external server 9 via the wireless router 7.

[0076] As shown in FIG. 3, the display section 53D of the touch panel 53 displays the detection result (container temperature) of the container temperature sensor 16 below the detection button 53B1.

[0077] <Cooking container> In Figure 1, cooking container 20 is shown as an unlidded pot as an example, but cooking container 20 may be a pot with a lid, a frying pan, or any other type of cooking container as long as it has a shape that can accommodate the food F1 to be cooked.

[0078] 2, cooking container 20 has container temperature sensor 26 and second communication unit 27. Container temperature sensor 26 is an example of the "information acquisition means" and the "first status sensor" of the present invention.

[0079] The container temperature sensor 26 and the second communication unit 27 are powered by a battery (not shown) built into the cooking container 20. The container temperature sensor 26 and the second communication unit 27 are normally in a standby state in which power consumption is minimal, and enter an active state when the cooking assistant app is executed on the mobile information terminal 50. The cooking container 20 may have a power switch that can be configured to switch between a power-off state and a standby or active state.

[0080] 1, the container temperature sensor 26 is built into the lower part of the cooking container 20. The container temperature sensor 26 is a well-known temperature sensor that uses a thermistor or the like.

[0081] When the container temperature sensor 26 is activated, it detects temperature information of the cooking container 20 as cooking state information of the cooking container 20. The temperature information of the cooking container 20 detected by the container temperature sensor 26 is an example of the "state of the cooking container" in the present invention.

[0082] The second communication unit 27 is disposed inside the handle of the cooking container 20. As shown in FIG. 2, the second communication unit 27 performs short-range wireless communication. In this embodiment, the second communication unit 27 performs short-range wireless communication using Bluetooth (registered trademark), similar to the first communication unit 17. The second communication unit 27 performs short-range wireless communication with the terminal communication unit 57 of the mobile information terminal 50 by pairing with the terminal communication unit 57.

[0083] When the cooking assistant app is executed, the cooking status information of the cooking container 20 acquired by the container temperature sensor 26 is transmitted to the cooking control unit 52 via the second communication unit 27 and the terminal communication unit 57, and further transmitted to the external server 9 via the wireless router 7.

[0084] <Ladle> 1, the ladle 30A is used to cook food F1 together with the cooking vessel 20. The ladle 30A is a so-called ladle consisting of a handle and a scooping part connected to the lower end of the handle.

[0085] 2, the ladle 30A has a weighing sensor 36A and a third communication unit 37A. The weighing sensor 36A is an example of the "information acquisition means" and the "second status sensor" of the present invention.

[0086] The weighing sensor 36A and the third communication unit 37A are powered by a battery (not shown) built into the ladle 30A. The weighing sensor 36A and the third communication unit 37A are normally in a standby state in which power consumption is minimal. When the cooking assistant app is executed on the mobile information terminal 50 and the user presses the detection button 53B1 on the touch panel 53 shown in FIG. 3, the weighing sensor 36A and the third communication unit 37A enter an active state for a predetermined period of time. The ladle 30A may have a power switch configured to switch between a power-off state, a standby state, and an active state. A vibration sensor or a contact detection sensor may be provided on the handle of the ladle 30A to detect when the user lifts the ladle 30A and switch between the standby state and the active state.

[0087] When an operation input to activate the weighing sensor 36A is made by voice input to the microphone 55, the cooking control unit 52 may activate the weighing sensor 36A and the third communication unit 37A for a predetermined period of time instead of operating the detection button 53B1 on the touch panel 53.

[0088] As shown in Fig. 1, the weighing sensor 36A is located in the middle of the handle of the ladle 30A. The weighing sensor 36A is a well-known weighing sensor that uses a strain gauge or the like. The weighing sensor 36A performs weighing when activated.

[0089] The third communication unit 37A is built into the upper end of the ladle 30A. As shown in FIG. 2, the third communication unit 37A performs short-range wireless communication. In this embodiment, the third communication unit 37A performs short-range wireless communication using Bluetooth (registered trademark), similar to the first communication unit 17. The third communication unit 37A performs short-range wireless communication with the terminal communication unit 57 of the mobile information terminal 50 by pairing with the terminal communication unit 57.

[0090] As shown in FIG. 3, the display unit 53D of the touch panel 53 displays the detection result (measurement value) of the weighing sensor 36A near the detection button 53B1.

[0091] The user can reset the weight value to zero by pressing detection button 53B1 while holding the handle of ladle 30A and suspending the scooping part in the air, and then measure the food F1 by scooping it with the scooping part of ladle 30A, or by dripping seasoning onto the scooping part of ladle 30A to measure the seasoning to be added to the food F1. The weight value may be reset by estimating the user's behavior in holding ladle 30A based on cooking image information captured by photographing device 40, which will be described later.

[0092] The weighing sensor 36A detects weight information of the food F1 and / or additional ingredients to be added to the food F1 as cooking state information of the food F1 and the ladle 30A. The weight information of the food F1 and / or additional ingredients to be added to the food F1 detected by the weighing sensor 36A is an example of the "state of the food F1 and / or the state of the ingredients to be added to the food F1" in the present invention.

[0093] When the cooking assistant app is executed, the weight information of the food F1 and / or additional ingredients to be added to the food F1 obtained by the weighing sensor 36A is transmitted to the cooking control unit 52 via the third communication unit 37A and the terminal communication unit 57, and further transmitted to the external server 9 via the wireless router 7.

[0094] <Cooking condition detection rod> As shown in FIG. 1, the cookedness detector rod 30B is used together with the cooking vessel 20 to cook food F1.

[0095] The cooked state detector rod 30B has a cylindrical handle and a small-diameter rod-like detector that protrudes downward from the lower end of the handle. The lower end of the detector is tapered.

[0096] 2, the cooked state detection rod 30B has a temperature sensor 36B and a third communication unit 37B. The temperature sensor 36B is an example of the "information acquisition means" and the "second status sensor" of the present invention.

[0097] The temperature sensor 36B and the third communication unit 37B are powered by a battery (not shown) built into the cooktop 30B. The temperature sensor 36B and the third communication unit 37B are normally in a standby state in which power consumption is minimal. When the cooking assistant app is executed on the mobile information terminal 50 and the user presses the detection button 53B2 on the touch panel 53 shown in FIG. 3, the temperature sensor 36B and the third communication unit 37B enter an active state for a predetermined period of time. The cooktop 30B may have a power switch configured to switch between a power-off state, a standby state, and an active state. A vibration sensor or a contact detection sensor may be provided on the handle of the cooktop 30B to detect when the user lifts the cooktop 30B and switch between the standby state and the active state.

[0098] When an operation input to activate the temperature sensor 36B is made by voice input to the microphone 55, the cooking control unit 52 may activate the temperature sensor 36B and the third communication unit 37B for a predetermined period of time instead of operating the detection button 53B2 on the touch panel 53.

[0099] As shown in Figure 1, temperature sensor 36B is built into the lower end of the detection part of cookedness detection rod 30B. Temperature sensor 36B is a well-known temperature sensor that uses a thermistor or the like. When activated, temperature sensor 36B detects the cookedness (°C) of the ingredients of food F1 by piercing the detection part into the ingredients of food F1.

[0100] The third communication unit 37B is disposed inside the upper end of the handle of the cookedness detection rod 30B. As shown in FIG. 2, the third communication unit 37B performs short-range wireless communication. In this embodiment, the third communication unit 37B performs short-range wireless communication using Bluetooth (registered trademark), similar to the first communication unit 17. The third communication unit 37B performs short-range wireless communication with the terminal communication unit 57 of the mobile information terminal 50 by pairing with the terminal communication unit 57.

[0101] As shown in FIG. 3, the display section 53D of the touch panel 53 displays the detection result (cooking level) of the temperature sensor 36B near the detection button 53B2.

[0102] The user presses the detection button 53B2 to activate the temperature sensor 36B, and then inserts the detection part of the cookedness detection rod 30B into the ingredients of the food item F1 to find out the cookedness (℃) of the ingredients of the food item F1.

[0103] The temperature sensor 36B detects information about the degree of doneness of the food F1 as cooking state information for the food F1 and the cookedness detection rod 30B. The information about the degree of doneness of the food F1 detected by the temperature sensor 36B is an example of "the state of the food F1 and / or the state of any additional ingredients added to the food F1" in the present invention.

[0104] When the cooking assistant app is running, information on the degree of doneness of the food F1 obtained by the temperature sensor 36B is transmitted to the cooking control unit 52 via the third communication unit 37B and the terminal communication unit 57, and further transmitted to the external server 9 via the wireless router 7.

[0105] <Photographic equipment> As shown in FIG. 1, the photographing device 40 has a power cable 40C that is connected to an outlet in the house H1.

[0106] As shown in Fig. 2, the photographing device 40 has a camera 46 and a fourth communication unit 47. The camera 46 and the fourth communication unit 47 are in a standby state with minimal power consumption when the power cable 40C is connected to an outlet, and are activated when a cooking assistant app is executed on the mobile information terminal 50. The photographing device 40 may have a power switch and be configured to switch between a power "off" state, a standby state, and an active state. A human presence sensor that detects whether or not a person is present in the kitchen space may be provided, and the camera 46 and the fourth communication unit 47 may be configured to switch between the standby state and the active state when the human presence sensor detects a person.

[0107] 1, the camera 46 can photograph the cooking appliance 10, the cooking container 20, and the food F1. The camera 46 can also photograph the upper surface of the system kitchen in which the cooking appliance 10 is installed, and cooking utensils placed on the upper surface, such as a ladle 30A, a cooking state detection rod 30B, a cutting board, a knife, and the like, as well as ingredients to be cooked, seasonings, etc. (not shown). The camera 46 can also photograph a user performing a cooking menu around the cooking appliance 10.

[0108] The fourth communication unit 47 is built in and disposed near the outer peripheral surface of the photographing device 40. As shown in FIG. 2 , the fourth communication unit 47 performs short-range wireless communication. In this embodiment, the fourth communication unit 47 performs short-range wireless communication using Bluetooth (registered trademark), similar to the first communication unit 17 and the like. The fourth communication unit 47 performs short-range wireless communication with the terminal communication unit 57 of the mobile information terminal 50 by pairing with the terminal communication unit 57.

[0109] As shown in FIG. 1, the photographing device 40 photographs the food F1, cooking container 20, and cooking appliance 10 using a camera 46, and obtains cooking image information as cooking status information of the food F1, cooking container 20, and cooking appliance 10.

[0110] The cooking image information includes image information of the preparation process for identifying a cooking menu. The cooking image information is also image information for analyzing the progress of the cooking menu process, and includes image information of the food F1, cooking utensils placed on the top surface of the system kitchen, cutting board, knife, ingredients to be cooked, seasonings, etc.

[0111] The photographing device 40 also photographs the user who is performing the cooking menu with the camera 46, and acquires user image information. The cooking image information includes the user image information.

[0112] The user image information includes a facial image for identifying the user, an image of the user's hands that reflects the user's cooking actions, and a full-body image.

[0113] When the cooking assistant app is executed, the cooking image information (including user image information) acquired by the photographing device 40 is transmitted to the cooking control unit 52 via the fourth communication unit 47 and the terminal communication unit 57, and further transmitted to the external server 9 via the wireless router 7.

[0114] <External server> As shown in FIG. 2, the external server 9 has a control unit which is an electronic circuit unit configured with a CPU, ROM, RAM, interface circuit, etc. (not shown), and is capable of high-speed, large-volume information processing.

[0115] The storage unit 9M may be an external storage device such as a hard disk drive or a memory card, or may be a non-volatile memory or the like built into the external server 9. The storage unit 9M is used as a long-term storage area for storing various pieces of information for the long term in response to a write command from the control unit, and for transmitting the information to the control unit in response to a read command from the control unit.

[0116] Furthermore, the external server 9 has a communication interface circuit and is capable of performing network communication with multiple information processing terminals 8 etc. via the network NW1, and of performing network communication with the mobile information terminal 50 via the network NW1 and the wireless router 7.

[0117] It should be noted that, with regard to the specific hardware configuration of the external server 9, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit of the external server 9 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, an FPGA (field-programmable gate array), or the like. The external server 9 may be configured with multiple information processing devices. Furthermore, the external server 9 may be an information processing device designed specifically for the services provided, as well as a general-purpose server device, a PC (Personal Computer), or the like. Furthermore, the external server 9 may be virtualized and constructed on a network.

[0118] As shown in FIG. 4, the external server 9 includes a cooking menu specification unit 105 and an analysis unit 101 in the cooking assistance system 100, which perform complex arithmetic processing involving a large amount of information.

[0119] The cooking menu specification unit 105 executes a process for specifying a cooking menu for the user to heat and cook the food F1 using the cooking appliance 10, and includes a cooking menu learning unit 150.

[0120] The analysis unit 101 includes a first learning unit 110, a second learning unit 120, a third learning unit 130, and a user database 140.

[0121] Information relating to multiple users is pre-registered in the user database 140. The information relating to multiple users includes identification information such as a facial image for identifying each user, and information such as each user's cooking skills and preferences.

[0122] <Cooking Menu Learning Department> As shown in FIG. 5, the cooking menu learning unit 150 has a learning data acquiring unit 150A and a learning processing unit 150B.

[0123] The learning data acquiring unit 150A acquires, as learning data 159, a set of input information 61T, cooking image information 68TP for the preparation process, and cooking menu information 69T corresponding to the input information 61T and cooking image information 68TP for the preparation process.

[0124] In addition, the preparation process is generally a process that precedes the heating process in which the food to be cooked F1 is heated using the heating cooking appliance 10, but if the next preparation is carried out in parallel with the heating process, the next preparation is also included in the preparation process.

[0125] The input information 61T and the preparation step cooking image information 68TP are used as input data, and the cooking menu information 69T is used as training data (correct answer data).

[0126] The input information 61T is input information previously acquired by the input unit 53A and microphone 55 of the touch panel 53, or input information acquired when multiple subjects performed cooking menus during the development stage of the cooking assistance system 100.

[0127] The cooking image information 68TP of the preparation process is cooking image information captured by the photographing device 40 in the preparation process when a cooking menu was executed in the past, or cooking image information captured in the preparation process when multiple subjects executed a cooking menu during the development stage of the cooking assistance system 100. The cooking image information 68TP of the preparation process includes user image information acquired by the photographing device 40 of a user performing work in the preparation process.

[0128] The cooking menu information 69T is cooking menu information when a cooking menu was executed in the past or cooking menu information when a plurality of subjects executed a cooking menu during the development stage of the cooking assistance system 100.

[0129] When the learning processing unit 150B receives the input information 61T and the cooking image information 68TP of the preparation step, it performs machine learning of the cooking menu learning unit 150 so as to output an output value corresponding to the cooking menu information 69T.

[0130] Cooking menu learning unit 150 has neural network 151T. Neural network 151T is configured by combining multiple types of neural networks.

[0131] In this embodiment, the neural network 151T includes a fully connected neural network 152T, a convolutional neural network 154T, a connection layer 156T, and an LSTM network 158T.

[0132] The fully connected neural network 152T and the convolutional neural network 154T are arranged in parallel on the input side of the neural network 151T.

[0133] Input information 61T is input to the fully connected neural network 152T. Cooking image information 68TP for the preparation process is input to the convolutional neural network 154T.

[0134] The coupling layer 156T couples the output of the fully coupled neural network 152T with the output of the convolutional neural network 154T. The LSTM network 158T receives the output from the coupling layer 156T and outputs an output value corresponding to the cooking menu information 69T.

[0135] The fully connected neural network 152T is a so-called multi-layered neural network, and includes, in order from the input side, an input layer 152TA, an intermediate layer (hidden layer) 152TB, and an output layer 152TC. However, the number of layers of the fully connected neural network 152T is not limited to this example, and may be selected appropriately depending on the embodiment.

[0136] Each layer 152TA, 152TB, 152TC of the fully connected neural network 152T has one or more neurons (nodes). The number of neurons included in each layer may be set appropriately depending on the embodiment. Each neuron included in each layer is connected to all neurons included in the adjacent layer, thereby forming the fully connected neural network 152T. A weight (connection load) is set appropriately for each connection.

[0137] The convolutional neural network 154T is a forward propagation neural network having a structure in which convolutional layers 154TA and pooling layers 154TB are alternately connected. In the convolutional neural network 154T according to this embodiment, a plurality of convolutional layers 154TA and pooling layers 154TB are alternately arranged on the input side. The output of the pooling layer 154TB arranged closest to the output side is input to a fully connected layer 154TC, and the output of the fully connected layer 154TC is input to an output layer 154TD.

[0138] The convolution layer 154TA is a layer that performs image convolution calculations. Image convolution corresponds to the process of calculating the correlation between an image and a specified filter. Therefore, by performing image convolution, for example, it is possible to detect a grayscale pattern similar to the grayscale pattern of the filter from the input image.

[0139] The pooling layer 154TB is a layer that performs pooling processing. The pooling processing discards some of the information at positions where the response to the image filter was strong, thereby realizing invariance of the response to minute changes in the position of features that appear in the image.

[0140] The fully connected layer 154TC is a layer that connects all neurons between adjacent layers. That is, each neuron included in the fully connected layer 154TC is connected to all neurons included in the adjacent layer. The fully connected layer 154TC may be composed of two or more layers. The number of neurons included in the fully connected layer 154TC may be set appropriately depending on the embodiment.

[0141] The output layer 154TD is the layer located at the most output side of the convolutional neural network 154T. The number of neurons included in the output layer 154TD may be set appropriately depending on the embodiment. Note that the configuration of the convolutional neural network 154T is not limited to this example, and may be set appropriately depending on the embodiment.

[0142] The coupling layer 156T is disposed between the fully connected neural network 152T, the convolutional neural network 154T, and the LSTM network 158T. The coupling layer 156T couples the output from the output layer 152TC of the fully connected neural network 152T with the output from the output layer 154TD of the convolutional neural network 154T. The output of the coupling layer 156T is input to the LSTM network 158T. The number of neurons included in the coupling layer 156T may be set appropriately depending on the number of outputs of the fully connected neural network 152T and the number of outputs of the convolutional neural network 154T.

[0143] The LSTM network 158T is a recurrent neural network including an LSTM block 158TB. A recurrent neural network is a neural network that has an internal loop, such as a path from a hidden layer to an input layer. The LSTM network 158T has a structure in which the hidden layer of a general recurrent neural network is replaced with the LSTM block 158TB.

[0144] The LSTM network 158T includes, in order from the input side, an input layer 158TA, an LSTM block 158TB, and an output layer 158TC, and has a path from the LSTM block 158TB back to the input layer 158TA in addition to a forward propagation path. The number of neurons included in the input layer 158TA and the output layer 158TC may be set appropriately depending on the embodiment.

[0145] The LSTM block 158TB is a block that includes an input gate and an output gate and is configured to be able to learn the timing of storing and outputting information. The LSTM block 158TB may also include a forget gate that adjusts the timing of forgetting information. The configuration of the LSTM network 158T can be set appropriately depending on the embodiment.

[0146] A threshold is set for each neuron, and the output of each neuron is basically determined depending on whether the sum of the products of each input and each weight exceeds the threshold.

[0147] The learning processing unit 150B inputs input information 61T to the fully connected neural network 152T and inputs preparation process cooking image information 68TP to the convolutional neural network 154T. The learning processing unit 150B then determines whether each neuron included in each layer fires, starting from the input side. In this way, the neural network 151T performs arithmetic processing on the input. The learning processing unit 150B obtains an output value corresponding to cooking menu information 69T from the output layer 158TC of the neural network 151T.

[0148] The learning processing unit 150B stores information indicating the configuration of the neural network 151T constructed in this way, the weights of the connections between the neurons, and the thresholds of the neurons, as cooking menu learning result data 159R in the storage unit 9M.

[0149] In other words, the cooking menu learning unit 150 has already performed machine learning to estimate a cooking menu based on input information 61T when a cooking menu was executed in the past and cooking image information 68TP captured during the preparation process when the cooking menu was executed in the past.

[0150] The cooking menu learning unit 150 periodically acquires learning data 159 from the cooking assistance system 100 via the network NW1 and updates the cooking menu learning result data 159R. The cooking menu learning unit 150 may also periodically acquire learning data 159 from each information processing terminal 8 via the network NW1 and update the cooking menu learning result data 159R.

[0151] <Cooking menu specification section> As shown in FIG. 6, the cooking menu identification unit 105 has a neural network 151 that uses cooking menu learning result data 159R from the cooking menu learning unit 150.

[0152] Like the neural network 151T of the cooking menu learning unit 150, the neural network 151 is configured by combining a plurality of types of neural networks.

[0153] In this embodiment, the neural network 151 includes a fully connected neural network 152, a convolutional neural network 154, a connection layer 156, and an LSTM network 158.

[0154] The specific configurations of the fully connected neural network 152, the convolutional neural network 154, the connection layer 156, and the LSTM network 158 are similar to those of the fully connected neural network 152T, the convolutional neural network 154T, the connection layer 156T, and the LSTM network 158T associated with the neural network 151T of the cooking menu learning unit 150, and therefore will not be described here.

[0155] When a user executes a cooking menu using the heating cooking support system 100, the neural network 151 inputs input information 61 to the fully connected neural network 152 and inputs cooking image information 68P of the preparation process to the convolutional neural network 154, thereby outputting cooking menu information 69 corresponding to the input information 61 and the cooking image information 68P of the preparation process, and identifying the cooking menu to be executed by the user.

[0156] The input information 61 is input information acquired by the input unit 53A of the touch panel 53 and the microphone 55 while the cooking menu is being executed. The cooking image information 68P of the preparation process is cooking image information captured by the photographing device 40 during the preparation process after the cooking menu has started.

[0157] When the user inputs the name of a cooking menu to be executed, the input becomes input information 61. Then, neural network 151 outputs cooking menu information 69 corresponding to the input cooking menu name, and identifies the cooking menu to be executed by the user.

[0158] If the user does not input the name of the cooking menu to be performed, the neural network 151 estimates the cooking menu to be performed by the user based on the input information 61 and the cooking image information 68P of the preparation process, outputs cooking menu information 69, and identifies the cooking menu to be performed by the user.

[0159] <First Study Section> As shown in FIG. 7, the first learning unit 110 includes a learning data acquisition unit 110A and a learning processing unit 110B.

[0160] The learning data acquisition unit 110A acquires as learning data 119 a set of cooking menu information 69U, input information 61T, cooking status information 62T, and cooking image information 68TA, and a progress status 71T of the cooking menu process corresponding to the cooking menu information 69U, input information 61T, cooking status information 62T, and cooking image information 68TA.

[0161] Cooking menu information 69U, input information 61T, cooking status information 62T, and cooking image information 68TA are used as input data. Progress status 71T is used as training data (correct answer data).

[0162] Cooking menu information 69U is information about each of a large number of cooking menus, including the name of the cooking menu, details of the recipe, and the like.

[0163] The input information 61T is input information acquired by the input section 53A and microphone 55 of the touch panel 53 when a cooking menu related to the cooking menu information 69U was executed in the past, or input information acquired when multiple subjects executed the cooking menu during the development stage of the heating cooking support system 100.

[0164] The cooking status information 62T is cooking status information acquired by the information acquisition means (heating control unit 12, container temperature sensors 16, 26, weighing sensor 36A, temperature sensor 36B) when a cooking menu related to cooking menu information 69U was executed in the past, or cooking status information acquired by the information acquisition means when multiple subjects executed the cooking menu during the development stage of cooking assistance system 100. The cooking status information 62T also includes the cooking results when the cooking menu related to cooking menu information 69U was executed. The cooking status information 62T does not include image information.

[0165] The cooking image information 68TA is cooking image information photographed by the photographing device 40 when the cooking menu related to the cooking menu information 69U was executed in the past, or cooking image information photographed when multiple subjects executed the cooking menu during the development stage of the heating cooking support system 100.

[0166] The cooking image information 68TA includes user image information 67T. The user image information 67T is user image information captured by the image capturing device 40 when a cooking menu related to the cooking menu information 69U was performed in the past, or user image information captured when multiple subjects performed the cooking menu during the development stage of the cooking assistance system 100.

[0167] When the learning processing unit 110B inputs cooking menu information 69U, input information 61T, cooking status information 62T, and cooking image information 68TA, it performs machine learning on the first learning unit 110 so that it outputs an output value corresponding to the progress status 71T of the cooking menu process.

[0168] The first learning unit 110 has a neural network 111T. The neural network 111T is configured by combining multiple types of neural networks.

[0169] In this embodiment, the neural network 111T includes a fully connected neural network 112T, a convolutional neural network 114T, a connection layer 116T, and an LSTM network 118T.

[0170] The specific configurations of the fully connected neural network 112T, the convolutional neural network 114T, the connection layer 116T, and the LSTM network 118T are similar to those of the fully connected neural network 152T, the convolutional neural network 154T, the connection layer 156T, and the LSTM network 158T associated with the neural network 151T of the cooking menu learning unit 150, and therefore will not be described here.

[0171] The learning processing unit 110B inputs cooking menu information 69U, input information 61T, and cooking status information 62T to the fully connected neural network 112T, and inputs cooking image information 68TA to the convolutional neural network 114T. The learning processing unit 110B then performs firing determinations for each neuron included in each layer, starting from the input side. In this way, the neural network 111T performs arithmetic processing on the input. The learning processing unit 110B obtains an output value corresponding to the progress status 71T from the output layer of the LSTM network 118T of the neural network 111T.

[0172] The learning processing unit 110B stores information indicating the configuration of the neural network 111T constructed in this way, the weights of the connections between the neurons, and the thresholds of the neurons, as first learning result data 119R in the storage unit 9M.

[0173] In other words, the first learning unit 110 has already performed machine learning to estimate the progress of the steps of the cooking menu currently being executed by the user based on the cooking menu information 69U, input information 61T, cooking status information 62T (including the cooking result) and cooking image information 68TA (including user image information 67T) when the cooking menu related to the cooking menu information 69U was executed in the past.

[0174] The first learning unit 110 periodically acquires learning data 119 from the cooking assistance system 100 via the network NW1 and updates the first learning result data 119R. The first learning unit 110 may also periodically acquire learning data 119 from each information processing terminal 8 via the network NW1 and update the first learning result data 119R.

[0175] <Second Study Section> As shown in FIG. 8, the second learning unit 120 includes a learning data acquisition unit 120A and a learning processing unit 120B.

[0176] The learning data acquisition unit 120A acquires as learning data 129 a set of cooking menu information 69U, input information 61T, cooking status information 62T, and cooking image information 68TA, and cooking support information 72T corresponding to the cooking menu information 69U, input information 61T, cooking status information 62T, and cooking image information 68TA.

[0177] The cooking menu information 69U, input information 61T, cooking state information 62T, and cooking image information 68TA are used as input data. The cooking support information 72T is used as training data (correct answer data).

[0178] The cooking menu information 69U, input information 61T, cooking status information 62T, and cooking image information 68TA are as described above.

[0179] The cooking support information 72T includes various information that is considered useful for improving the completeness of the cooking menu, such as information regarding tasks that the user should perform to prevent delays in progress or deterioration of cooking results.

[0180] When the learning processing unit 120B receives the cooking menu information 69U, the input information 61T, the cooking state information 62T, and the cooking image information 68TA, it performs machine learning on the second learning unit 120 so that it outputs an output value corresponding to the cooking assistance information 72T.

[0181] The second learning unit 120 has a neural network 121T. The neural network 121T is configured by combining a plurality of types of neural networks.

[0182] In this embodiment, the neural network 121T includes a fully connected neural network 122T, a convolutional neural network 124T, a connection layer 126T, and an LSTM network 128T.

[0183] The specific configurations of the fully connected neural network 122T, the convolutional neural network 124T, the connection layer 126T, and the LSTM network 128T are similar to those of the fully connected neural network 152T, the convolutional neural network 154T, the connection layer 156T, and the LSTM network 158T associated with the neural network 151T of the cooking menu learning unit 150, and therefore will not be described here.

[0184] The learning processing unit 120B inputs cooking menu information 69U, input information 61T, and cooking state information 62T to the fully connected neural network 122T, and inputs cooking image information 68TA to the convolutional neural network 124T. The learning processing unit 120B then determines whether each neuron included in each layer is fired, starting from the input side. In this way, the neural network 121T performs arithmetic processing on the input. The learning processing unit 120B obtains an output value corresponding to cooking assistance information 72T from the output layer of the LSTM network 128T of the neural network 121T.

[0185] The learning processing unit 120B stores information indicating the configuration of the neural network 121T constructed in this way, the weights of the connections between the neurons, and the thresholds of the neurons, as second learning result data 129R in the storage unit 9M.

[0186] In other words, the second learning unit 120 has already performed machine learning to estimate cooking support information related to the cooking menu currently being executed by the user based on the cooking menu information 69U, input information 61T, cooking status information 62T (including cooking results), and cooking image information 68TA (including user image information 67T) when the cooking menu related to the cooking menu information 69U was executed in the past.

[0187] The second learning unit 120 periodically acquires learning data 129 from the cooking assistance system 100 via the network NW1 and updates the second learning result data 129R. The second learning unit 120 may also periodically acquire learning data 129 from each information processing terminal 8 via the network NW1 and update the second learning result data 129R.

[0188] <Third Study Section> As shown in FIG. 9, the third learning unit 130 includes a learning data acquisition unit 130A and a learning processing unit 130B.

[0189] The learning data acquisition unit 130A acquires as learning data 139 a set of cooking menu information 69U, input information 61T, cooking status information 62T, and cooking image information 68TA, and notification timing 73T corresponding to the cooking menu information 69U, input information 61T, cooking status information 62T, and cooking image information 68TA.

[0190] The cooking menu information 69U, the input information 61T, the cooking status information 62T, and the cooking image information 68TA are used as input data. The notification timing 73T is used as training data (correct answer data).

[0191] The cooking menu information 69U, input information 61T, cooking status information 62T, and cooking image information 68TA are as described above.

[0192] The notification timing 73T is a timing suitable for the user to receive notification of cooking assistance information corresponding to the progress status.

[0193] When the learning processing unit 130B receives the cooking menu information 69U, the input information 61T, the cooking state information 62T, and the cooking image information 68TA, it performs machine learning on the third learning unit 130 so that it outputs an output value corresponding to the notification timing 73T.

[0194] The third learning unit 130 has a neural network 131T. The neural network 131T is configured by combining a plurality of types of neural networks.

[0195] In this embodiment, the neural network 131T includes a fully connected neural network 132T, a convolutional neural network 134T, a connection layer 136T, and an LSTM network 138T.

[0196] The specific configurations of the fully connected neural network 132T, the convolutional neural network 134T, the connection layer 136T, and the LSTM network 138T are similar to those of the fully connected neural network 152T, the convolutional neural network 154T, the connection layer 156T, and the LSTM network 158T associated with the neural network 151T of the cooking menu learning unit 150, and therefore will not be described here.

[0197] The learning processing unit 130B inputs cooking menu information 69U, input information 61T, and cooking state information 62T to the fully connected neural network 132T, and inputs cooking image information 68TA to the convolutional neural network 134T. The learning processing unit 130B then performs firing determinations for each neuron included in each layer, starting from the input side. In this way, the neural network 131T performs arithmetic processing on the input. The learning processing unit 130B obtains an output value corresponding to the notification timing 73T from the output layer of the LSTM network 138T of the neural network 131T.

[0198] The learning processing unit 130B stores information indicating the configuration of the neural network 131T constructed in this way, the weights of the connections between the neurons, and the thresholds of the neurons, in the storage unit 9M as third learning result data 139R.

[0199] In other words, the third learning unit 130 has performed machine learning to estimate the appropriate notification timing for the user to receive notification of cooking support information corresponding to the progress of the cooking menu to be performed by the user, based on the cooking menu information 69U, input information 61T, cooking status information 62T (including cooking results), and cooking image information 68TA (including user image information 67T) when the cooking menu related to the cooking menu information 69U was performed in the past.

[0200] The third learning unit 130 periodically acquires learning data 139 from the cooking assistance system 100 via the network NW1 and updates the third learning result data 139R. The third learning unit 130 may also periodically acquire learning data 139 from each information processing terminal 8 via the network NW1 and update the third learning result data 139R.

[0201] <Analysis Department> As shown in FIG. 10, the analysis unit 101 has a neural network 211 that uses the first learning result data 119R of the first learning unit 110.

[0202] Like the neural network 151T of the cooking menu learning unit 150, the neural network 211 is configured by combining multiple types of neural networks.

[0203] In this embodiment, the neural network 211 includes a fully connected neural network 212, a convolutional neural network 214, a connection layer 216, and an LSTM network 218.

[0204] The specific configurations of the fully connected neural network 212, the convolutional neural network 214, the connection layer 216, and the LSTM network 218 are similar to those of the fully connected neural network 152T, the convolutional neural network 154T, the connection layer 156T, and the LSTM network 158T associated with the neural network 151T of the cooking menu learning unit 150, and therefore will not be described here.

[0205] When a user executes a cooking menu using the heating cooking assistance system 100, the neural network 211 inputs the cooking menu information 69, input information 61, and cooking status information 62 into the fully connected neural network 212 and inputs the cooking image information 68A into the convolutional neural network 214, thereby outputting a progress status 71 corresponding to the cooking menu information 69, input information 61, cooking status information 62, and cooking image information 68A, and estimating the progress status of the steps of the cooking menu currently being executed by the user.

[0206] Cooking menu information 69 is output by cooking menu specification unit 105 after the user starts the cooking menu.

[0207] Input information 61 is input information acquired by input unit 53A of touch panel 53 and microphone 55 while a cooking menu is being executed.

[0208] The cooking status information 62 is cooking status information acquired by the information acquisition means (heating control unit 12, container temperature sensors 16, 26, weighing sensor 36A, temperature sensor 36B) while a cooking menu is being executed. The cooking status information 62 does not include image information.

[0209] The cooking image information 68A is cooking image information captured by the image capturing device 40 while a cooking menu is being performed. The cooking image information 68A includes user image information 67. The user image information 67 is user image information captured by the image capturing device 40 while a cooking menu is being performed.

[0210] The cooking status information 62 is cooking status information in the narrow sense, which does not include the cooking image information 68A acquired by the image capturing device 40. The cooking image information 68A is included in cooking status information in the broad sense.

[0211] The analysis unit 101 also includes a neural network 221 that uses the second learning result data 129R of the second learning unit 120.

[0212] Like the neural network 151T of the cooking menu learning unit 150, the neural network 221 is configured by combining multiple types of neural networks.

[0213] In this embodiment, the neural network 221 includes a fully connected neural network 222, a convolutional neural network 224, a connection layer 226, and an LSTM network 228.

[0214] The specific configurations of the fully connected neural network 222, the convolutional neural network 224, the connection layer 226, and the LSTM network 228 are similar to those of the fully connected neural network 152T, the convolutional neural network 154T, the connection layer 156T, and the LSTM network 158T associated with the neural network 151T of the cooking menu learning unit 150, and therefore will not be described here.

[0215] When a user executes a cooking menu using the heating cooking support system 100, the neural network 221 inputs the cooking menu information 69, input information 61, and cooking status information 62 into the fully connected neural network 222 and inputs the cooking image information 68A into the convolutional neural network 224, thereby outputting cooking support information 72 corresponding to the cooking menu information 69, input information 61, cooking status information 62, and cooking image information 68A, and estimating cooking support information related to the cooking menu currently being executed by the user.

[0216] The cooking support information 72 includes various information that is considered useful for improving the completeness of the cooking menu, such as information regarding the tasks that the user should perform to prevent delays in progress or deterioration of the cooking results for the cooking menu being executed.

[0217] The cooking menu information 69, the input information 61, the cooking status information 62, and the cooking image information 68A are as described above.

[0218] Furthermore, the analysis unit 101 has a neural network 231 that uses the third learning result data 139R of the third learning unit .

[0219] Like the neural network 151T of the cooking menu learning unit 150, the neural network 231 is configured by combining multiple types of neural networks.

[0220] In this embodiment, the neural network 231 includes a fully connected neural network 232, a convolutional neural network 234, a connection layer 236, and an LSTM network 238.

[0221] The specific configurations of the fully connected neural network 232, the convolutional neural network 234, the connection layer 236, and the LSTM network 238 are similar to those of the fully connected neural network 152T, the convolutional neural network 154T, the connection layer 156T, and the LSTM network 158T associated with the neural network 151T of the cooking menu learning unit 150, and therefore will not be described here.

[0222] When a user executes a cooking menu using the heating cooking assistance system 100, the neural network 231 inputs the cooking menu information 69, input information 61, and cooking status information 62 into the fully connected neural network 232 and inputs the cooking image information 68A into the convolutional neural network 234, thereby outputting notification timing 73 corresponding to the cooking menu information 69, input information 61, cooking status information 62, and cooking image information 68A, and estimating the notification timing appropriate for the user to be notified of the cooking assistance information 72 corresponding to the progress status 71.

[0223] The cooking menu information 69, the input information 61, the cooking status information 62, and the cooking image information 68A are as described above.

[0224] As will be described later, the analysis unit 101 notifies the display unit 53D and the speaker 54 of the touch panel 53 of cooking assistance information 72 corresponding to the progress status 71 based on the notification timing 73. The analysis unit 101 also performs image analysis on the user image information 67 and compares it with the user database 140 to identify the user currently performing the cooking menu and extract information such as the user's cooking skills and preferences.

[0225] <Cooking assistant app> 11, when a user executes the cooking assistant app on the mobile information terminal 50 to perform cooking with the assistance of the cooking assistance system 100, the cooking control unit 52 of the mobile information terminal 50 and the cooking menu identification unit 105 and analysis unit 101 of the external server 9 cooperate to execute the program shown in Fig. 11. The cooking menu identification unit 105 and analysis unit 101 send the processing results to the cooking control unit 52.

[0226] First, in step S101, the cooking control unit 52 executes a cooking menu specification process. During this process, the cooking control unit 52 transmits input information acquired by the input unit 53A of the touch panel 53 and the microphone 55 while the cooking menu is being executed as input information 61 to the external server 9. In addition, the cooking control unit 52 transmits cooking image information captured by the image capturing device 40 in the preparation process after the cooking menu is started as cooking image information for the preparation process 68P to the external server 9.

[0227] The neural network 151 of the cooking menu identification unit 105 receives input information 61 and cooking image information 68P of the preparation process, outputs cooking menu information 69 corresponding to the input information 61 and cooking image information 68P of the preparation process, and identifies the cooking menu to be performed by the user.

[0228] In addition, if the cooking control unit 52 is having difficulty in the cooking menu identification process in step S101, it may ask the user about the cooking menu when a time limit of, for example, several minutes has elapsed, and immediately identify the cooking menu based on the user's response, and then proceed to step S103 to start cooking assistance.

[0229] Next, the cooking control unit 52 proceeds to step S102 and executes a process for identifying the user who is executing the cooking menu. The cooking control unit 52 transmits user image information captured by the image capturing device 40 while the cooking menu is being executed as user image information 67 to the external server 9.

[0230] The analysis unit 101 performs image analysis on the user image information 67 and compares it with the user database 140 to identify the user who is currently carrying out the cooking menu.

[0231] Next, the cooking control unit 52 executes steps S103 to S105. At this time, the cooking control unit 52 transmits input information acquired by the input unit 53A of the touch panel 53 and the microphone 55 while the cooking menu is being executed as input information 61 to the external server 9. The cooking control unit 52 also transmits cooking status information acquired by the information acquisition means (heating control unit 12, container temperature sensors 16 and 26, weighing sensor 36A, temperature sensor 36B) while the cooking menu is being executed as cooking status information 62 to the external server 9. Furthermore, the cooking control unit 52 transmits cooking image information (including user image information) captured by the image capturing device 40 while the cooking menu is being executed as cooking image information 68A (including user image information 67) to the external server 9.

[0232] In step S103, the analysis unit 101 executes a process of estimating the progress of the steps of the cooking menu. The neural network 211 of the analysis unit 101 receives the cooking menu information 69, the input information 61, the cooking state information 62, and the cooking image information 68A, outputs the progress 71, and estimates the progress of the steps of the cooking menu currently being performed by the user.

[0233] In step S104, the analysis unit 101 executes a cooking assistance information estimation process. The neural network 221 of the analysis unit 101 receives the cooking menu information 69, the input information 61, the cooking status information 62, and the cooking image information 68A, and outputs cooking assistance information 72 to estimate cooking assistance information related to the cooking menu currently being performed by the user. Multiple pieces of cooking assistance information 72 are output and stored.

[0234] In step S105, the analysis unit 101 executes a notification timing estimation process. The neural network 231 of the analysis unit 101 receives the cooking menu information 69, the input information 61, the cooking status information 62, and the cooking image information 68A, and outputs the notification timing 73 to estimate the appropriate notification timing for the user to be notified of the cooking assistance information 72 corresponding to the progress status 71. The notification timing 73 is output and stored corresponding to each of the multiple pieces of cooking assistance information 72 stored.

[0235] Next, the cooking control unit 52 proceeds to step S106. Then, the analysis unit 101 refers to the user database 140, extracts information such as the identified user's cooking skills and preferences, and adjusts the cooking assistance information according to the identified user.

[0236] For example, if the identified user has low cooking skills, the analysis unit 101 adjusts the cooking support information to be easy to understand by dividing it into small steps and making it short sentences, and if the user has high skills, the cooking support information is made more modest.

[0237] Next, the cooking control unit 52 proceeds to step S107. The analysis unit 101 then determines whether there is cooking assistance information 72 for which the notification timing 73 has arrived. If the answer is "Yes" in step S107, the cooking control unit 52 proceeds to step S108. On the other hand, if the answer is "No" in step S107, the cooking control unit 52 proceeds to step S109.

[0238] When the process proceeds from step S107 to step S108, the analysis unit 101 transmits the cooking assistance information 72 for which the notification timing 73 has arrived to the cooking control unit 52, and causes the display unit 53D of the touch panel 53 and the speaker 54 to notify the information. After that, the cooking control unit 52 proceeds to step S109.

[0239] When the process proceeds from step S107 or step S108 to step S109, the analysis unit 101 checks the progress status 71 to determine whether cooking is complete. If the answer is "Yes" in step S109, the cooking control unit 52 proceeds to step S111. On the other hand, if the answer is "No" in step S109, the cooking control unit 52 repeats steps S103 to S109.

[0240] When cooking is completed and the process moves from step S109 to step S111, the analysis unit 101 analyzes and stores the cooking results.

[0241] Next, the cooking control unit 52 proceeds to step S112. Then, the analysis unit 101 executes update processing for the first to third learning units 110 to 130 based on the various information and cooking results acquired when the current cooking menu was executed. Thereafter, the cooking control unit 52 terminates the cooking assistant app.

[0242] <Action and effect> In the cooking assistance system 100 of the embodiment, the analysis unit 101 executes arithmetic processing by the first to third learning units 110 to 130 shown in Figures 7 to 9. As a result, when the user executes the cooking assistant app shown in Figure 11 to perform cooking with the assistance of the cooking assistance system 100, the analysis unit 101 can accurately estimate, in steps S103 to S105, the progress status 71 of the steps of the cooking menu being performed, cooking assistance information 72 related to the cooking menu being performed, and notification timing 73 appropriate for the user to be notified of the cooking assistance information 72 corresponding to the progress status 71. Then, in step S108, the analysis unit 101 causes the display unit 53D and speaker 54 of the touch panel 53 to notify the cooking assistance information 72 corresponding to the progress status 71 based on the notification timing 73.

[0243] For example, when the analysis unit 101 receives input information 61 requesting the next step while a cooking menu is being executed, the analysis unit 101 can accurately estimate cooking support information 72 regarding the next step and accurately estimate the notification timing 73 that will result in a quick response.

[0244] Furthermore, when the analysis unit 101 receives input information 61 during the execution of a cooking menu requesting advice on how to cut ingredients before heating, the cooking utensils to use, the amount and ratio of seasonings, etc., it can accurately estimate cooking support information 72 relating to the advice and accurately estimate the notification timing 73 that will result in a quick response.

[0245] Furthermore, when the analysis unit 101 acquires cooking status information 62 indicating that the amount of heat applied by the cooking appliance 10 is excessive during the execution of a cooking menu, the cooking results of the cooking menu are likely to deteriorate, so the analysis unit 101 can accurately estimate cooking assistance information 72 relating to operations to reduce the amount of heat to an appropriate level, and can accurately estimate notification timing 73 that will result in a quick response.

[0246] Furthermore, when the analysis unit 101 acquires cooking status information 62 indicating that the temperature of the cooking container 20 is too low while a cooking menu is being executed, there is still time before the cooking result of the cooking menu deteriorates, so the analysis unit 101 can accurately estimate cooking assistance information 72 related to an operation to raise the temperature to an appropriate level and accurately estimate notification timing 73 that will provide a response that allows the user sufficient time to think and respond. If the user thinks and responds, the cooking assistance information 72 and notification timing 73 are canceled.

[0247] On the other hand, the analysis unit 101 can suppress notification of cooking assistance information 72 to the user for the cooking menu being executed while the delay in progress 71 or the deterioration of the cooking result is not a problem.

[0248] This allows the user more room to think about and execute cooking menus on their own, making it easier to adjust the cooking menu to suit their preferences. Also, it reduces the effort required for the user to input prerequisite information to limit the cooking assistance information 72 that is notified to only the information that the user wants to know.

[0249] Therefore, the cooking assistance system 100 of the embodiment can improve the cooking skills of the user while minimizing the inconvenience felt by the user.

[0250] Furthermore, in this cooking assistance system 100, the cooking assistance information 72 includes information on tasks that the user should perform to prevent delays in the progress status 71 or deterioration of the cooking results. This configuration can improve the effectiveness of the cooking assistance information 72, further reducing the user's inconvenience and further improving the user's cooking skills.

[0251] Furthermore, in this cooking assistance system 100, the photographing device 40 photographs the food item F1, the cooking container 20, and the cooking appliance 10 to acquire cooking image information 68A. The broad-sense cooking status information includes the cooking image information 68A and the narrow-sense cooking status information 62. With this configuration, the cooking image information 68A appropriately reflects the progress status 71 of the cooking menu steps, thereby increasing the amount of information in the broad-sense cooking status information. This allows the analysis unit 101 to more accurately estimate the progress status 71 of the cooking menu steps being executed, the cooking assistance information 72, and the notification timing 73. As a result, this cooking assistance system 100 can further reduce user inconvenience while further improving the user's cooking skills.

[0252] In addition, in this cooking assistance system 100, the photographing device 40 also photographs the user performing a cooking menu to acquire user image information 67. The cooking image information 68A includes the user image information 67. With this configuration, the cooking image information 68A not only appropriately reflects the progress status 71 of the cooking menu steps, but also the user's cooking status, thereby further increasing the amount of cooking status information in a broad sense. This allows the analysis unit 101 to more accurately estimate the progress status 71 of the cooking menu steps being performed, the cooking assistance information 72, and the notification timing 73. For example, if the analysis unit 101 estimates from the user image information 67 that the user is behaving as if they are confused because they do not know the next step while a cooking menu is being executed, the analysis unit 101 can accurately estimate cooking assistance information 72 relating to the next step, and if the cooking result of the cooking menu is likely to deteriorate, the analysis unit 101 can accurately estimate the notification timing 73 that will provide a quick response, and if there is sufficient time before the cooking result of the cooking menu deteriorates, the analysis unit 101 can accurately estimate the notification timing 73 that will provide a response that allows the user sufficient time to think and respond on their own. As a result, the cooking assistance system 100 can further reduce the user's annoyance while further improving the user's cooking skills.

[0253] Furthermore, in this cooking assistance system 100, the analysis unit 101 has a user database 140 in which information related to multiple users is registered in advance, as shown in Fig. 4. Then, in step S102 shown in Fig. 11, the analysis unit 101 identifies the user currently performing the cooking menu based on the user image information 67, and in step S106, refers to the user database 140 and adjusts the cooking assistance information 72 according to the identified user. With this configuration, the analysis unit 101 can easily optimize the notification of the cooking assistance information 72 according to each user by registering the age, gender, cooking skill, etc. of multiple users in advance.

[0254] Furthermore, in this cooking assistance system 100, as shown in Fig. 5, the cooking menu learning unit 150 of the cooking menu identification unit 105 has already performed machine learning to estimate a cooking menu based on cooking image information 68TP captured during the preparation process when a cooking menu was previously executed. Then, as shown in Fig. 6, the cooking menu identification unit 105 also identifies a cooking menu based on cooking image information 68P of the preparation process captured by the photographing device 40 after the cooking menu was started. With this configuration, even when the user does not input the cooking menu to be executed into the input unit 53A of the touch panel 53 and the microphone 55, the cooking menu identification unit 105 identifies a cooking menu based on the cooking image information 68P of the preparation process, thereby further reducing the user's inconvenience.

[0255] Furthermore, in this cooking assistance system 100, the microphone 55 is a voice interface that receives voice input from the user to acquire input information. This configuration saves the user the trouble of operating a touch panel, keyboard, etc. to input information, further reducing the inconvenience to the user.

[0256] Furthermore, in this cooking assistance system 100, the container temperature sensors 16, 26 are first status sensors that detect the status of the cooking container 20. The cooking status information 62 includes the detection results of the container temperature sensors 16, 26. With this configuration, the detection results of the container temperature sensors 16, 26 appropriately reflect the progress status 71 of the cooking menu steps, thereby increasing the amount of information in the cooking status information 62. This allows the analysis unit 101 to more accurately estimate the progress status 71 of the cooking menu steps being executed, the cooking assistance information 72, and the notification timing 73. As a result, this cooking assistance system 100 can further improve the user's cooking skills while further reducing the user's inconvenience.

[0257] Furthermore, in this cooking assistance system 100, the weighing sensor 36A and the temperature sensor 36B are provided on the ladle 30A and the doneness detection rod 30B, respectively. These are second status sensors that detect the status of the food F1 and / or the status of additional ingredients added to the food F1. The cooking status information 62 includes the detection results of the weighing sensor 36A and the temperature sensor 36B. With this configuration, the detection results of the weighing sensor 36A and the temperature sensor 36B appropriately reflect the progress status 71 of the cooking menu steps, thereby increasing the amount of information in the cooking status information 62. This allows the analysis unit 101 to more accurately estimate the progress status 71 of the cooking menu steps being executed, the cooking assistance information 72, and the notification timing 73. As a result, this cooking assistance system 100 can further reduce user inconvenience and further improve the user's cooking skills.

[0258] Although the present invention has been described above with reference to the examples, it goes without saying that the present invention is not limited to the above examples and can be modified and applied as appropriate within the scope of the invention.

[0259] In the embodiment, the cooking appliance 10 is a gas stove, but the present invention is not limited to this configuration. For example, the cooking appliance may be an IH stove having a high-frequency induction heating type heating unit.

[0260] In the embodiment, the cooking utensils are the ladle 30A and the doneness detection rod 30B, and the second status sensor is the weighing sensor 36A and the temperature sensor 36B, but the present invention is not limited to this configuration. For example, the present invention also includes a configuration in which the cooking utensil is a spatula or a spatula, and the second status sensor provided on the spatula is a temperature sensor that detects the temperature of the food to be cooked, or a concentration sensor that detects the salt concentration of the food to be cooked.

[0261] In the embodiment, cooking container 20 has container temperature sensor 26 as the first status sensor, but the present invention is not limited to this configuration. For example, the configuration also includes a configuration in which the first status sensor is a weighing sensor that measures the weight of the food to be cooked or a concentration sensor that detects the salt concentration of the food to be cooked, and multiple first status sensors may be provided. In this case, instead of providing container temperature sensor 26 in cooking container 20, a container temperature sensor may be provided on the top surface of cooking appliance 10 on which cooking container 20 is placed. Furthermore, cooking container 20 does not necessarily have to have a first status sensor.

[0262] In the embodiment, first communication unit 17, second communication unit 27, third communication units 37A and 37B, and fourth communication unit 47 each perform short-range wireless communication with terminal communication unit 57, and all communications are performed using Bluetooth (registered trademark), but the present invention is not limited to this configuration. For example, the present invention also includes a configuration in which battery-operated cooking container 20, ladle 30A, and doneness detection rod 30B communicate using Bluetooth (registered trademark), which allows communication with low power consumption, and cooking appliance 10 and photographing device 40, which are connected to a commercial power source and transmit a relatively large amount of information, communicate using Wi-Fi (registered trademark).

[0263] In the embodiment, the neural network includes a fully connected neural network, a convolutional neural network, a connection layer, and an LSTM network. However, the configuration of the neural network is not limited to this example and may be determined appropriately depending on the embodiment. For example, the LSTM network may be omitted. Furthermore, if the input of cooking image information is omitted, the convolutional neural network and the connection layer may be omitted.

[0264] In the embodiment, neural networks 111T, 121T, 131T, and 151T are used as the first to third learning units 110 to 130 and the cooking menu learning unit 150. However, the type of learning unit is not limited to a neural network as long as it can use captured images as input, and may be selected appropriately depending on the embodiment. Examples of usable learning units include support vector machines, self-organizing maps, and learning units that perform machine learning using reinforcement learning.

[0265] In the examples, neural networks 211, 221, 231, and 151 are used as the analysis unit 101 and the cooking menu identification unit 105. However, the types of the analysis unit and cooking menu identification unit are not limited to neural networks as long as the captured image can be used as an input, and may be appropriately selected depending on the embodiment, such as a processing unit that performs machine learning using a support vector machine, a self-organizing map, or reinforcement learning to perform analysis and identification.

[0266] In the embodiment, the analysis unit 101 estimates the progress of the steps of the cooking menu based on the machine learning of the first learning unit 110. However, it is not necessary to use artificial intelligence for all of the processes, and some of the processes may use conditional judgment based on predetermined logic. For example, when the analysis unit is performing a step of kneading minced meat by hand in the cooking process of hamburger steak, it uses artificial intelligence to estimate that the meat is being kneaded by hand based on cooking image information, etc., but after recognizing that the meat is being kneaded by hand, it can handle the next steps by conditional judgment based on predetermined logic, rather than by using artificial intelligence. The same applies to a configuration in which the analysis unit 101 estimates cooking assistance information based on the machine learning of the second learning unit 120 and a configuration in which the analysis unit 101 estimates the timing of notification based on the machine learning of the third learning unit 130.

[0267] In the embodiment, the cooking menu specification unit 105 estimates the cooking menu based on the machine learning of the cooking menu learning unit 150, but it is not necessary to use artificial intelligence for all of the processes, and some of the processes may use conditional judgments based on predetermined logic, etc. As an example, the cooking menu specification unit uses artificial intelligence to estimate the recognition of what ingredients and tableware have been prepared based on cooking image information, etc., and after recognizing that beef, onions, and a bowl have been prepared, it can determine that the cooking menu is "beef bowl" based on the AND condition of "beef" + "onion" + "bowl."

[0268] In the embodiment, the analysis unit 101 has the neural network 111T of the first learning unit 110 shown in Fig. 7 and the neural network 211 shown in Fig. 10, but these may be replaced with a single neural network that performs machine learning for estimating the progress status and the estimation of the progress status. The same applies to the neural network 121T of the second learning unit 120 shown in Fig. 8 and the neural network 221 shown in Fig. 10. The same applies to the neural network 131T of the third learning unit 130 shown in Fig. 9 and the neural network 231 shown in Fig. 10.

[0269] In the embodiment, the cooking menu identification unit 105 has the neural network 151T of the cooking menu learning unit 150 shown in FIG. 5 and the neural network 151 shown in FIG. 6, but these may be replaced with a single neural network, and the neural network may be configured to perform machine learning for estimating the progress status and to estimate the progress status.

[0270] In the embodiment, the cooking menu learning unit 150 does not execute the cooking menu specification process after specifying the cooking menu in step S101 of Fig. 11, but the present invention is not limited to this configuration. For example, while steps S103 to S109 of Fig. 11 are repeatedly executed, the cooking menu learning unit 150 may execute the cooking menu specification process in parallel to update the cooking menu information 69 used in steps S103 to S105. [Industrial Applicability]

[0271] The present invention can be used, for example, in a cooking assistance system for carrying out cooking in a home, a cooking facility, or the like. [Explanation of symbols]

[0272] 100...Heat cooking support system F1…Item to be cooked 20...Cooking container 30A, 30B...Cooking tools (30A...Ladle, 30B...Cooking condition detector) 10…Heating means (cooking equipment) 53A, 55...input interface (53A...touch panel input section, 55...voice interface (microphone)) 40, 12, 16, 26, 36A, 36B...information acquisition means (40...photography device, 12...heating control unit, 16, 26...first status sensor (container temperature sensor), 36A, 36B...second status sensor (36A...measurement sensor, 36B...temperature sensor)) 53D, 54...Notification means (53D...Touch panel display unit, 54...Speaker) 105...Cooking menu specification section 101…Analysis Department 110...First Learning Section 120...Second Learning Section 130…Third Learning Section 140...User database

Claims

1. a cooking container for accommodating food to be cooked; a cooking utensil used together with the cooking container to cook the food; a heating means for heating the cooking vessel; an input interface that receives input from a user and acquires input information; an information acquisition means for acquiring cooking status information of at least one of the food to be cooked, the cooking container, the cooking utensil, and the heating means; a notification means for notifying the cooking status information; a cooking menu specification unit that specifies a cooking menu for the user to heat and cook the food using the heating means based on the input information; an analysis unit, The analysis unit a first learning unit that has performed machine learning to estimate the progress of the steps of the cooking menu currently being executed by the user based on the input information, the cooking status information, and the cooking results when the cooking menu was executed in the past; a second learning unit that has performed machine learning to estimate cooking assistance information related to the cooking menu currently being executed by the user based on the input information, the cooking status information, and the cooking results when the cooking menu was executed in the past; a third learning unit that has performed machine learning to estimate an appropriate notification timing for the user to be notified of the cooking assistance information corresponding to the cooking progress, based on the input information, the cooking status information, and the cooking results when the cooking menu was previously executed; The analysis unit The cooking menu currently being executed, the input information, and the cooking status information are input; The first learning unit, the second learning unit, and the third learning unit execute calculation processing to estimate the progress status, the cooking assistance information, and the notification timing; A cooking support system characterized in that the notification means is configured to notify the cooking support information corresponding to the progress status based on the notification timing.

2. The heating cooking assistance system according to claim 1 , wherein the cooking assistance information includes information regarding an operation that the user should perform to prevent the delay in the progress or the deterioration of the cooking result.

3. The information acquisition means has a photographing device that photographs the food, the cooking vessel, and the heating means to acquire cooking image information, The cooking assistance system according to claim 1 , wherein the cooking status information includes the cooking image information.

4. the photographing device photographs the user who is performing the cooking menu to acquire user image information; The cooking assistance system according to claim 3 , wherein the cooking image information includes the user image information.

5. The analysis unit A user database in which information relating to a plurality of users is registered in advance, Identifying a user who is currently performing the cooking menu based on the user image information; The cooking assistance system according to claim 4, wherein the cooking assistance information is adjusted according to the identified user by referring to the user database.

6. the cooking menu identification unit has already performed machine learning to estimate the cooking menu based on the cooking image information captured in a preparation step when the cooking menu was previously executed, The cooking assistance system according to claim 3 , wherein the cooking menu specifying unit specifies the cooking menu based also on the cooking image information in the preparation step.

7. 7. The cooking assistance system according to claim 1, wherein the input interface includes a voice interface for receiving voice input from a user and acquiring the input information.

8. The information acquisition means has a first status sensor that detects the status of the cooking vessel, The cooking assistance system according to claim 1 , wherein the cooking status information includes a detection result of the first status sensor.

9. The information acquisition means has a second status sensor provided in the cooking tool that detects the status of the food to be cooked and / or the status of additional ingredients to be added to the food to be cooked, The cooking assistance system according to claim 1 , wherein the cooking status information includes a detection result of the second status sensor.

Images