Cooking system, program, inference device and learning device

Abstract

To provide cooking information suited to foodstuff in accordance with a user's taste.SOLUTION: A cooking system 1 includes: a heating cooker 2 for heating and cooking foodstuff; a display part 41a which is provided with a photographing part 40 and an operation unit 41b, is communicably connected to the heating cooker 2, acquires foodstuff information from a foodstuff image photographed by the photographing part 40, acquires from the foodstuff information cooking information necessary for cooking the foodstuff with the heating cooker 2, and displays the same; and a terminal device 3 provided with cooking setting information transmitting means that transmits cooking setting information to the heating cooker with correction information correcting cooking information as the cooking setting information when the operation unit 41b is operated by a user and the cooking information is corrected and with the cooking information as the cooking setting information when the operation unit 41b is operated by the user but the cooking information is not corrected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a cooking system that controls a cooking appliance. [Background technology]

[0002] Conventional cooking appliances have a technology that estimates food ingredient information such as the type, size, and amount of food ingredients from images of the ingredients, and then cooks the food in a way that is appropriate for each ingredient. For example, Patent Document 1 discloses a cooking appliance that takes an image of multiple food ingredients placed in a heating chamber, estimates food ingredient information such as the type, size, and amount of food ingredients from the image, sets cooking conditions such as temperature and heating time for each ingredient, and simultaneously cooks the ingredients under cooking conditions appropriate for each ingredient. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-088138 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the cooking appliance disclosed in Patent Document 1 sets cooking conditions based on images of ingredients, so there is no way for users to adjust the heating level to suit their preferences. Furthermore, the cooking appliance disclosed in Patent Document 1 does not provide a way to update the cooking conditions for ingredients set by the appliance if the finished food is not suitable for many users.

[0005] In order to solve the above-mentioned problems, the present disclosure aims to provide a cooking system that controls a heating cooker and that can obtain cooking information suitable for ingredients according to the user's preferences. [Means for solving the problem]

[0006] The cooking system of the present disclosure comprises a heating cooker that heats and cooks ingredients, a display unit that has a photographing unit and an operating unit and is communicatively connected to the heating cooker, acquires ingredient information from ingredient images photographed by the photographing unit, and acquires and displays cooking information, which is information necessary for cooking ingredients in the heating cooker, from the ingredient information, and a terminal device that has a cooking setting information transmission means that, when a user operates the operating unit to modify the cooking information, sets the modified cooking information, which is the modified cooking information, as cooking setting information, and, when the user operates the operating unit to not modify the cooking information, sets the cooking information as cooking setting information and transmits the cooking setting information to the heating cooker.

[0007] In addition, the program according to the present disclosure is a program that is installed in a computer and controls a cooking appliance that heats and cooks ingredients, and includes a cooking information estimation means that analyzes an image to obtain ingredient information and estimates cooking information, which is information necessary to cook ingredients in the cooking appliance, from the ingredient information, and a cooking setting information transmission means that, if a user modifies the cooking information, sets the modified information as cooking setting information, and, if the user does not modify the cooking information, sets the cooking information as cooking setting information and transmits the cooking setting information to the cooking appliance.

[0008] In addition, the second learning device according to the present disclosure includes a second learning data acquisition unit that acquires second learning data including ingredient information and cooking information for the ingredient information, and a second model generation unit that uses the second learning data to generate a second trained model for inferring cooking information from the ingredient information.

[0009] In addition, the inference device according to the present disclosure includes an ingredient information storage unit that stores ingredient information, and a cooking information estimation unit that outputs cooking information from the ingredient information stored in the ingredient information storage unit using a second trained model for inferring cooking information from the ingredient information stored in the ingredient information storage unit. [Effects of the Invention]

[0010] According to the present disclosure, in a cooking system that controls a cooking appliance, cooking information suitable for ingredients according to the preferences of a user can be obtained. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram showing a configuration of a cooking system according to a first embodiment. [Figure 2] 1 is a cross-sectional view of a cooking device according to a first embodiment. [Figure 3] 1 is a block diagram showing a functional configuration of a cooking device according to a first embodiment. [Figure 4] 2 is a block diagram showing a functional configuration of a terminal device according to the first embodiment. FIG. [Figure 5] 3 is an example of a cooking information database according to the first embodiment. [Figure 6] 10 is a flowchart showing the operation of a cooking appliance control application. [Figure 7] 10 is a diagram showing an example of a display (initial selection screen) on the display unit when a cooking appliance control app is executed. [Figure 8] 10 is a diagram showing an example of a display (photographing screen) on the display unit when a cooking appliance control app is running. [Figure 9] 10 is a diagram showing an example of a display (image analysis failure screen) on the display unit when the cooking appliance control app is running. [Figure 10] 10 is a diagram showing an example of a display (image analysis result screen) on the display unit when a cooking appliance control app is running. [Figure 11] 10 is a diagram showing an example of a display (cooking information display screen) on the display unit when a cooking appliance control app is running. [Figure 12] 10 is a diagram showing an example of a display (ingredients information input screen) on the display unit when a cooking appliance control app is running. [Figure 13] 10 is a diagram showing an example of a display (cooking history selection screen) on the display unit when a cooking appliance control app is running. [Figure 14] 10 is a diagram showing an example of a display (photography method selection screen) on the display unit when a cooking appliance control app is running. [Figure 15] 10 is a diagram showing an example of a display (container use photographing screen) on the display unit when the cooking appliance control app is running. [Figure 16] FIG. 10 is a perspective view of a container of a cooking device according to a second embodiment. [Figure 17]10 is a diagram showing an example of a display (photographing screen using a cutting board) on the display unit when the cooking appliance control app is running. [Figure 18] FIG. 11 is a block diagram showing the relationship between a first learning device and a terminal device regarding ingredient image analysis in a cooking system according to embodiment 3. [Figure 19] FIG. 11 is a diagram illustrating an example of a neural network in a first learning device according to a third embodiment. [Figure 20] 10 is a flowchart showing a first learning procedure by a first learning device according to a third embodiment. [Figure 21] 10 is a flowchart showing the steps of food ingredient image analysis using the first trained model according to embodiment 3. [Figure 22] FIG. 11 is a block diagram showing the relationship between a second learning device and a terminal device regarding cooking information estimation in a cooking system according to embodiment 4. [Figure 23] 10 is a flowchart illustrating a learning process of a second learning device according to the fourth embodiment. [Figure 24] 10 is a flowchart showing the procedure for estimating cooking information using a second trained model according to embodiment 4. DETAILED DESCRIPTION OF THE INVENTION

[0012] The following description will discuss embodiments of the present disclosure with reference to the accompanying drawings. In each drawing, identical or corresponding parts are designated by the same reference numerals, and redundant descriptions are appropriately simplified or omitted. The present disclosure is not limited to the following embodiments, and any of the components of the embodiments may be modified, combined, or omitted without departing from the spirit of the present disclosure.

[0013] Embodiment 1 Fig. 1 is a diagram showing the configuration of a cooking system 1 according to embodiment 1. As shown in Fig. 1, the cooking system 1 includes a cooking appliance 2 that heats and cooks ingredients, a terminal device 3, and a control server 6. The cooking appliance 2 and the terminal device 3 are communicatively connected via a router 4. Furthermore, the cooking appliance 2 and the terminal device 3 are connected via the router 4 to a wide area network 5 such as the Internet, and are connected to a remote control server 6.

[0014] The cooking appliance 2 is a device that heats and cooks ingredients for a cooking time and temperature set by an attached operation display device 33 or terminal device 3. The cooking appliance 2 is, for example, a low-temperature cooker, rice cooker, oven, or grill. The cooking appliance 2 has a communication unit 34 and is communicatively connected to the terminal device 3 via a router 4. Note that the cooking appliance 2 and the terminal device 3 may communicate directly using short-range communication such as Bluetooth (registered trademark) or NFC (Near Field Communication) without using the router 4.

[0015] The terminal device 3 is a computer on which software runs, such as a smartphone, tablet terminal, or PC, and which has input / output means and communication means. Alternatively, the terminal device 3 may be a dedicated terminal for controlling the cooker 2. A cooking appliance control app 100 for controlling the cooker 2 is installed in the terminal device 3. The terminal device 3 can send and receive various information to and from other devices, including the cooker 2, via the communication means. The terminal device 3 may be connected to a wide area network 5 and communicate with the cooker 2 via the wide area network 5 and a router 4.

[0016] The router 4 is a relay device for connecting the cooking appliance 2 and the terminal device 3 to the wide area network 5.

[0017] The wide area network 5 is a network such as the Internet that allows communication between remote devices.

[0018] The control server 6 is communicatively connected to the terminal device 3 via the wide area network 5, executes some functions of the cooking appliance control app 100 that controls the cooking appliance 2 installed on the terminal device 3, and stores usage data of the cooking appliance 2 acquired from the terminal device 3. Note that a plurality of terminal devices 3 may be connected to the control server 6. The control server 6 is composed of at least one processor and at least one memory. The processor is also referred to as a CPU (Central Processing Unit), processing device, arithmetic device, microprocessor, microcomputer, or DSP. The memory is composed of, for example, non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM, or EEPROM.

[0019] Fig. 2 is a cross-sectional view of cooking device 2 according to embodiment 1. In this embodiment, cooking device 2 will be described as a rice cooker capable of low-temperature cooking. The configuration of the rice cooker, which is cooking device 2, will be described with reference to Fig. 2.

[0020] The cooking appliance 2 includes a main body 8, a container housing 9 fixedly attached to the inside of the main body 8, a pot-shaped container 10 that is detachably housed in the container housing 9 and is cylindrical with a bottom and an open top, an inner lid 11 that covers the top opening of the container 10, and an outer lid 12 that is openably and closably engaged with the main body 8 and is connected to the inner lid 11 to cover the inner lid 11. The inner lid 11 and the outer lid 12 form a lid body that covers the container 10. The cooking appliance 2 also includes an operation display device 33 that is a user interface that accepts user operations, a communication unit 34 that communicates with the terminal device 3, and a control unit 35 that controls the heating of ingredients based on cooking information input from the operation display device 33 and the communication unit 34. The cooking information is a parameter for cooking ingredients in the cooking appliance 2 and includes at least one of a cooking temperature and a cooking time.

[0021] A spirally wound heating coil 13 for electromagnetic induction heating is provided on the outer wall of the container housing portion 9. A high-frequency current is supplied to the heating coil 13, which generates a magnetic field that induction heats the container 10. Note that the heating device for the container 10 is not limited to the heating coil 13 for electromagnetic induction heating, and may be a heater or the like.

[0022] A through-hole is formed in the center of the bottom of the container-receiving portion 9, and a temperature sensor 14 is disposed within the through-hole. The temperature sensor 14 is supported from below by a compression spring 15 and disposed in contact with the bottom of the container 10, and measures the temperature of the container 10.

[0023] A handle 16 is provided around the opening of container 10, and handle 16 is engaged with a holder (not shown) provided on container storage section 9, thereby holding container 10 within main body 8. In addition, a flange 17 extending outward is formed around the top opening of container 10. A lid gasket 18, which is a sealing member, is provided on the periphery of inner lid 11, and lid gasket 18 provides a tight seal between flange 17 of container 10 and the inner wall surface of container 10.

[0024] An inner lid vent hole 19 is provided through the inner lid 11, and a communication flow path 20 is formed in the inner lid 11 and the outer lid 12, connecting the inside of the container 10 to the outside of the cooking appliance 2 via the inner lid vent hole 19. The communication flow path 20 is made up of a first communication flow path 20a, a second communication flow path 20b, a third communication flow path 20c, and a fourth communication flow path 20d, all of which are hollow. The first communication flow path 20a, which connects to the inner lid vent hole 19, has a path packing 21 disposed therein for sealingly connecting it to the inner lid vent hole 19. The functions of the first communication flow path 20a, the second communication flow path 20b, and the third communication flow path 20c will be described later.

[0025] Inner lid 11 is further formed with a steam hole 22, and a steam exhaust valve 23 is disposed in steam hole 22. Steam exhaust valve 23 is a valve that automatically opens and closes depending on the pressure inside container 10 to seal or unseal the inside of container 10. Downstream of steam exhaust valve 23 in the direction in which steam is blown out, a cartridge 25 equipped with a steam exhaust port 24 is disposed in connection with cartridge packing 26.

[0026] In addition, a sensor hole 27 is provided through the inner lid 11, and a lid sensor 28 that measures the temperature inside the container 10 through the sensor hole 27 is provided in the outer lid 12, and a lid sensor gasket 29 that seals the sensor hole 27 and the outer lid 12 is provided.

[0027] The outer lid 12 is equipped with a decompression pump 30 that draws air from within the container 10 through the inner lid vent 19 via the first and second communication channels 20a and 20b, thereby reducing the pressure inside the container 10. A three-way solenoid valve 31 is disposed between the first and second communication channels 20a and 20b. The three-way solenoid valve 31 switches the communication channel 20, which connects the inside of the container 10 to the outside of the cooking appliance 2, between channel A and channel B. Channel A is a channel that, when the decompression pump 30 is operated, exhausts air from within the container 10 through the inner lid vent 19, the first and second communication channels 20a and 20b, the decompression pump 30, and the third communication channel 20c, and is then discharged through a first outer lid vent 32a that opens on the outer surface of the outer lid 12. This discharges the air from within the container 10 to the outside, thereby reducing the pressure inside the container 10. Flow path B is a flow path that discharges air outside the cooking appliance 2 from the second outer lid vent 32b through the fourth communication flow path 20d, the second communication flow path 20b, the vacuum pump 30, and the third communication flow path 20c to the first outer lid vent 32a by operating the vacuum pump 30. This allows the moisture sucked by the vacuum pump 30 to be discharged.

[0028] When flow path A is connected by the three-way solenoid valve 31, the vacuum pump 30 draws in air from the container 10 and exhausts it through flow path A to the outside of the cooking appliance 2 via the first outer lid vent 32a. When flow path B is connected by the three-way solenoid valve 31, the vacuum pump 30 draws in outside air through the second outer lid vent 32b and exhausts the air from the container 10 to the outside of the cooking appliance 2 via flow path B via the first outer lid vent 32a. The first outer lid vent 32a and the second outer lid vent 32b are located on the side of the outer lid 12 rather than on the top, which prevents moisture and foreign matter from entering the vacuum pump 30 and reduces the risk of malfunction. Note that, in FIG. 2, the first outer lid vent 32a and the second outer lid vent 32b are located on the outer lid 12 as exhaust holes for exhausting air from the container 10 to the outside. However, the exhaust holes may also be located on the side or bottom of the main body 8. The operation of the decompression pump 30 is controlled by a control unit 35, which will be described later.

[0029] The communication unit 34 is disposed inside the outer lid 12. The communication unit 34 is a communication interface that transmits and receives signals wirelessly or via a wire. The communication unit 34 receives control information for the cooking appliance 2 from the terminal device 3. It also transmits the state of the cooking appliance 2 to the terminal device 3. The communication unit 34 outputs the received control information for the cooking appliance 2 to the control unit 35, which will be described later. Note that although the communication unit 34 is disposed inside the outer lid 12 in FIG. 2, it may also be disposed in the main body 8.

[0030] The operation and display device 33 is disposed on the side of the main body 8, and displays a screen for setting cooking-related settings and accepts operations by the user on the screen. The operation and display device 33 outputs the contents of operations by the user to the control unit 35, which will be described later. Although the operation and display device 33 is provided on the main body 8 in FIG. 2, it may also be provided on the outer lid 12.

[0031] The operation display device 33 has a cooker operation unit 33a that accepts operation input from the user, a cooker display unit 33b that displays the input information and the operating status of the cooker 2, and a sound output unit (not shown) that outputs operation sounds, alarm sounds, etc. Note that the operation display device 33 is not limited to one in which the operation unit and display unit are integrated, and may be configured so that the display unit is provided separately. Furthermore, the operation display device 33 may be placed in a location other than the cooker 2, such as the terminal device 3, and may communicate with the communication unit 34 of the cooker 2 to set the control content operated on the operation display device 33.

[0032] The control unit 35 is disposed on the side of the main body 8, and controls the heating coil 13 and the decompression pump 30, which are the heating means of the cooking appliance 2, based on input from the communication unit 34 or the operation display device 33. The control unit 35 is also electrically connected to the temperature sensor 14 and the lid sensor 28, and acquires the detection results of the temperature sensor 14 and the lid sensor 28.

[0033] 3 is a block diagram showing a functional configuration of the cooking appliance 2 according to embodiment 1. The configuration of the control unit 35 and the operation of the cooking appliance 2 will be described with reference to FIG.

[0034] The control unit 35 includes at least one processor and at least one memory. The processor is also called a CPU (Central Processing Unit), processing unit, arithmetic unit, microprocessor, microcomputer, or DSP. The memory is configured by, for example, non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM, or EEPROM. The control unit 35 includes an input unit 35a, a storage unit 35b, and a heating control unit 35c.

[0035] The input unit 35a acquires cooking information input from the communication unit 34 or the cooker operation unit 33a. The cooking information includes cooking temperature and cooking time. The cooking information may also include a set pressure.

[0036] The storage unit 35b temporarily stores the cooking information acquired by the input unit 35a. The storage unit 35b may also store the cooking history of the cooking appliance 2.

[0037] The heating control unit 35c controls the heating of the heating coil 13 based on the cooking information stored in the memory unit 35b. The heating control unit 35c acquires the detection results of the temperature sensor 14 and the lid sensor 28, and controls the heating of the heating coil 13 so that the container 10 maintains the set cooking temperature for the set cooking time. Furthermore, if the cooking information includes a set pressure, the heating control unit 35c controls the pressure reducing pump 30 so that the pressure inside the container 10 becomes the set pressure.

[0038] The cooking flow in the cooking appliance 2 will be explained. The user places ingredients in the container 10 and closes the inner lid 11 and outer lid 12. When cooking at low temperature, the ingredients are placed in a heat-resistant bag and the bag is then placed in the container 10 filled with water. This method allows the water to transfer heat evenly, so the ingredients can be heated evenly. Also, placing the ingredients in a bag prevents the flavor of the ingredients from dissolving into the water. The user operates the terminal device 3 and sends cooking information from the terminal communication unit 42. The cooking appliance 2 receives the cooking information from the terminal device 3 via the communication unit 34 and inputs it to the control unit 35. Alternatively, the user operates the cooker operation unit 33a of the operation and display device 33 to send the cooking information to the control unit 35.

[0039] Upon receiving the cooking information, the control unit 35 controls the heating of the heating coil 13 so that the temperatures detected by the temperature sensor 14 and the lid sensor 28 are maintained at the cooking temperature. If the cooking information includes a set pressure, the heating control unit 35c controls the pressure reduction pump 30 so that the pressure inside the container 10 becomes the set pressure. The control unit 35 ends the cooking process after the set cooking time has elapsed since the temperatures detected by the temperature sensor 14 and the lid sensor 28 reached the cooking temperature.

[0040] By the above operations, the ingredients placed in the container 10 of the cooking device 2 are cooked based on the cooking information.

[0041] Next, a description will be given of the cooking appliance control app 100 that controls the cooking appliance 2 and is installed in the terminal device 3. Fig. 4 is a block diagram showing the functional configuration of the terminal device 3 according to embodiment 1. The functions of the terminal device 3 and the configuration of the cooking appliance control app 100 will be described using Fig. 4.

[0042] The terminal device 3 includes a photographing unit 40 such as a camera that captures images, a terminal communication unit 42 that communicates with other devices, and a terminal screen 41 that has a display unit 41a that displays information to the user and an operation unit 41b that accepts user operations. The terminal device 3 also has installed thereon a cooking appliance control app 100, which is an application for controlling the cooking appliance 2. The functions of the cooking appliance control app 100 may be implemented by software or firmware pre-installed in the terminal device 3, or a combination of software and firmware.

[0043] The photographing unit 40 is a device that photographs an image by operation of a user, such as a camera mounted on the terminal device 3. The photographing unit 40 may be configured not to be mounted on the terminal device 3 but to exist as a separate device and to transmit the photographed image to the terminal device 3.

[0044] The terminal communication unit 42 is a communication means for the terminal device 3 to communicate with other devices. The terminal communication unit 42 communicates with the cooking appliance 2 and the control server 6 via the router 4, for example, using Wi-Fi (registered trademark). Alternatively, the terminal communication unit 42 may communicate directly with the cooking appliance 2, for example, using short-range wireless communication such as Bluetooth (registered trademark). Alternatively, the terminal communication unit 42 may communicate directly with the cooking appliance 2 via a wired connection, for example, using a cable such as a LAN (Local Area Network) cable.

[0045] The terminal screen 41 includes a display unit 41a that displays various information and an operation unit 41b that accepts operations by the user. The display unit 41a is, for example, a liquid crystal display. The operation unit 41b is, for example, a touch-type input key that allows input by touching the display unit 41a. Alternatively, the operation unit 41b may be a keyboard or physical buttons that are installed separately from the display unit 41a. The operation unit 41b may also be a voice input unit.

[0046] The cooking appliance control app 100 includes an analysis unit 110 that analyzes images to acquire ingredient information, an estimation unit 120 that estimates cooking information appropriate for the ingredient information, and a screen control unit 130 that controls the display unit 41a and operation unit 41b. The cooking appliance control app 100 transmits cooking information appropriate for the ingredient information to the cooking appliance 2. The ingredient information is information that includes at least one of the weight or size of the ingredient and the type of ingredient. The cooking information is information that indicates cooking conditions appropriate for each ingredient, and includes at least one of the amount of water to be used in the cooking appliance 2, cooking time, cooking temperature, and set pressure.

[0047] Analysis unit 110 is the analysis means of terminal device 3. Analysis unit 110 includes image storage unit 111, image analysis unit 112, and ingredient information storage unit 113. Image storage unit 111 stores images of ingredients photographed by the user using the photographing unit 40 of terminal device 3. Image analysis unit 112 analyzes the ingredient images stored in image storage unit 111 and outputs ingredient information. For example, image analysis unit 112 extracts ingredient images similar to those stored in image storage unit 111 from a database of ingredient information and ingredient images pre-stored in terminal device 3, and outputs the extracted ingredient information corresponding to the ingredient image stored in image storage unit 111. Alternatively, a database of ingredient information and ingredient images may be stored in control server 6, image analysis unit 112 may send ingredient images to control server 6 via terminal communication unit 42, control server 6 may extract ingredient images similar to those stored in image storage unit 111 from the database, and ingredient information corresponding to the extracted ingredient images may be sent from the database to image analysis unit 112. The ingredient information output by image analysis unit 112 is stored in ingredient information storage unit 113.

[0048] The ingredient information output by image analysis unit 112 may be configured so that it can be modified by the user. Image analysis unit 112 displays the ingredient information on display unit 41a via screen control unit 130. The user looks at the ingredient information displayed on display unit 41a, and if it differs from the ingredient photographed, operates operation unit 41b to modify the ingredient information.

[0049] The estimation unit 120 includes a cooking information storage unit 121, a cooking information estimation unit 122, and a correction information acquisition unit 123. The cooking information estimation unit 122 is a cooking information estimation means of the terminal device 3. The cooking information estimation unit 122 acquires the ingredient information analyzed by the analysis unit 110 from the ingredient information storage unit 113, and estimates and outputs cooking information corresponding to the ingredient information. The cooking information is estimated by extracting information that matches the type and weight or size of the ingredient contained in the ingredient information from a cooking information database for each ingredient pre-stored in the cooking information storage unit 121. FIG. 5 shows an example of the cooking information database according to the first embodiment. In the example of FIG. 5, the cooking information database includes data indicating the relationship between cooking time and cooking temperature for each weight of the ingredient when the ingredient is beef, and data indicating the amount of water to be used in the cooker 2 according to the weight of the ingredient. Note that in the example of FIG. 5, the cooking time and cooking temperature are not uniquely determined for each type and weight of the ingredient. The cooking information storage unit 121 presets one cooking time and one cooking temperature as a standard cooking time and one standard cooking temperature for, for example, the type and weight of ingredients. The cooking information estimation unit 122 outputs the standard cooking time and the standard cooking temperature as cooking information. The standard cooking time and the standard cooking temperature may be updated based on the cooking history of the cooker 2. The cooking information estimation unit 122 may also output cooking information corresponding to a cooking finish, such as firm, normal, or soft. The user pre-registers preference information indicating a preference for a cooking finish, such as firm, normal, or soft, in the cooking appliance control app 100. The cooking information estimation unit 122 outputs a cooking time and a cooking temperature according to the user's preference information as cooking information. This makes it possible to provide cooking information according to the user's preferences. The cooking information storage unit 121 may be installed in the control server 6. In this case, the cooking information estimation unit 122 transmits ingredient information to the control server 6 via the device communication unit 42 and requests that cooking information corresponding to the ingredient information be returned.

[0050] The correction information acquisition unit 123 of the estimation unit 120 acquires the correction information when the user corrects the cooking information output by the cooking information estimation unit 122. The correction information acquisition unit 123 is a correction information acquisition means of the terminal device 3. Specifically, the correction information acquisition unit 123 displays the cooking information output by the cooking information estimation unit 122 on the display unit 41a via the screen control unit 130. If the user wants to correct the cooking information displayed on the display unit 41a, the user operates the operation unit 41b to correct the cooking information. The corrected cooking information, that is, the correction information, is output to the correction information acquisition unit 123. The correction information acquisition unit 123 transmits the corrected cooking information changed by the correction information to the cooking information estimation unit 122. For example, if the user determines that the cooking time in the cooking information output by the cooking information estimation unit 122 is too long, the user shortens the cooking time. After correcting the cooking time, the cooking temperature corresponding to the corrected cooking time may be read from the cooking information storage unit 121 and displayed on the display unit 41a. The cooking information estimation unit 122 transmits the corrected cooking information to the cooker 2 via the terminal communication unit 42 as cooking setting information.

[0051] Furthermore, if the user determines that the cooking information displayed on display unit 41a does not need to be modified, they operate operation unit 41b to send a signal indicating that cooking will be performed using the displayed cooking information to cooking information estimation unit 122. Cooking information estimation unit 122 then transmits the cooking information as cooking setting information to cooker 2 via device communication unit 42. The cooking setting information is cooking information transmitted from cooking appliance control app 100 to cooker 2, and is either cooking information output by cooking information estimation unit 122 or modified cooking information modified by the user. Terminal communication unit 42 functions as cooking setting information transmission means of terminal device 3.

[0052] The screen control unit 130 controls the screen displayed on the display unit 41a in accordance with the output results of the analysis unit 110 and the estimation unit 120 and the user's operation of the operation unit 41b, and provides the user with a UI (User Interface).

[0053] Fig. 6 is a flowchart showing the operation of the cooking appliance control app 100. Figs. 7 to 13 show examples of displays on the display unit 41a when the cooking appliance control app 100 is running. The operation of the cooking appliance control app 100 will be described in detail using Figs. 6 and 7 to 13.

[0054] First, in step S101, when the cooking appliance control app 100 of the terminal device 3 is launched, the screen control unit 130 displays an initial selection screen 200 on the display unit 41a. FIG. 7 shows an example of the display (initial selection screen 200) on the display unit 41a when the cooking appliance control app 100 is running. As shown in FIG. 7, the initial selection screen 200 includes an ingredient photograph transition button 201 for transitioning to a mode in which ingredients are photographed and cooking information is acquired, an ingredient information input transition button 202 for transitioning to a mode in which ingredient information is directly entered and cooking information is acquired, and a history selection transition button 203 for transitioning to a mode in which cooking information is acquired from past history. The initial selection screen 200 is a screen that allows the user to select how to set cooking setting information. The user photographs ingredients to acquire ingredient information and presses the ingredient photograph transition button 201 to output the cooking information. If the user presses the ingredient photograph transition button 201, the process proceeds to step S102. The user inputs ingredient information by operating the operation unit 41b, and presses the ingredient information input transition button 202 when outputting cooking information. The user presses the history selection transition button 203 when selecting cooking information from history. If the user presses the ingredient information input transition button 202, the process proceeds to step S120. If the user presses the history selection transition button 203, the process proceeds to step S130. The ingredient photograph transition button 201, ingredient information input transition button 202, and history selection transition button 203 are displayed on the display unit 41a and are pressed by the user touching the position of each button. Note that the ingredient photograph transition button 201, ingredient information input transition button 202, and history selection transition button 203 may also be configured to be selected and pressed by operating a keyboard or physical buttons, which is the operation unit 41b installed separately from the display unit 41a. Furthermore, all of the UI parts such as buttons and text inputs described below may be touch-sensitive, that is, displayed on the display unit 41a and can be operated by the user by touching it, or may be operable using a keyboard or physical buttons.

[0055] First, the operation of the cooking appliance control application 100 when the user presses the food ingredient photograph transition button 201 on the initial selection screen 200 will be described.

[0056] In step S102, the user photographs ingredients using the photographing unit 40 of the terminal device 3. FIG. 8 shows an example of a display (photographing screen 210) on the display unit 41a when the cooking appliance control app 100 is running. The screen control unit 130 displays the photographing screen 210 on the display unit 41a. The photographing screen 210 includes a photographing guide 211 indicating the photographing range during photographing and a photographing button 212. In addition, an image recognized by the photographing unit 40 is displayed in the background of the photographing screen 210. The photographing guide 211 is a guide for specifying the position of ingredients when photographing to facilitate image analysis. The screen control unit 130 displays an image recognized by the camera, which is the photographing unit 40, on the photographing screen 210. The user moves the photographing unit 40 of the terminal device 3 so that the ingredients fit within the photographing guide 211. When the user presses the photographing button 212, the photographing unit 40 photographs an image. The photographed image is stored in the image storage unit 111. Next, the process proceeds to step S103.

[0057] In step S103, image analysis unit 112 analyzes the image captured in step S102 and stored in image storage unit 111, and outputs ingredient information. Next, the process proceeds to step S104.

[0058] In step S104, it is determined whether image analysis unit 112 was able to output the ingredient information. If image analysis unit 112 was able to output the ingredient information, image analysis unit 112 stores the ingredient information in ingredient information storage unit 113, and the process proceeds to step S106. If image analysis unit 112 determines that the ingredient information and ingredient image database does not contain an ingredient image similar to the ingredient image stored in image storage unit 111, and the ingredient information could not be output, the process proceeds to step S105.

[0059] In step S105, the user is prompted to select whether to photograph the ingredients again or to input ingredient information by operating the operation unit 41b. FIG. 9 shows an example of a display (image analysis failure screen 220) on the display unit 41a when the cooking appliance control app 100 is running. The screen control unit 130 displays the image analysis failure screen 220 on the display unit 41a. The image analysis failure screen 220 includes a re-photograph button 221 and an ingredient information input transition button 222. If the user wants to photograph the ingredients again, they press the re-photograph button 221. If the user presses the re-photograph button 221, the process returns to step S102, and the screen control unit 130 displays the photographing screen 210 again. If the user wants to input ingredient information by operating the operation unit 41b, they press the ingredient information input transition button 222. If the user presses the ingredient information input transition button 222, the process proceeds to step S120. The processing from step S120 onwards is the same as that when the ingredient information input transition button 202 is pressed on the initial selection screen 200 in step S101, and will be described in detail later.

[0060] In step S106, the screen control unit 130 displays the ingredient information stored in the ingredient information storage unit 113 by the image analysis unit 112 in step S104 on the display unit 41a as an image analysis result screen 230. FIG. 10 shows an example of the display (image analysis result screen 230) on the display unit 41a when the cooking appliance control app 100 is running. The image analysis result screen 230 includes ingredient type text 231, ingredient weight text 232, a correction button 233, and a confirmation button 234. The ingredient type text 231 is an area that displays the type of ingredient included in the ingredient information. The ingredient weight text 232 is an area that displays the weight of the ingredient included in the ingredient information. The ingredient weight text 232 may numerically represent the thickness of the ingredient instead of the weight of the ingredient. The ingredient weight text 232 may also be represented by a symbol such as S, M, or L that indicates the size of the ingredient. Furthermore, the weight text of the ingredients may be displayed differently depending on the type of ingredient, for example, meat may be displayed in grams, eggs may be displayed in number, etc. Next, the process proceeds to step S107.

[0061] In step S107, the user decides whether to correct the ingredient information displayed on the image analysis result screen 230 and presses the Correct button 233 or the Confirm button 234. If the user determines that the ingredient information displayed on the image analysis result screen 230 does not match the actual ingredient and wishes to make corrections, the user presses the Correct button 233. If the user presses the Correct button 233, the process proceeds to step S108. If the user determines that the ingredient information displayed on the image analysis result screen 230 corresponds to the actual ingredient and wishes to make no corrections, the user presses the Confirm button 234. If the user presses the Confirm button 234, the process proceeds to step S110.

[0062] In step S108, the user operates the operation unit 41b to modify the ingredient information. When the user presses the Modify button 233 in step S107, the screen control unit 130 changes the image analysis result screen 230 to enable input into the ingredient type text 231 and ingredient weight text 232. The user can change the display of the ingredient type text 231 and ingredient weight text 232 by operating the operation unit 41b. In the ingredient type text 231, the user operates the operation unit 41b to input the ingredient type using letters. Alternatively, the ingredient type text 231 may display candidate ingredient types, and the user may operate the operation unit 41b to select an ingredient type. In the ingredient weight text 232, the user operates the operation unit 41b to input the ingredient weight using numbers. Alternatively, the ingredient weight text 232 may display candidate ingredient weights in a selection form such as a drum roll, and the user may operate the operation unit 41b to select an ingredient weight. When the user has finished making the corrections, he or she presses the decision button 234. When the user presses the decision button 234, the corrected ingredient information is stored in the ingredient information storage unit 113. Next, the process proceeds to step S110.

[0063] In step S110, the cooking information estimation unit 122 estimates cooking information using the ingredient information stored in the ingredient information storage unit 113. The screen control unit 130 displays the cooking information as a cooking information display screen 240 on the display unit 41a. FIG. 11 shows an example of the display (cooking information display screen 240) on the display unit 41a when the cooking appliance control app 100 is running. The cooking information display screen 240 includes a cooking temperature text 241, a cooking time text 242, a water amount text 243, a correction button 244, and a send button 245. The cooking temperature text 241 is an area that displays the cooking temperature included in the cooking information. The cooking time text 242 is an area that displays the cooking time included in the cooking information. The water amount text 243 is an area that displays the amount of water used in the cooker 2, which is included in the cooking information. Next, proceed to step S111.

[0064] In step S111, the user determines whether or not to modify the cooking information displayed on the cooking information display screen 240, and presses the Modify button 244 or the Send button 245. If the user determines that the cooking information displayed on the cooking information display screen 240 is not appropriate for cooking the ingredients and wishes to modify it, the user presses the Modify button 244. If the user presses the Modify button 244, the process proceeds to step S112. If the user determines that the cooking information displayed on the cooking information display screen 240 is appropriate for cooking the ingredients and wishes to modify it, the user presses the Send button 245. If the user presses the Send button 245, the process proceeds to step S113.

[0065] In step S112, the user operates the operation unit 41b to modify the cooking information. When the user presses the modify button 244 in step S111, the screen control unit 130 changes the cooking information display screen 240 so that input is possible in the cooking temperature text 241, cooking time text 242, and water amount text 243. The user can change the display of the cooking temperature text 241, cooking time text 242, and water amount text 243 by operating the operation unit 41b. In the cooking temperature text 241, the user operates the operation unit 41b to input the cooking temperature using numbers. Alternatively, the cooking temperature text 241 may display candidate cooking temperatures in a selection form such as a drum roll, and the user may operate the operation unit 41b to select and input the cooking temperature. In the cooking time text 242, the user operates the operation unit 41b to input the cooking time using numbers. Alternatively, cooking time text 242 may display cooking time options in a selection form such as a drum roll, and the user may input the cooking time by operating operation unit 41b to select an option. When the user modifies the cooking temperature, it is preferable that screen control unit 130 automatically displays in cooking time text 242 the cooking time corresponding to the modified cooking temperature from the cooking information database corresponding to the type and weight of the ingredients stored in cooking information storage unit 121. When the user modifies the cooking time, it is preferable that screen control unit 130 automatically displays in cooking temperature text 241 the cooking temperature corresponding to the modified cooking time from the cooking information database corresponding to the type and weight of the ingredients stored in cooking information storage unit 121. This allows the cooking temperature and cooking time to be set appropriately, preventing the ingredients from being insufficiently cooked after cooking. Furthermore, screen control unit 130 may display on cooking information display screen 240 a message indicating the finished state of the food in an easy-to-understand manner for the user based on the cooking temperature and cooking time. For example, if the ingredient is beef, the message indicating the finished state may be "soft" or "firm." For example, if the food ingredient is an egg, the words indicating the finished state are "hard-boiled," "soft-boiled," etc. These words indicating the finished state are stored in the cooking information database in association with the cooking temperature and cooking time.Furthermore, in the water volume text 243, the user operates the operation unit 41b to input the water volume numerically. However, the water volume displayed in the water volume text 243 is a guide when the user manually adds water to the cooker 2, and is not included in the information sent to the cooker 2. Therefore, the screen control unit 130 may configure the cooking information display screen 240 so that the water volume text 243 cannot be modified. When the user has finished modifying the text, they press the send button 245. Next, proceed to step S113.

[0066] In step S113, if the user has made corrections, screen control unit 130 outputs corrected cooking information, which is the corrected cooking information, to correction information acquisition unit 123. If the user has made corrections, correction information acquisition unit 123 outputs a request to cook information estimation unit 122 to transmit the corrected cooking information as cooking setting information. If the user has not made corrections, correction information acquisition unit 123 outputs a request to cook information estimation unit 122 to transmit the cooking information estimated by cooking information estimation unit 122 as cooking setting information. If the user has made corrections, cooking information estimation unit 122 transmits the corrected cooking information to cooker 2 via terminal communication unit 42, or transmits the cooking information estimated by cooking information estimation unit 122 as cooking setting information if the user has not made corrections. Furthermore, if the user has made corrections to the cooking information, it is considered that the corrected cooking information will produce a finish that is more to the user's liking than the cooking information estimated by cooking information estimation unit 122. The correction information acquisition unit 123 may use the corrected cooking information to update the cooking information database in the cooking information storage unit 121. As a result, the next time the cooking appliance control app 100 outputs cooking information for the same ingredients, the cooking information estimated by the cooking information estimation unit 122 will be the user's preferred cooking information, reducing the hassle of the user having to make corrections. This completes the operation of the cooking appliance control app 100 when the user presses the ingredient photography transition button 201 on the initial selection screen 200.

[0067] Next, the operation of the cooking appliance control application 100 when the user presses the ingredient information input transition button 202 on the initial selection screen 200 will be described.

[0068] In step S120, the screen control unit 130 displays the ingredient information input screen 250. FIG. 12 shows an example of the display (ingredient information input screen 250) on the display unit 41a when the cooking appliance control app 100 is running. The ingredient information input screen 250 includes ingredient type text 251, ingredient weight text 252, and a confirm button 253. The user can input the ingredient type into the ingredient type text 251 by operating the operation unit 41b. The user can input the ingredient weight into the ingredient weight text 252 by operating the operation unit 41b. After completing the input, the user presses the confirm button 253. When the user presses the confirm button 253, the input ingredient information is stored in the ingredient information storage unit 113. Next, the process proceeds to step S110. The subsequent processing is the same as when the user presses the ingredient photograph transition button 201 on the initial selection screen 200, and therefore a description thereof will be omitted.

[0069] Next, the operation of the cooking appliance control application 100 when the user presses the history selection transition button 203 on the initial selection screen 200 will be described.

[0070] In step S130, the screen control unit 130 displays a cooking history selection screen 260. FIG. 13 shows an example of the display (cooking history selection screen 260) on the display unit 41a when the cooking appliance control app 100 is running. The cooking history selection screen 260 includes a cooking history display area 261 and an OK button 253. The cooking history display area 261 displays the cooking history sent by the user from the cooking appliance control app 100 to the cooker 2. The cooking history is a record of cooking information sent by the cooking appliance control app 100 to the cooker 2 in the past. When the cooking appliance control app 100 sends cooking information to the cooker 2, the cooking appliance control app 100 stores the food ingredient information and the cooking information in association with each other in a storage unit (not shown) of the cooking appliance control app 100. The screen control unit 130 displays the stored cooking history in the cooking history display area 261. It is desirable to add a date to the cooking history when storing it in the storage unit. The cooking history may also include the number of times it has been cooked. In this case, when the cooking appliance control app 100 transmits cooking information to the cooker 2, if there is a cooking history in which the ingredient information and cooking information are exactly the same, the number of times is incremented by 1. The user selects cooking information from the cooking history display area 261 and presses the confirm button 262. The cooking history may be displayed by type of ingredient. Alternatively, it may be displayed in descending order of the number of times the information has been transmitted to the cooker 2. Next, proceed to step S131.

[0071] In step S131, the screen control unit 130 sets the cooking information selected in step S130 as cooking setting information and outputs a request to send the cooking setting information to the cooking information estimation unit 122. The cooking information estimation unit 122 transmits the cooking setting information to the cooker 2 via the device communication unit 42. This completes the operation of the cooking appliance control app 100 when the user presses the history selection transition button 203 on the initial selection screen 200.

[0072] This concludes the description of the operation of the cooking appliance control application 100. Note that the control server 6 may be configured to perform some of the functions of the cooking appliance control application 100.

[0073] The cooking appliance control app 100 may also have a function for setting the user's cooking preference information as preference information. For example, the initial selection screen 200 may further include a button for transitioning to a screen for setting preference information. On the screen for setting preference information, the user operates the operation unit 41b to select and input their preference for each ingredient, such as soft or firm meat, or hard-boiled or soft-boiled eggs. The screen control unit 130 stores the preference information input by the user in a storage unit (not shown) of the cooking appliance control app 100. When predicting cooking information, the cooking information estimation unit 122 refers to the preference information and outputs cooking information suited to the user's preference information.

[0074] As described above, the cooking system 1 according to this embodiment comprises a heating cooker 2 that heats and cooks ingredients, a display unit 41a that has a photographing unit 40 and an operation unit 41b and is communicatively connected to the heating cooker 2, acquires ingredient information from ingredient images photographed by the photographing unit 40, and acquires and displays cooking information from the ingredient information, which is information necessary for cooking ingredients in the heating cooker 2, and a terminal device 3 that has a cooking setting information transmission means that, when a user operates the operation unit 41b to modify the cooking information, sets the modified information, which is the modified cooking information, as cooking setting information, and, when the user operates the operation unit 41b but does not modify the cooking information, sets the cooking information as cooking setting information and transmits the cooking setting information to the heating cooker.

[0075] With this configuration, the cooking information estimation means estimates cooking information suitable for the ingredient information, allowing the user to set cooking information suitable for the ingredients in the cooker 2. Furthermore, the cooking information estimated by the cooking information estimation means can be modified by the user using the modification information acquisition means, allowing the user to obtain cooking information that suits their preferences.

[0076] The ingredient information also includes a value indicating the type of ingredient and the weight or size of the ingredient. In the cooking device 2, cooking information suitable for the ingredient is available according to the type of ingredient and the weight or size of the ingredient. Therefore, cooking information suitable for the ingredient can be obtained by having the cooking information estimation means estimate cooking information using the type of ingredient and the weight or size of the ingredient.

[0077] The cooking information also includes at least one of the cooking time for cooking the ingredients and the cooking temperature at which the ingredients are cooked. In the heat cooker 2, the finished result of cooking ingredients varies depending on the temperature and time. Therefore, the cooking information that serves as the cooking setting information to be set in the heat cooker 2 uses at least one of the cooking time for cooking the ingredients and the cooking temperature at which the ingredients are cooked. This allows the heat cooker 2 to produce a consistent finished result when cooking ingredients.

[0078] Furthermore, if the photographing unit 40 is unable to analyze the image captured and obtain food ingredient information, the terminal device 3 displays on the display unit 41a a message requesting the user to photograph the food ingredient again. With this configuration, if the analysis unit 110 is unable to obtain food ingredient information by analyzing the food ingredient image captured by the user, the user can be prompted to photograph the food ingredient image again.

[0079] Furthermore, if the terminal device 3 is unable to obtain ingredient information by analyzing the image captured by the photographing unit 40, the display unit 41a displays a message requesting the user to input ingredient information. With this configuration, if the analysis unit 110 is unable to obtain ingredient information by analyzing the ingredient image captured by the user, the user can input ingredient information, and the analysis unit 110 can obtain the ingredient information.

[0080] The program installed in the terminal device 3 according to this embodiment is a program for controlling the cooker 2, which heats and cooks ingredients, and includes: cooking information estimation means for analyzing an image to acquire ingredient information and estimating cooking information necessary for cooking ingredients in the cooker 2 from the ingredient information; and cooking setting information transmission means for, if the user modifies the cooking information, setting the modified cooking information as cooking setting information; and, if the user does not modify the cooking information, setting the cooking information as cooking setting information and transmitting the cooking setting information to the cooker. With this configuration, the cooking information estimation means estimates cooking information appropriate for the ingredient information, allowing the user to set cooking information appropriate for the ingredients in the cooker 2. Furthermore, the user can modify the cooking information estimated by the cooking information estimation means using the modification information acquisition means, thereby obtaining cooking information that suits the user's preferences.

[0081] In addition to the cooking time and cooking temperature, the cooking information may also include the pressure used to cook the ingredients. In the cooking appliance 2, changing the pressure changes the finished product. Therefore, using pressure as cooking information allows the cooking appliance 2 to produce a consistent finished product when cooking ingredients. The cooking time and cooking temperature included in the cooking information may also be a temperature profile that shows temperature changes over time. For example, if beef is slowly cooked at a low temperature and then suddenly heated to a high temperature, contraction of the meat fibers is suppressed and the meat becomes tender. The cooking information storage unit 121 stores this relationship between the passage of time and cooking temperature for each ingredient in a cooking information database. This allows cooking to be performed in a way that is more suitable for the ingredients.

[0082] Embodiment 2 Next, a description will be given of a second embodiment. In this embodiment, the configuration of the cooking system 1 and the structure of the cooking appliance 2 are the same as those of the first embodiment, and therefore a description thereof will be omitted. In this embodiment, the terminal device 3 includes a cooking appliance control app 100 that displays image capture instruction information so that the user can capture an ingredient image suitable for the analysis unit 110 to infer ingredient information during the process of photographing the ingredient.

[0083] 6, the process of photographing ingredients in step S102 differs from that in embodiment 1. Therefore, the process of photographing ingredients using the photographing unit 40 of the terminal device 3 will be described.

[0084] 14, 15, and 17 are examples of the display on the display unit 41a when an application is executed to photograph an ingredient image. Photographing ingredients in this embodiment will be described using FIGS. 14, 15, and 17. FIG. 14 is an example of the display on the display unit 41a when the application is executed (photography method selection screen 270). The photography method selection screen 270 is a screen that is displayed on the display unit 41a after the ingredient photography transition button 201 is pressed on the initial selection screen 200. The photography method selection screen 270 includes a container selection button 271 and a cutting board selection button 272 for selecting what to place the ingredient on when photographing. The container selection button 271 is a button that is selected when placing ingredients in the container 10 of the cooking appliance 2 and photographing them. The cutting board selection button 272 is a button that is selected when placing ingredients on a cutting board and photographing them.

[0085] 15 is an example of the display (container-used photographing screen 280) on the display unit when the application is executed. The container-used photographing screen 280 is a screen that is displayed on the display unit 41a after the container selection button 271 is pressed on the photographing method selection screen 270. The container-used photographing screen 280 includes a container opening guide 281, a photographing marker guide 282, and a photographing button 283. Furthermore, on the container-used photographing screen 280, an image recognized by the photographing unit 40 is displayed in the background. The container opening guide 281 and the photographing marker guide 282 are photographing instruction information that instructs the distance and angle between the photographing unit 40 and the food ingredient, which is the subject.

[0086] FIG. 16 is a perspective view of a container 10 of a cooking device 2 according to the present embodiment. As shown in FIG. 16, the container 10 is pot-shaped, i.e., a substantially cylindrical shape with a flat bottom 10b and an opening 10a on the surface opposite the bottom 10b. The container 10 is provided with a scale 10c indicating the distance from the bottom 10b and a photographing mark 10d serving as a guide for photographing the food. When photographing food in the container 10, the user adjusts the positions of the photographing unit 40 and the container 10 so that the container opening guide 281 overlaps with the opening 10a of the container 10 included in the image recognized by the photographing unit 40 on the container-use photographing screen 280. Furthermore, the user adjusts the positions of the photographing unit 40 and the container 10 so that the photographing mark guide 282 overlaps with the photographing mark 10d of the container 10 included in the image recognized by the photographing unit 40. When the container opening guide 281 and the opening 10a overlap, and further when the photographing mark guide 282 and the photographing mark 10d overlap, the user presses the photographing button 283.

[0087] In this way, by capturing an image so that the container opening guide 281 and the opening 10a overlap, it is possible to capture an image in which the distance and angle between the container 10 and the imaging unit are always constant. Images of various ingredients placed in the container 10 and captured so that the container opening guide 281 and the opening 10a overlap are pre-registered as ingredient images in a database of ingredient information and ingredient images. This makes the ingredient images captured by the user similar to the ingredient images registered in the database, facilitating analysis of the ingredient images by the analysis unit 110. The size and shape of the container opening guide 281 may be changed depending on the cooking appliance 2 owned by the user. The type of cooking appliance 2 owned by the user is pre-registered in the cooking appliance control app 100. On the container use imaging screen 280, the container opening guide 281 is displayed according to the size and shape of the container 10 of the registered cooking appliance 2. By creating a container opening guide 281 corresponding to the cooking appliance 2 used in the cooking system 1, it is possible to accurately capture an image in which the distance and angle between the container 10 and the imaging unit are always constant. Although the container 10 has been described as being substantially cylindrical, the shape of the container 10 is not limited as long as it can accommodate food ingredients.

[0088] Furthermore, the photographing marker 10d and the scale 10c are positioned on the container 10 so that the scale 10c appears in an image captured when the container opening guide 281 and the opening 10a overlap, and the photographing marker guide 282 and the photographing marker 10d overlap. In other words, the food ingredient image captured when the container opening guide 281 and the opening 10a overlap, and the photographing marker guide 282 and the photographing marker 10d overlap will include the scale 10c. When the food ingredient image is analyzed by the analysis unit 110, comparing the scale 10c on the food ingredient image with the food ingredient makes it easier to analyze the weight or size of the food ingredient.

[0089] In order to obtain food image suitable for inferring food information, it is desirable to photograph the food directly in the container 10. However, when performing low-temperature cooking, the food may be photographed while placed in a heat-resistant bag to smoothly transition to the actual cooking process. In this case, an image of the food placed in the bag, placed in the container 10, and photographed so that the container opening guide 281 and the opening 10a overlap is registered in advance as a food image in the database of food information and food image.

[0090] 17 shows an example of the display (cutting board use photographing screen 290) on the display unit when the application is executed. Cutting board use photographing screen 290 is a screen that is displayed on display unit 41a after cutting board selection button 272 is pressed on photographing method selection screen 270. Cutting board use photographing screen 290 includes cutting board frame guide 291 and photographing button 292. Furthermore, on cutting board use photographing screen 290, an image recognized by photographing unit 40 is displayed in the background. Cutting board frame guide 291 is photographing instruction information that indicates the distance and angle between photographing unit 40 and the food ingredient that is the subject.

[0091] When photographing food placed on a cutting board, the user adjusts the positions of the photographing unit 40 and the cutting board on which the food is placed so that the cutting board frame guide 291 overlaps with the frame of the cutting board included in the image recognized by the photographing unit 40 on the photographing screen 290 using a cutting board. When the cutting board frame guide 291 overlaps with the frame of the cutting board, the user presses the photographing button 292.

[0092] In this way, by capturing an image of the cutting board frame guide 291 overlapping the cutting board on which the food is placed, it is possible to capture an image in which the distance and angle between the food and the photographing unit 40 are always constant. Images of various food items captured with the cutting board frame guide 291 overlapping the cutting board on which the food is placed are pre-registered as food image data in a database of food information and food images. This makes the food image captured by the user similar to the food image registered in the database, facilitating analysis of the food image by the analysis unit 110. Note that the size and shape of cutting boards vary depending on the product. Therefore, the cooking appliance control app 100 may be provided with a function that allows the user to register the size and shape of the cutting board. The cutting board frame guide 291 changes depending on the size and shape of the registered cutting board, and it is desirable that when capturing an image of the cutting board frame guide 291 overlapping the cutting board on which the food is placed, the distance and angle between the food placed on the cutting board and the photographing unit are constant.

[0093] As described above, the terminal device of the cooking system according to this embodiment is characterized in that, when a user photographs an ingredient, the display unit 41a displays shooting instruction information instructing the user on at least one of the angle and distance between the ingredient and the photographing unit. This configuration ensures that when a user photographs an ingredient image in accordance with the shooting instruction information, the image is captured with a constant distance and angle between the ingredient and the photographing unit 40. Photographs of various ingredients are taken in accordance with the shooting instruction information to create ingredient images for a database of ingredient information and ingredient images. This results in the ingredient images photographed by the user and the ingredient images registered in the database being similar, making it easier for the analysis unit 110 to analyze the ingredient images.

[0094] Furthermore, the cooking appliance 2 of the cooking system according to this embodiment includes a container 10 that contains ingredients, and the terminal device 3 is characterized in that, when the ingredients contained in the container 10 are photographed with the photographing unit 40, the terminal device 3 displays on the display unit 41a photographing instruction information that instructs the user on at least one of the angle and distance between the container 10 and the photographing unit 40. This makes it possible to display photographing instruction information according to the shape of the container 10, and to accurately photograph an image in which the distance and angle between the container 10 and the photographing unit are always constant.

[0095] The container 10 also has a scale 10c, which is positioned so that it will appear in the image when the image is taken in accordance with the photography instruction information. This configuration makes it easy to analyze the weight or size of the food ingredient by comparing the scale 10c on the food ingredient image with the food ingredient itself.

[0096] Embodiment 3 Next, a third embodiment will be described. In this embodiment, the configuration of the cooking system 1 and the structure of the cooking appliance 2 are the same as those in the first embodiment, and therefore their description will be omitted. In this embodiment, the analysis unit 110 of the terminal device 3 analyzes food ingredient images using a known learning algorithm. Figure 18 is a block diagram showing the relationship between the first learning device 500 and the terminal device regarding food ingredient image analysis in the cooking system according to the third embodiment. The food ingredient image analysis in this embodiment will be described using Figure 18.

[0097] 18, in this embodiment, the control server 6 of the cooking system 1 includes a first learning device 500 and a first trained model storage unit 503 that stores the first trained model generated by the first learning device 500. The first learning device 500 learns ingredient images and ingredient information and generates a first trained model that infers ingredient information from ingredient images. The first learning device 500 includes a first learning data acquisition unit 501 and a first model generation unit 502.

[0098] First learning data acquisition unit 501 acquires first learning data. The first learning data includes food image data captured by the user and food information related to the food image data. For example, the acquired food image data and food information include food image data captured by the user using cooking appliance control app 100 in steps S102 to S108 of the first embodiment and food information corrected by the user from the food information output by analysis unit 110.

[0099] The first model generation unit 502 learns information such as the type, weight, or size of ingredients depicted in the ingredient images from the ingredient images, for example, by so-called supervised learning in accordance with a neural network model. Here, supervised learning refers to a method of providing the first learning device 500 with a set of data of input and result (label), i.e., first learning data, to learn the features of the first learning data and infer the result from the input.

[0100] A neural network is composed of an input layer consisting of multiple neurons, an intermediate layer (hidden layer) consisting of multiple neurons, and an output layer consisting of multiple neurons. The intermediate layer may be one layer or two or more layers. For example, in a three-layer neural network as shown in Figure 19, when multiple inputs are input to the input layer (X1-X3), the values ​​are multiplied by weight W1 (w11-w16) and input to the intermediate layer (Y1-Y2), and the result is further multiplied by weight W2 (w21-w26) and output from the output layer (Z1-Z3). This output result varies depending on the values ​​of weights W1 and W2.

[0101] In this embodiment, the neural network learns information such as the type, weight, and size of the ingredients depicted in the ingredient images through so-called supervised learning based on the first learning data, which is a combination of ingredient images and ingredient information acquired by first learning data acquisition unit 501. That is, the neural network learns by inputting ingredient images to the input layer and adjusting the weights W1 and W2 so that the results output from the output layer approach the ingredient information.

[0102] The first model generation unit 502 generates and outputs a first trained model by performing the above-described learning. The output first trained model is stored in the first trained model storage unit 503 of the control server 6. Note that the learning algorithm used in the first learning device 500 may be deep learning, which learns to extract features themselves, or other known methods of machine learning, such as genetic programming, inductive logic programming, or support vector machines.

[0103] Next, the learning process performed by the first learning device 500 will be described with reference to Fig. 20. Fig. 20 is a flowchart showing the first learning procedure performed by the first learning device 500 according to this embodiment.

[0104] In step b1, the first learning data acquisition unit 501 acquires an ingredient image and ingredient information corresponding to the ingredient image. The ingredient image and ingredient information may be, for example, an ingredient image captured by the user using the cooking appliance control app 100 in steps S102 to S108 of the first embodiment, and ingredient information corrected by the user from the ingredient information output by the analysis unit 110. Alternatively, for example, the person operating the control server 6 may input the ingredient image and ingredient information into the first learning data acquisition unit 501 for learning purposes by the first learning device 500. Next, proceed to step b2.

[0105] In step b2, the first model generation unit 502 learns information such as the type, weight, or size of the ingredients shown in the ingredient images through so-called supervised learning in accordance with the first training data, which is a combination of ingredient images and ingredient information acquired by the first training data acquisition unit 501, and generates a first trained model. Next, the process proceeds to step b3.

[0106] In step b3, the first trained model storage unit 503 stores the first trained model generated by the first model generation unit 502.

[0107] With the above processing, the first learning procedure is completed.

[0108] Note that either or both of the first learning device 500 and the first trained model storage unit 503 may be located on the terminal device 3 rather than on the control server 6. However, because the accuracy of the generated first trained model improves when multiple users send first learning data to the first learning device 500 by operating their respective terminal devices 3, it is desirable to locate the first learning device 500 and the first trained model generated by the first learning device 500 on the control server 6 to which multiple terminal devices 3 are connected.

[0109] In this embodiment, in FIG. 18, the image storage unit 111 and the image analysis unit 112 of the analysis unit 110 function as an inference device using the first trained model.

[0110] As described in the first embodiment, the image storage unit 111 stores images of ingredients photographed by the user using the photographing unit 40.

[0111] The image analysis unit 112 uses the first trained model to infer ingredient information from the ingredient images stored in the image storage unit 111.

[0112] 21 is a flowchart showing the steps of analyzing an ingredient image using the first trained model according to this embodiment. The process by which the analysis unit 110 infers ingredient information will be described with reference to FIG.

[0113] First, in step c1, the image storage unit 111 stores an ingredient image. This process is included in step S102 in Fig. 6. Next, the process proceeds to step c2.

[0114] In step c2, the image analysis unit 112 sends the ingredient image to the first stored model in the first trained model storage unit 503 of the control server 6 to obtain ingredient information. The image analysis unit 112 may first obtain the first trained model from the first trained model storage unit 503 of the control server 6 and then input the ingredient image into the first trained model to obtain ingredient information. Next, the process proceeds to step c3.

[0115] In step c3, the image analysis unit 112 stores the ingredient information obtained by the first trained model in the ingredient information storage unit 113. Next, the process proceeds to step c4.

[0116] In step c4, the estimation unit 120 estimates cooking information using the ingredient information stored in the ingredient information storage unit 113. This process corresponds to step S110 in Fig. 6. This completes the process of inferring ingredient information.

[0117] As described above, cooking system 1 includes first learning device 500, which generates a first trained model that infers ingredient information from ingredient images. The ingredient images and ingredient information learned by first learning device 500 can be acquired by a user operating terminal device 3, and first learning device 500 can improve the accuracy of inferring ingredient information from ingredient images by updating the first trained model.

[0118] As described above, cooking system 1 according to the present embodiment further includes image analysis unit 112 that outputs ingredient information from ingredient images using a first trained model for inferring ingredient information from ingredient images. This configuration enables analysis unit 110 to infer ingredient information with high accuracy.

[0119] The first learning device 500 according to this embodiment includes a first learning data acquisition unit that acquires first learning data including food image data and food information related to the food image data, and a first model generation unit that uses the first learning data to generate a first trained model for inferring food information from food image data. As the user accumulates first learning data by photographing food image data and inputting food information using the terminal device 3, the first learning device 500 can update and improve the accuracy of the first trained model that infers food information from food image data. By incorporating the first learning device 500, the cooking system 1 can be configured to highly accurately infer food information from food image data. Furthermore, by using a first trained model that employs a learning algorithm to infer food information from food image data, highly accurate inference can be achieved even when the distance and angle between the food and the photographing unit 40 are not constant. Furthermore, the accuracy of inferring food information is improved even for food image data in which the food is packaged, allowing for greater flexibility when photographing food.

[0120] Embodiment 4 Next, a fourth embodiment will be described. In this embodiment, the configuration of the cooking system 1 and the structure of the cooking appliance 2 are the same as those in the first embodiment described above, and therefore their description will be omitted. In this embodiment, the estimation unit 120 of the terminal device 3 estimates cooking information using a known learning algorithm. FIG. 22 is a block diagram showing the relationship between the second learning device 600 and the terminal device 3 regarding cooking information estimation in the cooking system according to the fourth embodiment. Cooking information estimation in this embodiment will be described using FIG. 22.

[0121] As shown in FIG. 22, in this embodiment, the control server 6 of the cooking system 1 includes a second learning device 600 and a second learned model storage unit 603 that stores the second learned model generated by the second learning device 600. The second learning device 600 learns cooking setting information, which is cooking information set in the cooker 2, and generates a second learned model that infers cooking information for cooking ingredients in the cooker 2 from ingredient information. The second learning device 600 includes a second learning data acquisition unit 601 and a second model generation unit 602. Furthermore, in this embodiment, the cooking information is inferred using the second learned model and a cooking information database is not used, so the cooking information storage unit 121 in which the cooking information database is stored is not necessary.

[0122] The second learning data acquisition unit 601 acquires ingredient information and cooking setting information as second learning data. The ingredient information and cooking setting information are transmitted to the second learning data acquisition unit 601 of the control server 6 at the same time that the user operates the terminal device 3 to transmit the cooking setting information to the cooker 2, for example, in step S113 of Fig. 6. In addition, correction information obtained by manually correcting the cooking information by operating the terminal device 3 may also be transmitted to the second learning data acquisition unit 601 and used as second learning data.

[0123] Furthermore, the second learning data may be user preference information. The preference information is information that is registered in advance by the user in the cooking appliance control app 100 of the terminal device 3. The preference information is transmitted from the terminal device 3 to the second learning data acquisition unit 601 at the same time that the ingredient information and cooking setting information are transmitted.

[0124] The timing at which the second learning data is transmitted from the terminal device 3 to the second learning data acquisition unit 601 is not limited to the timing at which the cooking setting information is transmitted to the cooker 2. For example, when cooking by the cooker 2 is completed, the cooker 2 notifies the terminal device 3 that the cooking is completed. Upon being notified that cooking is completed, the terminal device 3 may prompt the user to evaluate the doneness of the cooking, and transmit the second learning data together with the evaluation of the doneness to the second learning data acquisition unit 601.

[0125] The second model generation unit 602 learns cooking information suitable for the ingredient information based on the second training data including the ingredient information and cooking setting information. That is, it generates a trained model that infers cooking setting information to be set in the cooker 2 from the ingredient information acquired by the analysis unit 110 through analysis of the ingredient images or the ingredient information input by the user through operation of the terminal device 3.

[0126] The second model generation unit 602 can use known algorithms such as supervised learning, unsupervised learning, reinforcement learning, etc. As an example, a case where reinforcement learning is applied will be described.

[0127] In reinforcement learning, an agent (acting subject) in a certain environment observes the current state (environmental parameters) and decides on the action to take. The environment changes dynamically depending on the agent's actions, and the agent is given a reward according to the changes in the environment. The agent repeats this process and learns the course of action that will obtain the most reward through a series of actions. Q-learning and TD-learning are known as representative reinforcement learning methods. For example, in the case of Q-learning, the general update formula for the action value function Q(s, a) is expressed as the following equation 1.

[0128]

number

[0129] In equation 1, st represents the state of the environment at time t, and at represents the action at time t. Action at changes the state to st+1. rt+1 represents the reward obtained due to the change in state, γ represents the discount rate, and α represents the learning coefficient. Note that γ is in the range of 0<γ≦1, and α is in the range of 0<α≦1. Action at occurs when the user determines the cooking setting information to be sent to the cooking appliance 2, and the combination of ingredient information and cooking setting information, together with preference information, represents state st, and the best action at for state st at time t is learned.

[0130] The update formula expressed by Equation 1 increases the action value Q if the action value Q of the action a with the highest Q value at time t+1 is greater than the action value Q of the action a executed at time t, and decreases the action value Q in the opposite case. In other words, the action value function Q(s, a) is updated so that the action value Q of the action a at time t approaches the best action value at time t+1. As a result, the best action value in a certain environment is propagated sequentially to the action value in the previous environment.

[0131] As described above, when the second trained model is generated by reinforcement learning, the second model generation unit 602 includes a reward calculation unit 602a and a function update unit 602b.

[0132] The reward calculation unit 602a calculates a reward for the combination of ingredient information and cooking setting information, and preference information. The reward calculation unit 602a calculates a reward r based on the user's level of satisfaction with the finished cooking. The user's satisfaction is determined, for example, by using an evaluation of the finished cooking input by the user into the terminal device 3 after cooking using the cooking appliance 2 is completed. If the user's evaluation of the finished cooking is high, the reward r is increased, and if the evaluation is low, the reward r is decreased.

[0133] The user's satisfaction level may also be determined based on the number of times the user has cooked using the cooker 2 with the same ingredient information and cooking setting information. That is, the reward calculation unit 602a increases the reward r if the user has previously cooked using the cooker 2 with the same ingredient information and cooking setting information. The more the user cooks using the cooker 2 with the same ingredient information and cooking setting information, the more the reward calculation unit 602a increases the reward r.

[0134] Furthermore, when correction information is used as the second learning data, the user's satisfaction level may be determined based on whether or not the user has made corrections. That is, if the user has made corrections to the cooking information estimated by the estimation unit 120, the reward calculation unit 602a determines that the user's satisfaction with the cooking information before the correction is low, and reduces the reward r for the cooking information before the correction. The more the user corrects the cooking information estimated by the estimation unit 120 and cooks using the cooker 2, the more the reward calculation unit 602a reduces the reward r.

[0135] Furthermore, if the user performs additional heating after cooking setting information is transmitted from the terminal device 3 and cooking is performed on the cooker 2, it may be determined that the user's satisfaction is low. In this case, when the user operates the cooker operation unit 33a of the cooker 2 to set additional heating, the cooker 2 notifies the terminal device 3 that additional heating will be performed. When the terminal device 3 receives the notification of additional heating, it notifies the second learning device 600 that the user's satisfaction with the cooking setting information is low. The reward calculation unit 602a reduces the reward r for the cooking setting information.

[0136] The function update unit 602b updates the function for inferring cooking information according to the reward calculated by the reward calculation unit 602a, and outputs the updated function to the second trained model storage unit 603. For example, in the case of Q-learning, the action value function Q(s t ,a t ) is used as a function to calculate cooking information.

[0137] The learning described above is repeated. The second learned model storage unit 603 stores the action-value function Q(s t ,a t ), i.e., store the second trained model.

[0138] 23 is a flowchart of the learning process of the second learning device 600 according to this embodiment. The learning process of the second learning device 600 will be described with reference to FIG.

[0139] First, in step d1, the second learning data acquisition unit 601 acquires the combination of ingredient information and cooking setting information, and preference information as second learning data. Next, the process proceeds to step d2.

[0140] In step d2, the second model generation unit 602 calculates a reward for the combination of ingredient information and cooking setting information and the preference information. Specifically, the reward calculation unit 602a acquires the combination of ingredient information and cooking setting information and the preference information, and determines whether to increase the reward (step d3) or decrease the reward (step d4) based on the user's satisfaction, which is the reward criterion.

[0141] If the reward calculation unit 602a determines that the reward should be increased, it increases the reward in step d3. On the other hand, if the reward calculation unit 602a determines that the reward should be decreased, it decreases the reward in step d4. Next, the process proceeds to step d5.

[0142] In step d5, the function update unit 602b updates the action value function Q(s t ,a t ) to update the

[0143] The second learning device 600 repeatedly executes the above steps d1 to d5, and stores the generated action-value function Q(st,at) in the second learned model storage unit 603 as the second learned model.

[0144] Note that either or both of the second learning device 600 and the second trained model storage unit 603 may be located on the terminal device 3 rather than on the control server 6. By arranging the second learning device 600 and the second trained model storage unit 603 on the control server 6 and having multiple users send second learning data to the second learning device 600 by operating their respective terminal devices 3, the generated second trained model will be a model that infers cooking information that will provide high satisfaction to many users. On the other hand, if the second learning device 600 and the second trained model storage unit 603 are arranged on the terminal device 3, the generated second trained model will be a model that infers cooking information that will provide high satisfaction to individual users of the terminal device 3.

[0145] Note that even if the second learning device 600 and the second learned model storage unit 603 are located in the control server 6, the second learning device 600 may learn from the ingredient information and cooking setting information sent by each user and generate a second learning model corresponding to each user. When generating a second learning model corresponding to each user, the second learning model will be a second learned model that infers cooking information that will highly satisfy the corresponding user, so the second learning device 600 can learn the ingredient information and cooking setting information and generate the second learning model without using preference information.

[0146] In this embodiment, in FIG. 22, the ingredient information storage unit 113 of the analysis unit 110 and the cooking information estimation unit 122 of the estimation unit 120 function as an inference device using the second trained model.

[0147] As described in the first or third embodiment, ingredient information storage unit 113 stores ingredient information used in cooking.

[0148] The cooking information estimation unit 122 uses the second trained model to infer cooking information from the ingredient information stored in the ingredient information storage unit 113.

[0149] 24 is a flowchart showing the procedure for inferring cooking information using the second trained model according to this embodiment. The process by which the inference unit 120 infers cooking information will be described with reference to FIG.

[0150] First, in step e1, ingredient information storage unit 113 stores ingredient information. This process is included in step S104 or step S120 in Fig. 6. Next, the process proceeds to step e2.

[0151] In step e2, the cooking information estimation unit 122 transmits the ingredient information stored in the ingredient information storage unit 113 to the second learned model in the second learned model storage unit 603 of the control server 6 to obtain cooking information. The cooking information estimation unit 122 may first obtain the second learned model from the second learned model storage unit 603 of the control server 6 and input the ingredient information into the second learned model to obtain cooking information. Alternatively, the cooking information estimation unit 122 may transmit the ingredient information and preference information to the second learned model. Next, the process proceeds to step e3.

[0152] In step e3, the cooking information estimation unit 122 stores the cooking information obtained by the second trained model in a temporary storage area (not shown) of the terminal device 3. Next, the process proceeds to step e4.

[0153] In step e4, the screen control unit 130 generates the cooking information display screen 240 using the cooking information stored in step e3 and displays it on the display unit 41a. This process corresponds to step S111 in Figure 6. This completes the process of displaying cooking information.

[0154] In this embodiment, a case where reinforcement learning is applied to the learning algorithm used by the cooking information estimation unit 122 has been described, but the present invention is not limited to this. As for the learning algorithm, other than reinforcement learning, supervised learning, unsupervised learning, semi-supervised learning, etc. can also be applied.

[0155] Furthermore, the learning algorithm used in the second model generation unit 602 can be deep learning, which learns to extract the features themselves, or machine learning can be performed according to other known methods, such as neural networks, genetic programming, inductive logic programming, and support vector machines.

[0156] As described above, the second learning device 600 according to this embodiment includes a second learning data acquisition unit 601 that acquires second learning data including ingredient information and cooking information for the ingredient information, and a second model generation unit 602 that uses the second learning data to generate a second trained model for inferring cooking information from ingredient information. By generating a second learning model that infers cooking information from ingredient information, it is possible to infer cooking information that will highly satisfy users.

[0157] Furthermore, ingredient information storage unit 113, which stores ingredient information, and cooking information estimation unit 122, which uses a second trained model for inferring cooking information from the ingredient information stored in ingredient information storage unit 113 and outputs cooking information from the ingredient information stored in ingredient information storage unit 113, function as an inference device. This allows cooking information estimation unit 122 to infer cooking information that will highly satisfy the user.

[0158] The second learning device 600 is also communicatively connected to a terminal device 3 that transmits cooking information to a cooker 2 that heats and cooks ingredients in response to a user's operation. The second model generation unit 602 includes a reward calculation unit 602a that calculates a reward such that the reward increases the more times the user transmits cooking information to the cooker 2, and a function update unit 602b that updates a function for inferring cooking information based on the reward. With this configuration, the second learning device 600 can determine that the corresponding cooking information is highly satisfying for the user when the user has cooked using the cooking information many times with the cooker 2. The reward calculation unit 602a increases the reward for cooking information that provides high user satisfaction, allowing the second learning device 600 to generate a second learning model that infers cooking information that provides high user satisfaction.

[0159] The second learning device 600 is also communicatively connected to a terminal device 3 that modifies cooking information in response to user operation. The second model generation unit 602 includes a reward calculation unit 602a that calculates a reward based on the number of times the user modifies the cooking information, and a function update unit 602b that updates a function for inferring cooking information based on the reward. With this configuration, the second learning device 600 can determine that cooking information that the user modifies frequently is of low satisfaction to the user. The reward calculation unit 602a reduces the reward for cooking information that provides low user satisfaction, allowing the second learning device 600 to generate a second learning model that does not infer cooking information that provides low user satisfaction.

[0160] Furthermore, the second learning device 600 is connected to multiple terminal devices 3, and the reward calculation unit 602a subtracts a reward the more times multiple users transmit cooking information to the cooking appliance 2. With this configuration, the second learning device 600 can generate a second learning model that infers cooking information that is highly satisfying to many users.

[0161] The configurations shown in the above embodiments are examples of the content of the present invention, and may be combined with other known technologies, and some of the configurations may be omitted or modified within the scope of the gist of the present invention.

[0162] Examples of aspects that may be included in the present disclosure are set forth below as appendices. (Appendix 1) A cooking device that heats and cooks ingredients; a terminal device comprising a photographing unit and an operation unit, connected to the cooking appliance so as to be able to communicate with the cooking appliance, a display unit which acquires food ingredient information from food ingredient images photographed by the photographing unit, and acquires and displays cooking information from the food ingredient information, which is information necessary for cooking the food ingredients in the cooking appliance; and a cooking setting information transmission means which, when a user operates the operation unit to modify the cooking information, sets the modified information as cooking setting information, and, when a user operates the operation unit to not modify the cooking information, sets the cooking information as cooking setting information and transmits the cooking setting information to the cooking appliance; A cooking system comprising: (Appendix 2) The cooking system of claim 1, wherein the ingredient information includes a value indicating the type of ingredient and a value indicating the weight or size of the ingredient. (Appendix 3) The cooking system described in Appendix 1, wherein the cooking information includes at least one of a cooking time for the ingredients and a cooking temperature for the ingredients. (Appendix 4) The terminal device When a user photographs the food material with the photographing unit, photographing instruction information instructing the user on at least one of the angle and the distance between the food material and the photographing unit is displayed on the display unit. 4. The cooking system of any one of claims 1 to 3. (Appendix 5) The cooking device includes a container for accommodating food to be cooked, When the user photographs the food material contained in the container with the photographing unit, the terminal device displays, on the display unit, photographing instruction information instructing the user on at least one of the angle and distance between the container and the photographing unit. 4. The cooking system of any one of claims 1 to 3. (Appendix 6) The cooking system described in Appendix 5, characterized in that the container is equipped with a scale, and the scale is positioned in a position that will appear in the captured image when photography is performed in accordance with the photography instruction information. (Appendix 7) The cooking system described in any one of Appendices 1 to 3, wherein if the terminal device is unable to obtain the ingredient information by analyzing the image captured by the photographing unit, the display unit displays a message requesting the user to photograph the ingredient again. (Appendix 8) The cooking system according to any one of appendices 1 to 3, wherein if the terminal device is unable to obtain the ingredient information by analyzing the image captured by the photographing unit, the terminal device displays on the display unit a request for the user to input the ingredient information. (Appendix 9) A program installed in a computer to control a cooking device that heats and cooks ingredients, cooking information estimation means for analyzing an image to acquire ingredient information and estimating cooking information, which is information necessary to cook the ingredients in the cooking appliance, from the ingredient information; and cooking setting information transmission means for, if a user modifies the cooking information, setting the modified information as cooking setting information, and, if the user does not modify the cooking information, setting the cooking information as cooking setting information and transmitting the cooking setting information to the cooking appliance. The program to be executed by the computer, comprising: (Appendix 10) 4. The cooking system of claim 1, further comprising an image analysis unit that outputs the ingredient information from the ingredient image using a first trained model for inferring the ingredient information from the ingredient image. (Appendix 11) a first learning data acquisition unit that acquires first learning data including an ingredient image and ingredient information related to the ingredient image; a first model generation unit that generates the first trained model for inferring the ingredient information from the ingredient image using the first training data; 11. The cooking system of claim 10, further comprising: (Appendix 12) a second learning data acquisition unit that acquires second learning data including ingredient information and cooking information for the ingredient information; a second model generation unit that generates a second trained model for inferring cooking information from ingredient information using the second training data; A learning device comprising: (Appendix 13) a cooking information estimation unit that acquires ingredient information and outputs the cooking information from the ingredient information using a second trained model for inferring cooking information from the ingredient information; An inference device comprising: (Appendix 14) the learning device is communicably connected to a terminal device that transmits the cooking information to a cooking appliance that heats and cooks ingredients in response to a user's operation; The learning device described in Appendix 12, wherein the second model generation unit includes a reward calculation unit that adds a reward the more times the user transmits the cooking information to the cooking appliance, and a function update unit that updates a function for inferring the cooking information based on the reward. (Appendix 15) The learning device is communicably connected to a terminal device that modifies the cooking information in response to a user's operation, The learning device described in Appendix 12, wherein the second model generation unit includes a reward calculation unit that subtracts a reward the more times the user modifies the cooking information, and a function update unit that updates a function for inferring the cooking information based on the reward. (Appendix 16) the learning device is connected to a plurality of the terminal devices, The learning device according to claim 14, wherein the reward calculation unit calculates the reward based on the number of times that multiple users transmit the cooking information to the cooking appliance. [Explanation of symbols]

[0163] 1 Cooking system, 2 Cooker, 3 Terminal device, 4 Router, 5 Wide area network, 6 Control server, 8 main body, 9 container housing portion, 10 container, 10a opening, 10b bottom, 10c scale, 10d photography mark, 11 inner lid, 12 outer lid, 13 heating coil, 14 temperature sensor, 15 compression spring, 16 handle portion, 17 flange portion, 18 lid gasket, 19 inner lid vent, 20 communication flow path, 20a first communication flow path, 20b second communication flow path, 20c third communication flow path, 20d fourth communication flow path, 21 path gasket, 22 steam hole, 23 steam exhaust valve, 24 steam exhaust port, 25 cartridge, 26 cartridge gasket, 27 sensor hole, 28 lid sensor, 29 lid sensor gasket, 30 pressure reducing pump, 31 three-way solenoid valve, 32a first outer lid vent, 32b second outer lid vent, 33 Operation display device, 33a cooker operation unit, 33b cooker display unit, 34 communication unit, 35 control unit, 35a input unit, 35b memory unit, 35c heating control unit, 40 imaging unit, 41 terminal screen, 41a display unit, 41b operation unit, 42 terminal communication unit, 100 Cooking appliance control application, 110 Analysis unit, 111 Image storage unit, 112 Image analysis unit, 113 Food ingredient information storage unit, 120 Estimation unit, 121 Cooking information retention unit, 122 Cooking information estimation unit, 123 Correction information acquisition unit, 130 Screen control unit, 200 Initial selection screen, 201 Ingredient photograph transition button, 202 Ingredient information input transition button, 203 History selection transition button, 210 Photographing screen, 211 Photographing guide, 212 Photographing button, 220 Image analysis failure screen, 221 Re-photograph button, 222 Ingredient information input transition button, 230 Image analysis result screen, 231 Ingredient type text, 232 Ingredient weight text, 233 Correction button, 234 Confirm button, 240 Cooking information display screen, 241 Cooking temperature text, 242 Cooking time text, 243 Water amount text, 244 Correction button, 245 Send button, 250 Ingredient information input screen, 251 Ingredient type text, 252 Ingredient weight text, 253 Confirm button, 260 Cooking history selection screen, 261 Cooking history display area, 262 Confirm button, 270 Shooting method selection screen, 271 Container selection button, 272 Cutting board selection button, 273 Ingredient registration button, 280 Container use shooting screen, 281 Container opening guide, 282 Shooting mark guide, 283 Shooting button, 290 Cutting board use shooting screen, 291 Cutting board frame guide, 292 Shooting button, 500: First learning device; 501: First learning data acquisition unit; 502: First model generation unit; 503: First trained model storage unit; 600: Second learning device, 601: Second learning data acquisition unit, 602: Second model generation unit, 602a: Reward calculation unit, 602b: Function update unit, 603: Second learned model storage unit.

Claims

1. A cooking device that heats and cooks ingredients; a terminal device comprising a photographing unit and an operation unit, connected to the cooking appliance so as to be able to communicate with the cooking appliance, a display unit which acquires food ingredient information from food ingredient images photographed by the photographing unit, and acquires and displays cooking information from the food ingredient information, which is information necessary for cooking the food ingredients in the cooking appliance; and a cooking setting information transmission means which, when a user operates the operation unit to modify the cooking information, sets the modified cooking information as cooking setting information, and, when a user operates the operation unit to not modify the cooking information, sets the cooking information as cooking setting information and transmits the cooking setting information to the cooking appliance; A cooking system comprising:

2. The cooking system according to claim 1 , wherein the ingredient information includes a value indicating the type of the ingredient and a value indicating the weight or size of the ingredient.

3. The cooking system according to claim 1 , wherein the cooking information includes at least one of a cooking time for the ingredients and a cooking temperature for the ingredients.

4. The terminal device When a user photographs the food material with the photographing unit, photographing instruction information instructing the user on at least one of the angle and the distance between the food material and the photographing unit is displayed on the display unit. The cooking system according to any one of claims 1 to 3.

5. The cooking device includes a container for accommodating food to be cooked, When the user photographs the food material contained in the container with the photographing unit, the terminal device displays, on the display unit, photographing instruction information instructing the user on at least one of the angle and distance between the container and the photographing unit. The cooking system according to any one of claims 1 to 3.

6. The cooking system according to claim 5 , wherein the container has a scale, and the scale is positioned so as to appear in a captured image when a photograph is taken in accordance with the photographing instruction information.

7. The cooking system according to any one of claims 1 to 3, wherein if the terminal device is unable to obtain the ingredient information by analyzing the image captured by the photographing unit, the terminal device displays on the display unit a request for the user to photograph the ingredient again.

8. The cooking system according to any one of claims 1 to 3, wherein if the terminal device is unable to obtain the ingredient information by analyzing the image captured by the photographing unit, the terminal device displays on the display unit a request for the user to input the ingredient information.

9. A program installed in a computer to control a cooking device that heats and cooks ingredients, cooking information estimation means for analyzing an image to acquire ingredient information and estimating cooking information, which is information necessary to cook the ingredients in the cooking appliance, from the ingredient information; and cooking setting information transmission means for, if a user modifies the cooking information, setting the modified cooking information as cooking setting information, and, if the user does not modify the cooking information, setting the cooking information as cooking setting information and transmitting the cooking setting information to the cooking appliance. The program to be executed by the computer, comprising:

10. The cooking system described in any one of claims 1 to 3, further comprising an image analysis unit that outputs the ingredient information from the ingredient image using a first trained model for inferring the ingredient information from the ingredient image.

11. a first learning data acquisition unit that acquires first learning data including an ingredient image and ingredient information related to the ingredient image; a first model generation unit that generates the first trained model for inferring the ingredient information from the ingredient image using the first training data; The cooking system of claim 10 further comprising:

12. a second learning data acquisition unit that acquires second learning data including ingredient information and cooking information of the ingredient information; a second model generation unit that generates a second trained model for inferring cooking information from ingredient information using the second training data; A learning device comprising:

13. a cooking information estimation unit that acquires ingredient information and outputs the cooking information from the ingredient information using a second trained model for inferring cooking information from the ingredient information; An inference device comprising:

14. the learning device is communicably connected to a terminal device that transmits the cooking information to a cooking appliance that heats and cooks ingredients in response to a user's operation; The learning device according to claim 12, characterized in that the second model generation unit includes a reward calculation unit that adds a reward the more times the user transmits the cooking information to the cooking appliance, and a function update unit that updates a function for inferring the cooking information based on the reward.

15. The learning device is communicably connected to a terminal device that modifies the cooking information in response to a user's operation, 13. The learning device according to claim 12, wherein the second model generation unit includes: a reward calculation unit that subtracts a reward as the number of times the user modifies the cooking information increases; and a function update unit that updates a function for inferring the cooking information based on the reward.

16. the learning device is connected to a plurality of the terminal devices, 15. The learning device according to claim 14, wherein the reward calculation unit calculates the reward based on the number of times a plurality of users transmit the cooking information to the cooking appliance.

Images