Cooking system

The heating cooker system addresses the challenge of inaccurate cooking level assessment in microwave ovens by using monochrome image analysis and user-specific models to enhance convenience and consistency.

JP7577597B2Active Publication Date: 2024-11-05MIDEA GROUP CO LTD
View PDF 11 Cites 0 Cited by

Patent Information

Application Number
JP2021064993
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-04-06
Publication Date
2024-11-05
Estimated Expiration
2041-04-06

AI Technical Summary

Technical Problem

Existing microwave ovens lack convenience in determining the cooking level of food due to the difficulty in accurately assessing the degree of cooking based on image analysis and the need for frequent user intervention.

Method used

A heating cooker system that includes an imaging unit to capture images of the food, converts them to monochrome for analysis using a learned model, and adjusts heating based on the determined cooking level, considering factors like lens dirt and user preferences.

Benefits of technology

Improves cooking convenience by reducing the need for constant user monitoring and enhancing the accuracy of cooking determination, ensuring consistent quality and user-specific preferences.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007577597000001
    Figure 0007577597000001
  • Figure 0007577597000002
    Figure 0007577597000002
  • Figure 0007577597000003
    Figure 0007577597000003
Patent Text Reader

Abstract

To provide a heating cooking system capable of achieving improvement in convenience.SOLUTION: A heating cooking system includes a housing, a heating part, an imaging part, a determination part, and a control part. The housing includes a placement part on which an object to be cooked can be placed. The heating part is provided in the housing, and heats the object to be cooked. The imaging part images the object to be cooked, and the color of the object to be cooked contained in the image to be imaged is different depending on the temperature of the heating part. The determination part converts the image imaged by the imaging part into a monochromatic image, and determines the cooking conditions of the object to be cooked based on the converted monochromatic image and a learnt model in which an internal variable is adjusted by learning. The control part controls the heating part based on the determination result of the determination part.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] An embodiment of the present invention relates to a cooking device system. [Background technology]

[0002] A microwave oven has been proposed that is equipped with a monitor camera that monitors the food items placed inside the oven and identifies cooking ingredients based on image information of the food items captured by the monitor camera. Meanwhile, further improvements in the convenience of cooking appliances are expected. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 6-257756 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a cooking device system that can improve convenience. [Means for solving the problem]

[0005] The cooking device system of the embodiment has a housing, a heating unit, an imaging unit, a determination unit, and a control unit. The housing includes a placement unit on which an object to be cooked is placed. The heating unit is provided within the housing and heats the object to be cooked. The imaging unit photographs the object to be cooked, and the color of the object to be cooked contained in the photographed image varies depending on the temperature of the heating unit. The determination unit converts the image photographed by the imaging unit into a monochrome image, and determines the degree of cooking of the object to be cooked based on the converted monochrome image and a trained model in which internal variables have been adjusted by learning. The control unit controls the heating unit based on the determination result of the determination unit. The housing has a housing body with an opening and a door that closes the opening. The imaging unit is provided on the ceiling of the housing body or on the inner surface of the door. The determination unit determines the cooking state based on information that directly or indirectly indicates the amount of dirt adhering to the lens of the imaging unit, the monochrome image, and the trained model. The information that directly or indirectly indicates the amount of dirt adhering to the lens of the imaging unit includes a usage history that indicates the types of ingredients that have been cooked in the past as the cooking target. [Brief description of the drawings]

[0006] [Figure 1] 1 is a diagram showing an overall configuration of a cooking device system according to a first embodiment; [Diagram 2] FIG. 1 is a front view showing a cooking device according to a first embodiment. [Diagram 3] 3 is a cross-sectional view taken along line F3-F3 of the cooking device shown in FIG. 2. [Figure 4] FIG. 1 is a block diagram showing a system configuration of a cooking device according to a first embodiment. [Diagram 5] FIG. 4 is a view showing the processing of an image processing unit according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a trained model according to the first embodiment. [Figure 7] 5 is a diagram showing an example of content displayed by an evaluation receiving unit according to the first embodiment. FIG. [Figure 8] 5 is a flowchart showing a flow of processing by a control unit according to the first embodiment. [Figure 9] FIG. 11 is a block diagram showing a system configuration of a cooking device according to a second embodiment. [Figure 10] FIG. 11 is a block diagram showing a system configuration of a cooking device according to a third embodiment. [Figure 11] FIG. 13 is a diagram showing an example of dirt information on a camera lens according to the third embodiment. [Figure 12] FIG. 13 is a block diagram showing a system configuration of a server according to a fourth embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a screen on which a correction is accepted by a correction accepting unit according to the fourth embodiment. [Figure 14] 13 is a flowchart showing a flow of processing by a server according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, the cooking device system of the embodiment will be described with reference to the drawings. In the following description, the same reference numerals are given to configurations having the same or similar functions. Then, duplicated descriptions of those configurations may be omitted. "Based on XX" means "based on at least XX" and may include cases based on other elements in addition to XX. "Based on XX" is not limited to cases where XX is directly used, and may include cases where XX is based on a calculation or processing performed thereon. "XX or YY" is not limited to cases where either XX or YY is used, and may include cases where both XX and YY are used. This also applies to cases where there are three or more selective elements. "XX" and "YY" are any elements (for example, any information).

[0008] (First embodiment) <1. Overall composition> FIG. 1 is a diagram showing the overall configuration of a cooking device system 1 of the first embodiment. The cooking device system 1 includes, for example, a cooking device 100 and a server 200. The Kaetsu cooking device system 1 can communicate with a terminal device 300 of a user U directly or via a network NW. The network NW may be, for example, the Internet, a cellular network, a Wi-Fi (registered trademark) network, a Low Power Wide Area (LPWA), a Wide Area Network (WAN), a Local Area Network (LAN), or other public lines or dedicated lines, depending on the situation.

[0009] The cooking device 100 is a cooking home appliance that heats and cooks a cooking target. The cooking device 100 is placed in the residence of a user U. The cooking device 100 is, for example, a toaster, a toaster oven, or a range oven, but may also be a microwave oven or other cooking home appliances that do not have an oven function. In the following, an example in which the cooking device 100 is a toaster oven will be described.

[0010] The cooking device 100 can be connected to a network NW via a wireless router R installed in the residence of a user U, for example. The cooking device 100 can communicate with a server 200 via the network NW. The cooking device 100 may be able to communicate with a terminal device 300 of a user U directly or via the network NW. However, in this embodiment, the cooking device 100 does not have a communication function and may not be able to communicate with the server 200. In other words, the server 200 may not exist. In this case, an example of a "cooking device system" is configured by the cooking device 100 alone.

[0011] The server 200 is a server that manages the operation of the cooking appliance 100. The server 200 is composed of one or more server devices (e.g., cloud servers) connected to the network NW. The server 200 may include an information processing unit that performs edge computing or fog computing, such as an information processing unit included in a router in the network NW. The server 200 is capable of communicating with a terminal device 300 of a user U via the network NW.

[0012] The terminal device 300 of the user U is, for example, a mobile terminal device such as a smartphone or a tablet terminal device. The terminal device 300 includes a display device 301 including a display screen capable of displaying various information, and an operation unit 302 capable of receiving an operation from the user U. The display device 301 is an example of a "display unit." The operation unit 302 is, for example, a touch sensor superimposed on the display screen of the display device 301. An application program for using the cooking device system 1 is installed in the terminal device 300, and the functions described below are supported. Note that the terminal device 300 is not limited to a mobile terminal device, and may be a personal computer.

[0013] <2.Heating cooker> <2.1 Overall configuration of cooking appliance> Next, the cooking device 100 will be described. Fig. 2 is a front view showing the cooking device 100. Fig. 3 is a cross-sectional view taken along line F3-F3 of the cooking device 100 shown in Fig. 2. As shown in Fig. 2 and Fig. 3, the cooking device 100 has, for example, a device main body (cooking device main body) 101, a communication unit 102, and a control unit 103.

[0014] The device body 101 has, for example, a housing 111, a placement section 112, a heating section 113, a display device 114, an operation section 115, a camera 116, a temperature sensor 117, and a humidity sensor 118.

[0015] The housing 111 has, for example, a housing main body 121 and a door 122. The housing main body 121 is formed in a box shape with a part of the front surface open. That is, the front surface of the housing main body 121 has an opening 121a that exposes the inside of the housing main body 121 to the outside. The door 122 is provided in front of the housing main body 121. The door 122 is rotatably connected to the housing main body 121, and closes the opening 121a in an openable and closable manner.

[0016] The door 122 has a door body 131, a window portion 132, and a handle 133. The door body 131 is formed in a plate shape that covers the opening 121a of the housing body 121. The window portion 132 is provided so as to penetrate the door body 131. The window portion 132 is formed of a transparent glass material or a plastic material. The user U can see the inside of the housing body 121 from the outside through the window portion 132. The handle 133 is provided on the front surface of the door 122 and protrudes forward compared to the front surface of the window portion 132. The handle 133 is a part that the user U grasps when opening and closing the door 122 with respect to the housing body 121. The handle 133 has a handle body 133a provided away from the door body 131, and a connection portion 133b provided at both ends of the handle body 133a and connected to the door body 131.

[0017] The placement portion 112 is provided inside the housing 111. The placement portion 112 is formed, for example, in a plate or net shape, and the cooking object O is placed on the placement portion 112.

[0018] The heating unit 113 is provided inside the housing 111 and heats the cooking object O placed on the placement unit 112. The heating unit 113 includes, for example, a first heater 113a and a second heater 113b. The first heater 113a is located above the cooking object O and heats the upper surface of the cooking object O. The second heater 113b is located below the cooking object O and heats the lower surface of the cooking object. Here, the heating unit 113 emits different colors depending on the heat generation state of the heating unit 113. For example, the heating unit 113 emits yellow or orange light when in a first temperature state, and emits a stronger orange color (a darker orange color) when in a second temperature state higher than the first temperature state. Therefore, the color of the surface of the cooking object O illuminated by the heating unit 113 differs depending on the temperature state of the heating unit 113.

[0019] The display device 114 is provided, for example, on the front surface of the housing main body 121. The display device 114 has a display screen capable of displaying various information. The display device 114 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display, but is not limited to these. The display device 114 is an example of a "display unit."

[0020] The operation unit 115 is provided, for example, on the surface of the housing body 121. The operation unit 115 can accept an operation by the user U. The operation unit 115 is, for example, a touch sensor provided on top of the display screen of the display device 114. However, the operation unit 115 may be a physical button provided on the surface of the housing body 121. The user U can input the type of cooking object O through an operation on the operation unit 115. The "type of cooking object" includes one or more of the following: type of ingredients (for example, types of bread such as white bread, muffin, wheat bread, etc.), number of cooking objects (1 slice / 2 slices / 3 slices of bread, etc.), size or thickness of the cooking object (6 slices of bread / 8 slices of bread, etc.), etc.

[0021] The camera 116 is provided in the housing 111 and captures the object O placed on the placement unit 112. The camera 116 captures an image of the surface (e.g., the top surface) of the object O. In this application, the term "image" is not limited to a still image, but also includes a video (video). The image captured by the camera 116 includes a content showing the degree of doneness of the object O. For example, the camera 116 captures a color image of the surface of the object O. In this embodiment, the color of the object O included in the image captured by the camera 116 differs depending on the temperature state of the heating unit 113. That is, when the heating unit 113 is in the first temperature state, the surface of the object O is illuminated in yellow or orange by the heating unit 113. On the other hand, when the heating unit 113 is in the second temperature state, the surface of the object O is illuminated in a stronger orange color by the heating unit 113. Camera 116 includes camera body 116a including an imaging element, and lens 116b positioned between camera body 116a and a subject (object O to be cooked). Camera 116 is an example of an "imaging section."

[0022] The camera 116 is provided, for example, on the ceiling surface 121b of the housing body 121, and is positioned above the cooking object O. The camera 116 is disposed in a location where the cooking object O can be photographed at an angle that allows the thickness of the cooking object O to be known in addition to the size of the cooking object O (for example, the size when viewed from above). The camera 116 may be provided on the inner surface of the door body 131 instead of the ceiling surface 121b of the housing body 121. The camera 116 may also be provided on the rear surface of the handle body 133a of the door 122, outside the housing body 121. In this case, the camera 116 photographs an image of the surface of the cooking object O accommodated inside the housing body 121 through the window portion 132 of the door 122.

[0023] Temperature sensor 117 is provided inside housing 111 and detects the temperature inside housing 111 . Humidity sensor (water vapor sensor) 117 is provided in housing 111 and detects the humidity inside housing 111. Cooking device 100 has, for example, a water vapor generating unit (not shown) that generates water vapor by heating water supplied to a water supply unit and supplies moisture to cooking object O during cooking. Humidity sensor 118 is provided, for example, in cooking device 100 having a water vapor generating unit and detects the moisture supplied to cooking object O.

[0024] In this embodiment, the camera 116, the temperature sensor 117, and the humidity sensor 118 constitute an example of the sensor unit SU. In the following description, the image captured by the camera 116, the detection result of the temperature sensor 117, and the detection result of the humidity sensor 118 may be collectively referred to as the "detection result of the sensor unit SU." Note that the sensor unit SU may include sensors other than those described above. Also, one or both of the temperature sensor 117 and the humidity sensor 118 may be omitted.

[0025] <2.2 System configuration of cooking appliance> FIG. 4 is a block diagram showing a system configuration of the cooking device 100. As shown in FIG. The communication unit 102 is, for example, a wireless communication module including an antenna and a high-frequency circuit. The communication unit 102 can communicate with the server 200 via the network NW. The communication unit 102 can communicate with the terminal device 300 via the network NW or directly using short-range wireless communication (for example, Bluetooth (registered trademark)).

[0026] The control unit 103 includes an information processing unit PU1 and a storage unit 157. The information processing unit PU1 includes a type receiving unit 151, an image processing unit 152, a cooking state determining unit 153, a control unit 154, an evaluation receiving unit 155, and a learning unit 156. Each functional unit included in the information processing unit PU1 is realized by a hardware processor such as a CPU (Central Processing Unit) mounted on the control unit 103 executing a program. However, some or all of these functional units may be realized by hardware (including circuitry) such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array), or may be realized by cooperation between software and hardware. The storage unit 157 is realized by a RAM (Random Access Memory), a ROM (Read Only Memory), a HDD (Hard Disk Drive), a flash memory, or a combination of a plurality of these. A plurality of learned models MA and threshold information TI are stored in the storage unit 157.

[0027] The type receiving unit 151 receives an input from the user U indicating the type of the object to be cooked O based on an operation performed by the user U on the operation unit 115 of the cooking appliance 100 or the operation unit 302 of the terminal device 300. For example, the type receiving unit 151 displays multiple candidates for the type of the object to be cooked O on the display device 114 of the cooking appliance 100 or the display device 301 of the terminal device 300, and receives an input from the user U to select one from the multiple candidates. As described above, the "type of object to be cooked" includes one or more of the type of ingredients, the number of objects to be cooked, the size or thickness of the object to be cooked, and the like. In this embodiment, the type receiving unit 151 constitutes an example of the input receiving unit IU.

[0028] Image processing unit 152 performs image processing on the image captured by camera 116. For example, image processing unit 152 performs image processing to convert a color image captured by camera 116 into a monochrome image. In this application, a "monochrome image" may be a black-and-white image (an image in which one pixel is expressed in two gradations (1 bit) of black and white) or a grayscale image (an image including intermediate gradations). In this embodiment, image processing unit 152 converts a color image captured by camera 116 into a black-and-white image. However, image processing unit 152 may also convert a color image captured by camera 116 into a grayscale image.

[0029] Fig. 5 is a diagram showing the processing of the image processing unit 152. (a) and (b) in Fig. 5 show cases where the temperature states of the heating unit 113 are different. The "before conversion" images IM shown in (a) and (b) in Fig. 5 are color images captured by the camera 116 and are images before processing by the image processing unit 152. As described above, the colors of the cooking object O included in the two images IM differ depending on the temperature state of the heating unit 113.

[0030] On the other hand, the "converted" images IMA shown in (a) and (b) in Fig. 5 are monochrome images that have been processed by the image processor 152. The colors of the objects O contained in the two images IM are almost the same. In other words, by converting the images IM into monochrome images, the influence of the colors of the objects O contained in the images IM can be reduced.

[0031] The cooking degree determination unit 153 determines the cooking degree of the cooking object O based on the monochrome image converted by the image processing unit 152 and the trained model MA in which internal variables have been adjusted by learning. The trained model MA is a model including a neural network, and is trained by, for example, deep learning.

[0032] 6 is a diagram showing an example of the trained model MA. The trained model MA is composed of a neural network including an input layer, a hidden layer (intermediate layer), and an output layer. For example, the trained model MA is trained by adjusting weighting coefficients w (internal variables) between nodes n included in the input layer, hidden layer, and output layer by deep learning using teacher data obtained from a previously conducted experiment.

[0033] The trained model MA is trained to output, as output information, a judgment result indicating the degree of cooking of the cooking object O when the detection results of the sensor unit SU (for example, the image from the camera 116, the detection results from the temperature sensor 117, and the detection results from the humidity sensor 118) are input as input information. The "judgment result indicating the degree of cooking" is expressed as a numerical value between 0 and 1, where the state at the start of cooking is "0" and an overcooked state (burnt state) is "1", for example.

[0034] Here, the threshold information TI stored in the memory unit 157 includes a threshold value (e.g., "0.7") corresponding to just-right doneness (e.g., just-right browning). In this embodiment, the cooking degree judgment unit 153 inputs the latest detection result of the sensor unit SU as input information to the trained model MA at a predetermined cycle (e.g., every 5 seconds). Then, the cooking degree judgment unit 153 judges that the cooking degree of the cooking object O is just-right when the numerical value as the judgment result output from the trained model MA exceeds the threshold a predetermined number of times (e.g., exceeds the threshold five times in a row). In this embodiment, the image processing unit 152 and the cooking degree judgment unit 153 constitute an example of a judgment unit DU. The cooking degree judgment unit 153 is an example of a "first judgment unit".

[0035] In this embodiment, the memory unit 157 stores multiple trained models MA corresponding to multiple types of cooking object O. The cooking degree determination unit 153 selects one trained model MA from the multiple trained models MA based on the type of cooking object O input by the user U and accepted by the type acceptance unit 151. Then, the cooking degree determination unit 153 judges the cooking degree of the cooking object O based on the selected trained model MA.

[0036] The control unit 154 controls the heating unit 113 based on the determination result of the cooking degree determination unit 153. For example, the control unit 154 starts the operation of the cooking device 100 and heats the heating unit 113 based on the operation of the user U on the operation unit 115 of the cooking device 100 or the operation unit 302 of the terminal device 300. The control unit 154 controls the heating state of the heating unit 113, for example, by controlling ON / OFF of a switching element that supplies power to the heating unit 113. Then, when the cooking degree determination unit 153 determines that the cooking degree of the cooking target O is just right, the control unit 154 ends the heating of the heating unit 113 and ends the operation of the cooking device 100. Note that the control unit 154 may adjust the temperature of the heating unit 113 based on the determination result of the cooking degree determination unit 153 instead of / in addition to ending the operation of the cooking device 100.

[0037] After the operation of the cooking appliance 100 is completed, the evaluation receiving unit 155 receives an evaluation by the user U regarding the cooking state of the cooking object O. For example, the evaluation receiving unit 155 displays questions regarding the cooking state in a questionnaire format on the display device 114 of the cooking appliance 100 or the display device 301 of the terminal device 300, and receives the answers to the questions as the user U's evaluation regarding the cooking state.

[0038] 7 is a diagram showing an example of the content displayed by the evaluation receiving unit 155. The evaluation receiving unit 155 asks the user U to, for example, rate the doneness on a five-point scale or rate the moistness on a five-point scale, and receives the user U's selection of the five-point scale in response to the question as the user U's evaluation.

[0039] The learning unit 156 performs additional learning of the trained model MA based on the evaluation of the user U received by the evaluation receiving unit 155. For example, the learning unit 156 performs reinforcement learning to maximize the evaluation of the user U, using the operation details (a combination of the detection result of the sensor unit SU and the judgment details of the cooking degree judgment unit 153) when the user U evaluated the cooking degree as good as new teacher data. As a result, the internal variables of the trained model MA are adjusted to match the preferences of the user U.

[0040] <3. Processing flow> 8 is a flowchart showing the flow of processing by the control unit 103. First, the type receiving unit 151 receives the type of cooking object O based on an operation of the user U on the operation unit 115 of the cooking appliance 100 (or the operation unit 302 of the terminal device 300) (S101). Then, the cooking degree determination unit 153 selects a trained model MA to be used from among a plurality of trained models MA based on the type of cooking object O received by the type receiving unit 151 (S102).

[0041] Next, the control unit 154 starts the operation of the cooking appliance 100 (S103). That is, the control unit 154 starts cooking the cooking object O by heating the heating unit 113. Then, the determination unit DU acquires the detection result of the sensor unit SU at a predetermined period (S104).

[0042] When the detection result of the sensor unit SU is input, the image processing unit 152 converts the color image captured by the camera 116 included in the detection result of the sensor unit SU into a monochrome image (S105). Then, the cooking degree determination unit 153 determines the cooking degree of the cooking object O using the monochrome image converted by the image processing unit 152 and the selected trained model MA, and outputs the determination result (S106).

[0043] Next, the control unit 154 determines whether or not the determination result output by the cooking degree determination unit 153 satisfies a predetermined end condition (S107). For example, the control unit 154 determines that the predetermined end condition is satisfied when the numerical value included in the determination result output by the cooking degree determination unit 153 exceeds a predetermined threshold value a predetermined number of times.

[0044] If the predetermined end condition is not satisfied (S107: NO), the control unit 154 returns to S104 and repeats the processes from S104 to S106. On the other hand, if the predetermined end condition is satisfied (S107: YES), the control unit 154 ends the operation of the cooking device 100 (S108). That is, the control unit 154 ends the heating of the heating unit 113. This completes the cooking of the cooking object O.

[0045] <4. Advantages> As a comparative example, consider a cooking device in which the degree of cooking is adjusted by a timer. In such a cooking device, the user may need to check the degree of cooking each time, or the degree of cooking may vary each time. In addition, when the degree of cooking is judged by directly using an image captured by a camera, if the color of the surface of the object to be cooked varies depending on the temperature state of the heating section, it may be difficult to judge the degree of cooking with high accuracy.

[0046] On the other hand, the cooking device system 1 of the present embodiment includes a determination unit DU that converts an image captured by the camera 116 into a monochrome image, and determines the degree of cooking of the cooking object O based on the converted monochrome image and the learned model MA, and a control unit 154 that controls the heating unit 113 based on the determination result of the determination unit DU. According to this configuration, the degree of cooking is first determined by the determination unit DU, and the heating unit 113 is controlled based on the determination result, so that the user U does not need to check the degree of cooking each time or the need to do so is reduced, and the difference in the degree of cooking each time is reduced. Furthermore, when the color of the surface of the cooking object O varies depending on the temperature state of the heating unit 113, the degree of cooking can be determined based on the monochrome image, thereby improving the accuracy of the determination of the degree of cooking. This can improve the convenience of the user U.

[0047] In this embodiment, the cooking device system 1 further includes a type receiving unit 151 that receives a user's input indicating the type of the cooking object O. The determination unit DU selects one trained model MA from among multiple trained models MA based on the type of the cooking object O received by the type receiving unit 151, and determines the cooking state of the cooking object O based on the selected trained model MA. With this configuration, the accuracy of determining the cooking state can be further improved by the trained model MA prepared for each cooking object. This can improve the convenience for the user U.

[0048] In this embodiment, the cooking device system 1 further includes an evaluation receiving unit 155 that receives an evaluation by the user U regarding the cooking state after the operation is completed, and a learning unit 156 that performs additional learning of the trained model MA based on the evaluation by the user U received by the evaluation receiving unit 155. With this configuration, the trained model MA can be updated so that the evaluation by the user U is higher. This can improve the convenience for the user U.

[0049] (Modification) In the above-described first embodiment, the type reception unit 151, the image processing unit 152, the cooking degree determination unit 153, the control unit 154, the evaluation reception unit 155, the learning unit 156, and the memory unit 157 are provided in the cooking appliance 100. Alternatively, one or more of the type reception unit 151, the image processing unit 152, the cooking degree determination unit 153, the control unit 154, the evaluation reception unit 155, the learning unit 156, and the memory unit 157 may be provided in the server 200. Furthermore, the evaluation reception unit 155 and the learning unit 156 may be omitted.

[0050] Second embodiment Next, a second embodiment will be described. The second embodiment is different from the first embodiment in that the cooking device system 1 has a preference receiving unit 161 that receives the preference of the user U regarding the degree of cooking. Note that the configuration other than that described below is the same as the configuration of the first embodiment.

[0051] FIG. 9 is a block diagram showing a system configuration of the cooking device 100A of the second embodiment. In this embodiment, the input reception unit IU includes a preference reception unit 161. The preference reception unit 161 receives the preference of the user U regarding the degree of cooking of the cooking object O based on the operation of the user U performed on the operation unit 115 of the cooking device 100 or the operation unit 302 of the terminal device 300. For example, the preference reception unit 161 displays multiple candidates of the degree of cooking on the display device 114 of the cooking device 100 or the display device 301 of the terminal device 300, and receives an input of the user U selecting one from the multiple candidates. The preferences of the user U regarding the degree of cooking are, for example, "crispy texture", "moist texture", "likes well-cooked", "doesn't like overcooked", etc.

[0052] In this embodiment, the storage unit 157 stores a plurality of threshold information TI corresponding to a plurality of preferences for cooking state. The plurality of threshold information TI have different thresholds (thresholds for terminating the operation of the cooking device 100A) for determining that the cooking state is just right. The cooking state determination unit 153 selects one of the plurality of threshold information TI according to the preference of the user U accepted by the preference acceptance unit 161, and determines the timing for terminating the operation of the cooking device 100A using the selected threshold information TI. In other words, the cooking state determination unit 153 varies the determination result regarding the cooking state of the cooking target (for example, the determination result indicating the timing for terminating the operation) based on the preference of the user U accepted by the preference acceptance unit 161.

[0053] According to such a configuration, the cooking target O can be cooked to a degree of doneness more suitable for the preference of the user U. This can improve the convenience of the user U. For example, according to the configuration of this embodiment, even if the evaluation receiving unit 155 and the learning unit 156 are omitted, the cooking target O can be cooked to a degree of doneness more suitable for the preference of the user U.

[0054] Third embodiment Next, a third embodiment will be described. The third embodiment differs from the first embodiment in that the cooking doneness is determined by taking into account the amount of dirt on the lens 116b of the camera 116. Note that the configuration other than that described below is the same as that of the first embodiment.

[0055] 10 is a block diagram showing a system configuration of a cooking device 100B of a third embodiment. The cooking device 100B is a home appliance that is cleaned less frequently than other home appliances such as a refrigerator, and dirt (e.g., oily dirt) may accumulate on the lens 116b of the camera 116. When dirt accumulates on the lens 116b, the image captured by the camera 116 may become discolored (e.g., brown). When the image captured by the camera 116 becomes discolored, the accuracy of determining the degree of doneness may decrease.

[0056] In this embodiment, information processing unit PU1 includes a state management unit 171. State management unit 171 manages information (hereinafter referred to as "dirt information") that directly or indirectly indicates the amount of dirt adhering to lens 116b of camera 116. For example, state management unit 171 collects dirt information on lens 116b of camera 116 at a predetermined cycle (for example, once a week) and stores it as part of history information HI stored in storage unit 157.

[0057] FIG. 11 is a diagram showing an example of dirt information on lens 116b of camera 116. The dirt information on lens 116b of camera 116 includes, for example, at least one of dirt information HIA that directly indicates the amount of dirt adhering to lens 116b of camera 116 and dirt information HIB that indirectly indicates the amount of dirt adhering to lens 116b of camera 116. The dirt information HIA includes, for example, a history of images captured by camera 116. For example, dirt information HIA includes image 1 captured at the beginning of use of cooking device 100B and image N captured most recently. The dirt information HIB includes a usage history of cooking device 100B. The usage history includes, for example, one or more pieces of information such as the type of food cooked, the heating temperature of heating unit 113, and the heating time of heating unit 113.

[0058] In this embodiment, the trained model MA is trained to output a determination result indicating the degree of cooking of the cooking object O when the detection result of the sensor unit SU and dirt information of the lens 116b of the camera 116 are input as input information. The dirt information of the lens 116b of the camera 116 input as input information may be, for example, the most recently captured image N (e.g., image N of a color image), a pair of image 1 and image N, or information indicating the difference between image 1 and image N. Alternatively / in addition to this, the dirt information of the lens 116b of the camera 116 may be the history of cooked ingredients, the integrated value of the heating temperature of the heating unit 113, or the accumulated value of the heating time of the heating unit 113. In this embodiment, the cooking degree determination unit 153 determines the degree of cooking based on the detection result of the sensor unit SU and the dirt information of the lens 116b of the camera 116.

[0059] According to such a configuration, it is possible to determine the doneness of the cooking object O while taking into consideration dirt adhering to the lens 116b of the camera 116. This makes it possible to improve the accuracy of the determination of the doneness.

[0060] (Modification) If camera 116 is provided on handle 133 of door 122 instead of inside housing 111, the dirt problem will be dirt on window 132 of door 122, not dirt on lens 116b of camera 116. For this reason, if camera 116 is provided on handle 133 of door 122, state management unit 171 manages dirt information that directly or indirectly indicates the amount of dirt adhering to window 132. For details of the dirt information in this modification, it is sufficient to read "camera lens" as "window" in the dirt information described in the third embodiment. For example, state management unit 171 in this modification manages information (dirt information HIA, HIB) that directly or indirectly indicates the amount of dirt adhering to window 132.

[0061] The dirt information HIA includes, for example, a history of images of the window portion 132 taken by the camera 116 or another camera. For example, the dirt information HIA includes an image 1 of the window portion 132 taken at the beginning of use of the cooking device 100B and an image N of the window portion 132 taken most recently. The dirt information HIB includes a usage history of the cooking device 100B. The usage history includes, for example, one or more pieces of information such as the type of food cooked, the heating temperature of the heating section 113, and the heating time of the heating section 113. This configuration can also improve the accuracy of determining the degree of cooking.

[0062] (Fourth embodiment) Next, a fourth embodiment will be described. The fourth embodiment differs from the first embodiment in that the type of the cooking object O is determined by the cooking device system 1. Note that the configuration other than that described below is the same as that of the first embodiment. For the sake of distinction, in this embodiment, the trained model MA described above is referred to as the "first trained model MA."

[0063] FIG. 12 is a block diagram showing a system configuration of the server 200C of the fourth embodiment. The server 200C has, for example, an information processing unit PU2 and a storage unit 206. The information processing unit PU2 includes, for example, an information acquisition unit 201, a type determination unit 202, an information output unit 203, a correction acceptance unit 204, and a learning unit 205. Each functional unit included in the information processing unit PU2 is realized by a hardware processor such as a CPU mounted on the server 200C executing a program. However, some or all of these functional units may be realized by hardware (including a circuit unit; circuitry) such as an ASIC, PLD, or FPGA, or may be realized by cooperation between software and hardware. The storage unit 206 is realized by a RAM, a ROM, an HDD, a flash memory, or a combination of a plurality of these. The second trained model MB is stored in the storage unit 206.

[0064] The information acquiring unit 201 acquires a color image captured by the camera 116 of the cooking appliance 100 (or a monochrome image converted by the image processing unit 152) from the cooking appliance 100. Hereinafter, the color image captured by the camera 116 and the monochrome image converted by the image processing unit 152 are collectively referred to as "images acquired by the information acquiring unit."

[0065] The type determination unit 202 determines the type of the cooking object O based on the image (e.g., a color image) acquired by the information acquisition unit 201 and the second trained model MB. The second trained model MB is a model including a neural network, and is trained, for example, by deep learning. The type determination unit 202 is an example of a "second determination unit." In this embodiment, an example of a determination unit DU is configured by the image processing unit 152 and the cooking degree determination unit 153 provided in the cooking appliance 100, and the type determination unit 202 provided in the server 200C.

[0066] For example, the second trained model MB is configured by a neural network including an input layer, a hidden layer (intermediate layer), and an output layer, similar to the first trained model MA shown in Fig. 6. For example, the trained model MB is trained by adjusting weighting coefficients w (internal variables) between nodes n included in the input layer, hidden layer, and output layer by deep learning using teaching data obtained from experiments conducted in advance.

[0067] The second trained model MB has been trained to output, as output information, a determination result indicating the type of the cooking object O when an image acquired by the information acquisition unit 201 is input as input information. As described above, the type of cooking object O includes one or more of the type of ingredients, the number of cooking objects, the size or thickness of the cooking object, etc. The output information output from the second trained model MB includes information indicating the likelihood of the determination result of the type of cooking object O (e.g., including information indicating the probability that the determination result is correct).

[0068] The information output unit 203 outputs information indicating the type of cooking object O determined by the type determination unit 202 to the heating cooker 100 or the terminal device 300, and displays the information indicating the type of cooking object O on the display device 114 of the heating cooker 100 or the display device 301 of the terminal device 300.

[0069] The correction accepting unit 204 accepts a correction to the determination result of the type of the cooking object O determined by the type determining unit 202 and output by the information output unit 203. For example, the correction accepting unit 204 accepts a correction to the determination result of the type of the cooking object O based on an operation by the user U on the operation unit 115 of the cooking appliance 100 or the operation unit 302 of the terminal device 300.

[0070] 13 is a diagram showing an example of a screen on which the correction receiving unit 204 receives a correction. In this embodiment, the determination result of the type of the cooking object O by the type determination unit 202 is displayed on the display device 114 of the cooking appliance 100 or the display device 301 of the terminal device 300. Then, the correction receiving unit 204 first receives whether or not there is a correction to the determination result of the type of the cooking object O. Then, if there is a correction, the correction receiving unit 204 receives the specific content of the correction.

[0071] The learning unit 205 performs additional learning of the second trained model MB based on the correction content of the user U accepted by the correction accepting unit 204. For example, the learning unit 205 performs additional learning of the second trained model MB using a combination of the image acquired by the information acquiring unit 201 and the correct type of the cooking object O corrected by the user U as new teacher data. This can improve the determination accuracy of the type determining unit 202.

[0072] In this embodiment, the control unit 154 of the cooking device 100 may adjust the temperature of the heating unit 113 based on the determination result of the type determination unit 202 instead of / in addition to ending the operation of the cooking device 100.

[0073] 14 is a flowchart showing the flow of processing by the server 200C. First, the type determination unit 202 determines the type of the cooking object O based on the image acquired by the information acquisition unit 201 and the second trained model MB (S201). Next, the type determination unit 202 determines whether the determination result is correct (whether the probability that the determination result is correct is equal to or greater than a threshold) (S202).

[0074] If the determination result is not incorrect (if the probability that the determination result is correct is equal to or greater than the threshold, S202: YES), the information output unit 203 outputs the determination result of the type of the cooking object O to the cooking device 100. In this case, the cooking device 100 automatically selects (automatically sets) the operating conditions of the cooking device 100 (such as the heating temperature and heating time of the heating unit 113) based on the determination result of the type of the cooking object O, and starts the operation of the cooking device 100 (S206). Note that the information output unit 203 may display the determination result indicating the type of the cooking object O on the display device 114 of the cooking device 100 or the display device 301 of the terminal device 300, even if the determination result is not incorrect (if the probability that the determination result is correct is equal to or greater than the threshold).

[0075] On the other hand, if there is a possibility that the determination result is incorrect (if the probability that the determination result is correct is less than the threshold, S202: NO), the information output unit 203 displays the determination result of the type of cooking object O on the display device 114 of the cooking appliance 100 or the display device 301 of the terminal device 300 (S203). Next, the correction accepting unit 204 accepts a correction to the determination result of the type of cooking object O (S204).

[0076] When the information output unit 203 receives an input from the user U indicating that there is no correction (S204: NO), it outputs the determination result of the type of the cooking object O to the cooking appliance 100. Then, the process proceeds to S206. On the other hand, when the correction receiving unit 204 receives an input from the user U to correct the type of the cooking object O (S204: YES), it corrects the type of the cooking object O (S205). Then, the information output unit 203 outputs the corrected type of the cooking object O to the cooking appliance 100. Then, the process proceeds to S206. The subsequent processes are similar to those in the first embodiment.

[0077] According to this configuration, similarly to the first embodiment, it is possible to improve the convenience for the user U. Furthermore, in this embodiment, the type determination unit 202 automatically determines the type of the cooking object O. This makes it possible to further improve the convenience for the user U.

[0078] In this embodiment, the device further includes an information output unit 203 that outputs the type of the cooking object O determined by the type determination unit 202 to the display device 114, 301, and a correction receiving unit 204 that can receive an input from the user U to correct the type of the cooking object O. With this configuration, when the type of the cooking object O is incorrectly determined, cooking can be performed based on the correct content corrected by the user U. This can further improve convenience for the user U.

[0079] In this embodiment, the cooking degree determination unit 153 is provided in the cooking device 100. The type determination unit 202 is provided in the server 200C. This allows the cooking device 100 to determine the cooking degree that requires a quick determination, and the server 200C to determine the type of cooking object O that has more time than the cooking degree. This allows the required capacity of the memory unit 157 of the cooking device 100 to be reduced, resulting in stable processing and reduced costs for the cooking device 100.

[0080] In this embodiment, the cooking degree determination unit 153 determines the cooking degree based on the monochrome image. The type determination unit 202 determines the type of the cooking object O based on the color image. With this configuration, when the cooking object O changes color depending on the temperature of the heating unit 113, the influence of the color difference of the cooking object O illuminated by the heating unit 113 is reduced, and the type of the cooking object O can be determined with high accuracy based on the color image.

[0081] (Modification) In the above-described fourth embodiment, the information acquisition unit 201, the type determination unit 202, the information output unit 203, the correction acceptance unit 204, the learning unit 205, and the storage unit 206 are provided in the server 200C. Alternatively, one or more of the information acquisition unit 201, the type determination unit 202, the information output unit 203, the correction acceptance unit 204, the learning unit 205, and the storage unit 206 may be provided in the cooking appliance 100.

[0082] Some embodiments and modifications have been described above. However, the embodiments and modifications are not limited to the above examples. For example, the above-mentioned embodiments and modifications may be realized in combination with each other. The "server" in this application is not limited to a cloud server, and may be a computer installed in the user U's home.

[0083] According to at least one of the embodiments described above, the cooking device system includes a determination unit that converts an image captured by the imaging unit into a monochrome image and determines the doneness of an object to be cooked based on the converted monochrome image and a trained model in which internal variables are adjusted by training. With this configuration, it is possible to improve convenience.

[0084] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0085] 1...heating cooker system, 100, 100A, 100B...heating cooker, 111...housing, 112...placement section, 113...heating section, 114...display device (display section), 116...camera (imaging section), 151...type reception section, 152...image processing section, 153...cooking condition judgment section (first judgment section), 154...control section, 155...evaluation reception section, 156...learning section, 161...preference reception section, 200, 200C...server, 202...type judgment section (second judgment section), 204...correction reception section, 205...learning section, 300...terminal device, 301...display device (display section), DU...judgment section, MA...first learned model, MB...second learned model, O...cooking object.

Claims

1. A housing including a placement section on which an object to be cooked is placed; A heating unit provided in the housing for heating the object to be cooked; an imaging unit that captures an image of the cooking target, the color of the cooking target included in the captured image being different depending on the temperature of the heating unit; A determination unit that converts the image captured by the imaging unit into a monochrome image and determines the degree of doneness of the object to be cooked based on the converted monochrome image and a trained model in which internal variables have been adjusted by training; A control unit that controls the heating unit based on a determination result of the determination unit; Equipped with The housing has a housing body having an opening and a door that closes the opening, the imaging unit is provided on a ceiling of the housing body or on an inner surface of the door, The determination unit determines the cooking state based on information directly or indirectly indicating an amount of dirt attached to the lens of the imaging unit, the monochrome image, and the trained model, The information directly or indirectly indicating the amount of dirt adhering to the lens of the imaging unit includes a usage history indicating the types of ingredients previously cooked as the cooking target. Heating cooker system.

2. A housing including a placement section on which an object to be cooked is placed; A heating unit provided in the housing for heating the object to be cooked; an imaging unit that captures an image of the cooking target, the color of the cooking target included in the captured image being different depending on the temperature of the heating unit; A determination unit that converts the image captured by the imaging unit into a monochrome image and determines the degree of doneness of the object to be cooked based on the converted monochrome image and a trained model in which internal variables have been adjusted by training; A control unit that controls the heating unit based on a determination result of the determination unit; Equipped with The housing has a housing body having an opening, and a door that closes the opening and includes a window portion and a handle that allow the inside of the housing body to be viewed from the outside, The imaging unit is provided on the handle and captures an image of the cooking target contained inside the housing body through the window, The determination unit determines the cooking state based on information directly or indirectly indicating an amount of dirt attached to the window portion, the monochrome image, and the trained model, The information directly or indirectly indicating the amount of dirt adhering to the window portion includes a usage history indicating the types of ingredients previously cooked as the cooking target. Heating cooker system.

3. A type receiving unit is further provided for receiving a user's input indicating a type of the object to be cooked, the trained model includes a plurality of trained models corresponding to the type of the object to be cooked, and the determination unit selects one trained model from the plurality of trained models based on the type of the object to be cooked accepted by the type acceptance unit, and determines the cooking doneness of the object to be cooked based on the selected trained model. The cooking device system according to claim 1 or 2.

4. The trained model includes a first trained model used to determine the degree of cooking and a second trained model used to determine the type of the object to be cooked, 3. The heating cooker system according to claim 1 or 2, wherein the determination unit includes: a first determination unit that determines the cooking state based on the monochrome image and the first trained model; and a second determination unit that determines the type of the object to be cooked based on an image captured by the imaging unit or the monochrome image and the second trained model.

5. An information output unit that outputs the type of the cooking object determined by the second determination unit to a display unit; a correction receiving unit capable of receiving a user's input for correcting the type of the cooking object displayed on the display unit when the type of the cooking object is incorrect; Further equipped with The cooking system according to claim 4 .

6. A cooking device including the housing, the heating unit, the imaging unit, and the control unit; A server connectable to the cooking device via a network; Equipped with The first determination unit is provided in the cooking appliance, The second determination unit is provided in the server. The cooking device system according to claim 4 or 5.

7. The first determination unit determines the cooking state based on the monochrome image and the first trained model, The second determination unit determines the type of the cooking object based on the color image captured by the imaging unit and the second trained model. The cooking device system according to any one of claims 4 to 6.

8. The image captured by the imaging unit includes content indicating the degree of doneness of the object to be cooked. The cooking device system according to any one of claims 1 to 7.

9. an evaluation receiving unit that receives a user's evaluation of the cooking state after the operation is completed; A learning unit that performs additional learning of the trained model based on the user's evaluation accepted by the evaluation acceptance unit; Further comprising: The cooking device system according to any one of claims 1 to 8.

10. A preference receiving unit is further provided for receiving a user's preference regarding the degree of cooking, The determination unit changes a determination result regarding the doneness of the object to be cooked based on the user's preference accepted by the preference accepting unit. The cooking device system according to any one of claims 1 to 9.

11. a cooking device including the housing, the heating unit, the imaging unit, and the control unit; The cooking appliance is a toaster or a toaster oven. The cooking device system according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Heat-cooking device

    JP1984038529A

  • High frequency heating device

    JP1992269489A

  • Cooking implement

    JP1993172337A

  • Radar range

    JP1994257756A

  • Heating cooker

    JP2005172267A