Information processing device, information processing method, and program

Abstract

This invention relates to an information processing device, information processing method, and program that make it possible to appropriately estimate the internal temperature of food being heated. This information processing device comprises a construction unit for constructing a three-dimensional model representing the shape and temperature distribution of an object being cooked on the basis of sensor data acquired by a sensor for measuring the states of a cooking tool and the object being cooked, and an internal temperature estimation unit for estimating the internal temperature of the object being cooked by carrying out thermal conduction analysis based on the three-dimensional model. This invention can be applied to a cooking assistance system for assisting with the cooking work of a cook.

Description

[Technical Field]

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that are capable of appropriately estimating the internal temperature of foodstuffs being heated. [Background technology]

[0002] When cooking steak or other large pieces of meat in a frying pan, it is extremely difficult to accurately control the degree of doneness inside the food. Experienced chefs have mastered the skill of pressing the meat with their finger while it is cooking to determine the degree of doneness inside, but this is not always possible with ingredients that vary greatly from one to another, and they may not be able to make a perfect judgment and may not be able to cook the food as intended.

[0003] Some recent cooking appliances have the function of controlling the temperature by inserting a thermometer probe into the meat and heating it, but this is not desirable due to hygiene risks and the risk of meat juices leaking. There is a high demand for technology that can non-destructively measure the internal temperature of food during cooking, not just meat, and such technology is highly sought after by amateur cooks and professional chefs who pursue precise cooking results.

[0004] Against this background, many technologies have been proposed for estimating the internal temperature of food ingredients while they are being heated. For example, Patent Document 1 proposes a technology in which sensors that measure the shape and surface temperature of an object are placed on the top and sides of the interior of a cooking appliance, a three-dimensional model is constructed based on the shape of the object, and the internal temperature of the object is estimated by heat conduction analysis using the boundary element method. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 8-15037 [Patent Document 2] Special Publication No. 2010-508493 [Patent Document 3] Japanese Patent Application Laid-Open No. 2015-206502 [Patent Document 4] Japanese Patent Application Publication No. 2019-200002 Summary of the Invention [Problem to be solved by the invention]

[0006] The technology of Patent Document 1 does not anticipate measuring (estimating) the temperature of the bottom side, which cannot be directly detected by a temperature sensor. Therefore, the technology described in Patent Document 1 is not suitable for cooking with heat in a frying pan or the like.

[0007] The present technology has been developed in light of these circumstances, and makes it possible to appropriately estimate the internal temperature of foodstuffs during heating. [Means for solving the problem]

[0008] An information processing device according to one aspect of the present technology includes a construction unit that constructs a three-dimensional model representing the shape and temperature distribution of the object to be cooked based on sensor data acquired by a sensor that measures the state of the cooking utensil and the object to be cooked, and an internal temperature estimation unit that estimates the internal temperature of the object to be cooked by performing a heat conduction analysis based on the three-dimensional model.

[0009] In one aspect of the present technology, a three-dimensional model representing the shape and temperature distribution of the object to be cooked is constructed based on sensor data obtained by a sensor that measures the state of the cooking utensil and the object to be cooked, and the internal temperature of the object to be cooked is estimated by performing heat conduction analysis based on the three-dimensional model. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a configuration example of a cooking assistance system according to an embodiment of the present technology. [Figure 2] FIG. 1 is a block diagram showing the configuration of a cooking assistance system. [Figure 3] FIG. 2 is a block diagram showing an example of the functional configuration of a calculation unit. [Figure 4] 10 is a flowchart illustrating processing by the information processing device. [Figure 5] 5 is a flowchart illustrating a position and shape recognition process performed in step S2 of FIG. 4. [Figure 6] FIG. 10 is a diagram illustrating an example of contour extraction. [Figure 7] 10A to 10C are cross-sectional views showing an example of a method for constructing a three-dimensional model of an object to be cooked. [Figure 8] 10A to 10C are cross-sectional views showing an example of a method for constructing a three-dimensional model of an object to be cooked. [Figure 9] 5 is a flowchart illustrating a surface temperature extraction process performed in step S3 of FIG. 4. [Figure 10] FIG. 10 is a diagram showing an example of a three-dimensional model of an object to be cooked. [Figure 11] FIG. 10 is a diagram illustrating an example of temperature extraction of a heating medium. [Figure 12] FIG. 10 is a diagram illustrating an example of temperature extraction of a heating medium. [Figure 13] 5 is a flowchart illustrating a heat conduction characteristic estimation process performed in step S5 of FIG. 4. [Figure 14] 1A and 1B are diagrams showing examples of thermal images of steak meat being fried in a frying pan before and after being flipped. [Figure 15] 5 is a flowchart illustrating an internal temperature estimation process performed in step S6 of FIG. 4. [Figure 16] 10A and 10B are cross-sectional views illustrating an example of reconstruction of a three-dimensional model. [Figure 17] FIG. 10 is a diagram illustrating an example of a heat conduction model. [Figure 18] FIG. 10 is a diagram showing an example of the measurement results of the temperature change of the exposed surface after a steak being cooked in a frying pan is flipped over. [Figure 19] FIG. 10 is a diagram showing an example of the measurement results of the temperature change of a frying pan after heating has stopped. [Figure 20] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0011] <Overview of this technology> This technology supports appropriate control of the heating state of an object to be cooked in a cooking method in which the object is brought into contact with a heating medium and heated by thermal conduction.

[0012] Hereinafter, embodiments of the present technology will be described in the following order. 1. Cooking assistance system 2. Configuration of each device 3. Operation of information processing device 4.Other

[0013] <<1. Cooking support system>> FIG. 1 is a diagram showing an example of the configuration of a cooking assistance system according to an embodiment of the present technology.

[0014] The cooking assistance system of the present technology is used, for example, in a scene where a cook U1 uses a frying pan as a cooking utensil 11 to grill steak meat as an object 12 to be cooked.

[0015] The cooking assistance system in FIG. 1 is composed of a heating device 1, a stereo camera 2, a thermography camera 3, a processor module 4, a network device 5, a server 6, an information terminal 7, and an air conditioner 8.

[0016] The heating device 1 is composed of a stove, an IH cooker, or the like that heats the cooking utensil 11. The heating device 1 is installed in a work area where the cooking object 12 is cooked using the cooking utensil 11.

[0017] The camera sensor including the stereo camera 2 and the thermographic camera 3 is installed, for example, above the work area, in a position that allows it to view the work area including the cooking utensils 11 and the food items 12. In a typical kitchen environment, for example, the camera sensor is attached near a ventilation opening installed on top of the heating device 1.

[0018] The stereo camera 2 captures images of the work area and acquires visible images including depth information. The thermography camera 3 captures images of the work area and acquires thermal images. The camera sensors are connected to a processor module 4 installed in a predetermined location, such as the kitchen, via a high-speed interface, and data is sent and received between the camera sensors and the processor module 4 in real time.

[0019] The processor module 4 is connected to the server 6 via the network device 5. The processor module 4 estimates the internal temperature of the object 12 by processing information in cooperation with the server 6. In addition, the processor module 4 automatically adjusts the heat of the heating device 1, automatically adjusts the air conditioning by the air conditioner 8, and presents information to the cook U1 via the information terminal 7, depending on the heating state of the object 12.

[0020] As indicated by the dashed lines, data is transmitted and received between the processor module 4 and each of the heating device 1, the network device 5, the information terminal 7, and the air conditioner 8, for example, by wireless communication.

[0021] The server 6 is a server on an intranet or the Internet.

[0022] The information terminal 7 is configured as a smartphone or tablet terminal with a display such as an LCD (Liquid Crystal Display). The information terminal 7, placed near the cook U1, detects the cook U1's operation and accepts the input of information. The information terminal 7 presents information to the cook U1 according to the control of the processor module 4.

[0023] The air conditioner 8, under the control of the processor module 4, adjusts the air conditioning of the kitchen environment.

[0024] <<2. Configuration of each device>> FIG. 2 is a block diagram showing the configuration of the cooking assistance system.

[0025] The cooking assistance system of FIG. 2 is composed of a sensor unit 21, an information processing device 22, and an effector unit 23.

[0026] The processing units shown in FIG. 2 show a logical configuration of functions and do not limit the physical device configuration. One processing unit may include multiple physical devices. Also, one physical device may constitute multiple processing units. The specific configurations of the interfaces connecting the insides of each processing unit and the interfaces connecting between processing units are also not limited. The communication paths between processing units may be configured as wired or wireless, and the communication paths may include the Internet.

[0027] The sensor unit 21 includes a temperature sensor 31, a distance sensor 32, and an image sensor 33. For example, the temperature sensor 31, the distance sensor 32, and the image sensor 33 are configured by camera sensors including a stereo camera 2 and a thermography camera 3.

[0028] The temperature sensor 31 is a sensor that measures the surface temperature distribution of an object. The distance sensor 32 is a sensor that measures the three-dimensional shape of an object. The image sensor 33 is a sensor that captures an image of the object in the visible light range.

[0029] Although each sensor of the sensor unit 21 is treated as a distinct logical function, it is not necessarily configured by three corresponding physical devices. Hereinafter, these three sensors will be collectively referred to as a basic sensor group, where appropriate.

[0030] Each sensor in the basic sensor group measures the state of the object in a non-contact and non-destructive manner, and transmits the measurement results as time-series data to the information processing device 22. The internal and external parameters of each sensor are calculated by so-called camera calibration, and pixels between sensors can be associated with each other by coordinate transformation. In other words, the data measured by the basic sensor group can be expressed in a common three-dimensional coordinate system (world coordinates) in the information processing device 22.

[0031] The information processing device 22 includes a calculation unit 41 and a storage unit 42. For example, the information processing device 22 is configured by a processor module 4.

[0032] The calculation unit 41 is configured by, for example, a general-purpose calculation unit such as a CPU, GPU, or DSP, or a dedicated calculation unit specialized for AI-related processing, etc. The calculation unit 41 estimates the three-dimensional shape, surface temperature, and heat conduction characteristics of the object 12 to be cooked based on the information measured by the sensor unit 21 and known information stored in the storage unit 42, and estimates the internal temperature of the object 12 by heat conduction analysis using a three-dimensional model.

[0033] The storage unit 42 is configured by a storage device such as a memory, a storage, etc. The storage unit 42 holds known information such as a database that indicates the heat conduction characteristics of the object 12 to be cooked.

[0034] The information processing device 22 may be configured by combining a local processor module 4 and a server 6 on a network.

[0035] The effector unit 23 is a peripheral device controlled by the information processing device 22 to control the cooking state. Note that the effector unit 23 also includes devices that are not included in the sensor unit 21, such as an input device for inputting information by the cook, and an information terminal used by a user in a remote location away from the work area.

[0036] Effector unit 23 includes, for example, UI device 51, heating device 1, and air conditioner 8. UI device 51 is configured by information terminal 7 in Fig. 1, a PC, etc. Effector unit 23 automatically controls the heating operation and presents information to the user.

[0037] FIG. 3 is a block diagram showing an example of the functional configuration of the calculation unit 41.

[0038] 3, the calculation unit 41 includes a sensor data input unit 101, a position and shape recognition unit 102, a surface temperature extraction unit 103, a process status recognition unit 104, a heat conduction characteristic estimation unit 105, an internal temperature estimation unit 106, and an effector control unit 107. The functions of each processing unit will be described in detail later.

[0039] The sensor data input unit 101 receives the sensor data transmitted from the sensor unit 21 and outputs it to the position and shape recognition unit 102 , the surface temperature extraction unit 103 , the process status recognition unit 104 , and the heat conduction characteristic estimation unit 105 .

[0040] The position and shape recognition unit 102 recognizes the position and shape of the object 12 to be cooked based on the sensor data supplied from the sensor data input unit 101. For example, the position and shape recognition unit 102 detects obstruction of the field of view due to the cooking work of the cook.

[0041] Furthermore, the position and shape recognition unit 102 detects the cooking utensil 11 and the object 12 and extracts its contours. The position and shape recognition unit 102 recognizes the shape of the object 12 and constructs a three-dimensional model. The three-dimensional model of the object 12 represents the shape and temperature distribution of the object 12. The recognition results by the position and shape recognition unit 102 are supplied to the surface temperature extraction unit 103, the process status recognition unit 104, the heat conduction characteristic estimation unit 105, and the internal temperature estimation unit 106.

[0042] The surface temperature extraction unit 103 extracts the surface temperatures of the object 12 and the heating medium based on the sensor data supplied from the sensor data input unit 101 and the contour information of the object 12 and the heating medium supplied from the position and shape recognition unit 102. The surface temperature extraction results by the surface temperature extraction unit 103 are supplied to the process status recognition unit 104, the heat conduction characteristic estimation unit 105, and the internal temperature estimation unit 106.

[0043] The process status recognition unit 104 recognizes the status of the cooking process based on the sensor data supplied from the sensor data input unit 101, the position and shape of the object 12 supplied from the position and shape recognition unit 102, and the surface temperature extraction result supplied from the surface temperature extraction unit 103. Specifically, the process status recognition unit 104 detects the insertion, removal, shape change, position and orientation change, etc. of the object 12. The recognition results by the process status recognition unit 104 are supplied to the heat conduction property estimation unit 105 and the internal temperature estimation unit 106.

[0044] The heat conduction property estimation unit 105 estimates the heat conduction property of the object 12 to be cooked based on the sensor data supplied from the sensor data input unit 101 and the position and shape of the object 12 supplied from the position and shape recognition unit 102, in accordance with the state of the cooking process recognized by the process state recognition unit 104. Specifically, the heat conduction property estimation unit 105 estimates the physical property value of the object 12 to be cooked.

[0045] The heat conduction characteristic estimation unit 105 estimates the contact heat resistance between the object 12 and the heating medium based on the contour information of the object 12 supplied from the position and shape recognition unit 102 and the surface temperature extraction result by the surface temperature extraction unit 103. Information representing the contact heat resistance estimated by the heat conduction characteristic estimation unit 105 is supplied to the internal temperature estimation unit 106.

[0046] The internal temperature estimation unit 106 estimates the temperature of the heated portion of the object 12 based on the information supplied from the heat conduction characteristic estimation unit 105. The internal temperature of the object 12 is estimated based on a three-dimensional model of the object 12 in which the temperature of the heated portion is set according to the cooking process status recognized by the process status recognition unit 104. The internal temperature estimation result by the internal temperature estimation unit 106 is supplied to the effector control unit 107.

[0047] The effector control unit 107 controls the effector unit 23 based on the result of the internal temperature estimation by the internal temperature estimation unit 106. The effector control unit 107 controls the heating device 1, controls the presentation of information to the cook, and the like.

[0048] <<3. Operation of the information processing device>> <Overall processing> The processing of the information processing device 22 having the above configuration will be described with reference to the flowchart of FIG.

[0049] The flowchart in Figure 4 shows the main processing contents in a serialized manner as one embodiment. The processing of each step is not necessarily performed in the order shown in Figure 4. Some processing is performed only under specific conditions.

[0050] In step S1, the sensor data input unit 101 receives input of sensor data from each sensor of the sensor unit 21. A thermal image representing a surface temperature is input from the temperature sensor 31, and an RGB image as a visible image is input from the distance sensor 32. Furthermore, a depth image as depth information is input from the image sensor 33.

[0051] Each piece of sensor data is time-series data captured in real time and input at any timing to the sensor data input unit 101. For ease of explanation, it is assumed below that all sensor data is input synchronously at a fixed cycle.

[0052] The series of processes described with reference to the flowchart in Fig. 4 are periodically repeated each time new sensor data is received. For example, if the frame rate of the sensor data is 30 fps, the series of processes is completed within approximately 33 ms.

[0053] In step S2, the position and shape recognition unit 102 performs a position and shape recognition process. In the position and shape recognition process, the cooking operation of the cook is detected, and the position, contour, and shape of the object 12 are recognized. Furthermore, a three-dimensional model of the object 12 is constructed based on the recognition results of the position and shape of the object 12. Details of the position and shape recognition process will be described later with reference to the flowchart in FIG. 5.

[0054] In step S3, the surface temperature extraction unit 103 performs a surface temperature extraction process. In the surface temperature extraction process, the surface temperatures of the object 12 and the heating medium are extracted. Details of the surface temperature extraction process will be described later with reference to the flowchart in FIG.

[0055] In step S4, the process status recognition unit 104 recognizes the status of the cooking process based on the information obtained up to the processing in step S4. Details of the recognition of the status of the cooking process will be described later.

[0056] In step S5, the thermal conduction property estimation unit 105 performs a thermal conduction property estimation process, which estimates the thermal conduction property of the object 12. Details of the thermal conduction property estimation process will be described later with reference to the flowchart of FIG.

[0057] In step S6, the internal temperature estimation unit 106 performs internal temperature estimation processing. In the internal temperature estimation processing, the temperature of the heated portion of the object 12 is estimated, and the internal temperature is estimated based on the result of estimating the temperature of the heated portion. Details of the internal temperature estimation processing will be described later with reference to the flowchart in FIG. 15.

[0058] In step S7, the effector control section 107 controls the effector section 23 based on the result of estimation of the internal temperature of the object 12 by the internal temperature estimation section 106. An example of the control of the effector section 23 will be described later.

[0059] After the effector unit 23 is controlled in step S7, the process ends. The above series of processes is executed every time sensor data is input.

[0060] <Position and shape recognition processing> Here, the position and shape recognition process performed in step S2 of FIG. 4 will be described with reference to the flowchart of FIG.

[0061] (2-1) Detection of obstruction of view due to cook's operation In step S21, the position and shape recognition unit 102 detects whether the field of view of the camera sensor is blocked by the cooking work of the cook.

[0062] When a cook is operating a cooking object 12, a human finger or tongs may enter the field of view of the camera sensor and block the object whose position and shape are being recognized. The object includes a cooking object 12 such as steak meat and a cooking utensil 11 such as a frying pan. If subsequent processing is performed in such a state, the accuracy of the processing may be reduced.

[0063] The position and shape recognition unit 102 detects that the cook's hand is inserted within the field of view (shooting range) of the camera sensor through image recognition of the RGB image and the depth image, and recognizes the position of the hand. If an important object is obscured by the cook's cooking work, subsequent processing related to object recognition is skipped. Skipping processing from step S22 onwards makes it possible to prevent a decrease in processing accuracy.

[0064] (2-2) Detection and contour extraction of the cooking utensil 11 and the cooking object 12 In step S22, the position and shape recognition unit 102 uses image processing to detect whether the cooking utensil 11 and the object 12 are present within the field of view. If the cooking utensil 11 and the object 12 are present within the field of view, the position and shape recognition unit 102 extracts the contours of the cooking utensil 11 and the object 12 on the image of the basic sensor group. Extracting the contours means identifying the contours of the objects.

[0065] FIG. 6 is a diagram showing an example of contour extraction.

[0066] 6 shows an RGB image obtained by capturing an image of a steak being cooked as an object to be cooked 12 using a frying pan as cooking utensil 11. The position and shape recognition unit 102 extracts the outline of the vessel (main body) of cooking utensil 11 and the outline of object to be cooked 12 from the RGB image, as shown enclosed by a thick line in FIG.

[0067] There are various methods for contour extraction using image processing technology based on sensor data acquired by a basic sensor group. Regardless of the method used, if a contour can be identified on an image acquired by one sensor, it can also be identified on images acquired by other sensors by coordinate transformation between the sensors.

[0068] The contour of the cooking utensil 11 is extracted, for example, by the following method.

[0069] (A) Example of using a database with machine learning Using a database (inference model) generated by machine learning, object detection of the cooking utensil 11 and contour extraction of the cooking utensil 11 are performed. When an RGB image is used as input information, information representing the contour of the cooking utensil 11 is obtained from the database as output information.

[0070] (B) Example of using known information When known information such as a three-dimensional model representing the three-dimensional shape of the cooking utensil 11 is prepared in the position and shape recognition unit 102, the known information is used to extract the contour of the cooking utensil 11. For example, the presence or absence, position, and orientation of the cooking utensil 11 are detected by registering the three-dimensional model with the scene point cloud generated based on the depth image. The contour of the cooking utensil 11 is extracted on the RGB image based on the detected position, orientation, and three-dimensional model.

[0071] (C) Example of using markers embedded in cooking utensils 11 When the three-dimensional shape of cooking utensil 11 and markers such as feature patterns embedded in specific areas of cooking utensil 11 are prepared as known information in position and shape recognition unit 102, the known information is used to detect the presence or absence and position and orientation of the markers on the RGB image. Based on the position and orientation of the markers, the position and orientation of cooking utensil 11 are identified, and the contours on the RGB image are extracted.

[0072] The position and shape recognition unit 102 detects the presence or absence of a cooking utensil 11 and extracts its contour using the method described above, and then detects the presence or absence of a cooking object 12 and extracts its contour, with the area within the contour of the cooking utensil 11 as the area of ​​interest.

[0073] The object to be cooked 12 that can be handled by the method proposed by the present technology is assumed to be a solid object whose overall shape and the shape of the heated portion can be identified. For example, a solid object such as a chunk of meat, a fish (whole fish or fillets), a pancake, or an omelet can be handled as the object to be cooked 12. Multiple objects to be cooked 12 may be placed on the cooking utensil 11, or different types of objects to be cooked 12 may be mixed and placed on the cooking utensil 11.

[0074] The detection of the presence or absence of the object 12 and the extraction of its outline are performed, for example, by the following method.

[0075] (D) Example of using a database with machine learning Using a database generated by machine learning, object detection of the object 12 and contour extraction of the object 12 are performed. When an RGB image is used as input information, information representing the contour of the object 12 is obtained from the database as output information.

[0076] (E) Example of using background subtraction When the three-dimensional shape of the cooking utensil 11 is known and the position and orientation of the cooking utensil 11 are detected using the above method (B) or (C), a depth image showing only the cooking utensil 11 is generated. By performing background subtraction processing on the depth image actually captured by the stereo camera 2 using the depth image showing only the cooking utensil 11 as the background, a depth image of the object 12 to be cooked is extracted as the foreground. This allows the contour of the object 12 to be extracted from the depth image.

[0077] (2-3) Shape recognition of the cooking object 12 and construction of a 3D model 5, in step S23, the position and shape recognition unit 102 constructs a three-dimensional model of the object 12. The position and shape recognition unit 102 functions as a construction unit that constructs the three-dimensional model of the object 12 based on sensor data.

[0078] When the object 12 to be cooked is placed in the cooking utensil 11, the position and shape recognition unit 102 constructs a three-dimensional model based on the shape information representing the shape of the object 12 to be cooked.

[0079] Furthermore, if any of the shape, volume, and position and orientation of the object 12 to be cooked changes during the cooking process, the position and shape recognition unit 102 reconstructs a three-dimensional model.

[0080] Because the contour of the object 12 on the depth image has been extracted by the processing in step S22, it is possible to create point cloud data of the object 12. The point cloud data represents the three-dimensional shape of the object 12 with respect to the exposed surface to which the distance sensor 32 can detect the distance. The three-dimensional shape and position and orientation of the cooking utensil 11 (particularly the heating surface in contact with the object 12) that serves as the heating medium are recognized. The three-dimensional shape and position and orientation of the cooking utensil 11 serve as the basis for the three-dimensional model of the object 12.

[0081] 7 and 8 are cross-sectional views showing an example of a method for constructing a three-dimensional model of the object to be cooked 12. Although two-dimensional cross sections are shown in Fig. 7 and Fig. 8, in reality, processing is carried out so as to construct a three-dimensional model.

[0082] 7A, it is assumed that steak meat as the object to be cooked 12 is placed on the cooking utensil 11. The position and shape recognition unit 102 generates point cloud data of the object to be cooked 12 based on the depth image.

[0083] As shown by the dashed line in B of Figure 7, point cloud data based on the depth image is generated to represent the exposed surface of the object to be cooked 12 that is included in the angle of view of the distance sensor 32 installed above the cooking utensil 11.

[0084] As shown in C of Fig. 7, the position and shape recognition unit 102 divides the space above the cooking utensil 11, including the object 12, into a mesh using the heating surface of the cooking utensil 11 as a reference. In C of Fig. 7, the space above the cooking utensil 11 is divided into voxels. Setting parameters that represent the shape and fineness of the mesh are set appropriately depending on the type, size, shape, etc. of the object 12.

[0085] As shown by the darkly filled voxels in A of FIG. 8, the position and shape recognition unit 102 determines the voxels including the point cloud data of the object 12 as components of the three-dimensional model.

[0086] As shown by the lightly filled voxels in B of Figure 8, the position and shape recognition unit 102 also determines all voxels below (in a direction perpendicular to the heating surface of the cooking utensil 11 and away from the distance sensor 32) the voxels determined as components of the three-dimensional model based on the point cloud data as components of the three-dimensional model.

[0087] The position and shape recognition unit 102 constructs a three-dimensional model of the object 12 by treating a set of voxels determined as components as the shape structure of the three-dimensional model of the object 12.

[0088] The above-described procedure for constructing a three-dimensional model is one example. A three-dimensional model of the object 12 is constructed in a representation format suitable for heat conduction analysis based on the shape information of the object 12 measured by the distance sensor 32. The accuracy of the contour extraction performed in step S22 and the construction of the three-dimensional model performed in step S23 is determined based on the accuracy of the internal temperature estimation required as a result of the heat conduction analysis.

[0089] After the three-dimensional model of the object 12 is constructed in step S23, the process returns to step S2 in FIG. 4, and the subsequent processes are carried out.

[0090] <Surface temperature extraction process> The surface temperature extraction process performed in step S3 of FIG. 4 will be described with reference to the flowchart of FIG.

[0091] (3-1) Extraction of the surface temperature of the object 12 to be cooked In step S31, the surface temperature extraction unit 103 extracts the surface temperature of the object 12 based on the thermal image acquired by the temperature sensor 31, and maps it onto the three-dimensional model of the object 12 constructed by the position and shape recognition unit 102. Extracting the temperature means detecting the temperature.

[0092] A position on the surface of the object 12 from which the temperature can be extracted based on the thermal image is called a “temperature extraction point.” The temperature extraction point is expressed by three-dimensional coordinates.

[0093] The positions on a 3D model where the temperature is defined are called "temperature definition points." Temperature definition points are set according to the construction method of the 3D model. For example, temperature definition points are set at the vertices or center points of each voxel.

[0094] The surface temperature extraction unit 103 determines the temperature of a temperature definition point based on the temperature values ​​of temperature extraction points near the temperature definition point. For example, the surface temperature extraction unit 103 determines an area within a certain distance from the temperature definition point as a nearby area, and determines the temperature of the temperature extraction point closest to the temperature definition point as the temperature of the temperature definition point.

[0095] The temperature of the temperature definition point may be determined using general sampling processes such as applying a filter to the thermal image when there is a lot of noise at the temperature extraction point, or linearly interpolating the temperature of the temperature definition point when there is a large temperature gradient near the temperature definition point.

[0096] For example, the temperature definition point is determined for the voxel indicated by diagonal lines in Fig. 10. The voxel indicated by diagonal lines corresponds to the voxel including the point cloud data of the object 12. Note that the voxel for which the temperature definition point is determined does not have to correspond to the voxel including the point cloud data of the object 12.

[0097] (3-2) Extraction of the surface temperature of the heating medium In step S32, the surface temperature extraction unit 103 extracts the surface temperature of the heating medium based on the thermal image acquired by the temperature sensor 31.

[0098] The contours of the cooking utensil 11 and the object 12 to be cooked are extracted from the thermal image by the position and shape recognition process in step S2 of Fig. 4. For example, by performing image processing on a thermal image obtained by capturing an image of steak being cooked in a frying pan, as shown in Fig. 11A, the contours of the cooking utensil 11 and the object 12 to be cooked are extracted as shown by the outlines in bold lines in Fig. 11B.

[0099] Surface temperature extraction unit 103 extracts the surface temperature of the heating medium based on the temperature of the area indicated by the hatched area A in Fig. 12, which is the area inside the outline of cooking utensil 11 excluding the area inside the outline of object 12. The heating medium whose temperature is actually measured is, for example, oil or water placed in cooking utensil 11.

[0100] As shown in the thermal image A of Figure 11, the temperature distribution of the heating medium varies depending on how the oil and fat accumulates on the frying pan. In order to obtain the temperature of the heating medium that directly contributes to the heat conduction to the object 12, it is preferable to focus on the area near the object 12.

[0101] Therefore, as shown in Fig. 12B, the surface temperature extraction unit 103 obtains an enlarged outline of the object 12 and focuses on a nearby region, which is the region inside the enlarged outline excluding the inside of the outline of the object 12. The enlarged outline is an outline obtained by enlarging the outline of the object 12 outward, and is set so as to be included inside the outline of the cooking utensil 11. The area indicated by diagonal lines in Fig. 12B is the nearby region.

[0102] The surface temperature extraction unit 103 calculates the average temperature of the nearby area of ​​the entire area of ​​the heating medium as the surface temperature T heat Extract as.

[0103] Heating medium surface temperature T heat After being calculated in step S32, the process returns to step S3 in FIG. 4, and the subsequent processes are carried out.

[0104] <Contextual awareness of the cooking process> The following describes in detail the recognition of the cooking process status performed in step S4 of Fig. 4. For example, the process status recognition unit 104 recognizes that the following status has occurred. (A) Putting the food 12 into the cooking utensil 11 (B) Removal of the object 12 to be cooked from the cooking utensil 11 (C) Change in position and orientation of the object 12 to be cooked within the cooking utensil 11 (D) Change in shape of the object 12 to be cooked in the cooking utensil 11

[0105] There are various possible methods for situation recognition using image recognition technology based on sensor data acquired by a basic sensor group. An example will be described below.

[0106] The processing in steps S2 and S3 recognizes the number of cooking objects 12 placed in the cooking utensil 11, as well as their respective positions, contours, shapes, and surface temperatures. The timing at which the cook performed the cooking task is also recognized as auxiliary information. The auxiliary information is information for assisting in the recognition of the status of the cooking process.

[0107] The occurrence of adding or removing the cooking objects 12 (the above (A) or (B)) is recognized when the number of cooking objects 12 changes before and after the cook performs cooking work.

[0108] When a weight sensor is provided as a sensor constituting the sensor unit 21, the addition or removal of an object to be cooked 12 may be recognized in response to a discontinuous change in weight detected by the weight sensor. When there are multiple objects to be cooked 12, the identity of each object to be cooked 12 must be identified and the correspondence with the three-dimensional model must be appropriately maintained.

[0109] The occurrence of a change in the position and orientation of the objects 12 (above (C)) is recognized based on changes in the position, outline, shape, surface temperature, surface image, etc. of the objects 12, regardless of whether there is a change in the number of objects. In particular, it is recognized that the part of the object 12 that is in contact with the heating medium (the heated part), such as when meat or fish is turned over while being grilled, has changed significantly.

[0110] In the example of flipping a steak, there may not be a significant change in the position, outline, shape, etc. of the steak, so it is preferable to refer to this information and determine the change in the position and posture of the steak based on the change in the surface temperature of the object to be cooked 12, as described below.

[0111] The shape of the object 12 may change during cooking. For example, pancakes or hamburger steaks, which are the object 12, expand when heated. If the posture or shape of the object 12 changes during cooking and the deviation from the three-dimensional model becomes significant, the three-dimensional model must be reconstructed. The reconstruction of the three-dimensional model will be described later.

[0112] <Heat conduction property estimation processing> The heat conduction characteristic estimation process performed in step S5 of FIG. 4 will be described with reference to the flowchart of FIG.

[0113] ·Generating thermal conductivity properties In step S41, the heat conduction characteristic estimation unit 105 determines, based on the process status recognition result by the process status recognition unit 104, whether or not the introduction of the object 12 to be cooked has been detected.

[0114] If it is determined in step S41 that the introduction of the object 12 has been detected, in step S42 the thermal conduction property estimation unit 105 estimates the thermal conduction properties of the object 12 introduced into the cookware 11. The thermal conduction properties are parameters necessary for thermal conduction analysis, and include, for example, the thermal conductivity, specific heat, density, and thermal diffusion coefficient of the object. The thermal diffusion coefficient is calculated based on the thermal conductivity, specific heat, and density of the object.

[0115] Specifically, the thermal conduction property estimation unit 105 identifies the food ingredient properties that indicate the type, part, quality, etc. of the food ingredient as the object to be cooked 12, and calculates the thermal conduction properties corresponding to the food ingredient properties using various known measurement data. The food ingredient properties are identified, for example, by the method described below.

[0116] (A) Example of a cook selecting recipe data The cook selects recipe data using the UI function of the effector unit 23. For example, when cooking according to navigation by an application installed on the information terminal 7, the cook selects recipe data for the dish to be made.

[0117] The heat conduction property estimation unit 105 identifies the property of an ingredient by directly acquiring the property information of an ingredient contained in the recipe data selected by the cook, or by acquiring the property information of an ingredient from a database.

[0118] (B) Example of a cook inputting ingredients' characteristics As a supplement to the above method (A), the cook can directly input the ingredient characteristics using the UI function. For example, if there is a difference between the ingredient characteristics of the ingredients presented in the recipe data and the ingredient characteristics of the ingredients actually used in cooking, the cook can input information on ingredient characteristics that are not included in the recipe data. The type of ingredient may also be set using buttons on the main body of cooking appliance 11, such as a microwave oven.

[0119] (C) Example of identifying food characteristics using image recognition The thermal conduction property estimation unit 105 identifies the food property of the object to be cooked 12 by image recognition based on information acquired by the sensor unit 21, such as an RGB image showing the object to be cooked 12. For food property such as the fat content of meat, which shows individual differences between ingredients, image recognition of an image showing the actual object to be cooked 12 is effective.

[0120] (D) Example of determining density The heat conduction property estimation unit 105 can identify the volume of the object 12 that has been placed in the oven based on a three-dimensional model of the object 12. If the sensor unit 21 is equipped with a weight sensor and can measure the weight of each object 12 individually, the heat conduction property estimation unit 105 identifies the density based on the volume and weight of the object 12. When identifying food ingredient properties using the above method (C), by using density as known information, it is possible to narrow down the candidates for food ingredient properties that are more likely to be present.

[0121] Based on the food material properties identified as described above, the heat conduction properties of the object 12 to be cooked are estimated.

[0122] - Updated heat transfer properties On the other hand, if it is determined in step S41 that the introduction of the object 12 has not been detected, the process proceeds to step S43.

[0123] In step S43, the heat conduction property estimation unit 105 determines, based on the process status recognition result by the process status recognition unit 104, whether or not a change in the shape of the object 12 has been detected.

[0124] If it is determined in step S43 that a change in the shape of the object 12 has been detected, the heat conduction property estimation unit 105 updates the heat conduction property of the object 12 in step S44.

[0125] The thermal conductivity characteristics of the object 12 generally change during the heating process. Therefore, it is desirable to update the thermal conductivity characteristics as needed, not only when the object is placed in the oven but also after it is placed in the oven. The thermal conductivity characteristic estimation unit 105 repeatedly updates the thermal conductivity characteristics during the cooking process, such as the heating process.

[0126] Some foods, such as pancakes, have a significant change in density during heating. Such a change in density affects the heat conduction characteristics. If the volume of the object 12 changes significantly during heating, the heat conduction characteristics estimation unit 105 updates the density estimate based on the state of the object 12.

[0127] When meat or fish is heated, it loses moisture. Because the specific heat of water is high, the moisture content of the object 12 to be cooked has a significant effect on the heat conduction characteristics. Therefore, it is also useful to detect changes in the moisture content.

[0128] For example, as described in Patent Document 4, if it is assumed that the weight change of the object to be cooked 12 is due to evaporation of water, the heat conduction characteristic estimation unit 105 can estimate the water content based on the weight change of the object to be cooked 12.

[0129] Alternatively, the moisture content may be detected using a database constructed by machine learning that inputs an RGB image acquired by the image sensor 33 and outputs the ingredient characteristics and moisture content of the object 12. In this case, estimation accuracy equivalent to that achieved by a skilled chef visually judging the condition of the object 12 can be expected.

[0130] The database may be constructed by machine learning using not only RGB images but also thermal images (surface temperatures of the objects 12) obtained by capturing images of the objects 12, internal temperatures estimated in subsequent processing, and the like.

[0131] When the sensor unit 21 is provided with a near-infrared spectrometer, the change in moisture content may be determined based on the infrared radiation spectrum of the object 12 measured by the near-infrared spectrometer. In this way, the moisture content may be measured directly.

[0132] Estimation of thermal contact resistance On the other hand, if it is determined in step S43 that no change in the shape of the object 12 has been detected, the process proceeds to step S45.

[0133] In step S45, the heat conduction characteristic estimation unit 105 determines, based on the process status recognition result by the process status recognition unit 104, whether or not a change in the posture of the object 12 to be cooked has been detected.

[0134] If it is determined in step S45 that a change in the posture of the object 12 has been detected, in step S46 the heat conduction characteristic estimation unit 105 estimates the contact heat resistance between the object 12 and the heating medium.

[0135] FIG. 14 shows an example of thermal images of steak meat being fried in a frying pan before and after being turned over.

[0136] As shown in the upper part of Fig. 14, before the steak meat serving as the object to be cooked 12 is turned over, the surface of the object to be cooked 12 has not yet been heated and remains at a temperature close to room temperature. In the upper part of Fig. 14, the surface of the object to be cooked 12 is shown in a dark color, which indicates that the surface temperature is lower than the temperature of the surrounding heating medium, etc.

[0137] On the other hand, after the object 12 is turned over, the surface of the object 12 becomes hot due to heating, as shown in the lower part of Fig. 14. In the lower part of Fig. 14, the surface of the object 12 is shown in a whitish color, which indicates that the temperature of the surface has become high, similar to the temperature of the surrounding heating medium, etc.

[0138] The temperature of the object 12 and the temperature of the heating medium near the object 12 are extracted by the processes of steps S2 and S3 in Fig. 4. Based on the temperatures of the object 12 and the heating medium, it is determined that the position of the object 12 has changed and the heated portion of the object 12, which was previously the backside, is now exposed as the front side, i.e., that the object 12 has been turned over.

[0139] For example, if the conditions defined by the following formulas (1) and (2) are satisfied, it is determined that the object 12 has been turned over. The determination of whether the object 12 has been turned over based on the following formulas (1) and (2) is made by the process status recognition unit 104.

[0140]

number

number

[0141] T before represents the surface temperature of the object 12 before the change in posture, and T after represents the surface temperature of the object 12 after the change in posture. heat represents the temperature of the heating medium near the object 12 to be cooked after the posture change.

[0142] Also, T flip represents the temperature difference threshold at which it is determined that an inversion has occurred, and T gap represents the threshold value of the temperature difference at the contact surface at which it is determined that the object 12 has been in contact with the heating medium for a sufficient period of time.

[0143] Satisfying the condition defined by equation (1) means that the change in surface temperature of the object to be cooked 12 is greater than a threshold value before and after the posture change, i.e., the surface of the object to be cooked 12 exposed by turning it over is sufficiently heated.

[0144] Furthermore, satisfying the condition defined by equation (2) means that the difference between the temperature of the heating medium and the surface temperature of the object to be cooked 12 after the posture change is smaller than a threshold value, that is, the surface of the object to be cooked 12 exposed by turning it over has been sufficiently heated to a temperature close to the temperature of the heating medium.

[0145] The surface temperature T when the conditions defined by the formulas (1) and (2) are satisfied and the heated portion of the object 12 is judged to have been sufficiently heated and immediately exposed to the surface. after is the temperature T heat This temperature can be considered to be equal to the temperature of the heated part that was in contact with the heating medium.

[0146] As shown in the thermal image at the bottom of Figure 14, the surface temperature T after and the temperature of the heating medium T heat Generally, the temperature of the heating medium T heat The surface temperature T afterThis is due to contact thermal resistance occurring at the contact surface between the heating medium and the object 12 to be cooked.

[0147] Contact thermal resistance R at the contact surface between the heating medium and the object to be cooked 12 contact is defined by the following equation (3).

[0148]

number

[0149] Q represents the heat flow rate transferred from the heating medium to the object 12. In the case of steady heat conduction, the heat flow rate Q is expressed by the following equation (4).

[0150]

number

[0151] A represents the area of ​​the contact surface where heat conduction occurs, and k represents the thermal conductivity of the object 12. z represents the direction in which heat is transferred. Here, z represents the vertical upward direction from the contact surface. T is the temperature of the object 12 and is expressed as a function of z.

[0152] The area A is calculated based on the contour of the object 12 extracted by the position and shape recognition process in step S2 of Fig. 4. The thermal conductivity k is calculated as part of the thermal conduction characteristics estimated by the thermal conduction characteristics estimation process in step S5.

[0153] Therefore, once ∂T / ∂z is determined, the heat flow rate Q can be estimated using equation (4). When the temperature of the heating medium is stable and heating has been performed for a sufficient period of time, the temperature gradient inside the object 12 to be cooked can be expected to be a relatively monotonic gradient from the contact surface toward the center (along the z direction).

[0154] When a linear temperature gradient occurs, ∂T / ∂z can be approximated by the value expressed by the following equation (5).

[0155]

number

[0156] L represents the thickness of the object 12 in the z direction, and is calculated based on the three-dimensional model constructed by the processing in step S2. center represents the temperature at the center of the object 12 to be cooked, which is located a distance L / 2 above the contact surface.

[0157] Core temperature T center Although the value of is unknown exactly, if equation (1) holds, T center and T before The values ​​of can be approximated as the same value. If we assume that the exposed surface before the posture change has not yet been directly heated and that the temperature of the exposed surface is close to room temperature (the temperature before heating began), it is likely that the temperature at the center will also remain at a similar temperature.

[0158] Therefore, the thermal contact resistance R contact can be approximately calculated using the following equation (6).

[0159]

number

[0160] The thermal contact resistance R obtained in this way contact is used to estimate the internal temperature. After the contact thermal resistance is estimated in step S46, the process returns to step S5 in FIG. 4 and the subsequent steps are carried out.

[0161] Similarly, after the thermal conductivity characteristics are determined in step S42, after the thermal conductivity characteristics are updated in step S44, or if it is determined in step S45 that no change in the posture of the object to be cooked 12 has been detected, the process returns to step S5 in Figure 4 and subsequent processing is performed.

[0162] <Internal temperature estimation process> The internal temperature estimation process performed in step S6 of FIG. 4 will be described with reference to the flowchart of FIG.

[0163] (6-1) Estimation of the temperature of the heated part In step S61, the internal temperature estimation unit 106 estimates the temperature of the heated portion in contact with the heating medium and maps it onto a three-dimensional model of the object 12. For example, the temperature of the heated portion is mapped to the temperature definition points of the voxels indicated by dots in Fig. 10.

[0164] In equation (6), the surface temperature T after The temperature of the heated part (bottom surface) T bottom and the surface temperature T before The temperature of the exposed surface (surface) T top By replacing it with and deforming it, the temperature of the heated part T bottom The following equation (7) is obtained:

[0165]

number

number

[0166] As shown in equation (8), r is a dimensionless constant. contact is due to the roughness, hardness, and pressing pressure of the contact surface. When the oil or fat serving as a heating medium is uniformly distributed on the contact surface and the heated portion of the object 12 is heated to a certain extent, the contact thermal resistance R contact is not expected to change abruptly.

[0167] Therefore, once the thermal contact resistance R is calculated, contact By treating as a constant, the temperature of the heating medium T heat , the temperature T of the exposed surface of the object 12 to be cooked top , and the constant r obtained from known parameters, the temperature T bottom can be estimated.

[0168] Contact thermal resistance R contactis unknown, the internal temperature estimation unit 106 calculates the contact thermal resistance R contact = 0, and the temperature T bottom As the temperature of the heating medium, T heat Map the following.

[0169] (6-2) Estimation of internal temperature In step S62, the internal temperature estimation unit 106 estimates the internal temperature of the object 12. Through the processing up to the previous stage, the temperatures of the surface portion and the heated portion of the object 12 are mapped onto the three-dimensional model.

[0170] Here, the inside of the object 12 corresponds to a region of the three-dimensional model where the temperature is not mapped. The internal temperature is estimated by a different method depending on the following conditions.

[0171] (A) When a three-dimensional model of the object 12 to be cooked is first constructed. In the initial state after the object 12 is placed in the cookware 11 and heating begins, the internal temperature of the object 12 is assumed to be uniform. In other words, the internal temperature is considered to be close to the surface temperature measured by the temperature sensor 31.

[0172] In this case, the internal temperature estimation unit 106 calculates the average value of the surface temperature of the object 12 and maps it as the internal temperature in the voxels corresponding to the inside of the object 12. This corresponds to the initial condition for the heat conduction analysis.

[0173] In this way, when a three-dimensional model of the object 12 is first constructed, a representative value of the surface temperature of the object 12 is estimated as the internal temperature. The representative value of the surface temperature of the object 12 includes a value determined based on the surface temperature of the object 12, such as the average or median of the surface temperature of the object 12.

[0174] (B) When a change in the position, posture, or shape of the object 12 to be cooked is detected If a change in the position, posture, or shape of the object 12 is recognized in step S4, a three-dimensional model is reconstructed.

[0175] FIG. 16 is a cross-sectional view showing an example of reconstruction of a three-dimensional model.

[0176] As shown in the upper part of Fig. 16, the posture and shape of the object 12 change when the object 12 is turned upside down, for example. As shown on the left side of the upper part of Fig. 16, before the posture and shape change, the bottom temperature of the object 12 is high and the surface temperature is low. On the other hand, as shown on the right side, after the posture and shape change, the surface temperature of the object 12 is high and the bottom temperature is low.

[0177] In response to changes in the posture and shape of the object 12 to be cooked, a three-dimensional model is reconstructed as shown in the lower part of FIG.

[0178] If the posture and shape of the object to be cooked 12 change, the creation of voxels for the object to be cooked 12 and the extraction of temperatures at temperature definition points on the surface portion of the three-dimensional model indicated by dots are performed in the same manner as in steps S2 and S3.

[0179] The bottom surface temperature when the posture and shape of the object 12 to be cooked changes is identified by the process in step S61 and mapped to the voxels indicated by dots.

[0180] Of the voxels that make up the three-dimensional model, ideally, for voxels other than the surface and heated parts, the temperature distribution estimated before reconstruction should be reproduced in order to continue the heat conduction analysis.

[0181] However, it is generally difficult to identify how the posture and shape of the object 12 have changed before and after the cook's cooking work. Therefore, it is not easy to associate the temperature distribution of the three-dimensional model on a voxel-by-voxel basis and map it from the pre-reconstruction three-dimensional model to the reconstructed three-dimensional model.

[0182] Therefore, the temperatures of the temperature definition points of the three-dimensional model corresponding to the inside of the object 12 are mapped by the following method.

[0183] First, the internal temperature estimation unit 106 estimates the internal energy U of the object 12 based on the temperature distribution before reconstruction. all (the total amount of heat held by the object 12 to be cooked) is calculated based on the following formula (9).

[0184]

number

[0185] c represents the specific heat, and T i represents the temperature. T0 represents the reference temperature. The subscript i represents the temperature definition point. Equation (9) calculates the sum of the internal energy for all temperature definition points of the 3D model before reconstruction.

[0186] Although the specific heat c is generally not uniform throughout the food material and is temperature dependent, it is treated as a constant here. If an accurate value of the specific heat c, including its dependence on the part of the food material and temperature, is obtained in step S5, the calculation may be performed using the accurate value of the specific heat c rather than treating it as a constant.

[0187] Next, the internal temperature estimation unit 106 calculates the internal energy U bound Ask for.

[0188]

number

[0189] The subscript j represents the temperature definition point. Equation (10) calculates the sum of the internal energies for the temperature definition points of the surface area where the temperature is extracted and the heated area in the reconstructed 3D model.

[0190] The internal temperature estimation unit 106 calculates the total number of temperature definition points whose temperatures are not specified in the reconstructed three-dimensional model as N bulk Then, the temperature T bulk Ask for.

[0191]

number

[0192] The internal temperature estimation unit 106 calculates the temperature T as the temperature value of the temperature definition point whose temperature is not specified in the reconstructed three-dimensional model. bulk Map the following.

[0193] In this way, when a change in the position, posture, or shape of the object 12 to be cooked is recognized, the three-dimensional model is reconstructed so that the sum of the internal energies before and after the reconstruction is preserved, and the temperature T bulk is estimated as the internal temperature. This makes it possible to substantially maintain the accuracy of the internal temperature estimation before and after the reconstruction of the three-dimensional model.

[0194] (C) Other times Using the three-dimensional model constructed by the above method (A) or (B), the internal temperature estimation unit 106 performs a heat conduction analysis by numerical analysis such as the finite element method. The heat conduction model, which is a mathematical model of heat conduction, is expressed by a three-dimensional unsteady heat conduction equation such as the following equation (12).

[0195]

number

[0196] κ represents the thermal diffusion coefficient, and t represents time. T(x, y, z, t) represents the temperature of the object 12 to be cooked, expressed as a function of time and space. Hereinafter, the arguments x, y, and z representing spatial coordinates will be omitted. The heat conduction model is as shown in Figure 17.

[0197] The temperature distribution T(0) at time t=0 is given as the initial condition of the unsteady heat conduction equation. Time t=0 is the time when the object 12 to be cooked is placed in the cooking utensil 11. In the above method (A), the temperature distribution mapped onto the three-dimensional model corresponds to the temperature distribution T(0) of the initial condition. Note that the reconstruction of the three-dimensional model performed in the above method (B) is performed at time t>0, but this essentially means resetting the initial conditions.

[0198] As a boundary condition at time t>0, the temperature distribution T of the surface temperature measured by the temperature sensor 31 is top (t) and the temperature distribution T of the bottom surface estimated in step S61 bottom (t) is given.

[0199] The temperature inside the object 12 to be cooked, which is not restricted by boundary conditions, can be calculated by numerical calculation based on the governing equation obtained by discretizing Equation (12).

[0200] Incidentally, in order to appropriately control the finished quality of the object 12, it is important to predict the change in the internal temperature of the object 12 during the residual heat process after heating has stopped. In particular, when the volume of the object 12 is large, the central temperature of the object 12 changes significantly during the residual heat process.

[0201] The internal temperature estimation unit 106 predicts the temperature change of the object 12 during the preheating process by applying the above-mentioned heat conduction model. For example, at time t=t stop When heating is stopped at time t=t stop The temperature distribution of the surface temperature T top (t) and temperature distribution T of bottom surface temperature bottom The time change of (t) is predicted and used as a boundary condition, and a similar numerical analysis is carried out in advance of real time.

[0202] First, the temperature distribution T top As described above, the temperature of the exposed surface of the object 12 is constantly measured by the temperature sensor 31. top(t) is predicted based on the time series data of the measured values ​​until heating is stopped.

[0203] 18 is a diagram showing an example of the measurement results of the temperature change of the exposed surface after a steak being cooked in a frying pan is turned over. The vertical axis represents the temperature of the exposed surface, and the horizontal axis represents the elapsed time.

[0204] As shown in Figure 18, the temperature of the exposed surface after being turned over continues to decrease monotonically. As the temperature of the exposed surface decreases and approaches room temperature, the temperature change becomes a gentle linear shape. Therefore, the slope of the temperature change at the time when heating is stopped can be calculated, and the temperature distribution T after heating is stopped can be calculated by simple extrapolation. top If a method for predicting the temperature change of (t) is used, the temperature distribution T top (t) is obtained.

[0205] Next, the temperature distribution T bottom Consider how to predict (t). Temperature distribution T bottom The prediction of (t) is made based on the predicted temperature change of the heating medium after heating is stopped.

[0206] 19 is a diagram showing an example of the measurement results of the temperature change of the frying pan after heating has stopped. The vertical axis represents the temperature of the frying pan, and the horizontal axis represents the elapsed time.

[0207] As shown in Figure 19, the temperature of the frying pan continues to decrease monotonically after heating is stopped. As the temperature of the frying pan decreases and approaches room temperature, the temperature change becomes a gentle linear pattern. However, the rate at which the temperature decreases varies depending on the heat conduction characteristics of the heating medium, such as the frying pan. The rate at which the temperature decreases is primarily affected by the heat capacity.

[0208] Therefore, to accurately predict the temperature change of the heating medium, the thermal conductivity characteristics of the cooking utensil 11 that is actually used are required. The thermal conductivity characteristics of the cooking utensil 11 are calculated based on physical properties related to heat conduction, such as the material, specific heat, volume, and surface area of ​​the cooking utensil 11. A method of estimating the thermal conductivity characteristics of the cooking utensil 11 that is actually used and predicting the temperature change of the heating medium based on the thermal conductivity characteristics is not a practical method because it is not versatile.

[0209] In order to predict the temperature change of the heating medium, for example, an effective method is to perform calibration as an initial setting for the cooking utensil 11 that will actually be used.

[0210] For example, a cook places only the cooking utensils 11 that they will actually use on the stove, heats them until they reach a sufficiently high temperature, then turns off the heat and leaves them to cool. The temperature of the cooking utensils 11 during the cooling process is measured by the temperature sensor 31, and a curve showing the temperature change is obtained, as shown in Figure 19. The room temperature is also measured along with the temperature of the cooking utensils 11.

[0211] The internal temperature estimation unit 106 stores the slope of the temperature decrease of the cooking utensil 11, which is determined depending on the difference between the temperature of the cooking utensil 11 and the room temperature, as a characteristic value of the cooking utensil 11. After calibration, the internal temperature estimation unit 106 can predict the temperature change of the cooking utensil 11 based on the measured value of the temperature of the cooking utensil 11 and the measured value of the room temperature when heating is stopped during the cooking process.

[0212] The internal temperature estimation unit 106 calculates the temperature distribution T based on the temperature change of the cooking utensil 11 and the above-mentioned equation (8). bottom The temperature change of (t) can be predicted.

[0213] After the internal temperature of the object 12 is estimated in step S62, the process returns to step S6 in FIG. 4, and the subsequent processes are carried out.

[0214] <Effector Control> The control of effector unit 23 performed in step S7 of Fig. 4 will be described in detail. The content of the control of effector unit 23 leads to the provision of value to the cook. Examples of typical applications related to the control of effector unit 23 will be described below.

[0215] (A) Information on heating status The effector control unit 107 communicates with the information terminal 7 in Fig. 1 and presents the cook with the heating state of the object 12. For example, the display of the information terminal 7 displays information that visualizes the temperature distribution of the object 12 and the cooking utensil 11.

[0216] For example, the effector control unit 107 displays the surface temperature and internal temperature of the object 12 to be cooked in real time from any viewpoint based on a three-dimensional model. The surface temperature and internal temperature are displayed using CG in the same way as the results of a heat conduction analysis are displayed on a screen using a CAE (Computer Aided Engineering) tool installed on a PC.

[0217] There is a strong need to visualize not only the temperature distribution at a certain time but also the heat flow (temperature gradient over time). Heat flow can be visualized in the same way that vector fields are visualized with CAE tools. By displaying the current temperature distribution as well as the rate at which heat passes through the food, the cook can predict changes in the object 12 to be cooked. The cooking assistance system assists the cook by enabling appropriate heating control to achieve the ideal result.

[0218] Although automatic adjustment functions such as those described in (B) below are expected to evolve in the future, there are many situations where human skill is advantageous. Cooking in the work area may be carried out under the judgment and guidance of a skilled chef who is in a remote location and views the information displayed on the information terminal 7. The information terminal 7 that constitutes the cooking assistance system is connected to the information processing device 22 via an intranet or the Internet.

[0219] (B) Automatic adjustment of heat The effector control unit 107 communicates with the heating device 1 in Fig. 1 and automatically adjusts the heating power. To ensure uniform heating, it is desirable to maintain a constant temperature of the heating medium depending on the cooking contents. For example, the effector control unit 107 adjusts the heating power of the heating device 1 by feedback control depending on the temperature of the heating medium extracted in step S3.

[0220] As will be described later, a function that automatically stops heating at a timing when it is expected that the core temperature of the object 12 to be cooked will reach the target temperature by simulating the residual heat process is also useful.

[0221] (C) Air conditioning control Environmental conditions such as temperature and humidity in the cooking environment can have a significant impact on the final dish. When cooking meat or fish, the preparatory step is to bring the ingredients back to room temperature. If the temperature of the ingredients is uneven when the cooking process begins, it will directly lead to uneven cooking, so it is important to maintain a constant room temperature (room temperature).

[0222] Even during the preheating process, the core temperature changes differently depending on the temperature of the place where the meat is placed. In order to accurately control the heating conditions in pursuit of cooking reproducibility, it seems that air conditioning control using a dedicated device specialized for the cooking environment would be effective.

[0223] The effector control unit 107 controls the operation settings of the air conditioner 8 as feedback control based on the temperature information measured by the thermographic camera 3 so that the conditions around the object 12 to be cooked are appropriate.

[0224] It is also possible to use data measured by auxiliary sensors including sensors built into other devices other than the basic sensor group, such as the heating device 1. For example, a thermometer or a hygrometer installed in a position where it can measure the vicinity of the object 12 to be cooked is provided as the auxiliary sensor.

[0225] An aroma sensor may be provided as an auxiliary sensor. When cooking food in which aroma is important, it is essential to remove unnecessary unpleasant odors from the environment by air conditioning in order to control the finished product.

[0226] As described above, the effector control unit 107 controls at least one of the heating device 1, the information terminal 7, and the air conditioner 8, for example.

[0227] (D) Storage of recognition data The sensor data measured by the sensor unit 21 and the information estimated by the information processing device 22 may be stored in the storage unit 42 for post-analysis. In other words, the use of the estimation results by the information processing device 22 is not limited to real-time heating control. The physical configuration of the storage unit 42 is arbitrary, such as being included in the processor module 4 or being provided in the server 6.

[0228] It is effective if the sensor unit 21 has a temperature sensor that can accurately measure the internal temperature of the object 12 to be cooked. For example, a cooking thermometer using a thermocouple probe is provided as a component of the cookware 11. The information processing device 22 acquires data measured by the cooking thermometer and stores it together with other sensor data in association with the estimation result information. Referencing this information as ground truth information for estimating the internal temperature is useful for developing technical methods to improve the accuracy of internal temperature estimation.

[0229] In addition, storing the RGB image acquired by the image sensor 33 together with the estimation results of the thermal conductivity properties of the object to be cooked 12 by the thermal conductivity property estimation unit 105 is useful for verifying the validity of the estimation results and is useful for technological development to improve accuracy.

[0230] Through the above processing, in the cooking assistance system, the temperature of the heated portion of the object to be cooked 12, which is important for heat conduction analysis, can be accurately determined based on the thermal image acquired by the temperature sensor 31 without destroying the object to be cooked 12.

[0231] By using the accurately determined temperature of the heated portion to estimate the internal temperature, the cooking assistance system can estimate the internal temperature with high accuracy.

[0232] In addition, in the cooking assistance system, the exposure of the heated portion of the object to be cooked 12 is recognized based on sensor data acquired by the sensor unit 21, and the contact thermal resistance between the heated portion and the heating medium is estimated based on a thermal image acquired immediately after the exposure.

[0233] By using the contact thermal resistance to estimate the temperature of the heated portion, the cooking assistance system can improve the accuracy of estimating the temperature of the heated portion and estimate the internal temperature with high accuracy.

[0234] In the cooking assistance system, a three-dimensional model of the object 12 to be cooked is constructed based on sensor data acquired by the distance sensor 32, and information on the boundary conditions and temperature that serve as the initial conditions for the heat conduction analysis is mapped onto the three-dimensional model.

[0235] The cooking assistance system can estimate the internal temperature with high accuracy by performing heat conduction analysis based on a three-dimensional model that reflects the actual shape of the object 12. The cooking assistance system can also use the same three-dimensional model to simulate temperature changes after heating is stopped.

[0236] <<4. Other>> Computer configuration example The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0237] FIG. 20 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program.

[0238] A CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, and a RAM (Random Access Memory) 203 are interconnected by a bus 204.

[0239] An input / output interface 205 is also connected to the bus 204. An input unit 206 including a keyboard, a mouse, etc., and an output unit 207 including a display, a speaker, etc. are connected to the input / output interface 205. In addition, a storage unit 208 including a hard disk, a nonvolatile memory, etc., a communication unit 209 including a network interface, etc., and a drive 210 that drives removable media 211 are also connected to the input / output interface 205.

[0240] In the computer configured as above, the CPU 201 loads a program stored in the storage unit 208 into the RAM 203 via the input / output interface 205 and the bus 204 and executes the program, thereby performing the above-described series of processes.

[0241] The program executed by the CPU 201 is installed in the storage unit 208 by being recorded on, for example, a removable medium 211 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0242] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0243] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0244] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0245] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0246] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0247] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0248] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0249] Configuration combination examples The present technology can also be configured as follows.

[0250] (1) a construction unit that constructs a three-dimensional model representing the shape and temperature distribution of the object to be cooked based on sensor data acquired by a sensor that measures the state of the cooking utensil and the object to be cooked; an internal temperature estimation unit that estimates the internal temperature of the object to be cooked by performing a heat conduction analysis based on the three-dimensional model; An information processing device comprising: (2) The construction unit constructs the three-dimensional model when the object to be cooked is placed in the cooking utensil. The information processing device according to (1) above. (3) The construction unit reconstructs the three-dimensional model when any of the shape, volume, and orientation of the object to be cooked changes during the cooking process. The information processing device according to (1) or (2). (4) The cooking method further includes a heat conduction characteristic estimation unit that estimates the heat conduction characteristics of the object to be cooked based on the sensor data, which is used in the heat conduction analysis. The information processing device according to any one of (1) to (3). (5) The thermal conduction properties include thermal conductivity, specific heat, density, and thermal diffusivity. The information processing device according to (4) above. (6) The heat conduction characteristic estimation unit repeatedly updates the heat conduction characteristics during the cooking process. The information processing device according to (4) or (5). (7) The cooking apparatus further includes an extracting unit that extracts the surface temperature of the object to be cooked and the temperature of the heating medium based on the sensor data. The information processing device according to any one of (4) to (6). (8) The extraction unit extracts the temperature of a region of the entire heating medium near the object to be cooked as the temperature of the heating medium. The information processing device according to (7) above. (9) The internal temperature estimation unit sets the temperature of the heating medium as the temperature of the heated portion of the object to be cooked in the three-dimensional model, and estimates the internal temperature of the object to be cooked. The information processing device according to (7) or (8). (10) When the posture of the object to be cooked changes, the heat conduction characteristic estimation unit estimates a contact thermal resistance occurring between the object to be cooked and the heating medium based on a surface temperature of the object to be cooked and a temperature of the heating medium after the posture change; The internal temperature estimation unit estimates the internal temperature of the object to be cooked using the contact thermal resistance. The information processing device according to (7) or (8). (11) The internal temperature estimation unit sets a temperature calculated based on the contact thermal resistance, the surface temperature of the object to be cooked, and the temperature of the heating medium in the three-dimensional model as the temperature of the heated portion of the object to be cooked, and estimates the internal temperature of the object to be cooked. The information processing device according to (10) above. (12) The cooking apparatus further includes a recognition unit that recognizes that the object has been turned over as a change in the position of the object to be cooked when the change in the surface temperature of the object to be cooked is greater than a threshold value and the difference between the temperature of the heating medium and the surface temperature of the object to be cooked after the change is smaller than a threshold value. The information processing device according to (10) or (11). (13) The internal temperature estimation unit estimates a representative value of the surface temperature of the object to be cooked as the internal temperature of the object to be cooked when the three-dimensional model is first constructed. The information processing device according to any one of (1) to (12). (14) The internal temperature estimation unit estimates the internal temperature of the object to be cooked based on the internal energy of the three-dimensional model before reconstruction when reconstructing the three-dimensional model. The information processing device according to any one of (1) to (13). (15) The internal temperature estimation unit estimates the internal temperature of the object to be cooked based on the thermal diffusion coefficient and a heat conduction equation expressed by a function representing the temperature at each position on the three-dimensional model. The information processing device according to any one of (5) to (12). (16) The cooking apparatus further includes a control unit that controls peripheral devices based on the result of estimating the temperature inside the object to be cooked. The information processing device according to any one of (1) to (15). (17) The control unit controls at least one of a heating device that heats the object to be cooked, an information terminal that displays the heating state of the object to be cooked, and an air conditioning device installed in a space where the object to be cooked is cooked. The information processing device according to (16) above. (18) A three-dimensional model is constructed based on sensor data acquired by a sensor that measures the state of the cooking utensil and the object to be cooked, which represents the shape and temperature distribution of the object to be cooked; The temperature inside the object to be cooked is estimated by performing a heat conduction analysis based on the three-dimensional model. Information processing methods. (19) On the computer, A three-dimensional model is constructed based on sensor data acquired by a sensor that measures the state of the cooking utensil and the object to be cooked, which represents the shape and temperature distribution of the object to be cooked; The temperature inside the object to be cooked is estimated by performing a heat conduction analysis based on the three-dimensional model. A program for executing a process. [Explanation of symbols]

[0251] 1 heating device, 2 stereo camera, 3 thermography camera, 4 processor module, 5 network equipment, 6 server, 7 information terminal, 8 air conditioner, 21 sensor unit, 22 information processing device, 23 effector unit, 31 temperature sensor, 32 distance sensor, 33 image sensor, 41 calculation unit, 42 memory unit, 51 UI device, 101 sensor data input unit, 102 position and shape recognition unit, 103 surface temperature extraction unit, 104 process status recognition unit, 105 heat conduction property estimation unit, 106 internal temperature estimation unit, 107 effector control unit

Claims

1. a construction unit that constructs a three-dimensional model representing at least the shape and temperature distribution of the surface of the object to be cooked other than the exposed surface whose state can be measured by the sensor, based on sensor data acquired by a sensor that measures the state of the cooking utensil and the object to be cooked; an internal temperature estimation unit that estimates the internal temperature of the object to be cooked by performing a heat conduction analysis based on the three-dimensional model; An information processing device comprising:

2. The construction unit constructs the three-dimensional model when the object to be cooked is placed in the cooking utensil. The information processing device according to claim 1 .

3. The construction unit reconstructs the three-dimensional model when any of the shape, volume, and orientation of the object to be cooked changes during the cooking process.

3. The information processing device according to claim 1.

4. The cooking method further includes a heat conduction characteristic estimation unit that estimates the heat conduction characteristics of the object to be cooked based on the sensor data, which is used in the heat conduction analysis.

4. The information processing device according to claim 1.

5. The thermal conduction properties include thermal conductivity, specific heat, density, and thermal diffusivity. The information processing device according to claim 4 .

6. The heat conduction characteristic estimation unit repeatedly updates the heat conduction characteristics during the cooking process.

6. The information processing device according to claim 4.

7. The cooking apparatus further includes an extracting unit that extracts the surface temperature of the object to be cooked and the temperature of the heating medium based on the sensor data.

7. The information processing device according to claim 4.

8. The extraction unit extracts the temperature of a region of the entire heating medium near the object to be cooked as the temperature of the heating medium. The information processing device according to claim 7 .

9. The internal temperature estimation unit sets the temperature of the heating medium as the temperature of the heated portion of the object to be cooked in the three-dimensional model, and estimates the internal temperature of the object to be cooked.

9. The information processing device according to claim 7 or 8.

10. When the posture of the object to be cooked changes, the heat conduction characteristic estimation unit estimates a contact thermal resistance occurring between the object to be cooked and the heating medium based on a surface temperature of the object to be cooked and a temperature of the heating medium after the posture change; The internal temperature estimation unit estimates the internal temperature of the object to be cooked using the contact thermal resistance.

9. The information processing device according to claim 7 or 8.

11. The internal temperature estimation unit sets a temperature calculated based on the contact thermal resistance, the surface temperature of the object to be cooked, and the temperature of the heating medium in the three-dimensional model as the temperature of the heated portion of the object to be cooked, and estimates the internal temperature of the object to be cooked. The information processing device according to claim 10.

12. The cooking apparatus further includes a recognition unit that recognizes that the object has been turned over as a change in the position of the object to be cooked when the change in the surface temperature of the object to be cooked is greater than a threshold value and the difference between the temperature of the heating medium and the surface temperature of the object to be cooked after the change is smaller than a threshold value. The information processing device according to claim 10 or 11.

13. The internal temperature estimation unit estimates a representative value of the surface temperature of the object to be cooked as the internal temperature of the object to be cooked when the three-dimensional model is first constructed. The information processing device according to claim 1 .

14. The internal temperature estimation unit estimates the internal temperature of the object to be cooked based on the internal energy of the three-dimensional model before reconstruction when reconstructing the three-dimensional model.

14. The information processing device according to claim 1.

15. The internal temperature estimation unit estimates the internal temperature of the object to be cooked based on a thermal diffusion coefficient as the thermal conductivity characteristic and a heat conduction equation expressed by a function representing the temperature at each position on the three-dimensional model.

13. The information processing device according to claim 5.

16. The cooking apparatus further includes a control unit that controls peripheral devices based on the result of estimating the temperature inside the object to be cooked.

16. The information processing device according to claim 1.

17. The control unit controls at least one of a heating device that heats the object to be cooked, an information terminal that displays the heating state of the object to be cooked, and an air conditioning device installed in a space where the object to be cooked is cooked. The information processing device according to claim 16.

18. Based on sensor data acquired by a sensor that measures the state of the cooking utensil and the object to be cooked, a three-dimensional model is constructed that represents at least the shape and temperature distribution of the surface of the object to be cooked other than the exposed surface whose state can be measured by the sensor; The temperature inside the object to be cooked is estimated by performing a heat conduction analysis based on the three-dimensional model. Information processing methods.

19. On the computer, Based on sensor data acquired by a sensor that measures the state of the cooking utensil and the object to be cooked, a three-dimensional model is constructed that represents at least the shape and temperature distribution of the surface of the object to be cooked other than the exposed surface whose state can be measured by the sensor; The temperature inside the object to be cooked is estimated by performing a heat conduction analysis based on the three-dimensional model. A program for executing a process.

Images