Apparatus and method for controlling a cooking process using a visual cooking guide

Abstract

The present invention relates to an apparatus and method for controlling a cooking process using a visual cooking guide. A cooking appliance according to one embodiment of the present invention is a cooking appliance for cooking food ingredients, and includes a main body for heating the food ingredients, a camera for photographing the food ingredients stored in the main body, and a control module for performing a food ingredient recognition process on the image of the food ingredients photographed by the camera and outputting image information corresponding to the recognized food ingredients to a display module of the cooking appliance, or transmitting the image information to a terminal device linked to the cooking appliance, and then setting cooking conditions of the main body according to a time point or image selected from the image information, or monitoring the cooking status of the main body and performing an alarm function.

Description

[Technical Field]

[0001] The present invention relates to a device and method for controlling a cooking process using a visual cooking guide. [Background technology]

[0002] An oven is a device that can cook a variety of foods, and users can cook various ingredients in the oven. However, cooking methods and temperatures vary depending on the ingredients, and there are limitations to how well a user can master all of them.

[0003] For example, different temperatures and times may be set for different types of food ingredients, such as meat, fish, and vegetables, and different types of dishes. Therefore, oven users need a method for providing a cooking guide suitable for each ingredient and controlling the oven accordingly. Summary of the Invention [Problem to be solved by the invention]

[0004] This specification aims to solve the above-mentioned problems by allowing the user to select the cooking state suitable for the food ingredients using an image or video, and the cooking appliance will cook accordingly or sound an alarm when the cooking state is reached, thereby improving cooking efficiency.

[0005] In addition, the present invention allows the user to check the previous cooking process on video, select the optimal cooking state, and receive an alarm.

[0006] The objects of the present invention are not limited to those mentioned above, and other objects and advantages of the present invention not mentioned above can be understood from the following description and can be more clearly understood from the examples of the present invention. Furthermore, it can be easily understood that the objects and advantages of the present invention can be achieved by the means and combinations thereof set forth in the claims. [Means for solving the problem]

[0007] A cooking appliance according to one embodiment of the present invention is a cooking appliance for cooking food ingredients, and includes (comprises; constitutes; constructs; sets; encompasses; contains; has); a main body for heating the food ingredients; a camera for photographing the food ingredients stored in the main body; and a control module for performing a food ingredient recognition process on the image of the food ingredient photographed by the camera and outputting image information corresponding to the recognized food ingredient to a display module of the cooking appliance, or transmitting the image information to a terminal device linked to the cooking appliance, and then setting cooking conditions of the main body according to a time point or image selected from the image information, or monitoring the cooking status of the main body and performing an alarm function.

[0008] A method for controlling a cooking process using a visual cooking guide according to one embodiment of the present invention includes the steps of: using a camera of a cooking appliance to capture images of food ingredients placed in the cooking appliance main body; a control module of the cooking appliance performing an ingredient recognition process on the image of the food ingredients captured by the camera and retrieving video information corresponding to the recognized food ingredients; the control module outputting the video information to a display module of the cooking appliance or transmitting the video information to a terminal device linked to the cooking appliance; and the control module setting cooking conditions of the main body or monitoring the cooking status of the main body and performing an alarm function in response to a time point or image selected from the video information.

[0009] According to another embodiment of the present invention, a method for controlling a cooking process using a visual cooking guide includes the steps of: using a camera of the cooking appliance to capture images of food ingredients placed in the cooking appliance; a control module of the cooking appliance transmitting the images of the food ingredients captured by the camera to a server; the server performing an ingredient recognition process using the transmitted images and retrieving video information corresponding to the recognized food ingredients and transmitting the video information to the control module; the control module transmitting information regarding a specific time point or image in the video information to the server and then controlling the cooking appliance to perform cooking; the control module transmitting images captured during the cooking process of the cooking appliance to the server; and the server generating a message for pausing cooking, issuing an alarm, or providing guidance on the cooking status using the received images and transmitting the message to the control module; and the control module controlling cooking, performing an alarm function, or outputting a guidance message on a display module of the cooking appliance according to the received message. [Effects of the Invention]

[0010] When applying the present invention, a user can select the cooking state suitable for the food ingredients using an image or video, and the cooking appliance will cook accordingly or sound an alarm when the cooking state is reached, thereby improving cooking efficiency.

[0011] In addition, when the present invention is applied, the user can check the previous cooking process on video, select the optimal cooking state, and receive an alarm.

[0012] The effects of the present invention are not limited to those described above, and various other effects of the present invention can be easily achieved in the configuration of the present invention. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 illustrates components of a cooking appliance including a display module according to one embodiment of the present invention. [Figure 2] FIG. 10 illustrates components of a cooking appliance not including a display module according to another embodiment of the present invention. [Figure 3] 1 is a diagram showing the configuration of a cooking appliance according to an embodiment of the present invention; [Figure 4] 1 is a diagram illustrating a process of controlling cooking according to an embodiment of the present invention. [Figure 5] 1 is a diagram illustrating a process for controlling a cooking process according to an embodiment of the present invention. [Figure 6] 10A and 10B are diagrams illustrating a process of buffering and comparing images according to an embodiment of the present invention; [Figure 7] 10A and 10B are diagrams illustrating an image comparison process according to an embodiment of the present invention; [Figure 8] 10 is a diagram illustrating a user selecting a specific time point in a time lapse image according to an embodiment of the present invention. [Figure 9] 10 is a diagram showing the results of food ingredient recognition displayed on a display module according to an embodiment of the present invention. FIG. [Figure 10] 10 is a diagram illustrating a process in which a cooking appliance according to an embodiment of the present invention provides an alarm or a cooking-related video to a terminal device via a server. [Figure 11] FIG. 10 is a diagram showing a server according to an embodiment of the present invention recognizing ingredients and comparing them with existing cooking videos or images in a database to check whether cooking conditions are met. DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily understand and practice the present invention. The present invention may be embodied in various different forms and is not limited to the embodiments set forth herein.

[0015] In order to clearly explain the present invention, parts that are not relevant to the description will be omitted, and the same reference numerals will be used throughout the specification to refer to identical or similar components. Some embodiments of the present invention will be described in detail with reference to the drawings. When referring to components in each drawing, the same reference numerals may be used as much as possible, even if the same components are displayed in different drawings. Furthermore, when describing the present invention, if a detailed description of related known structures or functions is deemed to obscure the gist of the present invention, such detailed description may be omitted.

[0016] In describing components of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are used to distinguish the component from other components, and do not limit the nature, order, sequence, or number of the components. When a component is described as being "coupled," "coupled," or "connected" to another component, it should be understood that the component may be directly coupled or connected to the other component, but that other components may be "intervening" between the components, or that each component may be "coupled," "coupled," or "connected" via other components.

[0017] Furthermore, when implementing the present invention, the components may be subdivided for ease of explanation, but these components may be embodied within a single device or module, or one component may be embodied across multiple devices or modules.

[0018] An example of a cooking appliance in this specification includes an oven, which is a home appliance that heats and cooks various food ingredients. Examples of ovens include various home appliances such as a lightwave oven, a microwave oven, and an air fryer.

[0019] The cooking appliance in this specification includes a camera that captures images of food ingredients stored inside. It may also optionally include a display that outputs the images. If the cooking appliance does not have a display, the images can be output on a display of a terminal device linked to the cooking appliance.

[0020] FIG. 1 is a diagram illustrating components of a cooking appliance including a display module according to one embodiment of the present invention.

[0021] The cooking appliance 100 includes a camera 110, a display module 120, a control module 150, and a main body 190. FIG. 1 is created with an emphasis on the configuration of the cooking appliance 100, which photographs food ingredients and processes the corresponding information, and includes the components necessary for the cooking appliance 100 to cook food. As an example of the components included in the cooking appliance 100, the components necessary for heating food, such as a door, a heating unit, and a lamp, are all included in the main body 190. The operation of the main body 190 is controlled by the control module 150.

[0022] The camera 110 is disposed inside the main body and captures the area where cooking is performed. The camera 110 may be disposed on the ceiling or side of the main body. One or more cameras 110 may be disposed, and when two or more cameras 110 are disposed, they can capture images of the food ingredients placed inside the main body at different positions. The camera 110 can capture images or videos of the cooking process of the food ingredients at regular time intervals. The camera 110 can also capture time-lapse videos.

[0023] The display module 120 is disposed outside the main body 190 and can visually output various information. The display module 120 may further include a touch module that can receive touch input, allowing a user to control the operation of the cooking appliance 100 by touching an image displayed on the display module 120.

[0024] The control module 150 is responsible for controlling the cooking appliance 100. When food ingredients are added to the main body 190, the control module 150 can control the camera 110 to take a picture of the added food ingredients. The control module 150 can also calculate the size and weight information of the photographed food ingredients. As an example, a weight sensor may be disposed in the main body 190, and the control module 150 can calculate the weight of the food ingredients using a value sensed by the weight sensor.

[0025] The control module 150 can also communicate with a server or a terminal device, and can therefore send and receive data using a specific communication protocol (e.g., LTE-A, 5G communication, Wi-Fi, Bluetooth, ZigBee, etc.).

[0026] The control module 150 may include a predetermined storage medium, or the control module 150 may include a database using the storage medium. Alternatively, the cooking appliance 100 may include a separate database. Alternatively, the database may be included in an external server, and the control module 150 may communicate with the server to check information stored in the server's database.

[0027] In addition, the information output to the display module 120 can be transmitted by the control module 150 to a server or a terminal device, and the display of the terminal device can output the information.

[0028] 2 is a diagram showing components of a cooking appliance without a display module according to another embodiment of the present invention. The description of the same components is superseded by the description of FIG. 1. This configuration is the same as that of FIG. 1, but without the display module. The control module 150 can transmit information about ingredients placed in the main body 190, information about the cooking status, etc., to a server or a terminal device, and can receive a message requesting that cooking be stopped or resumed.

[0029] The cooking appliance 100 shown in Figures 1 and 2 is used to cook food ingredients and includes a main body 190 that heats the food ingredients, a camera 110 that takes pictures of the food ingredients stored in the main body 190, and a control module 150.

[0030] The control module 150 performs an ingredient recognition process on the food ingredient images captured by the camera 110 and searches for video information corresponding to the recognized ingredients. The control module 150 then outputs the video information to the display module 120 of the cooking appliance 100 or transmits the video information to a terminal device (600 in FIG. 10) linked to the cooking appliance 100. The control module 150 then sets cooking conditions for the main body 190 according to the time or image selected by the user from the video information, or monitors the cooking status of the main body 190 and performs an alarm function.

[0031] Fig. 3 is a diagram showing the configuration of a cooking appliance according to one embodiment of the present invention. Fig. 3 corresponds to one embodiment, and the cooking appliance of the present invention may have a different shape or include different components from those shown in Fig. 3.

[0032] The main body includes an operating unit 21, doors 11 and 12, and cooking chambers 15 and 16 that are opened and closed by the doors 11 and 12.

[0033] A display module 120 is disposed on an operation unit 21 that operates the cooking appliance 100. A user can check the cooking guide through the display module 120. A camera 110 is disposed on the top end of the cooking appliance 100 and can capture images of the inside of the cooking chambers. The cooking chambers 15 and 16 may be opened and closed by doors 11 and 12. Inside the cooking chambers 15 and 16, shelves and supports for placing food ingredients may be disposed, and lamps for heating may also be disposed. The camera 110 can capture images of the inside of the cooking chambers 15 and 16.

[0034] Although the operation unit 21 and the display module 120 are separated in the drawings, an alternative embodiment in which an interface for operation is displayed on the display module 120 is also included in the embodiments of the present invention. In addition, if the display module 120 is removed from the cooking appliance 100 of Figure 3, it becomes the embodiment of Figure 2.

[0035] Below, we consider a method in which a camera installed in a cooking appliance 100, such as an oven, collects and stores images of food, and the user can directly add or remove items to be recognized, providing cooking guidance and controlling cooking.

[0036] When applying an embodiment of the present invention, cooking setting information (oven operation mode, time, temperature) of the recognized item can be stored in a database, and when the cooking appliance 100 recognizes the cooking object, it can automatically set the cooking environment.

[0037] In addition, a user can select a cooking state for which they want to receive an alarm using a previously recorded cooking video, and then confirm the alarm. Alternatively, the cooking appliance 100 can end the cooking process when the desired cooking state is reached.

[0038] Instead of setting cooking conditions specified by letters or numbers, the user can select the optimum cooking condition based on the image, thereby easily controlling the cooking process of the cooking appliance 100.

[0039] 4 is a diagram illustrating a cooking control process according to an embodiment of the present invention. When a food ingredient with a cooking video associated with it is placed into the cooking appliance 100, 200, the food ingredient is photographed, recognized, and the cooking process is controlled based on the corresponding cooking video or DB image.

[0040] When a user places food ingredients into the cooking appliance 100, 200, the camera 110 captures a photo of the food ingredients, and the control module 150 processes the captured image to recognize the food ingredients (S310). The control module 150 displays a stored cooking video or DB image of the food ingredients through the display module 120 (S330). In the case of a cooking appliance 200 that does not include a display module 120, a terminal device linked to the cooking appliance 200 can display the cooking video or DB image.

[0041] One example of a cooking video is a video or a set of images captured during the cooking process of food ingredients previously recognized by the cooking appliances 100 and 200. These videos or sets of images include information about a specific point in time (e.g., in units of 5, 10, or 30 seconds, or in units of 1, 2, or 3 minutes) and information about the heating temperature and cooking mode set during cooking. Another example of a cooking video is a time-lapse video that records the changes in food ingredients during the cooking process.

[0042] For example, when meat is heated, the camera 110 continuously captures the changes in color and volume of the surface of the meat during the heating process and stores them as a video or a set of images. In this case, information on the time when cooking is performed (cooking time) and information on the temperature at which the meat is heated during the cooking process (cooking temperature) correspond to meta information, and the meta information may be combined with each video or image.

[0043] In addition to the cooking video, other images or videos may be sequentially recorded in the database. For example, images or videos showing the color change of the surface of food ingredients for each type of meat, heating temperature, and heating time may be stored in the database in combination with information such as cooking time and cooking temperature.

[0044] Therefore, the control module 150 displays cooking videos or DB images stored by itself or stored externally, such as on a server, and then allows the user to select a specific time point. For example, the user can select a specific time point from a time-lapse of cooking videos or select a specific image from DB images (S350). These selections may include information about the heating temperature.

[0045] The control module 150 controls the main body 190 to heat and cook the added food ingredients at the corresponding heating temperature (or the heating temperature set automatically / manually), and when the camera 110 captures the food ingredients, it compares the captured image with the image at a specific time point in the time lapse or the DB image (S360). In the comparison process, the control module 150 can refer to an image at a previous time point in the time lapse of the video or an image at a previous time point among the DB images.

[0046] In addition, the image of the food ingredients photographed in step S360 is displayed on the display module 120 or the user's terminal device, allowing the user to check the current cooking status.

[0047] The control module 150 then compares the cooking state of the food ingredients with the image at a specific time point to see if it is similar, and if so, stops cooking or issues an alarm (S380). An example of an alarm is when the cooking appliance 100, 200 outputs a predetermined sound, or when a linked terminal device (600 in FIG. 10) outputs an alarm sound. Another example of an alarm is when text or an image is output to the terminal device (600 in FIG. 10) to notify that the cooking state has reached a specific state (a time point selected by the user).

[0048] If the user then takes a subsequent action, the control module 150 updates the information accordingly (S390). If the cooking appliance 100, 200 is stopped to end cooking, the control module 150's processes in S360 and S380 are valid, and the control module 150 stores the results so that the same decision process can be performed later if similar.

[0049] On the other hand, if the user resumes cooking after S380, the processes in S360 and S380 of the control module 150 may be ineffective or may need to be corrected, and the control module 150 stores the results so that, if similar, it can later perform a comparison and judgment process by reflecting the subsequent corresponding measures.

[0050] The user selects the ingredients and the cooking method for those ingredients, and then selects the menu for the cooking appliances 100 and 200. The user can set the heating temperature, heating mode, time, etc. However, it is troublesome for the user to manually check the optimal cooking conditions for the ingredients, and there are limitations to selecting cooking conditions that are appropriate for the amount of ingredients.

[0051] In addition, a user may open the door of the cooking appliance 100, 200 during cooking to check the cooking process, which may cause heat loss and increase cooking time.

[0052] Therefore, the user selects a cooking video or an image stored in the database that corresponds to the desired cooking state, and the cooking appliances 100 and 200 compare the image of the food ingredients taken during the cooking process with the image selected by the user to determine whether the food ingredients need further heating or provide the user with information to inform them.

[0053] 4, the cooking appliances 100 and 200 can automatically recognize the ingredients and automatically set cooking settings or provide related information to the user. Also, the cooking status information can be checked while the camera is photographing the ingredients.

[0054] Below, we will discuss the process of selecting / adding / deleting food ingredients to be recognized using images stored in the cooking appliance or server, selecting the video of the state for which the user wants to receive an alarm using cooking videos, and receiving an alarm at that state.

[0055] The video information in the above-described embodiments may be, for example, a time-lapse video of the cooking process, a series of images taken at regular time intervals during the cooking process, or a collection of images constructed from the photographs of the food ingredients.

[0056] For example, the video information may be a cooking video previously captured. In this case, the video information may be video information captured and stored by the cooking appliance 100 owned by the user, showing the cooking process of food ingredients previously stored in the main body 190.

[0057] The image information may be generated by another user or a server, or may be a database image received from another device and stored by the cooking appliance 100. In this case, the image information may be, for example, image information stored in the control module 150 of the cooking state of food ingredients received from another device and stored.

[0058] These database images may correspond to cooking guides provided by supermarkets or online shopping malls. For example, supermarkets or online shopping malls may sell specific products and distribute images or videos of how to cook the products. The control module 150 may register the distributed images or videos and select a specific point in time in the image or video that corresponds to the cooking state of the purchased food ingredients.

[0059] FIG. 5 is a diagram illustrating a process for controlling a cooking process according to an embodiment of the present invention.

[0060] When food ingredients are placed in the cooking appliance 100, 200, the camera 110 takes a picture of the food ingredients (S311). The control module 150 recognizes the food ingredients using the captured image. For this reason, the control module 150 can store an initial image of the existing food ingredients.

[0061] If the food ingredients are not recognized (S312), the control module 150 sets R_FLAG to 0 and registers cooking conditions under the control of the user (S313). The cooking conditions include cooking mode, heating temperature, heating time, etc.

[0062] On the other hand, if the food ingredient is recognized in S312, the control module 150 sets R_FLAG to 1 (S315). Video information corresponding to the recognized food ingredient may be displayed on the display module or the terminal device (600 in FIG. 10). If the user has not set a status alarm (S316), the control module 150 automatically sets cooking settings suitable for the recognized food ingredient (S317). The automatically set cooking conditions include the cooking mode, heating temperature, heating time, etc.

[0063] Meanwhile, if the user sets a status alarm in S316, the control module 150 can select whether to receive a status alarm by selecting a specific time point in the stored cooking video or by selecting a desired status from the stored image (S331).

[0064] If the user selects a specific image from the displayed image information corresponding to the recognized food ingredient and selects a method of receiving an alarm if the state of the food ingredient is similar to the selected image (No flow of S331), the control module 150 displays a number of images on the display module 120, and the user selects one or more prototype images from among them (S351).

[0065] On the other hand, if the user selects a specific time point in the stored cooking video and chooses to receive a status alarm (Yes flow in S331), the control module 150 displays the cooking video on the display module 120 and the user selects the specific time point while playing the video (playing the time-lapse video) (S353). The user can also select information such as the temperature and cooking time recorded at each time point while recording the cooking video in the past, as well as the process of selecting the specific time point.

[0066] Then, the control module 150 extracts one or more prototype images for each cooking stage centered around the selected time point (S354).

[0067] The prototype selected / extracted in S351 or S354 can be used as comparison data to check the cooking state of food ingredients later during the cooking process of the food ingredients.

[0068] Then, the control module 150 automatically sets cooking conditions for the main body 190 according to the selected / extracted prototype (S357). At this time, the set cooking conditions are cooking conditions that reflect the cooking time or cooking temperature set in the meta information combined with the video information.

[0069] After the cooking conditions are set in steps S313, S317, and S357, the cooking appliances 100 and 200 start cooking (S321 and S361), and the camera 110 captures images of the food ingredients and buffers the images (S322 and S362). The control module 150 then generates time lapse information corresponding to the captured images (S323 and S363).

[0070] Generating time lapse information includes embodiments in which the information is generated to correspond to the current state of the image of the food ingredients (buffered in the image) and information on the cooking conditions (heated temperature, current oven temperature, time elapsed since cooking began, weight change of the food ingredients, etc.).

[0071] Thereafter, cooking completion or alarm status can be confirmed.

[0072] For example, the flow of S321 to S324 is an embodiment in which a status alarm is not received during the cooking process. In this case, the control module 150 determines whether the end condition according to the cooking conditions set in S313 / S317 has been reached (S324). If the end condition has not been reached, the control module 150 repeats S322 to S324. If the end condition has been reached, the control module 150 ends the cooking operation of the cooking appliance (S385).

[0073] Meanwhile, the flow from S361 to S381 is an embodiment in which a status alarm is received during the cooking process. In this case, the control module 150 compares the status of the food ingredients selected in S351 / S353 with the prototype image and determines whether the cooking status of the food ingredients has reached the set prototype (S364).

[0074] If the food ingredients have not reached the target state, steps S361 to S364 are repeated. If the food ingredients are found to be the same or similar to the prototype image in step S364, the control module 150 issues a visual or audio alarm to the user (step S381) and terminates the cooking operation of the cooking appliance (step S385).

[0075] When the cooking operation is completed, depending on the value of R_FLAG, i.e., whether or not the ingredients have been recognized, if the ingredients have been recognized, the images and time lapse information generated in steps S322 / S323 or S362 / S363 are updated to the previously stored cooking video (S387). Of course, updating can be selectively performed.

[0076] On the other hand, if the ingredients are not recognized (S312-S313), if the user selects whether to register the ingredients and answers "Yes" (S388-Yes), the images and time lapse information generated in steps S322 / S323 are updated to the previously stored cooking video (S387).

[0077] The prototype is an example of images of ingredients at certain stages in the cooking process. For example, images of ingredients when they are first added, after cooking for 1 minute, after cooking for 2 minutes, after 3 minutes before cooking is complete, and after cooking is complete are stored in a database or generated during the cooking video generation process.

[0078] Additionally, images showing cooking states may be extracted from the cooking video at regular time intervals. For example, if the cooking video has a playback time of 10 minutes, images may be extracted every minute, resulting in a total of 11 prototype images, including the first image.

[0079] Using multiple prototypes, the control module 150 can compare the progression of food ingredients with images.

[0080] As described in S388, the control module 150 controls the camera 110 to capture the cooking status of the main body 190, and then, when the cooking of the main body 190 is completed, stores the image information captured by the camera 110, or replaces the previously stored image information with the captured image information, so that a better cooking process can be confirmed later from the image information, and automatic settings and alarm settings for the cooking process can be performed.

[0081] FIG. 6 is a diagram illustrating a process of buffering and comparing images according to an embodiment of the present invention.

[0082] While cooking is taking place, the camera 110 captures and buffers images. For example, the camera 110 may capture K images (x*y*K) at a specific resolution (x*y) (S341). The control module 150 then performs feature extraction from the N images (S342).

[0083] The feature extraction may be three-dimensional, and the control module 150 measures the similarity between the extracted feature and a prototype image previously stored in the database (selected in S351) or an image corresponding to a specific point in time within a time-lapse of an existing stored image (selected in S353) (S343).

[0084] In one embodiment, in the process of measuring the similarity, the control module 150 may extract high indexes based on the z-axis (S344). If the number of extracted indexes is equal to or greater than a threshold, which is a reference value, the control module 150 determines that the state is similar (S345) and issues an alarm (S346). If the determination in S345 corresponds to No, the control module 150 may repeat the process of capturing and comparing images during the cooking process.

[0085] 6, it is assumed that the images have a resolution of 1920*1080. In this case, the control module 150 captures K images at regular time intervals (10 seconds, 20 seconds, 30 seconds, 1 minute, etc.) (S341). Then, the control module 150 can extract 30*30*512 features from these K images with a resolution of 1920*1080 (S342).

[0086] Based on these features, the control module 150 can measure the similarity between the image of a specific cooking step selected by combining it with the background image (S343). At this time, the control module 150 can arrange the extracted features on the z-axis and extract an index based on the z-axis (S344). Then, the control module 150 can compare the similarity with the image of the cooking step for which an alarm was previously set and issue an alarm if the similarity is above a certain standard.

[0087] To issue an alarm at S380 of FIG. 4, S381 of FIG. 5, S346 of FIG. 6, etc., the control module 150 compares an image captured of the cooking of the stored food ingredients with an image at a selected point in time from the image information (a video captured of a previous cooking process or a video in the database). The control module 150 may then halt cooking if the similarity between the image at the selected point in time and the captured image (a video of the food ingredients currently being cooked) is equal to or greater than a reference value. Alternatively, the control module 150 may issue an alarm via the display module 120 or transmit an alarm message to the terminal device (600 of FIG. 10).

[0088] In addition, during the cooking process, the control module 150 may output an image of the food ingredients being cooked in the main body 190 via the display module 120 or transmit the captured image to the terminal device. In addition to the currently cooked state, information on the remaining time until the expected completion time (the time selected by the user through the image information) or information on the cooking progress rate may also be output via the display module 120 or the terminal device (600 in FIG. 10).

[0089] 7 is a diagram illustrating an image comparison process according to an embodiment of the present invention. In this specification, as an embodiment of image recognition, a target class is defined, and a network is configured and trained so that similarity between the same class is high and similarity between different classes is low. When a class is added, a prototype for the class can be added and recognized without retraining the network.

[0090] As shown in FIG. 7, the control module 150 inputs an image into a learning network and extracts a feature vector (S401, S402).

[0091] The control module 150 can extract the similarity between the extracted feature vector and each prototype (S403). Here, the prototype can be determined by the class of each food ingredient (broccoli, chicken fillet, shrimp, etc.), and the features extracted from the input image can be compared with the image of each food ingredient class.

[0092] One example of a training network is PANet (Prototype Alignment Network), which uses the features of trained prototypes so that each trained prototype can represent one class.

[0093] The classes of food ingredients that can be put into a cooking appliance such as an oven are limited, and similar types of food ingredients may be cooked repeatedly depending on the eating habits of the user.

[0094] Therefore, when a class is added, the control module 150 does not need to retrain the learning network, i.e., the control module 150 can recognize food ingredients by adding only the prototype of the class.

[0095] 8 is a diagram illustrating a user selecting a specific time point from a time-lapse video according to an embodiment of the present invention. The video may be composed of multiple images or may be a video. In the case of a video, when a user plays the video displayed on the display module 120 or a terminal device, the display module 120 or the terminal device displays an image of the initially added food ingredients (Food in S410) being cooked by heating over time (Food in S411).

[0096] The display module 120 or the terminal device may then display an image (Food in S412) of the food ingredients taken when they have reached a fully cooked state. If the user selects the time lapse point when the image in S412 is displayed, the control module 150 may set the image indicated by Food in S412 as the prototype image, and then the cooking appliance 100, 200 may take a picture of the food ingredients to be cooked and compare the taken image with the prototype image.

[0097] In addition, meta information such as cooking time (time elapsed since the first food ingredient was added) and heating temperature may be displayed on the left side of the image in Figure 8. The meta information may be displayed in combination with the image information. For example, this is an image of the state immediately after S410 adds food ingredients, and the cooking mode is displayed as bake, cooking time is 0 minutes, cooking temperature is 0 degrees, and cooking state is displayed as "Raw."

[0098] S411 is an image of the state after heating at 180 degrees for 5 minutes, with the cooking mode displayed as bake, cooking time as 5 minutes, cooking temperature as 180 degrees, and cooking state as "Medium."

[0099] S412 is an image of the state after heating at 180 degrees for 10 minutes, with the cooking mode being bake, the cooking time being 10 minutes, the cooking temperature being 180 degrees, and the cooking status being "Well-done."

[0100] Therefore, when a user selects a particular image, the control module 150 can apply the corresponding heating temperature and mode to cooking. In addition, the control module 150 can compare the physical characteristics (e.g., size, area, weight) of the food ingredients included in the prototype with the physical characteristics of the food ingredients to be cooked, and use this as a reference when comparing images.

[0101] That is, the control module 150 can set the cooking temperature or cooking time of the main body 190 by using the weight or size information of the food ingredients contained in the meta information of the video information, or by comparing the size of the food ingredients in the video information with the size of the food ingredients stored in the main body 190.

[0102] For example, the time required to reach the optimal cooking state may differ between heating 200g of pork belly and heating 500g of pork belly. Therefore, by comparing the meta information (weight, size, area, etc.) of the physical properties of the food ingredients in the cooking video with the physical properties of the food ingredients currently being cooked, the accuracy of the image comparison results can be improved by taking into account time and temperature factors.

[0103] If the user selects the image of S411 to add more seasonings in the cooking state of S411 and sets an alarm, the control module 150 cooks the ingredients reflecting the meta information of the image of S411. In addition, the control module 150 issues an alarm when the state of the ingredients reaches a state that is highly similar to the image of S411 during the cooking process. In this case, the control module 150 can predict the time information for the alarm to be reached based on the weight of the ingredients.

[0104] For example, S411 in FIG. 8 illustrates an example in which 200 g of food material is heated at 180° C. for 5 minutes. If 500 g of food material is added, the control module 150 can calculate the proportional time based on the weight as "5 minutes * (500 g / 200 g) - a." Here, a is the time required to prevent overcooking depending on the food material. For example, a may be 2 minutes for pork belly, 3 minutes for shrimp, etc. Of course, a may have a positive or negative value depending on the food material. That is, depending on the type of food material, the control module 150 can increase or decrease the proportional time calculated based on the weight or size to prevent the food material from being overcooked or undercooked.

[0105] Therefore, the control module 150 can increase the precision of the image processing as the predicted time approaches, comparing the prototype image with the current image of the food material to improve accuracy.

[0106] The control module 150 can also create a separate training database, which includes time-lapse footage or images for each food ingredient. Of course, the training database may also include meta-information about cooking conditions (mode, time, temperature, etc.) associated with each footage or image.

[0107] In one embodiment of the present invention, the control module 150 records and stores the cooking process of food ingredients that a user of the cooking appliance 100, 150 has previously cooked, and allows the user to set an alarm at a specific cooking point.

[0108] Furthermore, when a user instructs cooking based on a specific cooking time point, the control module 150 can control the cooking appliances 100, 150 to cook food ingredients based on the cooking conditions performed up to that time point.

[0109] According to one embodiment of the present invention, the control module 150 can perform image processing to recognize ingredients that are actually needed and used by the customer, thereby continuously improving its ingredient recognition performance in response to the user's cooking method.

[0110] Furthermore, since image processing is performed based on information about the food ingredients used by the user, the time required to recognize and compare food ingredients can be reduced.

[0111] This improves the image recognition performance of the control module 150, reduces the possibility of erroneous recognition, and enables accurate cooking guidance and alarms.

[0112] The user can then add or remove food items to be recognized, thereby further improving the recognition accuracy.

[0113] 9 is a diagram showing the recognition result of food ingredients displayed on a display module according to an embodiment of the present invention. Three food ingredient videos are presented as candidates corresponding to the stored food ingredients. If the user selects a past cooking video related to pork belly located in the middle, a triangle button (PLAY) is displayed, allowing the user to play the video, as shown in 120b. The cooking video is five minutes long, and a progress bar may be displayed at the bottom of 120b to allow the user to view specific points in the video.

[0114] Alternatively, multiple images may be displayed, such as 120c, to allow the user to select a specific image by displaying only a portion of the main image of the cooking video. The user can set an alarm by selecting a specific image (e.g., the image on the right side of 120c).

[0115] In the recognition process, the control module 150 extracts a feature vector from a captured image of the stored food ingredients, calculates the similarity between the extracted feature vector and the feature vector of the video information, and searches for video information corresponding to the food ingredients. At this time, the control module 150 can analyze words spoken by the user near the cooking appliance 100, convert them into text, and include them in one feature vector.

[0116] This can increase the accuracy of food ingredient recognition not only through the name of a specific food ingredient, but also through references to past cooking processes, such as "like the shrimp I had last week."

[0117] In addition, the control module 150 can improve the recognition performance of photographed food ingredients by accumulating and storing information selected by the user in Fig. 9. In addition, the camera may be installed on the side or upper side of the main body 190 in addition to the top of the interior thereof, and the accuracy of recognition can be improved by using images taken from multiple directions.

[0118] FIG. 10 is a diagram illustrating a process in which a cooking appliance according to an embodiment of the present invention provides an alarm or a cooking-related video to a terminal device via a server.

[0119] The control module 150 of the cooking appliance 100 / 200 photographs and recognizes the food ingredients (S310). This is explained in conjunction with the description of FIG. 4 above. The control module 150 of the cooking appliance 100 / 200 then searches for video information corresponding to the food ingredients (S421). The control module 150 of the cooking appliance 100 / 200 transmits the searched video information to the terminal device 600 (S422), and the terminal device 600 displays the video information.

[0120] The user then selects a specific time point from the video information displayed on the terminal device 600 and selects whether or not to receive an alarm at that time point (S423). The terminal device 600 transmits the selected information (specific time point and whether or not to set an alarm) to the cooking appliance 100 / 200 (S424). After that, the cooking appliance 100 / 200 performs steps S360 and S380 of FIG. 4, and if an alarm is set, transmits an alarm message to the terminal device 600 (S425). The terminal device 600 then issues the alarm (S426).

[0121] The control module 150 of the cooking appliance 100 / 200 and the terminal device 600 can directly transmit and receive data, or the server 500 can mediate communication between the control module 150 of the cooking appliance 100 / 200 and the terminal device 600.

[0122] Meanwhile, according to another embodiment of the present invention, the control module 150 of the cooking appliance 100, 200 can take a picture of the food ingredients and transmit it to the server 500, which can then search for the corresponding image information. In this case, the server 500 can store the image information according to the identification information of the cooking appliance 100, 200.

[0123] Videos taken by the cooking appliances 100 and 200 during the cooking process may also be uploaded to the server 500.

[0124] 11 is a diagram showing a process in which a server according to an embodiment of the present invention recognizes ingredients and compares them with existing cooking videos or images in a database to check whether cooking conditions are met. Some of the processes in FIGS. 4, 5, 6, etc. are performed by the server 500. Please refer to FIG. 4.

[0125] When a user places food ingredients into the cooking appliance 100, 200, the camera 110 captures a photo of the food ingredients, and the control module 150 transmits the captured image of the food ingredients to the server 500 (S310a, S310b). The server 500 recognizes the food ingredients using the received image (S310c). The server 500 then searches for cooking videos or DB images stored in association with the food ingredients (S330a) and transmits the retrieved video or image to the cooking appliance 100, 200 (S330b).

[0126] The display module 120 of the cooking appliance 100, 200 displays the cooking video or DB image (S330c).

[0127] The control module 150 can play back the displayed cooking video or DB image in time lapse. When the user selects a specific time point from the cooking video or a specific image from the DB image, the control module 150 transmits information about the selected time point or image to the server 500 (S350a, S350b). The server 500 stores the transmitted information (S350c).

[0128] The cooking appliances 100 and 200 then perform cooking. Under the control of the control module 150, the camera 110 takes a picture of the food ingredients during cooking (S360a). The control module 150 then transmits the captured image or video to the server 500 (S360b). During this process, the control module 150 can set cooking conditions using meta information included in the image or video for the selected time point.

[0129] The server 500 compares the captured image or video with the image or DB image of a specific time point selected by the user (S380a). The server 500 then compares whether the cooking state of the food ingredients is similar to the image of the specific time point. If the cooking state is similar, the server 500 generates a message instructing the cooking to be stopped or an alarm to be issued and transmits the message to the cooking appliance 100, 200 (S380b, S380c).

[0130] Alternatively, if cooking needs to continue, the server 500 can generate a message guiding the cooking status. For example, the cooking status guidance message may be a message guiding the time or the cooking progress, such as "5 minutes remaining until the set cooking status is reached" or "60% of the cooking has been completed so far."

[0131] Thereafter, the control module 150 that receives the message may stop cooking or issue an alarm (S380d). If it is determined in S380b that the ingredients are not similar, steps S360a to S380b may be repeated.

[0132] The embodiment of Fig. 11 can also be applied to the embodiment of Fig. 5 or 6, which recognizes food ingredients and checks whether cooking conditions are met. That is, the server 500 can perform some of the functions performed by the control module 150 and transmit the results to the control module 150.

[0133] In addition, the server 500 stores images of various food ingredients in a database, and when a new food ingredient is uploaded, the server 500 can update the database with related images, videos, etc.

[0134] In addition, the image information captured by the cooking appliances 100 and 200 can be stored together with the identification information of the cooking appliances 100 and 200, thereby improving the recognition rate in the process of recognizing food ingredients later.

[0135] The embodiment of FIG. 11 includes steps (S310a to S330b) in which the server 500 performs an ingredient recognition process using the ingredient images transmitted from the cooking appliances 100 and 200, searches for image information corresponding to the recognized ingredients, and transmits it to the control module.

[0136] The embodiment of FIG. 11 includes the steps of transmitting information about a specific time point or image in the video information received by the control module 150 to the server 500, and then controlling the cooking appliance body to perform cooking (S350a to S360a).

[0137] The embodiment of FIG. 11 includes a process (S360a-S380c) in which the control module 150 transmits an image captured during the cooking process of the cooking appliance 100, 200 to the server 500, and the server 500 uses the received image to generate a message for interrupting cooking, issuing an alarm, or guiding the cooking status, and transmits the message to the control module.

[0138] The embodiment of FIG. 11 includes a step (S380d) in which the control module 150 controls cooking, performs an alarm function, or displays a guide message according to the received message.

[0139] Although the present invention has been described as including all components constituting an embodiment of the present invention as being combined or operating in combination, the present invention is not necessarily limited to these embodiments. Within the scope of the present invention, all components may be selectively combined and operate in one or more combinations. Furthermore, all components may be implemented as independent hardware components, or some or all of the components may be selectively combined and implemented as a computer program having program modules that perform some or all of the functions of one or more pieces of hardware. Codes and code segments constituting such a computer program can be easily construed by those skilled in the art. These computer programs may be stored in computer-readable media and read and executed by a computer to implement embodiments of the present invention. Examples of storage media for computer programs include magnetic recording media, optical recording media, and semiconductor recording devices. Furthermore, a computer program embodying an embodiment of the present invention may include a program module transmitted in real time via an external device.

[0140] Although the present invention has been described above with reference to the preferred embodiment, various modifications and variations can be made by those skilled in the art, and it is understood that these modifications and variations are included within the scope of the present invention as long as they do not deviate from the scope of the present invention.

[0141] [Claims at the time of international application] [Claim 1] A cooking appliance for cooking food ingredients, a main body for heating the food material; A camera that photographs the food ingredients stored in the main body; a control module that performs a food ingredient recognition process on the food ingredient images captured by the camera, and outputs video information corresponding to the recognized food ingredients to a display module of the cooking appliance, or transmits the video information to a terminal device linked to the cooking appliance, and then sets cooking conditions of the main body according to a time or image selected from the video information, or monitors the cooking status of the main body and performs an alarm function. [Claim 2] 2. The cooking appliance according to claim 1, wherein the video information is video information captured and stored by the cooking appliance during the cooking process of food ingredients previously stored in the main body. [Claim 3] 2. The cooking appliance of claim 1, wherein the image information is stored image information of the cooking state of food ingredients received from another device and stored by the control module. [Claim 4] 2. The cooking appliance according to claim 1, wherein the video information includes at least one of cooking temperature and cooking time as meta-information. [Claim 5] The cooking appliance of claim 1 , wherein the control module sets a cooking temperature or a cooking time of the main body using meta information included in the video information. [Claim 6] The cooking appliance of claim 1, wherein the control module sets the cooking temperature or cooking time of the main body by using weight or size information of the food ingredients included in the meta information of the video information, or by comparing the size of the food ingredients in the video information with the size of the food ingredients stored in the main body. [Claim 7] 10. The cooking appliance of claim 1, wherein the control module controls the camera to capture a cooking status of the main body, and then, when the cooking of the main body is completed, stores image information captured by the camera or replaces previously stored image information with the captured image information. [Claim 8] 2. The cooking appliance of claim 1, wherein the recognition process includes the control module extracting a feature vector from an image of the stored food ingredient, calculating a similarity between the feature vector and a feature vector of the video information, and searching for video information corresponding to the food ingredient. [Claim 9] 2. The cooking appliance of claim 1, wherein the control module compares an image of the stored food ingredients being cooked with an image of a time selected from the video information, and if a similarity between the image of the selected time and the captured image is equal to or greater than a reference value, the control module stops cooking, issues an alarm via the display module, or transmits an alarm message to the terminal device. [Claim 10] 10. The cooking appliance of claim 9, wherein the control module outputs a photographed image of the food material cooked in the main body through the display module or transmits the photographed image to the terminal device. [Claim 11] 1. A method for controlling a cooking process using a visual cooking guide, comprising: A step in which a camera of the cooking appliance photographs the food ingredients stored in the main body of the cooking appliance; a control module of the cooking appliance performing an ingredient recognition process on the food ingredient images captured by the camera and retrieving video information corresponding to the recognized food ingredients; The control module outputs the image information to a display module of the cooking appliance or transmits the image information to a terminal device linked to the cooking appliance; and a step in which the control module sets cooking conditions for the main body or monitors the cooking status of the main body and performs an alarm function in response to a time point or image selected from the video information. [Claim 12] 12. The method for controlling a cooking process using a visual cooking guide according to claim 11, wherein the image information is image information captured and stored by the cooking appliance showing the cooking process of food ingredients previously stored in the main body, or image information stored by the control module showing the cooking state of food ingredients received from another device. [Claim 13] The video information is The meta information includes at least one of cooking temperature and cooking time, 12. The method for controlling a cooking process using a visual cooking guide according to claim 11, wherein the video information includes weight or size information of ingredients as meta-information. [Claim 14] 12. The method of claim 11, further comprising the step of: controlling the camera to capture a cooking status of the main body; and storing image information captured by the camera or replacing previously stored image information with the captured image information when the cooking of the main body is completed. [Claim 15] 12. The method of claim 11, further comprising the step of: comparing an image of the stored food ingredients being cooked with an image of a selected time point from the video information; and, if a similarity between the image of the selected time point and the captured image is equal to or greater than a reference value, interrupting the cooking, issuing an alarm through the display module, or transmitting an alarm message to the terminal device. [Claim 16] 12. The method of claim 11, further comprising the step of: outputting a photographed image of the food ingredients cooked in the main body through the display module or transmitting the photographed image to the terminal device. [Claim 17] 1. A method for controlling a cooking process using a visual cooking guide, comprising: A step in which a camera of the cooking appliance photographs the food ingredients stored in the main body of the cooking appliance; transmitting the image of the food material captured by the camera to a server by the control module of the cooking appliance; the server performs a food ingredient recognition process using the transmitted image, retrieves image information corresponding to the recognized food ingredient, and transmits the image information to the control module; the control module transmits information about a specific time point or image in the video information to the server, and then controls the cooking appliance to perform cooking; the control module transmits an image taken during the cooking process of the cooking appliance to a server, and the server generates a message for interrupting cooking, issuing an alarm, or guiding the cooking status using the received image, and transmits the message to the control module; and the control module controls cooking, performs an alarm function, or outputs a guide message to a display module of the cooking appliance according to the received message. [Claim 18] 18. The method for controlling a cooking process using a visual cooking guide according to claim 17, wherein the video information is video information that the cooking appliance captures and transmits to the server images of the cooking process of food ingredients previously stored in the main body, or video information that the server receives from another device and stores and stores images of the cooking state of food ingredients. [Claim 19] The video information is The meta information includes at least one of cooking temperature and cooking time, The method for controlling a cooking process by providing a cooking guide in the form of a video according to claim 17, wherein the video information includes weight or size information of ingredients as meta information. [Claim 20] 18. The method for controlling a cooking process using a visual cooking guide according to claim 17, further comprising the step of: the server comparing an image of the stored food ingredients being cooked with an image of the selected time point from the video information, and determining whether a similarity between the image of the selected time point and the captured image is equal to or greater than a reference value. [Explanation of symbols]

[0142] 100, 200 cooking equipment 110 Camera 120 Display Module 150 control module 500 servers 600 Terminal Equipment

Claims

1. A cooking appliance for cooking food ingredients, a main body for heating the food material; A camera that photographs the food ingredients stored in the main body; a control module that performs a food ingredient recognition process on the food ingredient images captured by the camera, and outputs video information corresponding to the recognized food ingredients to a display module of the cooking appliance, or transmits the video information to a terminal device linked to the cooking appliance, and then sets cooking conditions of the main body according to a time or image selected from the video information, or monitors the cooking status of the main body and performs an alarm function.

2. The cooking appliance according to claim 1 , wherein the image information is image information captured and stored by the cooking appliance during the cooking process of food ingredients previously stored in the main body.

3. The cooking appliance of claim 1 , wherein the image information is image information of the cooking state of food ingredients received from another device and stored by the control module.

4. The cooking appliance according to claim 1 , wherein the video information includes at least one of cooking temperature and cooking time as meta-information.

5. The cooking appliance of claim 1 , wherein the control module sets a cooking temperature or a cooking time of the main body using meta information included in the video information.

6. The cooking appliance of claim 1, wherein the control module sets the cooking temperature or cooking time of the main body by using weight or size information of the food ingredients included in the meta information of the video information, or by comparing the size of the food ingredients in the video information with the size of the food ingredients stored in the main body.

7. 2. The cooking appliance of claim 1, wherein the control module controls the camera to capture a cooking status of the main body, and then, when the cooking of the main body is completed, stores image information captured by the camera or replaces previously stored image information with the captured image information.

8. 2. The cooking appliance of claim 1, wherein the recognition process includes the control module extracting a feature vector from an image of the stored food ingredient, calculating a similarity between the feature vector and a feature vector of the video information, and searching for video information corresponding to the food ingredient.

9. 2. The cooking appliance of claim 1, wherein the control module compares an image of the stored food ingredients being cooked with an image of a time selected from the video information, and when a similarity between the image of the selected time and the captured image is equal to or greater than a reference value, the control module interrupts cooking, issues an alarm via the display module, or transmits an alarm message to the terminal device.

10. The cooking appliance of claim 9 , wherein the control module outputs a photographed image of the food material being cooked in the main body through the display module or transmits the photographed image to the terminal device.

11. 1. A method for controlling a cooking process using a visual cooking guide, comprising: A step in which a camera of the cooking appliance photographs the food ingredients stored in the main body of the cooking appliance; a control module of the cooking appliance performing an ingredient recognition process on the food ingredient images captured by the camera and retrieving video information corresponding to the recognized food ingredients; The control module outputs the image information to a display module of the cooking appliance or transmits the image information to a terminal device linked to the cooking appliance; and a step in which the control module sets cooking conditions for the main body or monitors the cooking status of the main body and performs an alarm function in response to a time point or image selected from the video information.

12. 12. The method of claim 11, wherein the image information is image information captured and stored by the cooking appliance showing the cooking process of food ingredients previously stored in the main body, or image information stored by the control module showing the cooking state of food ingredients received from another device.

13. The video information is The meta information includes at least one of cooking temperature and cooking time, The method for controlling a cooking process using a visual cooking guide according to claim 11 , wherein the image information includes weight or size information of ingredients as meta-information.

14. 12. The method of claim 11, further comprising the step of: controlling the camera to capture a cooking status of the main body; and storing image information captured by the camera or replacing previously stored image information with the captured image information when the cooking of the main body is completed.

15. 12. The method of claim 11, further comprising: comparing an image of the stored food ingredients being cooked with an image of a selected time point from the video information; and, if a similarity between the image of the selected time point and the captured image is equal to or greater than a reference value, interrupting the cooking, issuing an alarm through the display module, or transmitting an alarm message to the terminal device.

16. 12. The method of claim 11, further comprising the step of: outputting a photographed image of the food ingredients cooked in the main body through the display module or transmitting the photographed image to the terminal device.

17. 1. A method for controlling a cooking process using a visual cooking guide, comprising: A step in which a camera of the cooking appliance photographs the food ingredients stored in the main body of the cooking appliance; transmitting the image of the food material captured by the camera to a server by the control module of the cooking appliance; the server performs a food ingredient recognition process using the transmitted image, retrieves image information corresponding to the recognized food ingredient, and transmits the image information to the control module; the control module transmits information about a specific time point or image in the video information to the server, and then controls the cooking appliance to perform cooking; The control module transmits an image taken during the cooking process of the cooking appliance to a server, and the server generates a message for interrupting cooking, issuing an alarm, or guiding the cooking status using the received image, and transmits the message to the control module; and the control module controls cooking, performs an alarm function, or outputs a guide message to a display module of the cooking appliance according to the received message.

18. 18. The method of claim 17, wherein the image information is image information that the cooking appliance captures a cooking process of food ingredients previously stored in the main body and transmits to the server, or image information that the server receives from another device and stores, and stores, an image of the cooking state of food ingredients.

19. The video information is The meta information includes at least one of cooking temperature and cooking time, The method of claim 17 , wherein the image information includes weight or size information of ingredients as meta information.

20. 18. The method of claim 17, further comprising the step of: comparing an image of the stored food ingredients being cooked with an image of the selected time point from the video information; and determining whether a similarity between the image of the selected time point and the captured image is equal to or greater than a reference value.

Images