Apparatus and method for controlling cooking process using visual cooking guidance

Abstract

The present invention relates to an apparatus and a method for controlling a cooking process using visual cooking guidance, and a cooking apparatus according to an embodiment of the present invention, as a cooking apparatus for cooking a food material, comprises: a main body for heating the food material; the camera is used for shooting the food materials put into the main body; the control module is used for executing a food material identification process on the food material image shot by the camera, outputting image information corresponding to the identified food material to a display module of the cooking equipment or transmitting the image information to a terminal device linked with the cooking equipment; the cooking condition of the main body is set or the cooking condition of the main body is monitored according to the time point or the image selected from the image information, and a reminding function is executed.

Description

Technical Field

[0001] The present invention is a technology related to devices and methods for controlling a cooking process using visual cooking guides. Background Art

[0002] An oven is a device that can cook a variety of foods, allowing users to heat a variety of ingredients. However, depending on the ingredients, cooking methods, and temperatures, it is difficult for users to master all cooking methods and temperatures.

[0003] For example, the temperature and time required for heating the oven can be set differently depending on the type of ingredients and cooking methods, such as meat, fish, and vegetables. Therefore, a method is needed to provide oven users with cooking instructions that are suitable for the ingredients and control the oven accordingly. Summary of the Invention

[0004] Problems to be solved by the invention

[0005] This manual aims to solve the above problems. Users can select the cooking state suitable for the ingredients in the form of images or videos, so that the cooking equipment can cook according to the selection or execute reminders when the cooking state is reached, thereby improving cooking efficiency.

[0006] In addition, this manual allows you to confirm the past cooking process through images to select the best cooking state and receive reminders.

[0007] The purpose of the present invention is not limited to the above purpose. Other purposes and advantages of the present invention not mentioned above can be understood through the following description and will be more clearly understood through the embodiments of the present invention. In addition, it is easy to understand that the purposes and advantages of the present invention can be achieved by the means described in the scope of the claims and their combinations.

[0008] Technical solutions to the problem

[0009] A cooking device according to an embodiment of the present invention is a cooking device for cooking ingredients, comprising: a main body for heating ingredients; a camera for photographing ingredients placed in the main body; and a control module for executing an ingredient recognition process on the image of the ingredients photographed by the camera, and after outputting image information corresponding to the recognized ingredients to a display module of the cooking device or transmitting the image information to a terminal device linked to the cooking device, setting the cooking conditions of the main body or monitoring the cooking status of the main body corresponding to a time point or image selected from the image information to execute a reminder function.

[0010] According to an embodiment of the present invention, a method for controlling a cooking process using visual cooking guidance includes: a step in which a camera of a cooking device photographs ingredients placed in a main body of the cooking device; a step in which a control module of the cooking device performs an ingredient recognition process on the image of the ingredients photographed by the camera to retrieve image information corresponding to the recognized ingredients; a step in which the control module outputs the image information to a display module of the cooking device, or transmits the image information to a terminal device linked to the cooking device; and a step in which the control module sets cooking conditions for the main body or monitors the cooking status of the main body in correspondence with a time point or image selected from the image information to perform a reminder function.

[0011] Another embodiment of the present invention provides a method for controlling a cooking process using visual cooking guidance, including: a step in which a camera of a cooking device photographs ingredients placed in a main body of the cooking device; a step in which a control module of the cooking device transmits an image of the ingredients photographed by the camera to a server; a step in which the server executes an ingredient recognition process using the transmitted image, and transmits the image information corresponding to the recognized ingredients to the control module after retrieving the image information; a step in which the control module controls the main body of the cooking device to execute cooking after transmitting information of a specific time point or image in the image information to the server; a step in which the server generates a message for interrupting cooking or executing a reminder or guiding the cooking status using the received image after the control module transmits an image captured during the cooking process of the cooking device to the server, and transmits the message to the control module; and a step in which the control module controls cooking or executes a reminder function or outputs a guidance message on the display module of the cooking device according to the received message.

[0012] Effects of the Invention

[0013] When using the present invention, the user selects a cooking state suitable for the food in the form of an image or video, and the cooking device cooks according to the selection or issues a reminder when the cooking state is reached, thereby improving cooking efficiency.

[0014] In addition, when using the present invention, the past cooking process can be confirmed through images to select the best cooking state and receive reminders.

[0015] The effects of the present invention are not limited to the above-mentioned effects, and various effects of the present invention can be easily derived from the configuration of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 FIG. 1 is a diagram showing components of a cooking device including a display module according to an embodiment of the present invention.

[0017] Figure 2 FIG. 1 is a diagram illustrating components of a cooking apparatus that does not include a display module according to another embodiment of the present invention.

[0018] Figure 3This is a diagram showing the configuration of a cooking device according to an embodiment of the present invention.

[0019] Figure 4 FIG. 1 is a diagram illustrating a cooking control process according to an embodiment of the present invention.

[0020] Figure 5 FIG. 1 is a process diagram illustrating a cooking process of controlling the cooking process according to an embodiment of the present invention.

[0021] Figure 6 This is a diagram illustrating a process of buffering and comparing images according to an embodiment of the present invention.

[0022] Figure 7 FIG. 1 is a diagram illustrating an image comparison process according to an embodiment of the present invention.

[0023] Figure 8 This is a diagram showing a user selecting a specific time point from a time-lapse video according to an embodiment of the present invention.

[0024] Figure 9 FIG. 1 is a diagram showing a display module displaying identification results of food ingredients according to an embodiment of the present invention.

[0025] Figure 10 This diagram illustrates a process in which a cooking appliance provides a reminder or a cooking-related image to a terminal device via a server according to an embodiment of the present invention.

[0026] Figure 11 The server of one embodiment of the present invention identifies the ingredients or compares them with existing cooking images or images in a database to confirm whether the cooking conditions are met. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention. The present invention can be implemented in various ways and is not limited to the embodiments described herein.

[0028] In order to clearly illustrate the present invention, parts not related to the description are omitted. Throughout the specification, the same or similar components are given the same figure numerals. In addition, some embodiments of the present invention are described in detail with reference to exemplary drawings. In the process of assigning figure numerals to the components of each drawing, the same components may have the same symbols as much as possible even if they are shown in different drawings. In addition, in the process of describing the present invention, when it is determined that the detailed description of the relevant known structure or function may make the gist of the present invention unclear, its detailed description may be omitted.

[0029] When describing the constituent elements of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the constituent elements from other constituent elements and do not limit the nature, order, sequence, or quantity of the corresponding constituent elements. When it is described that a constituent element is "connected," "combined," or "coupled" to other constituent elements, it should be understood that the constituent element can be directly connected or coupled to the other constituent element, but another constituent element may be interposed between the constituent elements, or the constituent elements may be "connected," "combined," or "coupled" through other constituent elements.

[0030] In addition, in the process of implementing the present invention, for the sake of convenience, the components can be subdivided for description, but these components can also be defined as one device or one module for implementation, or one component can be divided into multiple devices or modules for implementation.

[0031] In this specification, an embodiment of a cooking device includes an oven as a household appliance for heating and cooking various ingredients. Ovens include various household appliances such as light wave ovens, microwave ovens, and air fryers.

[0032] In this specification, the cooking device includes a camera for capturing food placed inside. Additionally, it may optionally include a display for outputting images. If the cooking device is not equipped with a display, a display of a terminal device linked to the cooking device may output images.

[0033] Figure 1 FIG. 1 is a diagram showing components of a cooking device including a display module according to an embodiment of the present invention.

[0034] The cooking device 100 includes a camera 110 , a display module 120 , a control module 150 , and a main body 190 . Figure 1 This diagram focuses on the cooking device 100's ability to capture food and process the resulting information. It also includes the components necessary for cooking. As one example of the multiple components included in the cooking device 100, components necessary for heating food, such as the door, heating unit, and lamp, are all included in the main body 190. Furthermore, the operation of the main body 190 is controlled by a control module 150.

[0035] Camera 110 is located inside the main body and captures the cooking area. Camera 110 can be located on the ceiling or side of the main body, for example. Furthermore, more than one camera 110 can be provided. When two or more cameras 110 are provided, the food placed inside the main body can be captured from different positions. Camera 110 can capture images or videos of the cooking process at predetermined intervals. Furthermore, camera 110 can capture time-lapse videos.

[0036] The display module 120 can be configured outside the body 190 and visually output various information. The display module 120 can also include a touch module capable of receiving touch input, and the user can control the operation of the cooking device 100 by touching the image displayed on the display module 120.

[0037] The control module 150 is responsible for controlling the cooking device 100. When ingredients are added to the main body 190, the camera 110 can be controlled to capture the ingredients. Furthermore, the control module 150 can calculate the size or weight of the captured ingredients. As one embodiment, the main body 190 can be equipped with a weight sensor, and the control module 150 can use the value sensed by the weight sensor to calculate the weight of the ingredients.

[0038] In addition, the control module 150 can perform communication with a server or a terminal device. To this end, the control module 150 can transmit and receive data according to a specific communication protocol (such as LTE-A, 5G communication, Wi-Fi, Bluetooth, ZigBee, etc.).

[0039] The control module 150 may include a specific storage medium and a database using the storage medium. Alternatively, the cooking device 100 may include an additional database. Alternatively, the database may be located in an external server, and the control module 150 may communicate with the server to confirm information stored in the database.

[0040] In addition, the information outputted by the display module 120 may be transmitted by the control module 150 to a server or a terminal device, so that the display of the terminal device outputs the information.

[0041] Figure 2 1 is a diagram showing the components of a cooking device that does not include a display module according to another embodiment of the present invention. Figure 1 's instructions instead. Figure 2 It is from Figure 1 The control module 150 can transmit information related to the food put into the main body 190 and information on the cooking status to the server or terminal device, and receive a message requesting to stop or restart cooking.

[0042] Figure 1 and Figure 2 The cooking device 100 shown is a device for cooking food, and includes a main body 190 for heating food, a camera 110 for photographing food placed in the main body 190 , and a control module 150 .

[0043] The control module 150 performs an ingredient recognition process on the ingredient image captured by the camera 110, and retrieves the image information corresponding to the identified ingredient. In addition, the control module 150 outputs the information to the display module 120 of the cooking device 100 or to a terminal device ( Figure 10 Then, the control module 150 sets the cooking conditions of the main body 190 or monitors the cooking status of the main body 190 to perform the reminder function according to the time point or image selected by the user from the image information.

[0044] Figure 3 This is a diagram showing the configuration of a cooking device according to an embodiment of the present invention. Figure 3 and one In accordance with the embodiment, the cooking device of the present invention may have Figure 3 Different shapes or including Figure 3 Different components.

[0045] The main body includes an operation portion 21 , doors 11 , 12 , and cooking chambers 15 , 16 opened and closed by the doors 11 , 12 .

[0046] A display module 120 is provided on the operating unit 21 for operating the cooking device 100. The user can confirm cooking instructions via the display module 120. A camera 110 can be located at the top of the cooking device 100 and can capture images of the interior of the cooking chamber. The cooking chambers 15 and 16 can be opened and closed by doors 11 and 12. Shelves or racks for placing ingredients and heating lamps can be provided inside the cooking chambers 15 and 16. The camera 110 can capture images of the interior of the cooking chambers 15 and 16.

[0047] Although the operation unit 21 and the display module 120 are distinguished in the drawings, a configuration in which a user interface for operation is displayed on the display module 120 as a different embodiment is also included in the embodiments of the present invention. Figure 3 When the display module 120 is removed from the cooking device 100, the cooking device 100 becomes Figure 2 Example of .

[0048] Hereinafter, a method is described in which a camera installed in a cooking device 100 such as an oven collects and stores food images, and a user provides cooking guidance and controls cooking by directly adding or removing recognition target items.

[0049] When using the embodiment of the present invention, the cooking setting information (oven operation mode, time, temperature) of the identified item can be databased, and when the cooking device 100 recognizes the cooking target, the cooking environment can be automatically set.

[0050] In addition, using the cooking video previously taken, the user can select the cooking state that needs to receive reminders, and the user can confirm the reminder. Or, if the desired cooking state is reached, the cooking device 100 can end the cooking process.

[0051] The user can select the optimal cooking state based on the image instead of setting a specific cooking state with text or numbers, thereby being able to easily control the cooking process of the cooking apparatus 100.

[0052] Figure 4 This diagram illustrates a cooking control process according to an embodiment of the present invention. When ingredients with cooking images associated with them are placed in the cooking apparatus 100 or 200, the ingredients are photographed and recognized, and the cooking process is controlled using the corresponding cooking images or DB images.

[0053] When a user places ingredients into cooking apparatus 100 or 200, camera 110 captures the ingredients, and control module 150 processes the captured image to identify the ingredients (S310). Control module 150 displays a cooking image or a database image stored in association with the ingredients via display module 120 (S330). If cooking apparatus 200 does not include display module 120, a terminal device linked to cooking apparatus 200 can display the cooking image or database image.

[0054] In one embodiment, a cooking video refers to a collection of videos or images previously captured while cooking the identified ingredients using the cooking apparatus 100 or 200. These videos or images include information about specified time points (e.g., in units of 5, 10, or 30 seconds, or 1, 2, or 3 minutes, etc.), as well as information about the heating temperature or cooking mode used during cooking. Furthermore, one embodiment of a cooking video is a time-lapse video that captures the changes in ingredients during the cooking process.

[0055] For example, when heating meat, camera 110 continuously captures changes in surface color and volume during the heating process and stores them as a video or image collection. In this case, the cooking time information (cooking time) and the temperature information during the cooking process (cooking temperature) are equivalent to meta-information, which can be combined with each video or image.

[0056] In addition to cooking images, the database can also store additional images or videos. For example, images or videos showing the color changes on the surface of the food depending on the type of meat, cooking temperature, and cooking time can be combined with information such as time and cooking temperature and stored in the database.

[0057] Thus, after the control module 150 displays a cooking video or a database image stored in the control module 150 itself, or displays a cooking video or a database image stored externally on a server, the user can select a specific time point. For example, a specific time point can be selected in a time-lapse of a cooking video or a specific image can be selected from the database image (S350). Such selection can include information regarding the heating temperature.

[0058] If the control module 150 controls the main body 190 to heat the food to be cooked at the corresponding heating temperature (or the automatically / manually set heating temperature), and the camera 110 captures the food, the control module 150 compares the captured image with the image at the specific time point of the time-lapse photography or the DB image (S360). During the comparison process, the control module 150 may refer to the image at the previous time point of the time-lapse photography of the corresponding image or the image at the previous time point in the DB image.

[0059] In addition, the image of the food captured in the process S360 can be displayed on the display module 120 or the terminal device carried by the user, so that the user can confirm the current cooking status.

[0060] In addition, the control module 150 compares whether the cooking state of the food is similar to the image at a specific time point. If the state is similar, the control module 150 performs a cooking interruption or a reminder (Notification, Alarming) (S380). As an embodiment, the execution of the reminder is that the cooking device 100, 200 outputs a specified sound or a linked terminal device ( Figure 10 600) outputs a notification sound. In addition, as an embodiment, the reminder is in the terminal device ( Figure 10 600) outputs text or images notifying that the cooking status has reached a specific state (a time point selected by the user).

[0061] Then, if the user takes subsequent action, the control module 150 updates the information based on the user's action (S390). When the cooking device 100, 200 is turned off and cooking is finished, the processes in S360 and S380 of the control module 150 are effective, and the control module 150 stores the results so that the same judgment process can be performed in similar situations later.

[0062] On the contrary, if the user starts cooking after S380, the processes in S360 and S380 of the control module 150 are not valid, and the control module 150 stores the results so that subsequent response measures can be reflected and comparison and judgment processes can be performed in similar situations.

[0063] The user selects ingredients and a cooking method based on the ingredients, and then selects a menu from the cooking device 100 or 200. The user can set the heating temperature, heating mode, time, etc. However, it is cumbersome for the user to check the manual for the optimal cooking conditions for the ingredients, and there are limitations in selecting cooking conditions based on the amount of ingredients.

[0064] In addition, in order to confirm the cooking process, the user may open the door of the cooking apparatus 100, 200 during cooking. During this process, heat loss may occur, resulting in an increase in cooking time.

[0065] Among them, the user selects the cooking image corresponding to the user's desired cooking state or the image stored in the database. The cooking devices 100 and 200 can determine whether the ingredients need to be heated or provide notification information to the user by comparing the image of the ingredients captured during the cooking process with the image selected by the user.

[0066] In use Figure 4 In the case of the embodiment of the present invention, the cooking device 100, 200 can automatically identify the ingredients and automatically set the cooking settings or provide relevant information to the user. In addition, while the camera is shooting the ingredients, the cooking status information can be confirmed.

[0067] The following describes the process of using images stored in the cooking device or server to select / add / delete items of ingredients to be identified, using cooking videos to select images of the state in which the user wants to receive reminders, and receiving reminders at the corresponding state time point.

[0068] In the above embodiment, as one embodiment, the image information is a set of images captured as a time-lapse of the cooking process or captured at specified time intervals during the cooking process or a set of images constructed in association with the capture of corresponding ingredients.

[0069] For example, the image information may be a cooking image captured in the past. In one embodiment, the image information is image information captured and stored by the cooking device 100 of the user during the cooking process of the food placed in the main body 190.

[0070] In addition, the image information may be a database image generated by other users or servers or received and constructed by the cooking device 100 from other devices. In one embodiment, the image information is image information composed of images of the cooking state of ingredients received and stored by the control module 150 from other devices.

[0071] The database images described above can be equivalent to cooking guides provided by large supermarkets or online shopping malls. For example, large supermarkets or online shopping malls may sell specific products and provide images or videos of cooking the corresponding products. The control module 150 can register the provided images or videos and select a specific time point in the image or video that corresponds to the cooking status of the purchased ingredients.

[0072] Figure 5 FIG. 1 is a flow chart illustrating a cooking process of controlling the cooking process according to an embodiment of the present invention.

[0073] If food is put into the cooking device 100, 200, the camera 110 takes a picture of the food (S311). The control module 150 uses the captured image to identify the food. To this end, the control module 150 can store an existing initial image of the food.

[0074] If the food is not recognized (S312), the control module 150 sets R_FLAG to 0 and sets and registers cooking conditions according to user control (S313). The cooking conditions include cooking mode, heating temperature, heating time, etc.

[0075] On the other hand, if the food is recognized in step S312, the control module 150 sets R_FLAG to 1 (S315). The image information corresponding to the recognized food can be displayed on the display module or terminal device ( Figure 10 600). In addition, if the user does not set a status reminder (S316), the control module 150 automatically sets the cooking settings suitable for the identified ingredients (S317). The automatically set cooking conditions include cooking mode, heating temperature, heating time, etc.

[0076] On the other hand, if the user sets a status reminder in S316, the control module 150 can choose to: select a specific time point from the stored cooking images to choose whether to receive the status reminder, or select a desired status from the stored images and receive the reminder (S331).

[0077] If the user chooses to select a specific image from the image information displayed corresponding to the identified food and receive a reminder when the state of the food is similar to the selected image (the "No" process of S331), the control module 150 displays multiple images on the display module 120, and the user selects one or more images as prototypes (S351).

[0078] On the other hand, if the user selects a specific time point from the stored cooking video to receive a status reminder ("Yes" in S331), the control module 150 displays the cooking video on the display module 120, and the user selects a specific time point during the video playback process (time-lapse video playback) (S353). After selecting a specific time point, information such as temperature and cooking time at various time points previously recorded when the cooking video was recorded can also be selected.

[0079] In addition, the control module 150 extracts one or more prototype images of cooking states in different stages with the selected time point as the center (S354).

[0080] The prototype selected / extracted in S351 or S354 can be used as comparative data for confirming the cooking status of the food during the subsequent cooking process of the food.

[0081] In addition, the control module 150 automatically sets the cooking conditions of the main body 190 according to the selected / extracted prototype (S357). At this time, the set cooking conditions are cooking conditions that reflect the cooking time or cooking temperature set in the meta information combined with the image information.

[0082] If the cooking conditions are set in steps S313, S317, and S357, respectively, the cooking devices 100 and 200 begin cooking (S321, S361). During the cooking process, the camera 110 captures the food and buffers the images (S322, S362). Furthermore, the control module 150 generates time-lapse photography information corresponding to the captured images (S323, S363).

[0083] As an embodiment, generating time-lapse photography information includes correspondingly generating the current state of the food image (buffered as an image) and information on cooking conditions (heating temperature, current temperature in the oven, time elapsed after cooking starts, weight changes of the food, etc.).

[0084] Then, you can confirm the end of cooking or the reminder status.

[0085] For example, the process from S321 to S324 describes an example in which no status notification is received during the cooking process. In this case, the control module 150 determines whether the end condition based on the cooking conditions set in S313 / S317 has been met (S324). If not, S322 to S324 are repeated. If the end condition has been met, the control module 150 ends the cooking operation of the cooking device (S385).

[0086] On the other hand, the process of S361 to S381 is an embodiment of receiving a status reminder during the cooking process. In this case, the control module 150 compares the status of the ingredients selected in S351 / S353 with the image of the prototype to determine whether the cooking status of the ingredients reaches the set prototype (S364).

[0087] Otherwise, if the state of the food is not reached, S361 to S364 are repeated. If the state of the food is the same or similar to the prototype image in S364, the control module 150 provides a visual or auditory reminder to the user (S381) and ends the cooking operation of the cooking device (S385).

[0088] If the cooking process is finished, the image and time-lapse photography information generated during S322 / S323 or S362 / S363 when the food is recognized are updated to the existing stored cooking image (S387) based on the value of R_FLAG, i.e., based on whether the food is recognized. Of course, the update can be performed selectively.

[0089] On the other hand, when the ingredients are not recognized (S312~S313), depending on the user's choice of whether to register the ingredients, if the user chooses to register (S388-"Yes"), the image and time-lapse photography information generated during S322 / S323 will be updated to the existing stored cooking image (S387).

[0090] In one embodiment, the prototypes are images of the state of ingredients according to predetermined steps during the cooking process. For example, images of the ingredients immediately after they are added, images of the ingredients after one minute of cooking, images of the ingredients after two minutes of cooking, images of the ingredients three minutes before the end of cooking, and images of the ingredients at the end of cooking can be stored in a database or generated during the cooking image generation process.

[0091] In addition, images showing the cooking status can be extracted from the cooking video at preset time intervals. For example, if the cooking video is played for 10 minutes, images can be extracted in 1-minute units, and a total of 11 prototype images including the initial image can be extracted.

[0092] Using multiple prototypes, the control module 150 can compare the changing process of the ingredients with the image.

[0093] As described in S388, after the control module 150 controls the camera 110 to shoot the cooking status of the main body 190, if the cooking of the main body 190 is completed, the image information captured by the camera 110 is stored or the previously stored image information is replaced with the captured image information, so as to confirm a better cooking process through subsequent image information and execute automatic setting and reminder setting of the cooking process.

[0094] Figure 6 FIG. 1 is a diagram illustrating a process of buffering and comparing images according to an embodiment of the present invention.

[0095] While cooking, the camera 110 captures and buffers images. For example, the camera 110 may capture K images (x*y*K) of a specific resolution (x*y) (S341). In addition, the control module 150 may extract features from the N images (S342).

[0096] The feature extraction can be performed in a three-dimensional space. In addition, the control module 150 measures the similarity (S343) by comparing the extracted features with a prototype image previously stored in the database (selected in S351) or an image corresponding to a specific time point in the time-lapse photography within the existing stored images (selected in S353).

[0097] As one embodiment, during the similarity measurement process, the control module 150 may extract high indexes based on the z-axis (S344). Furthermore, if the number of extracted indexes exceeds a threshold value (Threshold), the control module 150 determines that a similar state exists (S345) and issues a reminder (S346). If the answer to S345 is "No," the control module 150 may repeat the image capture and comparison process during the cooking process.

[0098] Figure 6 The embodiment of the present invention primarily utilizes an image resolution of 1920*1080. In this case, the control module 150 captures K images at a predetermined time interval (e.g., 10 seconds, 20 seconds, 30 seconds, or 1 minute) (S341). Furthermore, the control module 150 may extract 30*30*512 features from the K images with a resolution of 1920*1080 (S342).

[0099] Furthermore, based on the aforementioned features, the control module 150 can measure the similarity of each cooking step with the image of the selected specific step, in conjunction with the background image (S343). At this point, the control module 150 can arrange the extracted features along the z-axis and extract an index value based on the z-axis (S344). Furthermore, the control module 150 can compare the similarity with the image of the cooking step for which a reminder has been previously set, and if the similarity exceeds a specified threshold, a reminder can be issued.

[0100] In the above Figure 4 S380, Figure 5 S381, Figure 6During the execution of reminders such as S346, the control module 150 performs image comparison using an image captured of the state of the ingredients being cooked and an image at a time point selected from the image information (previous images of the cooking process or images in the database). In addition, if the similarity between the image at the selected time point and the captured image (the image of the ingredients currently being cooked) is above a reference value, the control module 150 may interrupt cooking. Alternatively, the control module 150 may execute a reminder or send a notification to the terminal device ( Figure 10 600) sends a reminder message.

[0101] In addition, during the cooking process, the control module 150 can output the image of the food being cooked in the main body 190 through the display module 120, or transmit the image to the terminal device. At this time, in addition to the current cooking status, the display module 120 or the terminal device ( Figure 10 600) to output the remaining time information or cooking progress percentage information until the estimated completion time point (the time point selected by the user from the image information).

[0102] Figure 7 This diagram illustrates the image comparison process in one embodiment of the present invention. In this specification, as one embodiment of image recognition, target categories are defined and the network is configured to increase similarity between classes within the same category, while the network is configured and trained to decrease similarity between classes of different categories. If a new category is added, the network does not need to be retrained; recognition can be performed by adding prototypes of that category.

[0103] like Figure 7 As shown, the control module 150 may input an image into a learning network to extract a feature vector (S401, S402).

[0104] The control module 150 may extract similarities between the extracted feature vectors and each prototype (S403). The prototype may be determined based on the category of each ingredient (broccoli, chicken breast, shrimp, etc.), and the features extracted from the input image may be compared with the categorized images of each ingredient.

[0105] As an embodiment of the learning network, PANet (Prototype Alignment Network) can be used. This technology uses the features of learned prototypes to enable each learned prototype to represent a category.

[0106] There are limitations on the types of ingredients that can be put into a cooking device such as an oven, and in addition, similar kinds of ingredients may be cooked repeatedly according to the user's eating habits.

[0107] Therefore, when the control module 150 adds a new category, there is no need to retrain the learning network. In other words, the control module 150 can identify food ingredients by simply adding a prototype of the new category.

[0108] Figure 8 This is an image of a specific time point selected by the user from a time-lapse video according to an embodiment of the present invention. As one embodiment, the image is composed of multiple images or is a video. In the case of a video, if the user plays an image displayed on the display module 120 or the terminal device, the display module 120 or the terminal device displays an image of the food just added ("Food" in S410) being heated and cooked over time ("Food" in S411).

[0109] In addition, the display module 120 or the terminal device can display an image captured when the food is fully heated ("Food" in S412). When the image similar to S412 is displayed, if the user selects a time point corresponding to time-lapse photography, the control module 150 can set the image indicated by "Food" in S412 as the prototype image. The cooking apparatus 100 or 200 then captures the food being cooked and compares the captured image with the prototype image.

[0110] in addition, Figure 8 Meta-information such as cooking time (the time elapsed after the ingredients were initially added) and heating temperature can be displayed on the left side of the image. This meta-information can be displayed in conjunction with the image information. For example, in S410, an image immediately after the ingredients are added shows the cooking mode as "bake," the cooking time as 0 minutes, the cooking temperature as 0 degrees Celsius, and the cooking status as "raw."

[0111] S411 is an image of a state of heating at 180 degrees for 5 minutes, with the cooking mode displayed as baking, the cooking time displayed as 5 minutes, the cooking temperature displayed as 180 degrees, and the cooking state displayed as "Medium".

[0112] S412 is an image of a state of heating at 180 degrees for 10 minutes, with the cooking mode displayed as baking, the cooking time displayed as 10 minutes, the cooking temperature displayed as 180 degrees, and the cooking state displayed as "Well-done".

[0113] Therefore, when the user selects a specific image, the control module 150 can use the heating temperature and mode according to the specific image in cooking. In addition, the control module 150 can refer to the physical characteristics (such as size, area, weight, etc.) of the ingredients included in the prototype and the physical characteristics of the ingredients to be cooked when comparing the images.

[0114] That is, the control module 150 may set the cooking temperature or cooking time of the main body 190 by using the weight or size information of the food included in the meta information of the image information, or by comparing the size of the food in the image information with the size of the food put into the main body 190 .

[0115] For example, the time it takes to reach optimal cooking conditions may differ when heating 200g of pork belly versus 500g. Therefore, the physical characteristics of the ingredients in the cooking image (weight, size, area, etc.) can be compared with the physical characteristics of the ingredients currently being cooked, reflecting time or temperature factors, thereby improving the accuracy of the image comparison results.

[0116] If the user selects the image in step S411 to add seasoning to the cooking process and sets the user to receive reminders, the control module 150 will cook the ingredients based on the metadata of the image in step S411. Furthermore, if the cooking process reaches a state where the ingredients are highly similar to the image in step S411, the control module 150 will issue a reminder. In this case, the control module 150 can use the weight of the ingredients as a reference to predict the time when the corresponding reminder will be issued.

[0117] For example, Figure 8 S411 is an image of a case where 200g of food is heated at 180 degrees for 5 minutes. When 500g of food is put in, the control module 150 can calculate the proportional time based on the weight as "5 minutes × (500g / 200g) - a". In this case, a is the time to prevent the food from being overcooked. For example, in the case of pork belly, a can be 2 minutes, in the case of shrimp, a can be 3 minutes, and so on. Of course, a can be a positive or negative value depending on the material. That is, in order to prevent different types of food from being overcooked or undercooked, the control module 150 can increase or decrease the proportional time calculated based on weight or size, etc.

[0118] In addition, the control module 150 may increase the precision of image processing closer to the prediction time, and compare the image of the prototype with the current image of the food to improve accuracy.

[0119] The control module 150 can additionally construct a learning database that includes time-lapse photography images or pictures of each ingredient. Of course, the learning data can include meta-information of cooking conditions (mode, time, temperature, etc.) associated with each image or picture.

[0120] The control module 150 of one embodiment of the present invention records and stores the cooking process of ingredients cooked in the past by the user using the cooking devices 100 , 150 , so that the user can set a reminder at a specific cooking time point.

[0121] In addition, when the user instructs cooking based on a specific cooking time point, the control module 150 can control the cooking devices 100 and 150 to reflect the cooking conditions performed up to that time point to cook the ingredients.

[0122] According to an embodiment of the present invention, the control module 150 may perform image processing on the ingredients actually needed and used by the user as recognition objects, thereby continuously improving the ingredient recognition performance corresponding to the user's cooking method.

[0123] In addition, by configuring image processing based on information about the ingredients used by the user, it is possible to reduce the time required for ingredient recognition and comparison.

[0124] Therefore, the image recognition performance of the control module 150 can be improved, the possibility of misrecognition can be reduced, and cooking can be accurately guided and reminders can be executed.

[0125] In addition, the items of food that the user wants to identify can be added or removed, thereby further improving the recognition accuracy.

[0126] Figure 9 This figure shows the result of ingredient recognition displayed by the display module according to one embodiment of the present invention. Three food images are displayed as alternatives, corresponding to the ingredients placed. If the user selects the previous cooking image related to pork belly in the middle, a triangular button (PLAY) similar to 120b appears, allowing the corresponding image to be played. Furthermore, a progress bar with a length of 5 minutes corresponding to the cooking image can be located at the bottom of 120b, allowing the user to confirm a specific time point in the image.

[0127] Alternatively, multiple images may be displayed as in 120c, so that a portion of the main image in the cooking image is displayed, allowing the user to select a specific image. The user may select a specific image (e.g., the image on the right side of 120c) to set a reminder.

[0128] During the recognition process, control module 150 extracts a feature vector from the image of the placed ingredients and calculates the similarity between the extracted feature vector and the feature vector of the image information to retrieve the image information corresponding to the ingredients. At this point, control module 150 can analyze the user's speech near cooking device 100, convert it into text, and include it in a feature vector.

[0129] This not only enables the recognition of specific ingredient names, but also improves the accuracy of ingredient recognition through descriptions of historical cooking processes, such as the utterance "like the shrimp I had last week."

[0130] In addition, the control module 150 can accumulate and store the user's Figure 9In addition, in addition to the upper part of the main body 190, the camera can also be configured on the side or the side top surface, and the images captured in multiple directions can be used to improve the accuracy of recognition.

[0131] Figure 10 This diagram illustrates a process in which a cooking appliance provides a reminder or a cooking-related image to a terminal device via a server according to an embodiment of the present invention.

[0132] The control module 150 of the cooking device 100, 200 photographs and identifies the food (S310). Figure 4 In addition, the control module 150 of the cooking device 100 or 200 retrieves the image information corresponding to the food (S421). The control module 150 of the cooking device 100 or 200 transmits the retrieved image information to the terminal device 600 (S422), and the terminal device 600 displays the image information.

[0133] In addition, the user selects a specific time point from the video information displayed on the terminal device 600 and selects whether to receive a reminder at that time point (S423). The terminal device 600 transmits the selected information (specific time point and reminder) to the cooking device 100, 200 (S424). Then, the cooking device 100, 200 executes Figure 4 After the steps S360 and S380, if the reminder is set, a reminder message is transmitted to the terminal device 600 (S425). Thereafter, the terminal device 600 executes the reminder (S426).

[0134] The control module 150 of the cooking apparatus 100 or 200 and the terminal device 600 may directly transmit and receive data. Alternatively, the server 500 may serve as a communication medium between the control module 150 of the cooking apparatus 100 or 200 and the terminal device 600.

[0135] On the other hand, according to another embodiment of the present invention, after the control module 150 of the cooking device 100 or 200 takes a picture of the food, it transmits the picture to the server 500, which can retrieve the corresponding image information. In this case, the server 500 can store the image information according to the identification information of the corresponding cooking device 100 or 200.

[0136] In addition, images captured by the cooking devices 100 and 200 during the cooking process may also be uploaded to the server 500 .

[0137] Figure 11 This is a diagram of a server according to an embodiment of the present invention identifying ingredients or comparing them with existing cooking images or images in a database to confirm whether cooking conditions are met. Figure 4 、 Figure 5 、 Figure 6 Part of the process can be performed on the server 500. Figure 4 .

[0138] When a user places ingredients into cooking apparatus 100 or 200, camera 110 captures the ingredients, and control module 150 transmits the captured images of the ingredients to server 500 ( S310a , S310b ). Server 500 uses the received images to identify the ingredients ( S310c ). Furthermore, server 500 retrieves stored cooking images or database images related to the ingredients ( S330a ) and transmits the retrieved images or videos to cooking apparatus 100 or 200 ( S330b ).

[0139] The display module 120 of the cooking apparatus 100 or 200 displays a cooking image or a DB image ( S330 c ).

[0140] The control module 150 can display the displayed cooking video or the database image in a time-lapse video format. If the user selects a specific time point in the cooking video or a specific image in the database image, the control module 150 transmits the information of the selected time point or image to the server 500 (S350a, S350b). The server 500 stores the transmitted information (S350c).

[0141] The cooking devices 100 and 200 then cook. Under the control of the control module 150, the camera 110 captures the cooking ingredients (S360a). Furthermore, the control module 150 transmits the captured images or videos to the server 500 (S360b). During this process, the control module 150 can use the metadata included in the images or videos at the selected time point to set cooking conditions.

[0142] The server 500 compares the captured image or video with an image or a database image of a specific time point previously selected by the user for time-lapse photography (S380a). Furthermore, the server 500 compares the cooking state of the ingredients to see if it is similar to the image at the specific time point. If so, the server 500 generates a notification or alarm message instructing to interrupt cooking or issue a reminder, and transmits it to the cooking devices 100 and 200 (S380b and S380c).

[0143] Alternatively, if cooking needs to be continued, the server 500 may generate a message to guide the cooking status. For example, as one embodiment, the message to guide the cooking status is a message to guide the time or the cooking progress, such as "5 minutes left to the set cooking state" or "60% of the cooking is now completed."

[0144] Then, the control module 150 receiving the message may interrupt the cooking or execute a reminder (S380d). If S380b determines that the two are not similar, the steps S360a to S380b may be repeated.

[0145] Figure 11 The embodiments may also be applicable to Figure 5 or Figure 6 That is, the server 500 can execute a part of the functions previously executed by the control module 150 and transmit the results thereof to the control module 150.

[0146] In addition, the server 500 may store images of various ingredients in a database, and when new ingredients are uploaded, the database may be updated with related images or videos.

[0147] In addition, the image information captured by the cooking devices 100 and 200 is stored together with the identification information of the cooking devices 100 and 200, so that the recognition rate can be improved in the subsequent process of identifying the food.

[0148] Figure 11 The embodiment includes a process (S310a-S330b) in which the server 500 uses the image of the food transmitted from the cooking device 100, 200 to perform the food identification process and retrieve the image information corresponding to the identified food, and then transmits it to the control module.

[0149] Figure 11 The embodiment includes a process (S350a-S360a) of controlling the main body of the cooking device to perform cooking after the control module 150 transmits information of a specific time point or image in the received image information to the server 500.

[0150] Figure 11 An embodiment includes a process (S360a~S380c) in which the control module 150 transmits images captured during the cooking process of the cooking device 100, 200 to the server 500, and the server 500 uses the received images to generate a message for interrupting cooking or executing a reminder or guiding the cooking status, and transmits it to the control module.

[0151] Figure 11 The embodiment includes a process in which the control module 150 controls cooking or performs a reminder function or displays a guide message according to the received message (S380d).

[0152] All the constituent elements constituting the embodiments of the present invention are described as being combined into one or a combination of operations, but the present invention is not limited to these embodiments. Within the scope of the purpose of the present invention, all the constituent elements may be selectively combined with one or more constituent element operations. In addition, although all the constituent elements can be implemented as independent hardware, some or all of the constituent elements may also be selectively combined to form a computer program with a program module, which can execute some or all of the combined functions in a single or multiple hardware. The codes and code segments constituting the computer program can be easily derived by those skilled in the art. Such computer programs can be stored in a computer-readable storage medium, read and executed by a computer to implement an embodiment of the present invention. Storage media include magnetic recording media, optical recording media and semiconductor memory devices. The computer program that implements an embodiment of the present invention also includes a program module transmitted in real time by an external device.

[0153] The above description is centered around the embodiments of the present invention, but various changes or modifications may be made by those skilled in the art. Therefore, it should be understood that the above changes and modifications are all included within the scope of the present invention without departing from the scope of the present invention.

[0154] Description of Reference Numerals

[0155] 100, 200: Cooking equipment 110: Camera

[0156] 120: Display module 150: Control module

[0157] 500: Server 600: Terminal device

Claims

1. A cooking device is a cooking device for cooking ingredients, wherein: include: A main body, heating the ingredients; a camera, for photographing the food placed in the main body; as well as The control module executes an ingredient recognition process on the ingredient image captured by the camera, and after outputting image information corresponding to the recognized ingredient to the display module of the cooking device or transmitting the image information to a terminal device linked to the cooking device, sets the cooking conditions of the main body or monitors the cooking status of the main body corresponding to a time point or image selected from the image information to perform a reminder function.

2. The cooking device according to claim 1, wherein The image information is image information previously captured and stored by the cooking device during the cooking process of the food placed in the main body.

3. The cooking device according to claim 1, wherein The image information is image information including an image of a cooking state of food received and stored by the control module from another device.

4. The cooking device according to claim 1, wherein The image information includes one or more of cooking temperature and cooking time as meta-information.

5. The cooking device according to claim 1, wherein The control module sets a cooking temperature or a cooking time of the main body using meta information included in the image information.

6. The cooking device according to claim 1, wherein The control module uses the weight or size information of the ingredients included in the metadata of the image information to set the cooking temperature or cooking time of the main body, or compares the size of the ingredients in the image information with the size of the ingredients placed in the main body to set the cooking temperature or cooking time of the main body.

7. The cooking device according to claim 1, wherein After the control module controls the camera to capture the cooking status of the subject, if cooking of the subject is completed, the control module stores the image information captured by the camera or replaces previously stored image information with the captured image information.

8. The cooking device according to claim 1, wherein In the identification process, After extracting a feature vector from the image of the placed food, the control module calculates a similarity between the feature vector and a feature vector of the image information to retrieve image information corresponding to the food.

9. The cooking device according to claim 1, wherein After performing image comparison between an image of the cooking status of the placed ingredients and an image at a time point selected from the image information, the control module interrupts cooking, issues a reminder through the display module, or transmits a reminder message to the terminal device when the similarity between the image at the selected time point and the captured image reaches a benchmark value or above.

10. The cooking device according to claim 9, wherein The control module outputs the image of the food being cooked in the main body through the display module, or transmits the image to the terminal device.

11. A method for controlling a cooking process using visual cooking guidance, wherein: include: The step of photographing the food placed in the main body of the cooking device by a camera of the cooking device; The control module of the cooking device performs an ingredient recognition process on the ingredient image captured by the camera to retrieve image information corresponding to the identified ingredient; The control module outputs the image information to a display module of the cooking device, or transmits the image information to a terminal device linked to the cooking device; as well as The control module sets the cooking conditions of the main body corresponding to the time point or image selected from the image information, or monitors the cooking status of the main body to perform a reminder function.

12. The method for controlling a cooking process using visual cooking guidance according to claim 11, wherein: The image information is image information previously photographed and stored by the cooking device of the cooking process of the food placed in the main body, or image information containing an image of the cooking state of the food received and stored by the control module from another device.

13. The method for controlling a cooking process using visual cooking guidance according to claim 11, wherein: The image information includes one or more of cooking temperature and cooking time as meta-information, or includes weight or size information of the food in the image information as meta-information.

14. The method for controlling a cooking process using visual cooking guidance according to claim 11, wherein: Also includes: After the control module controls the camera to capture the cooking status of the subject, if cooking of the subject is completed, the control module stores the image information captured by the camera or replaces the previously stored image information with the captured image information.

15. The method for controlling a cooking process using visual cooking guidance according to claim 11, wherein: Also includes: After the control module performs image comparison using an image of the cooking status of the placed ingredients and an image at a time point selected from the image information, when the similarity between the image at the selected time point and the captured image reaches a benchmark value or above, the cooking is interrupted or a reminder is issued through the display module or a reminder message is transmitted to the terminal device.

16. The method for controlling a cooking process using visual cooking guidance according to claim 11, wherein: Also includes: The control module outputs the image of the food being cooked in the main body through the display module, or transmits the image to the terminal device.

17. A method for controlling a cooking process using visual cooking guidance, wherein: include The step of photographing the food placed in the main body of the cooking device by a camera of the cooking device; The control module of the cooking device transmits the food image captured by the camera to the server; The server performs an ingredient recognition process using the transmitted image, and after retrieving image information corresponding to the identified ingredient, transmits the image information to the control module; The control module controls the main body of the cooking device to perform cooking steps after transmitting information of a specific time point or image in the image information to the server; After the control module transmits the image captured by the cooking device during cooking to the server, the server generates a message for interrupting cooking or issuing a reminder or guiding the cooking status using the received image, and transmits the message to the control module; as well as The control module controls cooking or performs a reminder function or outputs a guide message on the display module of the cooking device according to the received message.

18. The method for controlling a cooking process using visual cooking guidance according to claim 17, wherein: The image information is image information that the cooking device has previously transmitted to the server after capturing a cooking process of the food placed in the main body, or is image information containing an image of the cooking state of the food received and stored by the server from another device.

19. The method of providing cooking guidance and controlling the cooking process by using images according to claim 17, wherein: The image information includes one or more of cooking temperature and cooking time as meta-information, or includes weight or size information of the food in the image information as meta-information.

20. The method of claim 17, wherein: Also includes: The server performs image comparison using an image of the cooking status of the placed ingredients and an image at a time point selected from the image information, and then determines whether the similarity between the image at the selected time point and the captured image reaches a baseline value or above.

Images