Device for controlling cooking and method therefor

Abstract

Proposed is a method for controlling cooking, and the method involves: periodically acquiring image data of an object put into a cooking chamber; periodically determining a cooking state according to a color change of the object based on the acquired image data; and terminating automatic cooking control as the cooking state reaches a set cooking state. The determining of the cooking state may be performed on one piece of the acquired image data in parallel by using an image processing model and a deep learning model, and may be based on a combination of the result according to the image processing model and the result according to the deep learning model.

Description

Device for cooking control and method therefor

[0001] The present invention relates to a cooking control device and method therefor. More specifically, the present invention relates to a cooking control device and method therefor based on an image captured of the interior of a cooking chamber of a cooking appliance.

[0002] An oven is a home appliance that heats and cooks food by placing it in a cooking chamber formed within a cavity. To heat the food, the oven includes at least one heating source. Depending on the heating method, heating sources can be categorized into high-frequency heating sources, radiant heating sources, and convection heating sources. The operation of the heating source is controlled according to the type of food being cooked and, in essence, the cooking method. Recently released ovens offer multiple cooking methods, increasing accessibility even for those unfamiliar with oven operation.

[0003] Recent ovens are increasingly incorporating smart features. Smart ovens connect to smartphones or tablets via Wi-Fi and Bluetooth, allowing remote control. Furthermore, smart ovens can download recipes and monitor the cooking process through dedicated apps. Furthermore, smart ovens integrate with voice assistants, allowing you to control the oven with voice commands.

[0004] In addition, attempts are being made to provide more diverse services using images acquired using image sensors, such as cameras that capture the cooking area inside an oven, to provide user convenience.

[0005] The present invention proposes a method for determining the cooking status of an object or cooking ingredient placed in a cooking appliance.

[0006] In addition, the present invention proposes a method for continuously controlling automatic cooking if the cooking state of an object or cooking ingredient placed in a cooking appliance does not reach a set cooking state.

[0007] In addition, the present invention proposes a method for automatically terminating cooking when the cooking state of an object or cooking ingredient placed in a cooking appliance reaches a set cooking state.

[0008] In addition, the present invention proposes a method of automatically terminating cooking or outputting a notification such as a caution or warning when an object or cooking ingredient placed in a cooking appliance is in an abnormal state or the cooking state cannot be determined.

[0009] The problems to be solved by the present invention are not limited to the problems to be solved above, and other problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0010] A method for cooking control is proposed, the method including periodically acquiring image data of an object brought into a cooking room, periodically determining a cooking state according to a color change of the object based on the acquired image data, and terminating automatic cooking control when the cooking state reaches a set cooking state, wherein the judgment of the cooking state is performed in parallel using an image processing model and a deep learning model for one acquired image data, and may be based on a combination of a result according to the image processing model and a result according to the deep learning model.

[0011] A control device for a cooking appliance for controlling cooking is proposed, the device including a processor and a user interface, wherein the processor periodically acquires image data of an object brought into a cooking chamber, periodically determines a cooking state according to a color change of the object based on the acquired image data, and terminates automatic cooking control when the cooking state reaches a set cooking state, and the judgment of the cooking state is performed in parallel using an image processing model and a deep learning model for one acquired image data, and may be based on a combination of a result according to the image processing model and a result according to the deep learning model.

[0012] The above problem solving methods are only some of the embodiments of the present invention, and various embodiments reflecting the technical features of the present invention can be derived and understood by a person having ordinary knowledge in the relevant technical field based on the detailed description of the present invention described below.

[0013] The present invention has the following technical effects.

[0014] The cooking status of ingredients or food inside the cooking device can be recognized, and automatic cooking control can be provided accordingly until the set cooking status is reached.

[0015] Additionally, by using the image processing pipeline and deep learning pipeline in parallel to determine the cooking status, accurate cooking status determination is possible.

[0016] The effects according to the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the detailed description of the invention below.

[0017] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide embodiments of the present invention and, together with the detailed description, explain the technical idea of ​​the present invention.

[0018] Figure 1 is a perspective view of a cooking appliance according to the present invention.

[0019] Figure 2 illustrates a procedure for automatic cooking control according to the present invention.

[0020] Figure 3 shows examples of multiple cooking states.

[0021] Figure 4 illustrates a user interface for inputting a preset recipe according to the present invention.

[0022] Figure 5 illustrates a specific process for judging the cooking status according to the present invention.

[0023] Figure 6 illustrates a process in an image processing model according to the present invention.

[0024] Figure 7 illustrates a process for detecting an abnormal state according to the present invention.

[0025] FIG. 8 is a diagram for explaining a procedure for determining weights applied to an image processing model and a deep learning model according to the present invention.

[0026] Figure 9 illustrates a flowchart of automatic cooking control according to the present invention.

[0027] Figure 10 illustrates a block diagram of a cooking appliance control device according to the present invention.

[0028] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0029] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0030] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0031] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0032] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0033]

[0034] Figure 1 is a perspective view of a cooking appliance according to the present invention.

[0035] The case (10) is provided in a shape having an internal space and is open at the front. For example, the case (10) may be formed in a predetermined box shape.

[0036] A cooking chamber (11) in which food is cooked is formed inside the case (10). A grill (not shown) for placing food on the inside of the cooking chamber (11) is provided.

[0037] A plurality of heating sources (or heaters) are installed inside the case (10) and outside the cooking chamber (11) to provide heat for cooking food.

[0038] An input unit (12) and a display unit (13) are provided on the outside of the case (10). The input unit (12) may be formed as a dial-type button so that the user can select various operation modes, set temperatures, and operation times. Here, the set temperature is the temperature input by the user for cooking food.

[0039] The display unit (13) displays certain information so that the user can determine the status of the cooking appliance (1).

[0040] The door (20) is installed on the open front of the case (10) and is arranged to open and close the cooking chamber (11). That is, the cooking chamber (11) can be opened and closed by the door (20). For convenience of explanation, the installation structure and locking device of the door (20) are omitted and illustrated.

[0041] The door (20) is installed on the front of the case (10) so as to be rotatable forward. In addition, the door (20) may be provided with a handle (21) that a user can grasp and rotate.

[0042] Meanwhile, although not shown, a temperature sensor (not shown) may be additionally installed inside the cooking chamber (11). The temperature sensor measures the internal temperature of the cooking chamber (11), i.e., the temperature inside the oven. The measured temperature inside the oven is transmitted to the control unit, which will be described later. For example, the temperature sensor may be a thermostat.

[0043] Additionally, an image sensor (17), such as a camera, for capturing images by photographing the inside of the cooking room (11) may be installed inside the cooking room.

[0044]

[0045] Figure 2 illustrates a procedure for cooking control according to the present invention. The cooking control illustrated in Figure 2 may be performed by a cooking appliance (1) or a control device (100, see Figure 9) of the cooking appliance, or by a processor of the cooking appliance (1) or the control device (100) of the cooking appliance. Hereinafter, it will be described as being performed by a processor.

[0046] The cooking appliance (1) according to the present invention may include a combination oven that combines a conventional microwave oven function and an oven function, but the present invention is not limited thereto.

[0047] The processor can periodically acquire image data of an object placed in the cooking chamber (11) (S10). Here, the object may include a plate, a bowl, or ingredients placed on a plate or bowl. Meanwhile, a turntable for placing the object is provided within the cooking chamber (11), and image data of the object can be acquired as the object rotates on the turntable. This enables acquisition of image data for the object from various angles, thereby enabling acquisition of more accurate image data for the object.

[0048] The reason for periodically acquiring video data is to continuously monitor the cooking status while performing automatic cooking control.

[0049] Meanwhile, the processor assumes that it can determine from the acquired image data whether the object is a cooking ingredient or the type of dish the user desires. Alternatively, it assumes that the type of dish is input by the user, and thus the processor can recognize the type of dish.

[0050] The processor can periodically determine the cooking status based on color changes in the object based on acquired image data (S20). Bakery products, especially bread, have a characteristic of changing color as they progress from raw materials (e.g., flour dough) to finished food. Therefore, the present invention proposes monitoring color information over time for objects placed in the cooking chamber (11).

[0051] Meanwhile, in addition to color changes, changes in volume and shape can be used to determine the cooking status.

[0052] Additionally, the deep learning model can extract and / or learn feature vectors reflecting various characteristics, such as color changes, volume changes, and texture changes, based on acquired image data to determine the cooking status. In other words, the processor can use feature vectors to determine the cooking status.

[0053] The cooking state can be expressed as a numerical value as described later, but the present invention is not limited thereto.

[0054] The processor may terminate or stop automatic cooking control when the cooking state reaches the set cooking state, i.e., the target cooking state (S30). When automatic cooking control is terminated, the processor may output an audible or visual notification, such as a warning sound, a guidance sound, or a message.

[0055] Meanwhile, the judgment of the cooking state (S20) may be performed by a server connected to the cooking device (1) or the like via a network, rather than by the cooking device (1). Accordingly, the judgment of the cooking state below may be performed by the control device of the cooking device (1) or the cooking device (10), or the processor thereof, as well as by the server, and this is included in the scope of the present invention.

[0056]

[0057] Figure 3 shows examples of multiple cooking states.

[0058] Figure 3 illustrates examples of cooking states for bakery products such as bread. While Figure 3 illustrates a total of eight cooking states, the present invention is not limited thereto. More or fewer cooking states may be utilized. Furthermore, neither the number of cooking states nor the specific cooking states (e.g., browning levels 1, 2, 3, etc.) limit the present invention.

[0059] Additionally, although not shown, the cooking device (1) or the user interface of the cooking device (1) may display multiple cooking states that can be set by the user and may receive a selection for one of the multiple cooking states. Through the user's selection of the cooking state, the cooking state for automatic cooking control can be set.

[0060] Additionally, the cooking state can be quantified as a cooking state value. The eight cooking states illustrated in Fig. 3 can be set to have respective cooking state values ​​as follows.

[0061] Cooking Status Cooking Status ValueT11.0T22.0T33.0T44.0T55.0T66.0T77.0T88.0

[0062] Accordingly, the cooking state according to the present invention can be set to have one state value among a plurality of state values ​​having a certain interval.

[0063] As an example, assume that the cooking state for any one of T3, T4, and T5 can be set by the user.

[0064] As another example, settings can be performed not only for cooking conditions but also for preset recipes. Alternatively, preset cooking conditions can be set for each preset recipe.

[0065] A preset recipe is a preset cooking method or cooking control method provided by a cooking device (1), which provides cooking for multiple dishes. Each preset recipe may be composed of a combination of set (heating) temperatures and times. Additionally, a preset recipe may include a combination of water injection and water injection times.

[0066] Preset recipes are output through a user interface (160) as shown in Fig. 4, and can be selected and set by a user through touch input. The preset recipes shown in Fig. 4 are for bread, and bread, rolls, pastries, and specialty items can be displayed on the user interface (160).

[0067]

[0068] Figure 5 illustrates a specific process for judging the cooking status according to the present invention.

[0069] The cooking state determination illustrated in FIG. 5 may be performed by the cooking appliance (1) or the control device (100, see FIG. 10) of the cooking appliance, or by the processor of the cooking appliance (1) or the control device (100) of the cooking appliance. Hereinafter, it will be described as being performed by the processor.

[0070] The processor can process in parallel an image processing model-based pipeline (S21) and a deep learning model-based pipeline (S22) for cooking status judgment (S20).

[0071] Typically, the shape of objects (or dishes) within a cooking device (1) is not standardized. Even for the same dish, recipes can vary from user to user, resulting in variations in the shape, quantity, and volume of ingredients. Therefore, both image processing model-based and deep learning model-based pipelines are used.

[0072] Image processing models can be used to assess cooking status by analyzing image data using color information. As mentioned earlier, changes in volume over time can be used to assess cooking status. Accordingly, image processing models can be used to assess cooking status by analyzing image data using volume information about objects within the image.

[0073] The deep learning model can be used to determine the cooking status using the results of previously performed cooking control. That is, according to the deep learning model, the processor can analyze image data acquired inside the cooking chamber (11) based on previously acquired image data corresponding to an object brought into the cooking chamber (11), and determine the cooking status therefrom. Alternatively, the processor can analyze image data acquired inside the cooking chamber (11) based on previously acquired image data corresponding to a preset recipe or cooking status set for an object brought into the cooking chamber (11), and determine the cooking status therefrom.

[0074] Image processing models and deep learning models can perform complementary functions. Image processing models can analyze image data to extract color information (or volume information) and determine cooking status.

[0075] Furthermore, the image processing model determines the cooking status by comparing or averaging the current image data with color information derived from previously acquired image data, making it possible to determine the cooking status even with a small amount of training data. Here, the color information output may include features based on color information extracted by applying the image processing model to a previously acquired database for each dish or recipe.

[0076]

[0077] On the other hand, deep learning models are learning-based models, so their accuracy can increase with the amount of training data (i.e., the amount of previously acquired image data). Accordingly, the processor can use the two models in parallel to determine cooking status.

[0078] The processor can combine, fuse, or merge the results of the image processing model and the results of the deep learning model to produce a cooking status value (S23). The cooking status value is a numerical value representing the cooking status. The target cooking status set by the user can also be digitized and represented as a target cooking status value.

[0079] Cooking status values ​​can be:

[0080]

[0081] Here, cv is a cooking state value according to the image processing model, dl is a cooking state value according to the deep learning model, α is a weight applied to the cooking state value according to the image processing model, and β is a weight applied to the cooking state value according to the deep learning model.

[0082] It can be defined as α+β=1. As mentioned earlier, since they have complementary functions or effects, either one of the two models' results can be used, or a portion of each of the two models' results can be combined and used.

[0083] The cooking state value of each model can be set so that it can be calculated as a value corresponding to the numerical value of each cooking state shown in Fig. 3.

[0084] Each cooking state can be quantified and represented by a value. For example, the eight states (T1 to T8) illustrated in FIG. 3 can be set to have cooking state values ​​ranging from 0.0 to 8.0, such that each state can be set to have a range of magnitude 1. The undercooked state (T1) can be set from 0.0 to 1.0, the undercooked state (T2) from 1.0 to 2.0, and the burnt state (T8) from 7.0 to 8.0.

[0085] Accordingly, the image processing model (S21) or the deep learning model (S22) also processes the image data to calculate the cooking status value, and the cooking status value can be determined as a value within a preset range, for example, between 0.0 and 8.0.

[0086] Meanwhile, the image processing model produces good results in judging the cooking state even when the number of previously acquired image data is small, while the deep learning model is expected to have problems in the reliability of the results in judging the cooking state if the number of previously acquired image data (or cooking state or cooking state value) is small. Therefore, the weight (α) applied to the image processing model can be set higher as the number of previously acquired image data is small, and the weight (β) applied to the deep learning model can be set higher as the number of previously acquired image data is large.

[0087] Additionally, the processor can filter the produced cooking status values.

[0088]

[0089] Figure 6 illustrates a process in an image processing model (S21) according to the present invention.

[0090] The cooking state determination illustrated in Fig. 6 may be performed by the cooking appliance (1) or the control device (100, see Fig. 10) of the cooking appliance, or by the processor of the cooking appliance (1) or the control device (100) of the cooking appliance. Hereinafter, it will be described as being performed by the processor.

[0091] The processor may crop the acquired image data to include a region of interest (S211). This is to use only the portions deemed significant for determining the cooking status for image processing.

[0092] The processor can apply a filter to the cropped image data (S212). For example, in the case of bread cooking, the brown color information of the object (i.e., the ingredient or the dish) can be detected to determine the cooking status. Accordingly, the processor can apply a brown filter to the cropped image. The brown filter can activate only the portions of the image data that contain brown colors. In other words, the color represented by each pixel can be changed using the H (color) value in the HSV color space.

[0093] A brown filter can be used for baking, and other color filters may be applied depending on the ingredient or dish.

[0094] For reference, the HSV color space is a method for expressing color, representing color with three elements: hue, saturation, and value. Hue represents the type of color and is expressed as an angle from 0 to 360 degrees. Red is defined as 0 degrees, yellow as 60 degrees, green as 120 degrees, blue as 240 degrees, and purple as 300 degrees. Hue represents the basic type of color and reflects the cyclical nature of color. The advantage of the HSV color space is that it is similar to the way humans perceive color. For this reason, it is frequently used in various fields such as image processing, computer vision, and graphic design. The HSV color space allows for intuitive color manipulation, making color adjustment work easy.

[0095] Additionally, the processor can segment only the food or ingredient area within the cropped image data through filtering.

[0096] Then, the processor can perform feature extraction on the image data (S213).

[0097] For example, when filtering using color information (S212), the processor can perform color information analysis, i.e., color-based feature extraction. The processor can perform color information analysis using information in multiple color spaces (i.e., color values ​​of HSV, Lab, and RGB). In addition, the processor can average color information analyzed through previously acquired image data and color information analyzed through currently acquired image data for multiple frames. In other words, the processor can perform filtering by averaging color information for multiple frames.

[0098] When analyzing image data through color-based feature extraction, the cooking status value calculated in S23 of FIG. 5 described above is ultimately obtained as a value corresponding to or related to color information in the image data.

[0099] The value obtained by feature extraction can be used as the result value (cv) according to the image processing model described above. However, as described above, if the (target) cooking state value corresponding to each (target) cooking state is predefined as in FIG. 3, the cooking state value calculated in S23 of FIG. 5 must also have a value within the predefined range, and therefore, additional processing may be required for the value obtained by feature extraction (i.e., the color information value).

[0100] For further processing, reference data that can match feature extraction values ​​based on currently acquired image data to cooking status values ​​is required. When feature extraction is performed using color information, color change information of image data acquired while performing previously performed cooking control can be used as reference data. The color change information may include color information values ​​of objects (or ingredients, dishes, etc.) that change over time according to the cooking control. The color change information may be acquired and pre-stored for each preset recipe or dish type. The pre-stored color change information may be referred to as reference color change information. In addition, the reference color change information may include the color information value and the cooking status value matched thereto. For example, if the color information value is “a”, the cooking status value matched thereto may be determined as “b”.

[0101] The processor can receive reference color change information from an external device such as a server and store it in storage such as memory.

[0102] Additionally, the reference color change information can be continuously updated using color change information based on image data acquired while the cooking appliance (1) or the cooking appliance control device (100) performs (automatically or manually) cooking control. The processor can transmit the color change information based on image data acquired while performing cooking control to an external device such as a server so that it can be shared with other users.

[0103] The processor can acquire color information values ​​from periodically acquired image data (S213) and compare the color information values ​​with reference color change information. In this case, the reference color change information may correspond to an object brought into the cooking chamber (11) included in the acquired image data. If a preset recipe is selected or input, the reference color change information may correspond to a preset recipe or type of dish corresponding to the acquired image data.

[0104] More specifically, the processor can calculate a distance between the reference color change information and the color information value calculated from the acquired image data. In addition, the processor can compare the time information of the reference color change information corresponding to the calculated color information with the cooking time information during automatic cooking control. If the calculated distance falls within a threshold range, and if a difference between the cooking time information during automatic cooking control and the cooking time information (i.e., the reference cooking time information) of the corresponding color information value of the reference color change information (i.e., the reference color information) falls within the threshold range, the processor can determine the cooking state value during automatic cooking control as the cooking state value corresponding to the reference color information or the reference cooking time information (S214).

[0105]

[0106] Figure 7 illustrates a process for detecting an abnormal state according to the present invention.

[0107] The cooking state determination illustrated in Fig. 7 may be performed by the cooking appliance (1) or the control device (100, see Fig. 10) of the cooking appliance, or by the processor of the cooking appliance (1) or the control device (100) of the cooking appliance. Hereinafter, it will be described as being performed by the processor.

[0108] The processor can periodically acquire image data within the cooking room (11) (S10).

[0109] Based on the acquired image data, the processor can determine whether an object (or ingredient, food) introduced into the cooking chamber (11) is a cooking target (S11). If it is determined not to be a cooking target, the processor can output a warning, caution, or other notification to indicate that automatic cooking control of the cooking device (1) is not possible (S40). Then, the processor can terminate automatic cooking control or switch to manual cooking control.

[0110] If the received object is determined to be a cooking target, the processor can determine whether the cooking status can be determined (S12). Whether the cooking status can be determined can be determined using reference color change information, which is related to the previously described cooking status determination. The processor can extract color information values ​​from the image data acquired through image data processing and compare the extracted color information with the reference color information in the reference color change information.

[0111] The processor can calculate the distance between the color information values ​​acquired from the reference color change information and the acquired image data. If the calculated distance falls outside a threshold range, the processor determines that the cooking state cannot be determined. If the cooking state cannot be determined, a warning or other notification can be output to indicate that the cooking state cannot be determined or that automatic cooking control is not possible (S40). The processor can then terminate automatic cooking control or switch to manual cooking control.

[0112] If the cooking status can be determined, the processor can determine the cooking status as described above (S20).

[0113] Meanwhile, in a case where the cooking state can be determined, the processor can obtain a probability value indicating the degree to which the color information value or color change information obtained from the obtained image data deviates from the reference color change information. This probability value is referred to as an abnormal state value (ab), and this value has a value between 0 and 1. The closer the color information value or color change information obtained from the obtained image data deviates from the reference color change information, the closer the value will be to 0, and the closer the degree to which it deviates, although within the critical range, the closer the value will be to 1.

[0114] This value can be added to mathematical expression 1 representing the cooking state value of the acquired image data. That is, the value of the abnormal state or the detection of the abnormal state can be included in the judgment of the cooking state (S20).

[0115]

[0116] ab is the non-stationary state value, γ is a constant, and is a rational number greater than 0.

[0117]

[0118] FIG. 8 is a diagram for explaining a procedure for determining weights applied to an image processing model and a deep learning model according to the present invention.

[0119] The procedure for determining the weights to be applied to each model illustrated in Fig. 8 may be performed by the cooking appliance (1) or the control device (100, see Fig. 10) of the cooking appliance, or by the processor of the cooking appliance (1) or the control device (100) of the cooking appliance. Hereinafter, it will be described as being performed by the processor.

[0120] We assume that the sum of the weights (α) applied to the image processing model and the weights (β) applied to the deep learning model is 1.

[0121] Additionally, the determination of the weights to be applied to each model can be performed on a per-dish or per-preset recipe basis.

[0122] The processor can obtain image data taken inside the kitchen (S100).

[0123] The processor can perform a performance evaluation of an image processing model on the acquired image data (S101) and also perform a performance evaluation of a deep learning model on the image data acquired in parallel (S102). More specifically, the processor can evaluate an indicator (P) indicating the reliability of the image processing model. cv ) and the processor also obtains an indicator (P) indicating the reliability of the deep learning model. dl ) can be obtained.

[0124] Methods for obtaining reliability indicators can be broadly categorized into static methods, dynamic methods, or methods using Kalman filters or Bayesian networks.

[0125] A static method can measure reliability by model performance against a pre-obtained DB (model evaluation DB) for each dish or preset recipe.

[0126] Additionally, dynamic methods can measure reliability based on the final probability value of each model. For example, reliability can be measured using the probability value obtained by applying a probability function to the distance value calculated in an image processing model and the maximal probability value in a deep learning model.

[0127] Alternatively, dynamic methods can measure reliability using voting methods within ensemble techniques. For example, reliability can be measured using estimated values ​​using hard voting or soft voting methods.

[0128] Then, the processor can compare the difference between the two performance indicators with a threshold value (S201, S202). The threshold value can be determined as a relatively large value so that the difference between the two performance indicators is greater than a certain threshold value. For example, the threshold value can be 0.5.

[0129] More specifically, the confidence index (P) of the deep learning model dl ) and reliability index (P) of the image processing model cv ) is greater than the threshold, the processor can set the weight (β) applied to the deep learning model to 1 and the weight (α) applied to the image processing model to 0. This is because the reliability index of the deep learning model is relatively higher than the reliability index of the image processing model.

[0130] Additionally, the reliability index (P) of the deep learning model dl ) and reliability index (P) of the image processing model cv ) is less than the threshold value, the processor can set the weight (α) applied to the image processing model to 1 and the weight (β) applied to the deep learning model to 0. This is because the reliability index of the image processing model is relatively higher than that of the deep learning model.

[0131] Additionally, the reliability index (P) of the deep learning model dl ) and reliability index (P) of the image processing model cv ) is neither greater nor less than the threshold, the processor can set the weight (α) applied to the image processing model and the weight (β) applied to the deep learning model based on the ratio of each indicator. For example, the confidence indicator (P) of the deep learning model dl ) and reliability index (P) of the image processing model cv ) is 2 to 3, the weight (α) applied to the image processing model and the weight (β) applied to the deep learning model can be 0.6 and 0.4, respectively.

[0132] The weights set in this way can be applied to the cooking status values ​​according to mathematical expressions 1 or 2.

[0133]

[0134] Figure 9 illustrates a flowchart of a cooking appliance control according to the present invention.

[0135] The flow chart of the cooking device control of Fig. 9 shows an embodiment in which the cooking device (1) or the cooking device control device (100) acquires and transmits an internal image of the cooking chamber (11) and operates a heater or fan according to a cooking control command, and the server (200) performs the cooking status judgment (S20) described above.

[0136] In the following description, the invention is described as being implemented by a cooking appliance (1) or a cooking appliance control device (100) and a server (200), but it is obvious that the invention can be implemented by a processor of the cooking appliance (1) or the cooking appliance control device (100) instead of the cooking appliance (1) or the cooking appliance control device (100).

[0137] The cooking appliance (1) or the cooking appliance control device (100) can receive a preset recipe for an object brought into the cooking room (11) and / or a target cooking state therefor (S81).

[0138] The cooking appliance (1) or the cooking appliance control device (100) can confirm the input preset recipe based on the input preset recipe and the acquired image data (S82). This corresponds to a procedure for confirming whether the input preset recipe is valid or has been entered correctly. The cooking appliance (1) or the cooking appliance control device (100) can use the acquired image data to confirm whether the preset recipe entered by the user corresponds to, is suitable for, or is valid for the object entered.

[0139] The cooking appliance (1) or the cooking appliance control device (100) can transmit the confirmed preset recipe to the server (200) via the network. Meanwhile, if the preset recipe input by the user is not confirmed (i.e., if the preset recipe input by the user does not correspond to, fit, or is not valid for the received object), the cooking appliance (1) or the cooking appliance control device (100) can output a warning or notification and stop the procedure.

[0140] The cooking device (1) or the cooking device control device (100) can periodically transmit acquired image data regarding an object brought into the cooking room (11) to the server (200) via a network (S84).

[0141] The server (200) can periodically determine the cooking status based on image data received from the cooking appliance (1) or the cooking appliance control device (100) (S85). That is, the server (200), rather than the cooking appliance, can perform the judgment of the cooking status described above.

[0142] The cooking appliance (1) or the cooking appliance control device (100) can receive information on the cooking status for a confirmed preset recipe from the server (200) (S86).

[0143] The cooking device (1) or the cooking device control device (100) can perform cooking control according to the cooking status received from the server (200) (S87). At this time, the target cooking status input in S81 can be used.

[0144] A command for cooking control may include a combination of a set temperature (heating or cooking) and time. Furthermore, the command for cooking control may include a combination of water injection and water injection time. Furthermore, the command for cooking control may include a command to terminate or stop cooking control.

[0145] As mentioned above, S84, S85, and S86 of Fig. 8 can be repeated periodically. Accordingly, the cooking control (S87) can also have its control value (temperature or time) change.

[0146]

[0147] Figure 10 illustrates a block diagram of a cooking appliance control device according to the present invention.

[0148] The cooking appliance control device (100) may include a transmitter / receiver (31), a processor (32), a memory (33), and a user interface or HMI (34). The cooking appliance control device (100) may be included in the cooking appliance (1).

[0149] The transceiver (31) can periodically transmit acquired image data inside the cooking room (11) to the server (200). In addition, the transceiver (31) can receive information on the cooking status from the server (200).

[0150] The processor (32) can perform the operations according to the present invention described with reference to FIGS. 2 to 9. As an example, the processor (32) can be configured to periodically acquire image data of an object brought into the cooking chamber, periodically determine the cooking state based on color changes of the object based on the acquired image data, and terminate automatic cooking control when the cooking state reaches a set cooking state.

[0151] Judgment of the cooking status is performed in parallel using an image processing model and a deep learning model for a single acquired image data, and can be based on a combination of the results from the image processing model and the results from the deep learning model.

[0152] The processor (32) may additionally detect an abnormal state from the acquired image data when determining the cooking state based on the acquired image data. Here, the detection of the abnormal state includes acquiring an abnormal state value, and the abnormal state value may include a probability value indicating the degree to which the color change information of the acquired image data deviates from the reference color change information corresponding to the preset recipe or dish set for the received object.

[0153] The processor (32) can obtain a cooking status value to determine the cooking status. The cooking status value is obtained based on a combination of a result value according to an image processing model and a result value according to a deep learning model, and individual weights are applied to the result value according to the image processing model and the result value according to the deep learning model, and the sum of the weights applied to the result value according to the image processing model and the weights applied to the result value according to the deep learning model may be 1.

[0154] The resulting values ​​from the image processing model may include feature values ​​extracted based on color information from the acquired image data. The resulting values ​​from the deep learning model may include values ​​obtained based on previously acquired image data corresponding to a preset recipe or dish set for an object brought into the kitchen.

[0155]

[0156] Even if not described with reference to FIG. 10, the cooking appliance control device (100) of the present invention may perform the operation according to the present invention according to FIGS. 2 to 9 described above.

[0157]

[0158] In addition, as another aspect of the present invention, the operation of the proposal or invention described above may be implemented, performed or executed by a “computer” (a comprehensive concept including a system on chip (SoC) or a (micro) processor, etc.), or may be provided as a code or a computer-readable storage medium storing or including the code or a computer program product, and the scope of the present invention may be extended to the code or the computer-readable storage medium storing or including the code or the computer program product.

[0159]

[0160] The detailed description of the preferred embodiments of the present invention disclosed above has been provided to enable those skilled in the art to implement and practice the present invention. While the above description has been made with reference to preferred embodiments of the present invention, those skilled in the art will appreciate that various modifications and variations of the present invention, as defined by the following claims, are possible. Accordingly, the present invention is not intended to be limited to the embodiments disclosed herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. As a method for controlling cooking, Periodically acquire image data of objects brought into the kitchen, Periodically determine the cooking status based on the color change of the object based on the image data acquired above, When the above cooking state reaches the set cooking state, the automatic cooking control is terminated, Judgment of the above cooking status: It is performed in parallel using an image processing model and a deep learning model for a single acquired image data. A cooking control method based on a combination of results according to the above image processing model and results according to the above deep learning model.

2. In paragraph 1, the judgment of the cooking state is: Includes acquisition of a cooking status value, wherein the cooking status value is acquired based on a combination of a result value according to the image processing model and a result value according to the deep learning model, Individual weights are applied to the result values ​​according to the above image processing model and the result values ​​according to the above deep learning model. A cooking control method wherein the sum of the weights applied to the result values ​​according to the above image processing model and the weights applied to the result values ​​according to the above deep learning model is 1.

3. In paragraph 1, The result value according to the above image processing model is Includes feature values ​​extracted based on color information of the image data obtained above, The result values ​​according to the above deep learning model are: A cooking control method comprising a value acquired based on previously acquired image data corresponding to the object received above.

4. In paragraph 1, the judgment of the cooking state is: A cooking control method, further comprising detection of an abnormal condition.

5. In paragraph 4, the detection of the abnormal state is: Includes acquisition of an abnormal state value, wherein the abnormal state value is: A cooking control method, wherein the color change information of the acquired image data includes a probability value indicating the degree to which the color change information deviates from the reference color change information corresponding to the received object.

6. In paragraph 1, the judgment of the cooking status is based on the cooking status value, The above cooking status values ​​are: It is determined by (1-ab)*(α*cv+β*dl) - γ*ab, Here, ab is an abnormal state value, cv is a result value according to the image processing model, dl is a result value according to the deep learning model, α is a weight applied to the result value according to the image processing model, β is a weight applied to the result value according to the deep learning model, and γ is a constant and a rational number greater than 0, a cooking control method.

7. In paragraph 1, Input a preset recipe for the above-mentioned object and / or a cooking state for the same, The input preset recipe is confirmed based on the input preset recipe and the acquired image data, A cooking control method for transmitting the above-determined preset recipe and the cooking status therefor to a server via a network.

8. In paragraph 7, Periodically transmit the acquired image data to the permanent server, A cooking control method for receiving information on a cooking state determined based on the periodically transmitted image data from the server.

9. In paragraph 1, A cooking control method that outputs a cooking completion notification when the above cooking state reaches a set cooking state.

10. In paragraph 1, A cooking control method that continues automatic cooking control when the above cooking state has not reached a set cooking state and is not in an abnormal state.

11. In paragraph 1, A cooking control method for stopping automatic cooking control and switching to manual cooking control when the above cooking state is abnormal.

12. A computer program stored on a computer-readable medium for performing a method according to any one of claims 1 to 11.

13. As a control device for a cooking appliance for controlling cooking, Includes a processor and a user interface, The above processor: Periodically acquire image data of objects brought into the kitchen, Periodically determine the cooking status based on the color change of the object based on the image data acquired above, When the above cooking state reaches the set cooking state, the automatic cooking control is terminated, Judgment of the above cooking status: It is performed in parallel using an image processing model and a deep learning model for a single acquired image data. A cooking appliance control device based on a combination of results according to the above image processing model and results according to the above deep learning model.

14. In paragraph 13, the judgment of the cooking state is: Includes acquisition of a cooking status value, wherein the cooking status value is acquired based on a combination of a result value according to the image processing model and a result value according to the deep learning model, Individual weights are applied to the result values ​​according to the above image processing model and the result values ​​according to the above deep learning model. A cooking appliance control device in which the sum of the weights applied to the result values ​​according to the above image processing model and the weights applied to the result values ​​according to the above deep learning model is 1.

15. In paragraph 13, The result values ​​according to the above image processing model are: Includes feature values ​​extracted based on color information of the image data obtained above, The result values ​​according to the above deep learning model are: A cooking appliance control device comprising a value acquired based on previously acquired image data corresponding to the object received above.

Images