Electronic device and control method thereof

Abstract

An electronic device includes at least one processor capable of obtaining information about a recipe corresponding to a selected cooking menu. The at least one processor may identify a first cooking step and at least one cooking step to be performed after the first cooking step. The at least one processor may load at least one first neural network model corresponding to a first cooking step and at least one cooking step to be performed after the first cooking step. The at least one processor may input the sensing data to the at least one first neural network model to obtain contextual information indicating a cooking condition included in the sensing data, and control an operation of the electronic device based on the recipe information and the contextual information. The at least one processor may load the at least one second neural network based on the neural network model change event.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a control method of the electronic device, and more particularly, to an electronic device that operates based on sensor information and a control method of the electronic device. Background Art

[0002] Recently, the use of electric cookers or electric stoves has increased, leading to accelerated progress in technologies for automatically controlling devices such as cookers. These advances include the latest developments in artificial intelligence (AI)-related technologies and Internet of Things (IoT)-related technologies.

[0003] In the related art, there is a technology for automatically controlling a cooker by using an image of the cooker. However, the related art cannot effectively perform control required for each step of a recipe by comprehensively considering various information about the current cooking situation.

[0004] For example, the related art cannot automatically control the cooker in consideration of a case where a recipe of a cooking menu is changed based on a cooker type selected by a user or a cooking history of the user.

[0005] Furthermore, the related art fails to consider a method of selectively loading or selectively using a specific neural network model among a plurality of neural network models for each cooking step required to perform proper cooking based on a current cooking situation. Summary of the invention

[0006] Technical Solutions

[0007] According to one aspect of the present disclosure, an electronic device includes: at least one sensor; a memory storing a plurality of neural network models; and at least one processor configured to: based on receiving a user input for selecting a cooking menu, obtain recipe information corresponding to a cooking menu, identify a first cooking step and at least one cooking step to be performed after the first cooking step from a plurality of cooking steps included in the recipe information, load at least one first neural network model among the plurality of neural network models based on the recipe information and resources of the electronic device, the at least one first neural network model corresponding to the first cooking step and at least one cooking step to be performed after the first cooking step, obtain sensing data from at least one sensor, the sensing data corresponding at least to the first cooking step, obtain contextual information about a cooking situation indicated in the sensing data by inputting the sensing data into the at least one first neural network model, control the operation of the electronic device based on the recipe information and the contextual information, detect a neural network model change event based on the contextual information, and based on detecting the neural network model change event, load at least one second neural network model among the plurality of neural network models based on the contextual information and resource information of the electronic device.

[0008] The neural network model change event corresponds to an event of detecting the execution of a second cooking step different from the first cooking step, and the at least one processor may also be configured to: based on contextual information, identify a second cooking step among multiple cooking steps included in the recipe information and at least one cooking step to be performed after the second cooking step, and based on the recipe information and resources of the electronic device, load at least one second neural network model among multiple neural network models, the at least one second neural network model corresponding to the second cooking step and at least one cooking step to be performed after the second cooking step.

[0009] At least one processor may also be configured to: determine whether to change the recipe information based on contextual information and the recipe information, obtain the changed recipe information based on determining that the recipe information is changed, identify a third cooking step among multiple cooking steps included in the changed recipe information and at least one cooking step to be performed after the third cooking step based on the contextual information, and load at least one third neural network model among multiple neural network models based on the changed recipe information and resources of the electronic device, the at least one third neural network model corresponding to the third cooking step and at least one cooking step to be performed after the third cooking step.

[0010] At least one processor may also be configured to: determine whether there is an intermediate cooking step between the first cooking step and the second cooking step, determine that the recipe information has not been changed based on the determination that there is no intermediate cooking step between the first cooking step and the second cooking step, determine whether the intermediate cooking step between the first cooking step and the second cooking step is a cooking step related to the next cooking step after the second cooking step based on the determination that there is an intermediate cooking step between the first cooking step and the second cooking step, and determine that the recipe information is changed based on the determination that the cooking step between the first cooking step and the second cooking step is a cooking step related to the next cooking step after the second cooking step.

[0011] The neural network model change event may correspond to detection of new cookware not included in the recipe information, and the at least one processor may be further configured to: based on detection of the new cookware not included in the recipe information, change the recipe information based on the new cookware, and based on the changed recipe information and resources of the electronic device, load at least one fourth neural network model among the plurality of neural network models, the at least one fourth neural network model corresponding to a fourth cooking step and at least one cooking step to be performed after the fourth cooking step.

[0012] At least one processor may also be configured to: identify a first cooking step included in the recipe information, obtain information about the probability of performing one or more cooking steps after the first cooking step based on the recipe information, and identify at least one cooking step among the next cooking steps after the first cooking step based on the information about the probability of performing one or more cooking steps after the first cooking step.

[0013] The recipe information may include a knowledge graph showing a recipe corresponding to a selected cooking menu, the knowledge graph may include multiple nodes representing multiple cooking steps for completing the cooking menu based on the recipe and multiple edges representing a sequential relationship between the multiple cooking steps, and the at least one processor may also be configured to obtain information about the probability of performing one or more cooking steps after the first cooking step based on a distance between a node representing the first cooking step and a node representing a next cooking step after the first cooking step.

[0014] The at least one processor may also be configured to, based on determining that there are multiple first neural network models: identify resources required to execute each of the multiple first neural network models, and determine a number of neural network models to be loaded simultaneously in the multiple first neural network models based on resources of the electronic device and the resources required to execute each of the multiple first neural network models, each of the multiple first neural network models may correspond to a first cooking step and at least one cooking step to be performed after the first cooking step, and the multiple first neural network models are included in the multiple neural network models.

[0015] The at least one processor may also be configured to: determine a weight value for each of a plurality of first neural network models based on information about a probability of performing at least one cooking step after a first cooking step; sequentially load each of the plurality of first neural network models based on the determined weight value; and obtain contextual information about a cooking situation indicated in the sensed data by inputting the sensed data into the plurality of sequentially loaded first neural network models.

[0016] The at least one sensor may include an illuminance sensor for sensing an illuminance value around the electronic device, and the at least one processor is configured to load at least one first neural network model among a plurality of neural network models based on a type of the at least one sensor and at least one of the illuminance values ​​obtained from the illuminance sensor, the at least one first neural network model corresponding to a first cooking step and at least one cooking step to be performed after the first cooking step.

[0017] According to one aspect of the present disclosure, a method for controlling an electronic device includes: obtaining recipe information corresponding to a cooking menu based on receiving a user input selecting a cooking menu; identifying a first cooking step and at least one cooking step to be performed after the first cooking step from a plurality of cooking steps included in the recipe information; loading at least one first neural network model among a plurality of neural network models based on the recipe information and resources of the electronic device, the at least one first neural network model corresponding to the first cooking step and at least one cooking step to be performed after the first cooking step; obtaining sensing data from at least one sensor, the sensing data corresponding to at least the first cooking step; obtaining contextual information about a cooking situation indicated in the sensing data by inputting the sensing data into the at least one first neural network model; controlling the operation of the electronic device based on the recipe information and the contextual information; detecting a neural network model change event based on the contextual information; and loading at least one second neural network model among the plurality of neural network models based on the contextual information and resource information of the electronic device based on detecting the neural network model change event.

[0018] The neural network model change event may correspond to an event of detecting execution of a second cooking step different from the first cooking step, and loading at least one second neural network model may include: based on contextual information, identifying a second cooking step among multiple cooking steps included in the recipe information and at least one cooking step to be executed after the second cooking step; and based on the recipe information and resources of the electronic device, loading at least one second neural network model among the multiple neural network models, the at least one second neural network model corresponding to the second cooking step and the at least one cooking step to be executed after the second cooking step.

[0019] Loading at least one second neural network model may include: determining whether to change the recipe information based on context information and the recipe information; obtaining the changed recipe information based on determining that the recipe information is changed; identifying a third cooking step among multiple cooking steps included in the changed recipe information and at least one cooking step to be performed after the third cooking step based on the context information; and loading at least one third neural network model among the multiple neural network models based on the changed recipe information and resources of the electronic device, the at least one third neural network model corresponding to the third cooking step and at least one cooking step to be performed after the third cooking step.

[0020] Determining whether to change recipe information may include: determining whether there is an intermediate cooking step between a first cooking step and a second cooking step; determining that the recipe information has not been changed based on determining that there is no intermediate cooking step between the first cooking step and the second cooking step; determining whether the intermediate cooking step between the first cooking step and the second cooking step is a cooking step related to a next cooking step after the second cooking step based on determining that there is an intermediate cooking step between the first cooking step and the second cooking step; and determining that the recipe information is changed based on determining that the cooking step between the first cooking step and the second cooking step is a cooking step related to a next cooking step after the second cooking step.

[0021] The neural network model change event corresponds to detection of new cookware not included in the recipe information, and loading at least one second neural network model may include: based on the detection of the new cookware not included in the recipe information, changing the recipe information based on the new cookware; and based on the changed recipe information and resources of the electronic device, loading at least one fourth neural network model among the plurality of neural network models, the at least one fourth neural network model corresponding to a fourth cooking step and at least one cooking step to be performed after the fourth cooking step. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0023] Figure 1 is a view showing an appearance of an electronic device according to one or more embodiments of the present disclosure;

[0024] Figure 2a is a block diagram of a configuration including an electronic device according to one or more embodiments of the present disclosure;

[0025] Figure 2b is a view showing a heater according to one or more embodiments of the present disclosure;

[0026] Figure 2c is a view showing a display according to one or more embodiments of the present disclosure;

[0027] Figure 2d is a block diagram of a configuration including an electronic device according to one or more embodiments of the present disclosure;

[0028] Figure 3 is a flow chart illustrating a control method of an electronic device including loading a neural network model based on recipe information and context information according to one or more embodiments of the present disclosure;

[0029] Figure 4ais a view showing recipe information in the form of a knowledge graph according to one or more embodiments of the present disclosure;

[0030] Figure 4b is a view showing information about a neural network model included in recipe information according to one or more embodiments of the present disclosure;

[0031] Figure 5 is a flow chart illustrating a method for identifying cooking steps according to one or more embodiments of the present disclosure;

[0032] Figure 6 is a flowchart illustrating a method of loading a first neural network model according to one or more embodiments of the present disclosure;

[0033] Figure 7a is a diagram showing an example of simultaneously loading multiple neural network models according to one or more embodiments of the present disclosure;

[0034] Figure 7b is a diagram showing an example of sequentially loading multiple neural network models according to one or more embodiments of the present disclosure;

[0035] Figure 8 is a flowchart illustrating a method for loading a second neural network model based on a neural network model change event according to one or more embodiments of the present disclosure;

[0036] Fig. 9 is a flowchart illustrating an example of changing recipe information according to one or more embodiments of the present disclosure;

[0037] Fig.10 , Fig.11 , Fig.12 , Fig.13 and Fig.14 is a diagram showing various examples of dynamically loading a neural network model based on a neural network model change event according to one or more embodiments of the present disclosure;

[0038] Fig.15 is a flowchart illustrating an example of controlling the operation of an electronic device based on context information according to one or more embodiments of the present disclosure;

[0039] Fig.16 is a diagram showing a control system including an electronic device and a user terminal according to yet another one or more embodiments of the present disclosure;

[0040] Fig.17 and Fig.18 are views each showing a user interface (UI) screen of a user terminal for controlling an electronic device according to still another one or more embodiments of the present disclosure; and

[0041] Fig.19 is a timing diagram illustrating a control method of a system for performing a cooking operation based on context information according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0042] The present disclosure may be variously modified and have various embodiments, and the specific embodiments of the present disclosure are therefore shown in the drawings and described in detail in the specific embodiments. However, it should be understood that the scope of the present disclosure is not limited to the specific embodiments, and includes various modifications, equivalents or substitutions according to the embodiments of the present disclosure. Throughout the drawings, similar components are represented by similar reference numerals.

[0043] In describing the present disclosure, the description omits a detailed description of a case where a detailed description of a known function or configuration related to the present disclosure is determined to unnecessarily obscure the gist of the present disclosure.

[0044] In addition, the following embodiments may be modified in various forms, and the scope and spirit of the present disclosure are not limited to the following embodiments. Instead, these embodiments are provided to make the present disclosure thorough and complete, and to fully convey the spirit of the present disclosure to those skilled in the art.

[0045] The terms used in the present disclosure are only used to describe specific embodiments rather than to limit the scope of the present disclosure.Unless otherwise clearly indicated in the context, a singular term may include its plural number.

[0046] In the present disclosure, the expressions “having”, “may have”, “including”, “may include”, etc. indicate the existence of corresponding features (e.g., values, functions, operations, or components such as parts), and do not exclude the existence of additional features.

[0047] In the present disclosure, the expressions "A or B", "at least one of A and / or B", "one or more of A and / or B", etc. may include all possible combinations of items listed together. For example, "A or B", "at least one of A and B", or "at least one of A or B" may mean all of the following: 1) the case of only including A, 2) the case of only including B, or 3) the case of both A and B.

[0048] The expressions "first", "second", etc. used in the present disclosure may represent various components regardless of the order and / or importance of the components. These expressions are only used to distinguish one component from another component and do not limit the corresponding components.

[0049] In the case where any component (e.g., a first component) is mentioned as being "(operably or communicatively) coupled to" / with another component (e.g., a second component) or "connected to" another component (e.g., a second component), it should be understood that any component is coupled to the other component directly or through yet another component (e.g., a third component).

[0050] On the other hand, in the case where any component (e.g., a first component) is mentioned as being "directly coupled to" or "directly connected to" another component (e.g., a second component), it should be understood that there is no further component (e.g., a third component) between either component and the other component.

[0051] Depending on the situation, the expression "configured (or set) to" used in the present disclosure may be replaced by the expression "suitable for", "capable of", "designed to", "suitable for", "manufactured to", or "capable of". The expression "configured (or set) to" may not necessarily mean "specially designed to" in hardware.

[0052] On the contrary, in any case, the statement that a device is "configured to" may mean that the device can "perform ~" together with another device or component. For example, "a processor configured (or set) to perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing the corresponding operations or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor) that can perform the corresponding operations by executing one or more software programs stored in a memory device.

[0053] In an embodiment, a "module" or "device" may perform at least one function or operation and may be implemented in hardware or software, or in a combination of hardware and software. In addition, multiple "modules" or multiple "devices" may be integrated with each other in at least one module and implemented by at least one processor other than the "module" or "device" that needs to be implemented in specific hardware.

[0054] The accompanying drawings schematically illustrate various elements and regions. Therefore, the spirit of the present disclosure is not limited by the relative sizes or intervals shown in the accompanying drawings.

[0055] Embodiments of the present disclosure are described in detail with reference to the accompanying drawings so that those skilled in the art to which the present disclosure pertains can easily practice the present disclosure.

[0056] Figure 1 1 is a diagram showing the appearance of an electronic device 100 according to one or more embodiments of the present disclosure. Figure 1As shown, the electronic device 100 can be implemented as a kitchen appliance including a heater (such as a cooker, a gas stove, an electric stove or an oven). In addition, in addition to the heater, the electronic device 100 can also include a cover located above the heater to discharge smoke or odor to the outside.

[0057] For example, based on receiving a user input for selecting a cooking menu, the electronic device 100 can obtain recipe information for the selected cooking menu. In one or more examples, the "recipe information" can include information about each step of a recipe for completing the cooking menu, and can include information about multiple cooking steps.

[0058] In addition, the electronic device 100 may obtain context information by using at least one sensor. Figure 1 As shown, the electronic device 100 can obtain sensory data collected by a camera or microphone installed in the engine hood, wherein the sensory data is input into the neural network model to obtain context information. In one or more examples, the "context information" may include information about the current cooking situation indicated in the sensory data, and may include the type or state of food ingredients indicated in the sensory data (e.g., image data or sound data), the type of cooking container, the surrounding environment of the electronic device 100 (e.g., ambient brightness), information about user actions, information about the heater, etc.

[0059] The electronic device 100 can control the operation of the electronic device 100 to execute the selected cooking menu based on the obtained recipe information and context information. For example, the electronic device 100 can control the heating temperature, heating time, and operation (on / off) of the heater based on the recipe information or context information. The electronic device 100 can automatically perform these operations without input from the user.

[0060] In one or more examples, the electronic device 100 may load at least one neural network model to obtain context information. However, the electronic device 100 in an on-device environment may be limited in resources and may therefore be limited in simultaneously loading all of the neural network models in the plurality of neural network models required to execute the selected cooking menu. In addition, based on simultaneously loading all of the neural network models in the plurality of neural network models required to execute the selected cooking menu, the electronic device 100 may have a significantly reduced processing speed.

[0061] Therefore, according to one or more embodiments of the present disclosure, the electronic device 100 can selectively load only the neural network model required for each cooking step among multiple neural network models based on at least one of the recipe information, the resource information of the electronic device 100, and the contextual information about the cooking situation. Therefore, the electronic device 100 can significantly reduce the resources used to load and process the neural network model. That is, even when the electronic device 100 does not have sufficient memory capacity or processing capacity to load more than one neural network model at the same time, the electronic device 100 can still recognize the cooking situation in real time by using the neural network model in the local environment, and perform control corresponding to the recognized cooking situation.

[0062] However, as understood by those of ordinary skill in the art, implementing the electronic device 100 as a kitchen appliance including a heater is only an example, and the electronic device 100 may be implemented as a cover, a user terminal (e.g., a smartphone), or a server. For example, the electronic device 100 may be a device that can remotely control a heater of an external device connected to the electronic device 100. For example, the electronic device 100 may be implemented as a portable terminal such as a smartphone or a tablet personal computer (PC), a household appliance such as a television or a refrigerator, or a server. In one or more examples, the electronic device 100 may control the operation of the external device through a communication connection with the external device including a heater.

[0063] Hereinafter, the electronic device 100 that may operate based on recipe information and context information about a cooking situation according to one or more embodiments of the present disclosure is described in more detail with reference to the accompanying drawings.

[0064] Figure 2a is a block diagram of a configuration of an electronic device including one or more embodiments of the present disclosure. Figure 2a As shown, the electronic device 100 may include at least one sensor 110, a memory 120, a heater 130, a communication interface 140, an input interface 150, an output device 160, and at least one processor 170. Figure 2a It is shown that the electronic device 100 is implemented as a kitchen appliance including a cooker and a hood, but this configuration is only an example, and in the implementation example, some components of the electronic device 100 may be deleted or added.

[0065] At least one sensor 110 may detect various information inside and outside the electronic device 100. In one or more examples, the at least one sensor 110 according to the present disclosure may include a camera 111, a microphone 113, and an illumination sensor 115, such as Figure 2a shown.

[0066] The camera 111 may obtain an image (or image data) of the interior and / or exterior of the electronic device 100 through an image sensor. For example, the camera 111 may be positioned in a hood located above the heater 130 to capture the heater 130 and a kitchen counter of the electronic device 100 or a user located around the electronic device 100, and a plurality of cameras 111 may be positioned to capture corresponding areas in a plurality of areas. The image obtained by the camera 111 may be a top view image or an image captured at an angle close to a top view image. In addition, at least one of the plurality of neural network models may perform learning based on a top view image or an image captured at an angle close to a top view image.

[0067] The microphone 113 can obtain sound (or sound data). In one or more examples, the microphone 113 can be located in the heater 130 or the cover. However, this configuration is only an example, and the microphone 113 can be located in one area in the kitchen when electrically connected to the electronic device 100. In one or more examples, multiple microphones 113 can be positioned to collect audio generated from multiple areas.

[0068] The illuminance sensor 115 may obtain illuminance data regarding illuminance of a surrounding environment of the electronic device 100 .

[0069] Although Figure 2a It is shown that the at least one sensor 110 includes a plurality of sensors, but this configuration is only an example, and the at least one sensor 110 may include only the camera 111, or only one microphone 113, or only one illumination sensor 115. In addition, the at least one sensor 110 may also include various types of sensors, such as a temperature sensor, a humidity sensor, an infrared sensor, and a biosensor.

[0070] According to one or more embodiments of the present disclosure, at least one processor 170 may obtain sensing data from at least one sensor 110, and obtain context information according to the present disclosure by inputting the sensing data into at least one neural network model of a plurality of neural network models. For example, when the sensing data is image data obtained by a camera 111, at least one processor 170 may input the image data into an object recognition model, and obtain information about food ingredients included in the image, and may also obtain information about the cookware used when preparing a cooking menu. In addition, at least one processor 170 may input the image data into an action recognition model to obtain information about user actions indicated in the image. In addition, when the sensing data is sound data obtained by a microphone 113, at least one processor 170 may input the sound data into a sound recognition model to obtain information about the type of sound included in the sound data. For example, the sound data may be related to a cooking situation such as frying an item or cutting vegetables.

[0071] The memory 120 may store at least one instruction for controlling the electronic device 100. In addition, the memory 120 may store an operating system (O / S) for driving the electronic device 100. In addition, the memory 120 may store various software programs or applications for operating the electronic device 100 according to various embodiments of the present disclosure. In addition, the memory 120 may include a semiconductor memory 120 such as a flash memory 120 or a magnetic storage medium such as a hard disk.

[0072] According to one or more embodiments, the memory 120 may store various software modules for operating the electronic device 100 according to various embodiments of the present disclosure, and at least one processor 170 may execute the various software modules stored in the memory 120 to control the operation of the electronic device 100. For example, the memory 120 may be accessed by the processor, and data may be read / written / modified / deleted / updated by the processor.

[0073] The memory 120 according to the present disclosure may include a non-volatile memory and a volatile memory. In one or more examples, the non-volatile memory refers to a memory that can maintain the stored information even when the power supply is interrupted, and the volatile memory refers to a memory that requires a continuous power supply to maintain the stored information. According to the present disclosure, the expression "load" may refer to an operation of loading and storing data stored in a non-volatile memory in a volatile memory. The volatile memory according to the present disclosure may be implemented not only as a separate component separate from at least one processor 170, but also as a component of at least one processor 170.

[0074] In addition, memory 120 may store data about multiple neural network models according to the present disclosure. Specifically, the non-volatile memory of memory 120 may store data about multiple neural network models according to the present disclosure. In addition, memory 120 may store data about multiple neural network models received from an external server. Although one neural network model may correspond to one cooking step, this configuration is only an example, and as understood by a person of ordinary skill in the art, more than one neural network model may correspond to the cooking step. In one or more examples, when more than one neural network model corresponds to one cooking step, these neural network models may be referred to as a neural network model set or a neural network model group.

[0075] In one or more examples, the memory 120 may not store a plurality of neural network models, and may request at least one neural network model corresponding to the cooking steps recognized by the electronic device 100 from the external server. The memory 120 (eg, a volatile memory) may store the requested neural network model.

[0076] In addition, the memory 120 may store recipe information, sensory data, and context information. In addition, the memory 120 may store user data according to the present disclosure, information about weight values ​​assigned to a neural network model, and information about the probability of performing a cooking step after a specific cooking step.

[0077] Furthermore, various information required to achieve the purpose of the present disclosure may be stored in the memory 120 , and the information stored in the memory 120 may be updated when received from a server or an external device or input by a user.

[0078] The heater 130 may refer to a component configured to heat food or one or more ingredients (e.g., raw materials) used in a cooking menu. For example, when the electronic device 100 is implemented as an electric stove, the heater 130 may generate heat by supplying electricity, and the heat source here may be a radiation heater or an induction heater. When the electronic device 100 is implemented as a gas stove, the heater 130 may generate heat by supplying gas. The heater 130 may be operated based on user input or the heating temperature and heating time of the heater 130 may be executed, which may be performed by at least one processor 170, particularly in various embodiments of the present disclosure. In one or more examples, the heater 130 may be operated automatically without user input.

[0079] In one or more examples, the heater 130 may include a plurality of burners 131, 132, and 133, such as Figure 2b As shown. In one or more examples, the burner can be implemented as a zone for providing heat to the cooker, and the plurality of burners can be operated in different ways. For example, the first burner 131 and the second burner 132 can be operated by a radiation heater method, while the third burner 133 can be operated by an induction heater method. In addition, the plurality of burners can also have different sizes. In one or more embodiments, at least one processor 170 can individually control the temperature, operating time, and operating mode of each of the plurality of burners 131, 132, and 133. For example, each of the plurality of burners 131, 132, and 133 can control the corresponding operating temperature in a plurality of stages (e.g., the first stage to the ninth stage).

[0080] In addition, the heater 130 may include an oven or a microwave. In one or more examples, the at least one sensor 110 may be located in the oven or the microwave to obtain sensing data about the state of food.

[0081] The communication interface 140 may include a circuit and perform communication with an external device (e.g., a server or a user terminal). In one or more examples, at least one processor 170 may receive various data or information from an external device connected thereto through the communication interface 140, and send various data or information to the external device.

[0082] The communication interface 140 may include at least one of a wireless fidelity (WiFi) module, a Bluetooth module, a wireless communication module, a near field communication (NFC) module, or an ultra-wideband (UWB) module. In addition, the wireless communication module may perform communication based on various communication protocols such as the Institute of Electrical and Electronics Engineers (IEEE), Zigbee, third generation (3G), third generation partnership project (3GPP), long term evolution (LTE), fifth generation (5G), etc.

[0083] According to one or more embodiments of the present disclosure, at least one processor 170 may receive data about multiple neural network models, recipe information, sensory data, context information, or any other suitable information known to a person of ordinary skill in the art from an external device through the communication interface 140. In addition, at least one processor 170 may receive user data, information about weight values ​​assigned to the neural network model, information about the probability of performing a cooking step after a specific cooking step, and the like through the communication interface 140. In addition, based on obtaining information about the user's past cooking actions or cooking history, at least one processor 170 may control the communication interface 140 to generate user data based on the information about the user's past cooking actions or cooking history, and transmit the generated user data to an external device. Past cooking actions or cooking history may refer to actions performed by a user within a predetermined time interval, or may refer to actions performed by a user when the user previously cooked the same menu ingredients that are currently being cooked.

[0084] The input interface 150 may include a circuit, and at least one processor 170 may receive a user input for controlling the operation of the electronic device 100 through the input interface 150. For example, the input interface 150 may be implemented as a touch screen included in a display. In addition, the input interface 150 may be a component such as a signal receiver that receives a user input from an external device. In one or more examples, the input interface 150 may include a camera or a microphone.

[0085] According to one or more embodiments of the present disclosure, at least one processor 170 may receive a user input for selecting a cooking menu through the input interface 150. In addition, at least one processor 170 may receive a user input for directly generating or changing recipe information from a user through the input interface 150, and generate or change the recipe information based on the received user input. In addition, the electronic device 100 according to the present disclosure may be implemented as a kitchen appliance including a heater 130 (such as a stove, a gas stove, or an electric stove). In one or more examples, according to the present disclosure, at least one processor 170 may receive a user input for controlling the heater 130. For example, at least one processor 170 may control the heater 130 based on context information, or may also control the heater 130 based on user input. The user input for controlling the heater 130 may be implemented in the form of a physical button, a touch button, or a control valve.

[0086] In one or more examples, in addition to the above-mentioned input interface 150, the electronic device 100 can receive user input through the camera 111 or the microphone 113. For example, based on receiving user input through motion recognition or voice recognition, the electronic device 100 can receive user motion captured by the camera 111 as user input or user voice obtained by the microphone 113 as user input.

[0087] The output device 160 may include a circuit, and the at least one processor 170 may output various functions that the electronic device 100 may perform through the output device 160. In addition, the output device 160 may include at least one of a display 161, a speaker 162, or an indicator 163.

[0088] In one or more examples, the display 161 may output image data under the control of at least one processor 170. For example, Figure 2c As shown, the display 161 may be located at the bottom of the heater 130 including the plurality of burners 131 , 132 , and 133 , and display information on an operating state of the electronic device 100 .

[0089] According to one or more embodiments of the present disclosure, the display 161 can display a user interface stored in the memory 120. The display 161 can be implemented as a liquid crystal display panel (LCD), an organic light emitting diode (OLED), or any other display type known to those of ordinary skill in the art. In addition, the display 161 can be implemented as a flexible display, a transparent display, or any other known display structure known to those of ordinary skill in the art. However, the display according to the present disclosure is not limited to any particular type of display. The speaker 162 can output sound data under the control of the processor 170, and the indicator 163 can be lit under the control of at least one processor 170.

[0090] According to one or more embodiments of the present disclosure, at least one processor 170 may output a guidance message indicating a control situation of the electronic device 100 or a guidance message for prompting a user action through the output device 160. For example, the cooking situation included in the contextual information may be a situation of "putting garlic, onions, mushrooms, bacon, tomatoes, and sauce into a pan." In this cooking situation, at least one processor 170 may control the heater 130 to operate at medium heat for 4 minutes, or also provide a guidance message guiding the user to put noodles into the pan after 4 minutes. In another example, the cooking situation included in the contextual information may be a situation of "putting steak into a pan." In this cooking situation, at least one processor 170 may output a guidance message for prompting the user to flip the steak every 30 seconds.

[0091] In one or more examples, the guidance message can be output in an auditory form through the speaker 162 (this is only an example), and can be sent in a visual form through the display 161 or the indicator 163. In one or more examples, based on the connection to the user terminal located outside, the electronic device 100 can send a guidance message to the user terminal. The user terminal can provide the received guidance message. In one or more examples, the electronic device 100 can provide a guidance message to at least one of the display 161, the speaker 162, or the indicator 163 based on the user setting.

[0092] At least one processor 170 may control one or more operations of the electronic device 100 based on at least one instruction stored in the memory. For example, the one or more processors may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), or a neural processing unit (NPU). According to one or more embodiments of the present disclosure, a general processor (e.g., a CPU) of at least one processor 170 may perform general operations, and an artificial intelligence processor (e.g., a GPU or an NPU) of at least one processor 170 may perform artificial intelligence operations. In one or more embodiments of the present disclosure, the artificial intelligence processor of at least one processor 170 may be used for an operation of obtaining contextual information about a cooking situation included in the sensed data by inputting sensed data obtained by at least one sensor 110 into at least one first neural network model. In addition, the general processor of at least one processor 170 may be used for other operations in addition to the operation of obtaining contextual information. However, the above case is only an example, and one processor may control the operation of the electronic device 100.

[0093] In one or more examples, at least one processor 170 may obtain recipe information corresponding to the selected cooking menu based on receiving a user input for selecting a cooking menu. In addition, at least one processor 170 may identify a first cooking step and at least one cooking step to be performed after the first cooking step among a plurality of cooking steps included in the recipe information. In addition, at least one processor 170 may load at least one first neural network model among a plurality of neural network models based on the recipe information and the resources of the electronic device 100, wherein the first neural network model corresponds to the first cooking step and at least one cooking step to be performed after the first cooking step. In addition, at least one processor 170 may obtain sensing data from at least one sensor and input the sensing data to the first neural network model to obtain contextual information about the cooking situation. In addition, at least one processor 170 may control the operation of the electronic device based on the recipe information and the contextual information. In addition, based on detecting a neural network model change event according to the contextual information, at least one processor 170 may load at least one second neural network model among a plurality of neural network models, wherein the at least one second neural network model is determined based on the contextual information and the resource information of the electronic device.

[0094] In one or more examples, at least one processor 170 may identify a first cooking step included in the recipe information. In addition, at least one processor 170 may obtain information about a probability of performing a cooking step after the first cooking step based on the recipe information. In addition, at least one processor 170 may identify at least one cooking step among the next cooking steps after the first cooking step based on the information about the probability of performing the cooking step after the first cooking step.

[0095] In one or more examples, the recipe information may include a knowledge graph showing the recipe. In one or more examples, the knowledge graph may include a plurality of nodes representing a plurality of cooking steps for completing a cooking menu based on the recipe and a plurality of edges representing a sequential relationship between the plurality of cooking steps. In addition, the at least one processor 170 may obtain information about the probability of performing a cooking step after the first cooking step based on a distance between a node representing the first cooking step and a node representing a next cooking step after the first cooking step.

[0096] In the case where there are multiple first neural network models (e.g., the multiple neural network models include multiple first neural network models), at least one processor 170 may identify resources required to execute each of the multiple first neural network models. At least one processor 170 may determine the number of neural network models to be loaded simultaneously among the multiple first neural network models based on resources of the electronic device and resources required to execute each of the multiple first neural network models.

[0097] For example, at least one processor 170 may determine a weight value of each of the plurality of first neural network models based on the probability of performing a cooking step after the first cooking step. In addition, at least one processor 170 may sequentially load the plurality of first neural network models based on the determined weight values. In addition, at least one processor 170 may obtain contextual information about a cooking situation by inputting sensing data to the plurality of sequentially loaded first neural network models.

[0098] In addition, at least one processor 170 can load at least one first neural network model corresponding to the first cooking step and at least one cooking step to be performed after the first cooking step from among the multiple neural network models based on the type of at least one sensor 110 and at least one of the illuminance values ​​obtained from the illuminance sensor 115.

[0099] In one or more examples, the neural network model change event may be an event of detecting a cooking situation in which a second cooking step different from the first cooking step is performed. Based on detecting the event, at least one processor 170 may identify a second cooking step among a plurality of cooking steps included in the recipe information and at least one cooking step to be performed after the second cooking step based on the context information. In addition, at least one processor 170 may load at least one second neural network model corresponding to the second cooking step and at least one cooking step to be performed after the second cooking step among a plurality of neural network models based on the recipe information and resources of the electronic device 100.

[0100] In one or more examples, at least one processor 170 may determine whether to change the recipe information based on the context information and the recipe information. The at least one processor 170 may obtain the changed recipe information based on determining that the recipe information has changed. The at least one processor 170 may identify the second cooking step and at least one cooking step to be performed after the second cooking step among the multiple cooking steps included in the changed recipe information based on the context information. In addition, the at least one processor 170 may load at least one second neural network model corresponding to the second cooking step and at least one cooking step to be performed after the second cooking step among the multiple neural network models based on the changed recipe information and the resources of the electronic device 100.

[0101] In addition, the at least one processor 170 may determine whether there is a cooking step between the first cooking step and the second cooking step. In the case where there is no cooking step between the first cooking step and the second cooking step, the at least one processor 170 may determine that the recipe information is not changed. The at least one processor 170 may determine whether the cooking step between the first cooking step and the second cooking step is a cooking step related to the next cooking step after the second cooking step based on determining that there is a cooking step between the first cooking step and the second cooking step. The at least one processor 170 may determine that the recipe information is changed based on determining that the cooking step between the first cooking step and the second cooking step is a cooking step related to the next cooking step after the second cooking step.

[0102] In one or more examples, the neural network model change event may be an event of detecting a new cookware not included in the recipe information. Based on detecting the new cookware not included in the recipe information according to the context information, the at least one processor 170 may change the recipe information based on the new cookware. In addition, based on the changed recipe information and the resources of the electronic device 100, the at least one processor 170 may load at least one second neural network model corresponding to the second cooking step and at least one cooking step to be performed after the second cooking step from among the plurality of neural network models.

[0103] Although Figure 2a The heater 130 is shown to be located in the electronic device 100, but this configuration is only an example, and the electronic device 100 may be implemented as a cover or a user terminal, wherein the electronic device 100 may not include the heater 130, such as Figure 2d In one or more examples, the electronic device 100 can communicate with and control another electronic device including the heater 130 .

[0104] Figure 3 is a flowchart illustrating a control method of an electronic device including loading a neural network model based on recipe information and context information according to one or more embodiments of the present disclosure.

[0105] The electronic device 100 may receive a user input for selecting a cooking menu (S310). For example, the electronic device 100 may receive a user input for selecting a cooking menu through the input interface 150 (eg, a touch screen). The cooking menu may specify cooking ingredients to be cooked.

[0106] In one or more examples, the electronic device 100 may receive a user input for selecting a cooking menu through an external device (e.g., a user terminal) connected to the electronic device 100. For example, when an application for controlling the electronic device 100 is executed in the external device, the external device may receive the user input through a screen on which the application is executed. The external device may transmit the user input for selecting a cooking menu to the electronic device 100.

[0107] In one or more embodiments, the electronic device 100 may receive a user input for selecting a cooking menu by photographing an identification code (e.g., a barcode or a QR code) included in the food raw material through the camera 113. For example, the electronic device 100 may obtain information about a cooking menu selected by a user by photographing a user input of a QR code located on the packaging of the food raw material. In one or more examples, the electronic device 100 may obtain information about a cooking menu corresponding to the identification code photographed by the user terminal. Specifically, based on photographing the identification code, the user terminal may obtain information about a cooking menu corresponding to the identification code. The user terminal may send the obtained information about the cooking menu to the electronic device 100, and the electronic device 100 may obtain information about the cooking menu from the user terminal.

[0108] The electronic device 100 may obtain recipe information corresponding to the selected cooking menu (S320). In one or more examples, the electronic device 100 may obtain recipe information corresponding to the selected cooking menu by searching for a recipe corresponding to the selected cooking menu in a pre-built recipe database. The recipe database may be stored in the electronic device 100 and an external device.

[0109] The "recipe information" may include information about each step of a recipe for completing a cooking menu. For example, the recipe information may include information about a plurality of cooking steps.

[0110] The recipe information may be implemented in the form of a knowledge graph showing a recipe. In one or more examples, a "knowledge graph" refers to a data structure including a plurality of nodes and a plurality of edges connecting the plurality of nodes to each other. A plurality of "nodes" may represent one of a plurality of cooking steps for completing a cooking menu based on a recipe, and a plurality of "edges" may represent a sequential relationship between the cooking steps corresponding to the plurality of nodes.

[0111] In addition, the “distance between nodes” may be determined based on the number of other nodes included in the minimum path connecting the two nodes to each other. For example, in the case where the first node and the second node are directly connected to each other, the distance between the first node and the second node may be zero, and in the case where the first node is connected to the second node through a third node, the distance between the first node and the second node may be 1.

[0112] In addition, the plurality of nodes may each include information about an object associated with each cooking step and information about an action on the object. In one or more examples, the information about the object may include information about a stove, a cooker, food ingredients, or any other suitable cooking information known to a person of ordinary skill in the art, and the information about the action on the object may include information about the user's cooking action, information about the time to perform the cooking action, or any other suitable cooking action information known to a person of ordinary skill in the art. This description is provided below with reference to Figure 4a and Figure 4b Describes the recipe information in detail.

[0113] The electronic device 100 may identify a first cooking step and at least one cooking step to be performed after the first cooking step (S330). In one or more examples, the "first cooking step" may represent the first cooking step of a plurality of cooking steps. However, the "first cooking step" may represent any one of a plurality of cooking steps. The at least one cooking step to be performed after the first cooking step may not only represent a cooking step that may be performed immediately after the first cooking step, but may also be identified from a plurality of cooking steps that may be performed after the first cooking step.

[0114] For example, the electronic device 100 may identify at least one cooking step among the next cooking steps after the first cooking step based on information about the probability of performing the cooking step after the first cooking step. In one or more examples, the electronic device 100 may identify a cooking step whose probability of performing the cooking step after the first cooking step is greater than or equal to a threshold as at least one cooking step. This description is provided below with reference to Figure 5 The configuration is described in detail.

[0115] The electronic device 100 may load at least one first neural network model (S340). In one or more examples, based on the recipe information and the resources of the electronic device 100, the electronic device 100 may load at least one first neural network model corresponding to the first cooking step and at least one cooking step to be performed after the first cooking step from among the plurality of neural network models.

[0116] In one or more examples, the electronic device 100 may identify at least one first neural network model corresponding to the identified first cooking step and at least one cooking step to be performed after the first cooking step. The electronic device 100 may determine the number of neural network models to be loaded simultaneously based on the resources of the electronic device 100 and the resources of the multiple first neural network models based on the determination that there are multiple first neural network models. In addition, the electronic device 100 may determine whether the sum of the resources of the multiple first neural network models is less than the resources of the electronic device. When the sum of the resources of the multiple first neural network models is less than the resources of the electronic device, the electronic device 100 may load the multiple first neural network models simultaneously. When the sum of the resources of the multiple first neural network models is greater than or equal to the resources of the electronic device, the electronic device 100 may load the multiple first neural network models in sequence based on the corresponding weight values ​​of each of the multiple first neural network models. In one or more examples, the electronic device 100 may determine the weight values ​​of the multiple first neural network models based on information about the probability of performing at least one cooking step after the first cooking step.

[0117] In addition, the electronic device 100 may load at least one first neural network model corresponding to the first cooking step and at least one cooking step to be performed after the first cooking step among the plurality of neural network models based on at least one of the type of the at least one sensor 110 and the illuminance value obtained from the illuminance sensor 115. Figure 6 to Figure 7b The method of loading at least one first neural network model by the electronic device 100 is described in more detail.

[0118] The electronic device 100 may input the sensing data into at least one neural network model to obtain context information about the cooking situation indicated in the sensing data (S350). The "context information" may include information about the cooking situation included in the sensing data. In one or more examples, when the sensing data includes an image, the context information may include information about the user action included in the image, information about the object included in the image, and the like. For example, the information about the user action included in the image may represent a user action, such as whether the user slices the food raw material, whether the user puts the food raw material into the cooker, or whether the user stir-fryes the food raw material. In one or more examples, the user action included in the image may include a user action (in the form of a verb) and an object (in the form of a noun) as a target of the user action, and the action recognition model may therefore output the user action included in the image (in the form of a verb-noun pair). For example, the action recognition model may output the user action included in the image, such as "the action of {putting} {pasta} in {pot}" or "the action of {putting} {pasta} in {pan}", followed by "the action of {stir-frying}". In one or more examples, the action recognition model can perform learning to classify user actions included in the image based on the labeled image in the form of verb-noun pairs. For example, the action recognition model can perform learning to classify user actions included in the image based on learning data, where the input data is image data and the output data has information about the action in the form of verb-noun pairs.

[0119] In addition, the information about the objects included in the image may include information about food ingredients (e.g., garlic, onions, mushrooms, or noodles) included in the image, information about changes in the state of food ingredients included in the image, or information about cookware (e.g., pots, coated pans, stainless steel pans, cast iron pans, or soup spoons) included in the image, etc. For example, the object recognition model may output information about the recognized object type in the form of a noun. In one or more examples, the object recognition model may output information about the object, including information about the recognized object type in the form of a noun and changes in the state of the object in the form of a verb. For example, the object recognition model may output information about the object, such as "{onion} becomes {transparent}" or "{water} {boils}".

[0120] In addition, when the sensing data includes a sound signal, the context information may include information about the type of sound included in the sound signal. For example, the information about the type of sound may be information about a sound indicating a cooking progress state (e.g., the sound of boiling water or the sound of a pressure cooker) or a sound indicating a cooking action of the user (e.g., the sound of flipping a steak in a pan or cutting vegetables).

[0121] In addition, the context information can be obtained as the final context information by combining the context information output by inputting multiple sensing data into each neural network model. For example, the electronic device 100 can obtain "water" as information about the object through the object recognition model, and obtain "boiling sound" as information about the cooking progress state through the sound recognition model. In one or more examples, the electronic device 100 can obtain "the state of boiling water" as the final context information based on the combination of the first context information output by the object recognition model and the second context information output by the sound recognition model.

[0122] In addition, the context information may include information about the surrounding environment (eg, ambient brightness) of the electronic device 100. In one or more examples, the electronic device 100 may obtain information about the surrounding environment of the electronic device 100 by inputting the obtained image into a neural network model.

[0123] In addition, the context information may include information about the heater. In one or more examples, the electronic device 100 may obtain information about the heater on which the cooker is located by inputting an image of the heater 130 captured into the neural network model. For example, when there is a heater 130 including three burners, such as Figure 2b As shown, the electronic device 100 may input the image captured of the heater 130 into the neural network model to identify that the burner on which the cookware is located is the second burner 132. In one or more examples, the electronic device 100 may analyze the pixels of the image captured of the heater 130 to identify that the second burner 132 is the burner on which the cookware is located. In addition, the electronic device 100 may control the operating temperature and operating time of the second burner 132 on which the cookware is located in the heater 130.

[0124] When the electronic device 100 monitors context information through at least one loaded neural network model, it may not be possible to detect context information through at least one neural network model loaded within a threshold time. For example, the electronic device 100 may identify the next cooking step after the cooking step corresponding to at least one currently loaded neural network model, and load the neural network model corresponding to the next cooking step. In one or more examples, the cooking steps corresponding to the at least one currently loaded neural network model may be the first cooking step and the second cooking step. In one or more examples, based on the fact that context information is not detected during the threshold time, the electronic device 100 may load the neural network models corresponding to the third cooking step and the fourth cooking step. In one or more examples, the threshold time may be predetermined, which is only an example, and may be set based on the cooking time of the corresponding cooking step.

[0125] The electronic device 100 may control the operation of the electronic device 100 based on the recipe information and the context information (S360). In one or more examples, the electronic device 100 may control the heater 130 by identifying which cooking step of the recipe the current cooking situation belongs to or identifying information about the state of the current food material based on the recipe information and the context information. Fig.15 This configuration is described in detail.

[0126] Then, the electronic device 100 may determine whether a neural network model change event is detected (S370). In one or more examples, when obtaining context information, the electronic device 100 may determine whether a neural network model change event is detected based on the obtained context information. For example, the neural network model change event may be at least one of an event in which a cooking situation in which a second cooking step different from the first cooking step is performed is detected or an event in which a new cooker not included in the recipe information is detected. Reference Figures 8 to 14 Describes neural network model change events in more detail.

[0127] In addition, based on detecting the neural network model change event (S370-Y), the electronic device 100 can load at least one second neural network model (S380), which will be described in more detail later with reference to the accompanying drawings.

[0128] Figure 4a is a view showing recipe information in the form of a knowledge graph according to one or more embodiments of the present disclosure. For example, Figure 4a An example of recipe information in which "tomato pasta" is a cooking menu selected by user input is shown. The recipe information may include objects related to each cooking step and information about actions on the objects, such as a node representing the step "boil water" or a node representing the step "cook noodles for 7 minutes" connected thereto. Figure 4a In addition to the nodes and edges shown in , the recipe information may include a greater number of nodes and edges.

[0129] Figure 4a It is shown that each node includes information in the form of natural language, such as "boil water", "cook noodles for 7 minutes", or "put garlic, onions, and mushrooms in a pan". These nodes are only examples, where the nodes can store information about {objects} (for example, store information about cooking utensils and food ingredients respectively) and information about {actions}. For example, the node corresponding to "boil water" can store the step in the form of {water} and {boil}, the node corresponding to "cook noodles for 7 minutes" can store the step in the form of {noodles} and {cook for 7 minutes}, and the node corresponding to "put garlic, onions, and mushrooms in a pan" can store the step in the form of {pan}, {garlic, onion, mushrooms}, and {put}.

[0130] According to one or more embodiments of the present disclosure, recipe information may include information about a neural network model required for each of a plurality of cooking steps. In one or more examples, the information about the neural network model may include various information such as the type of the neural network model, input / output data of the neural network model, or resources required by the neural network model. Figure 4b is a view showing information about a neural network model included in recipe information according to one or more embodiments of the present disclosure.

[0131] like Figure 4b As shown, based on the first cooking step 410 (e.g., the step of putting garlic, onions, and mushrooms into a pan), the neural network model required to identify the execution of the first cooking step may include a first object recognition model, a second object recognition model, and a first action recognition model. Based on the second cooking step 420 (e.g., the step of stir-frying onions until transparent), the neural network model required to identify the execution of the second cooking step may include a third object recognition model, a second action recognition model, and a first sound recognition model.

[0132] In one or more examples, the metadata of each neural network model may include information about the input / output data of the neural network model. For example, the first object recognition model may include {image} as input data and {whether the object is a pan} as output data.

[0133] In addition, Figure 4a In the example, the metadata of each neural network model may also include information about the resources required to load the neural network model, the version of the neural network model, etc.

[0134] In addition, the recipe information can be updated based on user data about the user's past cooking actions. In one or more examples, the electronic device 100 can obtain user data about the user's past cooking actions and store it in a memory of the electronic device 100 or a memory of a user terminal in which an application for controlling the electronic device 100 is installed.

[0135] In one or more examples, the electronic device 100 may determine whether a cooking action is performed according to a plurality of cooking steps included in the selected cooking menu when executing the selected cooking menu. For example, the electronic device 100 may determine whether some of the plurality of cooking steps are omitted, or whether food ingredients, cookware, cooking methods, or any other cooking processes in the plurality of cooking steps are changed. The electronic device 100 may determine not to perform cooking according to the plurality of cooking steps, for example, some of the plurality of cooking steps are omitted, and food ingredients, cookware, cooking methods, etc. in some of the plurality of cooking steps are changed. In one or more examples, the electronic device 100 may obtain information about the changed cooking action as user data.

[0136] In addition, although food ingredients 1, 2, and 3 are required based on the information about the food ingredients of the selected cooking menu A, food ingredients 1, 2, and 4 may be food ingredients currently detected by the camera 111, or food ingredients stored in the refrigerator. In one or more examples, the electronic device 100 may determine whether food ingredient 4 can replace food ingredient 3. For example, the electronic device 100 may search for recipe information of the cooking menu A including food ingredient 4 through a search server, and provide the retrieved recipe information to the electronic device 100 or a user terminal controlling the electronic device 100. However, based on determining that food ingredient 3 cannot be replaced with food ingredient 4, the electronic device 100 may change the cooking step including food ingredient 3 among the multiple cooking steps. In addition, the user may execute the selected cooking menu A by omitting the cooking step including food ingredient 3. For example, the electronic device 100 may store user identification information (e.g., ID information) and information about the omitted cooking steps. In addition, the number of times the cooking step including food ingredient 3 is omitted when executing the selected cooking menu A may be greater than or equal to the threshold number. In one or more examples, the electronic device 100 may store recipe information in which a cooking step including the food ingredient 3 is omitted from the recipe information of the cooking menu A as user data.

[0137] The electronic device 100 may identify at least one cooking step of the plurality of cooking steps based on user data regarding past cooking actions of the user. In one or more examples, the electronic device 100 may change some of the plurality of cooking steps included in the recipe information by adding or deleting some nodes in the recipe information or changing the contents of some nodes in the recipe information based on the user data.

[0138] For example, the electronic device 100 may obtain information that garlic, onion, mushroom, bacon, tomato, and sauce are not put into the pan at the same time when the user cooks "tomato spaghetti", and store it as user data. Therefore, as a result, the electronic device 100 may update the recipe information by deleting the node representing the step "putting garlic, onion, mushroom, bacon, tomato, and sauce into the pan" and deleting the node representing the next step "stir-frying the food ingredients for 4 minutes" after the step "putting garlic, onion, mushroom, bacon, tomato, and sauce into the pan".

[0139] In addition, the electronic device 100 may obtain the following information: when the user cooks "tomato spaghetti", the user puts the noodles into a pan, then stir-fries the noodles in the last 2 minutes, and adds Parmesan cheese powder thereto, and stores the information as user data. In one or more examples, in the recipe information before updating, the next node after the node representing the step "putting noodles into a pan" may represent the step "stir-frying for 1 minute". Therefore, as a result, the electronic device 100 may update the recipe information so that the node corresponding to the step "stir-frying for 1 minute" is updated to represent the step "stir-frying for 2 minutes", and a node representing the step "adding Parmesan cheese powder" is added after the node representing the step "stir-frying for 2 minutes".

[0140] Figure 5 is a flow chart illustrating a method for identifying a first cooking step and at least one cooking step to be performed after the first cooking step according to one or more embodiments of the present disclosure.

[0141] First, the electronic device 100 may identify a first cooking step included in the recipe information (S510). In one or more examples, the electronic device 100 may identify a first cooking step among a plurality of cooking steps included in the recipe information as the first cooking step. In one or more examples, the electronic device 100 may identify the first cooking step among the plurality of cooking steps based on pre-acquired context information. The description describes in detail below a method for identifying a cooking step based on context information.

[0142] The electronic device 100 may obtain information about the probability of performing a cooking step after the first cooking step (S520). In one or more examples, the electronic device 100 may obtain information about the probability of performing a cooking step after the first cooking step based on the distance between the node representing the first cooking step and the node representing the next cooking step after the first cooking step.

[0143] In one or more examples, such as Figure 4aAs shown, in the recipe information, the second node representing the second cooking step may be connected to the first node representing the first cooking step. In one or more examples, the electronic device 100 may determine that the probability of performing the second cooking step after the first cooking step is 50%. In addition, the third node representing the third cooking step may not be connected to the first node, but to the second node. Therefore, the electronic device 100 may determine that the probability of performing the third cooking step after the first cooking step is 25%. For example, the electronic device 100 may obtain information about the probability of performing a cooking step after the first cooking step based on the distance between the nodes included in the recipe information.

[0144] In one or more examples, in the recipe information, a first node representing a first cooking step may be connected to a second node representing a second cooking step, and a third node representing a third cooking step may also be connected to the first node. Therefore, as a result, the electronic device 100 may obtain information about the probability of performing a cooking step after the first cooking step by determining that the probability of performing the second cooking step after the first cooking step and the probability of performing the third cooking step after the first cooking step are the same as each other. For example, the electronic device 100 may determine that the probability of performing the second cooking step after the first cooking step is 50%, and the probability of performing the third cooking step after the first cooking step is 50%.

[0145] The electronic device 100 may identify at least one cooking step among the next cooking steps after the first cooking step based on the obtained probability (S530). In one or more examples, the electronic device 100 may identify, as at least one cooking step, a cooking step among the cooking steps that may be performed after the first cooking step, whose probability of being performed after the first cooking step is greater than or equal to a predetermined threshold (e.g., 20%).

[0146] For example, the electronic device 100 may determine that the probability of executing the second cooking step after the first cooking step is 50%, the probability of executing the third cooking step after the first cooking step is 25%, and the probability of executing the fourth cooking step after the first cooking step is 12.5%. Therefore, the electronic device 100 may identify the second cooking step and the third cooking step whose probability of execution after the first cooking step is greater than or equal to a predetermined threshold (e.g., 20%) as at least one cooking step.

[0147] In one or more examples, the electronic device 100 may identify at least one cooking step by identifying a predetermined number of cooking steps in order of higher probability of being performed after the first cooking step among the cooking steps that can be performed after the first cooking step. For example, the electronic device 100 may determine that the probability of performing the second cooking step after the first cooking step is 50%, the probability of performing the third cooking step after the first cooking step is 25%, and the probability of performing the fourth cooking step after the first cooking step is 12.5%. Therefore, the electronic device 100 may identify the second cooking step and the third cooking step corresponding to the predetermined number (e.g., two) as at least one cooking step in order of highest probability to lowest probability.

[0148] Figure 6 is a flowchart illustrating a method of loading a first neural network model according to one or more embodiments of the present disclosure.

[0149] In one or more examples, the electronic device 100 may identify at least one first neural network model corresponding to the identified first cooking step and at least one cooking step to be performed after the first cooking step (S610).

[0150] In one or more examples, the electronic device 100 may identify at least one first neural network model corresponding to a first cooking step and at least one cooking step to be performed after the first cooking step based on information about a neural network model required for each cooking step included in the recipe information. For example, the recipe information may include information about neural network models, such as "neural network model a corresponding to the first cooking step, neural network models b and c corresponding to the second cooking step, and neural network models c, d, and f corresponding to the third cooking step." In one or more examples, based on identifying the first to third cooking steps as the first cooking step and at least one cooking step to be performed after the first cooking step, the electronic device 100 may identify neural network models a, b, c, d, and f as at least one first neural network model.

[0151] In one or more examples, the recipe information may not include information about the neural network model required for each cooking step. When the information is not included in the recipe information, the electronic device 100 may identify at least one first neural network model corresponding to the first cooking step and at least one cooking step to be performed after the first cooking step based on the recipe information and the metadata of the neural network model. In one or more examples, the electronic device 100 may identify at least one first neural network model by comparing the food ingredients, cooking utensils, and user actions included in the image for each cooking step included in the recipe information with the information about the input / output data included in the metadata of the neural network model. For example, in the recipe information, the cooking utensils included in the first cooking step may be a "frying pan", the food ingredients may be "garlic, onions, and mushrooms", and the user action may be "put". Therefore, the electronic device 100 may identify at least one neural network model among the plurality of neural network models stored in the electronic device 100 based on the input / output data of the plurality of neural network models, and the at least one neural network model may identify the cooking utensils, food ingredients, and user actions of the first cooking step. For example, the electronic device 100 may identify a first object recognition model that can recognize a "frying pan", a second object recognition model that can recognize "onions, mushrooms, and garlic", and a first action recognition model that can recognize an action "put" as neural network models corresponding to the first cooking step. As a result of these recognitions, the electronic device 100 may recognize at least one first neural network model corresponding to the first cooking step and at least one cooking step to be performed after the first cooking step.

[0152] The electronic device 100 may determine whether there are a plurality of identified first neural network models (S620). For example, when it is determined that there are a plurality of neural network models to be loaded for the first cooking step and at least one cooking step performed after the first cooking step, it is determined that there are a plurality of first neural network models.

[0153] When there are multiple first neural network models (S620-Y), the electronic device 100 may determine the number of neural network models to be loaded simultaneously based on the resources of the electronic device 100 and the resources of the multiple first neural network models (S630). In one or more examples, the resources of the electronic device 100 may be information about resources currently available to the electronic device 100, including the amount of memory currently available to the electronic device 100, the current processing power of the electronic device 100 (CPU utilization, NPU or GPU utilization, etc.), etc. In addition, the resources of the first neural network model may be information about resources required to load the first neural network model. According to one or more embodiments of the present disclosure, the resources of the electronic device 100 may represent the total resources currently available to the electronic device 100. In one or more examples, the resources of the electronic device 100 may represent resources that take into account a critical ratio (e.g., 80%) of the total available resources for another task.

[0154] In one or more examples, the electronic device 100 may determine whether the sum of resources of the plurality of first neural network models is less than the resources of the electronic device 100 (S640).

[0155] When it is determined that the sum of the resources of the plurality of first neural network models is less than the resources of the electronic device 100 (S640-Y), the electronic device 100 may load the plurality of first neural network models simultaneously (S650). Figure 7a An example method for simultaneously loading multiple first neural network models is shown in more detail.

[0156] like Figure 7a As shown, the neural network model corresponding to the first cooking step 710 may be a neural network model a, the neural network model corresponding to the second cooking step 720 may be neural network models b and c, and the neural network model corresponding to the third cooking step 730 may be neural network models c, d, and f. In one or more examples, the electronic device 100 may identify the first cooking step (T1), and identify the second cooking step as the first cooking step and at least one subsequent cooking step after the first cooking step (T2). In addition, the electronic device 100 may identify the neural network models a, b, and c as a plurality of first neural network models corresponding to the first cooking step and the second cooking step. In one or more examples, when the sum of the resources of the neural network models a, b, and c is less than the resources of the electronic device 100, the electronic device 100 may load the neural network models a, b, and c simultaneously (T3).

[0157] Return to reference Figure 6, when the sum of the resources of the plurality of first neural network models is greater than or equal to the resources of the electronic device 100 (S640-Y), the electronic device 100 may determine the weight values ​​of the plurality of first neural network models (S660). For example, the electronic device 100 may determine the weight values ​​of the plurality of first neural network models based on the information on the probability of performing at least one cooking step after the first cooking step obtained in step S420 of FIG. 4 .

[0158] The electronic device 100 may sequentially load a plurality of first neural network models based on the weight values ​​(S670). Figure 7b An example method for sequentially loading multiple first neural network models is described in more detail.

[0159] like Figure 7b As shown, the neural network model corresponding to the first cooking step 710 may be a neural network model a, the neural network model corresponding to the second cooking step 720 may be a neural network model b and c, and the neural network model corresponding to the third cooking step 730 may be a neural network model c, d, and f. In one or more examples, the electronic device 100 may identify the first cooking step (T1), and identify the second cooking step as the first cooking step and at least one subsequent cooking step (T2) after the first cooking step. In addition, the electronic device 100 may identify the neural network models a, b, and c as a plurality of first neural network models corresponding to the first cooking step and the second cooking step. In one or more examples, when the sum of the resources of the neural network models a, b, and c is greater than or equal to the resources of the electronic device 100, the electronic device 100 may determine the weight values ​​of the neural network models a, b, and c based on the information about the probability of performing the second cooking step after the first cooking step. For example, the electronic device 100 may determine 1 as the weight value of the neural network model a, and determine 0.33 as the weight values ​​of the neural network models b and c based on the information about the probability of performing the second cooking step.

[0160] After determining the weight values ​​of the neural network models, the electronic device 100 may sequentially load the neural network models a, b, and c based on the weight values ​​of the neural network models (T3). Specifically, the electronic device 100 may load the neural network models in the order of aaabc, such as Figure 7bAs shown. Based on loading the neural network models in sequence, the electronic device 100 can obtain context information by inputting the sensing data obtained from at least one sensor to each of the neural network models loaded in sequence with a number of times corresponding to the weight value. For example, loading the neural network models in the order of aaabc can indicate that the neural network models b and c are used once for every three uses of the neural network model a. For example, based on loading the neural network models in the order of aaabc, the electronic device 100 can load the neural network model a, obtain the first context information by inputting the first sensing data into the loaded neural network model a, obtain the second context information by inputting the second sensing data into the loaded neural network model a, obtain the third context information by inputting the third sensing data into the loaded neural network model a, load the neural network model b, obtain the fourth context information by inputting the fourth sensing data into the loaded neural network model b, load the neural network model c, and obtain the fifth context information by inputting the fifth sensing data into the loaded neural network model c.

[0161] In one or more examples, the electronic device 100 may obtain sensing data from at least one sensor 110 at regular intervals and input it into the neural network model. In one or more examples, when a cooking step has an operation corresponding to a time period (e.g., stir-frying for 3 minutes), the electronic device 100 may obtain sensing data from at least one sensor 110 at a specific time point (e.g., the first time point, the middle time point, or the last time point) of the cooking step and input the sensing data into the neural network model.

[0162] In addition, based on loading the neural network model, the electronic device 100 can obtain sensing data by requesting sensing data from the sensor corresponding to the neural network model. For example, neural network model a can be an object recognition model, neural network model b can be a sound recognition model, and neural network model c can be an action recognition model. In one or more examples, based on loading the neural network model a, the electronic device 100 can request image data from the camera 111 corresponding to the object recognition model. In addition, when the object recognition operation is performed by the neural network model a or before the object recognition operation is performed by the neural network model a, the electronic device 100 can request sound data from the microphone 113 corresponding to the neural network model b.

[0163] In the above examples, the electronic device 100 is described as loading the multiple first neural network models one by one based on the sequential loading of the multiple first neural network models. However, in one or more examples, the electronic device 100 may load at least two of the multiple first neural network models simultaneously based on the resources of the electronic device 100 and the resources of the multiple neural network models. Figure 7bThe resources of the neural network models b and c corresponding to the second cooking step shown may be less than the resources of the electronic device 100. Therefore, the electronic device 100 may load the neural network models in the order of aaa-(b and c). For example, the electronic device 100 may load the neural network model a three times and simultaneously load the neural network models b and c once.

[0164] Return to reference Figure 6 , based on determining that there is a first neural network model (S620-N), the electronic device 100 can load a first neural network model (S680).

[0165] like Figure 6 As shown, the electronic device 100 can load at least one first neural network model based on the resources of the electronic device 100 and the resources of the at least one first neural network model. Therefore, even when the resources for loading the neural network model are insufficient, the electronic device 100 can recognize the cooking situation in real time by using the neural network model in the local environment, and perform control corresponding to the recognized cooking situation.

[0166] In addition, based on the type of at least one sensor 110 and at least one of the illuminance values ​​obtained from the illuminance sensor 115, the electronic device 100 can load at least one first neural network model among the multiple neural network models corresponding to the first cooking step and at least one cooking step to be performed after the first cooking step.

[0167] In one or more examples, based on comparing information about the type of at least one sensor 110 of the electronic device 100 with metadata (e.g., input data) of the plurality of neural network models, the electronic device 100 may load at least one first neural network model corresponding to the first cooking step and at least one cooking step to be performed after the first cooking step among the plurality of neural network models. For example, the electronic device 100 may include only the camera 111 without the microphone 113. Based on this configuration, the electronic device 100 may load only an object recognition model or an action recognition model among the plurality of neural network models based on the metadata (e.g., input data) without loading a sound recognition model.

[0168] In one or more examples, the electronic device 100 may identify at least one first neural network model based on loading the first neural network model by comparing information about the type of at least one sensor 110 with metadata of the plurality of neural network models. In one or more examples, the electronic device 100 may download and store only some of the plurality of neural network models based on the type of the at least one sensor 110. For example, the electronic device 100 may include only the camera 111 without the microphone 113. Based on this configuration, the electronic device 100 may download and store only an object recognition model or an action recognition model among the plurality of neural network models.

[0169] In one or more examples, based on comparing information about the performance of at least one sensor 110 included in the electronic device 100 with metadata (specifically, input data), the electronic device 100 may load at least one first neural network model corresponding to a first cooking step and at least one cooking step to be performed after the first cooking step among multiple neural network models. For example, the camera 111 included in the electronic device 100 may be a camera that captures high-definition (HD) images. Based on this configuration, the electronic device 100 may load only an object recognition model or an action recognition model that can obtain context information by inputting an HD image into a loaded neural network model among the multiple neural network models, without loading an object recognition model or an action recognition model that can obtain context information by inputting a full HD image.

[0170] In one or more examples, based on a comparison of the illuminance value obtained from the illuminance sensor 115 with the metadata of the plurality of neural network models, the electronic device 100 may load at least one first neural network model corresponding to the first cooking step and at least one cooking step to be performed after the first cooking step from among the plurality of neural network models. In one or more examples, in a bright environment where the illuminance value obtained from the illuminance sensor 115 is greater than or equal to a threshold value, the electronic device 100 may load at least one first neural network model having a first performance by referring to the metadata of the plurality of neural network models. In a dark environment where the illuminance value obtained from the illuminance sensor 115 is less than a threshold value, the electronic device 100 may load at least one first neural network model having a second performance that is better than the first performance by referring to the metadata of the plurality of neural network models.

[0171] In the above examples, the electronic device 100 is described as identifying at least one neural network model to be loaded based on the illuminance value. In one or more examples, when the illuminance value changes while cooking is performed, the electronic device 100 can identify and load a neural network model corresponding to the changed cooking value among the plurality of neural network models.

[0172] Then, the electronic device 100 may determine whether a neural network model change event (eg, Figure 3 In one or more examples, when obtaining context information, the electronic device 100 may determine whether a neural network model change event is detected based on the obtained context information. In one or more examples, the neural network model change event may be an event for changing the neural network model, and may be at least one of an event of detecting a cooking situation in which a second cooking step different from the first cooking step is performed or an event of detecting a new cookware not included in the recipe information.

[0173] Figure 8 is a flowchart illustrating a method for loading a second neural network model based on a neural network model change event according to one or more embodiments of the present disclosure.

[0174] The electronic device 100 may obtain context information (S810). For example, the electronic device 100 may obtain context information output from at least one first neural network model. In one or more examples, the electronic device 100 may obtain and monitor the context information output from at least one first neural network model in real time.

[0175] Then, the electronic device 100 may detect whether the second cooking step is performed based on the context information (S820). In one or more examples, the second cooking step may represent a cooking step that can be performed after the first cooking step. For example, the electronic device 100 may detect whether the second cooking step is performed by identifying a second cooking step that matches the obtained context information among a plurality of cooking steps included in the recipe information. For example, based on obtaining context information such as an action “stir-fry onions” while monitoring the context information output by the neural network model, the electronic device 100 may detect that the “second cooking step” among the plurality of cooking steps is performed, such as Figure 4a shown.

[0176] Based on detecting that the second cooking step is performed (S820-Y), the electronic device 100 may determine whether to change the recipe information (S840). In one or more examples, changing the recipe information may mean deleting or changing at least one of the multiple nodes in the graph representing the recipe information, or adding a new node. Fig. 9 A description describing whether the electronic device 100 changes the recipe information.

[0177] refer to Fig. 9 , the electronic device 100 may determine whether there is a cooking step between the first cooking step and the second cooking step (S910). For example, the electronic device 100 may determine whether the next cooking step is performed immediately after the first cooking step.

[0178] When it is determined that there is a cooking step between the first cooking step and the second cooking step (S910-Y), the electronic device 100 may determine whether the cooking step between the first cooking step and the second cooking step is related to the next cooking step after the second cooking step (S920). In one or more examples, the electronic device 100 may determine whether the food ingredients and cooking utensils included in the cooking step between the first cooking step and the second cooking step are used in the next cooking step after the second cooking step. For example, "onion" may be a food ingredient included in the cooking step between the first cooking step and the second cooking step, and "stir-fry onions" may be included in the next cooking step after the second cooking step. Therefore, as a result, the electronic device 100 may determine that the cooking step between the first cooking step and the second cooking step is related to the next cooking step after the second cooking step.

[0179] Based on determining that the cooking step between the first cooking step and the second cooking step is related to the next cooking step after the second cooking step (S920-Y), the electronic device 100 may change the recipe information (S930). For example, the electronic device 100 may identify the cooking steps related to the omitted cooking steps in the cooking steps after the second cooking step (e.g., the cooking steps between the first cooking step and the second cooking step), and delete or change the nodes corresponding to the identified cooking steps. For example, "onion" may be a food ingredient included in the cooking steps between the first cooking step and the second cooking step, and "stir-fry onions" may be included in the next cooking step after the second cooking step. Therefore, as a result, the electronic device 100 may delete the cooking step "stir-fry onions" related to "onion" among the multiple cooking steps included in the recipe information.

[0180] However, when it is determined that there is no cooking step between the first cooking step and the second cooking step (S910-N), or when it is determined that the cooking step between the first cooking step and the second cooking step is irrelevant to the next cooking step after the second cooking step (S920-N), the electronic device 100 may maintain the recipe information (S940).

[0181] Return to reference Figure 8 Based on detecting that the second cooking step is not performed (S820-N), the electronic device 100 may determine whether new cookware is detected based on the context information (S830). In one or more examples, the electronic device 100 may determine whether new cookware not included in the recipe information is detected based on the context information obtained through the object recognition model.

[0182] In one or more examples, the electronic device 100 may determine whether new cookware is detected based on context information obtained by an object recognition model for detecting cookware. For example, "pan" may be information about cookware obtained by an object recognition model for recognizing cookware, and "pot" is a cookware included in the recipe information. Therefore, as a result, the electronic device 100 may determine that new cookware is detected based on the context information.

[0183] In one or more examples, the electronic device 100 may not recognize the cookware included in the recipe information by using the context information during the threshold time when loading the object recognition model for recognizing the cookware included in the recipe information. Therefore, as a result, the electronic device 100 may load another object recognition model for recognizing new cookware, and may determine whether the new cookware is detected by the other object recognition model. For example, "pot" may be the cookware included in the recipe information. In one or more examples, the electronic device 100 may load the object recognition model for recognizing "pot", and then obtain the context information by the object recognition model for recognizing "pot". In one or more examples, the electronic device 100 may not recognize the "pot" during the threshold time when obtaining the context information. Therefore, as a result, the electronic device 100 may load the object recognition model for recognizing another cookware, and detect the new cookware by the loaded object recognition model. In one or more examples, the object recognition model used to recognize another cookware can be an object recognition model that can recognize cookware with a superordinate concept of "pot" (e.g., cookware similar to food ingredients), or an object recognition model that can recognize a cookware type different from the cookware type of "pot" (e.g., a frying pan or a ladle).

[0184] The above example describes a case where a new cookware not included in the recipe information is detected. However, embodiments of the present disclosure may be applied to a case where a new food ingredient not included in the recipe information is detected.

[0185] Return to reference Figure 8, based on determining that the recipe information changes (S840-Y) or detecting new cookware (S830-Y) after detecting whether the second cooking step is performed, the electronic device 100 may change the recipe information (S850). In one or more examples, as described above, the electronic device 100 may identify the cooking steps related to the omitted cooking steps (e.g., the cooking steps between the first cooking step and the second cooking step) in the cooking steps after the second cooking step, and delete or change the nodes corresponding to the identified cooking steps. In one or more examples, based on detecting new cookware or food ingredients, the electronic device 100 may request an external server to retrieve a recipe including the detected new cookware or food ingredients, and receive recipe information including the detected new cookware or food ingredients from the external server. For example, a user may select "cook instant noodles" as a cooking menu, and then "frying pan" may be detected as a cookware not included in the recipe information. Therefore, as a result, the electronic device 100 may change the recipe information by receiving recipe information about "cooking instant noodles using a frying pan" from an external server.

[0186] The electronic device 100 may identify the second cooking step and at least one next cooking step after the second cooking step (S860). In one or more examples, the electronic device 100 may identify the second cooking step and at least one next cooking step after the second cooking step based on the recipe information (or the changed recipe information) and the context information. For example, when the cookware is changed to a new cookware, the electronic device 100 may identify the second cooking step including the changed new cookware, and identify at least one cooking step in the cooking steps after the second cooking step. Figure 3 and Figure 5 This configuration is described in detail in step S330 of , so this description omits its redundant description.

[0187] In one or more examples, the information about the second cooking step detected in step S820 may be information about the completion of the second cooking step. Therefore, as a result, the electronic device 100 may only recognize at least one next cooking step after the second cooking step instead of the second cooking step. For example, based on recognizing the cooking step "boil water" in step S820, the electronic device 100 may recognize the cooking step "boil water" and the next cooking step because "boil water" is not completed. However, based on recognizing the cooking step "put garlic and onion in a pan" in step S820, the electronic device 100 may recognize the next cooking step after the cooking step "put garlic and onion in a pan" because the cooking step "put garlic and onion in a pan" is completed.

[0188] The electronic device 100 may load at least one second neural network model corresponding to the second cooking step and at least one next cooking step after the second cooking step (S870). In one or more examples, the electronic device 100 may load at least one second neural network model corresponding to the second cooking step and at least one cooking step to be performed after the second cooking step from among the plurality of neural network models based on the recipe information and the resources of the electronic device 100. Figure 3 Step S340 and Figure 6 to Figure 7b The configuration is described in detail in , so this description omits its redundant description.

[0189] The following description refers to Fig.10 , Fig.11 , Fig.12 , Fig.13 and Fig.14 An example of dynamically loading a neural network model based on a neural network model change event is described.

[0190] Fig.10 is a view showing a first example of an event of detecting a cooking situation in which a second cooking step different from a first cooking step is performed according to one or more embodiments of the present disclosure. First, the recipe information of the cooking menu selected by the user may include the first cooking step 1010 to the fifth cooking step 1050. According to one or more embodiments of the present disclosure, the neural network model for identifying the execution of the first cooking step 1010 may be a neural network model a, the neural network model for identifying the execution of the second cooking step 1020 may be neural network models b and c, the neural network model for identifying the execution of the third cooking step 1030 may be neural network models c, d and f, the neural network model for identifying the execution of the fourth cooking step 1040 may be neural network models f and g, and the neural network model for identifying the execution of the fifth cooking step 1050 may be neural network models h and i. In one or more examples, the neural network model for identifying the execution of the first cooking step is described as model a. However, model a_1 or a_2 may be further included based on the performance and type of the neural network model. Figures 10 to 14 The times T1 to T8 disclosed in the drawings may represent the order of operations rather than specific time points.

[0191] First, the electronic device 100 may identify the first cooking step 1010 (T1). For example, the electronic device 100 may identify the first cooking step 1010 as the first cooking step. In addition, the electronic device 100 may identify the second cooking step 1020 after the first cooking step 1010 based on the recipe information. For example, the electronic device 100 may identify the first cooking step 1010 and the second cooking step 1020 (T2).

[0192] The electronic device 100 may load at least one neural network model corresponding to the first cooking step 1010 and the second cooking step 1020. For example, the electronic device 100 may sequentially load the neural network models based on each weight value of the neural network model. For example, the electronic device 100 may sequentially load the neural network models in the order of aaabc (T3). Although the above example describes that the electronic device 100 sequentially loads the neural network models in the order of aaabc, as described above, when it is determined that the sum of the resources of the neural network models a, b, and c is less than the resources of the electronic device 100, the electronic device 100 may load the neural network models a, b, and c at the same time. However, for ease of explanation, the description is referred to. Figures 10 to 14 Only sequential loading is assumed and described by excluding simultaneous loading.

[0193] While monitoring the context information, the electronic device 100 may detect a cooking situation “cooking BB” corresponding to the second cooking step 1020 ( T4 ).

[0194] The electronic device 100 may identify the second cooking step 1020 and the third cooking step 1030 corresponding to the cooking situation “cooking BB”, the third cooking step 1030 being the next cooking step after the second cooking step 1020 ( T5 ).

[0195] The electronic device 100 may sequentially load models b, c, d, and f as neural network models corresponding to the second cooking step 1020 and the third cooking step 1030 in the order of bcbcdf (T6).

[0196] Fig.11 2 is a view showing a second example of an event of detecting a cooking situation in which a second cooking step different from a first cooking step is performed according to one or more embodiments of the present disclosure. Fig.10 As shown, the recipe information of the cooking menu selected by the user may include first to fifth cooking steps 1010 to 1050 .

[0197] First, the electronic device 100 may identify the first cooking step 1010 (T1). For example, the electronic device 100 may identify the first cooking step 1010 as the first cooking step. In addition, the electronic device 100 may identify the second cooking step 1020 and the third cooking step 1030 after the first cooking step 1010 based on the recipe information. For example, the electronic device 100 may identify the first cooking step 1010 to the third cooking step 1030 (T2).

[0198] The electronic device 100 may sequentially load models a, b, c, d, and f as neural network models corresponding to the first to third cooking steps 1010 to 1030 in the order of aaabcbcdf (T3).

[0199] The electronic device 100 may detect the cooking situation "Cooking CC" (T4) corresponding to the third cooking step 1030 based on the context information obtained when the neural network models a, b, c, d, and f corresponding to the first cooking step 1010 to the third cooking step 1030 are sequentially loaded. In one or more examples, the electronic device 100 skips the second cooking step 1020 and performs the third cooking step 1030, and thus may determine whether to change the recipe information. However, based on determining that the second cooking step 1020 is not related to the next cooking step, the electronic device 100 may maintain the recipe information unchanged.

[0200] The electronic device 100 may identify a third cooking step 1030 corresponding to the cooking condition “Make CC” and a fourth cooking step 1040 which is a next cooking step after the third cooking step 1030 ( T5 ).

[0201] The electronic device 100 may sequentially load models c, d, f, and g as neural network models corresponding to the third cooking step 1030 and the fourth cooking step 1040 in the order of cdfcdfg (T6).

[0202] Fig.12 2 is a view showing a third example of an event of detecting a cooking situation in which a second cooking step different from a first cooking step is performed according to one or more embodiments of the present disclosure. Fig.10 and Fig.11 As shown, the recipe information of the cooking menu selected by the user may include first to fifth cooking steps 1010 to 1050 .

[0203] First, the electronic device 100 may identify the first cooking step 1010 (T1). For example, the electronic device 100 may identify the first cooking step 1010 as the first cooking step. In addition, the electronic device 100 may identify the second cooking step 1020 to the fourth cooking step 1040 after the first cooking step 1010 based on the recipe information. For example, the electronic device 100 may identify the first cooking step 1010 to the fourth cooking step 1040 (T2).

[0204] The electronic device 100 may sequentially load models a, b, c, d, f, and g as neural network models corresponding to the first to fourth cooking steps 1010 to 1040 in the order of aaabcbcdfg (T3).

[0205] While monitoring the context information, the electronic device 100 may detect a cooking situation “Done DD” (T4) corresponding to the fourth cooking step 1040. In one or more examples, the electronic device 100 skips the second cooking step 1020 and the third cooking step 1030 and performs the fourth cooking step 1040, and thus may determine whether to change the recipe information.

[0206] Based on determining that at least one of the second cooking step 1020 and the third cooking step 1030 is related to the fifth cooking step 1050 as the next cooking step, the electronic device 100 may change the recipe information (T5). For example, the electronic device 100 may change the fifth cooking step 1050 to the fifth' cooking step 1060, and determine the models h' and i' as the neural network models corresponding to the fifth' cooking step 1060.

[0207] The electronic device 100 may identify the fourth cooking step 1040 corresponding to the cooking condition “Done DD” and the fifth cooking step 1060, which is the next cooking step after the fourth cooking step 1040, (T6).

[0208] The electronic device 100 may sequentially load models f, g, h', and i' as neural network models corresponding to the fourth cooking step 1040 and the fifth cooking step 1060 in the order of fgfg-h'-i' (T7).

[0209] Fig.13 2 is a diagram showing an example of detecting a new cooker according to one or more embodiments of the present disclosure. Figures 10 to 12 As shown, the recipe information of the cooking menu selected by the user may include first to fifth cooking steps 1010 to 1050 .

[0210] First, the electronic device 100 may identify the first cooking step 1010 (T1). For example, the electronic device 100 may identify the first cooking step 1010 as the first cooking step. In addition, the electronic device 100 may identify the second cooking step 1020 after the first cooking step 1010 based on the recipe information. For example, the electronic device 100 may identify the first cooking step 1010 and the second cooking step 1020 (T2).

[0211] The electronic device 100 may sequentially load models a, b, and c as neural network models corresponding to the first cooking step 1010 and the second cooking step 1020 in the order of aaabc (T3). In one or more examples, the neural network model a may be an object recognition model that can recognize cookware.

[0212] The electronic device 100 may detect a new cooker while monitoring the context information (T4). For example, the electronic device 100 may detect a new cooker that is not included in the context information through the neural network model a.

[0213] The electronic device 100 may change the recipe information (T5). For example, the electronic device 100 may change the third cooking step 1030 associated with the new cookware to the third 'cooking step 1070, and change the fourth cooking step 1040 to the fourth 'cooking step 1080. In one or more examples, the electronic device 100 may determine the models c', d', f' as the neural network models corresponding to the third 'cooking step 1070, and determine the models f' and g as the neural network models corresponding to the fourth 'cooking step 1080.

[0214] While monitoring the context information, the electronic device 100 may detect a cooking situation “cooking CC” corresponding to the third 'cooking step 1070 (T6).

[0215] The electronic device 100 may identify a third 'cooking step 1070 corresponding to the cooking situation 'Cook CC' and a fourth 'cooking step 1080 which is a next cooking step after the third 'cooking step 1070 (T7).

[0216] The electronic device 100 may sequentially load models c', d', f', and g as neural network models corresponding to the third' cooking step 1070 and the fourth' cooking step 1080 in the order of c'-d'-f'-c'-d'-f'-g (T8).

[0217] Fig.14 is a view showing an example of reducing the number of loaded neural network models based on context information after loading the neural network models corresponding to all cooking steps according to one or more embodiments of the present disclosure. The recipe information of the cooking menu selected by the user may include first to third cooking steps (1410 to 1430). For example, the neural network model for identifying the execution of the first cooking step 1410 may be neural network model a, the neural network model for identifying the execution of the second cooking step 1420 may be neural network models b and c, and the neural network model for identifying the execution of the third cooking step 1430 may be neural network models c, d, and f.

[0218] First, the electronic device 100 may recognize all the cooking steps 1410 to 1430 (T1). For example, the electronic device 100 may initially recognize all the cooking steps 1410 to 1430 instead of recognizing a single cooking step.

[0219] In addition, the electronic device 100 may sequentially load a plurality of neural network models based on each weight value of the neural network model corresponding to all cooking steps 1410 to 1430. For example, the electronic device 100 may sequentially load the neural network models in the order of aaaabcbcdf (T2).

[0220] While monitoring the context information, the electronic device 100 may detect a cooking situation “cooking BB” ( T3 ) corresponding to the second cooking step 1420 .

[0221] The electronic device 100 may identify the next cooking step after the second cooking step 1420 corresponding to the cooking condition "cooking BB", and sequentially load a plurality of neural network models corresponding to the identified cooking steps. For example, the electronic device 100 may sequentially load the neural network models in the order of bcbcdf (T4).

[0222] While monitoring the context information, the electronic device 100 may detect a cooking situation “cooking CC” corresponding to the third cooking step 1430 ( T5 ).

[0223] The electronic device 100 may sequentially load a plurality of neural network models corresponding to the third cooking step 1430 corresponding to the cooking situation “make CC”. For example, the electronic device 100 may sequentially load the neural network models in the order of df (T6).

[0224] like Figures 10 to 14 As shown, the electronic device 100 can dynamically load the neural network model based on the context information to obtain the context information as quickly and efficiently as possible by using the neural network model in the local environment.

[0225] Fig.15 is a flowchart illustrating an example of controlling the operation of an electronic device based on context information according to one or more embodiments of the present disclosure.

[0226] First, the electronic device 100 may obtain context information (S1510). In one or more examples, as Figure 3 As described in step S350 of , the context information may be information about the cooking situation obtained through at least one neural network model. For example, when the selected dish is "fried rice", the electronic device 100 may obtain through the neural network model that the cookware is a "stainless steel pan" or a "coated pan", and the burner on which the cookware is located is a "third burner 133".

[0227] The electronic device 100 may search the knowledge graph included in the recipe information based on the context information (S1520). For example, the electronic device 100 may search for a node corresponding to a cooking step including context information in the knowledge graph included in the recipe information. For example, based on obtaining context information about "the stainless steel pan is located on the third burner 133", the electronic device 100 may search for nodes corresponding to "preheating {stainless steel pan} with {medium-high fire} for {5 minutes}" and "stir-frying {rice, vegetables} in {stainless steel pan} with {medium fire}", which is a cooking step including information about the "stainless steel pan". In one or more examples, based on obtaining context information about "the coated pan is located on the third burner 133", the electronic device 100 may search for nodes corresponding to "preheating {coated pan} with {high fire} for {30 seconds}" and "stir-frying {rice, vegetables} in {coated pan} with {high fire}", which is a cooking step including information about the "coated pan".

[0228] Therefore, the electronic device 100 can control the heater 130 based on the search results (1530). In one or more examples, the electronic device 100 can identify the burner on which the cooker is located based on the context information. In addition, the electronic device 100 can identify the operation stage of the heater 130 corresponding to the fire intensity corresponding to the retrieved node (for example, one of the first to ninth stages), and the electronic device 100 can identify that the operation time of the heater 130 corresponds to the operation time corresponding to the retrieved node. In addition, the electronic device 100 can control the burner identified as operating at the identified operation stage during the identified operation time.

[0229] For example, based on obtaining the contextual information about "the stainless steel pan is located on the third burner 133", the electronic device 100 can control the third burner 133 to operate in the "sixth stage" corresponding to the medium-high fire for 5 minutes based on the retrieved node "preheat {stainless steel pan} with {medium-high fire} for {5 minutes}". In addition, based on obtaining the contextual information about "putting rice and vegetables into the stainless steel pan located on the third burner 133", the electronic device 100 can control the third burner 133 to operate in the "fifth stage" corresponding to the medium fire based on the retrieved node "stir-fry {rice, vegetables} in {stainless steel pan} with {medium fire}". For another example, based on obtaining the contextual information about "the coated pan is located on the third burner 133", the electronic device 100 can control the third burner 133 to operate in the "eighth stage" corresponding to the high fire for 30 seconds based on the retrieved node "preheat {coated pan} with {high fire} for {30 seconds}". In addition, based on obtaining the contextual information about "putting rice and vegetables into the coated pan located on the third burner 133", the electronic device 100 can control the third burner 133 to operate in the "eighth stage" corresponding to high fire based on the retrieved node "stir-fry {rice, vegetables} in {coated pan} at {high fire}".

[0230] In one or more examples, based on the contextual information "the color of the onion becomes transparent" obtained through the object recognition model when executing the cooking step of "frying the onions until transparent", the electronic device 100 can determine that the retrieved node "frying the onions until transparent" is completed, and control the heater 130 to stop the operation of the electronic device 100 or lower the temperature.

[0231] For example, the electronic device 100 may actively control the heater 130 based on the obtained context information.

[0232] In addition, multiple dishes can be cooked on at least two of the multiple burners 131 to 133. Therefore, as a result, the electronic device 100 can identify each position of the burner on which the multiple dishes are cooked, and control the corresponding burner. For example, based on obtaining context information that the first cooking is performed on the first burner 131 and the second cooking is performed on the second burner 132, the electronic device 100 can control the first burner 131 and the second burner 132 to operate differently based on the context information and the recipe information.

[0233] In addition, the electronic device 100 can control not only the operation of the heater 130 based on the context information, but also the operation of another component (e.g., a camera). For example, when a cooking step (e.g., cutting vegetables) that requires sensing of a user's operation is identified based on the context information, the electronic device 100 can control the position photographed by the camera 111 to change from the heater 130 to the area where the user is located (e.g., a shelf area).

[0234] In addition, the electronic device 100 may output a guide message for the operation of the electronic device 100 before and after automatically performing the operation of the electronic device 100. For example, before preheating the stainless steel pan at medium-high heat for 5 minutes, the electronic device 100 may output a guide message indicating "the stainless steel pan will be preheated at medium-high heat for 5 minutes" through the display or through the speaker 162, or send the guide message to an external terminal device. In addition, after preheating the stainless steel pan at medium-high heat for 5 minutes, the electronic device 100 may output a guide message indicating "the stainless steel pan was preheated at medium-high heat for 5 minutes" through the display or through the speaker 162, or send the guide message to an external terminal device.

[0235] In one or more examples, the electronic device 100 may output only a guidance message including information about the operation of the electronic device 100 without automatically controlling the operation of the electronic device 100 based on the user setting. For example, in the case where the automatic control function is turned off by the application, the electronic device 100 may output only a guidance message including information about the operation of the electronic device 100 without automatically controlling the operation of the electronic device 100.

[0236] In one or more examples, the electronic device 100 is described as controlling the operation of the electronic device 100 based on the context information and the recipe information, which is merely an example, and the electronic device 100 may control the operation of the electronic device 100 based on the context information.

[0237] For example, based on detecting the context information “water is boiling” through the object recognition model, the electronic device 100 may control the heater 130 to stop the operation of the electronic device 100 regardless of the recipe information.

[0238] In the above examples, the electronic device 100 is described as controlling the operation of the electronic device 100 or dynamically loading the neural network model based on the context information obtained through the neural network model, which is only an example, and the electronic device 100 can be operated externally in conjunction with the user terminal.

[0239] Fig.16 2 is a diagram showing a control system including an electronic device and a user terminal according to one or more embodiments of the present disclosure. Fig.16As shown, the control system 10 may include an electronic device 100 and a user terminal 1600 .

[0240] In one or more examples, the user terminal 1600 may receive, through an application for controlling the electronic device 100, a user command for selecting a cooking menu, a user command for controlling the operation of the electronic device 100, a user command for checking a current cooking status, or any other user command known to a person of ordinary skill in the art.

[0241] The user terminal 1600 may control the operation of the electronic device 100 based on the received user command.

[0242] Specifically, based on receiving a user command to select a cooking menu, the user terminal 1600 may send information about the selected cooking menu to the electronic device 100. The electronic device 100 may obtain recipe information based on the received information about the selected cooking menu. In one or more examples, the user terminal 1600 may send the recipe information of the selected cooking menu together with the information about the selected cooking menu.

[0243] In addition, based on receiving a user command to control the operation of the electronic device 100, the user terminal 1600 may send information for controlling the operation of the electronic device 100 to the electronic device 100. For example, in the display such as Fig.17 17. When the user terminal 1600 displays the user interface (UI) screen shown in FIG. 17, an icon 1710 corresponding to the electronic device 100 is selected. Therefore, as a result, the user terminal 1600 may display a control screen for controlling the operation of the electronic device 100. Based on receiving a user command to control the electronic device 100 through the control screen, the user terminal 1600 may transmit information corresponding to the selected command to the electronic device 100. For example, based on receiving a user command to stop the operation of the heater 130, the user terminal 1600 may transmit information including "stop the operation of the heater 130" to the electronic device 100.

[0244] In addition, based on receiving a user command to check the current cooking state of the electronic device 100, the user terminal 1600 may transmit a signal for requesting information about the current cooking state to the electronic device 100. Based on receiving the information about the current cooking state from the electronic device 100, the user terminal 1600 may display information about the current cooking state based on the information about the current cooking state received from the electronic device 100. Fig.18 The UI screen 1810 is shown. For example, information about the current cooking step, information about the control status of the plurality of burners 131 to 133, and information about the cooking status may be displayed on the UI screen 1810.

[0245] In the above examples, when it is determined that the user terminal 1600 transmits a signal for requesting information about the current cooking state to the electronic device 100, the electronic device 100 is described as transmitting information about the current cooking state to the user terminal 1600. In one or more examples, the electronic device 100 may transmit the information about the current cooking state to the user terminal 1600 at predetermined intervals. In one or more examples, when it is determined that a cooking step end event or a cooking end event may occur, the electronic device 100 may transmit the information about the current cooking state to the user terminal 1600.

[0246] As described above, the user may remotely control the electronic device 100 or check information on the cooking status by controlling the electronic device 100 or checking the current cooking status via the user terminal 1600 .

[0247] In the above examples, a plurality of neural network models are described as being stored in the electronic device 100. In one or more examples, at least some of the neural network models may be stored in the server 10, and the electronic device 100 may obtain context information by receiving the neural network models from the server 10.

[0248] Fig.19 is a timing diagram illustrating a control method of a system for performing a cooking operation based on context information according to one or more embodiments of the present disclosure.

[0249] First, the user terminal 200 may receive a user input for selecting a cooking menu (S1905). In one or more examples, the user input for selecting a cooking menu may be a user input for directly inputting text of the cooking menu or selecting an icon. In one or more examples, the user input may be a user input for photographing identification information (e.g., a QR code or a barcode) of the cooking menu.

[0250] The user terminal 200 may transmit information about a cooking menu to the electronic device 100 (S1910).

[0251] The electronic device 100 may obtain recipe information corresponding to the selected cooking menu (S1915). In one or more examples, the electronic device 100 may obtain recipe information pre-stored in the memory 120, which is only an example, and may receive the recipe information from an external recipe server.

[0252] The electronic device 100 may identify a cooking step (S1920). In one or more examples, as shown in FIG. Figures 3 to 5 As shown, the electronic device 100 may identify a first cooking step and at least one cooking step to be performed after the first cooking step.

[0253] The electronic device 100 may search for a neural network model stored in the electronic device 100 (S1925). In one or more examples, the electronic device 100 may search for at least one neural network model corresponding to the identified cooking step among a plurality of pre-stored neural network models based on information about the first cooking step and at least one cooking step to be performed after the first cooking step. In one or more examples, the information about the first cooking step and at least one cooking step to be performed after the first cooking step may include information about {food ingredients}, {cooking utensils}, and {actions} included in a node corresponding to each cooking step.

[0254] Based on the failure to retrieve the neural network model for identifying the first cooking step and at least one cooking step to be performed after the first cooking step (S1925-N), the electronic device 100 may send information about the cooking step to the server 10 (S1930). In one or more examples, the electronic device 100 may send information about the first cooking step and at least one cooking step to be performed after the first cooking step to the server 10. In one or more examples, the electronic device 100 may send identification information, resource information, and surrounding environment information (e.g., illumination value) of the current electronic device 100 together.

[0255] The server 10 may search for a neural network model based on the received information (S1940). In one or more examples, the server 10 may search for at least one neural network model that can output data matching information about {food ingredients}, {cooking utensils}, and {actions} included in nodes corresponding to the first cooking step and at least one cooking step to be performed after the first cooking step. In addition, the server 10 may search for at least one neural network model having resources or performance corresponding to the electronic device 100 based on the identification information, resource information, and surrounding environment information of the electronic device 100.

[0256] The server 10 may transmit information about the retrieved neural network model to the electronic device 100 (S1945).

[0257] Based on retrieving the neural network model (S1925-Y) or receiving information about the retrieved neural network model from the server 10 (S1945), the electronic device 100 may load the neural network model (S1950). In one or more examples, the electronic device 100 may load at least one of the at least one neural network model retrieved from the electronic device 100 or the at least one neural network model received from the server 10. In one or more examples, as Figure 6 As shown, the electronic device 100 can load at least one neural network model simultaneously or sequentially.

[0258] The electronic device 100 may obtain context information (S1955). For example, the electronic device 100 may obtain context information about the current cooking situation by inputting the sensing data into at least one loaded neural network model.

[0259] The electronic device 100 may transmit the context information to the user terminal 200 (1960), and the user terminal 200 may display the context information (S1965). Fig.18 User interface (UI) 1810 is shown displaying contextual information.

[0260] Then, the electronic device 100 may control the operation of the electronic device 100 based on the context information (S1970). Fig.15 As shown, the electronic device 100 can control the operation of the electronic device 100 based on the context information.

[0261] According to the present disclosure, functions of the electronic device 100 related to artificial intelligence technology may be operated by a processor and a memory of the electronic device 100 .

[0262] The processor may include one or more processors. In one or more examples, the one or more processors may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), or a neural processing unit (NPU), and is not limited to the examples of the above processors.

[0263] The CPU may be a general-purpose processor that can perform not only general operations but also artificial intelligence operations, and can efficiently execute complex programs through a multi-layer cache structure. The CPU may be advantageous for a serial processing method that enables an organic link between the previous operation result and the next operation result through sequential operations. Unless specified as the above-mentioned CPU, the general-purpose processor is not limited to the above-mentioned examples.

[0264] A GPU may be a processor for large-scale operations such as floating-point operations for graphics processing, and may perform large-scale operations in parallel by integrating a large number of cores. Specifically, a GPU may be advantageous for parallel processing methods such as convolution operations, etc., compared to a CPU. In addition, a GPU may be used as a coprocessor to supplement the functions of a CPU. Unless specified as the above-mentioned GPU, the processor for large-scale operations is not limited to the above-mentioned examples.

[0265] NPU is a processor dedicated to artificial intelligence operations using artificial neural networks, and each layer included in the artificial neural network can be implemented in hardware (e.g., silicon). In one or more examples, the NPU is specially designed based on the company's requirements and may have a lower degree of freedom than a CPU or GPU. However, the NPU can efficiently handle the artificial intelligence operations required by the company. As a processor dedicated to artificial intelligence operations, the NPU can be implemented in various forms such as a tensor processing unit (TPU), an intelligent processing unit (IPU), a visual processing unit (VPU), etc. Unless specified as the above-mentioned NPU, the artificial intelligence processor is not limited to the above examples.

[0266] Furthermore, one or more processors may be implemented in a system on chip (SoC). In one or more examples, in addition to one or more processors, the SoC may also include a memory and a network interface, such as a bus for data communication between the processor and the memory.

[0267] When a system on chip (SoC) included in an electronic device includes multiple processors, the electronic device may use some of the multiple processors to perform artificial intelligence operations (e.g., operations related to learning or inference of an artificial intelligence model). For example, the electronic device may perform artificial intelligence operations by using at least one of a GPU, NPU, VPU, TPU, or a hardware accelerator dedicated to artificial intelligence operations (such as convolution operations and matrix multiplication operations) among the multiple processors. However, in one or more examples, a general-purpose processor such as a CPU may be used to process artificial intelligence operations.

[0268] In addition, the electronic device can perform operations related to artificial intelligence functions by using multiple cores (e.g., dual cores or quad cores) included in one processor. For example, the electronic device can use the multiple cores included in the processor to perform artificial intelligence operations such as convolution operations and matrix multiplication operations in parallel.

[0269] One or more processors can control the input data to be processed based on predefined operating rules or artificial intelligence models stored in the memory. The predefined operating rules or artificial intelligence models can be obtained through learning.

[0270] In one or more examples, "obtained by learning" can mean a predefined operating rule or artificial intelligence model that obtains a desired function by applying a learning algorithm to a large amount of learning data. Such learning can be performed on the device itself on which artificial intelligence is performed according to the present disclosure, or by a separate server / system.

[0271] The artificial intelligence model may include multiple neural network layers. At least one layer has at least one weight value, and the operation of the layer may be performed by the operation result of the previous layer and at least one defined operation. The neural network may include, for example, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q network, and a transformer. However, unless otherwise stated, the neural network of the present disclosure is not limited to the above examples.

[0272] A learning algorithm is a method of training a predetermined target device (e.g., a robot) to make decisions or predictions for itself by using a large amount of learning data for the predetermined target device. The learning algorithm may include, for example, a supervised learning algorithm, an unsupervised learning algorithm, a semi-supervised learning algorithm, or a reinforcement learning algorithm. However, unless otherwise specified, the learning algorithm of the present disclosure is not limited to the above examples.

[0273] The methods according to various embodiments may be provided by being included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a storage medium (e.g., a compact disc read-only memory (CD-ROM)) that can be read by a machine, or may be available through an application store (e.g., PlayStore). TM ) online distribution (e.g., downloading or uploading) or directly distributed between two user devices (e.g., smart phones). Based on online distribution, at least a part of the computer program product (e.g., downloadable application) can be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a relay server), or temporarily provided.

[0274] Various embodiments of the present disclosure may be implemented in software including instructions stored in a machine-readable storage medium (e.g., a computer-readable storage medium). The machine may be a device that calls the stored instructions from the storage medium and can operate based on the called instructions, and may include an electronic device (e.g., a television) according to the disclosed embodiments.

[0275] The machine-readable storage medium may be provided in the form of a non-transitory storage medium. In one or more examples, "non-transitory storage medium" may refer to a tangible device and simply means that the storage medium does not include signals (e.g., electromagnetic waves), and the term does not distinguish between a case where data is semi-permanently stored in the storage medium and a case where data is temporarily stored in the storage medium. For example, a "non-transitory storage medium" may include a buffer that temporarily stores data.

[0276] When the instruction is executed by the processor, the processor may directly execute the function corresponding to the instruction, or other components may execute the function corresponding to the instruction under the control of the processor. The instruction may include a code provided or executed by a compiler or an interpreter.

[0277] Although the exemplary embodiments of the present disclosure have been shown and described, the present disclosure is not limited to the above specific embodiments, and various modifications may be made by those skilled in the art to which the present disclosure belongs without departing from the gist of the present disclosure disclosed in the appended claims. These modifications should also be understood to fall within the scope and spirit of the present disclosure.

Claims

1. An electronic device, include: at least one sensor; Memory, storing multiple neural network models; as well as At least one processor configured to: based on receiving a user input for selecting a cooking menu, obtaining recipe information corresponding to the cooking menu, identifying a first cooking step and at least one cooking step to be performed after the first cooking step from among a plurality of cooking steps included in the recipe information, loading at least one first neural network model among the plurality of neural network models based on the recipe information and the resource of the electronic device, the at least one first neural network model corresponding to the first cooking step and the at least one cooking step to be performed after the first cooking step, obtaining sensory data from at least one sensor, the sensory data corresponding to at least a first cooking step, obtaining contextual information about a cooking condition indicated in the sensory data by inputting the sensory data into the at least one first neural network model, Controlling the operation of the electronic device based on the recipe information and the context information, Detecting neural network model change events based on contextual information, and Based on detecting a neural network model change event, at least one second neural network model among the plurality of neural network models is loaded based on context information and resource information of the electronic device.

2. The device according to claim 1, in, The neural network model change event corresponds to an event of detecting execution of a second cooking step different from the first cooking step, and Wherein, the at least one processor is further configured to: identifying, based on the context information, a second cooking step among the plurality of cooking steps included in the recipe information and at least one cooking step to be performed after the second cooking step, and Based on the recipe information and the resource of the electronic device, at least one second neural network model among the plurality of neural network models is loaded, the at least one second neural network model corresponding to the second cooking step and the at least one cooking step to be performed after the second cooking step.

3. The device according to claim 2, in, The at least one processor is further configured to: Determine whether to change the recipe information based on the context information and the recipe information, Based on determining that the recipe information is changed, obtaining the changed recipe information, identifying, based on the context information, a third cooking step among the plurality of cooking steps included in the changed recipe information and at least one cooking step to be performed after the third cooking step, and Based on the changed recipe information and the resource of the electronic device, at least one third neural network model among the plurality of neural network models is loaded, the at least one third neural network model corresponding to a third cooking step and the at least one cooking step to be performed after the third cooking step.

4. The device according to claim 3, in, The at least one processor is further configured to: determining whether there is an intermediate cooking step between the first cooking step and the second cooking step, determining that the recipe information has not been changed based on determining that there is no intermediate cooking step between the first cooking step and the second cooking step, determining whether the intermediate cooking step between the first cooking step and the second cooking step is a cooking step related to a next cooking step after the second cooking step based on determining that the intermediate cooking step exists between the first cooking step and the second cooking step, and Based on determining that the cooking step between the first cooking step and the second cooking step is a cooking step related to a next cooking step after the second cooking step, it is determined that the recipe information is changed.

5. The device according to claim 1, in, The neural network model change event corresponds to detecting a new cooker that is not included in the recipe information, and The at least one processor is further configured to: based on detecting a new cookware not included in the recipe information, changing the recipe information based on the new cookware, and Based on the changed recipe information and the resource of the electronic device, at least one fourth neural network model among the plurality of neural network models is loaded, the at least one fourth neural network model corresponding to a fourth cooking step and at least one cooking step to be performed after the fourth cooking step.

6. The device according to claim 1, in, The at least one processor is further configured to: identifying a first cooking step included in the recipe information, obtaining information about a probability of performing one or more cooking steps after the first cooking step based on the recipe information, and At least one cooking step among the next cooking steps after the first cooking step is identified based on the information about the probability of performing one or more cooking steps after the first cooking step.

7. The device according to claim 6, in, The recipe information includes a knowledge graph showing a recipe corresponding to the selected cooking menu, The knowledge graph includes a plurality of nodes representing the plurality of cooking steps for completing a cooking menu based on a recipe and a plurality of edges representing a sequential relationship between the plurality of cooking steps, and The at least one processor is further configured to obtain information about a probability of performing one or more cooking steps after the first cooking step based on a distance between a node representing the first cooking step and a node representing a next cooking step after the first cooking step.

8. The device according to claim 1, in, The at least one processor is further configured to, based on determining that there are a plurality of first neural network models: identifying resources required to execute each of the plurality of first neural network models, and determining the number of neural network models to be loaded simultaneously in the plurality of first neural network models based on the resources of the electronic device and the resources required to execute each of the plurality of first neural network models, wherein each of the plurality of first neural network models corresponds to a first cooking step and the at least one cooking step to be performed after the first cooking step, and Among them, the multiple first neural network models are included in the multiple neural network models.

9. The device according to claim 6, in, The at least one processor is further configured to: determining a weight value for each of the plurality of first neural network models based on information about a probability of performing at least one cooking step after the first cooking step; sequentially loading each of the plurality of first neural network models based on the determined weight values; as well as By inputting the sensory data into a plurality of sequentially loaded first neural network models, contextual information about the cooking conditions indicated in the sensory data is obtained.

10. The device according to claim 1, in, The at least one sensor includes an illumination sensor for sensing an illumination value around the electronic device, and Wherein, the at least one processor is configured to load the at least one first neural network model among the multiple neural network models based on the type of at least one sensor and at least one of the illuminance values ​​obtained from the illuminance sensor, and the at least one first neural network model corresponds to the first cooking step and the at least one cooking step to be performed after the first cooking step.

11. A method for controlling an electronic device, the method include: based on receiving a user input selecting a cooking menu, obtaining recipe information corresponding to the cooking menu; identifying a first cooking step and at least one cooking step to be performed after the first cooking step from among a plurality of cooking steps included in the recipe information; loading at least one first neural network model among the plurality of neural network models based on the recipe information and the resource of the electronic device, the at least one first neural network model corresponding to the first cooking step and the at least one cooking step to be performed after the first cooking step; obtaining sensory data from at least one sensor, the sensory data corresponding to at least a first cooking step; obtaining contextual information about a cooking condition indicated in the sensory data by inputting the sensory data into the at least one first neural network model; controlling the operation of the electronic device based on the recipe information and the context information; Detecting neural network model change events based on contextual information; as well as Based on detecting a neural network model change event, at least one second neural network model among the plurality of neural network models is loaded based on context information and resource information of the electronic device.

12. The method according to claim 11, in, The neural network model change event corresponds to an event of detecting execution of a second cooking step different from the first cooking step, and Wherein, loading the at least one second neural network model comprises: identifying, based on the context information, a second cooking step among the plurality of cooking steps included in the recipe information and at least one cooking step to be performed after the second cooking step; and Based on the recipe information and the resource of the electronic device, at least one second neural network model among the plurality of neural network models is loaded, the at least one second neural network model corresponding to the second cooking step and the at least one cooking step to be performed after the second cooking step.

13. The method according to claim 12, in, Loading the at least one second neural network model comprises: determining whether to change the recipe information based on the context information and the recipe information; Based on determining that the recipe information is changed, obtaining the changed recipe information; identifying, based on the context information, a third cooking step among the plurality of cooking steps included in the changed recipe information and at least one cooking step to be performed after the third cooking step; and Based on the changed recipe information and the resource of the electronic device, at least one third neural network model among the plurality of neural network models is loaded, the at least one third neural network model corresponding to a third cooking step and the at least one cooking step to be performed after the third cooking step.

14. The method according to claim 13, in, Determining whether to change recipe information includes: determining whether an intermediate cooking step exists between the first cooking step and the second cooking step; determining that the recipe information has not been changed based on determining that there is no intermediate cooking step between the first cooking step and the second cooking step; determining whether the intermediate cooking step between the first cooking step and the second cooking step is a cooking step related to a next cooking step after the second cooking step based on determining that the intermediate cooking step exists between the first cooking step and the second cooking step; and Based on determining that the cooking step between the first cooking step and the second cooking step is a cooking step related to a next cooking step after the second cooking step, it is determined that the recipe information is changed.

15. The method according to claim 11, in, The neural network model change event corresponds to detecting a new cooker that is not included in the recipe information, and Wherein, loading the at least one second neural network model comprises: based on detecting a new cookware not included in the recipe information, changing the recipe information based on the new cookware; and Based on the changed recipe information and the resource of the electronic device, at least one fourth neural network model among the plurality of neural network models is loaded, the at least one fourth neural network model corresponding to a fourth cooking step and at least one cooking step to be performed after the fourth cooking step.

Images