Cooking apparatus and control method thereof

Abstract

A cooking apparatus includes an image taker and a processor. The processor is configured to identify a food object from a plurality of images obtained through the image taker, adjust a photographing interval of images of the food object obtained through the image taker in real time based on information about a change in a cooking state of the identified food object, and generate a video based on photographed images obtained according to the adjusted photographing interval.

Description

Technical Field

[0001] This disclosure relates to cooking equipment and control methods thereof, and more specifically, to cooking equipment and control methods thereof for generating videos of food cooking in the cooking equipment.

[0002] Cross-reference to related applications

[0003] This application claims priority to Korean Patent Application No. 10-2020-0125099, filed with the Korean Intellectual Property Office on September 25, 2020, the disclosure of which is incorporated herein by reference in its entirety. Background Technology

[0004] Cooking equipment can directly record the cooking process of food. If the cooking process is recorded, it is possible to observe how the food changes during cooking, and this can be used to identify whether the food components are suitable and to adjust the cooking temperature and heating time, etc.

[0005] However, observing all the actual cooking times can be very time-consuming for users. For example, when watching a video recording a cooking process that takes an hour, a user would need to watch the video for an hour or perform a manual action to issue a time-lapse command or fast-forward command.

[0006] When watching a video using the replay time-shift command, the user should be able to select any time. Therefore, the user may not be able to select important recording times and may skip crucial parts of the video.

[0007] When watching a video using the fast-forward command, users must watch the unnecessary parts at the same speed. Furthermore, if a user needs to fast-forward through parts of the video that are almost unchanged before watching the video at normal speed, they must give a separate fast-forward cancellation command, which is inconvenient for the user. Summary of the Invention

[0008] [Technical Solution]

[0009] This disclosure provides a cooking apparatus and its control method that take into account changes in cooking state when generating a video of a food cooking process.

[0010] According to one embodiment, a cooking device includes an image capture device and a processor. The processor is configured to: identify food objects from a plurality of images acquired by the image capture device; adjust the capturing interval (or capturing time interval) of images of the food objects acquired in real time by the image capture device based on information about changes in the cooking state of the identified food objects (or about changes in the cooking state corresponding to the identified food objects); and generate a video based on the images acquired according to the adjusted capturing interval.

[0011] Information regarding changes in cooking state includes the amount of the change; and the processor is also configured to adjust the shooting interval inversely to the amount of the change in the cooking state of the food object.

[0012] The processor is also configured to: increase the shooting interval of the image capturer based on the change in the cooking state of the food object being less than a first threshold; and decrease the shooting interval of the image capturer based on the change in the cooking state of the food object being greater than or equal to a second threshold.

[0013] The processor is also configured to: analyze the food object using at least one feature from the acquired multiple images; and adjust the amount of at least one feature to be proportional to the amount of change in the cooking state of the food object.

[0014] At least one feature information includes at least one of contour, edge, corner, histogram or brightness; and the processor is further configured to: extract at least one feature information from the acquired plurality of images, and obtain information about a food object and changes in the cooking state of the food object based on the extracted at least one feature information.

[0015] The processor is also configured to: identify regions from multiple acquired images where the amount of change in the cooking state of a food object is greater than or equal to a third threshold as target regions; and based on the identified target regions, obtain the amount of change in the cooking state of the identified food object.

[0016] The processor is also configured to change the resolution of the image obtained from the image capture device based on the amount of change in the cooking state of the food object being greater than or equal to a fourth threshold.

[0017] The cooking device also includes a memory configured to store food information corresponding to each of a plurality of foods. The processor is further configured to: obtain a predicted cooking state change of a food object based on cooking state change amounts included in the stored food information and information about the cooking state change of the food object; and adjust the capturing interval of the image capture device based on the predicted cooking state change of the food object.

[0018] The processor is also configured to: obtain a cooking state change of the identified food object based on at least one of a change in the size of the food object and a change in the color of the food object; and adjust the image capture interval based on the obtained cooking state change.

[0019] The cooking equipment also includes a display that is controlled to show a user interface (UI) for changing the shooting interval.

[0020] The control method of the cooking device according to the embodiment includes: acquiring multiple images; identifying a food object from the acquired multiple images; adjusting the shooting interval of acquiring images of the food object in real time based on information about changes in the cooking state of the identified food object; and generating a video based on the captured images acquired according to the adjusted shooting interval.

[0021] Information regarding changes in cooking state includes the amount of change in cooking state; and adjusting the shooting interval includes adjusting the shooting interval to be inversely proportional to the amount of change in the cooking state of the food object.

[0022] Adjusting the shooting interval includes: increasing the shooting interval when the change in the cooking state of the food object is less than a first threshold; and decreasing the shooting interval when the change in the cooking state of the food object is greater than or equal to a second threshold.

[0023] The method further includes: analyzing the food object using at least one feature from the acquired multiple images; and adjusting the number of at least one feature to be proportional to the amount of change in the cooking state of the food object.

[0024] At least one feature includes at least one of contour, edge, corner, histogram, or brightness; and the method further includes: extracting at least one feature from the acquired plurality of images, and obtaining information about a food object and changes in the cooking state of the food object based on the extracted at least one feature. Attached Figure Description

[0025] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, wherein:

[0026] Figure 1 This is a block diagram illustrating a cooking apparatus according to an embodiment;

[0027] Figure 2 It is shown Figure 1 A block diagram of a specific configuration of the cooking equipment;

[0028] Figure 3 It is shown Figure 1 A three-dimensional diagram of a specific configuration of the cooking equipment;

[0029] Figure 4 This is a diagram showing the internal layout of a cooking chamber used for cooking food;

[0030] Figure 5 This is a flowchart illustrating a method for generating video for food according to an embodiment;

[0031] Figure 6 It is a diagram showing cooking status information based on the time flow;

[0032] Figure 7 This is a flowchart illustrating the operation of a cooking device used to control the shooting interval;

[0033] Figure 8 It is shown in Figure 7 A graph showing the relationship between the shooting interval and the amount of change in cooking state during the control operation;

[0034] Figure 9 This is a flowchart illustrating the operation used to control the threshold for obtaining the image;

[0035] Figure 10 It is a graph showing the relationship between the number of image frames per unit time and the amount of change in cooking state during control operation;

[0036] Figure 11 This is a diagram illustrating the operations that control the amount of image feature information;

[0037] Figure 12 It is shown Figure 11 A graph showing the relationship between the amount of image feature information and the amount of change in cooking state during the control operation.

[0038] Figure 13 This is a flowchart illustrating the operation of a preset area that controls the cooking state of identified food objects;

[0039] Figure 14 It is shown Figure 13 A diagram of the preset area in the control operation;

[0040] Figure 15 It is shown Figure 13 A diagram illustrating the control operations of multiple food objects;

[0041] Figure 16 This is a flowchart illustrating the operation of controlling the resolution;

[0042] Figure 17 It is a diagram illustrating the process of estimating the cooking state of food;

[0043] Figure 18 It is a graph showing the cumulative change in the cooking state of food over time;

[0044] Figure 19 This is a diagram showing the UI displayed when reproducing video;

[0045] Figure 20 This is a diagram illustrating the video recording settings UI according to an implementation method;

[0046] Figure 21 This is a diagram illustrating a video recording settings UI according to another embodiment;

[0047] Figure 22 This is a diagram illustrating the video recording setup UI according to yet another embodiment;

[0048] Figure 23 This is a diagram illustrating a video recording settings user interface (UI) according to another embodiment; and

[0049] Figure 24 This is a flowchart illustrating a control method for a cooking apparatus according to an embodiment. Detailed Implementation

[0050] This disclosure will be described in more detail with reference to the accompanying drawings.

[0051] The terminology used in this disclosure and the claims is general terminology determined in consideration of the functionality of embodiments of this disclosure. However, these terms may be changed based on the intent of those skilled in the art, legal or technical interpretations, the emergence of new technologies, etc. Furthermore, in some cases, the terminology may be chosen by the applicant, in which case the term will be described in detail in the description of the corresponding disclosure. Therefore, the terminology used in this disclosure should be defined based on the meaning of the term rather than its simple name, and on its content throughout this disclosure.

[0052] Expressions such as “have,” “may have,” “include,” and “may include” indicate the existence of corresponding numbers, functions, operations, or parts, but do not exclude the existence of additional features.

[0053] Expressions such as “at least one of A or B” and “at least one of A and B” should be understood to mean “A”, “B”, or “A and B”.

[0054] As used herein, terms such as “first” and “second” may identify corresponding components regardless of their order and / or importance, and are used to distinguish components from other components rather than to limit those components.

[0055] Furthermore, the description of an element (e.g., a first element) being operatively or communicatively coupled to / "or connected to" another element (e.g., a second element) should be interpreted as including the first element being directly coupled to the second element, and the first element being indirectly coupled to the second element through another element (e.g., a third element).

[0056] Unless otherwise stated, singular expressions include plural expressions. It should be understood that terms such as “comprising” or “consisting of” are used herein to indicate the presence of a feature, number, step, operation, element, component, or combination thereof, and do not preclude the presence or possibility of adding one or more other features, numbers, steps, operations, elements, components, or combinations thereof.

[0057] Terms such as “module,” “unit,” and “part” are used to refer to an element that performs at least one function or operation and can be implemented as hardware or software, or a combination of hardware and software. Except where each of multiple “modules,” “units,” “parts,” etc., must be implemented in a single piece of hardware, a part can be integrated into at least one module or chip and can be implemented in at least one processor (not shown).

[0058] In the following description, "user" may refer to a person using the cooking equipment or the equipment using the cooking equipment (e.g., an artificial intelligence electronic device).

[0059] In the following description, embodiments will be described in more detail with reference to the accompanying drawings.

[0060] Figure 1 This is a block diagram illustrating a cooking apparatus according to an embodiment.

[0061] refer to Figure 1 The cooking device 100 includes an image capture device 110 and a processor 120.

[0062] Cooking equipment 100 can refer to an apparatus for cooking food (or an object to be cooked). Cooking equipment 100 can refer to one of an oven, microwave oven, electric stove (induction oven, induction cooker, high-intensity oven) or food imaging device.

[0063] Image capture device 110 is configured to generate captured images from a captured object, wherein the captured images include both moving images and still images. Image capture device 110 can acquire images from at least one external device and can be implemented as a camera, lens, infrared sensor, etc. Furthermore, image capture device 110 can be implemented as a video capturing device (or video capture device).

[0064] Image capture device 110 may include a lens and an image sensor. Lens types include general-purpose lenses, wide-angle lenses, zoom lenses, etc., and can be determined based on the type, characteristics, and usage environment of cooking equipment 100. Complementary metal-oxide-semiconductor (CMOS) and charge-connected device (CCD) can be used as image sensors.

[0065] Image capture device 110 outputs an image signal corresponding to the light incident upon it. Specifically, image capture device 110 may include a lens, pixels, and an analog-to-digital (AD) converter. The lens collects light from an object to form an optical image on the shooting area, and the pixels output an analog image signal from the light incident through the lens. The AD converter converts the analog image signal into a digital image signal and outputs the converted image signal. Image capture device 110 is configured to capture images from the front of cooking appliance 100 and generates captured images by capturing images of a user in front of cooking appliance 100.

[0066] The processor 120 can control the overall operation of the cooking appliance 100. The processor 120 can be used to control the overall operation of the cooking appliance 100.

[0067] Processor 120 may be implemented by, but is not limited to, digital signal processors (DSPs), microprocessors, and time controllers (TCONs), and may include at least one of a central processing unit (CPU), microcontroller unit (MCU), microprocessor unit (MPU), controller, application processor (AP), graphics processing unit (GPU), communication processor (CP), and advanced reduced instruction set computing (RISC) machine (ARM) processor, or may be defined by the corresponding terms. Processor 120 may be implemented as a system-on-a-chip (SoC), a large-scale integration (LSI) type in which processing algorithms are built, or a field-programmable gate array (FPGA) type. Processor 120 can perform various functions by executing computer-executable instructions stored in memory.

[0068] The processor 120 can identify food objects from multiple images acquired by the image capturer 110, control (or adjust) the shooting interval of images acquired in real time by the image capturer 110 based on information about the cooking state of the identified food objects, and generate video based on the captured images acquired according to the adjusted shooting interval.

[0069] The processor 120 can control the image capture device 110 to capture images of food (or cooked objects). (See reference...) Figure 4 Describe in detail the process of photographing food.

[0070] Processor 120 can identify food objects based on multiple images captured using image capture device 110. Artificial intelligence (AI) models can be stored in memory 160 of cooking appliance 100. Processor 120 can identify the presence and type of food objects. For example, processor 120 can identify whether the food is meat or bread based on an image.

[0071] The processor 120 can obtain cooking state change information of the identified food object. The cooking state change information can represent changes in the food's state. The cooking state change information of the food object can include the amount of cooking state change indicating whether the food has changed, and the amount of cooking state change indicating the degree of change.

[0072] The processor 120 can adjust the shooting interval based on the amount of change in the cooking state. The processor 120 can be adjusted so that the shooting interval is inversely proportional to the amount of change in the cooking state of the acquired food object.

[0073] Changes in cooking state can be information indicating the degree of change in food. The degree of change in food can include changes in color, size, texture, or surface temperature.

[0074] According to one embodiment, the image itself can be determined to measure changes in the surface temperature of the food; and according to another embodiment, a separate temperature sensor (not shown) can be used to determine this.

[0075] The processor 120 can measure the amount of change in the cooking state in real time, and if it detects that the amount of change in the cooking state is small, the processor 120 can increase the shooting interval; and if it detects that the amount of change in the cooking state is large, the processor 120 can decrease the shooting interval.

[0076] The shooting interval can represent the time interval between images captured by the image capture device 110. For example, if the image capture device 110 captures one food image every 1 second, the shooting interval can be 1 second; and if the image capture device 110 captures one food image every 10 seconds, the shooting interval can be 10 seconds.

[0077] If the change in the cooking state of the food object is less than a first threshold, the processor 120 may increase the shooting interval of the image capturer 110; and if the change in the cooking state of the obtained food object is greater than or equal to a second threshold, the processor 120 may decrease the shooting interval of the image capturer 110.

[0078] The following will refer to Figures 6 to 8 Describe in detail the relationship between changes in cooking state and shooting intervals.

[0079] The processor 120 can analyze a food object using at least one feature from multiple acquired images, and can adjust the amount of at least one feature to be proportional to the amount of change in the cooking state of the acquired food object.

[0080] The feature information may include at least one of contour, edge, corner, histogram or brightness, and the processor 120 may extract the feature information from the acquired multiple images and obtain information about the food object and information about changes in the cooking state of the food object based on the extracted feature information.

[0081] Feature information can represent feature points used in image analysis techniques. Processor 120 can analyze an image based on key feature points to obtain information such as the presence of food objects, the cooking state of the food objects, and the amount of change in the cooking state. Here, the feature points can be set differently according to user settings.

[0082] The following will refer to Figure 6 , Figure 11 and Figure 12Describe in detail the relationship between the amount of change in cooking state and the image feature information of the image.

[0083] Processor 120 can use a target region when analyzing a food object. Processor 120 can perform image analysis operations based solely on the target region among all regions of the acquired image. Selectively analyzing the target region instead of all regions can reduce data throughput and data processing speed. Processor 120 can perform resizing operations on small-sized images that only include the target region so that only the target region can be analyzed.

[0084] The processor 120 can identify regions where the amount of change in the cooking state of a food object obtained from multiple images is greater than or equal to a third threshold as target regions, and can obtain the amount of change in the cooking state of the identified food object based on the identified target regions.

[0085] Processor 120 can obtain the amount of cooking state change based solely on the target area, thereby reducing data throughput and data processing time. (See reference...) Figures 13 to 15 Describe the target area in more detail.

[0086] The processor 120 can adjust the image resolution based on the amount of change in the cooking state. When the amount of change in the cooking state is high, the processor 120 can adjust the image sharpness to be greater.

[0087] If the change in the cooking state of the food object is greater than or equal to the fourth threshold, the processor 120 may change the sharpness of the image obtained by the image capturer 110.

[0088] When the changes in cooking status are small, users may not be interested in the corresponding cooking process. Therefore, the resolution can be automatically adjusted to reduce the image size. Conversely, if the changes in cooking status are large, users may be interested in the corresponding cooking process, so the resolution can be automatically adjusted to obtain a high-definition image.

[0089] Reference Figure 16 Describe in detail the operation of changing the resolution.

[0090] It may also include a memory 160 for storing food information corresponding to each of the plurality of foods, and the processor 120 may obtain the expected cooking state change of the food object based on the cooking state change amount included in the stored food information and the cooking state change amount of the food object, and may change the shooting interval of the image capturer 110 based on the obtained expected cooking state change amount of the food object.

[0091] The food information corresponding to each of the multiple foods may refer to information already stored in memory 160, rather than values ​​measured by the cooking device 100 itself.

[0092] For example, if the food is meat, the stored food information can be stored in memory 160 based on the conventional cooking process of the meat. The meat food information may include at least one of cooking time, cooking temperature, food cooking state according to cooking time, and cooking state change according to the amount of meat. This is not information directly captured by the image capturer 110 of the cooking device 100, and may correspond to previously stored basic information. To distinguish the amount of cooking state change obtained from the image captured by the image capturer 110, the already stored amount of cooking state change may be described as a basic change or a fundamental cooking state change, and the amount of cooking state change obtained from the image may be described as a measured cooking state change. Food information corresponding to each of the plurality of foods may be corresponding to... Figure 18 The value of D_step.

[0093] The processor 120 can obtain the predicted cooking state change by considering both the basic cooking state change and the measured cooking state change. This will refer to... Figure 18 Provide a detailed description.

[0094] The processor 120 can identify the cooking state change of a food object by obtaining at least one of the change in the size of the food object or the change in the color of the food object, and can adjust the image capture interval based on the obtained cooking state change.

[0095] Food can change size depending on the cooking process (e.g., bread expands), and food color can change depending on the degree of cooking (e.g., bread dough changes from white to brown).

[0096] The cooking appliance 100 may also include a display 130, and the display 130 may be controlled to display a guided UI for changing the shooting interval.

[0097] The processor 120 can display a guidance UI to instruct the user to select the shooting interval. Specifically, the processor 120 can display at least one of the following guidance UIs on the display 130: guidance UIs 2011 and 2012, displaying mode names with different shooting intervals; guidance UIs 2111, 2112, and 2113, indicating the degree of adjustment of the shooting interval; guidance UIs 2211, 2212, and 2213, displaying video generation capabilities; and guidance UIs 2311, 2312, and 2313, displaying video playback time.

[0098] When the guided UI is displayed, users can intuitively adjust video recording settings, thereby improving user convenience.

[0099] In the above description, the cooking device 100 is described as capturing food by controlling the shooting interval of the image capture device 110, etc. After the video is recorded, the final video can be generated through editing operations.

[0100] According to another embodiment, the cooking device 100 can generate a video by shooting food in a conventional manner, and the cooking device 100 can directly edit the generated video. The shooting interval can represent the interval at which frames (images) are extracted (obtained) at a specific point in time, rather than the interval at which images are captured by an image capture device.

[0101] According to another embodiment, the cooking device 100 can generate video by filming food in a conventional manner. The cooking device 100 can send the generated video to an external device via a communication interface 140. The external device can edit the received video. The filming interval can represent the interval at which frames (images) are extracted (obtained) at a specific point in time, rather than the interval at which an image capture device films the images. The cooking device 100 can record the cooking process of the food and can edit the video recorded by the external device.

[0102] Reference Figure 9 and Figure 10 Describe the editing process after recording a video.

[0103] By adjusting the shooting interval, the number of image frames per unit time, the amount of image feature information, sharpness, and target area, the resulting food cooking video can have different reproduction times depending on the amount of change in the cooking state. Therefore, more frames including the parts the user expects can be seen, thus improving user satisfaction. A method for generating or editing video can improve data throughput and data processing speed.

[0104] Since users can intuitively change the methods for generating or editing videos through various guided UIs, the methods used to generate or edit videos can increase user convenience.

[0105] Although only a simple structure of the cooking appliance 100 is shown above, various other structures can be provided during implementation. These will be referenced below. Figure 2 Describe it.

[0106] Figure 2 It is shown Figure 1 A block diagram of a specific configuration of the cooking equipment.

[0107] refer to Figure 2The cooking device 100 may include an image capture device 110, a processor 120, a display 130, a communication interface 140, a user interface 150, a memory 160, and a speaker 170.

[0108] The description of the same operations as described above in the operation of the image capturer 110 and the processor 120 will be omitted.

[0109] The display 130 can be implemented as various types of displays, such as liquid crystal displays (LCDs), organic light-emitting diode (OLED) displays, plasma display panels (PDPs), etc. The display 130 may also include a backlight unit, which can be implemented as a driving circuit for a-si TFTs, low-temperature polycrystalline silicon (LTPS) TFTs, organic TFTs (OTFTs), etc. Furthermore, the display 130 can be implemented as a touchscreen connected to a touch sensor, a flexible display, a three-dimensional (3D) display, etc.

[0110] According to one embodiment, the display 130 may include a display panel for outputting images and a cover plate for housing the display panel. According to another embodiment, the cover plate may include a touch sensor (not shown) for sensing user interaction.

[0111] The communication interface 140 performs communication with various types of external devices according to various communication methods. The communication interface 140 may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Each communication module can be implemented as at least one hardware chip.

[0112] The Wi-Fi module and Bluetooth module communicate using Wi-Fi and Bluetooth methods, respectively. When using either the Wi-Fi or Bluetooth module, various connection information such as Service Set Identifier (SSID) and session key can be sent and received first, and various information can be sent and received after the communication connection is established.

[0113] The infrared communication module performs communication based on Infrared Data Association (IrDA) technology, which uses infrared light, which falls between visible light and millimeter waves, to wirelessly transmit data to a local area.

[0114] In addition to the communication methods mentioned above, the wireless communication module may also include at least one chip that performs communication according to various communication standards, such as Zigbee, 3G, 3GPP, LTE, LTE-A, 4G, 5G, etc.

[0115] The communication interface 140 may include at least one of a local area network (LAN) module, an Ethernet module, or a wired communication module that uses a cable, coaxial cable, optical fiber to perform communication, or an ultra-wideband (UWB) module.

[0116] According to an implementation, the communication interface 140 can use the same communication module (e.g., a Wi-Fi module) to communicate with external devices such as remote controls and external servers.

[0117] According to another example, communication interface 140 can utilize different communication modules (e.g., a Wi-Fi module) to communicate with external devices such as remote controls and external servers. For example, communication interface 140 can use at least one of an Ethernet module or a Wi-Fi module to communicate with an external server, and can use a Bluetooth (BT) module to communicate with external devices such as remote controls. However, this is merely exemplary, and communication interface 140 can use at least one of various communication modules when communicating with multiple external devices or external servers.

[0118] The user interface 150 can be implemented as buttons, a touchpad, a mouse, and a keyboard, or it can be implemented as a touchscreen that can also perform display and operation input functions. Here, the buttons can be various types of buttons formed in at least one arbitrary area such as the front, side, or rear of the outer surface of the main body of the cooking appliance 100, such as mechanical buttons, touchpads, and scroll wheels.

[0119] The memory 160 can be implemented as internal memory, such as read-only memory (ROM) (e.g., electrically erasable programmable read-only memory (EEPROM)) and random access memory (RAM), or memory separate from the processor 120. In this case, the memory 160 can be implemented as at least one of memory embedded within the cooking appliance 100 or memory removable from the cooking appliance 100 depending on the use of the data storage. For example, data for driving the cooking appliance 100 can be stored in memory embedded within the cooking appliance 100, and data for upgrading the cooking appliance 100 can be stored in memory removable from the cooking appliance 100.

[0120] The memory embedded in the cooking appliance 100 can be implemented as at least one of volatile or non-volatile memory, wherein the volatile memory includes dynamic random access memory (DRAM), static random access memory (SRAM), synchronous dynamic random access memory (SDRAM), and non-volatile memory includes one-time programmable ROM (OTPROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), JROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard disk drive (HDD), or solid-state drive (SSD). When the memory is removably mounted to the cooking appliance 100, the memory can be implemented as a memory card (e.g., small flash memory (CF), secure digital card (SD), micro-secure digital card (Micro-SD), mini-secure digital card (mini-SD), extreme digital card (xD), multimedia card (MMC), etc.) or an external memory (e.g., universal serial bus (USB) memory) connectable to a USB port.

[0121] The cooking appliance 100 may include a speaker 170. The speaker (170) may be a component that outputs various audio data, various alarm sounds, voice messages, etc., processed by the input and output interfaces.

[0122] The cooking appliance 100 may also include a microphone (not shown). A microphone is an element that receives a user's voice or other sounds and converts them into audio data.

[0123] A microphone (not shown) can receive active user voice. For example, the microphone can be integrally formed as a single unit on at least one of the top, front, or side surfaces of the cooking appliance 100. The microphone can include various configurations, such as a microphone for collecting user voice in analog format, an amplifier circuit for amplifying the collected user voice, an audio-to-digital (A / D) converter circuit for sampling the amplified user voice to convert it into a digital signal, and a filter circuit for removing noise elements from the converted digital signal.

[0124] Figure 3 It is shown Figure 1 A three-dimensional diagram of a specific configuration of cooking equipment.

[0125] refer to Figure 3 The cooking device 100 is merely an example, and the cooking device can be implemented in various forms according to various embodiments.

[0126] like Figure 3 As shown, the cooking device 100 may include a body 10 that forms the external shape.

[0127] The cooking apparatus 100 also includes a cooking chamber 20 that opens toward one side thereto. The cooking chamber 20 may refer to a space for containing food (i.e., a holding space). The cooking chamber 20 of the main body 10 may be a cooking chamber 20 for cooking an object (i.e., food), and the cooking chamber 20 may open toward the front of the cooking apparatus 100.

[0128] The cooking chamber 20 is box-shaped, and its front is opened to remove or place cooking objects. The front of the body 10 may include an opening connected to the cooking chamber 20.

[0129] The front surface of the cooking chamber 20 can be opened and closed via a door 21 connected to the main body 10. The door 21 can be hinged to the lower part of the main body 10 so that it can rotate relative to the main body 10. A door handle 23 that can be gripped by a user can be provided on the front surface of the door 21, and the user can grip the door handle 23 to open and close the cooking chamber 20.

[0130] The cooking chamber 20 may include a heater for heating the object being cooked. In this case, the heater may be an electric heater including a resistor. The heater is not limited to an electric heater and may also be a gas heater that generates heat by burning gas.

[0131] The control panel 30 may be located on the upper part of the main body 10. The control panel 30 may include a display 31, which is used to display various operating information of the cooking equipment 100 and receive user commands for controlling the operation of the cooking equipment 100. The control panel 30 may include buttons 32, which are used to receive various user commands for controlling the operation of the cooking equipment 100.

[0132] According to the implementation, the cooking device 100 can perform cooking on the object to be cooked, taking into account the size and cooking state of the object to be cooked in the cooking chamber.

[0133] The cooking device 100 can determine the cooking time of a cooking object based on the type of cooking object selected via the control panel 30. The cooking device 100 can identify the size of the cooking object and determine the cooking time based on the identified size. For example, even for the same type of cooking object, a smaller size of the cooking object can result in a relatively shorter cooking time, while a larger size of the cooking object can result in a relatively longer cooking time.

[0134] The cooking device 100 can identify the cooking state of the object being cooked during cooking and can control the cooking process based on the cooking state.

[0135] If the cooking of an object is determined to be complete based on its ripeness, the cooking device 100 may even terminate the cooking process before the set cooking time ends. For example, the cooking device 100 may identify the ripeness of the object, and if the cooking device 100 determines that the cooking of the object is complete based on its ripeness, the cooking device 100 may even terminate the cooking process before the set cooking time ends.

[0136] As described above, according to various embodiments of this disclosure, cooking time is not determined solely by the type of food, but rather, for the same type of food, different cooking times can be determined based on the size of the food being cooked, and the cooking termination time can be determined based on the cooking state of the food during cooking. Therefore, even if the user is unaware of the precise cooking method due to differences in the size of the food being cooked, the user can optimally cook the food without monitoring the cooking process when selecting the type of food to be cooked.

[0137] Figure 4 This is a diagram showing the internal configuration of a cooking chamber used for cooking food.

[0138] refer to Figure 4 Food 410 can be present in the cooking chamber 20 of the cooking equipment 100.

[0139] Food 410 can be in container 405, and container 405 can be placed on substrate 22. The image capture device 110 of cooking device 100 can capture images of at least one of substrate 22, container 405, or food 410.

[0140] The field of view can vary depending on the type of image capture device 110, but the image capture device 110 can be arranged to capture the entire area within the cooking room 20.

[0141] According to the implementation method, such as Figure 4 As shown, the image capture device 110 can be attached to the wall surface of the cooking chamber 20 and can be configured to tilt downwards at a predetermined angle. For example, the image capture device 110 can be configured to tilt downwards at a 45-degree angle while attached to the wall surface of the cooking chamber 20.

[0142] According to another embodiment, the image capture device 110 can be attached to the upper plate (or top plate) of the cooking chamber 20 and can be configured to face downwards at a predetermined angle. For example, the image capture device 110 can be attached to the upper plate of the cooking chamber 20 while facing downwards.

[0143] when Figure 4 When the image capture device 110 is shown in a prominent form, Figure 4The image capture device 110 can be implemented in the form of being mounted on the wall of the cooking chamber 20 or inside the upper plate of the cooking chamber 20, so as to protect the lens of the image capture device during actual implementation.

[0144] Figure 5 This is a flowchart illustrating a method for generating video of food according to an embodiment.

[0145] refer to Figure 5 In operation S505, the cooking device 100 can acquire an image including food. In operation S510, the cooking device 100 can identify food objects in the acquired image. In operation S515, the cooking device 100 can analyze the identified food objects. The operation of analyzing food objects can represent analyzing the cooking state of the food objects. In operation S520, the cooking device 100 can predict the cooking state of the identified food objects as an analysis result. The cooking device 100 can control itself based on the predicted cooking state. Here, controlling the cooking device 100 can mean controlling various hardware or software included in the cooking device 100. For example, the cooking device 100 can adjust the heating temperature and heating time based on the predicted cooking state. The cooking device 100 can control the image capture device 110 based on the predicted cooking state. The cooking device 100 can perform video editing based on the predicted cooking state.

[0146] During operation S530, the cooking device 100 can acquire multiple images based on the above operation, and can generate a video based on the acquired multiple images.

[0147] Figure 6 This is a diagram showing cooking status information based on the time flow.

[0148] refer to Figure 6 Table 605 includes information about the food cooking process. Specifically, it is assumed that the food is bread and the cooking time is 35 minutes. The cooking process can be divided into four parts. These four parts can be arbitrarily divided and can be changed depending on the food or cooking method.

[0149] During the bread cooking process, the first portion of the cooking time can be 20 minutes (0 to 20 minutes of the total cooking time), and the change in cooking state can be 5%. The cooking device 100 can set the shooting interval to 20 seconds based on the change in cooking state in the first portion. The cooking device 100 can set the number of image frames per unit time to 0.05. The unit time in Table 605 corresponds to 1 second and can be changed according to the actual implementation example. The cooking device 100 can set the number of image feature information to 2.

[0150] During the bread cooking process, the second portion can last for 5 minutes (20 to 25 minutes of the total cooking time), and the cooking state change can be 15%. The cooking device 100 can set the shooting interval to 5 seconds based on the cooking state change in the second portion. The cooking device 100 can set the number of image frames per unit time to 0.2. The unit time in Table 605 corresponds to 1 second and can be changed according to the actual implementation example. The cooking device 100 can set the number of image feature information to 8.

[0151] During the bread cooking process, the third portion can last for 5 minutes (25 to 30 minutes of the total cooking time), and the change in cooking state can be 20%. The cooking device 100 can set the shooting interval to 1 second based on the change in cooking state in the third portion. The cooking device 100 can set the number of image frames per unit time to 1. The unit time in Table 605 corresponds to 1 second and can be changed according to the actual implementation example. The cooking device 100 can set the number of image feature information to 20.

[0152] During the bread cooking process, the fourth section can last for 5 minutes (30 to 35 minutes of the total cooking time), and the change in cooking state can be 5%. The cooking device 100 can set the shooting interval to 20 seconds based on the change in cooking state in the fourth section. The cooking device 100 can set the number of image frames per unit time to 0.05. The unit time in Table 605 corresponds to 1 second and can be changed according to the actual implementation example. The cooking device 100 can set the number of image feature information to 2.

[0153] Regarding the first four stages of the bread-cooking process, the amount of change can be very small in the first 20 minutes, but can be increased after about 10 minutes. After 30 minutes, the amount of change can be reduced, and finally the cooking process can be completed.

[0154] The cooking device 100 can control the shooting interval based on the amount of change in the cooking state. According to the data in Table 605, the shooting interval and the number of image frames per unit time can represent the same information. According to another embodiment (different from...) Figure 6 In this implementation, the number of shooting intervals and the number of image frames per unit time may not correspond to each other. For example, the cooking device 100 may selectively acquire multiple images for generating a video based on the number of image frames per unit time after acquiring images based on the shooting interval. The cooking device 100 may selectively acquire 80 images based on the number of image frames per unit time after acquiring 100 images based on the shooting interval. The cooking device 100 may generate a video based on 80 images.

[0155] The cooking device 100 can control the amount of image feature information related to the cooking state. The cooking device 100 can use more image feature information in areas of significant change to identify or analyze food objects. (See below for further details.) Figure 11 Describe a detailed description related to it.

[0156] Figure 7 This is a flowchart illustrating the operation of a cooking device used to control the shooting interval.

[0157] refer to Figure 7 In operation S705, the cooking device 100 can acquire multiple images according to the shooting interval. In operation S710, the cooking device 100 can identify food objects from the acquired multiple images and identify the cooking state of the identified food objects. In operation S715, the cooking device 100 can obtain the amount of change in the cooking state of the food objects based on the cooking state of the food objects corresponding to each of the multiple images.

[0158] In operation S720, the cooking device 100 can identify whether the cooking of the food is completed during operation S720. Whether the cooking of the food is complete can be identified through various methods. For example, the cooking device 100 can identify whether cooking is complete based on the amount of change in the cooking state of the food. For example, in the initial cooking process according to the cooking procedure, the amount of change in the cooking state may be 0%, and the amount of change in the cooking state may increase. Finally, the amount of change in the cooking state may decrease back to 0%. The cooking device 100 can also identify whether cooking is complete based on the time it takes for the amount of change in the cooking state to rise from 0% to a specific value and then return to 0%. As another example, the cooking device 100 can identify whether cooking is complete based on a cooking set time. For example, the cooking set time of the cooking device 100 may be predetermined. The cooking time may be set based on a mode selected by the user or by the user directly setting the cooking time.

[0159] If the cooking of the product is identified as complete, then in operation S750, the cooking device 100 can generate a video based on the acquired multiple images.

[0160] If it is determined that the cooking of the food is not complete, then in operation S725, the cooking device 100 can determine whether the change in cooking state is greater than or equal to a first threshold. If the change in cooking state is less than the first threshold, then in operation S730, the cooking device 100 can increase the shooting interval. The cooking device 100 can acquire multiple images based on the increased shooting interval.

[0161] If the change in cooking state is greater than or equal to a first threshold, then in operation S735, the cooking device 100 can identify whether the change in cooking state is greater than or equal to a second threshold. If the change in cooking state is less than the second threshold, then in operation S740, the cooking device 100 can maintain the previously established shooting interval. The cooking device 100 can acquire multiple images based on the maintained shooting interval.

[0162] If the change in cooking state is greater than or equal to a second threshold, then in operation S745, the cooking device 100 can reduce the shooting interval. The cooking device 100 can acquire multiple images based on the reduced shooting interval.

[0163] After performing operations S730, S740, and S745, the cooking device 100 can acquire images based on increasing, maintaining, or decreasing the shooting interval, and can repeat operations S705 to S720. If it is determined that the cooking of the food is complete, then in operation S750, the cooking device 100 can generate a video based on the acquired multiple images.

[0164] Figure 8 It is shown in Figure 7 A graph showing the relationship between the shooting interval and the amount of change in cooking state during the control operation.

[0165] refer to Figure 8 The cooking device 100 can control the shooting interval so that the amount of change in the cooking state and the shooting interval are inversely proportional. The smaller the amount of change in the cooking state, the more the cooking device 100 can increase the shooting interval; and the greater the amount of change in the cooking state, the more the cooking device 100 can decrease the shooting interval.

[0166] The reason why the amount of cooking state change is inversely proportional to the shooting interval is that when the amount of cooking state change is small, it is not necessary to obtain a large number of images. The images that the user expects are those with changes, rather than those without changes. By reducing the number of images without changes and increasing the number of images with significant changes, the final video can include more of the desired images.

[0167] Figure 9 This is a flowchart illustrating the operation of controlling the threshold used to obtain an image.

[0168] refer to Figure 9 In operation S905, the cooking apparatus 100 can acquire a video including the food object. The cooking apparatus 100 can acquire an unedited video. For example, an unedited video can refer to a video acquired by the image capture device 110. For example, if the shooting interval of the image capture device 110 is not adjusted, the video can be a video composed of images acquired at regular intervals.

[0169] As another example, if the shooting interval of the image capture device 110 is adjusted, the video can be referenced to the video formed by the images whose shooting interval has been adjusted. The cooking device 100 can also further edit the video after the shooting interval has been adjusted.

[0170] During operation S910, the cooking device 100 can identify the cooking state of a food object identified in the acquired video. The cooking device 100 can analyze each of the multiple images constituting the video to identify (or analyze) the cooking state of the food object.

[0171] In operation S915, the cooking device 100 can obtain the amount of change in the cooking state of the food object based on the cooking state of the food object obtained in operation S910. The amount of change in cooking state can represent the amount of change corresponding to a feature frame. For example, the amount of change in cooking state of frame 1, frame 2, and frame 60 in the video can be obtained. The cooking device 100 can obtain the amount of change in cooking state for each frame included in the video.

[0172] In operation S925, the cooking device 100 can identify whether the cooking state change amount of a specific frame is greater than or equal to a first threshold. If the cooking state change amount is less than the first threshold, then in operation S930, the cooking device 100 can acquire an image every first threshold time interval. The cooking device 100 can identify frames whose cooking state change amount is less than the first threshold among all frames included in the video. The cooking device 100 can acquire one frame (image) every first threshold time interval from the frames whose cooking state change amount is less than the first threshold. The cooking device 100 can remove (or delete or skip) frames (images) corresponding to the first threshold interval from the frames whose cooking state change amount is less than the first threshold.

[0173] For example, suppose there are 100 frames (corresponding to 100 seconds) where the cooking state change is less than a first threshold, and the first threshold time is 20 seconds. The cooking device 100 can obtain (or extract or select) only five frames from the 100 frames that correspond to 100 seconds. Finally, the cooking device 100 can assign the five frames to the region corresponding to 100 seconds.

[0174] If the cooking state change in a specific frame is greater than or equal to a first threshold, then in operation S935, the cooking device 100 can identify whether the cooking state change is greater than or equal to a second threshold. If the cooking state change is less than the second threshold, the cooking device 100 can acquire an image every second threshold time interval. The second threshold time interval can be less than the first threshold time interval. The cooking device 100 can identify frames among all frames included in the video whose cooking state change is greater than or equal to the first threshold and less than the second threshold. The cooking device 100 can acquire one frame (image) every second threshold time interval from frames greater than or equal to the first threshold and less than the second threshold. The cooking device 100 can remove (or delete or skip) frames (images) corresponding to the second threshold interval from frames greater than or equal to the first threshold and less than the second threshold.

[0175] If the cooking state change in a specific frame is greater than or equal to a second threshold, then in operation S945, the cooking device 100 can acquire an image every third threshold time interval. The third threshold time interval can be less than the second threshold time interval. The time intervals of the first, second, and third threshold times can be progressively smaller. Specifically, the cooking device 100 can identify frames among all frames included in the video whose cooking state change is greater than or equal to the second threshold. The cooking device 100 can acquire one frame (image) every third threshold time interval from frames having a second threshold or a higher threshold. The cooking device 100 can remove (or delete or skip) frames (images) corresponding to the third threshold interval from frames greater than or equal to the second threshold.

[0176] Operations S930, S940, and S945 can be performed in consecutive frames. Since the video should be generated according to chronological order rather than changes in cooking state, the operation of acquiring (or extracting or selecting) images can also be performed in consecutive frames. In operation S950, the cooking device 100 can generate video based on multiple images acquired in operations S930, S940, and S945.

[0177] Figure 10 This is a graph showing the relationship between the number of image frames per unit time and the amount of change in cooking state during control operation.

[0178] refer to Figure 10 The number of image frames per unit time and the amount of change in cooking state can be proportional to each other. Specifically, the cooking device 100 can reduce the number of image frames per unit time as the amount of change in cooking state decreases. To reduce the number of image frames per unit time, the cooking device 100 can increase the threshold time used to acquire images in the video.

[0179] The cooking device 100 can increase the number of image frames per unit time as the cooking state changes more significantly. To increase the number of image frames per unit time, the cooking device 100 can reduce the threshold time used to acquire images from the video.

[0180] The greater the change in cooking state, the more likely the image will be the one the user desires. The cooking device 100 can capture more images in frames of the video with large changes in cooking state and fewer images in sections with smaller changes, thus generating a new video. A thumbnail version of the video, including more of the desired images, can then be regenerated from the original video.

[0181] Figure 11 This is a diagram illustrating the operations that control the amount of image feature information.

[0182] refer to Figure 11 In operation S1105, the cooking device 100 can acquire multiple images including a food object. Furthermore, in operation S1110, the cooking device 100 can identify the cooking state of the food object based on image feature information in the acquired images. In operation S1115, the cooking device 100 can acquire the amount of change in the cooking state of the food object. In operation S1120, the cooking device 100 can identify whether the cooking of the food is complete. Operation S1120 corresponds to operation S720, therefore, a repeated description will be omitted.

[0183] If the cooking of the food object is identified as complete, then in operation S1150, the cooking device 100 can generate a video based on the acquired multiple images.

[0184] If it is determined that the cooking of the food is incomplete, in operation S1125, the cooking device 100 can determine whether the amount of change in the cooking state is greater than or equal to a first threshold. If the amount of change in the cooking state is less than the first threshold, in operation S1130, the cooking device 100 can reduce the amount of image feature information. The cooking device 100 can obtain multiple images based on the reduced image feature information.

[0185] If the change in cooking state is greater than or equal to a first threshold, then in operation S1135, the cooking device 100 can identify whether the change in cooking state is greater than or equal to a second threshold. If the change in cooking state is less than the second threshold, then in operation S1140, the cooking device 100 can maintain a preset amount of image feature information. The cooking device 100 can obtain multiple images based on the maintained image feature information.

[0186] If the change in cooking state is equal to or greater than the second threshold, then in operation S1145, the cooking device 100 can increase the amount of image feature information. The cooking device 100 can obtain multiple images based on the increased image feature information.

[0187] After performing operations S1130, S1140, and S1145, the cooking device 100 can acquire images based on reduced, maintained, or increased image feature information, and operations S1105 to S1120 can be repeated. If it is determined that the cooking of the food is complete, in operation S1150, the cooking device 100 can generate a video based on the acquired multiple images.

[0188] Figure 12 It is shown in Figure 11 The graph shows the relationship between the amount of image feature information and the amount of change in cooking state during the control operation.

[0189] refer to Figure 12 The cooking device 100 can control the amount of image feature information so that the amount of change in cooking state is proportional to the amount of image feature information. When the amount of change in cooking state is small, the cooking device 100 can reduce the amount of image feature information, and when the amount of change in cooking state is large, the cooking device 100 can increase the amount of image feature information.

[0190] A smaller amount of image feature information allows for the application of a minimum criterion to analyze the image. A larger amount of image feature information may indicate an increase in image recognition rate. Therefore, the cooking device 100 can perform image analysis based on more image feature information during periods of high cooking state change. Since high-recognition-rate analysis is performed only during periods of high cooking state change, rather than in all periods, the total data processing time can be reduced.

[0191] Figure 13 This is a flowchart illustrating the operation of a preset area that controls the cooking state of a food object.

[0192] refer to Figure 13 In operation S1305, the cooking device 100 can acquire an image including a food object. In operation S1310, the cooking device 100 can identify the cooking state of the identified food object based on a predetermined area in the acquired image. The predetermined area (or target area) may refer to a specific area of ​​the cooking chamber 20 that can be captured by the image capture device 110. In operation S1315, the cooking device 100 can acquire the amount of change in the cooking state of the food object. In operation S1320, the cooking device 100 can identify whether the cooking of the food is complete. Operation S1320 may correspond to operation S720, and repeated descriptions will be omitted.

[0193] When the cooking of food is recognized as complete, in operation S1350, the cooking device 100 can generate a video based on the acquired multiple images.

[0194] When the cooking of food is identified as incomplete, in operation S1325, the cooking device 100 can identify whether the change in cooking state is greater than or equal to a third threshold. If the change in cooking state is less than the third threshold, then in operation S1330, the cooking device 100 can determine a predetermined area as all areas displaying the food object. The cooking device 100 can identify the cooking state of the food object based on all areas displaying the food object.

[0195] If the change in cooking state is greater than or equal to a third threshold, then in operation S1335, the cooking device 100 can determine that a predetermined area is an area where the change in cooking state is greater than or equal to the third threshold. The cooking state of the food object can be identified based on the area where the change in cooking state is greater than or equal to the third threshold.

[0196] The cooking device 100 can identify the cooking state of a food object based on a predetermined area determined according to operations S1330 and S1335, and can repeat S1305 to S1320. If it is identified that the cooking of the food is complete, the cooking device 100 can generate a video based on multiple acquired images obtained in operation S1350.

[0197] exist Figure 13 In one implementation, the cooking device 100 may perform an image resizing operation in order to analyze only a predetermined area (or target area).

[0198] Figure 14 It is shown Figure 13 A diagram of the preset area in the control operation.

[0199] refer to Figure 14 The cooking device 100 can obtain the first image 1410, the second image 1420, the third image 1430 and the fourth image 1440 in chronological order.

[0200] The cooking device 100 can identify the food object 1405 in the first image 1410.

[0201] The cooking device 100 can identify object regions (or target regions) based on the food object 1405 identified in the first image 1410. The reason for identifying target regions is to reduce data processing time by analyzing only the target regions instead of the entire image.

[0202] The identified target region can be used to analyze the second image 1420. The cooking device 100 can analyze the food object 1405 based on the target region 1421 in the second image 1420.

[0203] The target area can be changed based on the amount of change in the cooking state. The first target area can be the entire area of ​​the food object in the second image 1420. In the early stages of cooking, the entire food may hardly change. Therefore, the cooking device 100 can analyze the entire area of ​​the food to identify whether any changes exist in any part.

[0204] If the food is meat, its cooking state can be changed externally according to the cooking process. That is, since meat cooks from the outside in according to the cooking process, the amount of external cooking state change can be increased. The cooking device 100 can change the target area to a region with a high amount of cooking state change throughout the entire area of ​​the food. The cooking device 100 can define the external area of ​​the food as a new target area.

[0205] The altered target area can be used to analyze the third image 1430. The cooking device 100 can analyze the food object 1405 based on the altered target area 1431 in the third image 1430.

[0206] During cooking, the meat's interior gradually cooks after the exterior is fully cooked. The amount of cooking state change on the exterior of the already cooked meat decreases, while the amount of cooking state change on the interior increases. The cooking device 100 can reshape the target area. The cooking device 100 can reshape the target area into an area with a high degree of cooking state change.

[0207] The modified target area can be used to analyze the fourth image 1440. The cooking device 100 can analyze the food object 1405 based on the modified target area 1441 in the fourth image 1440.

[0208] Figure 15 It is shown in Figure 13 A diagram showing the analysis operations on multiple food objects during the control process.

[0209] refer to Figure 15 ,and Figure 14 Unlike other cooking devices, cooking equipment 100 can cook multiple foods simultaneously. Therefore, there can be multiple food objects. Cooking equipment 100 can acquire a first image 1510, a second image 1520, a third image 1530, and a fourth image 1540 in a time sequence.

[0210] The cooking device 100 can recognize multiple food objects 1505-1, 1505-2, 1505-3, 1505-4 and 1505-5.

[0211] The cooking apparatus 100 can identify target areas based on multiple identified food objects 1505-1 to 1505-5. The cooking apparatus 100 can determine the entire area of ​​the food corresponding to each of the food objects 1505-1 to 1505-5 as the target area. Since there are multiple food objects 1505-1 to 1505-5, the cooking apparatus 100 can identify multiple target areas.

[0212] The identified target regions can be used to analyze the second image 1520. The cooking device 100 can analyze food objects 1505-1 to 1505-5 based on the multiple target regions 1521-1, 1521-2, 1521-3, 1521-4 and 1521-5 in the second image 1520.

[0213] When the food is bread, its cooking state can change from the inside out depending on the cooking process. The cooking apparatus 100 can change the target area to a region with a high degree of cooking state change throughout the food. The cooking apparatus 100 can also define the internal region of the food as a new target area.

[0214] The altered target area can be used to analyze the third image 1530. The cooking device 100 can analyze food objects 1505-1 to 1505-5 based on the altered target areas 1531-1, 1531-2, 1531-3, 1531-4 and 1531-5 altered in the third image 1530.

[0215] The cooking device 100 can ultimately determine whether the cooking of food objects 1505-1 to 1505-5 is complete. If the cooking state of food objects 1505-1 to 1505-5 has not changed based on the fourth image 1540, the cooking device 100 can determine that the cooking is complete.

[0216] Figure 16 This is a flowchart illustrating the operation of controlling the resolution.

[0217] refer to Figure 16 In operation S1605, the cooking device 100 can acquire an image including a food object with a predetermined resolution. In operation S1610, the cooking device 100 can identify the food object from the acquired image and can identify the cooking state of the identified food object. In operation S1615, the cooking device 100 can acquire the amount of change in the cooking state of the food object. In operation S1620, the cooking device 100 can identify whether the cooking of the food is complete. Since operation S1620 corresponds to S720, repeated descriptions will be omitted.

[0218] If the cooking of the food is identified as complete, in operation S1650, the cooking device 100 can generate a video based on the acquired multiple images.

[0219] If it is determined that the cooking of the food is incomplete, in operation S1625, the cooking device 100 can determine whether the change in cooking state is greater than or equal to a fourth threshold. If the change in cooking state is less than the fourth threshold, in operation S1630, the cooking device 100 can determine a predetermined resolution as a first resolution. The cooking device 100 can obtain an image based on the determined first resolution.

[0220] If the change in cooking state is equal to or greater than the fourth threshold, then in operation S1635, the cooking device 100 can determine a predetermined resolution as a second resolution. Here, the second resolution can be greater than the first resolution. The cooking device 100 can obtain an image based on the determined second resolution.

[0221] The cooking device 100 can identify the cooking state of the food object based on a predetermined resolution determined according to operations S1630 and S1635, and repeat operations S1605 to S1620. When the cooking of the food is identified as complete, in operation S1650, the cooking device 100 can generate a video based on the acquired multiple images.

[0222] If the amount of change in cooking state is high, the cooking device 100 can analyze the food object using a higher quality image by increasing the resolution.

[0223] Figure 17 It is a diagram illustrating the process of estimating the cooking state of food.

[0224] refer to Figure 17 During operation S1705, the cooking apparatus 100 can acquire images using the image capture device 110. The cooking apparatus 100 can acquire images based on changes in the cooking state by adjusting settings associated with the image capture device 110, such as exposure, gain, white balance, and frames per second (FPS). The cooking apparatus 100 can acquire images within the cooking chamber 20 using the image capture device 110.

[0225] In operation S1710, the cooking apparatus 100 can analyze the image based on the acquired image. The cooking apparatus 100 can measure information about the food object (food). The cooking apparatus 100 can measure at least one of the food's brightness, color, gradient, size, or texture. The cooking apparatus 100 can perform a measurement operation based on the amount of change in the food by changing the measurement information and the measurement position.

[0226] The cooking apparatus 100 can identify changes in the amount of food (changes in cooking state) by comparing time intervals. The cooking apparatus 100 can perform a change identification operation based on the amount of change by changing at least one of the time interval or the number of comparisons.

[0227] The cooking apparatus 100 can update the feature model of the food's cooking state. The cooking apparatus 100 can identify a basic feature model of the food. The basic feature model can include feature information and can be identified based on at least one of color, size distribution, or change in cooking state. The basic feature model can represent a feature model determined without considering the change in cooking state. For example, the basic feature model can be identified by user selection, while a predetermined feature model is stored in memory. The cooking apparatus 100 can change the feature model of the food's cooking state according to the food's cooking state. For example, the cooking apparatus 100 can analyze an image based on the basic feature model and change the basic feature model to another feature model based on the change in cooking state.

[0228] The cooking equipment 100 can analyze the current cooking state of food by comparing the existing feature model with the changed feature model.

[0229] As a result of the cooking state analysis operation, the cooking device 100 can predict changes in the food. The cooking device 100 can predict changes (or the amount of change) in the food based on the current cooking step.

[0230] The cooking apparatus 100 can predict changes in the food and calculate the next measurement time point. In operation S1705, the cooking apparatus 100 can obtain an image based on the calculated next measurement time point.

[0231] Figure 18 It is a graph showing the cumulative change in the cooking state of food over time.

[0232] refer to Figure 18 Equation 1805 can represent the calculation process used to predict the amount of change at the next time point.

[0233] D_n+1 can refer to the cumulative change predicted at the next time point.

[0234] D_n can refer to the cumulative change in cooking state measured at the current point in time.

[0235] D_step can represent the cumulative amount of cooking state changes based on predetermined cooking steps. D_step can represent the cumulative amount of cooking state changes corresponding to food already stored in memory, rather than information obtained through image analysis. For example, if the food is meat, information about the conventional cooking process for meat can be stored in memory. The cooking device 100 can identify which food to cook based on a user's selection. The cooking device 100 can identify which food to cook based on an image obtained before cooking begins. The cooking device can obtain a basic cumulative amount of cooking state changes corresponding to the identified food. The obtained basic cumulative amount of cooking state changes can represent information about the cumulative amount of cooking state changes over time. For example, basic cooking change information could include information about bread changing from white to brown according to cooking steps.

[0236] D_pred can refer to the predicted amount of change in cooking state.

[0237] D_pred can be represented as the value obtained by summing a_x*diff(Dn,Dx) over a specific time point (nm) to the immediately preceding time point (n-1). Here, x can be a time point variable. Here, m can represent a fixed constant and can refer to the value of the number of data points to be reflected. For example, if 10 data points are to be reflected, then m can be 10 (m=10).

[0238] Here, a_x can represent the weight corresponding to time point x. Cooking equipment 100 can adjust the weight so that a higher weight is applied when cooking equipment 100 is closer to the current time point. diff(Dn,Dx) can be the difference between the amount of cooking state change (cumulative) at the current time point (n) and the amount of cooking state change (cumulative) at a specific time point (x).

[0239] Therefore, for the most recent m time points, D_pred can be a value obtained by adding the difference between the amount of cooking state change (cumulative) at the current time point (N) and the amount of cooking state change (cumulative) at a specific time point (x).

[0240] Cooking equipment 100 can obtain D_n+1 by adding the calculated D_n, D_step and D_pred.

[0241] Equation 1810 can represent a formula for determining the shooting interval (or time interval). The shooting interval can be calculated as b / (D_n+1). Here, b can be a constant or a weight. If the user wishes to increase the shooting interval, they can set the value of b to be greater than the shooting value calculated based on the user settings; and if the user wishes to decrease the shooting interval, they can set the value of b to be less than the calculated shooting interval. Here, D_[n+1] can represent the predicted change in cooking state at the next time point (cumulative) calculated in Equation 1805.

[0242] Figure 1815 can represent the cumulative change in cooking state over time points. The current time point can be t_n. Since the change is cumulative, the gradient in Figure 1815 can represent the change in cooking state at a specific time point. Referring to Figure 1815, the change in cooking state can be very low at initial time points t_0, t_1, and t_2. If some time has elapsed, the change in cooking state can increase at time t_n-5. If a predetermined time has elapsed, the change in cooking state can decrease again from time t_n-2. Since the food changes almost nothing in the initial cooking steps and also almost nothing in the final cooking completion steps, the cooking device 100 can obtain data similar to that in Figure 1815.

[0243] Figure 19 This is a diagram showing the UI displayed when reproducing a video.

[0244] refer to Figure 19 The cooking device 100 can generate a video including a time slideshow UI 1920 representing the flow of time. The time slideshow UI 1920 can be included in one image 1910 of the video. The time slideshow UI 1920 can include a current position UI 1921 to indicate the current point in time in the currently replayed image. The current position UI 1921 can move to the right based on the video replay. One image 1910 of the video can include a replay-related UI 1922, which is used to replay the current video from a previous point in time or from the next point in time, or to pause (or replay) the current video. Changes to the food 1923 can be displayed based on the video replay.

[0245] Typically, the time slideshow UI 1920 can have the same number of frames corresponding to a predetermined interval. However, the cooking device 100 can generate video such that there are different numbers of frames within the predetermined interval.

[0246] Assume the entire cooking process is 35 minutes and divided into four parts. Referring to Table 605, assume the first part is 20 minutes and remains almost unchanged during cooking, with a shooting interval of 20 seconds. The cooking device 100 can acquire 60 frames (images) over 20 minutes based on the 20-second shooting interval.

[0247] Assuming the second part is 5 minutes and the amount of change in cooking status increases, and the shooting interval is 5 seconds, cooking device 100 can obtain 60 frames (images) in 5 minutes using a 5-second shooting interval.

[0248] Assuming the third segment is 5 minutes, and the amount of change in cooking status further increases so that the shooting interval is 1 second, cooking device 100 can acquire 300 frames (images) in 5 minutes using a 1-second shooting interval.

[0249] Assuming the fourth part is 5 minutes, and the amount of change in cooking state decreases so that the shooting interval is 20 seconds, the cooking device 100 can obtain 15 frames (images) in 5 minutes using a 20-second shooting interval.

[0250] The cooking device 100 can adjust the number of frames based on the amount of change in cooking state, rather than allocating the same frame based on the playback time.

[0251] Since the second part is more important than the first part (which is 20 minutes long and the second part is 5 minutes long), but both can have a frame rate of 60, the cooking device 100 can allocate more frames to the desired portion. The cooking device 100 can then generate a summary video.

[0252] To visually inform the user of distorted time information during the summary process, the cooking appliance 100 can use a time slideshow UI 1920. Although both the first and second parts are 60 frames, the time slideshow UIs corresponding to each part can be displayed at different sizes. For example, the time slideshow UI corresponding to the first part can be displayed as a first size 1921-1, while the time slideshow UI corresponding to the second part can be displayed as a second size 1921-2, which is smaller than the first size 1921-1. The size of the time slideshow UI corresponding to each part can be proportional to the reproduced time information before editing.

[0253] The current position UI 1921 can be displayed as the cooking state change moving relatively quickly in the portion where the cooking state change is low, and can be displayed as the cooking state change moving relatively slowly in the portion where the cooking state change is high.

[0254] Figure 20 This is a diagram illustrating the video recording settings UI according to an implementation method.

[0255] refer to Figure 20The cooking appliance 100 may display a video recording settings UI. The cooking appliance 100 may also display a screen 2010 on the display 130 for adjusting the image capture interval. The screen 2010 for adjusting the image capture interval may include a UI that includes a mode name. For example, the screen 2010 for adjusting the image capture interval may include a UI 2012 corresponding to a regular interval that controls the capture interval to a predetermined value, or a UI 2012 corresponding to an adaptive interval where the image capture interval is changed.

[0256] To enhance user convenience, the cooking device 100 can provide notifications of the current setting mode. Specifically, the cooking device 100 may additionally display UI 2013-1 to emphasize the current setting mode. The cooking device 100 may additionally display text information 2013-2 indicating the current setting mode. Each UI 2011 and UI 2012 may be a UI that allows the user to select the shooting interval.

[0257] Figure 21 This is a diagram illustrating the video recording settings UI according to another embodiment.

[0258] refer to Figure 21 The cooking appliance 100 can display a screen 2110 on the display 130 for adjusting the image capturing interval. The screen 2110 for adjusting the image capturing interval can display a UI indicating the degree of adjustment. For example, the cooking appliance 100 can display a UI 2111 corresponding to "insensitive," a UI 2112 corresponding to "normal," and a UI 2113 corresponding to "sensitive." Each UI 2111, UI 2112, and UI 2113 can be a UI that guides the user to select the capturing interval.

[0259] Figure 22 This is a diagram illustrating the video recording settings UI according to yet another embodiment.

[0260] refer to Figure 22 The cooking appliance 100 may display a screen 2210 on the display 130 to guide the selection of video size. The screen 2210 guiding the selection of video size may include at least one UI corresponding to the size information of the final generated video. For example, the cooking appliance 100 may display UI 2211 corresponding to "250MB", UI 2212 corresponding to "500MB", and UI 2213 corresponding to "750MB". The cooking appliance 100 may additionally display size change information (50%, 100%, 150%) to notify the extent of the video size change based on a reference size of 500MB along with information indicating the video size. Each UI 2211, UI 2212, and UI 2213 may be a UI guiding the user to select the size of the final generated video.

[0261] Figure 23 This is a diagram illustrating a video recording settings user interface (UI) according to another embodiment.

[0262] refer to Figure 23 The cooking appliance 100 may display a screen 2310 on the display 130 to guide the selection of a video playback time. The screen 2310 guiding the selection of a video playback time may include at least one UI corresponding to the timing information of the final generated video. For example, the cooking appliance 100 may display a UI 2311 corresponding to "30 seconds," a UI 2312 corresponding to "60 seconds," and a UI 2313 corresponding to "90 seconds." The cooking appliance 100 may additionally display playback time change information (50%, 100%, and 150%) to indicate the degree of change in video playback time based on a reference playback time (60 seconds) along with information indicating the video playback time. Each UI 2311, 2312, and 2313 may be a UI guiding the user to select the playback time of the final generated video.

[0263] Figure 24 This is a flowchart illustrating a control method for a cooking apparatus according to an embodiment.

[0264] refer to Figure 24 The control method of the cooking device 100 includes: acquiring multiple images in operation S2405; identifying food objects from the acquired multiple images in operation S2410; adjusting the shooting interval of acquiring food object images in real time based on information about changes in the cooking state of the identified food objects in operation S2415; and generating video based on the captured images acquired according to the adjusted shooting interval in operation S2420.

[0265] Information on changes in cooking status can include the amount of change in cooking status, and adjusting the shooting interval can include adjusting the shooting interval to be inversely proportional to the amount of change in the cooking status of the food object.

[0266] Adjusting the shooting interval in operation S2415 may include: increasing the shooting interval based on the amount of change in the cooking state of the food object being less than a first threshold, and decreasing the shooting interval based on the amount of change in the cooking state of the food object being greater than or equal to a second threshold.

[0267] The method may further include: analyzing the food object using at least one feature from the acquired multiple images, and adjusting the number of at least one feature to be proportional to the amount of change in the cooking state of the food object.

[0268] The feature information may include at least one of contour, edge, corner, histogram or brightness; and the control method of the cooking device 100 may also include: extracting feature information from multiple images obtained, and obtaining information on the food object and the cooking state change of the food object based on the extracted feature information.

[0269] The method for controlling the cooking device 100 may further include the following steps: identifying a region in which the change in the cooking state of a food object obtained from multiple images is greater than or equal to a third threshold as a target region, and obtaining the change in the cooking state of the identified food object based on the identified target region.

[0270] The control method of the cooking device 100 may also include: changing the resolution of the obtained image based on the fact that the change in the cooking state of the food object is greater than or equal to a fourth threshold.

[0271] The control method of the cooking device 100 may further include: obtaining a predicted cooking state change of the food object based on the cooking state change amount included in the stored food information and the cooking state change information of the food object; and changing the shooting interval of the image capturer based on the predicted cooking state change of the food object.

[0272] The control method of the cooking device 100 may further include: obtaining the cooking state change of the identified food object based on at least one of the size change (or size state change) and color change (or color state change) of the food object; and adjusting the image capturing interval based on the obtained cooking state change.

[0273] The control method of the cooking device 100 may also include: displaying a user interface (UI) for changing the shooting interval.

[0274] like Figure 24 As shown, the method of electronic devices can be used in having Figure 1 or Figure 2 It can be executed on cooking equipment with a certain structure, and it can also be executed on cooking equipment with other structures.

[0275] The method described above can be implemented as an application that can be executed in existing cooking equipment (electronic devices).

[0276] The method described above can be implemented as software or hardware for a cooking device (electronic device).

[0277] The above-described embodiments can be implemented by an embedded server disposed in the cooking device (electronic device), or an external server of at least one of the cooking device (electronic device) or display device.

[0278] The implementation can be software-based, including instructions stored on a machine-readable storage medium (e.g., a computer). A device can read and execute the instructions from the storage medium, including cooking appliances (electronic devices). When an instruction is executed by a processor, the processor can perform the function corresponding to the instruction directly or by utilizing other components controlled by the processor. The instructions may include computer-readable code generated by a compiler or code executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium, meaning that the storage medium is tangible and does not distinguish between cases where data is stored semi-permanently and cases where data is temporarily stored in the storage medium.

[0279] The method described above can be included in a computer program product. The computer program product can be traded between a seller and a consumer. The computer program product can be distributed online in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)) or through application storage (e.g., a Playstore). TM The computer program product may be distributed either directly online or temporarily stored. When distributed online, at least a portion of the computer program product may be temporarily stored or generated at least temporarily on the manufacturer's server, the application store's server, or a machine-readable storage medium such as a relay server storage.

[0280] The aforementioned elements (e.g., modules or programs) may comprise a single entity or multiple entities. At least one of the corresponding elements or operations may be omitted, or at least one other element or operation may be added. Alternatively or additionally, components (e.g., modules or programs) may be combined to form a single entity. In this configuration, the integrated entity may perform at least one function of each of the multiple elements in the same or similar manner as each of the multiple elements prior to integration. Modules, program modules, or operations performed by other elements according to the implementation may be performed sequentially, in parallel, repeatedly, or heuristically, or at least some operations may be performed in a different order, may be omitted, or additional operations may be added to them.

[0281] While this disclosure has been specifically shown and described with reference to certain embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the scope of the subject matter as defined by the appended claims and their equivalents.

Claims

1. A cooking appliance, comprising: Image capture device; as well as The processor is configured as follows: Identify food objects from multiple images acquired by the image capture device. Based on information about changes in the cooking state of the identified food object, the capturing interval of images of the food object acquired in real time by the image capture device is adjusted, and Based on the adjusted shooting interval, a video is generated from the acquired images.

2. The cooking apparatus according to claim 1, wherein, Information regarding the change in cooking state includes the amount of change in cooking state, and The processor is further configured to adjust the shooting interval to be inversely proportional to the amount of change in the cooking state of the food object.

3. The cooking apparatus according to claim 2, wherein, The processor is also configured to: Based on the fact that the change in the cooking state of the food object is less than a first threshold, the shooting interval of the image capture device is increased, and Based on the fact that the change in the cooking state of the food object is greater than or equal to a second threshold, the shooting interval of the image capture device is reduced.

4. The cooking apparatus according to claim 2, wherein, The processor is also configured to: The food object is analyzed using at least one feature from multiple acquired images, and The quantity of the at least one feature information is adjusted to be proportional to the amount of change in the cooking state of the food object.

5. The cooking apparatus according to claim 4, wherein, The at least one feature includes at least one of contour, edge, corner, histogram, or brightness, and The processor is further configured to: extract at least one feature information from the acquired plurality of images, and obtain information about the food object and changes in the cooking state of the food object based on the extracted at least one feature information.

6. The cooking apparatus according to claim 2, wherein, The processor is also configured to: From the acquired multiple images, regions whose changes in the cooking state of the food object are greater than or equal to a third threshold are identified as target regions. Based on the identified target area, the amount of change in the cooking state of the identified food object is obtained.

7. The cooking apparatus according to claim 2, wherein, The processor is also configured to change the resolution of the image obtained from the image capture device based on the fact that the change in the cooking state of the food object is greater than or equal to a fourth threshold.

8. The cooking apparatus according to claim 2, further comprising: The memory is configured to store food information corresponding to each of the multiple foods. The processor is further configured as follows: Based on the cooking state change amount included in the stored food information and the information about the cooking state change of the food object, a predicted cooking state change amount of the food object is obtained, and The shooting interval of the image capture device is changed based on the predicted change in the cooking state of the food object.

9. The cooking apparatus according to claim 2, wherein, The processor is also configured to: Based on at least one of the changes in size and color of the food object, the cooking state change of the identified food object is obtained, and The image capture interval is adjusted based on the amount of change in cooking state obtained.

10. The cooking apparatus according to claim 1, further comprising: monitor, The display is controlled to show a user interface (UI) for changing the shooting interval.

11. A method for controlling a cooking device, the method comprising: Obtain multiple images; Identify food objects from multiple acquired images; Based on information about changes in the cooking state of the identified food object, the shooting interval for acquiring images of the food object in real time is adjusted; as well as Based on the adjusted shooting interval, a video is generated from the acquired images.

12. The method according to claim 11, wherein, Information regarding the change in cooking state includes the amount of change in cooking state, and Adjusting the shooting interval includes adjusting the shooting interval to be inversely proportional to the amount of change in the cooking state of the food object.

13. The method according to claim 12, wherein, Adjusting the shooting interval includes: Based on the fact that the change in the cooking state of the food object is less than a first threshold, the shooting interval is increased, and The shooting interval is reduced based on the fact that the change in the cooking state of the food object is greater than or equal to a second threshold.

14. The method of claim 12, further comprising: The food object is analyzed using at least one feature from multiple acquired images; as well as The quantity of the at least one feature information is adjusted to be proportional to the amount of change in the cooking state of the food object.

15. The method according to claim 14, wherein, The at least one feature includes at least one of contour, edge, corner, histogram, or brightness, and The method further includes: extracting at least one feature information from the acquired plurality of images, and obtaining information about the food object and changes in the cooking state of the food object based on the extracted at least one feature information.

Images