Artificial intelligence device and operation method thereof

Abstract

An artificial intelligence range hood includes: a main body; a camera disposed at a lower end of the main body and configured to capture an image of a cooking bench located below the main body; a display on the front surface of the main body; a memory storing an object recognition model; and a processor configured to control the camera to capture an image of the cooking bench, generate object identification information for identifying a cooking object included in the captured image of the cooking bench using an object identification model stored in the memory, setting a user region of interest of the captured image corresponding to the identified cooking object specified by the generated object information, controlling an operation of the camera to capture the user region of interest of the captured image, and controlling the display to display the user region of interest of the captured image captured.

Description

Technical Field

[0001] This application claims the benefit of and priority to the earlier filing date of U.S. patent application Ser. No. 18 / 104,422, filed on February 1, 2023, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to an artificial intelligence range hood, which includes a camera that captures a cooking process performed on a top plate of a stove based on user input. Background Art

[0003] Artificial intelligence (AI) is a field of computer engineering and information technology that involves computer thinking, learning, and self-improvement. Research is currently underway to leverage AI to recognize and learn from surrounding situations, provide users with desired information, or perform desired operations or functions. Electronic devices that provide these various operations and functions are also referred to as AI devices.

[0004] In addition, an over-the-range (OTR) or range hood includes a microwave oven installed on a home stove. Furthermore, a range hood located on a stovetop can exhaust fumes and odors generated by the cooking utensils to the outside through a fan. The range hood is also installed with the stovetop. However, existing stovetops with cameras do not allow users to easily capture and share various recipe photos and videos. Summary of the Invention

[0005] Technical issues

[0006] Accordingly, one aspect of the present disclosure is to address the above-mentioned and other problems of related art methods of capturing and providing cooking content.

[0007] Another aspect of the present disclosure is to provide an artificial intelligence range hood and a control method thereof, wherein the artificial intelligence range hood captures a cooking process performed on a countertop of a stove based on user input.

[0008] Yet another aspect of the present disclosure is to provide an artificial intelligence range hood and a control method thereof, wherein the artificial intelligence range hood includes a camera for photographing a tabletop of a stove and a display for receiving user input while displaying the photographed image.

[0009] Yet another aspect of the present disclosure is to provide an artificial intelligence range hood and a control method thereof, wherein the artificial intelligence range hood recognizes food on a cooking zone, cooking utensils, or a countertop of a stove to provide a user with various camera user experiences / user interfaces (UX / UI).

[0010] Another aspect of the present disclosure is to provide an artificial intelligence range hood and a control method thereof, wherein the artificial intelligence range hood identifies an object located in a user's area of ​​interest and automatically changes shooting conditions according to features of the identified object.

[0011] Yet another aspect of the present disclosure is to provide an artificial intelligence range hood and a control method thereof, wherein the artificial intelligence range hood adjusts the audio focus to a cooking zone area identified as being cooking.

[0012] Yet another aspect of the present disclosure is to provide an artificial intelligence range hood and a control method thereof, wherein the artificial intelligence range hood provides various recipes using an open API (application programming interface).

[0013] Problem Solution

[0014] To achieve the above and other objectives, an artificial intelligence range hood according to an embodiment of the present disclosure includes: a camera for capturing an image of a stovetop located below the range hood; a display located on the front surface of the range hood body for displaying images captured by the camera; and a processor for acquiring image data from the images captured using an object recognition model to generate object recognition information using the object recognition model. Furthermore, the processor receives user input based on screen information generated in association with the object recognition information and displayed on the display, and controls the operation of the camera based on the received user input.

[0015] In addition, the object recognition information may include object identification information (regarding an object included in the acquired image data) and object position information. The object recognition information includes, for example, at least one of a cooking vessel, food, a cooking utensil, and a user's hand.

[0016] In addition, the processor can use the object recognition information to identify one or more cooking zones, cooking utensils and at least one of the foods being cooked, display an area on the display including the identified one or more cooking zones, cooking utensils and at least one of the foods being cooked, and then set an area of ​​the stove selected by the user from an area displayed on the display as the user's area of ​​interest.

[0017] The processor can also receive user input including time-lapse shooting speed and shooting mode, and control the camera to shoot the user's area of ​​interest based on the received user input.

[0018] In addition, the processor can use the object recognition information and the object position information to respectively calculate the proportion of the area occupied by food and the proportion of the area occupied by objects other than food within the user's area of ​​interest, increase the time-lapse shooting speed when the proportion of the area occupied by food is greater than a preset first threshold, and reduce the time-lapse shooting speed when the proportion of the area occupied by food is less than the first threshold and the proportion of the area occupied by objects other than food is greater than a preset second threshold. In addition, the first threshold is set as a reference value for determining the area occupied by food within the user's area of ​​interest, and the second threshold is set as a reference value for determining the area occupied by objects other than food within the user's area of ​​interest.

[0019] The memory may also store a food recognition model, and when food is included in the object recognition information, the processor generates food recognition information using the food recognition model, and the food recognition information identifies the type of food included in the acquired image data. In addition, the processor may generate a list of color correction filters to display on the display, and may also receive one of the color correction filter lists as user input.

[0020] The memory can also store a motion recognition model, and when the user's hand is included in the object recognition information, the processor generates motion recognition information using the motion recognition model, and the motion recognition information recognizes the motion type of the user's hand included in the acquired image data, and the motion recognition information includes at least one of a motion recognition start motion, a thumb up motion, a thumb down motion, a zoom in motion, and a zoom out motion.

[0021] When the motion recognition information is a motion recognition start action, the processor can change the motion recognition mode to the on (ON) state; when the motion recognition information is one of a thumbs-up action and a thumbs-down action and the motion recognition mode is in the on state, the processor increases or decreases the time-lapse shooting speed; and when the motion recognition information is one of a zoom-in action and a zoom-out action and the motion recognition mode is in the on state, the processor zooms in or out of the screen displaying the user's area of ​​interest.

[0022] In addition, the AI ​​range hood may also include multiple microphones located at the lower end of the main body, such as a directional microphone that receives audio signals generated from the stove top. When a user's area of ​​interest is set, the processor may move the microphone's audio focus to the set user's area of ​​interest.

[0023] Furthermore, the processor may obtain first image data and second image data for a current frame and a previous frame from an image captured by the camera, calculate a similarity between the current frame and the previous frame, and determine the current frame as a first change point frame when the calculated similarity is less than a preset threshold similarity. The processor may generate a change point frame list including the first change point frame, display the generated change point frame list on a display, and receive a start frame and an end frame in the change point frame list as user input to generate a highlight segment.

[0024] In addition, the similarity between the current frame and the previous frame can be calculated based on at least one of color, edge, histogram, correlation, and motion vector of optical flow extracted from the first image data and the second image data for the current frame and the previous frame. In addition, the processor can generate first object recognition information and second object recognition information from the first image data and the second image data to respectively calculate the proportion of the food area within the user's region of interest, and increase or decrease the threshold similarity based on the calculated proportions.

[0025] The processor may also generate a recipe database (DB) based on the recipe information collected through the open API to store the generated recipe database in the memory, wherein a plurality of recipes including one or more entries of dish names, cooking methods, ingredient information, multiple images showing the cooking methods, and multiple texts describing the cooking methods are stored in the recipe database, and a cooking utensil entry corresponding to the cooking method is generated and also included in the recipe.

[0026] The memory may further store a cooking vessel recognition model, and when the cooking vessel is included in the object recognition information, the processor generates cooking vessel recognition information using the cooking vessel recognition model, and the cooking vessel recognition information recognizes a type of cooking vessel included in the acquired image data.

[0027] The processor may also retrieve a cooking vessel entry corresponding to the cooking vessel identification information from the recipe database, extract one or more dish names associated with the retrieved cooking vessel entry, generate a recommended dish list including the one or more extracted dish names, and display the list on the display. Furthermore, the processor may receive a dish from the recommended dish list as user input, extract a recipe associated with the dish received as user input from the recipe database, and display on the display at least one of a plurality of images illustrating a cooking method included in the extracted recipe and a plurality of texts describing the cooking method.

[0028] The AI ​​range hood may further include a plurality of speakers, and the processor may convert at least one of the plurality of texts describing the cooking method into speech using a text-to-speech (TTS) engine stored in the memory, and output the converted speech through the plurality of speakers. The memory may further include an ingredient recognition model, and when one or more ingredients are included in the object recognition information, the processor may generate one or more ingredient recognition information using the ingredient recognition model, and the ingredient recognition information may identify the type of the ingredient included in the acquired image data.

[0029] The processor may also generate additional information related to the ingredient identification information, and display the generated additional information on the display, wherein the additional information is retrieved and generated from a recipe database or an external server. The processor may also generate a recommended ingredient list based on the ingredient information included in the recipe received as user input and the ingredient identification information of the ingredients included in image data obtained by capturing the user's area of ​​interest, and display the generated recommended ingredient list on the display.

[0030] In order to achieve the above and other purposes, according to one aspect of the present disclosure, the present disclosure also provides a method for controlling an artificial intelligence range hood, the method comprising photographing a tabletop of a stove located below the artificial intelligence range hood to display the photographed tabletop on a display; acquiring image data from the photographed image to generate object recognition information using an object recognition model; receiving user input based on screen information generated in association with the generated object recognition information to be displayed on the display; and controlling the operation of a camera according to the received user input.

[0031] In addition, receiving user input may include: using object recognition information to identify at least one of one or more cooking zones, cooking utensils, or foods being cooked; displaying an area on the display including the identified one or more cooking zones, cooking utensils, or at least one of foods being cooked; setting an area of ​​the stovetop selected by the user in an area displayed on the display as a user area of ​​interest; and receiving user input including a time-lapse shooting speed and a shooting mode, wherein controlling the operation of the camera includes controlling the camera to shoot the user area of ​​interest based on the received user input.

[0032] Advantageous Effects of the Invention

[0033] According to embodiments of the present disclosure, the following effects can be achieved: when capturing a cooking process on a stovetop, one or more cooking zones and an object being cooked on the stovetop can be identified, thereby providing various camera user experience / user interface (UX / UI) forms along with the captured image.

[0034] Furthermore, a user's area of ​​interest (ROI) can be set, encompassing at least one of one or more cooking zones, cooking utensils, and the food being cooked. One or more objects within the ROI can be identified, automatically changing the capture conditions based on the type and state of the identified objects. Furthermore, the audio focus can be moved to the user's area of ​​interest, encompassing at least one of one or more cooking utensils, cooking utensils, and the food being cooked, primarily recording the sounds generated by the cooking zone area.

[0035] Furthermore, a recipe database may be generated using an open API, and a cooking vessel identified as being cooked may be identified, thereby providing the user with recipes related to the identified cooking vessel. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention will be more fully understood from the detailed description given below and the accompanying drawings, which are illustrative only and therefore do not limit the present invention, in which:

[0037] Figure 1 is a block diagram illustrating an artificial intelligence (AI) device according to an embodiment of the present disclosure.

[0038] Figure 2 is a block diagram illustrating an AI server according to an embodiment of the present disclosure.

[0039] Figure 3 is an overview diagram showing an AI system according to an embodiment of the present disclosure.

[0040] Figure 4 is a block diagram illustrating an AI device according to another embodiment of the present disclosure.

[0041] Figure 5 is a perspective view showing an artificial intelligence range hood and a stove including a camera and a display according to an embodiment of the present disclosure.

[0042] Figure 6a 1 is an operational flowchart for controlling a camera and a display provided in an artificial intelligence range hood according to an embodiment of the present disclosure.

[0043] Figure 6b Included is an overview of the cooktop area captured by the camera and displayed on the display.

[0044] Figure 6c is a conceptual diagram illustrating a method of generating object recognition information using an object recognition model.

[0045] Figure 7a is an operational flowchart illustrating a method for automatically adjusting a time-lapse shooting speed based on object recognition information according to an embodiment of the present disclosure.

[0046] Figure 7b Included is an overview of a screen showing automatic adjustment of time-lapse speed.

[0047] Figure 8a FIG. 4 is an operational flowchart illustrating a method for providing a color correction filter list based on food recognition information according to an embodiment of the present disclosure.

[0048] Figure 8b is an operational flow diagram illustrating a method of applying a color correction filter to stored food-related images.

[0049] Figure 8c Included is an overview showing a screen providing a list of color correction filters.

[0050] Figure 8d is a conceptual diagram illustrating a method of generating food recognition information using a food recognition model.

[0051] Figure 9a is an operational flowchart illustrating a method of recognizing a user's hand motion to automatically adjust a cooking screen and a time-lapse shooting speed according to an embodiment of the present disclosure.

[0052] Figure 9b and Figure 9c An overview diagram showing a cooking screen and a screen that automatically adjusts the time-lapse shooting speed based on recognized hand motions of the user is included.

[0053] Figure 9d is a conceptual diagram illustrating a method of generating motion recognition information using a motion recognition model.

[0054] Figure 10a is an operational flow chart illustrating a method of recording sounds generated during a cooking process using a directional microphone according to an embodiment of the present disclosure.

[0055] Figure 10b Included is an overview diagram showing the microphone's audio focus being moved to the user's area of ​​interest.

[0056] Figure 11a 1 is an operational flowchart illustrating a method for identifying a portion in a cooking process captured by a camera where a large change occurs, according to an embodiment of the present disclosure.

[0057] Figure 11b FIG. 1 is an operational flowchart illustrating a method for generating a highlight segment after shooting is completed.

[0058] Figure 12a is an operational flow chart illustrating a method for providing recommended recipes and information about ingredients used in the recipes using an open API according to an embodiment of the present disclosure.

[0059] Figure 12b is an exemplary view showing recipe information collected using an open API.

[0060] Figure 12c is an exemplary view showing a recipe database generated based on collected recipe information.

[0061] Figure 12d is an exemplary view illustrating a screen on which a list of recommended dishes and recipes are provided.

[0062] Figure 13a is a flowchart illustrating an operation of a method for identifying one or more ingredients contained in food to provide a recommended ingredient list.

[0063] Figure 13b : is an exemplary view showing a screen on which a recommended ingredient list is provided. DETAILED DESCRIPTION

[0064] The following description will now be described in detail with reference to the accompanying drawings according to the exemplary embodiments disclosed herein. For the convenience of brief description with reference to the accompanying drawings, the same or equivalent elements may use the same or similar figure marks, and the description thereof will not be repeated. The suffixes "module" or "unit" used for the elements disclosed in the following description are merely for the convenience of describing this specification, and the suffixes themselves do not give any special meaning or function. In addition, the drawings are provided to better understand the embodiments disclosed herein, rather than to limit the technical concepts disclosed herein, and therefore, it should be understood that the drawings cover all modifications, equivalents and alternatives within the scope of the concepts and techniques disclosed herein.

[0065] Terms including ordinal numbers such as first, second, etc. may be used to describe various elements, but the elements are not limited by these terms. These terms are only used to distinguish one element from another. When an element is referred to as being "connected to" or "coupled to" another element, the element may be connected to the other element, or there may be intervening elements. In contrast, when an element is "directly connected to" or "directly coupled to" another element, there are no other elements between them.

[0066] Artificial Intelligence (AI)

[0067] Artificial intelligence refers to the field that studies artificial intelligence or methods that can produce it. Machine learning defines the various problems addressed in the field of artificial intelligence and studies methods for solving them. Machine learning is also defined as algorithms that improve their performance on tasks through continuous experience with those tasks.

[0068] An artificial neural network (ANN) is a model used in machine learning. It is configured with artificial neurons (nodes) that form a network through a combination of synapses, meaning that the entire model has the ability to solve problems. An artificial neural network can be defined by the connection pattern between neurons in different layers, the learning process that updates the model parameters, and the activation function used to generate output values.

[0069] An artificial neural network can include an input layer, an output layer, and optionally one or more hidden layers. Each layer includes one or more neurons. An artificial neural network can also include synapses connecting neurons. In an artificial neural network, each neuron can output a function value of an activation function based on an input signal, weight, and bias input via a synapse.

[0070] Model parameters are those determined through learning, including the weights of synaptic connections and the biases of neurons. Furthermore, hyperparameters are those that need to be configured before learning in a machine learning algorithm, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0071] The learning object of the artificial neural network can determine the model parameters that minimize the loss function. The loss function can be used as an indicator to determine the optimal model parameters during the learning process of the artificial neural network.

[0072] Based on the learning method, machine learning can be divided into supervised learning, unsupervised learning and reinforcement learning. Supervised learning refers to a method of training an artificial neural network in a state where the labels of the learning data have been given. The label can refer to the answer (or result value) that the artificial neural network must deduce when the learning data is input into the artificial neural network. Unsupervised learning refers to a method of training an artificial neural network without giving the labels of the learning data. Reinforcement learning refers to a learning method in which an agent (agent) defined in an environment is trained to select an action or sequence of actions that maximizes the accumulated compensation (accumulated reward) in each state.

[0073] In artificial neural networks, machine learning implemented as a deep neural network (DNN) with multiple hidden layers is also called deep learning. Deep learning is a part of machine learning. In the following, machine learning is used to include deep learning.

[0074] Object detection models using machine learning include the single-stage "You Only Look Once (YOLO)" model and the two-stage "Faster Regions with Convolution Neural Networks (Faster R-CNN)" model. The YOLO model is a model that can predict the objects present and their locations in an image after observing the image only once.

[0075] The YOLO model divides the original image into grids of equal size. For each grid, the number of bounding boxes specified in a predetermined format centered around the grid center is predicted, and the confidence level is calculated accordingly. It can then determine whether the image contains an object or just background, and select locations with high object confidence to identify the object category. The R-CNN model is a model that can detect objects faster than the RCNN model and the Fast RCNN model.

[0076] The R-CNN model is described in detail. First, a convolutional neural network (CNN) model extracts feature maps from an image. Based on the extracted feature maps, multiple regions of interest (RoIs) are extracted. RoI pooling is performed on each region of interest.

[0077] RoI pooling is a process that sets a grid so that the feature map of the region of interest projected onto it conforms to a predetermined H×W size, and extracts the maximum value for each space included in each grid to extract a feature map of size H×W. A feature vector can be extracted from the feature map of size H×W, and object recognition information can be obtained from the feature vector.

[0078] Extended Reality (XR)

[0079] Extended reality broadly refers to virtual reality (VR), augmented reality (AR), and mixed reality (MR). VR technology presents real-world objects or backgrounds solely as CG images. AR technology overlays virtual CG images on real-world images. MR technology is a computer graphics technology that blends and combines virtual objects with the real world for presentation.

[0080] MR technology is similar to AR technology in that both can display real and virtual objects. However, in AR, virtual objects are used to complement real objects. Conversely, unlike AR, in MR, virtual and real objects are used as the same character.

[0081] XR technology can be applied to head-mounted displays (HMDs), head-up displays (HUDs), mobile phones, tablets, laptops, desktops, TVs, and digital signage. Devices that apply XR technology can be called XR devices.

[0082] Next, Figure 1 1 is a block diagram illustrating an AI device 100 according to an embodiment of the present disclosure. The AI ​​device 100 can be implemented as a fixed device or a mobile device, such as a TV, a projector, a mobile phone, a smartphone, a desktop computer, a laptop computer, a digital broadcast terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a tablet PC, a wearable device, a set-top box (STB), a digital multimedia broadcasting (DMB) receiver, a radio, a washing machine, a refrigerator, a desktop computer, a digital signage, a robot, and a vehicle.

[0083] Reference Figure 1 , the AI ​​device 100 may include a communication unit 110, an input unit 120, a learning processor 130, a sensing unit 140, an output unit 150, a memory 170, and a processor 180. The communication unit 110 may utilize wired and wireless communication technologies to communicate with external devices such as Figure 3 The communication unit 110 can transmit and receive data with the other AI devices 100a to 100e or the AI ​​server 200 shown. For example, the communication unit 110 can transmit and receive sensor information, user input, learning models, and control signals to and from external devices.

[0084] In addition, the communication technology used by the communication unit 110 includes Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Long Term Evolution (LTE), 5G, Wireless Local Area Network (WLAN), Wireless Fidelity (Wi-Fi), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), ZigBee, Near Field Communication (NFC), etc.

[0085] The input unit 120 can obtain various types of data and includes a camera for inputting image signals, a microphone for receiving audio signals, and a user input unit 123 for receiving information from a user. Here, the camera or the microphone may be regarded as a sensor, and the signal obtained from the camera or the microphone may be referred to as sensing data or sensor information.

[0086] In addition, the input unit 120 can obtain learning data for model learning and input data to be used when obtaining output using the learning model. The input unit 120 can also obtain unprocessed input data, and the processor 180 or the learning processor 130 can extract input features by preprocessing the input data.

[0087] Furthermore, the learning processor 130 can be trained using a model configured with an artificial neural network utilizing learning data. A trained artificial neural network can be referred to as a learning model. Specifically, the learning model is used to derive resultant values ​​for new input data rather than the learned data. The derived values ​​can then be used as the basis for performing a given operation.

[0088] The learning processor 130 can be used with Figure 2 and Figure 3 The AI ​​processing is performed together with the learning processor 240 of the AI ​​server 200 shown. The learning processor 130 may include a memory integrated in the AI ​​device 100 or implemented in the AI ​​device 100. Alternatively, the learning processor 130 may be implemented using the memory 170, an external memory directly connected to the AI ​​device 100, or a memory maintained in an external device.

[0089] In addition, the sensing unit 140 can use various sensors to obtain at least one of internal information of the AI ​​device 100, surrounding environment information of the AI ​​device 100, or user information. In this case, the sensors included in the sensing unit 140 include a proximity sensor, an illumination sensor, an acceleration sensor, a magnetic sensor, a gyro sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, a lidar, and a radar.

[0090] In addition, the output unit 150 can generate outputs related to vision, hearing, or touch. The output unit 150 can include a display module for outputting visual information, a speaker for outputting auditory information, and a haptic module for outputting tactile information.

[0091] In addition, the memory 170 may store data supporting various functions of the AI ​​device 100. For example, the memory 170 may store input data obtained by the input unit 120 and store learning data, a learning model, a learning history, and the like.

[0092] The processor 180 may also determine at least one executable operation of the AI ​​device 100 based on information determined or generated using a data analysis algorithm or a machine learning algorithm. In addition, the processor 180 may perform the determined operation by controlling elements of the AI ​​device 100.

[0093] In addition, the processor 180 can request, search, receive, and utilize data from the learning processor 130 or the memory 170, and control the elements of the AI ​​device 100 to perform the predicted operation or the operation determined to be preferred among at least one executable operation. If the determined operation is performed using the association with the external device, the processor 180 can generate a control signal for controlling the corresponding external device and transmit the generated control signal to the corresponding external device.

[0094] The processor 180 may also obtain the intent information of the user input and send the user demand based on the obtained intent information. For example, the processor 180 may utilize at least one of a speech-to-text (STT) engine for converting voice input into a text string or a natural language processing (NLP) engine for obtaining intent information in natural language to obtain the intent information corresponding to the user input.

[0095] At least one of the STT engine or the NLP engine may be configured as an artificial neural network trained based on a machine learning algorithm. In addition, at least one of the STT engine or the NLP engine may have been trained by the learning processor 130, or may have been trained by the learning processor 130. Figure 2 and Figure 3 The learning processor 240 of the AI ​​server 200 is shown trained, or may have been trained by its distributed processing.

[0096] In addition, the processor 180 may collect historical information including the operation content of the AI ​​device 100 or the user's feedback on the operation, store the historical information in the memory 170 or the learning processor 130, or transmit the historical information to an external device such as Figure 2 and Figure 3 The AI ​​server 200 is shown. The collected historical information can be used to update the learning model.

[0097] The processor 180 may control at least some elements of the AI ​​device 100 to execute an application program stored in the memory 170. The processor 180 may also combine and drive two or more elements included in the AI ​​device 100 to execute an application program.

[0098] Next, Figure 2 is a block diagram showing an AI server 200 according to an embodiment of the present disclosure. Figure 2 , AI server 200 corresponds to a device trained by an artificial neural network using a machine learning algorithm or a device using a trained artificial neural network. AI server 200 is configured with multiple servers and can perform distributed processing and is defined as a 5G network. AI server 200 can also be included as part of the configuration of AI device 100 and can perform at least some AI processing.

[0099] In addition, if Figure 2 As shown, the AI ​​server 200 may include a communication unit 210, a memory 230, a learning processor 240, and a processor 260. The communication unit 210 may transmit and receive data to and from an external device such as the AI ​​device 100. In addition, the memory 230 may include a model storage unit 231 that stores a model (or an artificial neural network 231a) that is being trained or has been trained by the learning processor 240.

[0100] The learning processor 240 can also use the learning data to train the artificial neural network 231a. The learning model can be used when it is already installed on the AI ​​server 200 of the artificial neural network, or it can be installed on an external device such as the AI ​​device 100 and used.

[0101] Furthermore, the learning model may be implemented as hardware, software, or a combination of hardware and software. If some or all of the learning model is implemented as software, one or more instructions for configuring the learning model may be stored in the memory 230. The processor 260 may utilize the learning model to derive a result value for new input data and generate a response or control command based on the derived result value.

[0102] Next, Figure 3 1 is an overview diagram showing an AI system 1 according to an embodiment of the present disclosure. Figure 3 , the AI ​​system 1 is connected to at least one of the AI ​​server 200, the robot 100a, the autonomous driving vehicle 100b, the XR device 100c, the smartphone 100d, or the home appliance 100e through the cloud network 10. The robot 100a, the autonomous driving vehicle 100b, the XR device 100c, the smartphone 100d, or the home appliance 100e to which AI technology has been applied may be referred to as AI devices 100a to 100e.

[0103] In addition, the cloud network 10 may configure a portion of the cloud computing infrastructure, or may refer to a network existing within the cloud computing infrastructure. For example, the cloud network 10 may be configured using a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network. That is, the devices 100a to 100e (200) configuring the AI ​​system 1 may be interconnected via the cloud network 10. In particular, the devices 100a to 100e and 200 may communicate with each other via a base station, but may also communicate directly with each other without the intervention of a base station.

[0104] In addition, the AI ​​server 200 includes a server for performing AI processing and a server for performing calculations on a large amount of data. Figure 3As shown, the AI ​​server 200 is connected to at least one of the robot 100a, the autonomous driving vehicle 100b, the XR device 100c, the smartphone 100d, or the home appliance 100e (i.e., the AI ​​device configuring the AI ​​system 1) via the cloud network 10, and can assist at least some AI processing of the connected AI devices 100a to 100e.

[0105] Furthermore, the AI ​​server 200 can train an artificial neural network based on a machine learning algorithm on behalf of the AI ​​devices 100a to 100e, directly store the learning model, or transmit the learning model to the AI ​​devices 100a to 100e. The AI ​​server 200 can also receive input data from the AI ​​devices 100a to 100e, derive the result value of the received input data using the learning model, generate a response or control command based on the derived result value, and transmit the response or control command to the AI ​​devices 100a to 100e. Alternatively, the AI ​​devices 100a to 100e can directly derive the result value of the input data using the learning model, and generate a response or control command based on the derived result value.

[0106] Hereinafter, various implementations of the AI ​​devices 100 a to 100 e applying the above-described technology will be described. Figure 3 The AI ​​devices 100a to 100e shown can be considered as Figure 1 The specific implementation of the AI ​​device 100 is shown.

[0107] <AI+XR>

[0108] AI technology is applied to the XR device 100c, and the XR device 100c can be implemented as a head-mounted display, a head-up display set in a vehicle, a television, a mobile phone, a smartphone, a computer, a wearable device, a home appliance, a digital signage, a vehicle, a fixed robot, or a mobile robot. The XR device 100c can generate position data and attribute data of three-dimensional points by analyzing three-dimensional point cloud data or image data obtained via various sensors or from external devices, obtain information about the surrounding space or real objects based on the generated position data and attribute data, and output the XR object by rendering the XR object. For example, the XR device 100c can output an XR object including additional information of the recognized object by making the XR object correspond to the corresponding recognized object.

[0109] Furthermore, the XR device 100c can utilize a learning model configured with at least one artificial neural network to perform the above operations. For example, the XR device 100c can utilize the learning model to identify real objects in three-dimensional point cloud data or image data and provide information corresponding to the identified real objects. The learning model can have been trained directly in the XR device 100c or in an external device such as the AI ​​server 200.

[0110] In addition, the XR device 100 c may directly generate a result using a learning model and operate, but may also operate by sending sensor information to an external device such as the AI ​​server 200 and receiving a result generated in response thereto.

[0111] Next, Figure 4 1 is a block diagram illustrating an AI device 100 according to an embodiment of the present disclosure. Hereinafter, the AI ​​device 100 may also be referred to as an artificial intelligence cooking device or an artificial intelligence range hood (over-the-range (OTR)).

[0112] As shown, the input unit 120 may include a camera 121 for receiving image signals, a microphone 122 for receiving audio signals, and a user input unit 123 for receiving information from a user. Voice data or image data collected by the input unit 120 may be analyzed and processed by a user's control command.

[0113] In addition, the input unit 120 can receive video information (or signals), audio information (or signals), data or user input information. In order to receive video information, the AI ​​device 100 may include one or more cameras 121. The camera 121 can process image frames of static or dynamic images obtained by the image sensor in video call or image capture mode. The processed image frames can then be displayed on the display module 151 or stored in the memory 170.

[0114] Furthermore, the microphone 122 processes the external sound signal into electronic voice data. The processed voice data can then be utilized in various ways depending on the function being executed (or the application being executed) in the AI ​​device 100. The microphone 122 may also include various noise reduction algorithms to remove noise generated during the process of receiving the external sound signal.

[0115] The user input unit 123 can receive information input by the user, and the processor 180 can control the operation of the AI ​​device 100 to correspond to the input information. The user input unit 123 may include one or more mechanical input elements (or mechanical keys, for example, buttons, dome switches, jog wheels, micro switches, etc. located on the front surface and / or rear surface or side of the AI ​​device 100) and touch-sensitive input elements. For example, the touch-sensitive input element can be a virtual key, soft key, or visual key displayed on the touch screen through software processing, or a touch key located at a location other than the touch screen on the mobile terminal.

[0116] The output unit 150 may include at least one of a display module 151, a sound output unit 152, a haptic module 153, and an optical output unit 154. The display module 151 may display (output) information processed in the AI ​​device 100. For example, the display module 151 may display execution screen information of an application running on the AI ​​device 100, or user interface (UI) and graphical user interface (GUI) information in response to the execution screen information.

[0117] The display module 151 may have a sandwich structure or an integrated structure with a touch sensor to implement a touch screen. The touch screen may provide an output interface between the AI ​​device 100 and the user, and serve as the user input unit 123 providing an input interface between the electronic device 100 and the user.

[0118] In addition, the sound output unit 152 can output audio data received from the communication unit 110 or stored in the memory 170 in a call signal reception mode, a call mode, a recording mode, a voice recognition mode, a broadcast reception mode, etc. The sound output unit 152 may include at least one of a receiver, a speaker, and a buzzer.

[0119] In addition, the haptic module 153 can generate various tactile effects that the user can feel. One example of the tactile effect generated by the haptic module 153 is vibration.

[0120] The optical output unit 154 may output a signal indicating event generation using light from the light source of the AI ​​device 100. Examples of events generated in the AI ​​device 100 include message reception, call signal reception, missed call, alarm, schedule notification, email reception, information reception via an application, and the like.

[0121] Next, Figure 5 1 is a perspective view showing an artificial intelligence range hood 100 and a stove 550 according to an embodiment of the present disclosure. Figure 5The artificial intelligence range hood 100 is located on a tabletop 550 a (eg, +z-axis direction) of a stovetop 550 , and includes a main body 50 , a camera 510 , a display 520 , a plurality of microphones 530 , and a plurality of speakers 540 .

[0122] In one embodiment, the display 520 is located on the front surface (eg, +y axis direction) of the main body 50. The display 520 corresponds to Figure 4 The user input unit 123 of the input unit 120 and the display module 151 of the output unit 150 are shown. For example, a touch screen capable of receiving user input and displaying execution screen information of an application running on the artificial intelligence range hood 100 may be implemented.

[0123] In addition, the camera 510 may be located in an area of ​​the bottom surface (eg, -z axis direction) of the body 50. Figure 5 In the embodiment, the area where the camera 510 is located is only one embodiment, and the camera 510 can be set in various positions to capture the tabletop 550a of the stovetop 550 located at the lower end of the artificial intelligence range hood 100, such as the central area of ​​the bottom surface of the artificial intelligence range hood 100. The camera 510 corresponds to Figure 4 The camera 121 of the input unit 120 is shown. Therefore, cooking vessels, cooking utensils, food, and user's hands located in the area of ​​the table top 550a of the cooktop 550 can be photographed to obtain image data, such as still or dynamic images.

[0124] A plurality of microphones 530 may be located on the bottom surface (eg, -z direction) of the body 50. The microphones 530 correspond to Figure 4 The microphone 122 of the input unit 120 is shown. For example, the plurality of microphones 530 may include one or more directional microphones and be disposed at different positions to record various cooking sounds when cooking on the tabletop 550a of the stovetop 550 located at the lower end of the range hood 100.

[0125] Reference Figure 5 , the artificial intelligence range hood 100 may further include a plurality of speakers 540. The speakers 540 correspond to Figure 4 The output unit 120 is shown as a sound output unit 152. For example, cooking sounds recorded during cooking can be output, or voice guidance related to a recipe can be output.

[0126] In addition, the stove 550 located below the artificial intelligence range hood 100 (eg, in the -z axis direction) may include one of an electric stove, a gas stove, and an induction stove. In addition, the stove 550 may include a combination of an electric stove, a gas stove, and an induction stove.

[0127] As shown, the cooktop 550 located below the AI ​​range hood 100 may include multiple cooking zones 551. If the cooktop 550 is a gas cooking appliance, the multiple cooking zones 551 may burn the gas supplied to it. Furthermore, if the cooktop 550 is an electric range, the multiple cooking zones 551 may generate heat using the electricity supplied to it, and use the generated heat to heat the tabletop 550a. Furthermore, if the cooktop 550 is an induction heating cooking appliance, the multiple cooking zones 551 may generate an induced current using the electricity supplied to it, and use the generated induced current to directly heat the cooking vessel or the food within the cooking vessel.

[0128] In addition, the oven 560 located at the lower end of the stove 500 can generate high-temperature heat using gas or electricity and cook the food ingredients in the cooking cavity through air convection. The door 61 located in front of the main body 60 of the oven stove 500 (including the stove 550 and the oven 560) can rotate based on the hinge axis. The operation panel and display module for controlling or inputting the operation and / or function of the oven 560 can be located at the upper end of the door 61. In addition, the artificial intelligence range hood 100 can absorb smoke, cooking steam and / or cooking odors generated during the cooking process of the cooking utensils placed on the table panel 550a of the stove 550.

[0129] In the following, reference will be made to Figures 6a to 6c Describes the basic operation of the AI ​​range hood. Figure 5 Specifically, Figure 6a is an operation flow chart in which the processor of the artificial intelligence range hood 100 controls the camera 510 and the display 520 provided in the artificial intelligence range hood 100, and Figure 6b The display image includes a countertop area of ​​the cooktop 550 captured by the camera 510 .

[0130] Reference Figure 6a , the processor of the artificial intelligence range hood 100 can control the camera 510 (S601) to obtain image data including at least one of multiple cooking areas, cooking utensils, or surrounding objects located in the countertop area of ​​the stove 550 (S602). The camera 510 can capture multiple cooking areas, cooking utensils, cooking utensils, food, and the user's hands located in the countertop area of ​​the stove 550 to obtain image data, such as static or dynamic images. Here, food can refer to objects to be cooked and can be located inside or outside the cooking vessel. In other words, food can include ingredients located in the countertop area of ​​the stove.

[0131] In addition, the camera 510 may include at least one of an RGB camera that generates RGB image data, an IR camera that generates IR image data, a depth camera (or 3D camera) that generates depth image data, or an RGB-D camera that generates RGB-D image data. The depth camera may refer to a time-of-flight (ToF) camera.

[0132] Subsequently, the processor of the AI ​​range hood 100 may use the object recognition model stored in the memory to generate object recognition information for the objects included in the image data (S603). The object recognition model may be received from the AI ​​server via the communication unit of the AI ​​range hood 100 and stored in the memory. In addition, the object recognition model, including an artificial neural network, may be trained using a deep learning algorithm or a machine learning algorithm.

[0133] The processor then uses the object recognition model to determine whether the image data includes cooking utensils, cooking utensils, food, a user's hand, etc., and, if so, in which area of ​​the cooktop 550a the cooking utensils, cooking utensils, food, or a user's hand are located. In other words, the processor can use the object recognition model to generate object recognition information about the objects included in the image data, and the object recognition information can include object identification information and object location information. The object recognition information can include at least one of the cooking utensils, cooking utensils, food, a user's hand, etc. Furthermore, the object location information can include coordinate information (x, y) indicating the location of the object located on the cooktop 550.

[0134] The processor then determines whether the user has started cooking (S604). More specifically, based on the object recognition information, the processor can identify at least one of one or more cooking zones, cooking utensils, or food being cooked located in the countertop area of ​​the cooktop 550 to determine whether the user has started cooking. That is, upon identifying one or more cooking zones determined to be cooking, or one or more cooking utensils or food located in the countertop area of ​​the cooktop 550, the processor can determine that cooking has started. Here, the cooking zone determined to be cooking can refer to a cooking zone among the multiple cooking zones located on the countertop 550a of the cooktop 550 that is heated by the energy source supplied thereto.

[0135] When the processor determines that the user has not started cooking food ("No" in S604), the processor displays a preview image 611 of the tabletop area of ​​the stovetop 550 on the display 520, such as Figure 6b(a), and returns to step S602 of acquiring image data. The preview image 611 may include an image generated by photographing the countertop area of ​​the stove 550 by the camera 510, and a user interface for receiving photographing conditions, such as a time control button 613a, a video shooting button 613b, and a photo shooting button 613c. Figure 6b As shown in (a).

[0136] When the processor determines that the user has started cooking food ("Yes" in S604), that is, when at least one of one or more cooking zones, cooking utensils, or food being cooked is identified in the countertop area of ​​the cooktop 550, the processor can set the user's area of ​​interest and receive user input including shooting conditions (such as time-lapse shooting speed and shooting mode) (S605).

[0137] Here, the user's area of ​​interest may be an area of ​​the stovetop surface area including at least one of one or more cooking zones, cooking utensils, or food identified as being cooked. Figure 6b (b) An area of ​​the cooktop surface area, including at least one of one or more cooking zones, cooking utensils, or food being cooked, may be displayed as areas 612a to 612c of a specific color. Furthermore, a user interface for receiving shooting conditions, such as a time control button 613a, a video capture button 613b, and a photo capture button 613c, may also be displayed. Here, the time control button 613a is a user interface for setting the time-lapse shooting speed, and the video capture button 613b and the photo capture button 613c are user interfaces for selecting a shooting mode.

[0138] In addition, the user can select one of the areas 612a to 612c that are determined to be cooking and displayed in a specific color. Figure 6b As shown in (b), the user-selected area 614a can be distinguished from the areas 612a to 612c that are determined to be cooking and displayed in a specific color and displayed in another specific color (e.g., highlighted). In addition, the time control button 613a can be used to set the time-lapse shooting speed, and the video shooting button 613b or the photo shooting button 613c can be used to record or shoot the user-selected area 614a. Hereinafter, for convenience of description, the area 614a selected by the user from the areas 612a to 612c of the tabletop area of ​​the stove 550 is referred to as the user region of interest (ROI).

[0139] Next, return to the reference Figure 6aThe processor of the artificial intelligence range hood 100 can control the camera 510 to capture the user's area of ​​interest based on user input including shooting conditions (such as time-lapse shooting speed and shooting mode) (S606). In addition, the processor can control the still or dynamic image generated by the camera 510 capturing the user's area of ​​interest to be displayed on the display 520.

[0140] Specifically, an area including at least one of a cooking area, cooking utensils, or food selected by a user, that is, only the area of ​​interest to the user, may be photographed in a cropped manner, or the recording time may be displayed only in the area of ​​interest to the user, while the remaining area may be processed in a blurred or darkened manner. Figure 6b (b) When the user selects one of the areas 612a to 612c that are determined to be cooking and displayed in a specific color 614a and presses a photo shooting button 613c, the user interest area 614a may be processed and displayed in a cropped screen 614b.

[0141] In addition, in Figure 6b In an alternative embodiment shown in (c), the cooking time may be displayed only in the user interest area 614c for video recording, and the remaining areas may be processed and displayed in a blurred or darkened manner, but the present disclosure is not limited thereto.

[0142] In the following, reference will be made to Figure 6c A method for generating object recognition information using an object recognition model is described. Specifically, Figure 6c (a) shows a method of generating object recognition information about an object included in image data using an object recognition model, and Figure 6c (b) shows an embodiment of training an object recognition model.

[0143] Reference Figure 6c (a) The processor of the AI ​​range hood 100 uses the object recognition model 620 to recognize one or more objects included in the image data 610 and generates object recognition information 630 about the recognized objects. The object recognition information 630 may include object recognition information 631 that identifies the type of the object and object location information 632 that indicates the location of the object.

[0144] The object recognition model 620 may be trained using a plurality of image data including objects and learning data annotated with object recognition information about the objects included in the image data. Figure 6c(b) shows an embodiment of training the object recognition model 620 using training image data 610a to 610d and training object recognition information 631a to 631d annotated on the training image data 610a to 610d. Here, the training object recognition information 631a to 631d may include at least one of cooking utensils, cooking utensils, food, and a user's hand.

[0145] In the following, Figure 7a and Figure 7b The method for automatically adjusting the time-lapse shooting speed in the artificial intelligence range hood 100 is shown. Specifically, Figure 7a is a flowchart showing an operation of automatically adjusting the time-lapse shooting speed according to an embodiment of the present disclosure, Figure 7b Includes a display showing that the time-lapse speed is being automatically adjusted.

[0146] Reference Figure 7a The processor of the artificial intelligence range hood 100 can control the camera 510 (S701) to capture the user's area of ​​interest based on the user input, thereby obtaining image data (S702). The user input may include a time-lapse shooting speed and a shooting mode.

[0147] Subsequently, the processor of the AI ​​range hood 100 may use the object recognition model stored in the memory to generate object recognition information for the objects included in the image data (S703). The processor may use the object recognition model to determine whether the image data includes cooking utensils, cooking utensils, food, a user's hand, etc., and where they are located. In other words, the processor may use the object recognition model to generate object recognition information about the objects included in the image data, and the object recognition information may include object identification information and object location information.

[0148] Subsequently, the processor of the artificial intelligence range hood 100 can respectively calculate the proportion of the food area and the proportion of the non-food area in the user's area of ​​interest (S704). Specifically, the processor can identify the food and objects other than food (for example, cooking utensils or user hands) located in the user's area of ​​interest based on the object recognition information. In addition, based on the object position information, the positions of the identified food and objects other than food can be respectively identified. In addition, in the user's area of ​​interest, the processor can respectively calculate the proportion of the food area (i.e., the area occupied by food) and the non-food area (i.e., the area occupied by objects other than food).

[0149] When the proportion of the food area in the user's area of ​​interest is greater than a preset first threshold ("Yes" in S705), the processor can increase the time-lapse shooting speed (S706) to shoot. Figure 7bAs shown in (a), when the calculated proportion of food area 711b within user's area of ​​interest 711a is 70% or greater, the processor can determine that there is little change in the cooking process. For example, the processor can determine that the cooking process is almost complete. In this case, the time-lapse shooting speed can be increased to 5x speed.

[0150] On the other hand, when the proportion of the food area in the user's area of ​​interest is less than the first preset threshold and the proportion of the non-food area is greater than the second preset threshold ("No" in S707), the processor can control the camera 510 to reduce the time-lapse shooting speed (S708). Figure 7b As shown in (b), when the area 712b occupied by food within the user's area of ​​interest 712a accounts for 70% or less, and the area occupied by objects other than food (such as cooking utensils area 712c or user's hand area 712d) accounts for 50% or more, the processor can determine that there is a significant change in the cooking process. In this case, the processor can control the camera 510 to automatically reduce the time-lapse shooting speed to 1x speed for shooting.

[0151] The aforementioned first threshold value may refer to a reference value for determining whether the cooking process has experienced a large or small change based on the percentage of the area occupied by the food within the user's region of interest. For example, when the percentage of the area occupied by the food within the user's region of interest is greater than the first threshold value (70%), the processor may determine that there is a small change in the cooking process. Specifically, the processor may determine that the cooking process is nearing completion.

[0152] In addition, the second threshold value may refer to a reference value for determining whether the cooking process has a large or small change based on the proportion of the area occupied by objects other than food in the user's area of ​​interest. For example, when the proportion of the area occupied by objects other than food (e.g., cooking utensils or the user's hands) in the user's area of ​​interest is greater than the second threshold value (50%), the processor may determine that there is a large change in the cooking process.

[0153] Next, we will refer to Figures 8a to 8c The method of providing a color correction filter based on the food type in the artificial intelligence range hood 100 is described. Specifically, Figure 8a is an operational flowchart illustrating controlling the generation and application of a color correction filter list based on the type of food according to an embodiment of the present disclosure, Figure 8b Includes a display that provides and applies recommended filters based on the type of food.

[0154] Reference Figure 8a, the processor of the artificial intelligence range hood 100 can control the camera 510 to shoot the user's area of ​​interest (S801) according to the user input, thereby acquiring image data (S802). Subsequently, the processor can use the object recognition model stored in the memory to generate object recognition information about the objects included in the image data, and determine whether the object recognition information included in the object recognition information includes food. When the result of the determination is that food is included in the object recognition information, the food recognition model stored in the memory can be used to further generate food recognition information about the food (S803). Specifically, the processor can use the food recognition model to extract the food area included in the image data to generate food recognition information for identifying the type of food. For example, the food recognition information may include at least one dish name (such as kimchi soup, pizza, steak, and ginseng chicken soup).

[0155] The food recognition model can be received from the AI ​​server through the communication unit of the artificial intelligence range hood 100 and stored in the memory. In addition, the food recognition model including the artificial neural network can be trained using a deep learning algorithm or a machine learning algorithm.

[0156] Subsequently, the processor of the artificial intelligence range hood 100 can generate a color correction filter list based on the generated food recognition information and provide the color correction filter list to the user (S804). The color correction filter list provided to correct the color of the food area can include one or more basic filters and recommended filters according to the food type.

[0157] Subsequently, the processor may receive user input through the display 520, the user input also including shooting conditions such as time-lapse shooting speed, shooting mode, and a color correction filter from the color correction filter list (S805). Subsequently, the processor may control the camera 510 to shoot the user's area of ​​interest based on the received user input (S806). In other words, the color correction filter received as user input may be applied to the food area within the user's area of ​​interest and then shot.

[0158] In addition, the color correction filter selected by the user can be applied to the food area, photographed and stored in the memory of the artificial intelligence range hood 100, and can be edited by the user through the gallery application installed in the artificial intelligence range hood.

[0159] like Figure 8bAs shown, when a user enters an application related to photo and video editing (e.g., a gallery application) installed in the artificial intelligence range hood 100, the processor of the artificial intelligence range hood 100 may execute the gallery application (S811) to extract a still or dynamic image related to food stored in the memory and display the extracted still or dynamic image on the display 520 (S812). When the user selects one of the still or dynamic images displayed on the display 520, the processor may obtain image data from the selected image (S813).

[0160] The processor may then generate food recognition information about the food using the food recognition model stored in the memory (S814). Specifically, the processor may extract the food region included in the image data using the food recognition model to generate food recognition information for recognizing the type of food.

[0161] Subsequently, the processor of the artificial intelligence range hood 100 can generate a color correction filter list based on the generated food recognition information and display the color correction filter list on the display (S815). The color correction filter list provided for correcting the color of food can include one or more basic filters and recommended filters according to the food type.

[0162] Then, when the user selects a color correction filter from the color correction filter list (S816), the processor may control the selected color correction filter to be applied to the food area of ​​the selected image for editing (S817). In addition, the photo or video taken with the color correction filter applied may be sent to and stored in the communication unit of the artificial intelligence range hood 100 through the communication unit of the artificial intelligence range hood 100. Figure 3 The mobile electronic device 100d is shown.

[0163] It can then be edited by the user and stored in a gallery application running on the mobile electronic device 100d. Specifically, the mobile electronic device 100d can receive a food recognition model from the AI ​​server and store the food recognition model in the memory of the mobile electronic device 100d. When the user executes the gallery application installed in the mobile electronic device 100d to select a food-related image, the food recognition model can be used to generate food identification information. In addition, based on the food type identification information included in the food identification information, a filter list for color correction can be provided, and the color correction filter selected by the user can be applied to the food area of ​​the food-related image for editing.

[0164] Next, Figure 8c(a) shows a process in which a list of one or more color correction filters is displayed on the display 520 of the artificial intelligence range hood 100 and a user selects and applies a filter. First, an area of ​​the stove top area including at least one of one or more cooking zones, cooking utensils, or food being identified as being cooked can be displayed as areas 812a to 812c of a specific color. When the user selects one of the areas 812a to 812c that are identified as being cooked and displayed in a specific color, the selected area 814 can be distinguished from the cooking zone areas 812a to 812c identified as being cooked and displayed in another specific color (e.g., highlighted, etc.).

[0165] Subsequently, the processor of the artificial intelligence range hood 100 can generate a list of color correction filters 813a to 813c based on the food recognition information generated using the food recognition model, and provide or display them to the user. When the user selects one of the color correction filter lists 813a to 813c, the selected color filter can be applied to the food area in the user's area of ​​interest 814. In addition, the processor can control the user's area of ​​interest to be processed and displayed in a cropped screen 815 in an area of ​​the display 520. The food area within the user's area of ​​interest 815 displayed on the display 520 can include an image with the user's selected color correction filter applied.

[0166] Figure 8c (b) shows the process of editing food-related photos or videos by using the gallery application running on the mobile electronic device 100d. A similar process can be performed on the gallery application installed in the artificial intelligence range hood 100. Figure 8c (b) The user may execute a gallery application running on the mobile electronic device 100d. When a food-related photo or video 821 is selected, a user interface including an edit button 822a and a delete button 822b may be displayed.

[0167] When the user selects the edit button 822a, food recognition information may be generated using a food recognition model stored in the memory, and a color correction filter list 823a to 823c may be displayed based on the food recognition information. When the user selects one of the color correction filter lists 823a to 823c, the selected color correction filter may be applied to the food area of ​​the food-related image 821 for editing.

[0168] In the following, reference will be made to Figure 8d Describes a method for generating food recognition information using a food recognition model. Figure 8d (a) The processor of the AI ​​range hood 100 generates food recognition information 830 for recognizing the type of food included in the image data 810 using the food recognition model 820 .

[0169] The food recognition model 820 may be trained using image data 810 including food regions and learning data labeled with food recognition information 830 about the food included in the image data 810. For example, Figure 8d (b) illustrates an embodiment of training a food recognition model 820 using training image data 810a to 810d and training food recognition information 831a to 832d annotated on the training image data 810a to 810d. Here, the training food recognition information 831a to 832d may include at least one dish name (e.g., kimchi jjigae, pizza, steak, and samgyetang). When receiving image data containing a food region captured by a camera, food recognition information corresponding to the relevant food type is generated.

[0170] In the following, reference will be made to Figures 9a to 9c Describes a method for recognizing user hand movements in an AI range hood to zoom in or out of the cooking screen or automatically adjust the speed of time-lapse photography. Figure 9a , the processor of the artificial intelligence range hood 100 can control the camera 510 (S901) to obtain image data including the user's area of ​​interest (S902). Subsequently, the processor can use the motion recognition model stored in the memory to extract the user's hand area from the obtained image data to generate motion recognition information (S903).

[0171] A deep learning algorithm or a machine learning algorithm may be used to train a motion recognition model including an artificial neural network. In addition, the motion recognition information for recognizing the user's hand motion may include at least one of motion recognition start, thumb up, thumb down, zoom in, zoom out, and OK.

[0172] Subsequently, the processor may determine whether the user has made a gesture indicating a motion recognition operation within the user's area of ​​interest based on the generated motion recognition information (S904). When the processor determines that the gesture is a motion recognition start operation ("Yes" in S904), the processor changes the motion recognition mode to an on state (S905) and then returns to acquiring image data (S902).

[0173] When the processor determines that the gesture is not a motion recognition start operation ("No" in S904), the processor can determine whether the motion recognition information is one of thumbs up, thumbs down, zoom in, zoom out, and OK, and whether the motion recognition mode is in the on state. When the motion recognition information is one of thumbs up, thumbs down, zoom in, zoom out, and OK, and the motion recognition mode is in the on state ("Yes" in S906), the operation can be controlled based on the motion recognition information (S907), and the motion recognition mode can be changed to the off state (S908).

[0174] When the motion recognition information is not thumbs up, thumbs down, zoom in, zoom out, or OK, or the motion recognition mode is not in the on state (No in S906 ), the processor returns to acquiring image data ( S902 ).

[0175] Next, Figure 9b and Figure 9c An embodiment of recognizing the user's hand motion to automatically adjust the time-lapse shooting speed or zoom in or out of the cooking screen is shown. Figure 9b (a) When the user makes a gesture 912a indicating the start of motion recognition within the user's area of ​​interest, the processor of the artificial intelligence range hood 100 can recognize that the user's hand motion is a motion recognition start operation based on the generated motion recognition information, and change the motion recognition mode to the on state.

[0176] When the user makes a thumbs-up gesture 912b within the user's area of ​​interest, the processor can identify the user's hand motion as a thumbs-up gesture based on the generated motion recognition information. In addition, in response to the thumbs-up gesture, the processor can control the camera 510 to increase the time-lapse shooting speed by a certain preset range to shoot, and change the motion recognition mode to off.

[0177] When the user makes a gesture 912c indicating the start of motion recognition within the user's area of ​​interest, the motion recognition mode is changed back to the on state, and then when the user makes a gesture 912d indicating OK within the user's area of ​​interest, the motion recognition model can be used to recognize the gesture indicating OK to stop the time-lapse shooting speed adjustment and change the motion recognition mode back to the off state.

[0178] When the time-lapse shooting speed reaches a preset maximum or minimum value during time-lapse shooting speed adjustment, or when no gesture indicating the start of motion recognition, thumbs up / down, or OK is recognized within a subsequent preset time period (for example, within 10 seconds), the time-lapse shooting speed adjustment can be automatically stopped.

[0179] In addition, if Figure 9bAs shown in (b), when the user makes a gesture 913a indicating the start of motion recognition within the user's area of ​​interest, the motion recognition mode can be changed to the on state, and then when the user makes a thumbs-down gesture within the user's area of ​​interest, the motion recognition model can be used to recognize the thumbs-down gesture. The processor can control the camera 510 to reduce the time-lapse shooting speed by a certain preset range for shooting, and change the motion recognition mode to the off state. Then, when no gesture indicating the start of motion recognition, thumbs up / down, or OK is recognized within a preset time period (for example, within 10 seconds), the time-lapse shooting speed adjustment can be automatically stopped, thereby continuing to shoot with the currently adjusted shooting speed value.

[0180] Furthermore, according to an embodiment of the present disclosure, the processor of the AI ​​range hood 100 can determine whether the user has made gestures in a preset sequence, thereby controlling the operation of the camera 510 based on the preset sequence of hand movements. For example, when the user sequentially makes gestures indicating the start of motion recognition, zooming in, and zooming out, the camera 510 can be controlled to zoom in on the screen displaying the user's area of ​​interest at a preset ratio for photographing. Furthermore, when the user sequentially makes gestures indicating the start of motion recognition, zooming in, and zooming out, the camera 510 can be controlled to zoom out on the screen displaying the user's area of ​​interest at a preset ratio for photographing.

[0181] like Figure 9c As shown in FIG. 1 , when a user makes a gesture 914a indicating the start of motion recognition within the user's area of ​​interest, the motion recognition mode may be turned on. Then, when the user subsequently makes a gesture 914b indicating zooming out within the user's area of ​​interest, the motion recognition model may be used to recognize the gesture indicating zooming out, thereby changing the zoom-in / zoom-out motion recognition mode to the on state. When the user makes a gesture 914c indicating zooming in within the user's area of ​​interest, it may be checked whether the zoom-in / zoom-out motion recognition mode is in the on state. If the recognition mode is in the on state, the processor may control the camera 510 to zoom in on the cooking screen by a preset ratio for shooting in response to the gesture 914c indicating zooming in, and change the motion recognition mode and the zoom-in / zoom-out motion recognition mode to the off state.

[0182] In addition, reference Figure 9c(b) When the user makes a gesture 915a indicating the start of motion recognition within the user's area of ​​interest, the motion recognition mode may be changed to an on state. Then, when the user subsequently makes a gesture 915b indicating zooming in within the user's area of ​​interest, the motion recognition model may be used to recognize a gesture indicating zooming out, so as to change the zoom-in / zoom-out motion recognition mode to an on state. When the user makes a gesture 915c indicating zooming out within the user's area of ​​interest, it may be determined whether the zoom-in / zoom-out motion recognition mode is in an on state. If the mode is in an on state, the processor may control the camera 510 to zoom out the screen by a preset ratio for shooting in response to the gesture 915c indicating zooming in, and change the motion recognition mode and the zoom-in / zoom-out motion recognition mode to an off state.

[0183] Next, we will refer to Figure 9d Describes a method for generating action recognition information using an action recognition model. Specifically, Figure 9d (a) shows a method for generating motion recognition information about a user's gesture included in image data using a motion recognition model, and Figure 9d (b) shows an embodiment of training an action recognition model.

[0184] Reference Figure 9d (a) The processor of the AI ​​range hood 100 uses the motion recognition model 920 to recognize the hand region included in the image data 910 to generate motion recognition information 930. The motion recognition model 920 can be trained using the image data 910 including the hand region and the learning data annotated with the motion recognition information 930 regarding the hand region included in the image data 910.

[0185] For example, Figure 9d (b) shows an embodiment of training the motion recognition model 920 using training image data 910a to 910f and training motion recognition information 931a to 931f annotated on the training image data 910a to 910f. Here, the training motion recognition information 931a to 931f may include at least one of motion recognition start, OK, thumbs up, thumbs down, and zoom in / out. Then, when image data including an image of a user's hand motion captured by the camera 510 is received, motion recognition information corresponding to the relevant gesture is generated.

[0186] In the following, reference will be made to Figure 10a and Figure 10b The method of using a directional microphone to record the sound generated during the cooking process of food in the artificial intelligence range hood 100 is described. Figure 10a, the processor of the artificial intelligence range hood 100 can control the camera 510 (S1001) to capture the countertop area of ​​the stovetop 550 and obtain image data (S1002). Subsequently, object recognition information about the multiple cooking areas, cooking utensils, cooking utensils, food, and user hands included in the image data can be generated (S1003).

[0187] The processor may then determine whether the user has started cooking food (S1004). When the processor determines that the user has started cooking food ("Yes" in S1004), one of the areas of the countertop area of ​​the cooktop 550 that includes at least one of one or more cooking zones, cooking utensils, or food being cooked as identified by the display may be received as user input and set as a user area of ​​interest (S1005).

[0188] When the processor determines that the user's area of ​​interest has been set, the processor may move the audio focus of the plurality of microphones 530 provided in the artificial intelligence range hood 100 to the user's area of ​​interest (S1006). That is, the audio focus of the microphone 530 may be moved to the user's area of ​​interest including the cooking zone identified as being cooked, thereby mainly recording cooking-related sounds generated from the cooking zone of interest to the user.

[0189] In addition, the processor can control the camera 510 to further receive shooting conditions including time-lapse shooting speed and shooting mode as user input to shoot the user's area of ​​interest. Cooking-related sounds focused and recorded in the user's area of ​​interest can also be included in the image captured by the camera 510 and stored in the memory.

[0190] Figure 10b An embodiment is shown in which the audio focus of the plurality of microphones 530 is moved to the user's area of ​​interest based on the user's area of ​​interest being set. Figure 10b (a) and Figure 10b (b) A plurality of directional microphones 530 for receiving audio signals generated from the tabletop 550 a of the cooktop 550 may be located at a lower end of the main body 50 of the artificial intelligence range hood 100 .

[0191] When cooking is started on the tabletop 550a of the stovetop 550 and the user interest areas 1010a and 1010b are set, the audio focus of the microphone 530 can be moved to the user interest areas 1010a to 1010b. That is, the audio focus of the microphone 530 can be adjusted to mainly receive sounds generated from the user interest areas 1010a to 1010b.

[0192] In the following, reference will be made to Figure 11a and Figure 11bA method for analyzing a cooking video shot in an AI range hood 100 to generate highlight clips is described. First, Figure 11a A method for identifying a portion where a significant change occurs during a cooking process in a cooking video captured by the camera 510 in the artificial intelligence range hood 100 is shown.

[0193] Reference Figure 11a , the processor of the artificial intelligence range hood 100 can control the camera 510 (S1101) to obtain image data including the user's area of ​​interest (S1102). The image data may include first image data for a current frame captured and received by the camera 510 and second image data for a previous frame received before the current frame.

[0194] Subsequently, the processor of the artificial intelligence range hood 100 may analyze the acquired first image data and second image data to extract features of the first image data and the second image data, including at least one of color, edge, histogram, correlation, and motion vector of optical flow (S1103). The processor may then calculate the similarity between the current frame and the previous frame based on the features of the first image data and the second image data (S1104), and determine whether the similarity is higher than a preset threshold similarity (S1105).

[0195] When the processor determines that the similarity between the current frame and the previous frame is greater than the preset threshold similarity ("Yes" in S1105), the processor determines that the current frame has fewer change points than the previous frame, and then returns to acquiring image data (S1102).

[0196] When the processor determines that the similarity between the current frame and the previous frame is less than the preset threshold similarity ("No" in S1105), the processor determines that the current frame has more change points than the previous frame, and then determines the current frame as a change point frame (S1106) and stores it in the memory.

[0197] In addition, a threshold similarity can be determined based on the first object recognition information and the second object recognition information generated from the first image data and the second image data of the current frame and the previous frame, where the threshold similarity is used to determine the similarity calculation between the current frame and the previous frame. Specifically, the processor of the artificial intelligence range hood 100 can use the object recognition model to generate the first object recognition information and the second object recognition information about the objects included in the first image data and the second image data (S1107).

[0198] Subsequently, the processor may calculate the percentage of the food region in the user's region of interest based on the first object recognition information and the second object recognition information, and increase or decrease the threshold similarity by a predetermined range. The threshold similarity is a reference for calculating the similarity between the current frame and the previous frame based on the percentage of the food region (S1108). For example, when the percentage of the food region in the user's region of interest is 70% or greater, it is likely that the one or more ingredients for the food are already mostly contained in the cooking vessel and are thus in the final stages of cooking (e.g., stewing or baking). In other words, this is most likely a highlight in the cooking process.

[0199] The threshold similarity can be lowered by a preset first range to determine a change point frame, thereby increasing the probability of generating an important process as a highlight segment in the cooking process. In addition, when the food area accounts for 70% or less of the user's area of ​​interest, the processor determines that the change in the cooking process is small and can increase the threshold similarity by a preset second range to determine a change point frame. The first range can be set to be smaller than the second range, but the present disclosure is not limited to this.

[0200] Next, Figure 11b A method for generating highlight segments using one or more change point frames stored in a memory after video capture is completed is shown. Figure 11b When the video shooting is completed, the processor of the artificial intelligence range hood 100 can display a popup message on the display, asking the user whether to generate a highlight clip (S111).

[0201] When a user requests the generation of a highlight segment (S1112), the processor may extract one or more change point frames stored in the memory. The processor may then generate a change point frame list including the extracted one or more change point frames and display the list on the display 520 (S1113). The processor may then receive the start frame and end frame in the change point frame list displayed on the display as user input (S1114) to generate a highlight segment (S1115).

[0202] In the following, reference will be made to Figure 12a and Figure 12b The present invention describes a method for providing recommended recipes and information about ingredients used in the recipes using an open API in the artificial intelligence range hood 100. Figure 12a The processor of the artificial intelligence range hood 100 can use the open API provided by the Ministry of Food and Drug Safety to collect recipe information from the cooking food recipe database on the server of the Ministry of Food and Drug Safety. Subsequently, the processor can generate a recipe database based on the collected recipe information (S1201) and store the generated recipe database in the memory.

[0203] In the following, reference will be made to Figure 12b An embodiment of using an open API to collect recipe information from the Ministry of Food and Drug Safety database to generate a recipe database is described. Figure 12b The processor of the artificial intelligence range hood 100 may allow a cooking name 1251 to be input through an open API provided by the Ministry of Food and Drug Safety to collect recipe information 1270, which includes at least one of a cooking method 1252, ingredient information 1253, nutritional information 1254a to 1254c, calories 1255, or one or more images 1256a to 1256b (specifically showing the cooking method for each cooking step) and text 1257a to 1257b (describing the cooking method). The cooking method 1252 included in the recipe information 1270 may include at least one of baking, stewing, steaming, roasting, frying, etc., and the nutritional information 1254a to 1254c may include at least one of carbohydrate information 1254a, fat information 1254b, protein information 1254c, etc.

[0204] The processor can then Figure 12c The collected recipe information 1270 is shown as generating a recipe database 1280, and the recipe database 1280 is stored in the memory of the artificial intelligence range hood 100. The generated recipe database 1280 includes a dish name 1291, a cooking method 1292, ingredient information 1293, one or more nutritional information 1294a to 1294c, a calorie 1295, a plurality of images 1296a to 1296b showing the cooking method, and a plurality of texts 1297a to 1297b describing the cooking method.

[0205] In addition, the processor of the artificial intelligence range hood 100 may generate a database entry 1298 related to a cooking vessel corresponding to the cooking method 1252 included in the collected recipe information 1270, and further include it in the recipe database 1280. For example, in response to the cooking method of "steaming", a cooking vessel entry called "steamer" may be generated and further included in the recipe database 1280. Similarly, among the cooking methods included in the recipe information 1270, baking, stewing, steaming, roasting, and frying may correspond to one or more cooking vessel entries 1298 categorized as a frying pan, a soup pot, a steamer, and a cooker, respectively, and stored in the recipe database 1280.

[0206] The recipe database 1280 may include a first database set (a), a second database set (b), and a third database set (c). Here, the first database set (a) to the third database set (c) may include entries of dish names, cooking methods, ingredient information, one or more nutritional information, calories, one or more images showing cooking methods, and one or more texts describing cooking methods, respectively, related to the first to third recipes.

[0207] Next, return to the reference Figure 12a The processor of the AI ​​range hood 100 may control the camera 510 (S1202) to capture the user's area of ​​interest and acquire image data (S1203). The processor may then utilize a cooking vessel recognition model stored in memory to generate cooking vessel recognition information for the cooking vessel included in the image data (S1204). The cooking vessel recognition information may include at least one of a frying pan, a soup pot, a steamer, and a pot.

[0208] Furthermore, the processor may generate a list of recommended dishes by referring to the generated cooking vessel identification information and a recipe database stored in the memory, and then display the list on the display (S1205). Specifically, the processor may retrieve a cooking vessel entry that matches the cooking vessel identification information from the recipe database to extract one or more dish names corresponding to the retrieved cooking vessel entry. For example, if the cooking vessel identification information is "steamer," the processor may extract one or more dish names corresponding to the cooking vessel "steamer" from the recipe database. The one or more dish names extracted by the aforementioned method may be included in the list of recommended dishes and presented to the user via the display.

[0209] In addition, the processor of the artificial intelligence range hood 100 can generate a list of recommended dishes based on weather information collected from the meteorological bureau's server or at least one of the search results collected by the user by directly searching the network through the display of the artificial intelligence range hood 100, so as to further display the list of recommended dishes on the display.

[0210] When the user selects a dish from the list of recommended dishes (S1206), the processor may extract the recipe corresponding to the name of the selected dish from the recipe database (S1207). Subsequently, the processor may generate a recipe list for each cooking step by referring to the extracted recipe, and display the recipe list on the display (S1208). Specifically, the recipe list for each cooking step may be generated based on multiple images showing the cooking method and multiple texts describing the cooking method included in the recipe database.

[0211] For example, reference Figure 12cThe recipe database set (a) corresponding to the dish name "Steamed Snails with Cabbage and Soybean Paste" may include at least one of a first image 1296a showing a cooking method performed in a first cooking step and a first text 1297a describing the cooking method performed in the first cooking step. Furthermore, the recipe database set may also include at least one of a second image 1296b showing a cooking method performed in a second cooking step and a second text 1297b describing the cooking method performed in the second cooking step.

[0212] The processor can generate a first cooking step recipe including a first image 1296a or a first text 1297a and a second cooking step recipe including a second image 1296b and a second text 1297b, and generate recipes for each cooking step (including first and second cooking step recipes) in the form of a list and provide the list to the user.

[0213] Return Reference Figure 12a , when the user selects a cooking step from the recipe list for each cooking step displayed on the display 520 (S1209), the processor of the artificial intelligence range hood 100 may display text describing the cooking method used in the selected cooking step on the display (S1210).

[0214] In addition, the text describing the cooking method can be converted into speech to be output through one or more speakers provided in the artificial intelligence range hood 100 (S1211). Specifically, the processor of the artificial intelligence range hood 100 can convert the text data into speech data using a text-to-speech (TTS) engine that converts text strings into speech. The TTS engine can be configured with an artificial neural network trained according to a machine learning algorithm. In addition, it can be trained by a learning processor, or by a learning processor of an AI server, or by its distributed processing.

[0215] In addition, the processor of the artificial intelligence range hood 100 can store one or more TTS engines in the memory. For example, a TTS engine corresponding to a news style of a male / female announcer, a TTS engine corresponding to a storytelling style of a voice actor, and a TTS engine corresponding to an entertainment style of an entertainer can be stored therein.

[0216] Therefore, the processor can convert the text describing the cooking method into voices corresponding to various styles using one of the multiple TTS engines stored in the memory through the aforementioned method, and output the voices through the multiple speakers provided in the artificial intelligence range hood 100. In addition, the voices describing the cooking method output through the multiple speakers can be included in the image captured by the camera 510 to be stored in the memory.

[0217] Subsequently, the processor may further generate additional information about the selected dish and one or more ingredients used in the dish, and display the additional information on the display (S1212). The additional information about the dish and the ingredients used in the dish may be collected from a recipe database stored in the memory, a server of the Ministry of Food and Drug Safety, or an external network server, and may include news related to the dish and the ingredients, price information, nutritional information, allergy information, etc., but the present disclosure is not limited thereto.

[0218] Figure 12d An embodiment is shown in which the artificial intelligence range hood 100 displays a list of recommended dishes based on recipe information collected using an open API on a display and provides information about the dish selected by the user. In particular, Figure 12d (a) shows an embodiment in which the processor of the artificial intelligence range hood 100 displays a list 1291a of recommended dishes related to cooking utensils located in the user's area of ​​interest on the display 520.

[0219] like Figure 12d As shown in FIG. 1 , the processor may also display a list 1291 b of recommended dishes generated based on weather information collected from a weather bureau server. In addition, a list 1291 c of recommended dishes generated based on search results collected by the user directly searching the network through the display 520 may also be displayed on the display.

[0220] Next, Figure 12d (b) shows an embodiment in which, in response to a user selecting a dish from a list of recommended dishes, recipes related to the selected dish are displayed. Specifically, the recipes for the selected dish may be displayed in the form of a list including an image showing a cooking method for each cooking step and text describing the cooking method.

[0221] For example, in Figure 12d In (a), when the user selects "tteokbokki" from the list of recommended dishes, a recipe corresponding to the dish name "tteokbokki" can be extracted from the recipe database. The recipe corresponding to "tteokbokki" can include an image showing the cooking method for each cooking step and a text describing the cooking method. Specifically, during the process of cooking "tteokbokki", a first image showing the cooking method performed in the first cooking step and a first text describing the cooking method performed in the first cooking step can be included to generate a first cooking step recipe. Similarly, second and third cooking step recipes can be generated and displayed on the display 520, and the second and third cooking step recipes respectively include second and third images showing the cooking method performed in the second and third cooking steps and second and third text describing the cooking method performed in the second and third cooking steps.

[0222] Reference Figure 12d (b), the first to third cooking step recipes 1292a to 1292c generated by the aforementioned method may be displayed in the form of a list on the display 520 and may be scrolled by a user's touch input, but the present disclosure is not limited thereto. Figure 12d (c) shows an embodiment in which, when a user selects a cooking step recipe from a recipe list for each cooking step, text 1294 describing the cooking method included in the selected cooking step recipe is displayed on the display 520. In addition, the text 1294 describing the cooking method can be converted into speech using a TTS engine and output through multiple speakers simultaneously. In this case, when the user selects the video capture button 1295, the speech output through the multiple speakers can be included in a file and stored in the memory, while the image captured and generated by the camera 510 is stored as a file in the memory.

[0223] In addition, additional information related to the dish called "tteokbokki" and one or more ingredients used in "tteokbokki" may be collected and further displayed on the display 520 as thumbnails 1296 or text 1297. For example, news, price information, nutritional information, allergy information, etc. related to the dish called "tteokbokki" or one or more ingredients used in "tteokbokki" may be retrieved from a recipe database stored in a memory, a server of the Ministry of Food and Drug Safety, or an external network server, and displayed on the display 520 as thumbnails 1296 or text 1297.

[0224] Furthermore, additional information related to the food displayed on the display 520 can be generated by identifying the type of food included in the image data obtained by capturing the user's area of ​​interest. Specifically, the processor of the artificial intelligence range hood 100 can use the food recognition model stored in the memory to generate food recognition information, which is used to identify the type of food included in the image data captured and obtained by the camera 510.

[0225] Based on the generated food identification information, related news, price information, nutritional information, allergy information, etc. can be extracted from the recipe database stored in the memory, the server of the Ministry of Food and Drug Safety, or the external network server, and further displayed on the display 520.

[0226] In the following, reference will be made to Figure 13a and Figure 13b The method of identifying one or more ingredients included in food in the artificial intelligence range hood 100 to provide a recommended ingredient list is described. Figure 13a, the processor of the artificial intelligence range hood 100 can execute the camera 510 (S1301) to obtain image data by photographing the user's area of ​​interest (S1302).

[0227] Subsequently, the processor may generate food recognition information for recognizing the food included in the image data using the food recognition model stored in the memory (S1303). In addition, the food recognition model stored in the memory may be used to generate food recognition information for recognizing the type of one or more food ingredients included in the image data (S1304).

[0228] Subsequently, the processor may obtain the ingredient information corresponding to the generated food recognition information from the recipe database stored in the memory (S1305). Figure 12b , "Soybean paste and cabbage roll snails" corresponding to the food identification information can be input into the recipe database 1280 as a dish name to obtain ingredient information including pork (50g), cabbage (5 leaves), leek (30g), rice (100g), onion (50g), zucchini (1 / 2), snails (100g) and soybean paste (30g).

[0229] Return Reference Figure 13a , the processor of the artificial intelligence range hood 100 can compare the ingredient information obtained from the recipe database with the ingredient recognition information for identifying the type of ingredients included in the image data obtained by photographing the user's area of ​​interest, so as to generate a basic ingredient list and a recommended ingredient list (S1306). For example, when the ingredient information obtained from the recipe database includes "pork (50g), cabbage (5 leaves), leek (30g), rice (100g), onion (50g), zucchini (1 / 2), snail (100g), soybean paste (30g)", basic ingredient information including "pork (50g), cabbage (5 leaves), leek (30g), rice (100g), onion (50g), zucchini (1 / 2), snail (100g), soybean paste (30g)" can be generated in sequence.

[0230] In addition, a recommended ingredient list can be generated by changing the order of one or more ingredients included in the basic ingredient list and provided to the user. For example, when the ingredient information obtained from the recipe database includes "pork (50g), cabbage (5 leaves), leek (30g), rice (100g), onion (50g), zucchini (1 / 2), snails (100g), soybean paste (30g)", and the ingredient identification information generated from the image data includes "pork, cabbage leaves, leek and rice", "onion, zucchini, snails and soybean paste" will be identified as the ingredients to be added. In this case, a recommended ingredient list can be generated by arranging the ingredients identified as the ingredients to be added in a higher (higher) order in the basic ingredient list.

[0231] Subsequently, the processor according to an embodiment of the present disclosure may determine whether the current cooking step is a finishing step (S1307) to display a basic ingredient list or a recommended ingredient list on the display. Whether the current cooking step is a finishing step may be determined based on the proportion of ingredients identified as ingredients to be added. Specifically, when the number of ingredients included in the basic ingredient list that are identified as ingredients to be added is less than a predetermined proportion, the processor may determine that the number of ingredients to be added is small. Therefore, by determining that the current cooking step is a finishing step, arranging the ingredients to be added in a higher order, and displaying a recommended ingredient list on the display 520, the user can easily identify the ingredients to be added.

[0232] In addition, whether the current cooking step is a finishing step can be determined based on the proportion of the food area included in the image data obtained by shooting the user's area of ​​interest, but the present disclosure is not limited to this. In addition, one or more ingredients included in the basic ingredient list and the recommended ingredient list generated by the aforementioned method can be displayed in the form of labels (stickers) respectively and moved to the screen displaying the user's area of ​​interest by the user's drag input. In addition, the ingredient label moved to the user's area of ​​interest can be included in the photo or video taken by the camera 510 and stored in the memory.

[0233] Figure 13b An embodiment of displaying a basic ingredient list and a recommended ingredient list on a display is shown. Figure 13b As shown in (a), the processor of the artificial intelligence range hood 100 can generate a basic ingredient list 1311, in which the ingredient information obtained from the recipe database is arranged in sequence to display the generated basic ingredient list on the display.

[0234] In addition, if Figure 13bAs shown in (b), a recommended ingredient list 1323 may be generated and displayed on the display 520. In the recommended ingredient list 1323, the ingredients to be added are arranged in a higher order 1324. The processor may generate guidance information 1325 for guiding the ingredients to be added, and further display the guidance information on the display 520.

[0235] The present disclosure described above can be implemented in a program recording medium as a computer-readable code. Computer-readable media include all types of recording devices that can store computer system-readable data. Examples of such computer-readable media include hard disk drives (HDDs), solid-state drives (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage elements, and the like. In addition, a computer may include a processor of an artificial intelligence range hood.

[0236] The above embodiments are merely exemplary and should not be considered as limiting the present disclosure. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included within the scope of the present invention.

Claims

1. An artificial intelligence range hood, comprising: main body; a camera disposed at a lower end of the body and configured to capture an image of a stove located below the body; a display located on the front surface of the main body; a memory storing an object recognition model; and The processor is configured to: controlling the camera to capture an image of the stove; generating object recognition information using the object recognition model stored in the memory, the object recognition information being used to recognize a cooking object included in a captured image of the cooktop; setting a user interest region of the captured image, the user interest region corresponding to the recognized cooking object specified by the generated object information; controlling the operation of the camera to capture a user's area of ​​interest in the captured image; and The display is controlled to display a user's interested area of ​​the captured image.

2. The artificial intelligence range hood according to claim 1, wherein: The object recognition information includes object identification information and object position information about a cooking object included in the captured image, and The object recognition information specifies at least one of a cooking vessel, food, a cooking utensil, and a user's hand.

3. The artificial intelligence range hood according to claim 1, wherein: The identified cooking objects include at least one of one or more cooking zones, cooking utensils, and food being cooked, and The user's area of ​​interest includes the identified cooking object.

4. The artificial intelligence range hood according to claim 2, wherein: The processor controls the operation of the camera to capture a user's region of interest of the captured image in response to user input including a time-lapse capture speed and a capture mode.

5. The artificial intelligence range hood according to claim 4, wherein: The processor calculates, by using the object recognition information and the object position information, a ratio of an area occupied by food and a ratio of an area occupied by objects other than food within the user's area of ​​interest; When the proportion of the area occupied by the food is greater than a preset first threshold, the processor increases the time-lapse shooting speed; When the proportion of the area occupied by food is less than the first threshold and the proportion of the area occupied by objects other than food is greater than a preset second threshold, the processor reduces the time-lapse shooting speed, and The first threshold is set as a reference value for determining an area occupied by food within the user's area of ​​interest, and the second threshold is set as a reference value for determining an area occupied by objects other than food within the user's area of ​​interest.

6. The artificial intelligence range hood according to claim 2, wherein: The memory also stores a food recognition model, and When food is included in the object recognition information, the processor generates food recognition information using the food recognition model, and the food recognition information recognizes a type of food included in the captured image.

7. The artificial intelligence range hood according to claim 1, wherein: The processor controls the display to display a color correction filter list and applies a user-selected color correction filter to a user-interested region of the captured image.

8. The artificial intelligence range hood according to claim 4, wherein: The memory also stores an action recognition model. When the user's hand is included in the object recognition information, the processor generates the action recognition information using the action recognition model. wherein the motion recognition information recognizes the type of motion of the user's hand included in the captured image, and The action recognition information includes at least one of an action recognition start action, a thumb up action, a thumb down action, a zoom in action, and a zoom out action.

9. The artificial intelligence range hood according to claim 8, wherein: When the motion recognition information is a motion recognition start action, the processor changes the motion recognition mode to an on state; when the motion recognition information is one of a thumbs-up action and a thumbs-down action and the motion recognition mode is in an on state, the processor increases or decreases the time-lapse shooting speed; and when the motion recognition information is one of a zoom-in action and a zoom-out action and the motion recognition mode is in an on state, the processor zooms in or out on the screen displaying the user's area of ​​interest.

10. The artificial intelligence range hood according to claim 1, further comprising: A plurality of directional microphones, located at the lower end of the body, The processor controls the directional microphone to directionally record sounds from cooking objects included in the user's area of ​​interest.

11. The artificial intelligence range hood according to claim 1, wherein: The processor calculates the similarity between the current frame and the previous frame from the captured image; when the calculated similarity is less than a preset threshold similarity, the processor determines the current frame as a first change point frame; and the processor generates a highlight segment including the first change point frame.

12. The artificial intelligence range hood according to claim 11, wherein: The processor calculates a similarity between the current frame and the previous frame based on at least one of a color, an edge, a histogram, a correlation, and a motion vector of optical flows extracted from image data for the current frame and the previous frame.

13. The artificial intelligence range hood according to claim 11, wherein: The processor generates first object recognition information and second object recognition information for the current frame and the previous frame to respectively calculate proportions occupied by the food area within the user's area of ​​interest, and increases or decreases the threshold similarity based on the calculated proportions.

14. The artificial intelligence range hood according to claim 2, wherein: The processor generates a recipe database based on the recipe information collected through the open application programming interface, and stores the generated recipe database in the memory, The memory further stores a plurality of recipes in the recipe database, the recipes including one or more entries of a dish name, a cooking method, ingredient information, an image showing the cooking method, and a text describing the cooking method, and The processor further generates a cooking vessel entry corresponding to the cooking method.

15. The artificial intelligence range hood according to claim 14, wherein: The memory also stores a cooking vessel recognition model. When a cooking vessel is included in the object recognition information, the processor generates the cooking vessel recognition information using the cooking vessel recognition model, and The cooking vessel identification information identifies the type of cooking vessel included in the captured image.

16. The artificial intelligence range hood according to claim 15, wherein: The processor retrieves a cooking vessel entry corresponding to the cooking vessel identification information from the recipe database to extract one or more dish names associated with the retrieved cooking vessel entry, and controls the display to display a recommended dish list including the extracted one or more dish names.

17. The artificial intelligence range hood according to claim 16, wherein: The processor receives a dish from the recommended dish list as user input to extract a recipe related to the dish received from the recipe database, and controls the display to display at least one image showing a cooking method included in the extracted recipe and text describing the cooking method.

18. The artificial intelligence range hood according to claim 17, further comprising: Multiple speakers, The processor converts the text describing the cooking method into speech using a text-to-speech engine stored in the memory, and outputs the converted speech through the plurality of speakers.

19. The artificial intelligence range hood according to claim 14, wherein: The memory also includes a food recognition model, and When one or more food ingredients are included in the object recognition information, the processor generates one or more food ingredient recognition information using the food ingredient recognition model, and the food ingredient recognition information recognizes the type of food ingredients included in the acquired image data.

20. A method for controlling an artificial intelligence range hood, the artificial intelligence range hood comprising a main body; a camera disposed at a lower end of the main body and configured to capture an image of a stove located below the main body; a display located on the front surface of the main body; a memory for storing an object recognition model; and a processor, the method comprising: capturing an image of the cooktop via the camera; generating, via the processor, object recognition information using the object recognition model stored in the memory, the object recognition information being used to identify a cooking object included in a captured image of the cooktop; setting, via the processor, a user interest region of the captured image, the user interest region corresponding to the recognized cooking object specified by the generated object information; controlling, via the processor, operation of the camera to capture a user's area of ​​interest in the captured image; and A user's area of ​​interest of the captured image is displayed on the display.

Images