Smart refrigerator system and food information provision method of smart refrigerator

Abstract

A smart refrigerator system according to an embodiment comprises: a server which implements a food determination model by training a deep learning neural network with a training data set and distributes the food determination model; a smart refrigerator which, by means of the food determination model received from the server, recognizes detailed food information including type, quantity, expiration date, and manufacturing date of food items, tracks the location of food inside or outside the refrigerator, and displays the detailed food information and the location of food; and a user terminal which operates in conjunction with the smart refrigerator to output the detailed food information and the location of food.

Description

Smart refrigerator system and method for providing food information in a smart refrigerator

[0001] The present disclosure relates to a smart refrigerator system and a method for providing food information of a smart refrigerator, and more particularly, to a smart refrigerator system and a method for providing food information that provide detailed food information including the type, location, expiration date, and manufacturing date of food stored in a smart refrigerator.

[0002] Unless otherwise indicated herein, the materials described in this section are not prior art to the claims of this application, and their inclusion in this section is not intended to be admitted as prior art.

[0003] A refrigerator maintains a low temperature for various foods by repeatedly cooling them over a set period of time, preventing spoilage. Because we typically store a wide variety of foods in our refrigerators, we often forget what we've stored there after a certain amount of time. Therefore, when cooking or consuming food, we must open the refrigerator and check the contents of the refrigerator to determine the type of food stored there.

[0004] Furthermore, because people don't remember all the foods stored in their refrigerators and their expiration dates, they often end up purchasing the same items multiple times. This practice increases the frequency of discarding food and negatively impacts household finances. Furthermore, it reduces refrigerator space efficiency.

[0005] Meanwhile, smart refrigerators connect to the Internet via a touchscreen interface and wireless network, offering a variety of additional features. Furthermore, smart refrigerators utilize the Internet of Things, allowing them to be controlled via smart devices or linked with other home appliances, such as smart TVs, smart dishwashers, smart microwaves, serving robots, and robot vacuum cleaners.

[0006] A smart refrigerator system according to an embodiment and a method for providing food information of a smart refrigerator sense a container for storing food, a food type, and a storage start time, and provides detailed food information including the type, location, expiration date, and manufacturing date of food stored in a smart refrigerator, as well as the storage period and location of each food.

[0007] In addition, the smart refrigerator system and the food information provision method of the smart refrigerator according to the embodiment recognize medicines, ointments, cosmetics, and various processed foods from around the world stored in the refrigerator, read letters and pictures written on the packaging of the processed foods, and provide detailed information including the expiration date to the user through voice and display.

[0008] Additionally, the smart refrigerator according to the embodiment can receive the type of food and the date of manufacture via voice or text input, and can track the storage location of food inside the refrigerator and the food movement path within the house by linking with various IoT devices in the home.

[0009] In addition, the smart refrigerator according to the embodiment can provide food detail information to the user terminal by linking with the user terminal and control the refrigerator temperature through the user terminal.

[0010] A smart refrigerator system according to an embodiment may include: a server that trains a deep learning neural network with a training data set to implement a food judgment model and distributes the food judgment model; a smart refrigerator that recognizes food detailed information including the type, amount, expiration date, and manufacturing date of food through the food judgment model received from the server, tracks the location of food inside or outside the refrigerator, and displays the food detailed information and the food location; a user terminal that outputs the food detailed information and the food location in conjunction with the smart refrigerator; and an Internet of Things home appliance that transmits a food image to the smart refrigerator in conjunction with the smart refrigerator to track the location of food outside the refrigerator.

[0011] In an embodiment, a smart refrigerator includes: a sensor for sensing food, text and pictures of the food packaging container, and the container containing the food; a communication unit for receiving a food judgment model from a server and linking with a user terminal and an Internet of Things home appliance; an extraction unit for identifying the type and amount of food sensed by the sensor through the food judgment model, and extracting an expiration date and a manufacturing date from the text and pictures of the food packaging container; and a display unit for displaying detailed food information including the type and amount of food, expiration date, and manufacturing date, and the food storage period and the location of the food.

[0012] The smart refrigerator system and the method for providing food information in the smart refrigerator described above enable users to know detailed information such as the type, location, and expiration date of food stored in the refrigerator, thereby reducing the frequency of repeat purchases or disposal of food.

[0013] In addition, the example increases the utilization of food stored in the refrigerator and tracks the location of food inside and outside the refrigerator, thereby enabling food to be stored more freshly and efficiently.

[0014] The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the detailed description of the present invention or the composition of the invention described in the claims.

[0015] Figure 1 is a drawing showing the configuration of a smart refrigerator system according to an embodiment.

[0016] Figure 2 is a diagram showing the data processing configuration of a smart refrigerator (100) according to an embodiment.

[0017] Figure 3 is a drawing showing an example of an internal sensor configuration.

[0018] Figure 4 is a drawing showing the data processing configuration of a server (200) according to an embodiment.

[0019] Figure 5 is a diagram showing a data processing configuration of a smart terminal according to an embodiment.

[0020] Figure 6 is a signal flow diagram of a smart refrigerator system according to an embodiment.

[0021] A smart refrigerator system according to the present invention comprises: a server for implementing a food judgment model by training a deep learning neural network with a training data set and distributing the food judgment model; a smart refrigerator for recognizing food detailed information including the type, amount, expiration date, and manufacturing date of food through the food judgment model received from the server, tracking the location of food inside or outside the refrigerator, and displaying the food detailed information and the food location; and a user terminal for outputting the food detailed information and the food location in conjunction with the smart refrigerator.

[0022] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

[0023] When describing embodiments of the present invention, detailed descriptions of known functions or configurations will be omitted if they are deemed to unnecessarily obscure the gist of the invention. Furthermore, the terms described below are defined in light of their functions in the embodiments of the present invention and may vary depending on the intent or custom of the user or operator. Therefore, their definitions should be based on the overall content of this specification.

[0024] Fig. 1 is a drawing showing the configuration of a smart refrigerator system according to an embodiment.

[0025] Referring to FIG. 1, a smart refrigerator system according to an embodiment may be configured to include a smart refrigerator (100), a server (200), a user terminal (300), and an Internet of Things home appliance (400).

[0026] The server (200) trains a deep learning neural network with a training data set to implement a food judgment model and distributes the food judgment model. The food judgment model is an artificial intelligence model that determines the type and amount of food based on food images, food shapes, packaging containers, and wrapping paper sensed in a smart refrigerator, and extracts detailed food information such as manufacturing date and expiration date. In an embodiment, the server (200) trains the food judgment model using a training data set that includes the type of food matched to the food image, expiration date, storage period for each food, etc. In addition, the server (200) can update the food judgment model by training the food judgment model with new food information received from an external server.

[0027] The smart refrigerator (100) recognizes food detailed information through a food judgment model received from a server (200). In an embodiment, the food detailed information is information related to food stored in the refrigerator. For example, the food detailed information may include the type of food stored, expiration date, manufacturing date, storage period, etc. In an embodiment, the smart refrigerator (100) recognizes food detailed information sensed from a sensor through the food judgment model, tracks the location of each food located inside or outside the refrigerator, and displays food detailed information and food location.

[0028] The user terminal (300) outputs food details and food locations in conjunction with the smart refrigerator (100). In an embodiment, at least one user terminal (300) may be implemented as a computer capable of accessing a remote server or terminal via a network. Here, the computer may include, for example, a notebook, desktop, or laptop equipped with a navigation system and a web browser. In this case, at least one user terminal (300) may be implemented as a terminal capable of accessing a remote server or terminal via a network. At least one user terminal (300) may include, for example, all kinds of handheld-based wireless communication devices such as navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminals, smartphones, smartpads, tablet PCs, etc., as wireless communication devices that ensure portability and mobility.

[0029] The Internet of Things (IoT) home appliance (400) is an Internet of Things-enabled home appliance installed in a home. In an embodiment, the IoT home appliance (400) is linked to a user terminal (300) and a smart refrigerator (100), and tracks the location of food taken out of the refrigerator and transmits the location information to the smart refrigerator (100) and the user terminal (300). In addition, in an embodiment, the IoT home appliance (400) transmits a food image to the smart refrigerator (100), thereby enabling the smart refrigerator to track the location of food outside the refrigerator through the food image.

[0030] In an embodiment, the IoT home appliance (400) may include any home appliance controllable via the Internet of Things, such as a wall pad, a television, a robot vacuum cleaner, a serving robot, a cooking robot observation camera, and lighting. In an embodiment, the smart refrigerator (100) may, when food is prepared and stored by a cooking robot, link with the cooking robot to store the manufacturing date and time of the food.

[0031] In addition, in an embodiment, when a series of tasks with a load exceeding a certain level is processed through artificial intelligence in a smart refrigerator (100), the tasks can be divided or collaborated among multiple Internet of Things home appliances (400), including desktops, TVs, and robots in the home.

[0032] A smart refrigerator system according to an embodiment and a method for providing food information of a smart refrigerator sense a container for storing food, a food type, and a storage start time, and provides detailed food information including the type, location, expiration date, and manufacturing date of food stored in a smart refrigerator, as well as the storage period and location of each food.

[0033] In addition, the smart refrigerator system and the food information provision method of the smart refrigerator according to the embodiment recognize medicines, ointments, cosmetics, and various processed foods from around the world stored in the refrigerator, read letters and pictures written on the packaging of the processed foods, and provide detailed information including the expiration date to the user through voice and display.

[0034] Additionally, the smart refrigerator according to the embodiment can receive the type of food and the date of manufacture via voice or text input, and can track the storage location of food inside the refrigerator and the food movement path within the house by linking with various IoT devices in the home.

[0035] In addition, the smart refrigerator according to the embodiment can provide food detail information to the user terminal by linking with the user terminal and control the refrigerator temperature through the user terminal.

[0036] Fig. 2 is a diagram showing a data processing configuration of a smart refrigerator (100) according to an embodiment.

[0037] Referring to FIG. 2, a smart refrigerator (100) according to an embodiment may be configured to include a sensor (110), a communication unit (120), an extraction unit (130), an output unit (140), a display unit (150), a text input unit (160), a location tracking unit (170), and a personal information protection unit (180). The term 'unit' used in this specification should be interpreted to include software, hardware, or a combination thereof, depending on the context in which the term is used. For example, the software may be machine language, firmware, embedded code, and application software. As another example, the hardware may be a circuit, a processor, a computer, an integrated circuit, an integrated circuit core, a sensor, a MEMS (Micro-Electro-Mechanical System), a passive device, or a combination thereof.

[0038] The sensor (110) senses users and food around the refrigerator. In an embodiment, the sensor (110) may be configured as a camera, etc. In an embodiment, when the sensor (110) is configured as a camera, an abnormal signal from the camera may be transmitted to the server (200) via a wireless communication network system from a communication unit within the camera to facilitate maintenance of the camera. In addition, in an embodiment, the camera may be powered by wireless power transmission technology from the main power supply unit of the smart refrigerator so that it can operate without a battery.

[0039] In one embodiment, the sensor (110) senses text and images on food and food packaging. In one embodiment, the sensor (110) may be comprised of internal and external sensors. In one embodiment, the internal sensor senses the location of food containers and each food item stored inside the refrigerator. The external sensor senses the user's motion, voice, food images, and food shapes.

[0040] In the embodiment, an external sensor is provided as a voice recognition sensor, so that detailed information about food can be asked to the user without operating a camera, and detailed information about food can be determined through voice recognition of the user's response.

[0041] In addition, in the embodiment, the sensor (110) may be configured with a plurality of cameras, and the number of cameras equipped in the smart refrigerator may be set differently depending on the size of the smart refrigerator (100) and the installation site, so that multiple people can use the smart refrigerator (100) simultaneously.

[0042] In an embodiment, when configuring multiple cameras with sensors (110), each camera tracks a different target and can communicate with the other cameras to prevent duplication of the tracking targets of each camera. At this time, the cameras generate a duplication estimation prevention signal and transmit and receive it to other cameras. The duplication estimation prevention signal is a signal for preventing duplication of the tracking targets of each camera, and can include information about the target being tracked by each camera, the field of view of each camera, and the shooting boundary information of each camera.

[0043] These cameras can be used for monitoring other devices for malfunctions, relaying signals, and other tasks.

[0044] In one embodiment, the internal sensor can identify the location of a container by identifying a tag configured within the container. Specifically, in one embodiment, a tag is inserted into a non-microwaveable container, such as a kimchi container, and the internal sensor recognizes the inserted tag to track the location of the container storing the food. Furthermore, in one embodiment, the external sensor senses the user's voice, allowing detailed food information, such as the food type and expiration date, to be input and stored.

[0045] In addition, in an embodiment, when the external sensor is a camera and the user blocks the view of the camera in the refrigerator, the user's shape and user motion are sensed, and the accurate location of the food is estimated by linking with the internal sensor. For example, even when the view is blocked, the internal sensor tracks the internal movement location of the food in real time and transmits the accurate internal location of the food to the external sensor, which is the camera. Referring to FIG. 3, which illustrates an embodiment of the internal sensor configuration, a plurality of internal sensors (1, 2, 3, 4, 5, 6) according to the embodiment are included, and each internal sensor may be composed of a movable camera (1), an angle adjustment sensor (2), and a camera (3, 4, 5).

[0046] In the embodiment, the angle adjustment sensor (2) includes an infrared sensor, etc., to enable tracking of the approximate shape or real-time location of food. In the embodiment, the location of food is tracked even when the refrigerator is closed through an internal sensor, and in conjunction with photographic data from an external sensor, a camera, it provides image data on the location of the food.

[0047] The mobile camera (1) according to the embodiment can be angle-adjusted, and the lens can be dehumidified in the parking area (6). In the embodiment, the mobile camera (1) is used for food shape recognition and food location tracking on the inner door and inner shelves (7, 8, 9). The parking area (6) is a device that removes moisture or foreign substances from the camera lens, which is an internal sensor.

[0048] In an embodiment, a mobile camera (1) and an angle adjustment sensor (2) track the shape and location of food inside the refrigerator and inside the open door from the outside. In an embodiment, the external cameras, which are external sensors, can be used to track food movement and location outside the refrigerator.

[0049] The communication unit (120) receives a food judgment model from the server (200) and connects with a user terminal (300) and an IoT home appliance (400). The communication unit (120) transmits food detailed information to the user terminal (300) and receives a food image from an IoT home appliance (400), thereby enabling the tracking of the location of food outside the refrigerator.

[0050] The extraction unit (130) determines the type and quantity of food, expiration date, and manufacturing date. In the embodiment, the extraction unit (130) determines the type and quantity of stored food sensed by the sensor (110) through the food judgment model received from the server (200), and extracts the expiration date and manufacturing date from the text and pictures written on the food packaging.

[0051] Additionally, the extraction unit (130) can extract information about the food contained in a container based on the shape and pattern of the container. For example, if it is recognized that a food stored in a container exceeding a certain size is kimchi, the food contained in a container whose size and appearance pattern are similar to the kimchi container to a certain degree or more can be extracted as kimchi. Furthermore, the extraction unit (130) can identify the type of food contained in a transparent container through image recognition and a food judgment model.

[0052] The calculation unit (140) calculates the storage period for each food based on the initial storage time of the food stored in the refrigerator. In addition, the calculation unit (140) calculates the food edible period. In the embodiment, the food edible period is the period from the present time to the expiration date, which is the period during which the food can be consumed before it spoils. If the expiration date of the food is set, the calculation unit (140) calculates the food edible period based on the period from the present time to the expiration date. If the expiration date of the food is not set, the calculation unit (140) calculates the food edible period based on the period set according to the type of the food.

[0053] In the embodiment, the calculation unit (140) can calculate the food consumption period in approximate time units for food without a specified expiration date. For example, the food consumption period can be calculated in time units such as one week, two weeks, or one month.

[0054] Additionally, in the embodiment, the output unit (140) predicts the food to be retrieved by the user based on the user's food retrieval pattern information and identifies the predicted food location. In the embodiment, the user's food retrieval pattern information refers to the type of food the user takes out of the refrigerator over time. Specifically, if the user repeatedly takes out milk and eggs around 7:00 AM and meat around 6:00 PM according to the user's regular schedule, the time of food retrieval and the type of food can be stored as food retrieval pattern information.

[0055] In the embodiment, the calculation unit (140) stores information on the user's food retrieval pattern for a certain period of time, identifies information on food repeatedly retrieved by the user at certain points in time, and predicts the type of food the user intends to retrieve when the user approaches the refrigerator at a certain point in time. The calculation unit (140) then calculates detailed information on the food, including the predicted food's location, storage period, expiration date, and remaining amount. Thereafter, the calculation unit (140) provides the user with detailed information on the corresponding food.

[0056] The display unit (150) displays information generated by the smart refrigerator (100). For example, the display unit (150) may output detailed information about food stored in the refrigerator, the food storage period, location information about food to be retrieved, detailed information about food to be retrieved, the food consumption period for each food, and the location of the food outside the refrigerator. In an embodiment, the detailed information about food may include the type and amount of food, expiration date, manufacturing date, etc. In addition, in an embodiment, the display unit (150) may periodically output the food consumption period and the food expiration date.

[0057] Additionally, the display unit (150) enlarges the image of food and containers captured by the sensor (110) and separates the food and container objects from the image, allowing the user to easily identify the separated objects. Furthermore, a virtual screen is displayed on the refrigerator display to facilitate individual clicking of the objects corresponding to the separated food and containers. This allows the user to accurately identify food details and container shapes on the external display panel.

[0058] In addition, the display unit (150) can be configured as a full-length mirror in the form of a touch screen, and can provide user-customized information such as how to dress, how to put on makeup, how to cook food, and how to decorate a table.

[0059] In addition, when a full-length mirror is installed opposite the smart refrigerator (100), the display unit (150) can use an image that is an inverted image reflected in the full-length mirror opposite the refrigerator through an external camera installed on the top of the refrigerator to track the shape and location of food on the refrigerator shelf or inside the door. In addition, the display unit (150) can 3D scan the user and, by linking with an artificial intelligence program, output the user's full body image on the refrigerator display and provide 3D scanned images of the user wearing various clothing.

[0060] In one embodiment, a full-length mirror opposite the smart refrigerator (100) serves as an auxiliary function of the interior camera. For example, the camera on top of the smart refrigerator (100) can be used to capture images of food on refrigerator shelves or inside a wide-open door, as reflected by the full-length mirror, and to track their location.

[0061] In addition, the display unit (150) can combine overlapping foods or food containers into one in the display list so that only the type and quantity of food can be output.

[0062] The text input unit (160) receives detailed food information as text from the user. In an embodiment, the text input unit (160) outputs a text input button on the display panel of the smart refrigerator (100) according to user settings, thereby allowing the user to input detailed food information, such as the type of food and expiration date, in text form.

[0063] The location tracking unit (170) tracks the location of food outside the refrigerator by linking with an IoT home appliance (400) installed in the home. For example, the location tracking unit (170) can receive an image of food taken from an IoT home appliance (400) equipped with a camera and track the current location of the food. In addition, the location tracking unit (170) receives real-time image information from the IoT home appliance (400) and determines the location of the food.

[0064] In an embodiment, the location tracking unit (170) can track the location of food outside the refrigerator by recognizing food in a food image through an image recognition process using artificial intelligence machine learning. Artificial intelligence image recognition is a computer vision technology that allows a machine to recognize objects from a photograph and understand a scene, much like a human. In an embodiment, the location tracking unit (170) performs a data processing process for image recognition, including classification and detection of food objects included in the image, and segmenting the objects by identifying them on a pixel-by-pixel basis. In an embodiment, the location tracking unit (170) can perform an out-of-distribution detection process for processing unlearned patterns other than noise response. Out-of-distribution detection identifies whether an image input to the artificial intelligence is learned probability distribution data. In an embodiment, out-of-distribution detection filters out or processes exceptions to images that are difficult for the artificial neural network to judge, thereby increasing stability and reliability. In the embodiment, the probability value indicating the degree of confidence in the deep learning judgment is calibrated for detecting non-learning distribution data, or the non-learning distribution data is generated and trained using a generative adversarial network (GAN) to improve the accuracy of food detection.

[0065] In addition, in the embodiment, the location tracking unit (170) uses a lightweight deep learning technology that simplifies the calculation to reduce the size of the model while maintaining the accuracy of recognizing the food image received from the Internet of Things home appliance (400), so as to finally determine the location of the food outside the refrigerator. In the embodiment, the convolution filter in the convolutional neural network (CNN) is modified for image recognition to reduce the calculation dimension, or the weight of the neural network that does not have a large influence is deleted through pruning, and the floating point of the weight value is reduced through a quantization process to simplify the calculation, thereby enabling data lightweighting. In addition, in the embodiment, the output of a large pre-trained neural network is imitated and learned in a small neural network, thereby simplifying the calculation and maintaining accuracy.

[0066] In addition, in the embodiment, the location tracking unit (170) can calculate the food location based on the real-time location information of the Internet of Things home appliance (400) that captured the food image, and thus track the location of the food outside the refrigerator. In addition, in the embodiment, the location tracking unit (170) can receive the food location from the Internet of Things home appliance (400) that captured the food image, and collect the received food location to track the location of the food outside the refrigerator.

[0067] In addition, in the embodiment, the location tracking unit (170) transmits a request for transmitting the location of the food to the user terminal (300) when the location tracking of the food outside is lost while tracking the location of the food in conjunction with the Internet of Things home appliance (400). Thereafter, when the location tracking unit (170) receives the location of the food from the user terminal (300), it can confirm and update the location of the food and transmit a message to the user terminal (300) recommending that the food be brought into the refrigerator.

[0068] In addition, in the embodiment, the location tracking unit (170) can adjust the intensity of location tracking of food outside the refrigerator. In the embodiment, the location tracking unit (170) can adjust the intensity of location tracking of food outside the refrigerator according to the location tracking intensity setting such as high or low. For example, the location tracking unit (170) can adjust the number of Internet of Things home appliances linked to the smart refrigerator (100) according to the location tracking intensity. Specifically, when the location tracking intensity is high, the location tracking unit (170) can set the number of Internet of Things home appliances linked to the smart refrigerator (100) to a certain number or more (e.g., 5 or more), and when the location tracking intensity is low, the number of Internet of Things home appliances linked to the smart refrigerator (100) can be set to a certain number or less. That is, in the embodiment, when the tracking intensity is high, the location tracking unit (170) increases the number of Internet of Things home appliances linked to the smart refrigerator (100), thereby allowing more data such as food images to be collected for obtaining location information of food outside the refrigerator.

[0069] Additionally, the location tracking unit (170) can set the number of linked IoT home appliances to 0 when the location tracking function is set to off.

[0070] Additionally, the location tracking unit (170) can set the tracking intensity depending on the type of food. For example, for food that does not need to be returned to the refrigerator after being removed from the refrigerator, such as beverage cans, the location tracking function for the food can be turned off. On the other hand, for food that needs to be returned to the refrigerator for storage, such as large-capacity milk or large-capacity food, the tracking intensity can be set high. If location tracking of the food outside is lost, a location transmission request for the food can be transmitted to the user terminal (300).

[0071] The personal information protection unit (180) limits the location range of the Internet of Things home appliance (400) that is linked to the smart refrigerator (100), so that food location and image information can be transmitted and received only in a limited space within the house. To this end, the personal information protection unit (180) can limit the range in which the location of the Internet of Things home appliance can be included, or preset the type of the Internet of Things home appliance (400) that is linked to the smart refrigerator (100). In addition, the personal information protection unit (180) sets the tracking range of the food location to only indoors, so that food location and image information can be transmitted and received only in a limited space within the house. In addition, in the embodiment, the personal information protection unit (180) allows the transmission and reception of the food location through the user terminal (300) when the food location is not recognized inside the house or the food location is outside the house.

[0072] In addition, in the embodiment, the personal information protection unit (180) sets the format of food location information as coordinates and text when storing the food location, thereby preventing personal information leakage indoors.

[0073] FIG. 4 is a diagram showing a data processing configuration of a server (200) according to an embodiment.

[0074] Referring to FIG. 4, a server (200) according to an embodiment may be configured to include a deep learning unit (210) and a communication unit (220).

[0075] The deep learning unit (210) trains a neural network using a training data set to implement a food judgment model. In the embodiment, the food judgment model is an artificial intelligence model that identifies the type and quantity of food based on its image, shape, packaging, etc., and recognizes detailed food information such as the manufacturing date and expiration date.

[0076] In an embodiment, the neural network includes, but is not limited to, at least one of a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), and a Bidirectional Recurrent Deep Neural Network (BRDNN). In an embodiment, the deep learning unit (210) can learn food information, including food images, expiration dates by food type, storage methods, etc., as a training data set, thereby implementing a food judgment model.

[0077] The communication unit (220) distributes a food judgment model implemented in a smart refrigerator, communicates with an external server, and updates the food judgment model. In particular, the communication unit (220) updates the food judgment model with global languages, new food brands, and expiration date labeling methods. Furthermore, in one embodiment, the communication unit (220) collects abnormal signals from the smart refrigerator and automatically requests an inspection of the smart refrigerator from an after-sales service center based on the collected abnormal signals. In one embodiment, the abnormal signals may include abnormal signals related to a series of data processing processes occurring in the smart refrigerator, such as temperature maintenance, food, and user detection abnormalities.

[0078] Fig. 5 is a diagram showing a data processing configuration of a smart terminal according to an embodiment.

[0079] Referring to FIG. 5, a smart terminal (300) according to an embodiment may be configured to include an information provision unit (310) and a recipe extraction unit (320).

[0080] The information providing unit (310) outputs food detailed information generated by the smart refrigerator, food storage period, food ingestion period, food detailed information and food location according to the user's food return pattern. In an embodiment, the information providing unit (310) may sort and output food detailed information according to the food's storage period, expiration date, food ingestion period, and food quantity. For example, foods with a long storage period may be output in order, or foods with a short expiration date may be output first. Specifically, foods may be sorted and output in order of the food ingestion period, which is the time between the expiration date and the current point in time. In addition, the information providing unit (310) may output food detailed information according to the food quantity and expiration date. For example, foods may be sorted and output in order of the food ingestion period to the food quantity. In addition, the information providing unit (310) may periodically output the food ingestion period and food expiration date in units of time.

[0081] The recipe extraction unit (320) provides recipe information for dishes that can be made using food stored in the smart refrigerator. For example, if the food stored in the refrigerator is kimchi, eggs, and butter, the recipe extraction unit (320) can provide recipe information for dishes such as kimchi fried rice that can be made using these foods.

[0082] Additionally, the recipe extraction unit (320) can provide shopping information. In one embodiment, the recipe extraction unit (320) can provide recipe information for a dish that can be made if there is more of a specific food item stored in the refrigerator, and can provide shopping information, which is a list of food items to be purchased for the dish. Furthermore, if the amount of food regularly consumed is low, the recipe extraction unit (320) can provide shopping information for the remaining food items. For example, if the amount of milk or eggs is determined to be below a certain level, the recipe extraction unit (320) can provide shopping information for milk and eggs.

[0083] In addition, when the number of foods stored in the refrigerator is a certain number or more, the recipe extraction unit (320) can extract foods with a storage period of a certain level or longer and provide cooking recipe information using the extracted foods. For example, the recipe extraction unit (320) can extract foods with a storage period exceeding a certain period (e.g., one month), such as pork belly or meat, and provide cooking recipe information for using up the extracted foods. Specifically, the recipe extraction unit (320) can provide recipe information for kimchi stew, stir-fried meat, etc. for using up meat with a storage period exceeding one month.

[0084] Below, the information provision method of a smart refrigerator is sequentially described. Since the function (function) of the information provision method of a smart refrigerator according to an embodiment is essentially the same as the function of a smart refrigerator system, any description overlapping with that of FIGS. 1 to 4 will be omitted.

[0085] Fig. 6 is a signal flow diagram of a smart refrigerator system according to an embodiment.

[0086] Referring to FIG. 6, in step S100, a deep learning neural network is trained on a server using a training data set to implement a food judgment model, and the food judgment model is distributed to a smart refrigerator. In step S200, the smart refrigerator receives the food judgment model from the server, and in step S300, the food detailed information, including the type, amount, expiration date, and manufacturing date of the food, is recognized through the food judgment model. In an embodiment, step S300 may include a step of sensing food, text and pictures of a food packaging container, and a container containing the food by a sensor; a step of receiving the food judgment model from the server by a communication unit and linking it with a user terminal and an Internet of Things home appliance; and a step of identifying the type and amount of food sensed by the sensor through the food judgment model by an extraction unit, and extracting the expiration date and manufacturing date from the text and pictures of the food packaging container.

[0087] In step S400, the user terminal receives detailed food information from the smart refrigerator and outputs the detailed food information. In step S500, the Internet of Things appliance generates an image of the food outside the refrigerator and transmits it to the smart refrigerator. In step S600, the food image is collected from the Internet of Things appliance, and in step S700, the smart refrigerator tracks the location of the food inside or outside the refrigerator. In an embodiment, step S700 recognizes the food in the food image received from the Internet of Things electronic device through an image recognition process using artificial intelligence machine learning, and tracks the location of the food outside the refrigerator. Thereafter, in step S800, the smart refrigerator displays detailed food information and the food location. In step S900, the user terminal receives the food location from the smart refrigerator and outputs the food location.

[0088] The smart refrigerator system and the method for providing food information in the smart refrigerator described above enable users to know detailed information such as the type, location, and expiration date of food stored in the refrigerator, thereby reducing the frequency of repeat purchases or disposal of food.

[0089] In addition, the example increases the utilization of food stored in the refrigerator and tracks the location of food inside and outside the refrigerator, thereby enabling food to be stored more freshly and efficiently.

[0090] The disclosed content is merely an example, and various modifications and implementations can be made by a person skilled in the art without departing from the gist of the claims claimed in the patent, so the scope of protection of the disclosed content is not limited to the specific embodiments described above.

Claims

1. In the smart refrigerator system, A server that implements a food judgment model by training a deep learning neural network with a training data set and distributes the food judgment model; A smart refrigerator that recognizes food detailed information including the type, amount, expiration date, and manufacturing date of food through a food judgment model received from the above server, tracks the location of food inside or outside the refrigerator, and displays the food detailed information and food location; and A smart refrigerator system including a user terminal that outputs food details and food locations in conjunction with the smart refrigerator.

2. In paragraph 1, The above smart refrigerator system A smart refrigerator system further comprising an Internet of Things home appliance that links with the smart refrigerator to transmit food images to the smart refrigerator to track the location of food outside the refrigerator.

3. In paragraph 1, The above smart refrigerator Food, text and pictures on the food packaging container, a sensor for sensing the container containing the food; A communication unit that receives a food judgment model from the server and connects with the user terminal and an Internet of Things home appliance; An extraction unit that identifies the type and amount of food sensed from the sensor through the food judgment model and extracts the expiration date and manufacturing date from the text and picture of the food packaging container; A smart refrigerator system characterized by including a display unit that displays food details including the type and amount of the food, expiration date, manufacturing date, food storage period, and food location.

4. In paragraph 3, The above sensor An internal sensor that recognizes the container and location of food stored inside the refrigerator; and A smart refrigerator system comprising an external sensor that recognizes a user's motion and voice.

5. In paragraph 2, The above smart refrigerator A text input section for entering food details including the type of food, expiration date, and manufacturing date in text format; and A smart refrigerator system further comprising a location tracking unit that tracks the location of food outside the refrigerator by linking with an Internet of Things device in the home.

6. In paragraph 5, The above location tracking unit A smart refrigerator system characterized by recognizing food in food images received from Internet of Things electronic devices through an image recognition process using artificial intelligence machine learning, and tracking the location of food outside the refrigerator.

7. In the method of providing information on a smart refrigerator, (A) A step of implementing a food judgment model by training a deep learning neural network with a training data set on a server and distributing the food judgment model; (B) A step of recognizing food detailed information including the type, amount, expiration date, and manufacturing date of food through a food judgment model received from the server in a smart refrigerator; (C) a step of tracking the location of food inside or outside the refrigerator in a smart refrigerator and displaying the food details and location; and (D) A method for providing information of a smart refrigerator, including a step of linking a user terminal with a smart refrigerator to output food details and food location.

Images