Refrigerator, server and control method thereof
By using a combination of cameras and servers in the refrigerator to identify and classify food inside the refrigerator, the problem of difficulty in identifying overlapping foods in the existing refrigerators is solved, and efficient food management and health information provision is achieved.
Patent Information
- Application Number
- CN202410972366.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-06-15
- Filing Date
- 2019-06-07
- Publication Date
- 2024-11-08
AI Technical Summary
Existing refrigerators are difficult to efficiently identify and manage food in storage rooms, especially when foods are placed overlapping, and the identification efficiency is inefficient and cannot be accurately identified.
By installing a camera and a server in the refrigerator, the processor is used to compare the captured internal images of the refrigerator, identify the changing areas, combine the food change sensing model and the identification model, identify and classify food, and send corresponding food storage information to the refrigerator.
It realizes efficient identification and management of food inside the refrigerator, can identify overlapping foods, improves identification accuracy and efficiency, provides food category, location and processing information, and supports recipe recommendations and health information display.
Smart Images

Figure CN118912796A_ABST
Abstract
Description
[0001] This application is a divisional application of the invention patent application with the application date of June 7, 2019, Chinese application number 201980040258.5, and invention name “Refrigerator, server and control method thereof”. Technical Field
[0002] The present disclosure relates to a refrigerator, a server, and a method of identifying food stored in the refrigerator by using the refrigerator and the server. Background Art
[0003] A refrigerator is an electronic device (or household appliance) used to store refrigerated or frozen food. A refrigerator can store not only food, but also medicines, alcohol or cosmetics.
[0004] Advances in technology have made it possible to use a display to display the operating mode of the refrigerator. In addition, the refrigerator displays a user interface on the display to display information or receive user input. In addition, recent refrigerators may include a communication interface that can be connected to an external device (e.g., a server connected to the Internet).
[0005] In this way, the refrigerator has provided various services to the user through the display and the communication interface.
[0006] The above information is provided as background information only to assist in understanding the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above may be applied as prior art with respect to the present disclosure. Summary of the invention
[0007] Technical issues
[0008] Provided are a refrigerator, a server, and a control method thereof.
[0009] Technical Solutions
[0010] Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented embodiments.
[0011] According to one aspect of the present disclosure, a server includes a memory storing one or more instructions and at least one processor configured to execute the instructions to obtain at least one refrigerator interior image taken in the refrigerator, and identifies whether there is a changed area in the at least one refrigerator interior image obtained, in which a change occurs in the at least one refrigerator interior image obtained, based on the fact that the changed area does not exist in the at least one refrigerator interior image obtained, using a first result of identifying the pre-stored image to obtain first food storage information, based on the fact that the changed area exists in the at least one refrigerator interior image obtained, using the obtained first food storage information and a second result of identifying an object contained in the at least one refrigerator interior image obtained to obtain second food storage information, and sending the at least one refrigerator interior image obtained and one or both of the obtained first food storage information and the obtained second food storage information to the refrigerator.
[0012] The at least one processor is also configured to identify whether there is a changed area in at least one obtained image of the interior of the refrigerator based on a difference between a first value of each first pixel contained in the pre-stored image and a second value of each second pixel contained in at least one obtained image of the interior of the refrigerator.
[0013] The at least one processor is also configured to apply, to at least one obtained image of the interior of the refrigerator, either or both of a first image correction process for correcting blur of the at least one obtained image of the interior of the refrigerator and a second image correction process for increasing clarity of the at least one obtained image of the interior of the refrigerator, and to identify whether the change area exists in the at least one obtained image of the interior of the refrigerator to which either or both of the first image correction process and the second image correction process are applied.
[0014] The at least one processor is also configured to use a food change sensing model to identify changed areas in at least one obtained image of the interior of the refrigerator, and wherein the food change sensing model is a data recognition model that is trained using a first image, a second image that is identical to the first image but also includes a partial area with changes, and location information with changes as learning data.
[0015] The at least one processor is also configured to change the at least one obtained image of the interior of the refrigerator into a new image that emphasizes the edge component of the at least one obtained image of the interior of the refrigerator, and based on the at least one obtained image of the interior of the refrigerator being changed into the new image, identify the object contained in the new image based on the shape of the object identified based on the edge component.
[0016] The at least one processor is further configured to identify an object contained in the at least one obtained refrigerator interior image in the change region based on the change region existing in the at least one obtained refrigerator interior image.
[0017] The at least one processor is also configured to use a food recognition model to identify a first type of food contained in at least one obtained image of the interior of the refrigerator, wherein the food recognition model is a first data recognition model that is trained using a food image containing information about the food, the food category to which the food belongs, and the location of the food in the food image as first learning data.
[0018] The at least one processor is also configured to, based on the object being identified as processed food, use a processed food recognition model to identify a second category of processed food contained in at least one obtained image of the interior of the refrigerator, wherein the processed food recognition model is a second data recognition model trained using multiple images of the same processed food, processed food categories of the same processed food, and brand names of the same processed food as second learning data.
[0019] At least one of the obtained refrigerator interior images includes a first refrigerator interior image and a second refrigerator interior image, and wherein the at least one processor is also configured to obtain a third result of identifying another object contained in the second refrigerator interior image but not identified in the first refrigerator interior image.
[0020] According to one aspect of the present disclosure, a refrigerator includes a camera configured to photograph a storage room where food is stored, a communication interface, a processor; and a memory storing one or more instructions, wherein the instructions enable the processor to control the camera to photograph an image of the interior of the refrigerator based on the occurrence of a predetermined event, and control the communication interface to send the photographed image of the interior of the refrigerator to a server, and receive from the server one or both of first food storage information and second food storage information corresponding to the sent image of the interior of the refrigerator.
[0021] The refrigerator may further include a display, and the one or more instructions further cause the processor to control the display to display the captured image of the interior of the refrigerator, control the display to display at least one candidate category to which the selected food belongs based on a first user input selecting a food contained in the image of the interior of the refrigerator displayed on the display, and identify one of the selected candidate categories as a category of food based on a second user input selecting one of the at least one displayed candidate categories.
[0022] The refrigerator may further include a display, and the one or more instructions may also enable the processor to control the display to display a recipe providing an execution object, provide an execution object based on user input selecting the recipe displayed on the display, control the communication interface to send the categories of food contained in the captured image of the interior of the refrigerator to the server, and receive a recipe obtained by the server using the sent categories, and control the display to display the received recipe.
[0023] The refrigerator may further include a display, and the one or more instructions also cause the processor to control the display to display a food purchase website link execution object, and based on user input selecting the food purchase website link execution object displayed on the display, control the communication interface to access a food purchase website that sells food categories corresponding to foods that were previously included in the captured image of the interior of the refrigerator but are not currently included in the captured image of the interior of the refrigerator, and control the display to display the accessed food purchase website.
[0024] The refrigerator may further include a display, and the instructions may also cause the processor to control the display to display a health information display execution object, and based on user input selecting the health information display execution object displayed on the display, control the display to display the user's health information using the calories of food previously included in the captured image of the interior of the refrigerator but not currently included in the captured image of the interior of the refrigerator.
[0025] According to one aspect of the present disclosure, a refrigerator includes a camera configured to photograph a storage room where food is stored, a communication interface, a processor, and a memory storing one or more instructions, wherein the instructions enable the processor to control the camera to photograph an image of the interior of the refrigerator based on the occurrence of a predetermined event, identify whether there is a changed area in the photographed image of the interior of the refrigerator in which a change occurs in the photographed image of the interior of the refrigerator, obtain first food storage information by comparing a pre-stored image with the photographed image of the interior of the refrigerator using a first result of identifying the pre-stored image based on the absence of the changed area in the photographed image of the interior of the refrigerator, and control the communication interface to send the photographed image of the interior of the refrigerator to a server based on the presence of the changed area in the photographed image of the interior of the refrigerator, and receive from the server second food storage information obtained by the server using the first food storage information and a second result of identifying an object contained in the sent image of the interior of the refrigerator.
[0026] The one or more instructions cause the processor to identify whether a changed area exists in the captured image of the interior of the refrigerator based on a difference between a first value of each first pixel contained in the pre-stored image and a second value of each second pixel contained in the captured image of the interior of the refrigerator.
[0027] The one or more instructions cause the processor to apply, to the captured image of the interior of the refrigerator, either or both of a first image correction process for correcting blur of the captured image of the interior of the refrigerator and a second image correction process for increasing clarity of the captured image of the interior of the refrigerator, and to identify whether a changed area exists in the captured image of the interior of the refrigerator to which either or both of the first image correction process and the second image correction process are applied.
[0028] The one or more instructions cause the processor to use a food change sensing model to identify changed areas in the captured image of the interior of the refrigerator, wherein the food change sensing model is a data recognition model that is trained using a first image and a second image that is identical to the first image but also contains a changed partial area and location information of the change as learning data.
[0029] According to one aspect of the present disclosure, a method for controlling a server includes acquiring at least one refrigerator interior image taken in the refrigerator, determining whether there is a changed area in the at least one refrigerator interior image acquired by comparing a pre-stored image with the at least one refrigerator interior image acquired, acquiring first food storage information using a first result of identifying the pre-stored image based on the changed area being determined to be present in the at least one refrigerator interior image acquired, acquiring second food storage information using the acquired first food storage information and a second result of identifying an object contained in the at least one refrigerator interior image acquired, and sending the at least one refrigerator interior image acquired and one or both of the acquired first food storage information and the acquired second food storage information to the refrigerator.
[0030] The method may further include applying the pre-stored image and at least one acquired image of the interior of the refrigerator to a food change sensing model configured to identify a changed area in at least one acquired image of the interior of the refrigerator, and the food change sensing model may be a data recognition model that uses the first image and a second image that is identical to the first image but also includes a changed partial area and changed location information as learning data for training.
[0031] The method may further include applying the acquired at least one image of the interior of the refrigerator to a food recognition model configured to recognize a first type of food contained in the acquired at least one image of the interior of the refrigerator, and the food recognition model may be a first data recognition model that is trained using a food image including information about the food, the food category to which the food belongs, and the location of the food in the food image as first learning data.
[0032] The method may further include, based on the object being identified as a processed food, applying the acquired at least one image of the interior of the refrigerator to a processed food recognition model configured to recognize a second type of processed food contained in the acquired at least one image of the interior of the refrigerator, and the processed food recognition model may be a second data recognition model trained using multiple images of the same processed food, processed food categories of the same processed food, and brand names of the same processed food as second learning data.
[0033] According to one aspect of the present disclosure, a non-transitory computer-readable medium is configured to store one or more computer programs including instructions, which, when executed by a processor of an electronic device, cause the electronic device to acquire at least one refrigerator interior image taken in the refrigerator, determine whether there is a changed area in the at least one refrigerator interior image acquired by comparing the pre-stored image with the at least one refrigerator interior image acquired, acquire first food storage information using a first result of identifying the pre-stored image based on the changed area being determined to be present in the at least one refrigerator interior image acquired, acquire second food storage information using the acquired first food storage information and a second result of identifying an object contained in the at least one refrigerator interior image acquired based on the changed area being determined to be present in the at least one refrigerator interior image acquired, and send the at least one refrigerator interior image acquired and one or both of the acquired first food storage information and the acquired second food storage information to the refrigerator.
[0034] According to one aspect of the present disclosure, a refrigerator includes a first camera, a second camera, a first storage chamber, a second storage chamber, a communication interface, a processor, and a memory configured to store one or more instructions, wherein the instructions enable the processor to control the first camera to capture a first refrigerator interior image of the first storage chamber, control the second camera to capture a second refrigerator interior image of the second storage chamber, determine whether an area of each of the captured first refrigerator interior image and the captured second refrigerator interior image has changed by comparing a pre-stored image with a corresponding one of the captured first refrigerator interior image and the captured second refrigerator interior image, obtain first food storage information using a first result of identifying the pre-stored image based on the area being determined to have not changed in the captured first refrigerator interior image, and control the communication interface to send the captured second refrigerator interior image to a server based on the area being determined to have changed in the captured second refrigerator interior image, and receive second food storage information obtained from the server by the server using the first food storage information and a second result of identifying an object contained in the sent second refrigerator interior image.
[0035] The one or more instructions may further enable the processor to obtain first food storage information based on the area being determined to be unchanged in the captured second image of the interior of the refrigerator, and to control the communication interface to send the captured first image of the interior of the refrigerator to the server based on the area being determined to have changed in the captured first image of the interior of the refrigerator, and to receive from the server third food storage information obtained by the server using the first food storage information and a third result of identifying another object contained in the sent first image of the interior of the refrigerator.
[0036] According to one aspect of the present disclosure, a refrigerator includes: a display; a camera for photographing the interior of the refrigerator; a communicator for communicating with a server storing a recognition model, the recognition model being trained to recognize information about the category of food included in an image; and a processor configured to: acquire a first image using the camera, based on receiving a user input for inputting the category of food included in the first image, send the first image and the information about the category of food to the server through the communicator so that the recognition model is trained based on the first image and the information about the category input by the user input, and based on sending a second image acquired by the camera to the server through the communicator, acquire information about the category of food recognized from the second image by the trained recognition model from the server through the communicator, wherein the recognition model is configured to: based on the food corresponding to the food included in the first image being included in the second image, recognize that the category of food included in the second image is the category input by the user input according to learning.
[0037] According to one aspect of the present disclosure, a method for controlling a refrigerator, the refrigerator communicates with a server storing a recognition model, the recognition model being trained to recognize information about a category of food included in an image, the method comprising: acquiring a first image using a camera, sending the first image and information about the category of food to the server based on receiving a user input for inputting the category of food included in the first image, so that the recognition model is trained based on the first image and the information about the category input by the user input, and acquiring from the server information about the category of food recognized from the second image by the trained recognition model based on sending a second image acquired by the camera to the server, wherein the recognition model is configured to: based on the food corresponding to the food included in the first image being included in the second image, recognize that the category of food included in the second image is the category input by the user input according to learning.
[0038] According to one aspect of the present disclosure, a non-transitory computer-readable medium includes instructions, the instructions are based on being executed by a processor of a refrigerator, causing the refrigerator to perform operations, the refrigerator communicates with a server storing a recognition model, the recognition model is trained to recognize information about the category of food included in an image, the operation includes: using a camera to acquire a first image, based on receiving a user input for inputting the category of food included in the first image, sending the first image and the information about the category of food to the server, so that the recognition model is trained based on the first image and the information about the category input by the user input, and based on sending a second image acquired by the camera to the server, acquiring from the server information about the category of food recognized from the second image by the trained recognition model, wherein the recognition model is configured to: based on the food corresponding to the food included in the first image being included in the second image, recognize that the category of food included in the second image is the category input by the user input according to learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The above and other aspects and advantages of certain embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings, in which:
[0040] Figure 1a and 1b is a schematic diagram for explaining a food identification method according to an embodiment;
[0041] Figure 2a and 2b is a block diagram of a brief configuration of a refrigerator and a server according to one embodiment;
[0042] Figure 3 is a block diagram of a processor according to one embodiment;
[0043] Figure 4a and 4b is a schematic diagram showing a case where a plurality of cameras are used in a food recognition method according to an embodiment;
[0044] Figure 5a , 5b 5c are schematic diagrams showing a case where a food category is determined using a result of a food recognition method according to an embodiment;
[0045] Figure 6a and 6b is a schematic diagram showing a case where a recipe is obtained using a result of a food recognition method according to an embodiment;
[0046] Figure 7a and 7bis a schematic diagram showing a situation where a commodity is purchased using a result of a food recognition method according to an embodiment;
[0047] Figure 8a and 8b is a schematic diagram showing a case where health information is provided using a result of a food recognition method according to an embodiment;
[0048] Figure 9a , 9b 9c are schematic diagrams showing a food identification method according to an embodiment;
[0049] Fig.10 is a flow chart of a process for performing a food identification method according to one embodiment;
[0050] Fig.11 is a flowchart of a process of executing a food identification method according to an embodiment;
[0051] Fig.12 is a perspective view schematically showing a refrigerator having a camera and a display according to one embodiment;
[0052] Fig.13 is a front view schematically showing a refrigerator with all doors opened according to one embodiment; and
[0053] Fig.14 is a schematic diagram briefly showing a refrigerator according to one embodiment. DETAILED DESCRIPTION
[0054] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it will be appreciated that the present disclosure is not limited to the embodiments described below, but includes various modifications, equivalents and / or alternative forms of the embodiments of the present disclosure. With regard to the explanation of the accompanying drawings, similar reference numerals may be used for similar constituent elements.
[0055] In the specification, the terms “have”, “may have”, “include” or “may include” indicate the existence of corresponding features (eg, values, functions, operations, or constituent elements such as components), but do not exclude the existence of additional features.
[0056] In the specification, the term "A or B", "at least one of A or / and B", or "one or more of A or / and B" may include all possible combinations of the items listed together. For example, the term "A or B" or "at least one of A or / and B" may mean (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.
[0057] In the specification, the terms "first, second", etc. are used to describe different elements regardless of their order and / or importance, and are used to distinguish one element from other elements, without being limited to the corresponding elements. For example, a first user device and a second user device may indicate different user devices regardless of their order or importance. For example, a first element may be referred to as a second element, or similarly, a second element may be referred to as a first element without departing from the scope described herein.
[0058] Terms such as "module", "unit", "part", etc. are used to refer to elements that perform at least one function or operation, and such elements may be implemented as hardware or software, or a combination of hardware and software. In addition, except when each of a plurality of "modules", "units", "parts", etc. needs to be implemented in separate hardware, the components may be integrated in at least one module or chip, and implemented in at least one processor.
[0059] If it is described that one element (e.g., a first element) is “operably or communicatively coupled” to / is “operably or communicatively coupled to” or is “connected to” another element (e.g., the second element), it is understood that the element may be connected to the other element directly or through another element (e.g., a third element). When it is mentioned that one element (e.g., a first element) is “directly coupled” to or is “directly connected to” another element (e.g., the second element), it is understood that there is no element (e.g., the third element) between the element and the other element.
[0060] The terms used in the specification are used to describe one embodiment, but may not be intended to limit the scope of other embodiments. Unless otherwise specifically defined, singular expressions may cover plural expressions. All terms (including technical and scientific terms) used in the specification may be used as meanings commonly understood by those of ordinary skill in the art to which the present disclosure belongs. The terms used in the present disclosure and defined in a general dictionary may be used as meanings identical or similar to the meanings of the terms in the context from the relevant technology, and unless they have been clearly and specifically defined, they are not ideally or excessively interpreted. Depending on the circumstances, even the terms defined in the embodiments of the present disclosure may not be interpreted as excluding the embodiments of the present disclosure.
[0061] One of the methods of using the display included in the refrigerator may be to display the food stored in the refrigerator on the display. For example, the refrigerator may display an image acquired using a camera capturing the refrigerator storage compartment on the display, identify the food included in the image, and display the name of the food, etc., together with the image on the display.
[0062] The camera can shoot the storage room to identify the food included in the image generated by shooting the storage room, so it is inconvenient to perform food recognition every time the image of the inside of the refrigerator is generated. In addition, there may be a situation where the food placed overlapping in the refrigerator cannot be accurately recognized.
[0063] Therefore, a method is needed to maintain the efficiency of food identification and identify foods that are overlapped and placed in a refrigerator.
[0064] Figure 1a and 1b is a schematic diagram for explaining a food identification method according to an embodiment.
[0065] exist Figure 1a and 1b In the present invention, the food recognition method may be executed in a server (or cloud) 20 including one or more servers, or may be executed in the refrigerator 10.
[0066] According to an embodiment, the refrigerator 10 and the server 20 can establish a communication network by using a wired or wireless communication method. The refrigerator 10 and the server 20 may include a wireless communication interface (e.g., a cellular communication module, a near-field wireless communication module, or a global navigation satellite system (GNSS) system communication module) or a wired communication interface (e.g., a local area network (LAN) communication module, or a power line communication module), and can communicate through a first network (e.g., a near-field communication network such as Bluetooth, Wi-Fi Direct, or an infrared data association (IrDA)) or a second network (e.g., a far-field communication network such as a cellular network, the Internet, or a computer network (e.g., LAN or WAN)) by using the corresponding communication interface.
[0067] According to one embodiment, the refrigerator 10 may be a household appliance including a storage chamber for storing food, and a cooling air supply device for supplying cooling air to the storage chamber, and capable of keeping the food fresh. The refrigerator 10 may further include a camera. According to one embodiment, the refrigerator 10 may capture the food stored in the storage chamber and obtain an image by using a camera.
[0068] According to one embodiment, the server 20 may include multiple servers, including a first server, a second server, and a third server. For example, the first server may compare multiple images and identify the changed area. The second server may, for example, identify the food in the image. The third server may, for example, identify the processed food.
[0069] According to an embodiment, the server 20 may further include a storage server for storing a plurality of images.
[0070] Hereinafter, it is assumed that the food recognition method is performed in the server 20 .
[0071] In the exam Figure 1a In the example, the server 20 may be in a state of pre-storing the image 120 to be compared. Figure 1a In the embodiment, the refrigerator 10 can obtain the refrigerator interior image 110.
[0072] According to one embodiment, the refrigerator interior image 110 may be an image acquired using a camera included in the refrigerator 10. For example, the refrigerator interior image 110 may be an image generated by capturing a storage room within the refrigerator using a camera included in the refrigerator.
[0073] According to one embodiment, the image to be compared 120 may be an image that is recognized by receiving an image of the interior of the refrigerator from the refrigerator 10 in the server 20 and detecting an object contained in the image of the interior of the refrigerator. Figure 1a The refrigerator interior image 110 is an image previously captured by a camera in the refrigerator 10 , transmitted to the server 20 , and recognized and stored by the server 20 .
[0074] According to one embodiment, when the occurrence of an event is sensed, the refrigerator 10 can acquire at least one image of the interior of the refrigerator. For example, when an event occurs in which the door is opened and then closed, the refrigerator 10 can acquire at least one image of the interior of the refrigerator. However, this example is not limited to this. For example, the refrigerator 10 can capture at least one image of the interior of the refrigerator according to a predetermined cycle, or can sense that someone is located in front of the refrigerator 10 and capture at least one image of the interior of the refrigerator.
[0075] According to one embodiment, the refrigerator 10 may transmit at least one acquired refrigerator interior image 110 to the server 20 .
[0076] The server 20 can identify the changed area that has changed by comparing the pre-stored image to be compared 120 with the refrigerator interior image 110. For example, the server 20 can change the image to be compared 120 and the refrigerator interior image 110 into images that emphasize edge components, and then compare the acquired object shapes based on the edge components and identify the changed area.
[0077] According to one embodiment, the server 20 may identify the changed area by using pixel difference values. For example, the server 20 may calculate the corresponding differences between the red pixel value (R), green pixel value (G), and blue pixel value (B) included in the image to be compared 120 and the R, G, and B values of the refrigerator interior image 110, and identify the changed area.
[0078] According to one embodiment, the server 20 may identify the changed area by using an artificial intelligence algorithm. For example, the server 20 may apply at least one refrigerator interior image 110 to a food change sensing model trained with an image, an image identical to the image but including at least some parts with changes, and information on the change position as learning data, and compare the food change sensing model applied to at least one refrigerator interior image 110 with the image to be compared 120. Thus, the server 20 may identify the changed area in the refrigerator interior image 110.
[0079] refer to Figure 1a , when there is no changed area in the refrigerator interior image during identification, the server 20 may send the first food storage information obtained by identifying the pre-stored image to be compared and the refrigerator interior image 110 to the refrigerator. The first food storage information may be, for example, information generated using the result of sensing the object in the image to be compared 120. In addition, the first food storage information may include, for example, any one or any combination of the location information of the food sensed in the image to be compared 120, the information of the food category to which the food belongs, the information of the processed food category to which the processed food belongs, and the brand name of the processed food.
[0080] refer to Figure 1a , the server 20 may not obtain the changed area obtained by comparing the refrigerator interior image 110 with the image to be compared 120. Accordingly, the server 20 may obtain the first food storage information as the information to be sent to the refrigerator 10. For example, the server 20 may transmit grapes as food category information and position information about grapes 122 in the image to be compared 120 as the first food storage information to the refrigerator 10. In addition, the server 20 may transmit the refrigerator interior image 110 received from the refrigerator 10 to the refrigerator 10.
[0081] According to an embodiment, the server 20 may perform image correction processing on the refrigerator interior image 110 received from the refrigerator 10, and then transmit the refrigerator interior image 110 on which the image correction processing is performed to the refrigerator 10. The server 20 may perform image processing to correct blur that occurs when the camera performs capture. For example, the server 20 may correct blur by using a cross correction algorithm between the previous image and the current image.
[0082] According to an embodiment, the server 20 may apply image processing to the refrigerator interior image 110 to improve the clarity of an opaque image that occurs due to condensation or the like.
[0083] By performing image processing, the server can detect an object with high accuracy from the refrigerator interior image 110. The refrigerator 10 can always or selectively receive the refrigerator interior image 110 corrected by the server 20 and display the received refrigerator interior image on the display 11.
[0084] According to one embodiment, the server 20 can correct the refrigerator interior image by using an artificial intelligence model. For example, the server 20 can obtain an image with condensation eliminated by applying at least one refrigerator interior image 110 to a food change sensing model trained using an image, an image to which a condensation effect is added, and an image expressing only the condensation effect as learning data.
[0085] According to one embodiment, the refrigerator 10 may display the refrigerator interior image 110 and the first food storage information received from the server on the display 11 .
[0086] exist Figure 1b In the example, the server 20 may be in a state of pre-storing the image 140 to be compared. Figure 1b In the embodiment of the present invention, the refrigerator 10 can obtain the refrigerator interior image 130.
[0087] As reference Figure 1a , when an event (eg, an event in which a door is opened and then closed) occurs, the refrigerator 10 may acquire at least one refrigerator interior image 130 .
[0088] According to one embodiment, the refrigerator 10 may send at least one refrigerator interior image 130 to the server 20. The server 20 may identify a changed region in which a change has occurred by comparing the pre-stored image to be compared 140 with the refrigerator interior image 130.
[0089] For example, the server 20 may use reference Figure 1a The described changed region recognition method compares the refrigerator interior image 130 with the to-be-compared image 140 , and senses that the object 134 is added to the refrigerator interior image 130 , and recognizes the changed region 132 .
[0090] According to one embodiment, as a result of recognition, when a changed area exists in the refrigerator interior image, the server 20 can send the second food storage information obtained using the result of recognizing the object contained in the refrigerator interior image 130 and the first food storage information, as well as the refrigerator interior image 130 to the refrigerator.
[0091] According to one embodiment, the server 20 may change the image to be compared 140 and the refrigerator interior image 130 into images that emphasize edge components, then find the shape of the acquired object based on the edge components, compare the shape of the object with the data stored in the server 20, and identify the food category to which the object belongs. The food category may be, for example, a name that can represent similar foods.
[0092] According to one embodiment, when the object is identified as a processed food, the server 20 may identify a category of processed food to which the processed food belongs and / or a brand name of the processed food by using additional processed food data additionally stored in the server 20 .
[0093] According to one embodiment, the server 20 can sense an object by using an artificial intelligence algorithm and identify the food category to which the sensed object belongs. For example, the server 20 can sense an object by applying the refrigerator interior image 130 to a food recognition model trained using the image of the food, the food category to which the food belongs, and information about the location of the food in the image as learning data, and can identify the food category to which the object belongs.
[0094] According to one embodiment, as a result of identifying the object by using the food recognition model, when the object is identified as a processed food, the server 20 can identify the category of the processed food to which the object belongs and / or the brand name of the processed food by using the processed food recognition model. For example, the server 20 can identify the category of the processed food detected in the refrigerator interior image 130 and / or the brand name of the processed food in the processed food database by using the processed food recognition model trained with the image similarity based on multiple images of the same processed food, the category of the processed food, and the brand name of the processed food as learning data. The processed food database can be, for example, a database that stores the category of the processed food and the brand name of the processed food using images of various processed foods.
[0095] refer to Figure 1b , the server 20 may sense and identify the object 134 added to the refrigerator interior image 130. For example, the server 20 may sense and identify the coffee beverage as the category information of the object 134 added to the refrigerator interior image 130. In addition, the server 20 may acquire information about the location of the coffee beverage in the refrigerator interior image 130.
[0096] According to an embodiment, when sensing an object in the refrigerator interior image, the server 20 may sense only the object in the changed area. For example, the server 20 may sense and recognize only the object in the changed area 132. For this reason, the server 20 may reduce the data processing load compared to sensing and recognizing the object of the entire area of the refrigerator interior image 130.
[0097] As described above, the server 20 may recognize an object in the refrigerator interior image, and further acquire second food storage information by using the first food storage information.
[0098] According to one embodiment, the server 20 may identify the presence of the object 142 covered by the coffee beverage in the refrigerator interior image 130. The server 20 may obtain information of the objects sensed in the first food storage information and location information of each object. For example, the server 20 may identify the presence of grapes where the object 142 is placed in the image to be compared 140.
[0099] As described above, the server 20 can obtain the second food storage information by using the result of detecting a new object in the refrigerator interior image 130 (e.g., information that a coffee beverage is added) and the information obtained from the first food storage information (e.g., information that grapes exist where the coffee beverage is placed). The server 20 can send the obtained second food storage information and the refrigerator interior image 130 to the refrigerator 10.
[0100] According to one embodiment, the refrigerator 10 may display the refrigerator interior image 130 and the second food storage information received from the server on the display 11 .
[0101] According to an embodiment, the refrigerator 10 may include a plurality of cameras. For example, the refrigerator 10 may include a first camera for photographing a first storage chamber, and a second camera for photographing a second storage chamber. However, this example is not limited thereto. For example, the refrigerator 10 may include a number of cameras corresponding to the number of storage chambers included in the refrigerator 10.
[0102] The refrigerator 10 may acquire the first refrigerator interior image and the second refrigerator interior image by using the first camera and the second camera. The refrigerator 10 may transmit the first refrigerator interior image and the second refrigerator interior image to the server 20. The server 20 may transmit the first refrigerator interior image and the second refrigerator interior image by using the reference image. Figure 1a and 1b A process is described to identify change regions for a first refrigerator interior image and a second refrigerator interior image.
[0103] For example, in the case where the changed area does not exist in the first refrigerator interior image, the server 20 may send 1-1 food storage information (e.g., information obtained using the object included in the first image to be compared) and the first refrigerator interior image to the refrigerator with respect to the first storage room. In addition, in the case where the changed area exists in the second refrigerator interior image, the server 20 may send 2-2 food storage information obtained using the result of identifying the object included in the object included in the second refrigerator interior image and 1-2 food storage information (e.g., information obtained using the object included in the second image to be compared) and the second refrigerator interior image to the refrigerator with respect to the second storage room.
[0104] Thus, according to one embodiment, the food recognition method can use the pre-stored data according to the recognition result of comparing the pre-stored image to be compared with the newly acquired refrigerator interior image, or can use the data obtained from the newly acquired refrigerator interior image and the pre-stored data at the same time. Thus, the recognition rate of the object contained in the refrigerator interior image can be improved, and the load of the server 20 can be reduced.
[0105] Figure 2a and 2b is a block diagram of a brief configuration of a refrigerator and a server according to one embodiment.
[0106] refer to Figure 2a According to one embodiment, the refrigerator 10 may include a processor 210, a communication interface 220, a camera 230, and a memory 240. However, the embodiments are not limited to any particular example. For example, the refrigerator 10 may include additional elements, or may not include some elements.
[0107] According to one embodiment, the processor 210 may control the overall operation of the refrigerator 10. For example, the processor 210 may receive user input through the communication interface 220. The processor 210 may control the memory 240 to execute a stored program and call or store information.
[0108] According to one embodiment, when a predetermined event occurs, the processor 210 may control the camera 230 to capture at least one image of the interior of the refrigerator. The processor 210 may compare the pre-stored image to be identified with the image of the interior of the refrigerator, and identify the changed area where the change has occurred. In the case where the changed area does not exist in the image of the interior of the refrigerator, the processor 210 may obtain the first food storage information as a result of identifying the image to be compared. In the case where the changed area exists in the image of the interior of the refrigerator, the processor 210 may control the communication interface 220 to send the image of the interior of the refrigerator to the server 20, and when the server 20 receiving the image of the interior of the refrigerator uses the result of identifying the object in the image of the interior of the refrigerator and the first food storage information to obtain the second food storage information, the obtained second food storage information is received from the server 20.
[0109] According to one embodiment, the communication interface 220 can be controlled by the processor 210 to connect the refrigerator 10 to an external device. The communication interface 220 may include one of the elements implementing various wired or wireless communication methods such as wireless LAN, Bluetooth, wired Ethernet, etc. to correspond to the performance and structure of the refrigerator 10.
[0110] According to one embodiment, the memory 240 may include at least one type of storage medium selected from flash memory type, hard disk type, multimedia card micro and card type memory (e.g., Secure Digital (SD) memory, XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk and optical disk.
[0111] According to one embodiment, when a predetermined event occurs, the memory 240 can control the camera 230 to capture at least one image of the interior of the refrigerator, compare the pre-stored image to be compared with the image of the interior of the refrigerator, and obtain the changed area that has changed. When the changed area does not exist in the image of the interior of the refrigerator, the first food storage information is obtained. When the changed area exists in the image of the interior of the refrigerator, the communication interface 220 is controlled to send the image of the interior of the refrigerator to the server 20, and an instruction is stored in the server 20 for receiving food storage information obtained by the server 20 using the result of identifying an object from the image of the interior of the refrigerator and the first food storage information.
[0112] According to one embodiment, the camera 230 may be included in the refrigerator 10 and photograph the storage room of the refrigerator 10. However, this example is not limited thereto. For example, the camera 230 may photograph the storage room in the refrigerator 10, as well as the outside of the refrigerator 10. The camera 230 may include a first camera, a second camera, or a third camera. For example, the first camera may photograph the first storage room. The second camera may photograph the second storage room, and the third camera may photograph the third storage room.
[0113] According to an embodiment, the camera 230 may be provided on the door of the refrigerator 10. However, the embodiments are not limited to any specific example.
[0114] refer to Figure 2b , the server 20 may include a memory 250 and a processor 260 .
[0115] According to one embodiment, the processor 260 may acquire at least one image of the interior of the refrigerator taken in the refrigerator. For example, the processor 260 may receive at least one image of the interior of the refrigerator from the refrigerator 10 through a communication interface or module.
[0116] According to one embodiment, the processor 260 may compare the pre-stored image to be compared with the image inside the refrigerator, and identify the changed area.
[0117] According to one embodiment, when the change area does not exist, the processor 260 may transmit the first food storage information and the refrigerator interior image as a result of recognizing the pre-stored image to be compared to the refrigerator.
[0118] According to one embodiment, when a changed area exists in the refrigerator interior image, the processor 260 may obtain second food storage information using the result of identifying an object contained in the refrigerator interior image and the first food storage information, and the processor 260 may send the second food storage information and the refrigerator interior image to the refrigerator.
[0119] Figure 3 is a block diagram of a processor included in at least one of the refrigerator 10 or the server 20 according to one embodiment.
[0120] refer to Figure 3 According to the embodiment, the processor 310 may include a data learning part 320 and a data identification part 330. The processor 310 may include Figure 2a A processor 210, and Figure 2b Processor 260.
[0121] According to one embodiment, the data learning part 320 can train the food change sensing model to establish a standard for identifying the changed area in the image. The data learning part 320 can train the food change sensing model to establish a standard about which learning data will be used to identify the changed area in the image or how to use the learning data to identify the changed area in the image.
[0122] According to one embodiment, the data learning part 320 is data for learning, which can train a food change sensing model by using an image, an image that is the same as the image but contains changes in at least some regions, and location information of the changes.
[0123] For example, data used for learning may be an image of a storage room with milk and apples, an image of cucumbers being added to the storage room, and information related to the position where the cucumbers were added.
[0124] According to one embodiment, the data learning part 320 can train the food change sensing model to have a standard for correcting the image inside the refrigerator. The data learning part 320 can train the food change sensing model to establish a standard about which learning data will be used to correct the image or how to use the learning data to correct the image.
[0125] According to one embodiment, the data learning part 320 is data for learning, which can train a food change sensing model by using an image, an image to which a coagulation effect is added, and an image in which only the coagulation effect is expressed.
[0126] According to one embodiment, the data recognition part 330 can recognize the changed area in the image based on various types of data for recognition. The data recognition part 330 can compare at least one refrigerator interior image input using the trained food change sensing model with the image to be compared and recognize the changed area.
[0127] According to one embodiment, the data recognition part 330 can use the result of recognizing the changed area in the at least one refrigerator interior image and the user response (or feedback) to the recognition result to update the food change sensing model that uses the at least one refrigerator interior image and the image to be compared as input values of the food change sensing model. Therefore, the food change sensing model can increase the probability of recognizing the changed area according to the user's tendency.
[0128] For example, when an image to be compared including oranges and apples and an image of the interior of a refrigerator which is the same as the image but receives an input of an image of the interior of a refrigerator containing melons are input, the data recognition part 330 can identify an area where melons are added from the image containing melons.
[0129] According to one embodiment, the data recognition part 330 may correct the image based on various types of data for recognition. The data recognition part 330 may correct at least one refrigerator interior image input using the trained food change sensing model.
[0130] According to one embodiment, the data recognition part 330 can use the result of correcting at least one refrigerator interior image and the user response (or feedback) to the correction result to update the food change sensing model that uses the input at least one refrigerator interior image as an input value of the food change sensing model. Therefore, the food change sensing model can correct the image according to the user's inclination.
[0131] For example, in the case where an image containing bananas in which condensation has occurred is input as the image of the interior of the refrigerator, the data recognition part 330 can correct the image into an image in which the bananas are clearly visible by eliminating the condensation.
[0132] According to one embodiment, the data learning part 320 can train the food change sensing model to establish a standard for detecting and identifying food categories in an image. The data learning part 320 can train the food recognition model to establish a standard about which learning data will be used to detect and recognize (or identify) food categories in an image or how to use the learning data to detect and recognize food categories in an image.
[0133] According to one embodiment, the data learning part 320 is data for learning, which can train a food recognition model using information about images containing food, food categories to which the food belongs, and locations of the food in the images.
[0134] For example, the data used for learning may be an image of a pantry containing cucumbers and apples, food categories called "cucumbers" and "apples" (e.g., the names of cucumbers and apples), and location information of the cucumbers and apples in the image.
[0135] According to one embodiment, the data recognition part 330 may recognize the category of food included in the image based on various types of data for recognition. The data recognition part 330 may sense and recognize (or identify) the category of food included in the image based on the image input using the trained food recognition model.
[0136] According to one embodiment, the data recognition part 330 can use the result of recognizing the food category included in the image and the user's response to the recognition result to update the food recognition model that uses the input image as the input value of the food recognition model. Therefore, the food recognition model can increase the probability of recognizing the object according to the user's inclination.
[0137] For example, in the case where an image containing oranges and apples is input, the data recognition part 330 may sense and recognize the oranges and apples, and acquire food categories "oranges" and "apples" from the image.
[0138] According to one embodiment, the data learning part 320 can train the processed food recognition model to establish a standard for finding the same product image in the processed food database. The data learning part 320 can train the processed food recognition model to establish a standard about which learning data will be used to search (or identify) processed food in an image or how to use the learning data to search and identify food in an image.
[0139] According to one embodiment, the data learning part 320 is data for learning, which can train a processed food search model using a plurality of images of processed foods, categories of processed foods, and brand names of processed foods.
[0140] For example, the data used for learning could be images of various types of milk, a category of processed food called "milk," and the brand name of each type of milk.
[0141] The data recognition part 330 may recognize the category of processed food contained in the image based on various types of data for recognition. The data recognition part 330 may sense and recognize (or identify) the category of processed food contained in the image based on the image input using the trained processed food recognition model.
[0142] According to one embodiment, the data recognition part 330 can update the processed food recognition model that uses the input image as an input value of the processed food recognition model using the result of recognizing the category of the processed food contained in the image and the user's response to the recognition result. Therefore, the processed food recognition model can increase the probability of recognizing the object according to the user's inclination.
[0143] For example, in the case where an image containing milk is input, the data recognition part 330 can sense and recognize the milk, and acquire the food category "milk" and the brand name of the milk from the image.
[0144] The food change sensing model, the food recognition model, and the processed food recognition model can be constructed taking into account the application field of the recognition model, the learning purpose, or the computer performance of the device. The food change sensing model, the food recognition model, and the processed food recognition model can be, for example, a model based on a neural network. The food change sensing model, the food recognition model, and the processed food recognition model can include, for example, a neural network model or a deep learning model developed from a neural network. Multiple network nodes in the deep learning model can be located at different depths (or layers) from each other, and data can be exchanged according to a convolutional connection relationship. For example, food change sensing models, food recognition models, and processed food recognition models such as a deep neural network (DNN), a recurrent neural network (RNN), and a bidirectional recurrent deep neural network (BDNR) can be used, but are not limited to this.
[0145] One or both of the data learning part 320 and the data identification part 330 can be manufactured in the form of at least one hardware chip and installed on the electronic device. For example, one or both of the data learning part 320 and the data identification part 330 can be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or can be manufactured as part of an earlier general-purpose processor (e.g., CPU or application processor) or a graphics-dedicated processor (e.g., GPU) and installed in the above-mentioned various electronic devices.
[0146] According to one embodiment, a dedicated hardware chip for artificial intelligence may be a dedicated processor dedicated to probabilistic computing and may exhibit higher performance than earlier general-purpose processors to facilitate processing of computational operations in artificial intelligence fields such as machine learning.
[0147] The data learning part 320 and the data identification part 330 may be installed on the same electronic device or on different electronic devices respectively. For example, one of the data learning part 320 and the data identification part 330 may be included in the refrigerator 10, and the other may be included in the server 20. In addition, one of the data learning part 320 and the data identification part 330 may be included in a first server, and the other may be included in a second server different from the first server. The data learning part 320 and the data identification part 330 may provide the data identification part 330 with information about the food change sensing model, the food identification model, and the processed food identification model constructed by the data learning part 320, or the data input to the data identification part 330 may be provided to the data learning part 320 as additional learning data via wired or wireless.
[0148] One or both of the data learning part 320 and the data identification part 330 may be implemented as a software module. When one or both of the data learning part 320 and the data identification part 330 are implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable medium. In addition, in this case, at least one software module may be provided by an operating system (OS) or a predetermined application program. Alternatively, a portion of at least one of the at least one software module may be provided by an operating system (OS), and some of the at least one software module may be provided by a predetermined application program.
[0149] Figure 4a and 4b is a schematic diagram illustrating a case where a plurality of cameras are used in a food recognition method according to an embodiment.
[0150] According to one embodiment, the refrigerator 10 may include a plurality of cameras for photographing the storage compartment. For example, the refrigerator 10 may include a first and a second camera for photographing the storage compartment. Each camera may include a lens module capable of photographing a viewing angle of approximately 150 to 180 degrees. Fig.13 Description Provides example locations for cameras.
[0151] According to one embodiment, the first camera may photograph the first storage compartment 401, and the second camera may photograph the second storage compartment 402 located below the first storage compartment. According to an embodiment, the shelf 430 between the first storage compartment 401 and the second storage compartment 402 may be made of transparent glass or plastic material.
[0152] Figure 4a The first refrigerator interior image 410 is a first refrigerator interior image 410. The first refrigerator interior image 410 may be, for example, an image of the first storage room 401 captured using a first camera. Figure 4bis a partial area of the second refrigerator interior image 420. The second refrigerator interior image 420 may be, for example, an image of the second storage room 402 taken using a second camera. Figure 4b An indication of the second refrigerator interior image 420 may be a partial area of an area where the first storage chamber 401 is visible through the transparent shelf 430 from the second refrigerator interior image 420 acquired by the second camera.
[0153] refer to Figure 4a , the server 20 is in a state of identifying the changed region 412 in the first refrigerator interior image 410. For example, the server 20 may identify the changed region in the first refrigerator interior image 410 using the first refrigerator interior image 410 and the pre-stored first image to be compared.
[0154] According to an embodiment, the refrigerator 10 may identify a changed area in the first refrigerator interior image 410. For example, the server 20 may identify a changed area in the first refrigerator interior image 410 using the first refrigerator interior image 410 acquired using a camera and a first image to be compared pre-stored in the refrigerator 10. In this case, the first image to be compared may be an image captured using a first camera and transmitted to the server that detects and identifies food, and then transmitted to the refrigerator 10.
[0155] As mentioned above, refer to Figure 1b , in the case where the changed area in the refrigerator interior image is obtained, the server 20 can transmit the second food storage information obtained using the result of identifying the object contained in the refrigerator interior image and the first food storage information and the refrigerator interior image to the refrigerator.
[0156] According to an embodiment, when the server 20 stores a plurality of refrigerator interior images, the server 20 may use the plurality of refrigerator interior images to identify objects contained in the refrigerator interior images. That is, when an object contained in a first refrigerator interior image is identified, a second refrigerator interior image may be used.
[0157] For example, the server 20 may sense and recognize the wine bottle 414 as category information of processed food in the first refrigerator interior image 410. The server 20 may recognize the presence of an object (eg, apple 422) covered by the wine bottle 414 in the first refrigerator interior image.
[0158] The server 20 may sense and recognize objects in the second refrigerator interior image 420. For example, the server 20 may sense and recognize wine bottle 414 and apple 422 in the second refrigerator interior image 420. The server 20 may recognize that the wine bottle 414 and apple 422 are foods or processed foods stored in the first storage chamber 401 based on the fact that the wine bottle 414 and apple 422 are recognized on the upper side of the second refrigerator interior image 420. Alternatively, the server 20 may recognize the shelf 430, and recognize that the wine bottle 414 and apple 422 are foods stored in the first storage chamber 401 based on the fact that the wine bottle 414 and apple 422 are placed on the shelf 430.
[0159] According to one embodiment, server 20 may identify object 422 placed behind wine bottle 414 in first refrigerator interior image 410 based on the result of identifying apple 422 in second refrigerator interior image 420 .
[0160] Therefore, a food item hidden by another food item can be identified by combining multiple images of the refrigerator interior.
[0161] Figure 5a , 5b 5C are schematic diagrams showing a case where a food category is determined using a result of a food recognition method according to an embodiment.
[0162] Referring to FIG. 5a, the refrigerator 10 may display the food stored in the refrigerator 10 on the display 510 based on the acquired at least one refrigerator interior image and the food storage information. The display 510 may include Figure 1a Display 11.
[0163] According to one embodiment, the refrigerator 10 may display food and the food category identified using the food identification method on the display 510. For example, regarding the categories of apple 501, spinach 503, and grape 505 being identified, the refrigerator 10 may display the categories "apple", "spinach", and "grape" adjacent to the corresponding food.
[0164] According to an embodiment, the refrigerator 10 may use a food recognition method to provide candidate categories in which the recognized food may be included.
[0165] refer to Figure 5a , the refrigerator 10 may receive a user input 530 to select an object 520 from the food displayed on the display 510 .
[0166] refer to Figure 5b , the refrigerator 10 may display candidate categories 540 that may include the selected object 520 on the display 510. The candidate categories 540 may be, for example, as shown in FIG. Figure 1bAs described, the categories are identified by the server 20 by sensing the objects in the image of the interior of the refrigerator.
[0167] According to one embodiment, the refrigerator 10 may receive a user input to select one of the candidate categories 540. That is, the user may determine and select a category to which the object 520 belongs from the candidate categories 540 displayed on the display 510.
[0168] refer to Figure 5c , the refrigerator 10 may determine the category 541 input by the user as the category of the detected object 520 , and display the determined category on the display 510 .
[0169] According to one embodiment, the category 541 of the object 520 selected by the user can be used as data for learning. For example, the refrigerator 10 can send the object 520 of the category 541 selected by the user for the object 520 to the server 20. The server 20 can apply the received data to the data learning part 320 as learning data. The data learning part 320 can train the food recognition model by using the received data. As the data generated by the user feedback is used as the number of learning data of the food recognition model increases, the possibility of the food recognition model recognizing the object according to the user's tendency can be increased.
[0170] Figure 6a and 6b is a schematic diagram illustrating a case where a recipe is acquired using a result of a food recognition method according to an embodiment.
[0171] refer to Figure 6a , the refrigerator 10 can display the food stored in the refrigerator 10 on the display 510 based on the at least one refrigerator interior image acquired. The refrigerator 10 can display the food and the food category identified using the food identification method on the display 510.
[0172] According to one embodiment, the refrigerator 10 may provide a recipe by using the recognized category of food.
[0173] refer to Figure 6a , the refrigerator 10 may display an execution object 610 that provides a recipe to the display 510. The refrigerator 10 may receive a user input 620 to select the execution object 610 that provides a recipe.
[0174] refer to Figure 6b , the refrigerator 10 may display a recipe 630 of food that can be prepared using the stored food on the display 510 according to a user input of selecting the execution object 610 .
[0175] For example, the refrigerator 10 may transmit the categories of food stored in the server providing the recipe, and receive the recipe 630 acquired by the server using the received categories of food, and display the received recipe on the display 510 .
[0176] According to an embodiment, the refrigerator 10 can set a priority for a food category and send the set priority to the server. For example, the refrigerator 10 can set a high priority for a food category with the earliest date of the food stored in the refrigerator 10. Thus, the refrigerator 10 can first provide the user with food with a longer shelf life.
[0177] According to an embodiment, the refrigerator 10 may store various recipes in a memory (eg, the memory 240 of FIG. 2 ). In this case, the refrigerator 10 may display a recipe 630 of food that can be prepared using the food category received from the server 20 on the display 510 according to a user request.
[0178] Figure 7a and 7b FIG. 1 is a schematic diagram showing a case where a product is purchased using a result of a food recognition method according to an embodiment.
[0179] refer to Figure 7a , the refrigerator 10 can display the food stored in the refrigerator 10 on the display 510 based on the at least one refrigerator interior image acquired. The refrigerator 10 can display the food and the food category identified using the food identification method on the display 510.
[0180] According to one embodiment, the refrigerator 10 may use the identified food categories to provide guidance for food purchases.
[0181] refer to Figure 7a , the refrigerator 10 may display an execution object 710 linked to a food purchase website on the display 510. The refrigerator 10 may receive a user input 720 to select the execution object 710 linked to the food purchase website.
[0182] refer to Figure 7b , the refrigerator 10 may display a food purchase website 730 on the display 510 according to a user input 720 selecting the execution object 710 .
[0183] For example, the refrigerator 10 may access a food purchase website that sells food categories corresponding to food that has been previously stored in the refrigerator 10 but is not currently stored in the refrigerator 10. Alternatively, the refrigerator 10 may access a food purchase website that sells food categories suitable for the current season using weather information.
[0184] According to an embodiment, the refrigerator 10 can access a food purchasing website that uses information on foods used in recipes displayed on the display 510 of Figure 6 to identify foods used in recipes frequently selected by the user but not currently stored in the refrigerator 10, and sell foods included in the identified categories.
[0185] Figure 8a and 8b is a schematic diagram illustrating a case where health information is provided using a result of a food recognition method according to an embodiment.
[0186] refer to Figure 8a , the refrigerator 10 can display the food stored in the refrigerator 10 on the display 510 based on the at least one refrigerator interior image acquired. The refrigerator 10 can display the food and the food category identified using the food identification method on the display 510.
[0187] According to one embodiment, the refrigerator 10 may provide guidance of the user's health information using the identified food category.
[0188] refer to Figure 8a , the refrigerator 10 may display an execution object 810 displaying health information on the display 510. The refrigerator 10 may receive a user input 820 to select the execution object 810 displaying health information.
[0189] refer to Figure 8b , the refrigerator 10 may display the health information 830 on the display 510 according to the user input of selecting the execution object 810 .
[0190] For example, the refrigerator 10 may store calories contained in food stored in the refrigerator 10. The refrigerator 10 may estimate the user's calorie intake according to the reduction or disappearance of food, and provide the estimated calories as health information.
[0191] According to an embodiment, the refrigerator 10 can estimate the calorie intake of the user in breakfast, lunch and dinner using the time when the food decreases or disappears, and provide the estimated calories as health information. Alternatively, the refrigerator 10 can estimate the calorie intake of the user during a week or a month using the time when the food decreases or disappears, and provide the estimated calories as health information.
[0192] According to an embodiment, the refrigerator 10 may receive the user's exercise amount information from a personal device or a wearable device capable of measuring exercise amount. The refrigerator 10 may include the received exercise amount information (eg, consumed calories, etc.) in the health information 830 and provide the health information.
[0193] In FIGS. 5-8 , it is assumed that the display 510 included in the refrigerator 10 is used, but the example is not limited thereto. For example, the server 20 may transmit the first food storage information or the second food storage information to another electronic device (e.g., a smart phone, etc.) instead of the refrigerator 10. In this case, the user may use the display included in the other electronic device to perform the operations described in FIGS. 5-8 .
[0194] Figure 9a , 9b 9c are schematic diagrams illustrating a food identification method according to an embodiment.
[0195] Reference Figures 9a to 9c , the refrigerator 10 can use the continuously acquired refrigerator interior images to improve the performance of identifying food types.
[0196] Reference Figure 9a , the refrigerator 10 can obtain the refrigerator interior image 901 at time point t. For example, the refrigerator 10 can obtain the refrigerator interior image 901 at time point t when the door is open. The refrigerator 10 can send the refrigerator interior image 901 at time point t to the server 20, and receive food storage information generated by the server 20 by identifying the food in the refrigerator interior image 901 at time point t. However, the present example is not limited to this. For example, food storage information can be generated by using a processor included in the refrigerator 10 (e.g., the processor 210 of Figure 2) to identify the food included in the refrigerator interior image 901 at time point t. Figures 9a to 9c In the example, it is assumed that food is recognized by the refrigerator 10.
[0197] According to one embodiment, the refrigerator 10 may detect and recognize an object in the refrigerator interior image 901 at time point t. For example, the refrigerator 10 may detect the processed food category of milk 910. In this case, the refrigerator 10 may not detect an object placed behind the milk 910.
[0198] Figures 9b to 9c The image is generated by photographing the state where milk 910 is moved by the user. For example, the refrigerator 10 may obtain the refrigerator interior image 903 at time t+1, and obtain the refrigerator interior image 905 at time t+2. The refrigerator 10 may detect milk 910 from each of the refrigerator interior image 903 at time t+1 and the refrigerator interior image 905 at time t+2.
[0199] In addition, the refrigerator 10 can combine the shape of a portion of the object 920 placed behind the milk 910 in the refrigerator interior image 903 at time point t+1 with the shape of a portion of the object 920 placed behind the milk 910 in the refrigerator interior image 905 at time point t+2, and obtain the complete shape of the object 920.
[0200] The refrigerator 10 may use the acquired shape of the object 920 to recognize that the object 920 belongs to the food category of “apple”.
[0201] Therefore, the refrigerator 10 may recognize the kind of food using the continuous images acquired through the camera.
[0202] Fig.10 is a flowchart of a process of executing a food identification method according to an embodiment.
[0203] Referring to operation 1010, the server 20 may acquire at least one refrigerator interior image. For example, the server 20 may receive a refrigerator interior image generated by photographing a storage room of the refrigerator 10 using a camera included in the refrigerator 10. For example, when an event that a door is opened and then closed occurs, the refrigerator 10 may photograph an image of the interior of the refrigerator.
[0204] Referring to operation 1020, the server 20 may identify a changed region in the refrigerator interior image using the refrigerator interior image and the image to be compared.
[0205] For example, the server 20 may change the image to be compared and the image inside the refrigerator into images emphasizing edge components, and then compare the acquired object shapes based on the edge components and identify the changed area.
[0206] According to one embodiment, the server 20 may obtain the changed area by using pixel difference values. For example, the server 20 may calculate the corresponding difference values between the red pixel value (R), the green pixel value (G), and the blue pixel value (B) included in the image to be compared and the R, G, and B values of the refrigerator interior image, and identify the changed area.
[0207] According to one embodiment, the server 20 may obtain the changed region by using an artificial intelligence algorithm. For example, the server 20 may identify the changed region that has changed in at least one image of the interior of the refrigerator by applying at least one image of the interior of the refrigerator and the image to be compared to a food change sensing model, wherein the food change sensing model is trained using an image, an image that is the same as the image but includes at least some parts with changes, and information on the location of the changes as learning data.
[0208] In operation 1030 , the server 20 may identify whether a change area exists.
[0209] In operation 1040, if the changed area exists, the server 20 may acquire second food storage information using the result of recognizing the object contained in the refrigerator interior image and the first food storage information.
[0210] According to one embodiment, the server 20 can change the image to be compared and the image inside the refrigerator into images that emphasize edge components, and then find the shape of the acquired object based on the edge components, compare the shape of the object with the data stored in the server 20, and identify the food category to which the object belongs.
[0211] According to one embodiment, when the object is identified as a processed food, the server 20 may identify the category to which the processed food belongs and / or the brand name of the processed food by using additional data additionally stored in the server 20 .
[0212] According to one embodiment, the server 20 can sense an object by using an artificial intelligence algorithm and identify the food category to which the sensed object belongs. For example, the server 20 can sense an object by applying an image of the interior of a refrigerator to a food recognition model trained using information about the image of food, the food category to which the food belongs, and the location of the food in the image as learning data, and can identify the food category to which the object belongs.
[0213] According to one embodiment, as a result of identifying the object by using the food recognition model, when the object is identified as a processed food, the server 20 can identify the category of the processed food to which the object belongs and / or the brand name of the processed food by using the processed food recognition model. For example, the server 20 can identify the category of the processed food detected in the refrigerator interior image and / or the brand name of the processed food in the processed food database by using the processed food search model trained based on the image similarity of multiple images for the same processed food, the category of the processed food, and the brand name of the processed food as learning data.
[0214] According to one embodiment, in a case where unrecognizable food is detected due to being hidden by other food, the server 20 may acquire new food storage information by using the food information acquired from the first food storage information and the refrigerator interior image.
[0215] In operation 1050, the server 20 may transmit the food storage information and the refrigerator interior image to the refrigerator.
[0216] In operation 1060, in a case where the change area does not exist, the server 20 may acquire first food storage information.
[0217] The first food storage information may be, for example, information generated using a result of sensing an object in the image to be compared. In addition, the first food storage information may include, for example, any one or any combination of location information of the food sensed in the image to be compared, information of the category to which the food belongs, or information of the processed food category to which the processed food belongs, and a brand name of the processed food.
[0218] In operation 1070, the server 20 may transmit the first food storage information and the refrigerator interior image to the refrigerator.
[0219] Fig.11 is a flowchart of a process of executing a food identification method according to an embodiment.
[0220] In operation 1110, the refrigerator 10 may acquire a first refrigerator interior image generated by photographing the first storage chamber, and a second refrigerator interior image generated by photographing the second storage chamber.
[0221] In operation 1120, the refrigerator 10 may use the first refrigerator interior image, the second refrigerator interior image, the first image to be compared, and the second image to be compared, and identify a changed region where a change is made in the refrigerator interior image.
[0222] The first image to be compared and the second image to be compared may be, for example, images generated by photographing the first storage room and the second storage room, and these images may be images transmitted to the server 20 where food is sensed and recognized and delivered to the refrigerator 10 .
[0223] In operation 1130, the refrigerator 10 may identify whether a change area exists.
[0224] In operation 1140, when the changed area does not exist in the second refrigerator interior image, the refrigerator 10 may acquire 1-2 food storage information of the second storage chamber.
[0225] In operation 1150 , when acquiring the changed area from the first refrigerator interior image, the refrigerator 10 may request object recognition while transmitting the first refrigerator interior image to the server 20 .
[0226] In operation 1160, the server 20 may acquire 2-1 food storage information of the first storage chamber by using a result of recognizing an object included in the first refrigerator interior image and 1-1 food storage information of the first storage chamber.
[0227] In operation 1170 , the server 20 may transmit 2-1 food storage information of the first storage chamber to the refrigerator 10 .
[0228] In operation 1180, the refrigerator 10 may store 2-1 food storage information of the first storage chamber and 1-2 food storage information of the second storage chamber.
[0229] Fig.12 is a perspective view briefly showing a refrigerator having a camera and a display according to one embodiment. Fig.13 is a front view schematically showing a refrigerator with all doors opened according to one embodiment.
[0230] Reference Fig.12 and 13 , the refrigerator 10 includes a main body 1210, storage chambers 1211, 1212, 1213, doors 1220, 1230, 1240, 1250, and a hinge 1260 connecting each door 1220, 1230, 1240, 1250 to the main body 1210. A display (or touch screen, 1270) displaying content may be disposed on at least one of the plurality of doors (e.g., one or both of the right door 1230 and the left door 1220).
[0231] The camera may be placed on at least one of the plurality of doors (e.g., one or both of the right door 1230 and the left door 1220). The proximity sensor component may be positioned adjacent to the camera (e.g., within a radius of 500 mm). In addition, a microphone may be placed on at least one of the plurality of doors (e.g., one or both of the right door 1230 and the left door 1220).
[0232] The refrigerator 10 may be classified into various types according to the shapes of the storage chamber and the door. A top mounted freezer (TMP) type refrigerator includes a freezer formed on the upper surface of the storage chamber horizontally partitioned by a partition and a cold room formed on the lower side of the storage chamber. A bottom mounted freezer (BMF) type refrigerator includes a refrigerator formed on the upper side of the storage chamber horizontally partitioned by a partition and a freezer formed on the lower side of the storage chamber.
[0233] A side by side (SBS) type refrigerator includes a freezer compartment formed on one side of a storage compartment vertically partitioned by a partition and a refrigerating compartment formed on the other side of the storage compartment. A French door refrigerator (FDR) refrigerator includes a refrigerating compartment formed on the upper side of a storage compartment horizontally partitioned by a partition and a freezer compartment formed on the lower side of the storage compartment. The refrigerating compartment on the upper side can be opened and closed by a pair of doors. In an FDR refrigerator, both the upper refrigerating compartment and the lower refrigerating compartment can be opened and closed by a pair of doors.
[0234] The main body 1210 includes an inner shell forming storage chambers 1211 to 1213, an outer shell forming the outside of the refrigerator, and an insulator for maintaining a temperature difference between the inner shell and the outer shell. The insulating material can prevent cold internal air from leaking to the outside and prevent external warm air from entering the storage chambers 1211 to 1213.
[0235] The main body 1210 includes a cold air supply part that supplies cold air to the storage chambers 1211 to 1213. The cold air supply part may include a compressor for compressing a refrigerant, a condenser, an expansion valve, an evaporator, and a pipe.
[0236] The storage chambers 1211 to 1213 are separated by a partition 1214. The storage chambers 1211 to 1213 are divided into a lower freezing storage chamber (e.g., storage chambers 1212 and 1213) (hereinafter referred to as a "freezer") and a refrigerated storage chamber (e.g., storage chamber 1211) (hereinafter referred to as a "refrigerator") above the freezing chamber (e.g., storage chambers 1212 and 1213). The storage chamber 1212 can be set to a temperature above zero (e.g., between 0°C and 7°C) or below zero (e.g., between -1°C and 5°C), and stores water, beverages, food ingredients, and refrigerated or frozen foods. Water or beverages can be stored in a beverage container.
[0237] A refrigerating chamber (eg, the storage chamber 1211 ) among the storage chambers 1211 to 1213 partitioned by the partition 1214 may include one or more shelves 1215 and one or more storage boxes 1216 .
[0238] The refrigerating chamber (e.g., storage chamber 1211) is connected to one side (e.g., left side) of the storage chamber 1211 and the second door 1230 adjacent to the first door 1220 and located at the other side (e.g., right side) of the storage chamber 1211. The first door 1220 and / or the second door 1230 may rotate at an angle (e.g., 300° or less) set by each hinge 1260, and open or close (e.g., connect or separate) the front surface of the storage chamber 1211. The first door 1220 may rotate in a direction opposite to the rotation direction of the second door 1230, and open or close the storage chamber 1211. The positions of the first door 1220 and the second door 1230 may be changed to each other.
[0239] The first door 1220 rotates at an angle (eg, 300° or less) set by the hinge 1260 , and opens or closes a portion of the front surface of the storage chamber 1211 (eg, 35% to 70% of the front surface of the storage chamber 1211 ).
[0240] A dispenser of water, ice, or sparkling water and / or a graspable handle is provided to the front surface (eg, +y-axis direction) of the first door 1220 .
[0241] The second door 1230 rotates at an angle (e.g., 300° or less) set by the hinge 1260, and opens or closes a portion of the front surface of the storage chamber 1211 (e.g., 35% to 70% of the front surface of the storage chamber 1211). The second door 1230 may include a grippable handle 1231. The handle 1221 of the first door 1220 and the handle 1231 of the second door 1230 are positioned to be spaced apart in the left and right directions based on the central area of the storage chamber 1211.
[0242] The functions and storage settings of the refrigerator 10 may be displayed on the front surface of the second door 1230 (e.g., +y-axis direction). User input (e.g., touch or selection of a button) may be received on the front surface of the second door 1230. A display (or touch screen 1270) capable of displaying (or executing) a screen of an application (including a widget) may be placed on the front surface of the second door 1230. The button may be included in the display, or may be configured separately from the display. The button may be a button displayed on the touch screen, a touch button, or a physical button.
[0243] The vegetable compartments 1211d1 and 1211d2 are located below the storage compartment 1211. The vegetable compartments 1211d1, 1211d2 may be drawn out (eg, slid or rolled) toward the front side (eg, y-axis direction).
[0244] The storage chamber 1212 may have a door 1240 on one side. The storage chamber 1213 may have a door 1250 on one side. The storage chambers 1212 and 1213 may be combined into one storage chamber (e.g., like the storage chamber 1311). Like the storage chamber 1211, one storage chamber may have a door on the left and right sides. According to an embodiment, the storage chambers 1211, 1212, and 1213 may be equally divided by shelves into, for example, a first storage chamber 1211a, a second storage chamber 1211b, a third storage chamber 1211c, etc. In addition, the refrigerator may also have a drawer.
[0245] The refrigerator 10 may further include a front camera 1251 located outside the second door 1230. The refrigerator 10 may use the front camera 1251 to recognize a person located on the front surface of the refrigerator 10. In addition, the refrigerator 10 may further include a display 1270. The display 1270 may include Figure 1a The display 11 of FIG. 5 and the display 510 of FIG. 5 .
[0246] Fig.14 is a block diagram briefly illustrating a refrigerator according to one embodiment.
[0247] Reference Fig.14 , the refrigerator 10 may be functionally connected with the external device by using the communication interface 1420. The communication interface 1420 may include the communication interface 220 of FIG. 2. The external device may include one or both of the user terminal and the server 20.
[0248] The refrigerator 10 can send operation information corresponding to the operation of the refrigerator (for example, temperature control of each storage chamber) and status information corresponding to the status of the refrigerator 10 (for example, normal, abnormal, etc.) to an external device through the communication interface 1420, and can also receive control information from the external device (for example, a control command corresponding to quick freezing of the refrigerator 10).
[0249] The refrigerator 10 may include a processor 1410 , a communication interface 1420 , a camera 1450 , a sensor portion 1460 , a display (or display portion) 1270 , a driver portion 1480 , a memory 1490 , and a power supply portion 1495 .
[0250] The processor 1410 (or controller) may include one or more processors. The processor 1410 may include the processor 210 of FIG. 2 . In addition, the processor 1410 may store a non-volatile memory (the non-volatile memory includes a ROM storing a control program for controlling the refrigerator 10) and a volatile memory (the volatile memory includes a RAM storing a signal or data input from outside the refrigerator 10 or used as a storage area for various operations performed on the refrigerator 10). The RAM may serve as a storage area for control information received from an external source, operation information of the refrigerator 10, or status information of the refrigerator 10.
[0251] The processor 1410 is used to control the overall operation of the refrigerator 10 and the signal flow between the internal components of the refrigerator 10, and process data. The processor 1410 can control the power supplied to the internal elements by using the power supply part 1495.
[0252] The processor 1410 may control the communication interface 1420 , the camera 1450 , the sensor portion 1460 , the display 1270 , the driver portion 1480 , the memory 1490 , and the power supply portion 1495 .
[0253] The communication interface 1420 can be connected to an external device via a mobile communication network, a wireless LAN communication network, or a near field communication network by using one antenna or two or more antennas under the control of the processor 1410. Wireless LAN communication can be wirelessly connected to an access point (AP) at a location where the AP is installed under the control of the processor 1410. For example, wireless LAN communication may include Wi-Fi communication. Near field communication may include Bluetooth communication, Bluetooth low energy communication, infrared data association (IrDA) communication, ultra-wideband (UWB) communication, magnetic security transmission (MST) communication, and / or NFC communication. According to an embodiment, the term "communication interface" may be connected to an external device via mobile communication, wireless LAN communication, and / or near field communication.
[0254] The camera 1450 may capture a still image or record a video under the control of the processor 1410. The camera 1450 may include the camera 230 of FIG. 2 or Fig.12The camera 1450 may capture a still image or record a video for registering and / or managing food. The camera 1450 may capture the user's iris under the control of the processor 1410. The camera 1450 may include a front camera 1251 located on the front surface of the refrigerator 10, and an indoor camera located inside the refrigerator 10. In addition, an iris capturing camera for capturing the user's iris may be located on one side of the front camera 1251.
[0255] The front camera 1251 may be tilted toward the front (e.g., +y-axis direction) at a set angle (e.g., less than 85°) according to the front (or front surface) of the door of the refrigerator 10. The set angle may be less than 75 degrees or more than 35 degrees. In addition, the set angle may be less than 65 degrees or more than 15 degrees.
[0256] One of the front camera 1251 and the indoor camera may include an auxiliary light (eg, flash) that provides light for shooting. In addition, an auxiliary light source for capturing irises (eg, an LED for iris recognition) may be located on the front surface of the refrigerator 10.
[0257] One front camera 1251 or a plurality of front cameras 1251 may be included. One or more indoor cameras may be located on the rear surfaces of the doors 1220 to 1250 facing the storage rooms 1211 to 1213 according to the area of the storage room. For example, one or more indoor cameras may be located on the rear surfaces of the doors 1220 and 1230 facing the storage room 1211. One or more indoor cameras may be located on the rear surface of the door 1240 facing the storage room 1212. In addition, one or more indoor cameras may be located on the rear surface of the door 1250 facing the storage room 1213.
[0258] The processor 1410 may control an image photographed through one of the front camera 1251 and the indoor camera to be stored in the memory 1490 .
[0259] The sensor portion 1460 may detect a peripheral state of the refrigerator 10 (eg, illumination) and / or an internal state of the refrigerator 10 (eg, temperature of the storage chamber) through one or more sensors.
[0260] The sensor part 1460 may include any one or any combination of a proximity sensor component for detecting whether a user is close to the refrigerator 10, and one or more temperature sensor components for detecting the temperature of a storage chamber (e.g., a freezer, a refrigerator, a vegetable chamber, etc.) of the refrigerator 10. In addition, an illumination sensor component for detecting the light intensity around the refrigerator 10 corresponding to the light change of the display 1270 may be included. One of the proximity sensor component and the illumination sensor component in the sensor part 1460 may be located on the front surface of the door of the refrigerator 10. In addition, the temperature sensor component may be provided inside the refrigerator to correspond to the storage chamber.
[0261] Those skilled in the art will readily appreciate that the types of sensors included in the sensor portion 1460 may be added, modified, or deleted depending on the performance of the refrigerator 10 .
[0262] The display 1270 may provide (or display) a graphical user interface (GUI) corresponding to various services (e.g., voice call, video call, data transmission, broadcast reception, photo shooting, video content viewing, electronic payment including mobile payment, etc.). In one embodiment, the display 1270 may be configured to include a touch screen.
[0263] In the touch screen, the display panel and the touch panel 172 may be implemented as one body (eg, an in-cell type touch screen or an on-cell type touch screen).
[0264] The touch screen may include an edge touch panel for receiving a user input and an edge display panel for displaying content. The edge touch panel and the edge display panel may be implemented as one body as described above.
[0265] The touch screen can send analog signals corresponding to single-point touch or multi-point touch input through the main screen or GUI to the touch screen controller. The touch screen can receive single-point touch or multi-point touch through the user's body (e.g., fingers including thumb) or input pen (e.g., stylus).
[0266] The driver portion 1480 may include any one or any combination of a compressor, a fan, a filter, and a heater that operate under the control of the processor 1410. The driver portion 1480 may further include lighting (or a deodorizing device).
[0267] The compressor 1481 may compress the refrigerant as the working fluid of the refrigeration cycle under the control of the processor 1410. The refrigeration cycle may include a condenser that converts the gaseous compressed refrigerant into a liquid refrigerant, an expander that decompresses the liquid refrigerant, and an evaporator that evaporates the decompressed liquid refrigerant. The processor 1410 may control the temperature of the storage chamber by evaporation of the liquid refrigerant. In addition, the refrigerator 10 may control the temperature of the storage chamber by a Peltier module using the Peltier effect and a magnetic cooling device using the magnetocaloric effect.
[0268] The fan 1482 may circulate external air under the control of the processor 1410. The air heated by the cooling cycle may be cooled by heat exchange with the external air.
[0269] The air cleaning component 1483 may sterilize (or eliminate) bacteria floating in or attached to the storage chamber under the control of the processor 1410. The air cleaning component 1483 may include an ion sterilization cleaning component.
[0270] The heater 1484 may perform defrosting under the control of the processor 1410. The heater 1484 may include a defrosting heater.
[0271] The memory (or storage) 1490 may store input or output of a signal or data (e.g., corresponding to food management (or food recognition)) to correspond to the operation of the element under the control of the processor 1410. The memory 1490 may include the memory 240 of FIG. 2. According to one embodiment, the memory 1490 may store food storage information related to the food stored in the refrigerator 10.
[0272] The term "module" used herein includes a unit composed of hardware, software or firmware, and can be used interchangeably with terms such as logic, logic block, component or circuit. A "module" can be a component or minimum unit of an integrated structure that performs one or more functions, or a portion thereof. For example, according to one embodiment, the module can be implemented in the form of an application-specific integrated circuit (ASIC).
[0273] Embodiments of the present disclosure may be implemented as software including one or more instructions stored in a storage medium (e.g., a memory) that can be read by a machine (e.g., a refrigerator 10 or a server 20). For example, a processor (e.g., processor 210) of a machine (e.g., a refrigerator 10) may call at least one instruction from one or more instructions stored in a storage medium and execute the called instruction. This enables the machine to be managed to perform at least one function according to at least one instruction called. The one or more instructions may include code generated by a compiler or code executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" only means that the storage medium does not include a signal (e.g., an electromagnetic field), but is tangible, and does not distinguish between a case where data is semi-permanently stored in the storage medium and a case where data is temporarily stored in the storage medium.
[0274] According to one embodiment, the method according to the above embodiment may be provided as being included in a computer program product. The computer program product may be traded between a seller and a consumer as a product. The computer program product may be distributed online in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)) or through an application store (e.g., Play Store TM and App Store TM), or distributed online directly between users (e.g., smartphones) (e.g., downloaded or uploaded). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium such as a manufacturer's server, an application store's server, or a memory of a relay server.
[0275] According to an embodiment, each element (e.g., module or program) of the above-mentioned elements may include a single entity or multiple entities. According to an embodiment, at least one element or operation in the above-mentioned corresponding elements may be omitted, or at least one other element or operation may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be combined to form a single entity. In this case, the integrated entity may perform at least one function of each element in the multiple elements in the same or similar execution mode as the corresponding element in the multiple elements before integration. Modules, program modules, or operations performed by other elements according to various embodiments may be performed continuously, in parallel, repeatedly, or heuristically, or at least some operations may be performed in different orders, may be omitted, or other operations may be added thereto.
Claims
1. A refrigerator, include: monitor; a camera for photographing the interior of the refrigerator; a communicator for communicating with a server storing a recognition model, the recognition model being trained to recognize information about a category of food included in an image; and The processor is configured as: acquiring a first image using the camera, based on receiving a user input for inputting a category of food included in the first image, transmitting the first image and information about the category of the food to the server through the communicator so that the recognition model is trained based on the first image and the information about the category input through the user input, and Based on transmitting a second image acquired by the camera to the server through the communicator, acquiring information about the category of the food recognized from the second image by the trained recognition model from the server through the communicator, Wherein, the recognition model is configured as follows: Based on the fact that food corresponding to food included in the first image is included in the second image, the category of the food included in the second image is recognized according to the learning as the category input by the user input.
2. The refrigerator according to claim 1, in, The processor is configured to: The display is controlled to display information about a plurality of candidate categories corresponding to the food, and based on a user input selecting one of the plurality of candidate categories being input, information about the category input by the user input is transmitted to the server through the communicator.
3. The refrigerator according to claim 1, in, The processor is configured to: Based on information on the category of at least one food stored in the refrigerator, a recipe of a food that can be prepared by using the at least one food stored in the refrigerator is acquired, and the display is controlled to display the acquired recipe.
4. The refrigerator according to claim 1, in, The processor is configured to: Based on a user input selecting an object for connecting to the food purchase website displayed on the display being input, the display is controlled to display the food purchase website based on the user input.
5. A method for controlling a refrigerator, the refrigerator communicating with a server storing a recognition model, the recognition model being trained to recognize information about a category of food included in an image, the method include: Acquire a first image using a camera, based on receiving a user input for inputting a category of food included in the first image, sending the first image and information about the category of the food to the server so that the recognition model is trained based on the first image and the information about the category input by the user input, and Based on sending a second image acquired by the camera to the server, acquiring from the server information about the category of the food recognized from the second image by the trained recognition model, Wherein, the recognition model is configured as follows: Based on the fact that food corresponding to food included in the first image is included in the second image, the category of the food included in the second image is recognized according to the learning as the category input by the user input.
6. The control method according to claim 5, in, Sending the first image and information about the category of the food includes: Information about a plurality of candidate categories corresponding to the food is displayed on a display of the refrigerator, and based on a user input selecting one of the plurality of candidate categories being input, information about the category input by the user input is transmitted to the server.
7. The control method according to claim 5, further comprising: include: acquiring a recipe of a food that can be prepared by using the at least one food stored in the refrigerator based on information about the category of the at least one food stored in the refrigerator; and The acquired recipe is displayed on a display of the refrigerator.
8. The control method according to claim 5, further comprising: include: A user input based on selecting an object for connecting to a food purchase website displayed on the display of the refrigerator is input, and the food purchase website is displayed on the display based on the user input.
9. A non-transitory computer-readable medium comprising instructions, the instructions being executed by a processor of a refrigerator so that the refrigerator performs an operation, the refrigerator communicating with a server storing a recognition model, the recognition model being trained to recognize information about a category of food included in an image, the operation include: Acquire a first image using a camera, based on receiving a user input for inputting a category of food included in the first image, sending the first image and information about the category of the food to the server so that the recognition model is trained based on the first image and the information about the category input by the user input, and Based on sending a second image acquired by the camera to the server, acquiring from the server information about the category of the food recognized from the second image by the trained recognition model, Wherein, the recognition model is configured as follows: Based on the fact that food corresponding to food included in the first image is included in the second image, the category of the food included in the second image is recognized according to the learning as the category input by the user input.
10. The non-transitory computer readable medium of claim 9, in, Sending the first image and information about the category of the food includes: Information about a plurality of candidate categories corresponding to the food is displayed on a display of the refrigerator, and based on a user input selecting one of the plurality of candidate categories being input, information about the category input by the user input is transmitted to the server.
11. The non-transitory computer readable medium of claim 9, further comprising: include: acquiring a recipe of a food that can be prepared by using the at least one food stored in the refrigerator based on information about the category of the at least one food stored in the refrigerator; and The acquired recipe is displayed on a display of the refrigerator.
12. The non-transitory computer readable medium of claim 9, further comprising: include: A user input based on selecting an object for connecting to a food purchase website displayed on the display of the refrigerator is input, and the food purchase website is displayed on the display based on the user input.
