Refrigerator, server, and method for controlling refrigerator

Abstract

A refrigerator for identifying an object to be stored and automatically opening a door based on the identified object and a method thereof are provided. The refrigerator includes: a main body; at least one door provided on a front surface of the main body; one or more storage chambers provided in the main body or at least one door and configured to store the object; a cooler configured to cool the one or more storage chambers; a door driver configured to generate power for automatically opening and closing at least one door; a camera configured to capture an image of a surrounding environment near the at least one door; and at least one processor configured to, in the case of recognizing an object from an image of the surrounding environment captured by the camera, send a control signal for opening at least one door to the door driver based on the recognized object.

Description

Technical Field

[0001] The present disclosure relates to a refrigerator capable of automatically opening or closing a door, a server for managing the refrigerator, and a method for controlling the refrigerator. Background Art

[0002] Generally speaking, a refrigerator is a device that supplies cold air to a storage chamber where food is stored to keep the food fresh for a long time. The storage chamber is maintained within a preset temperature range required to keep the food fresh.

[0003] The storage compartment of the refrigerator is opened at the front side, and the opened front side is closed by a door.

[0004] The door is opened by the user. However, when the user's hands are not free, such as when the user puts food into the storage room or takes food out of the storage room, it is not easy for the user to open or close the door by himself. In order to cope with this situation, a refrigerator capable of automatically opening and closing the door is being developed.

[0005] The above information is presented as background information only to assist in understanding the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above may be applicable as prior art to the present disclosure. Summary of the invention

[0006] Technical issues

[0007] Aspects of the present disclosure at least solve the above-mentioned problems and / or disadvantages and provide at least the following advantages. Therefore, one aspect of the present disclosure is to provide a refrigerator for identifying an object to be stored and automatically opening a door based on the identified object, a method for controlling a refrigerator, and a server for identifying an object and sending a door opening signal to the refrigerator based on the identified object.

[0008] Technical Solution

[0009] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0010] According to one aspect of the present disclosure, a refrigerator is provided. The refrigerator includes: a main body; at least one door, which is arranged on the front surface of the main body; a plurality of storage chambers, which are arranged in the main body or at least one door and are configured to store objects; a cooler, which is configured to cool the plurality of storage chambers; a door driver, which is configured to generate power for automatically opening or closing at least one door; a camera, which is configured to capture an image of the surrounding environment near at least one door; and at least one processor, which is configured to, in the case where an object is recognized from the environment image captured by the camera, send a control signal for opening at least one door to the door driver based on the recognized object.

[0011] The at least one processor may be further configured to determine a storage room corresponding to the identified object from among a plurality of storage rooms, and transmit a control signal to a door driver for opening a door of the storage room corresponding to the identified object among the at least one door.

[0012] The at least one processor may be further configured to determine a door of the at least one door corresponding to the recognized object, and transmit a control signal for opening the door corresponding to the recognized object to a door driver.

[0013] The refrigerator may further include a temperature sensor configured to sense a temperature of an object located around the at least one door.

[0014] The at least one processor may be further configured to determine a storage compartment among the plurality of storage compartments and a storage temperature corresponding to the identified object based on the temperature sensed by the temperature sensor and the object identified from the image of the surrounding environment.

[0015] The at least one processor may be further configured to send a control signal to the cooler to control the temperature of the storage chamber corresponding to the identified object to a storage temperature.

[0016] The refrigerator may further include a proximity sensor configured to sense approach of a user, wherein the at least one processor may further be configured to activate the camera in response to an output of the proximity sensor indicating approach of the user.

[0017] The refrigerator may further include a speaker, wherein the at least one processor may be further configured to control the speaker to output a notification sound when the at least one door is automatically opened.

[0018] The at least one processor may also be configured to determine a relative position of the user relative to a door that is expected to be opened among the at least one door, and control the speaker to output a notification sound differently according to the relative position of the user.

[0019] The at least one processor may also be configured to determine a relative position of a user relative to a door of at least one door that is intended to be opened, and control an opening angle of the door based on the relative position of the user.

[0020] The refrigerator may further include a microphone, wherein the at least one processor may be further configured to, in response to a name of an object recognized from a user voice input to the microphone, send a control signal to a door driver for opening at least one door based on the name of the recognized object.

[0021] The refrigerator may further include an internal sensor including at least one of a weight sensor, a distance sensor, or an internal camera. The internal sensor may be disposed in at least one storage compartment among the plurality of storage compartments.

[0022] The at least one processor may be further configured to determine whether a storage room corresponding to the identified object can accommodate the identified object based on an output of the internal sensor.

[0023] The refrigerator may further include at least one of a speaker or a display, wherein the at least one processor may further be configured to control at least one of the speaker or the display to output a guidance message related to the determined result based on the determined result that the storage room corresponding to the identified object cannot accommodate the identified object.

[0024] The refrigerator may also include a display arranged in at least one door, wherein the at least one processor may also be configured to control the display to display information related to the entry or release of the object when the object enters at least one of the multiple storage chambers or is released from at least one of the multiple storage chambers.

[0025] The refrigerator may also include a drawer and a drawer drive, wherein the drawer is configured to slide to be taken out of or placed into at least one of the plurality of storage chambers, and the drawer drive is configured to generate power for at least one of automatically placing the drawer into the at least one storage chamber or automatically taking the drawer out of at least one of the plurality of storage chambers.

[0026] The at least one processor may be further configured to transmit a control signal for taking the drawer out of the storage chamber to the drawer driver in a case where the drawer is disposed in the storage chamber corresponding to the recognized object.

[0027] The refrigerator may further include a transceiver configured to communicate with an external device.

[0028] The transceiver may transmit an image of the surrounding environment captured by the camera to the server, and, if the server recognizes an object, receive information about the object or information about a door corresponding to the object from the server.

[0029] The refrigerator may further include a transceiver configured to communicate with another refrigerator, wherein, based on a result of determining that the storage chamber corresponding to the identified object cannot accommodate the identified object, the at least one processor may send information about the identified object or a signal for opening a door to the other refrigerator.

[0030] The transceiver may receive, from an appliance having a camera, an image captured by the camera or information on an object recognized from the image captured by the camera.

[0031] The at least one processor may also be configured to determine an opening time of the at least one door based on a distance between the device and the refrigerator.

[0032] The storage room corresponding to the recognized object may include a storage room storing objects related to the recognized object or estimated to store objects related to the recognized object.

[0033] According to an embodiment of the present disclosure, a server includes: a transceiver configured to receive an image of a surrounding environment from a refrigerator; and at least one processor configured to identify an object from the image of the surrounding environment, determine whether to open a door set in the refrigerator based on the identified object, and control the transceiver to send a signal for opening the door to the refrigerator based on a result of the determination that the door needs to be opened.

[0034] The refrigerator may include multiple storage rooms and multiple doors, and the at least one processor may also be configured to determine a storage room among the multiple storage rooms corresponding to the identified object, and control the transceiver to send a signal to the refrigerator for opening a door of the storage room among the multiple doors corresponding to the identified object.

[0035] The transceiver may receive information about the temperature of the identified object from the refrigerator, and the at least one processor may be further configured to determine a storage temperature corresponding to the identified object based on the information about the temperature of the identified object.

[0036] The at least one processor may be further configured to determine at least one of a storage room corresponding to the identified object or a storage temperature corresponding to the identified object based on a deep learning model generated by learning data obtained from a plurality of users.

[0037] The at least one processor may also be configured to control the transceiver to transmit a signal for opening a door to another refrigerator registered by a user of the refrigerator, if the storage chamber corresponding to the identified object cannot accommodate the identified object.

[0038] According to another aspect of the present disclosure, a method for controlling a refrigerator is provided, the refrigerator comprising a camera, a plurality of storage chambers, and at least one door located on a front surface of the plurality of storage chambers. The method comprises taking an image of a surrounding environment near the at least one door with the camera, and in response to recognizing an object from the environment image, opening the at least one door based on the recognized object.

[0039] Opening the at least one door may include determining a storage room corresponding to the recognized object among the plurality of storage rooms, and opening a door of the storage room corresponding to the recognized object among the at least one door.

[0040] The method may further include sensing a temperature of an object located around at least one door, wherein opening of the at least one door may include determining a storage room corresponding to the identified object and a storage temperature corresponding to the identified object among a plurality of storage rooms based on the sensed temperature and an object identified from an image of the surrounding environment.

[0041] The method may further include sensing proximity of a user, and activating the camera in response to sensing the proximity of the user.

[0042] The opening of at least one door may include determining whether a storage room corresponding to the identified object can accommodate the identified object, and opening a door of the storage room corresponding to the identified object based on a result of the determination that the storage room corresponding to the identified object can accommodate the identified object.

[0043] The method may further include outputting a guide message related to a result of the determination according to a result of the determination that the storage room corresponding to the identified object cannot accommodate the identified object.

[0044] The method may also include: triggering voice recognition when no object is recognized from the image of the surrounding environment; receiving the user's voice through a microphone set in the refrigerator; and opening at least one door based on the name of the recognized object according to the name of the object recognized from the voice input from the user to the microphone.

[0045] Other aspects, advantages, and salient features of the disclosure will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses various embodiments of the disclosure.

[0046] Beneficial Effects

[0047] In a refrigerator, a server, and a refrigerator control method according to one aspect of the present disclosure, an object to be stored may be recognized, and a door may be automatically opened based on the recognized object. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] These and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0049] Figure 1 The external appearance of a refrigerator with its door closed according to an embodiment of the present disclosure is shown;

[0050] Figure 2 The external appearance of a refrigerator with its door opened according to an embodiment of the present disclosure is shown;

[0051] Figure 3 is a front sectional view schematically showing the spatial division of a refrigerator according to an embodiment of the present disclosure;

[0052] Figure 4 is a control block diagram of a refrigerator according to an embodiment of the present disclosure;

[0053] Figure 5 is a control block diagram of a refrigerator further including a proximity sensor for sensing a user approaching according to an embodiment of the present disclosure;

[0054] Figure 6 and Figure 7 An example of a refrigerator opening a door corresponding to a recognized object according to an embodiment of the present disclosure is shown;

[0055] Figure 8 is a control block diagram of a refrigerator further including a temperature sensor according to an embodiment of the present disclosure;

[0056] Fig. 9 is a control block diagram of a refrigerator further including a speaker according to an embodiment of the present disclosure;

[0057] Fig.10 and Fig.11 shows an example of a notification sound output from a speaker of a refrigerator according to an embodiment of the present disclosure;

[0058] Fig.12 An example of a refrigerator that adjusts the opening angle of a door based on the relative position between the door and a user according to an embodiment of the present disclosure is shown;

[0059] Fig.13 is a control block diagram of a refrigerator further including an internal sensor according to an embodiment of the present disclosure;

[0060] Fig.14 is a control block diagram of a refrigerator further including a drawer drive according to an embodiment of the present disclosure;

[0061] Fig.15 is a control block diagram of a refrigerator further including a microphone according to an embodiment of the present disclosure;

[0062] Fig.16 An example of recognizing an object from a voice input by a user through a microphone according to an embodiment of the present disclosure is shown;

[0063] Fig.17 is a control block diagram of a refrigerator further including a display according to an embodiment of the present disclosure;

[0064] Fig.18 shows an example of a screen displayed on a display of a refrigerator according to an embodiment of the present disclosure;

[0065] Fig.19 is a control block diagram of a refrigerator further comprising a transceiver according to an embodiment of the present disclosure;

[0066] Figure 20 to Figure 22 illustrates an example of an external device communicating with a refrigerator according to various embodiments of the present disclosure; and

[0067] Figure 23 to Figure 27 is a flowchart illustrating a method for controlling a refrigerator according to various embodiments of the present disclosure.

[0068] Throughout the drawings, like reference numerals will be understood to refer to like parts, components and structures. DETAILED DESCRIPTION

[0069] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. It includes various specific details to assist in understanding, but these are to be regarded as exemplary only. Therefore, it will be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and structures may be omitted for clarity and conciseness.

[0070] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and not for limiting the present disclosure as defined by the appended claims and their equivalents.

[0071] It should be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.

[0072] Throughout the specification, the same reference numerals will refer to the same components. This specification does not describe all components of the embodiment, and will not describe the general information of the technical field to which the disclosure belongs or the overlapping information between the embodiments. As used herein, the term "part", "component", "module", "member" or "block" can be implemented as software or hardware, and according to an embodiment, multiple "parts", "components", "modules", "members" or "blocks" can be implemented as single components, or a single "part", "component", "module", "member" or "block" can include multiple components.

[0073] Throughout this specification, it should be understood that when a part is referred to as being "connected" to another component, it may be directly or indirectly connected to the other component. When a component is indirectly connected to another component, it may be connected to the other component through a wireless communication network, or may be electrically connected to the other component through a wire.

[0074] Furthermore, it should be understood that when a part “includes” a certain component, the part does not exclude another component but may also include another component unless the context clearly states otherwise.

[0075] Throughout the specification, it will also be understood that when an element is referred to as being “on” or “over” another element, it can be directly on the other element or intervening elements may also be present.

[0076] Throughout the specification, it should also be understood that when a component transmits or sends a signal or data to another component, it does not exclude the situation where other components exist between the corresponding component and the other component and the corresponding component transmits or sends the signal or data through the other component, unless the context clearly indicates.

[0077] Throughout the specification, ordinal terms “first”, “second”, etc. are used to distinguish between multiple components but do not indicate the arrangement order of the components, the manufacturing order of the components, the importance of the components, etc.

[0078] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0079] The reference numbers used in the operations are provided for convenience of description and do not describe the order of the operations, and the operations may be performed in an order different from the described order unless a specific order is clearly specified in the context.

[0080] Throughout the disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0081] Hereinafter, a refrigerator, a server, and a control method of a refrigerator according to one aspect will be described in detail with reference to the accompanying drawings.

[0082] A refrigerator according to one aspect may include at least one door. Therefore, a refrigerator according to one aspect may include one, two or three doors. In the embodiments of the present disclosure to be described below, a refrigerator including four doors will be described as an example to describe various examples.

[0083] The refrigerator according to one aspect may be a side-by-side type refrigerator in which a freezing chamber and a refrigerating chamber are arranged side by side, a bottom freezer type refrigerator in which a refrigerating chamber is formed at an upper portion and a freezing chamber is formed at a lower portion, a top mounted type refrigerator in which a freezing chamber is formed at an upper portion and a refrigerating chamber is formed at a lower portion, or a combination type refrigerator in which features of the above types are mixed and applied. However, for the sake of detailed description, in the embodiments of the present disclosure to be described below, a bottom freezer type refrigerator in which a freezing chamber is formed at a lower portion will be described as an example.

[0084] Figure 1 shows the appearance of a refrigerator with its door closed according to an embodiment of the present disclosure, Figure 2 shows the appearance of a refrigerator with its door open according to an embodiment of the present disclosure, Figure 3 is a front sectional view schematically showing space division of a refrigerator according to an embodiment of the present disclosure.

[0085] Reference Figure 1-Figure 3 The refrigerator 1 may include a main body 10 , a plurality of storage chambers 30 , 31 , 32 , and 33 formed inside the main body 10 to store food, and a door 40 for opening or closing the plurality of storage chambers 30 , 31 , 32 , and 33 .

[0086] In the current example, a case where the door 40 includes four doors 41 , 42 , 43 , and 44 will be described as an example.

[0087] The plurality of storage chambers 30 , 31 , 32 , and 33 may include an upper storage chamber 30 , lower storage chambers 31 and 32 , and a central storage chamber 33 formed between the upper storage chamber 30 and the lower storage chambers 31 and 32 .

[0088] The upper storage chamber 30 may be a refrigerating chamber 30 for refrigerating food. The refrigerating chamber 30 may be maintained at a temperature of about 0° C. to about 5° C. to keep the food refrigerated.

[0089] The lower storage chambers 31 and 32 may include a first freezing chamber 31 for keeping food frozen, and a temperature-adjusted first temperature conversion chamber 32. The first freezing chamber 31 may be maintained at a temperature of about 0°C to about -30°C to keep food frozen.

[0090] The temperature of the first temperature conversion chamber 32 can be adjusted between a refrigeration temperature and a freezing temperature. For example, the temperature of the first temperature conversion chamber 32 can be adjusted within a temperature zone among four temperature zones, which are a freezing temperature zone of -23°C to -17°C, a thin ice temperature zone of -5°C, a soft freezing temperature zone of -1°C, and a refrigeration temperature zone of 2°C.

[0091] The refrigerator 1 may include an input device (not shown) for receiving an input of a temperature zone selected by a user from a plurality of temperature zones. In response to the input of the temperature zone selected by the user, the temperature of the first temperature conversion chamber 32 may be adjusted to the selected temperature zone.

[0092] However, an embodiment of the refrigerator 1 may not necessarily include the first temperature conversion chamber 32, and a freezing chamber may be provided by changing the design instead of the first temperature conversion chamber 32 so that all of the lower storage chambers 31 and 32 may be implemented as freezing chambers.

[0093] The central storage chamber 33 may include a second freezing chamber 35 and a second temperature conversion chamber 36 placed side by side.

[0094] Like the first freezing chamber 31, the second freezing chamber 35 may be maintained at a temperature of 0°C to -30°C to keep food frozen.

[0095] The second freezing chamber 35 may have a relatively smaller size than the first freezing chamber 31 to serve as an auxiliary freezing chamber. The first freezing chamber 31 may be opened or closed by the first lower door 43, and the second freezing chamber 35 may be opened or closed by the first upper door 41. For example, by storing relatively less frequently used and larger-sized food in the first freezing chamber 31 and storing relatively more frequently used and smaller-sized food in the second freezing chamber 35, management and storage efficiency may be improved while minimizing unnecessary outflow of cold air.

[0096] Similar to the first temperature conversion chamber 32 , the temperature of the second temperature conversion chamber 36 can also be adjusted between a refrigerating temperature and a freezing temperature.

[0097] However, an embodiment of the refrigerator 1 may not necessarily include the second temperature conversion chamber 36, and a freezing chamber may be provided by changing the design instead of the second temperature conversion chamber 36 so that the entire central storage chamber 33 may be implemented as a freezing chamber.

[0098] The body 10 may be substantially box-shaped and the front side of the body 10 may be open. The body 10 may include an inner shell 11, an outer shell 12 coupled to the outer side of the inner shell 11, and an insulation 13 disposed between the inner shell 11 and the outer shell 12.

[0099] The inner case 11 may be injection molded with a resin material. The inner case 11 may form therein a refrigerating chamber 30, a first freezing chamber 31, a first temperature conversion chamber 32, a second freezing chamber 35, and a second temperature conversion chamber 36. That is, the inner case 11 may limit the space of each storage chamber.

[0100] The outer shell 12 may be formed of a metal material. The insulator 13 disposed between the inner shell 11 and the outer shell 12 may be a polyurethane foam insulator, and may be used together with a vacuum insulation panel as needed.

[0101] The body 10 may include a top wall 20 , a bottom wall 21 , a left wall 22 , a right wall 23 , a rear wall 24 , a first partition wall 25 , a second partition wall 26 , a first middle wall 27 , and a second middle wall 29 .

[0102] The first partition wall 25 may horizontally partition the inner space of the body 10. That is, the first partition wall 25 may partition the central storage chamber 33 from the refrigerating chamber 30. The first partition wall 25 may extend substantially horizontally from the left wall 22 to the right wall 23, and include an inner case 11 and an insulator 13.

[0103] The second partition wall 26 may horizontally partition the inner space of the body 10. That is, the second partition wall 26 may partition the central storage chamber 33. The second partition wall 26 may extend substantially horizontally from the left wall 22 to the right wall 23, and include the inner shell 11 and the insulator 13.

[0104] The first intermediate wall 27 may partition the central storage chamber 33 side by side. That is, the first intermediate wall 27 may partition the second temperature conversion chamber 36 from the second freezing chamber 35. The first intermediate wall 27 may extend substantially vertically from the first partition wall 25 to the second partition wall 26, and include the inner shell 11 and the insulator 13.

[0105] The second intermediate wall 29 may partition the lower storage chambers 31 and 32 side by side. That is, the second intermediate wall 29 may partition the first temperature conversion chamber 32 from the first freezing chamber 31. The second intermediate wall 29 may extend substantially vertically from the second partition wall 26 to the bottom wall 21, and include the inner shell 11 and the insulator 13.

[0106] Each storage chamber may include a shelf 37 on which food is placed, an airtight container 38 that seals and stores food, and a drawer 39 that slides forward and backward to be inserted into or taken out of the refrigerator 1 .

[0107] In the second freezing chamber 32, a small freezer box 90 having a storage space for storing food may be provided. The small freezer box 90 may be provided to easily store food and put food into / take out the second freezing chamber 32 having a relatively narrow space. The small freezer box 90 may include an intermediate wall 28.

[0108] The door 40 for opening or closing the plurality of storage chambers 30, 31, 32, and 33 may include four doors of a first upper door 41, a second upper door 42, a first lower door 43, and a second lower door 44. The doors 41, 42, 43, and 44 may be rotatably coupled to the body 10.

[0109] The first upper door 41 and the second upper door 42 may be rotatably coupled to the body 10 by an upper hinge and a middle hinge 15. The middle hinge 15 may be coupled to the second partition wall 26 to support the first upper door 41 and the second upper door 42. The first upper door 41 and the second upper door 42 may rotate in opposite directions to be opened or closed.

[0110] The first upper door 41 and the second upper door 42 can open or close the refrigerating chamber 30 and the central storage chamber 33 together. More specifically, the first upper door 41 can open or close a portion of the refrigerating chamber 30 and the second freezing chamber 35, and the second upper door 42 can open or close the remaining portion of the refrigerating chamber 30 and the second temperature conversion chamber 36.

[0111] The portion of the refrigerating chamber 30 opened or closed by the first upper door 41 may be an area facing the closed first upper door 41 , and the remaining portion of the refrigerating chamber 30 opened or closed by the second upper door 42 may be an area facing the closed second upper door 42 .

[0112] Therefore, by opening the first upper door 41, the refrigerating chamber 30 and the second freezing chamber 35 may be opened at the same time, and by opening the second upper door 42, the refrigerating chamber 30 and the second temperature conversion chamber 35 may be opened at the same time.

[0113] A filler 48 may be provided on the first upper door 41 to prevent cold air from flowing out between the first upper door 41 and the second upper door 42 when the first upper door 41 and the second upper door 42 are closed.

[0114] On the rear surfaces of the first upper door 41 and the second upper door 42, an upper seal 45 may be provided to prevent the outflow of cold air between the first upper door 41 and the second upper door 42 and the main body 10 when the first upper door 41 and the second upper door 42 are closed. On the rear surfaces of the first lower door 43 and the second lower door 44, a lower seal 46 may be provided to prevent the outflow of cold air between the first lower door 43 and the second lower door 44 and the main body 10 when the first lower door 43 and the second lower door 44 are closed. The upper seal 45 and the lower seal 46 may be formed of a rubber material.

[0115] The first lower door 43 and the second lower door 44 may be rotatably coupled to the body 10 through the middle hinge 15 and the lower hinge. The first lower door 43 and the second lower door 44 may be rotated in opposite directions to be opened or closed.

[0116] The first lower door 43 may open or close the first freezing chamber 31. The second lower door 44 may open or close the first temperature conversion chamber 32.

[0117] Meanwhile, a storage room for storing food may be provided in the door 40 as well as the main body 10. For example, door racks 44 of various shapes for placing food may be installed at the rear side of the door 40. For example, the door racks 44 may separately store liquid food such as beverages, alcohol, bottled water, and milk, and fragile food having a specific shape such as eggs.

[0118] The above-mentioned structure and configuration of the refrigerator 1 may be an example applicable to the refrigerator 1 according to the embodiment of the present disclosure, and the refrigerator 1 according to the embodiment of the present disclosure may have another structure and configuration different from the above-mentioned structure and configuration. However, for the sake of detailed description, the embodiment described below is based on the refrigerator having Figures 1 to 3 The refrigerator 1 has the structure and construction shown.

[0119] Figure 4 is a control block diagram of a refrigerator according to an embodiment of the present disclosure.

[0120] Reference Figure 4 According to an embodiment of the present disclosure, the refrigerator 1 may include a door driver 300 for generating power for automatically opening or closing at least one of a plurality of doors 41, 42, 43, and 44, a camera 110 for capturing an image of a surrounding environment, a controller 200 (e.g., at least one processor) for sending a control signal for opening at least one of the doors 41, 42, 43 to the door driver 300 based on an object recognized in the image of the surrounding environment captured by the camera, and a cooler 400 for supplying cold air to a plurality of storage chambers 30, 31, 32, and 33.

[0121] In the current embodiment, the description is given based on the refrigerator 1 including four doors 41, 42, 43 and 44, however, as described above, the embodiments of the present disclosure can also be applied to refrigerators having one, two or three doors. In addition, the embodiments of the present disclosure can be applied to refrigerators having five or more doors. That is, the number of doors is not limited.

[0122] Usually, the user opens or closes the refrigerator door by himself to put food into the refrigerator or take food out of the refrigerator. However, the refrigerator 1 according to the embodiment of the present disclosure can be opened or closed automatically, and can also be opened or closed manually by the user. Therefore, by automatically opening or closing the door in a case where the user's hands are not free due to the user holding the food to be stored in the refrigerator 1 or the object related to the food to be taken out of the refrigerator 1 in his / her hands, etc., the convenience of the user can be improved.

[0123] The door driver 300 for generating power to automatically open or close the door 40 may include a motor 320 and a driver circuit 310 for driving the motor 320. The driver circuit 310 may rotate the motor 320 forward or backward according to a control signal transmitted from the controller 200, and the door 40 may be opened or closed by the forward or backward rotation of the motor 320.

[0124] In the case where multiple doors 40 are provided, multiple door drivers 300 may be provided to correspond to the number of doors 40 to provide power to the respective doors 40, or a single door driver 300 may provide power to the multiple doors 40 by using a switching device capable of switching power transmission.

[0125] The power generated by the door driver 300 can be transmitted to the door 40 through a power transmission device including a plurality of gears. In addition, the power transmission device can be switched between a power transmission state for transmitting the power of the motor 320 to the door 40 and a power non-transmission state for not transmitting power to the door 40, so as to enable switching between automatic opening / closing and manual opening / closing of the door 40.

[0126] In the embodiment of the refrigerator 1, the method of automatically opening or closing the door 40 is not limited, and thus, any one of various technologies for automatically opening or closing the refrigerator may be applied.

[0127] Meanwhile, the objects stored in the refrigerator 1 according to the embodiment of the present disclosure are not limited to food. Objects stored in the refrigerator 1, such as containers for storing food, objects that need to be kept refrigerated or frozen for some purpose, and objects that are kept refrigerated or frozen regardless of some purpose according to the user's intention, are not limited. Therefore, in the following embodiments of the present disclosure, food and non-food are collectively referred to as objects.

[0128] The camera 110 may be located at a position capable of capturing an image of the surrounding environment of the door 40. For example, the camera 110 may be located on at least one of the plurality of doors 41, 42, 43, and 44, or on the top of the body 10.

[0129] The camera 110 may be installed to face the front direction of the refrigerator 1. The camera 110 may be located at a position capable of photographing an object located in front of the refrigerator 1, and the viewing angle or tilt angle of the camera 110 is not limited. Therefore, the image of the surrounding environment photographed by the camera 110 may include an image of the surrounding environment located beside the refrigerator 1, and an image of the surrounding environment located in front of the refrigerator 1. In addition, the tilt angle of the camera 110 may be tilted downward.

[0130] Meanwhile, a camera equipped in a mobile device such as a mobile phone or a tablet personal computer (PC) may perform the function of the camera 110. For example, a stand for supporting the mobile device may be provided on the refrigerator 1, and the mobile device equipped with the camera may be held on the refrigerator 1 to capture images of the surrounding environment. The images of the surrounding environment captured by the mobile device may be transmitted to the refrigerator 1 or a server 800 (see FIG. 1 ) that manages the refrigerator 1. Fig. 22 ). In order to send an image to the refrigerator 1 or the server 800, a refrigerator management application may be installed in the mobile device.

[0131] The image of the surrounding environment captured by the camera 110 may include an object carried by a user located around one of the plurality of doors 41 , 42 , 43 , and 44 .

[0132] The controller 200 itself may recognize an object from an image of the surrounding environment captured by the camera 110, or transmit the image of the surrounding environment to the server 800 (see Fig. 22 ) and receives the result of object recognition from the server 800. First, a case where the controller 200 recognizes an object by itself will be described as an example.

[0133] The controller 200 may apply an object recognition algorithm to the image of the surrounding environment captured by the camera 110 to recognize the object. The controller 200 may pre-set objects that can be stored in the refrigerator 1 and recognize a preset object among the preset objects from the image of the surrounding environment, or the controller 200 may recognize an object from the image of the surrounding environment regardless of the type of the object and then determine whether the recognized object is a preset object.

[0134] In all cases, the controller 200 may transmit a control signal for opening at least one of the plurality of doors 41 , 42 , 43 , and 44 to the door driver 300 according to recognition of a preset object from an image of the surrounding environment captured by the camera 110 .

[0135] The objects that can be stored in the refrigerator 1 may be set during the manufacturing process of the refrigerator 1, or may be set by a user who has purchased the refrigerator 1. In addition, after the objects that can be stored in the refrigerator 1 are set during the manufacturing process of the refrigerator 1, the user may add new objects, or the user may delete or replace some objects. In addition, in the case where the server 800 manages a plurality of refrigerators, objects that can be stored in the refrigerator 1 may be added based on information about objects stored in refrigerators of other users, which will be described later.

[0136] As described above, the refrigerator 1 may include a plurality of storage chambers 30, 31, 32, and 33 for storing food. The controller 200 may determine a storage chamber corresponding to the identified object from among the plurality of storage chambers 30, 31, 32, and 33, and transmit a control signal to the door driver 300 for opening the door of the storage chamber corresponding to the identified object.

[0137] The storage room corresponding to the identified object may be a storage room suitable for storing the identified object. For example, based on the result of identifying the object to be kept refrigerated, the controller 200 may determine that the storage room corresponding to the identified object is the refrigerating room 30, and based on the result of identifying the object to be kept frozen, the controller 200 may determine that the storage room corresponding to the identified object is the freezing room 31 and 35. In addition, based on the result of identifying the object to be kept at a specific temperature, neither refrigerated nor frozen, the controller 200 may determine that the storage room corresponding to the identified object is the temperature conversion room 32 and 36.

[0138] Information about the storage rooms corresponding to the identified objects may have been pre-stored in the controller 200. More specifically, information about the storage rooms corresponding to the respective objects may be stored to match the objects, or storage temperatures suitable for the respective objects may be stored to match the objects, and temperatures of the respective storage rooms may be stored to match the storage rooms.

[0139] In addition, information about a storage room corresponding to the recognized object may be obtained from the server 800 .

[0140] In addition, in the case where the object includes an identification mark including information about the object, such as a barcode or a Quick Response (QR) code, the controller 200 can recognize such a graphic object from the image of the surrounding environment and obtain information about the object from the recognized identification mark. The information about the object may include information about the type of the object, that is, what the object is, and information about the storage environment of the object (storage temperature, expiration date, etc.).

[0141] The door of the storage room corresponding to the recognized object may be a door capable of opening and closing the corresponding storage room. Figure 2 , the first upper door 41 may open or close a portion of the refrigerating chamber 30 and the second freezing chamber 35, and the second upper door 42 may open or close the remaining portion of the refrigerating chamber 30 and the second temperature conversion chamber 36. Therefore, according to the result of determination that the storage chamber corresponding to the recognized object is the second freezing chamber 35, a control signal for opening the first upper door 41 may be transmitted to the door driver 300, and according to the result of determination that the storage chamber corresponding to the recognized object is the second temperature conversion chamber 36, a control signal for opening the second upper door 42 may be transmitted to the door driver 300. In addition, according to the result of determination that the storage chamber corresponding to the recognized object is the refrigerating chamber 31, a control signal for opening at least one of the first upper door 41 or the second upper door 42 may be transmitted to the door driver 300.

[0142] At the same time, the controller 200 may determine the door corresponding to the recognized object and send a control signal for opening the door corresponding to the recognized object to the door driver 300. The door corresponding to the recognized object may be a door capable of opening a storage room suitable for storing the recognized object. To this end, the controller 200 or the server 800 may match each object with the door corresponding to the corresponding object and store the matching result.

[0143] The controller 200 may include at least one memory storing a program for performing the above-mentioned operations and operations to be described later and various data required to execute the program, and at least one processor for executing the stored program.

[0144] In the case where multiple memories and multiple processors are provided, the memories and processors may be integrated into one chip, or some memories and processors may be physically separated.

[0145] The cooler 400 may supply cool air to the storage chambers 30 , 31 , 32 , and 33 by utilizing evaporation of a refrigerant, thereby maintaining the temperatures of the storage chambers 30 , 31 , 32 , and 33 within a preset temperature range.

[0146] The cooler 400 may include a compressor 410 for compressing gas refrigerant, a condenser 420 for converting the compressed gas refrigerant into liquid refrigerant, an expander 430 for decompressing the liquid refrigerant, and an evaporator 440 for converting the decompressed liquid refrigerant into a gas state.

[0147] The compressor 410 can compress the sucked gas refrigerant at low temperature and low pressure, and discharge the gas at high temperature and high pressure. To this end, the compressor 410 can forcibly suck in the refrigerant and compress the sucked refrigerant to become a high-temperature and high-pressure gas, wherein the refrigerant can be sucked in by the rotation force of the installed motor.

[0148] The refrigerant can circulate in the cooling cycle of the refrigerator 1 by the force of the refrigerant sucked by the compressor 410. Therefore, the circulation amount or circulation speed of the refrigerant can depend on the driving level of the compressor 410, and further, the cooling efficiency of the refrigerator 1 can also depend on the driving level of the compressor 410.

[0149] The high-temperature and high-pressure refrigerant compressed by the compressor 410 may be transferred to the condenser 420 .

[0150] The condenser 420 may be connected to the high-pressure side exhaust pipe of the compressor 410 to heat-exchange the high-temperature and high-pressure refrigerant compressed by the compressor 410 with the surrounding air, thereby condensing the refrigerant into a liquid state. In the condenser 420, the refrigerant may be condensed into a liquid to dissipate heat to the outside, and thus, the temperature of the refrigerant may be reduced.

[0151] The expander 430 may decompress the room temperature and high pressure liquid refrigerant condensed by the condenser 420 by expanding the refrigerant into a two-phase refrigerant of low temperature and low pressure and a mixture of liquid and gas components. The expander 430 may be an expansion valve.

[0152] In addition, as the expander 430 , a capillary tube may be used instead of the expansion valve. The capillary tube may be implemented as a thin tube, and the refrigerant passing through the capillary tube may be decompressed and transferred to the evaporator 440 .

[0153] The evaporator 440 may evaporate the low-temperature and low-pressure liquid refrigerant expanded by the expander 430 into a gas state to supply cool air.

[0154] In this manner, the cooler 400 may supply cool air to the storage chambers 30 , 31 , 32 , and 33 by utilizing a phenomenon in which the decompressed liquid refrigerant absorbs heat energy of surrounding air when becoming liquid.

[0155] At the same time, the temperatures of a portion of the plurality of storage chambers 30, 31, 32, and 33 may be independently controlled. To this end, a plurality of coolers 400 may be provided. The plurality of coolers 40 may be independently configured or may share some components.

[0156] For example, three coolers 400 may be provided. One of the three coolers 400 may supply cold air to the refrigerating chamber 30, another may supply cold air to the first and second freezing chambers 31 and 35, and the remaining one may supply cold air to the first and second temperature conversion chambers 32 and 36.

[0157] The controller 200 can send a control signal to the cooler 400 for supplying cold air to the refrigerating chamber 30 to control the temperature of the refrigerating chamber 30, send a control signal to the cooler 400 for supplying cold air to the first freezer chamber 31 and the second freezer chamber 35 to control the temperature of the first freezer chamber 31 and the second freezer chamber 32, and send a control signal to the cooler 400 for supplying cold air to the first temperature conversion chamber 32 and the second temperature conversion chamber 36 to control the temperature of the first temperature conversion chamber 32 and the second temperature conversion chamber 36.

[0158] For example, based on the result of determining that the identified object is an object that needs to be stored at a specific temperature, the controller 200 can determine a storage room corresponding to the identified object and a storage temperature corresponding to the identified object, and control the temperature of the storage room corresponding to the identified object to the storage temperature corresponding to the identified object.

[0159] After the object is stored in the storage room, the temperature of the storage room may be controlled. For example, after opening and closing the door of the storage room corresponding to the recognized object, it may be assumed that the recognized object has been stored in the corresponding storage room, and the temperature of the storage room may be controlled. In addition, the temperature of the storage room may be controlled based on the internal sensor 150 (see FIG. 15A ) provided in the storage room. Fig.13 ) output to determine whether the object has been stored in the storage room, which will be described later.

[0160] Hereinafter, various embodiments in which the refrigerator 1 recognizes an object and automatically opens or closes a door will be described in detail.

[0161] Figure 5 is a control block diagram of a refrigerator further including a proximity sensor for sensing a user approaching according to an embodiment of the present disclosure, and Figure 6 and Figure 7 An example is shown in which a refrigerator opens a door corresponding to a recognized object according to an embodiment of the present disclosure.

[0162] Reference Figure 5According to an embodiment of the present disclosure, the refrigerator 1 can activate the camera 110 in response to a user approaching the refrigerator 1 to protect the user's privacy and reduce power consumption.

[0163] For this purpose, refer to Figure 5 The refrigerator 1 according to the embodiment of the present disclosure may further include a proximity sensor 120 for sensing the approach of a user. The proximity sensor 120 may be one of various types of sensors, such as an ultrasonic sensor, an infrared sensor, an optical sensor, and a magnetic sensor, which changes its output when an object approaches.

[0164] The controller 200 may activate the camera 110 based on the output of the proximity sensor 120. More specifically, the controller 110 may activate the camera 110 in response to the output of the proximity sensor 120 indicating that the user is approaching.

[0165] The output of the proximity sensor 120 itself may be a value indicating that the user is approaching or not approaching according to the specification of the proximity sensor 120, or the controller 200 may determine the approach of the user based on the output of the proximity sensor 120. In the latter case, the controller 200 may store a reference value indicating that the user is approaching according to the type of the proximity sensor 120, and activate the camera 110 in response to the output of the proximity sensor 120 being greater than or equal to the reference value or less than or equal to the reference value.

[0166] The activation and deactivation of the camera 110 may be implemented in software or hardware. In the case where the activation and deactivation of the camera 110 is implemented in software, the camera 110 may be activated by starting to shoot. In the case where the activation and deactivation of the camera 110 is implemented in hardware, a shutter structure may be located in front of the camera 110, the shutter structure may move in response to the activation of the camera 110 to deviate from the field of view (FOV) of the camera 110, and in response to the deactivation of the camera 110, the shutter structure may move again in front of the camera 110 to be within the FOV of the camera 110.

[0167] According to activation of the camera 110 , the camera 110 may capture an image of the surrounding environment, and the controller 200 may recognize an object from the image of the surrounding environment.

[0168] Reference Figure 6 , in the case where the user carries a Kimchi container, the image of the surrounding environment captured by the camera 110 may include the Kimchi container carried by the user, and the controller 200 may apply an object recognition algorithm to the image of the surrounding environment to recognize the Kimchi container.

[0169] Information about the storage chamber corresponding to the kimchi container or the door corresponding to the kimchi container may be stored in the memory of the controller 200. For example, the first temperature conversion chamber 32 whose temperature can be controlled to a specific temperature may be stored as the storage chamber corresponding to the kimchi container, and the controller 200 may send a control signal to the door driver 300 to open the first temperature conversion chamber 32, thereby opening the second lower door 44.

[0170] In this way, in the case where the storage chamber corresponding to the identified object is a temperature conversion chamber whose temperature can be independently controlled, the controller 200 can control the temperature of the temperature conversion chamber to the storage temperature corresponding to the identified object. To this end, in the memory of the controller 200, information about the storage temperature corresponding to each object, that is, the storage temperature at which the corresponding object is properly stored, can be stored, and the processor of the controller 200 can determine the storage temperature corresponding to the identified object based on the stored information.

[0171] Reference Figure 6 , the controller 200 may determine the storage temperature corresponding to the Kimchi container, and transmit a control signal for supplying cold air to the first temperature conversion chamber 32 to the cooler 400 to control the temperature of the first temperature conversion chamber 32 to the storage temperature corresponding to the Kimchi container.

[0172] In addition, refer to Figure 7 , in the case where the user carries a can of beer, the image of the surrounding environment captured by the camera 110 may include the can of beer carried by the user, and the controller 200 may apply an object recognition algorithm to the image of the surrounding environment to recognize the can of beer.

[0173] Information about a storage chamber corresponding to canned beer or a door corresponding to canned beer may be stored in the memory of the controller 200. For example, the refrigerating chamber 30 may be stored as a storage chamber corresponding to canned beer, and the controller 200 may send a control signal to the door driver 300 to open the refrigerating chamber 30, thereby opening the first upper door 41.

[0174] Meanwhile, the door corresponding to the refrigerating chamber 30 may be the first upper door 41 and the second upper door 42. The controller 200 may select and open any one or both of the first upper door 41 and the second upper door 42. The controller 200 may select and open any one or both of the first upper door 41 and the second upper door 42 based on the internal sensor 150 (see FIG. 1 ) to be described later. Fig.13 ) output to determine a location capable of receiving canned beer and to open a door corresponding to the determined location.

[0175] Figure 8 is a control block diagram of a refrigerator further including a temperature sensor according to an embodiment of the present disclosure.

[0176] Reference Figure 8The refrigerator 1 according to the embodiment of the present disclosure may further include a temperature sensor 130 for sensing the temperature of an object located around at least one door 40. The temperature sensor 130 may be a temperature sensor that senses the external temperature of the refrigerator 1, particularly the temperature of an object located around the door 40, rather than sensing the internal temperature of the refrigerator 1. The temperature sensor 130 may be a contactless temperature sensor.

[0177] The controller 200 may determine at least one of a storage chamber or a storage temperature corresponding to the recognized object based on the temperature sensed by the temperature sensor 130 and the object recognized from the image of the surrounding environment captured by the camera 110 .

[0178] For example, the temperature sensor 130 may be a thermal imaging camera. The controller 200 may determine the temperature of an object identified from the image based on the image of the surrounding environment captured by the thermal imaging camera, and determine at least one of the storage room or the storage temperature based on the determined temperature of the object. The controller 200 may select a storage room or change the storage temperature of the same storage room according to the temperature of the object.

[0179] According to a detailed example, in the case where the recognized object is a kimchi container, as described above, the storage chamber corresponding to the recognized object may be the first temperature conversion chamber 32. However, the storage temperature may be controlled differently according to the temperature of the kimchi container measured by the temperature sensor 130. The controller 200 may control the storage temperature to a low temperature according to the temperature of the kimchi container that is higher than or equal to a reference value, and may control the storage temperature to a high temperature according to the temperature of the kimchi container that is lower than or equal to another reference value. As a preset time passes after the kimchi container is stored, the temperature of the first temperature conversion chamber 32 may be controlled again to a normal temperature range, that is, a temperature range suitable for normal storage of kimchi.

[0180] According to another example, in the case where the identified object is canned beer, the storage chamber corresponding to the identified object may be the refrigerating chamber 30. However, the storage chamber corresponding to the canned beer may be changed according to the temperature of the canned beer measured by the temperature sensor 130. In response to the temperature of the canned beer measured by the temperature sensor 130 being higher than or equal to the reference value, the controller 200 may determine the storage chamber corresponding to the canned beer as the second temperature conversion chamber 36. In this case, the temperature of the second temperature conversion chamber 36 may also be controlled based on the current temperature of the canned beer.

[0181] As described above, since the refrigerator 1 according to the embodiment of the present disclosure recognizes the object by itself and automatically opens the door, the user carrying the object himself or herself does not need to operate the door, and since the door of the storage chamber suitable for storing the recognized object is opened, the object can be properly stored. In addition, by performing temperature control suitable for the object, the object can be kept at an optimal temperature.

[0182] Fig. 9 is a control block diagram of a refrigerator further including a speaker according to an embodiment of the present disclosure, Fig.10 and Fig.11 An example of a notification sound output from a speaker of a refrigerator according to an embodiment of the present disclosure is shown.

[0183] Reference Fig. 9 The refrigerator 1 according to the embodiment of the present disclosure may further include a speaker 510. The speaker 510 may be provided on at least one of the plurality of doors 41, 42, 43, and 44 or on the body 10.

[0184] The controller 200 may control the speaker 510 to output a notification sound when the door 40 is automatically opened, for notifying the user that the door 40 is automatically opened. Therefore, an accident that may occur when the door 40 is opened may be prevented in advance.

[0185] The notification sound output from the speaker 510 may depend on the relative position of the user with respect to the door 40 expected to be opened. The door 40 expected to be opened may be a door corresponding to an object recognized from the image of the surrounding environment, or a door of a storage room corresponding to an object recognized from the image of the surrounding environment.

[0186] The controller 200 may determine a relative position of the user with respect to the door 40 to be opened, and control the speaker 510 to output a notification sound differently according to the relative position of the user.

[0187] The relative position of the user with respect to the door 40 that is expected to be opened may include the direction in which the user is located with respect to the door 40 that is expected to be opened, and the distance between the user and the door 40 that is expected to be opened.

[0188] The controller 200 may determine the relative position of the user based on the image of the surrounding environment captured by the camera 110. In addition, in the case where the refrigerator 1 further includes a proximity sensor 120, the controller 200 may determine the relative position of the user based on the output from the proximity sensor 120 or by using the image of the surrounding environment and the output from the proximity sensor 120.

[0189] The warning level of the notification sound may be changed according to the relative position of the user with respect to the door 40 expected to be opened. In a case where the user is located at a position where he / she will not hit the door 40 expected to be opened even after the door 40 is opened (a position where collision is not possible), the warning level of the notification sound may be set to the first level, and in a case where the user is located at a position where he / she may hit the door 40 after the door 40 expected to be opened (a position where collision is possible), the warning level of the notification sound may be set to the second level.

[0190] Reference Fig.10 , in a case where the door expected to be opened is the second lower door 44 and the user is located next to the second lower door 44 so that the user will not bump into the second lower door 44 even after the second lower door 44 is opened, the warning level of the notification sound can be set to the first level, and accordingly, the notification sound "The right lower door is open, please be careful not to bump into the door" can be output through the speaker 510.

[0191] In addition, in such a case, although the user is located in front of the second lower door 44, he / she is at a distance from the second lower door 44 at which he / she will not hit the second lower door 44 after the second lower door 44 is opened, and the warning level of the notification sound can still be set to the first level.

[0192] Reference Fig.11 , in the case where the door expected to be opened is the second lower door 44 and the user is located in front of the second lower door 44 at a distance from the second lower door 44, at which the user may bump into the second lower door 44 after the second lower door 44 is opened, the warning level of the notification sound may be set to the second level. In this case, a notification sound “right lower door is open, please step back to avoid bumping into the door” may be output through the speaker 510.

[0193] After a preset time has passed after the notification sound is output, the controller 200 may send a control signal to the door driver 300 to open the door 40 that is expected to be opened. However, in the case where the user is already in the possible collision position, the controller 200 may determine the user's position again after outputting the notification sound, and open the door 40 according to the result of the determination that the user has moved to the impossible collision position. In the case where the user is still in the possible collision position, the controller 200 may output the notification sound again.

[0194] The content of the notification sound output through the speaker 510 is not limited to the above examples. A notification sound having a content different from the above examples may be output, or a melody or buzzer sound without any content may be output as long as the notification sound is output to prevent a collision between the user and the door 40. In addition, the warning level of the notification sound may be subdivided into warning levels of more than two warning levels.

[0195] Meanwhile, in addition to the notification sound for opening the door 40, the speaker 510 may be used to output a notification sound for other information. For example, in the case where an object appearing on an image of the surrounding environment is not correctly recognized, a notification sound may be output through the speaker 510. More specifically, in the case where the object is not recognized although the camera 110 is activated in response to the user's approach to capture an image of the surrounding environment, or in the case where the existence of an object is recognized but it is not determined what the recognized object is, the controller 200 may control the speaker 510 to output a notification sound. The notification sound may be output as a voice including information indicating that the object is not correctly recognized, or may be output as a melody or a buzzer sound.

[0196] Alternatively, there may be a case where the user moves away from the refrigerator 1 in a state where object recognition of the refrigerator 1 is not completed or the door 4 is not fully opened. In this case, the controller 200 may control the speaker 510 to output a notification sound. The notification sound may be output as a voice including information indicating that object recognition is not completed or the door 40 is not fully opened, or as a melody or a buzzer. In the case where the notification sound is output as a buzzer, a buzzer with greater intensity may be output as the user moves away from the refrigerator 1.

[0197] The refrigerator 1 may include an input for turning on / off the speaker 510. A user who does not want to hear the notification sound output from the speaker 510 may operate the input to turn off the speaker 510.

[0198] The controller 200 can store abnormal situations, such as situations where the refrigerator 1 fails to recognize an object, and provide the user with historical information about the abnormal situation. The user can view the abnormal situation through the display 520 (see FIG. Fig.17 ) or through a mobile device equipped with a display (such as a mobile phone or tablet) to check historical information about abnormal situations.

[0199] In addition, the refrigerator 1 may receive feedback from the user to obtain information about an unrecognized object, and the obtained information may be used for object recognition to be performed later.

[0200] Fig.12 An example of a refrigerator that adjusts the opening angle of a door based on the relative position between the door and a user according to an embodiment of the present disclosure is shown.

[0201] Reference Fig.12 , the refrigerator 1 according to the embodiment of the present disclosure can control the opening angle of the door 40 based on the relative position of the user with respect to the door 40 expected to be opened to prevent the collision between the door 40 and the user.

[0202] According to the above method, the controller 200 can determine the relative position of the user with respect to the door 41 expected to be opened, and according to the result of the determination that the user is in a position where he / she may collide with the door 41 expected to be opened, the controller 200 can open the door 41 to an angle α1 at which the door 41 will not collide with the user, such as Fig.12 shown.

[0203] The controller 200 may calculate the opening angle of the door 41 based on the relative position of the user, the length of the door 41, etc., or pre-store the opening angle of the door 41 corresponding to the relative position of the user. In the latter case, the controller 200 may classify the opening angle of the door 41 into three levels of 30 degrees, 60 degrees, and 90 degrees, store the three levels, and control the opening angle of the door 41 to one of the three levels according to the relative position of the user.

[0204] Furthermore, in a case where the user is in a position where he / she will not hit the door 41 expected to be opened, the controller 200 may open the door 41 to the opening angle α2 set as a default value regardless of the relative position of the user.

[0205] The opening angle α2 set as a default value may depend on the storage compartment, or may be a fixed value regardless of the storage compartment.

[0206] In the former case, in the case where the storage room corresponding to the identified object is a storage room set on the rear surface of the door 41, 42, 43 or 44, the door 41 can be opened to a larger opening angle than the case where the storage room corresponding to the identified object is a storage room set in the main body 10.

[0207] In the latter case, the door 41 can be opened to the maximum opening angle regardless of the storage room corresponding to the identified object. In addition, the controller 200 can open the door 41 to an angle less than the maximum opening angle and allow the user to open the door 41 significantly with his / her body.

[0208] In addition, the user may change the opening angle α2 set as a default value or select an opening angle to be set as a default value through an input device provided in the refrigerator 1 .

[0209] Meanwhile, in addition to the opening angle of the door 40, the refrigerator 1 may also control the opening speed of the door 40. The opening speed of the door 40 may be controlled according to the relative position of the user. For example, when the relative position of the user is closer to the door 40, the refrigerator 1 may control the door 40 to open at a low opening speed to provide the user with time to avoid the door 40 before hitting the door 40.

[0210] The controller 200 may calculate the opening speed of the door 40 based on the relative position of the user, the length of the door 40, etc., or pre-store the opening speed of the door 40 corresponding to the relative position of the user. As an example of the latter case, the controller 200 may classify the opening speed of the door 40 into three levels, store the three levels, and control the opening speed of the door 40 to one of the three levels according to the relative position of the user.

[0211] Alternatively, the controller 200 can control the opening speed of the door 40 according to the user's selection. In this case, the user can select a level from a plurality of levels indicating the opening speed of the door 40, change the opening speed of the door 40 set to a default value, or input a value of the opening speed of the door 40.

[0212] The above example has been described assuming that the object recognized from the image of the surrounding environment is an object that the user wants to store in the refrigerator 1. However, in an embodiment of the refrigerator 1, the object recognized from the image of the surrounding environment may be an object that the user wants to take out of the refrigerator 1.

[0213] For example, in a case where the object recognized from the image of the surrounding environment is an empty cup, the object that the user wants to take out from the refrigerator 1 may be liquids such as bottled water, milk, and beverages, and in a case where the object recognized from the image of the surrounding environment is an empty kimchi container, the object that the user wants to take out from the refrigerator 1 may be kimchi.

[0214] In the above examples, a description has been given that the controller 200 can pre-store information about the storage rooms corresponding to the respective objects. In the above examples, the storage rooms corresponding to the objects are storage rooms suitable for storing the corresponding objects. However, in the current example, the storage rooms corresponding to the objects may be storage rooms that store objects related to the corresponding objects or estimate the storage of related objects.

[0215] For example, in a case where the object recognized from the image of the surrounding environment is an empty cup, the object related to the empty cup may be bottled water or beverages, and the storage room corresponding to the empty cup may be a storage room for storing liquids such as bottled water and beverages.

[0216] Furthermore, in the case where the object recognized from the image of the surrounding environment is an empty kimchi container or an empty side dish container, the object associated with the empty kimchi container or the empty side dish container may be kimchi, and the storage chamber corresponding to the empty kimchi container or the empty side dish container may be the second temperature conversion chamber 36 for storing kimchi.

[0217] As described above, such information may be pre-stored in the controller 200, and the information stored in the controller 200 may be updated according to the use history of the refrigerator 1. In addition, the information may be obtained from the server 800 that manages a plurality of refrigerators.

[0218] In addition, in the case where the refrigerator 1 includes the internal sensor 150, a storage room in which an object corresponding to an object recognized from an image of the surrounding environment is stored may be determined based on an output of the internal sensor 150, which will be described later.

[0219] Fig.13 is a control block diagram of a refrigerator further including an internal sensor according to an embodiment of the present disclosure.

[0220] Reference Fig.13 , the refrigerator 1 according to the embodiment of the present disclosure may further include an internal sensor 150 disposed inside the refrigerator 1 to obtain various information related to the storage chamber.

[0221] For example, the internal sensor 150 may include at least one of a weight sensor that senses the weight of an object stored on a shelf set in at least one of the multiple storage chambers 30, 31, 32 and 33, a distance sensor that senses an empty space of at least one of the multiple storage chambers 30, 31, 32 and 33, or an internal camera that obtains an internal image of the refrigerator 1.

[0222] The controller 200 can determine whether the storage room corresponding to the object identified from the image of the surrounding environment can accommodate the identified object based on the output of the internal sensor 150. In addition, the controller 200 can also use the image of the surrounding environment taken by the camera 110 to determine whether the storage room corresponding to the identified object can accommodate the identified object.

[0223] More specifically, the controller 200 may calculate the size of the recognized object based on the image of the surrounding environment, and calculate the available space of the storage room corresponding to the recognized object based on the output of the internal sensor 150. The controller 200 may compare the size of the recognized object with the available space of the storage room corresponding to the recognized object to determine whether the storage room can accommodate the recognized object.

[0224] In the case where the controller 200 determines that the storage room corresponding to the identified object can accommodate the identified object, as described above, the controller 200 can automatically open the door of the storage room corresponding to the identified object, and in the case where the controller 200 determines that the storage room corresponding to the identified object cannot accommodate the identified object, the controller 200 can provide the user with information related to the determination result.

[0225] In the case where the refrigerator 1 includes the speaker 510, a guide message related to the determined result may be output through the speaker 510. The guide message related to the determined result may include information indicating that the storage room corresponding to the identified object does not have enough free space, and in the case where there is another refrigerator in the user's home, it is suggested that the user store the identified object in the other refrigerator.

[0226] The refrigerator 1 also includes a display 520 (see Fig.17 ), a guidance message related to the determined result can be output through the display 520, which will be described later.

[0227] Meanwhile, the output of the interior sensor 150 may be used to automatically close the door 40 or deactivate the camera 110. After the door 40 is automatically opened as described above, the user may manually close the door 40 again. However, the controller 200 may automatically close the door 40, at which time the controller 200 may use the output of the interior sensor 150. More specifically, after the door 40 of the storage room corresponding to the identified object is opened, the controller 200 may determine whether an object has been added in the storage room corresponding to the identified object based on the output of the interior sensor 150, and according to the result of the determination that the object has been added, the controller 200 may send a control signal for automatically closing the door 40 to the door driver 300.

[0228] Furthermore, according to a result of the determination that the object has been added, the controller 200 may deactivate the camera 110 .

[0229] However, embodiments of the refrigerator 1 are not limited thereto, and the controller 200 may automatically close the door 40 or turn off the camera 110 when a preset time has passed after the door 40 is opened, regardless of the output of the interior sensor 150 .

[0230] At the same time, the controller 200 can recognize users and objects from the image of the surrounding environment. In order to recognize the user, the user's characteristics, such as the user's physical characteristics (e.g., height, body shape, hair length, face size, etc.) or other external characteristics of the user (e.g., whether the user wears glasses, whether the user uses a cane or a wheelchair, etc.) may have been pre-stored in the controller 200. In the case where multiple users (e.g., multiple family members) use the refrigerator 1, the characteristics of each user may be stored separately.

[0231] The controller 200 may compare the features detected from the image of the surrounding environment with the pre-stored features to identify the user appearing in the image of the surrounding environment, and control the door 40 to open based on the identified user. More specifically, the controller 200 may match each user with the storage room preferred by the corresponding user, and pre-store the matching results. For example, user 1 may prefer to store a specific object in the cold storage room 30, and user 2 may prefer to store the same object in the first temperature conversion chamber 32. The matching results may have been pre-stored in the controller 200, and the controller 200 may open the door of the storage room that matches the identified user.

[0232] In addition, the controller 200 may match each user with the storage temperature preferred by the corresponding user in advance and store the matching result. For example, in the case where the storage chamber corresponding to the object recognized from the image of the surrounding environment is the first temperature conversion chamber 32 or the second temperature conversion chamber 36, the controller 200 may control the temperature of the first temperature conversion chamber 32 or the second temperature conversion chamber 36 according to the storage temperature matched with the user recognized from the image of the surrounding environment.

[0233] Information about storage rooms or storage temperatures preferred by respective users may be stored when the refrigerator 1 is manufactured, may be acquired from the server 800 , or may be updated according to a usage history of the refrigerator 1 .

[0234] In addition, the opening angle or opening speed of the door 40 can be controlled according to the identified user. For example, the controller 200 can control the door 40 to open more slowly for the identified elderly user, and to open to a larger opening angle for the identified larger user.

[0235] The controller 200 may identify the user by using the user's voice input to the microphone 140. The controller 200 may pre-register each user's voice, extract and store features of the user's voice, and then compare the features extracted from the voice input to the microphone 140 with the stored features to identify the user. The operations performed by the controller 200 after the controller 20 identifies the user have been described above.

[0236] The above operations related to user identification may be performed by the server 800 .

[0237] Fig.14 is a control block diagram of a refrigerator further including a drawer drive according to an embodiment of the present disclosure.

[0238] As described above, the refrigerator 1 according to an embodiment of the present disclosure may include at least one drawer 39 arranged in at least one of a plurality of storage chambers 30, 31, 32 and 33, and according to the example of the refrigerator 1, the at least one drawer 39 may be automatically inserted into or taken out of the at least one storage chamber.

[0239] For this reason, Fig.14 As shown, the refrigerator 1 according to the embodiment of the present disclosure may further include a drawer driver 600 that generates power for automatically taking out or inserting the drawer 39. The drawer driver 600 may include a motor 620 for generating power and a driver circuit 610 for driving the motor 620.

[0240] In the case where the drawer 39 is provided in the storage room corresponding to the recognized object, the controller 200 may send a control signal to the drawer driver 600 for automatically taking out the drawer 39. The user may manually insert the drawer 39 into the storage room after storing the object, or the controller 200 may automatically insert the drawer 39 into the storage room when a preset time has passed after automatically taking out the drawer 39. In addition, in the case where the internal sensor 150 is provided, the controller 200 may determine whether the object has been accommodated in the drawer 39 based on the output of the internal sensor 150, and according to the result of the determination that the object has been accommodated in the drawer 39, the controller 200 may automatically insert the drawer 39 into the storage room.

[0241] By automatically taking out the drawer 39 provided in the storage chamber, the convenience of the user can be further improved.

[0242] Fig.15 is a control block diagram of a refrigerator further including a microphone, and Fig.16 An example of recognizing an object from a voice input by a user through a microphone according to an embodiment of the present disclosure is shown.

[0243] Reference Fig.15 The refrigerator 1 according to the embodiment of the present disclosure may further include a microphone 140. The microphone 140 may be attached to the outer surface of the refrigerator 1 to receive a user's voice.

[0244] As described above, there may be a case where the controller 200 fails to recognize any preset object although the controller 200 activates the camera 110 based on the output of the proximity sensor 120 and applies the object recognition algorithm to the image of the surrounding environment captured by the camera 110. In this case, the controller 200 may trigger voice recognition.

[0245] When the microphone 140 is turned on, the controller 200 may trigger the voice recognition, and when the microphone 140 is turned off, the controller 200 may turn on the microphone 140 and then trigger the voice recognition. That is, when the output of the proximity sensor 120 indicates the approach of the user and no preset object is recognized from the image of the surrounding environment, the voice recognition may be triggered even though the wake-up word or trigger word for the voice recognition is not spoken.

[0246] The controller 200 may transmit a control signal for opening the door 40 to the door driver 300 based on the recognized object name in response to the object name recognized from the user's voice input to the microphone 140 .

[0247] More specifically, the user may speak the name of an object he / she wants to take out of the refrigerator 1 to input the name of the object to the microphone 140, and the controller 200 may determine a storage room in which the corresponding object is stored or estimated to be stored. The storage room in which the corresponding object is stored may be determined based on the output of the internal sensor 150, and the storage room in which the corresponding object is estimated to be stored may be determined based on information stored in a memory of the controller 200 or the server 800.

[0248] Reference Fig.16 , in the case where the preset object is not recognized from the image of the surrounding environment although the user approaches the refrigerator 1 so that the camera 110 is activated and captures an image of the surrounding environment, the controller 200 may trigger voice recognition. Then, the user may utter "beer" with the user's voice to input "beer" to the microphone. In this case, voice recognition may be performed on the user's voice. The voice recognition may be performed directly by the controller 200, or an external server may perform the voice recognition and send the result of the voice recognition to the refrigerator 1.

[0249] For example, the controller 200 may determine a storage room in which beer (beer is an object having a name spoken by the user) is stored or estimated to be stored as the refrigerating room 30 and transmit a control signal for opening the first upper door 41 to the door driver 300 .

[0250] As described above, information about the storage room corresponding to the object may be pre-stored in the controller 200, the information stored in the controller 200 may be updated according to the use history of the refrigerator 1, or the information may be obtained from the server 80 that manages a plurality of refrigerators.

[0251] Furthermore, in the case where the refrigerator 1 includes the internal sensor 150 , a storage room storing an object having a name spoken by the user may be determined based on an output of the internal sensor 150 .

[0252] In addition, there may be a case where the controller 200 fails to recognize what the object carried by the user is. In the case where the controller 200 recognizes the object carried by the user from the image of the surrounding environment, but fails to recognize what the corresponding object is, the controller 200 may determine the name of the object spoken by the user as the name of the corresponding object. Therefore, the controller 200 may determine the storage room corresponding to the object with the name spoken by the user, that is, the storage room suitable for storing the object with the name spoken by the user, and send a control signal to the door driver 300 for opening the door 40 of the determined storage room.

[0253] The controller 200 can match the features (such as shape, size, color, etc.) of the object identified from the image of the surrounding environment with the name of the object and store the matching results. In the case where the controller 200 is capable of performing deep learning, the features of the object identified from the image of the surrounding environment and the name of the object spoken by the user can be used as learning data. Thereafter, in the case where the image of the surrounding environment includes objects with the same features, the controller 200 can identify what the corresponding object is. Alternatively, in the case where the server 800 performs object recognition, the server 800 can perform operations.

[0254] Meanwhile, there may be a case where it is unclear whether the object carried by the user is an object to be stored in the refrigerator 1 or an object to be taken out of the refrigerator 1. That is, in the case where the object recognized by the controller 200 from the image of the surrounding environment is an object storable in the refrigerator 1 and is related to the object stored in the refrigerator 1, the controller 200 may turn on the microphone 140 or activate voice recognition to obtain additional information from the user.

[0255] The controller 200 may determine whether the object recognized from the image of the surrounding environment is an object to be stored in the refrigerator 1 or an object related to an object to be taken out of the refrigerator 1 based on the name of the object included in the user voice input to the microphone 140 .

[0256] In the case where the name of the object included in the user voice is the same as the object recognized from the image of the surrounding environment, the controller 200 can determine that the recognized object is an object to be stored in the refrigerator 1 and open the door 40 of the storage room suitable for storing the object.

[0257] In a case where the name of an object included in the user voice is different from the object recognized from the image of the surrounding environment, the controller 200 can determine that the recognized object is an object related to the object to be taken out from the refrigerator 1, and open the door 40 of the storage room storing the object with the name included in the user voice.

[0258] For example, in the case where the user carries an empty beer glass, the user's intention may be to keep the beer glass cool or to take out beer from the refrigerator 1. Therefore, in the case where an empty beer glass is recognized from the image of the surrounding environment, the controller 200 may turn on the microphone 140 or activate voice recognition to receive the user's voice. In the case where the name of the object included in the user's voice is "beer glass", the controller 200 may open the door 40 to enable the user to store the beer glass, and in the case where the name of the object included in the user's voice is "beer", the controller 200 may open the door 40 to enable the user to take out the beer.

[0259] Fig.17 is a control block diagram of a refrigerator further including a display, and Fig.18 An example of a screen displayed on a display of a refrigerator according to an embodiment of the present disclosure is shown.

[0260] Reference Fig.17 and Fig.18 The refrigerator 1 may further include a display 520 disposed on the door 40 .

[0261] The display 520 may be one of various types of displays including a liquid crystal display (LCD) panel, a light emitting diode (LED) panel, an organic light emitting diode (OLED) panel, and a quantum dot LED (QLED) panel.

[0262] like Fig.18 As shown, the display 520 may be located on one of the plurality of doors 40 , or a plurality of displays 520 may be located on each of the doors 40 , respectively.

[0263] In a case where the object enters at least one of the plurality of storage rooms or is released from at least one of the plurality of storage rooms, the controller 200 may control the display 520 to display information related to the entry or the release.

[0264] Reference Fig.18 In the case where food enters the temperature conversion chamber, the display 520 may display information indicating that the food has entered the temperature conversion chamber. The display 520 may display information related to the entry and release of food as text, such as Fig.18 as shown, or as an image.

[0265] Alternatively, as above with reference to Fig.13As described, in the case where it is determined that the storage room corresponding to the object recognized from the image of the surrounding environment cannot accommodate the recognized object due to insufficient available space in the storage room, the display 520 may display a guidance message related to the determined result. For example, the display 520 may display information indicating that the storage room corresponding to the recognized object does not have sufficient available space, or in the case where there is another refrigerator in the user's home, the display 520 may display information for suggesting that the user store the recognized object in another refrigerator.

[0266] This information can be displayed on the user's mobile device. In this case, the refrigerator 1 or the server 800 managing the refrigerator 1 can send the information to the user's mobile device through a communicator (e.g., a transceiver), and the user's mobile device can display the information.

[0267] Fig.19 is a control block diagram of a refrigerator which also includes a communicator, and Figure 20 to Figure 22 An example of an external device communicating with a refrigerator according to an embodiment of the present disclosure is shown.

[0268] Reference Fig.19 , the refrigerator 1 according to the embodiment of the present disclosure may further include a communicator 700 (eg, a transceiver) for communicating with an external device.

[0269] The communicator 700 may include a short-range communication module using a method such as Wireless-Fidelity (Wi-Fi), Zigbee, or Near Field Communication (NFC), a wireless communication module using a method such as the third generation (3G) rd generation, 3G), fourth generation (4 th generation, 4G) or fifth generation (5 th generation, 5G) method, or at least one of a wireless communication module using a wired communication module using a communication cable.

[0270] The communicator 700 may communicate with at least one of a mobile device, another device in a home, or a server to transmit / receive data.

[0271] The mobile device may include a portable and communicative electronic device such as a user's mobile phone and a tablet PC. The communicator 700 may communicate directly with the mobile device to send / receive data to / from the mobile device, or the communicator 700 may send / receive data to / from the mobile device through the server 800 that manages the refrigerator 1. Fig.17 and Fig.18The information displayed on the display 520 of the refrigerator 1 can be displayed on the mobile device. In order to display information on the pre-registered mobile device, the controller 200 can send corresponding information to the mobile device through the communicator 700.

[0272] Reference Fig. 20 , the refrigerator 1 can transmit / receive data to / from an artificial intelligence (AI) speaker A, which is one of the other devices located at home and including a camera, through the communicator 700. In the current example, the AI ​​speaker A is described as an example, however, other devices such as a television (TV), a computer, a robot vacuum cleaner, etc. may also be included in the other devices located at home and including a camera.

[0273] The AI ​​speaker A may include a speaker A10, a microphone A20, a camera A40, and a display A30.

[0274] The AI ​​speaker A may transmit an image captured by the camera A40 to the refrigerator 1 , identify an object from the image captured by the camera A40 , and transmit information about the identified object to the refrigerator 1 .

[0275] Alternatively, the AI ​​speaker A may transmit images captured by the camera A40 to the server 800 .

[0276] The AI ​​speaker A can send an image captured by the camera A40 or information about the recognized object to the refrigerator 1 or the server 800 in real time. In addition, the AI ​​speaker A can send an image or information to the refrigerator 1 or the server 800 when a preset object is recognized from the image or when a preset condition is met.

[0277] For example, when the status information is determined based on the image captured by the camera A40 and a preset condition is satisfied according to the status information indicating that an object enters or is released from the refrigerator 1, the AI ​​speaker A may transmit the captured image or information about the recognized object to the refrigerator 1 or the server 800. The AI ​​speaker A may determine the status information by further using a voice input to the microphone A20.

[0278] As a detailed example, in the case where the camera A40 captures an image of a user carrying a kimchi container and moving in a direction toward the refrigerator 1, it can be determined that the situation information determined by the corresponding image represents a situation where the user puts an object into the refrigerator 1. In addition, in the case where the voice input to the microphone A20 includes an intention to put kimchi into the refrigerator 1 or a command to put kimchi into the refrigerator 1, it can be determined that the situation information represents a situation where the object is put into the refrigerator 1.

[0279] The controller 200 of the refrigerator 1 may determine a storage chamber corresponding to the information about the recognized object received from the AI ​​speaker A, and transmit a control signal to the door driver 300 to open the door 40 of the corresponding storage chamber.

[0280] At this time, the controller 200 may control the opening time of the door 40 based on the distance between the AI ​​speaker A and the refrigerator 1. For example, the controller 200 may estimate the arrival time of the user at the refrigerator 1 based on the distance between the AI ​​speaker A and the refrigerator 1 and the moving speed of the user, and open the door 40 at the estimated arrival time. The moving speed of the user may be determined based on the walking speed of a normal person, or may be the average walking speed of the user determined based on an image captured by the AI ​​speaker A, the refrigerator 1, or another device having a camera.

[0281] As described above, in the case where the refrigerator 1 receives an image from the AI ​​speaker A, the controller 200 may recognize an object from the received image.

[0282] Meanwhile, in the case where the AI ​​speaker A is managed by the server 800 that manages the refrigerator 1 , images captured by the AI ​​speaker A may be transmitted to the server 800 , and the server 800 may perform operations such as object recognition, determination of a storage room, and the like.

[0283] Reference Fig.21 In the case where other refrigerators 2 and 3 are also provided in the home, the refrigerator 1 can communicate with the other refrigerators 2 and 3 through the communicator 700. The other refrigerators 2 and 3 may be refrigerators that perform the same functions as the refrigerator 1, or dedicated refrigerators for specific functions, such as kimchi refrigerators, wine refrigerators, and freezers.

[0284] The refrigerator 1 and the other refrigerators 2 and 3 can communicate with each other directly or through the server 800 that manages the refrigerator 1 and the other refrigerators 2 and 3 together. In the case where the refrigerator 1 and the other refrigerators 2 and 3 communicate directly with each other, the refrigerator 1 can be used as a hub device, and the other refrigerators 2 and 3 can send data to each other / receive data from each other through the refrigerator 1.

[0285] As described above, the controller 200 can determine whether the storage room corresponding to the identified object can accommodate the corresponding object, and based on the result of the determination that the storage room cannot accommodate the corresponding object, the controller 200 can output information through the speaker 510 or the display 520 for guiding the user to other refrigerators 2 and 3 that can accommodate the corresponding object.

[0286] The controller 200 may determine a storage room corresponding to the object among a plurality of storage rooms provided in the other refrigerators 2 and 3, and transmit a signal for opening the door of the determined storage room to the other refrigerators 2 and 3 through the communicator 700. At this time, the controller 200 may control the opening time of the doors of the other refrigerators 2 and 3 based on the distance between the refrigerator 1 and the other refrigerators 2 and 3. The operation of controlling the opening time of the doors of the other refrigerators 2 and 3 may be performed by the refrigerator 1 or the other refrigerators 2 and 3 whose doors are opened.

[0287] Alternatively, the controller 200 may transmit information about the recognized object to the other refrigerators 2 and 3 through the communicator 700, and each of the other refrigerators 2 and 3 may determine whether the recognized object can be received.

[0288] Meanwhile, in the case where the object recognized by the controller 200 of the refrigerator 1 is an object to be released from the refrigerator 1 rather than an object to be entered into the refrigerator 1, the controller 200 may determine whether the corresponding object exists in the refrigerator 1 based on the output of the internal sensor 150, receive the outputs of the internal sensors of the other refrigerators 2 and 3 according to the result of the determination that the corresponding object does not exist in the refrigerator 1, and determine whether the corresponding object exists in the other refrigerators 2 and 3. Alternatively, the controller 200 may transmit a signal requesting confirmation as to whether the corresponding object exists in the other refrigerators 2 and 3 to the other refrigerators 2 and 3, and receive the confirmation result from the other refrigerators 2 and 3.

[0289] In the case where the corresponding object exists in the other refrigerators 2 and 3, the controller 200 can display a guide message notifying that the corresponding object exists in the other refrigerators 2 and 3 on the display 520, and open the doors of the other refrigerators 2 and 3 in which the corresponding object exists.

[0290] Reference Fig. 22 , the refrigerator 1 can communicate with the server 800 through the communicator 700.

[0291] The server 800 may be a server that manages a plurality of refrigerators 1, 2, and 3 used by a plurality of users. In addition, the server 800 may manage other devices such as a display and an AI speaker together with the refrigerators 1, 2, and 3.

[0292] The server 800 may include a communicator 810 (eg, a transceiver) for communicating with the refrigerator 1 and a controller 820 (eg, at least one processor) for processing data received from the refrigerator 1 .

[0293] The refrigerator 1 may transmit an image of the surrounding environment captured by the camera 110 to the server 800. The communicator 810 may receive the image of the surrounding environment from the refrigerator 1, and the controller 820 may recognize an object from the received image of the surrounding environment and determine whether to open the door 40 provided in the refrigerator 1 based on the recognized object. According to the result of the determination that the door 40 needs to be opened, the controller 820 may control the communicator 810 to transmit a signal for opening the door 40 to the refrigerator 1.

[0294] In the case where the refrigerator 1 includes a plurality of storage chambers and a plurality of doors, the controller 820 may determine a storage chamber corresponding to the identified object from among the plurality of storage chambers and control the communicator 810 to transmit a signal for opening the door of the storage chamber corresponding to the identified object.

[0295] The operation of identifying an object from an image of the surrounding environment and determining whether to open the door 40 based on the identified object may be the same as the above-mentioned operation performed by the controller 200 of the refrigerator 1. However, since the server 800 manages a plurality of refrigerators used by a plurality of users, the server 800 may collect and manage big data on the usage patterns of the refrigerators. Therefore, the controller 200 may learn the data obtained from a plurality of users to generate a deep learning model, and determine the storage room corresponding to the identified object based on the deep learning model.

[0296] In addition, the controller 820 may determine a storage temperature corresponding to the recognized object based on a deep learning model. The information on the storage chamber and the storage temperature determined by the controller 820 may be transmitted to the refrigerator 1 through the communicator 810.

[0297] The communicator 810 may receive temperature information sensed by the temperature sensor 130 of the refrigerator 1. The temperature information sensed by the temperature sensor 130 may be temperature information of the identified object. The controller 820 may determine a storage temperature corresponding to the identified object by using the temperature information sensed by the temperature sensor 130.

[0298] In the event that the storage room corresponding to the identified object does not have sufficient available space and thus cannot accommodate the identified object, the controller 1 may control the communicator 810 to transmit a signal for opening the door to other refrigerators 2 and 3 registered by the user of the refrigerator 1 .

[0299] The controller 820 may determine the available space of the storage chamber by receiving the output of the internal sensor 150 from the refrigerator 1, or the controller 820 may request the refrigerator 1 to determine whether the storage chamber has the available space and receive the result of the determination.

[0300] In addition, the communicator 810 may receive an image captured by a camera provided in another device from another apparatus (such as an AI speaker and a TV), and the controller 820 may recognize an object from the received image and determine a storage room corresponding to the recognized object. The communicator 810 may send a signal to the refrigerator 1 for opening the door of the storage room corresponding to the recognized object.

[0301] In addition, as mentioned above Fig. 20 The described operations of the controller 200 , such as determination of status information and object recognition, may be performed by the controller 820 of the server 800 .

[0302] The communicator 810 may receive a user's voice input to the microphone 140 of the refrigerator 1, and the controller 820 may perform voice recognition on the user's voice to recognize the name of the object and send a signal for opening the door to the refrigerator 1 based on the recognized object name.

[0303] In addition, all or part of the various operations performed by the controller 200 of the refrigerator 1 may be performed by the controller 820 of the server 800 .

[0304] Hereinafter, a method for controlling a refrigerator according to an embodiment of the present disclosure will be described. The method for controlling a refrigerator according to an embodiment of the present disclosure can be applied to the above-mentioned refrigerator 1, and in some cases, a part of the method can be executed by the server 800. Figures 1 to 22 The described contents may be applied to the method of controlling a refrigerator according to an embodiment of the present disclosure unless otherwise specified.

[0305] Fig.23 is a flowchart illustrating a method of controlling a refrigerator according to an embodiment of the present disclosure.

[0306] Reference Fig.23 , the method of controlling a refrigerator according to an embodiment of the present disclosure may capture an image of the surrounding environment using the camera 110 in operation 1010, and send a control signal for opening at least one door 40 to the door driver 300 based on an object recognized from the image of the surrounding environment in operation 1020. In operation 1030, in response to the control signal, the door driver 300 may generate power and transmit the power to the at least one door 40, thereby opening the at least one door 40.

[0307] When the door 40 is opened, a notification sound may be output from the speaker 510 provided in the refrigerator 1, and the controller 200 may determine the relative position of the user relative to the door 40 expected to be opened, and control the speaker 510 to output the notification sound differently according to the relative position of the user. Fig.10 and Fig.11 Details regarding output of notification sound are described.

[0308] In addition, the controller 200 can control the opening angle of the door 40 based on the relative position of the user relative to the door 40 to be opened. Fig.12 Details regarding the control of the opening angle of the door 40 are described.

[0309] The operation of recognizing an object from the image of the surrounding environment captured by the camera 110 may be performed by the controller 200 of the refrigerator 1 or by the controller 820 of the server 800 .

[0310] The object recognized from the image of the surrounding environment may be a preset object that can be stored in the refrigerator 1. The objects that can be stored in the refrigerator 1 may be preset, and then the preset object may be recognized from the image of the surrounding environment, or the object may be recognized from the image of the surrounding environment regardless of the kind of the object, and then it may be determined whether the recognized object is the preset object.

[0311] According to the recognition of the preset object, the controller 200 may determine a storage room corresponding to the recognized object from among the plurality of storage rooms 30, 31, 32, and 33, and transmit a control signal to the door driver 300 for opening the door of the storage room corresponding to the recognized object.

[0312] Fig.24 is another flow chart illustrating a method of controlling a refrigerator according to an embodiment of the present disclosure.

[0313] Reference Fig.24 In operation 1110, the method for controlling a refrigerator according to an embodiment of the present disclosure may activate the camera 110 in response to the approach of the user sensed by the proximity sensor 120 ("Yes" in operation 1100). In response to the output of the proximity sensor 120 indicating the approach of the user, the controller 200 may activate the camera 110. Depending on the specifications of the proximity sensor 120, the output of the proximity sensor 120 itself may indicate the approach or absence of the user, or the controller 200 may determine the approach of the user based on the output of the proximity sensor 120.

[0314] The activation and deactivation of the camera 110 may be implemented in software or hardware, and details on the activation and deactivation of the camera 110 have been described above.

[0315] In response to the activation of the camera 110, the controller 200 may capture an image of the surrounding environment using the camera 110 in operation 1120, and transmit a control signal for opening at least one door 40 to the door driver 300 based on an object recognized from the image of the surrounding environment in operation 1130. In operation 1140, in response to the control signal, the door driver 300 may generate power and transmit the power to the at least one door 40, thereby opening the at least one door 40.

[0316] Fig.25 is another flow chart illustrating a method of controlling a refrigerator according to an embodiment of the present disclosure.

[0317] Reference Fig.25 , the method for controlling a refrigerator according to an embodiment of the present disclosure may capture an image of the surrounding environment using the camera 110 in operation 1200, and sense the temperature of an object in operation 1210. The temperature of the object may be sensed by the temperature sensor 130, which senses the temperature of the object located around at least one door. For example, the temperature sensor 130 may be a thermal imaging camera.

[0318] In operation 1220, the controller 200 may determine a storage chamber and a storage temperature corresponding to the identified object based on the object identified from the image of the surrounding environment and the temperature of the identified object. In the case where the temperature sensor 130 is a thermal imaging camera, the controller 200 may determine the temperature of the identified object based on the image taken by the thermal imaging camera.

[0319] In operation 1230, the controller 200 may open the door of the storage room corresponding to the recognized object, and in operation 1240, control the temperature of the storage room corresponding to the recognized object to a preset storage temperature. In order to open the door of the storage room, the controller 200 may send a control signal for transmitting power to the corresponding door to the door driver 300, and in order to control the temperature of the storage room to the preset storage temperature, the controller 200 may send a control signal for supplying cold air to the corresponding storage room to the cooler 400.

[0320] Fig.26 is another flow chart illustrating a control method according to an embodiment of the present disclosure.

[0321] Reference Fig.26 According to an embodiment of the present disclosure, the control method may capture an image of the surrounding environment with the camera 110 in operation 1300, determine a storage room corresponding to an object identified from the image of the surrounding environment in operation 1310, determine an available space of the storage room in operation 1320, and determine whether the storage room can accommodate the identified object based on the available space in operation 1330.

[0322] In order to determine the available space of the storage room, the controller 200 may use the output of the internal sensor 150 provided in the refrigerator 1. For example, the internal sensor 150 may include at least one of a weight sensor that senses the weight of an object stored on a shelf provided in at least one of the plurality of storage rooms 30, 31, 32, 33, 35, and 36, a distance sensor that senses an empty space of at least one of the plurality of storage rooms 30, 31, 32, 33, 35, and 36, or an internal camera that obtains an internal image of the refrigerator 1.

[0323] More specifically, the controller 200 may calculate the size of an object recognized from the image of the surrounding environment, and calculate the available space of the storage room corresponding to the recognized object based on the output of the internal sensor 150. The controller 200 may compare the size of the recognized object with the available space of the storage room corresponding to the recognized object to determine whether the storage room can accommodate the recognized object.

[0324] Based on the result of determining that the storage room can accommodate the identified object ("Yes" in operation 1330), the controller 200 can open the door of the storage room in operation 1340, and based on the result of determining that the storage room cannot accommodate the identified object ("No" in operation 1330), the controller 200 can output a guidance message in operation 1350.

[0325] The guide message may be outputted audibly through the speaker 510 or visually through the display 520. The guide message may include information indicating that the storage room corresponding to the identified object does not have sufficient available space, and if there is another refrigerator at the user's home, it is suggested that the user store the identified object in another refrigerator.

[0326] Fig. 27 is another flow chart illustrating a method of controlling a refrigerator according to an embodiment of the present disclosure.

[0327] Reference Fig. 27 , the method of controlling a refrigerator according to an embodiment of the present disclosure may, in response to sensing the approach of a user ("Yes" in operation 1400), activate the camera 110 in operation 1410, capture an image of the surrounding environment with the camera 110 according to the activation of the camera 110 in operation 1420, and in the case where a preset object is not recognized from the image of the surrounding environment ("No" in operation 1430), trigger voice recognition in operation 1440. In the case where the preset object is not recognized in operation 1430, the door is opened based on the object recognized from the image of the surrounding environment in operation 1460.

[0328] When the microphone 140 provided in the refrigerator 1 is in the turned-on state, the voice recognition may be triggered, and when the microphone 140 is in the turned-off state, the microphone 140 may be turned on and then the voice recognition may be triggered. That is, when the output of the proximity sensor 120 indicates the approach of the user and the preset object is not recognized from the image of the surrounding environment, the voice recognition may be triggered even though the wake-up word or trigger word for the voice recognition is not spoken.

[0329] In operation 1450, the controller 200 may open a door based on the name of an object recognized from the user's voice input to the microphone 140. More specifically, the controller 200 may open the door of a storage room that stores or is expected to store an object having a name recognized from the user's voice. The storage room in which the object is stored may be determined based on the output of the internal sensor 150. In addition, a storage room suitable for storing the corresponding object or mainly storing the corresponding object may be estimated as a storage room in which the object is stored. Information about a storage room suitable for storing the corresponding object or mainly storing the corresponding object may be pre-stored in the controller 200, or may be obtained from the server 800.

[0330] The name of the object recognized from the user's voice may be the name of the object that the user wants to take out from the refrigerator 1, or the name of the object that the user wants to store in the refrigerator 1. For example, in the case where the controller 200 recognizes that the user carries an object from the image of the surrounding environment, but fails to recognize what the corresponding object is, the controller 200 may determine the name of the object spoken by the user as the name of the corresponding object. Therefore, the controller 200 may determine the storage room corresponding to the object having the name spoken by the user, that is, the storage room suitable for storing the object having the name spoken by the user, and send a control signal to the door driver 300 for opening the door 40 of the determined storage room.

[0331] The controller 200 can match the shape, size, color, etc. of the object recognized from the image of the surrounding environment with the name of the object spoken by the user, and store the matching result. In the case where the controller 200 is capable of performing deep learning, the features of the object recognized from the image of the surrounding environment and the name of the object spoken by the user can be used as learning data. Thereafter, in the case where the image of the surrounding environment includes an object with the same features, the controller 200 can identify what the corresponding object is.

[0332] In a refrigerator, a server, and a refrigerator control method according to one aspect of the present disclosure, an object to be stored may be recognized, and a door may be automatically opened based on the recognized object.

[0333] While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. A refrigerator, comprising: main body; a plurality of doors disposed on the front surface of the body; a plurality of storage chambers disposed in the body or the plurality of doors and configured to store objects; a cooler configured to cool the plurality of storage chambers; a door driver configured to generate power for automatically opening or closing each door of the plurality of doors; a camera configured to capture images of surroundings near the plurality of doors; as well as at least one processor configured to, when an object is recognized from the image of the surrounding environment captured by the camera, send a control signal for opening one of the plurality of doors to the door driver based on the recognized object, Wherein, the at least one processor is further configured to: determining a door of the plurality of doors that corresponds to the identified object, and A control signal for opening a door corresponding to the recognized object is sent to the door driver.

2. The refrigerator according to claim 1, wherein: The at least one processor is further configured to: determining a storage room corresponding to the identified object from among the plurality of storage rooms, and A control signal for opening a door of a storage room corresponding to the recognized object among the plurality of doors is transmitted to the door driver.

3. The refrigerator according to claim 2, further comprising: a temperature sensor configured to sense a temperature of an object located near the plurality of doors, The at least one processor is further configured to determine a storage chamber and a storage temperature among the plurality of storage chambers corresponding to the identified object based on the temperature sensed by the temperature sensor and the object identified from the image of the surrounding environment.

4. The refrigerator according to claim 3, wherein: The at least one processor is further configured to send a control signal to the cooler to control the temperature of the storage chamber corresponding to the identified object to the storage temperature.

5. The refrigerator according to claim 1, further comprising: A proximity sensor configured to sense the approach of a user, The at least one processor is further configured to activate the camera in response to an output of a proximity sensor indicating that a user is approaching.

6. The refrigerator according to claim 1, further comprising: speaker, Wherein, the at least one processor is further configured to: controlling the speaker to output a notification sound when the plurality of doors are automatically opened, determining a relative position of the user relative to a door of the plurality of doors that is intended to be opened, and The speaker is controlled to output a notification sound differently according to a relative position of the user.

7. The refrigerator according to claim 1, wherein: The at least one processor is further configured to: determining a relative position of the user relative to a door of the plurality of doors that is intended to be opened, and At least one of an opening angle or an opening speed of the door is controlled based on the relative position of the user.

8. The refrigerator according to claim 1, further comprising: microphone, The at least one processor is further configured to, in response to a name of an object recognized from a user's voice input to the microphone, send a control signal for opening a door among the plurality of doors to the door driver based on the recognized name of the object.

9. The refrigerator according to claim 2, further comprising: an internal sensor, comprising at least one of a weight sensor, a distance sensor, or an internal camera, the internal sensor being disposed in at least one storage compartment of the plurality of storage compartments; as well as at least one of a speaker or a display, Wherein, the at least one processor is further configured to: determining whether a storage room corresponding to the identified object can accommodate the identified object based on the output of the internal sensor, and According to a result of determination that a storage room corresponding to the identified object cannot accommodate the identified object, at least one of the speaker or the display is controlled to output a guide message related to the result of the determination.

10. The refrigerator according to claim 1, further comprising: a display disposed in one of the plurality of doors, The at least one processor is further configured to control the display to display information related to the entry or release of the object when the object enters at least one of the plurality of storage rooms or is released from at least one of the plurality of storage rooms.

11. The refrigerator according to claim 2, further comprising: a drawer configured to slide to take out or put into at least one of the plurality of storage compartments; as well as A drawer drive is configured as at least one of the following: generating a power for automatically placing the drawer into the at least one storage chamber, or automatically taking out the drawer from at least one storage compartment of the plurality of storage compartments, The at least one processor is further configured to send a control signal to the drawer driver for taking the drawer out of the storage chamber when the drawer is disposed in the storage chamber corresponding to the identified object.

12. The refrigerator according to claim 1, further comprising: A transceiver configured to communicate with an external device, Wherein, the transceiver is further configured as: sending the image of the surrounding environment captured by the camera to a server, and In a case where the server recognizes the object, information about the object or information about a door corresponding to the object is received from the server.

13. The refrigerator according to claim 9, further comprising: a transceiver configured to communicate with another refrigerator, Wherein, the at least one processor is further configured to control the transceiver to send information about the identified object or a signal for opening the door to the other refrigerator based on a determination result that the storage room corresponding to the identified object cannot accommodate the identified object.

14. The refrigerator according to claim 12, in, The transceiver is further configured to receive, from a device having a camera, an image captured by the camera or information about an object recognized from the image captured by the camera, and Wherein, the at least one processor is further configured to determine the opening time of the plurality of doors based on the distance between the device and the refrigerator.

Images