Refrigerator and control method therefor

Abstract

Provided is an example of a method for controlling a refrigerator comprising one or more doors, a door control module, a proximity sensor, and a camera sensor. The method comprises the operations of: activating the camera sensor on the basis of identifying that a user is located within a first distance from the refrigerator using the proximity sensor; obtaining an input image through the camera sensor on the basis of identifying that the user is located within a second distance shorter than the first distance from the refrigerator using the proximity sensor; identifying a hand image corresponding to a hand of the user from the input image; performing preprocessing on the hand image; determining a hand gesture of the user from the hand image; and opening one door, selected on the basis of the gesture, among the one or more doors in response to determining that the hand gesture is a gesture for selecting a door to be opened.

Description

Refrigerator and method of controlling the same

[0001] Various embodiments of the present disclosure relate to a refrigerator, and more particularly, to a refrigerator that automatically opens at least one door and a method for controlling the refrigerator.

[0002] A refrigerator is a device that uses a refrigeration cycle to keep objects such as food fresh, and may include a freezer that stores objects at sub-zero temperatures and a refrigerator that stores objects at above-zero temperatures.

[0003] Recently, with the spread of smart homes and the advancement of IoT (Internet of Things) technology, refrigerators can perform various innovative functions, such as voice control, remote operation via smartphone apps, and food management using internal cameras.

[0004] Meanwhile, a refrigerator may include at least one door, depending on the number or structure of storage compartments, such as a refrigerator and / or freezer. The refrigerator door can be used to load or unload items into or out of the storage compartment. When closed, the refrigerator door prevents cold air from leaking out of the storage compartment, thereby maintaining a constant temperature within the storage compartment.

[0005] Opening the refrigerator door is typically performed by a separate user action, typically based on physical contact. However, as refrigerators become more intelligent, refrigerators capable of automatically opening the door based on the user's intention, without requiring physical contact, are being proposed.

[0006] Various embodiments of the present disclosure can provide a refrigerator and a control method thereof that automatically opens a door in accordance with a user's intention by simultaneously detecting a user's hand using an infrared (IR) camera and a red-green-blue (RGB) camera, and clearly identifying structural information of the user's hand through preprocessing of the hand image.

[0007] According to embodiments of the present disclosure, a refrigerator may include at least one storage compartment, at least one door for opening and closing the at least one storage compartment, a door control module for controlling opening or closing of the at least one door, a sensor unit including a proximity sensor and a camera sensor, at least one processor, and a memory including one or more storage media for storing instructions, wherein the instructions, when individually or collectively executed by the at least one processor, may cause the refrigerator to perform operations. The above operations may include an operation of activating the camera sensor based on the identification that the user is located within a first distance from the refrigerator using the proximity sensor, an operation of acquiring an input image through the camera sensor based on the identification that the user is located within a second distance shorter than the first distance from the refrigerator using the proximity sensor, an operation of identifying a hand image corresponding to the user's hand from the input image, an operation of performing preprocessing on the hand image, an operation of determining a hand gesture of the user from the hand image, and an operation of controlling the door control module to open one of the at least one door selected based on the first gesture in response to determining that the hand gesture is a first gesture for selecting a door to be opened.

[0008] According to embodiments of the present disclosure, a method for controlling a refrigerator may include an operation of activating a camera sensor based on identifying that a user is located within a first distance from the refrigerator using a proximity sensor, an operation of acquiring an input image through the camera sensor based on identifying that the user is located within a second distance shorter than the first distance from the refrigerator using the proximity sensor, an operation of identifying a hand image corresponding to a hand of the user from the input image, an operation of performing preprocessing on the hand image, an operation of determining a hand gesture of the user from the hand image, and an operation of opening one of the at least one door selected based on the gesture in response to determining that the hand gesture is a gesture for selecting a door to be opened.

[0009] According to various embodiments of the present disclosure, the refrigerator of the present disclosure can detect a user's hand using an IR camera and an RGB camera simultaneously, thereby recognizing the user's hand with high accuracy even in various environments. Furthermore, the refrigerator of the present disclosure can increase the accuracy of determining the user's hand gesture by clearly identifying the structural information of the user's hand through preprocessing of the hand image. Therefore, the refrigerator of the present disclosure can increase the convenience and efficiency of refrigerator use by accurately and automatically opening the door in accordance with the user's intention.

[0010] The effects that can be obtained from the exemplary embodiments of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly derived and understood by those skilled in the art to which the exemplary embodiments of the present disclosure pertain from the following description. In other words, unintended effects resulting from implementing the exemplary embodiments of the present disclosure can also be derived by those skilled in the art from the exemplary embodiments of the present disclosure.

[0011] FIG. 1 is a front view showing a refrigerator according to one embodiment of the present disclosure.

[0012] FIG. 2 is a perspective view showing the interior of a refrigerator according to one embodiment of the present disclosure.

[0013] FIG. 3 is a block diagram showing the configuration of a refrigerator according to one embodiment of the present disclosure.

[0014] FIG. 4 is an exemplary diagram showing a door control module and sensor unit of a refrigerator according to one embodiment of the present disclosure.

[0015] FIG. 5 is an exemplary diagram showing various arrangements of camera sensors according to one embodiment of the present disclosure.

[0016] FIG. 6 is a flowchart showing the operation of a refrigerator according to one embodiment of the present disclosure.

[0017] FIG. 7 is a diagram illustrating an operation of a proximity sensor according to one embodiment of the present disclosure to sense a user's approach to a refrigerator.

[0018] FIG. 8 is an exemplary diagram showing the configuration of a camera sensor according to one embodiment of the present disclosure.

[0019] FIG. 9 is an exemplary diagram showing an operation of a refrigerator identifying a hand image from an input image according to one embodiment of the present disclosure.

[0020] FIG. 10 is an exemplary diagram showing a first preprocessing of a refrigerator according to one embodiment of the present disclosure.

[0021] FIG. 11 is an exemplary diagram showing a second pretreatment of a refrigerator according to one embodiment of the present disclosure.

[0022] FIG. 12 is an exemplary diagram showing a third preprocessing of a refrigerator according to one embodiment of the present disclosure.

[0023] FIG. 13 is a diagram illustrating a process in which an artificial intelligence model determines a user's hand gesture from a hand image according to one embodiment of the present disclosure.

[0024] FIG. 14 is a drawing showing an operation of a refrigerator according to one embodiment of the present disclosure to determine a door to be opened based on the position of a user's hand.

[0025] FIG. 15 is a diagram illustrating a confirmation notification for a first gesture and a second gesture according to one embodiment of the present disclosure.

[0026] FIG. 16 is a diagram illustrating a confirmation notification for a first gesture and a third gesture according to one embodiment of the present disclosure.

[0027] FIG. 17 is a diagram illustrating a user device and a server connected to a refrigerator according to one embodiment of the present disclosure.

[0028] It should be understood that the various embodiments and terms used in this document are not intended to limit the technical features described in this document to specific embodiments, but rather to include various modifications, equivalents, or substitutes of the embodiments.

[0029] In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0030] The singular form of a noun corresponding to an item may include one or more items, unless the context clearly indicates otherwise.

[0031] In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof.

[0032] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish one component from another and do not qualify the components in any other respect (e.g., importance or order).

[0033] When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0034] The terms "include" or "have" are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in this document, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0035] When a component is said to be “connected,” “coupled,” “supported,” or “in contact with” another component, this includes not only cases where the components are directly connected, coupled, supported, or in contact, but also cases where the components are indirectly connected, coupled, supported, or in contact through a third component.

[0036] When we say that a component is "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where there is another component between the two components.

[0037] The term "and / or" includes any combination of a plurality of related described elements or any one of a plurality of related described elements.

[0038] The operating principle and embodiments of the present invention will be described with reference to the attached drawings below.

[0039] FIG. 1 is a front view showing a refrigerator (1) according to one embodiment of the present disclosure, and FIG. 2 is a perspective view showing the inside of a refrigerator (1) according to one embodiment of the present disclosure.

[0040] A refrigerator (1) may include a main body (10), a storage compartment provided inside the main body (10) so that the front is open, and a door (300) rotatably coupled to the main body (10) so as to open and close the open front of the storage compartment.

[0041] The main body (10) can form the exterior of the refrigerator (1). The main body (10) can include an inner case (11) forming a storage compartment, and an outer case (12) coupled to the outside of the inner case (11) to form the exterior. In addition, the main body (10) can further include a cold air supply device (not shown) that supplies cold air to the storage compartment.

[0042] The cold air supply device may be configured to include a compressor, a condenser, an expansion valve, an evaporator, a blower fan, a cold air duct, etc. An insulating material (not shown) may be filled between the inner case (11) and the outer case (12) of the main body (10) to prevent cold air from leaking from the storage room.

[0043] A machine room (not shown) in which a compressor for compressing refrigerant and a condenser for condensing the compressed refrigerant are installed may be provided at the lower rear side of the main body (10).

[0044] The storage room (20) can be divided into a plurality of storage rooms by horizontal partitions (21) and vertical partitions (22). In the present embodiment, the storage room (20) can include an upper storage room (20a) and a lower storage room (20b). The storage room (20) can be provided with a shelf (23) on which food can be placed and a sealed container (24) for sealingly storing food. The storage room (20) is provided with an open front so that food can be taken in and out, and the open front can be opened and closed by a door (300).

[0045] The upper storage chamber (20a) can be opened and closed by a plurality of doors (300a, 300b). The lower storage chamber (20b) can be opened and closed by a plurality of doors (300c, 300d).

[0046] The refrigerator (1) may further include a handle (100) provided on the door (300). A user can easily open and close the door (300) by gripping the handle (100). The handle (100) may be formed to be long along the vertical direction (Z) of the door (300).

[0047] The refrigerator (1) may further include a dispenser (not shown). The dispenser may be installed in the door (300). For example, the dispenser may be installed in the left upper door (300a). Through the dispenser, a user can directly dispense water or ice to the outside without opening the door (300a). The dispenser may include a cavity formed recessed into the inside of the door (300a) to form a dispensing space. The cavity may be provided with a dispensing port through which water or ice is dispensed, and a dispensing lever for dispensing the water or ice. When the dispensing lever is pressed, water or ice is dispensed from the dispensing port. The dispenser may further include a dispenser status display window that displays the operating status of the dispenser. The dispenser status display window may also have a touch function.

[0048] In one embodiment, the refrigerator (1) may further include a display (200).

[0049] The display (200) may be installed on the door (300) for the convenience of the user. Specifically, the display (200) may be installed on the front (301) of the door (300).

[0050] Hereinafter, a case where the display (200) is installed in the upper right door (300b) is illustrated. However, the location where the display (200) can be installed is sufficient as long as it is the door (300), and is not limited to the upper right door (300b). However, the following description focuses on a case where the display (200) is installed in the upper right door (300b).

[0051] The upper part of the display (200) can be placed at the same position as the upper part of the handle (100) in the vertical direction (Z) of the door (300).

[0052] The lower portion of the display (200) may be positioned at the same position as the lower portion of the dispenser (40) in the vertical direction (Z) of the door (300). One side end of the display (200) adjacent to the handle (100) may be spaced apart from the handle (100) by a certain distance. The other side end facing the one side end of the display (200) adjacent to the handle (100) may be spaced apart from the edge of the door (300) by a certain distance.

[0053] In another aspect, the display (200) may have a rectangular shape (e.g., portrait orientation) with a long side in the vertical direction (Z) of the door (300). The display (200) may include a right long side facing the right side of the door (300), a left long side facing the left side of the door (300), an upper short side facing the upper side of the door (300), and a lower short side facing the lower side of the door (300).

[0054] The right long side may be spaced apart from the right edge of the door (300) by a certain distance in the left direction of the door (300). The left long side may be spaced apart from the handle (100) by a certain distance in the right direction of the door (300). The upper short side may be positioned at the same position as the upper end of the handle (100) in the vertical direction (Z) of the door (300). The lower short side may be positioned at the same position as the lower end of the dispenser (40) in the vertical direction (Z) of the door (300).

[0055] Through the arrangement of the display (200) in this way, it is possible to implement a neat and stable design of the refrigerator (1).

[0056] The display (200) may include a display panel (220) and a touch panel (221). However, according to one or more embodiments, the display (200) may include only the display panel (220). The display (200) may be equipped with a wake-up function that is automatically activated when a user approaches within a certain range. For example, the wake-up function may be implemented through a proximity sensor (e.g., proximity sensor (162) of FIG. 3).

[0057] Specifically, when the proximity sensor (162) detects the approach of a user within a certain range, the display (200) may be activated. That is, the display (200) may be turned on. Conversely, when the proximity sensor (162) does not detect the approach of a user within a certain range, the display (200) may not be activated. That is, the display (200) may be maintained in an off state. When the display (200) is activated, various images or videos may be displayed on the display (200).

[0058] For example, the display (200) may be provided with a function to pause a video and turn off the display (200) when a user opens a door equipped with the display (200). In addition, the display (200) may be provided with a function to resume playing a video and turn on the display (200) when a user closes a door equipped with the display (200). For example, the power off and on functions of the display (200) may be implemented through a door open / close sensor (e.g., the open / close sensor (164) of FIG. 3).

[0059] The display (200) may include a display panel (220). The display (200) may include a liquid crystal display (LCD). The display panel (220) may be positioned on the front of the display. A touch panel (221) may be formed on the display panel (220).

[0060] A user can play or pause a video by touching the touch panel (221). The touch panel (221) may be implemented as a capacitive or pressure-sensitive type. However, the method of forming the touch panel (221) is not limited to the above example.

[0061] The display panel (220) may be provided with at least one input UI component (User Interface Component). The at least one input UI component may include, for example, a camera UI component that executes a camera (e.g., a camera sensor (163) of FIG. 3), a list UI component that lists various lists related to the functions of the refrigerator (1), a home UI component that returns to the start screen, a revert UI component that returns to a previous execution stage, and an information provision UI component that provides information on the overall functions of the refrigerator (1) or the overall functions of the display (200).

[0062] At least one input UI component may be formed on the display panel (220). Preferably, the at least one input UI component may be formed on an external area of ​​the display panel (220) so as not to interfere with an image or video displayed on the display (200).

[0063] The refrigerator (1) may further include a camera sensor (163) capable of capturing images of people or objects. For example, images or videos captured by the camera sensor (163) may be displayed on the display (200).

[0064] The refrigerator (1) may further include at least one microphone for implementing a voice recognition function. A voice command input through the at least one microphone is transmitted to a processor (e.g., processor (191) of FIG. 3), and the processor (191) controls the display (200) to display the voice command result.

[0065] The refrigerator (1) may further include a light sensor (not shown). The light sensor may, for example, adjust the display's lighting brighter in bright places or during the day, and may, for example, adjust the display's lighting darker in dark places or during the night, thereby reducing power loss of the refrigerator (1). The detection result of the light sensor is transmitted to the processor (191), and the processor (191) controls the display panel (220) to adjust the lighting of the display (200).

[0066] The refrigerator (1) may include a door open / close sensor (e.g., a door open / close sensor (164) of FIG. 3). The door open / close sensor (164) may be provided on a hinge (not shown) that connects the door (300) and the main body (10), or may be provided on a portion of the door (300) or the main body (10) where the door (300) and the main body (10) come into contact.

[0067] At least one of a proximity sensor (162), a camera sensor (163), at least one microphone, and a light sensor may be placed on the front of the door (300). For example, at least one of the proximity sensor (162), the camera sensor (163), at least one microphone, and a light sensor may detect a change in the front of the refrigerator (1) located in the X direction.

[0068] At least one of a proximity sensor (162), a camera sensor (163), at least one microphone, and a light sensor may be positioned on the rear of the door (300). For example, at least one of the proximity sensor (162), the camera sensor (163), at least one microphone, and a light sensor may detect a change in the interior of the refrigerator (1) (e.g., the storage compartment (20)).

[0069] FIG. 3 is a block diagram showing the configuration of a refrigerator (1) according to one embodiment of the present disclosure.

[0070] Referring to FIG. 3, the refrigerator (1) may include a door control module (150), a sensor unit (160), a cooling unit (170), a communication unit (180), a control unit (190), and a display (200).

[0071] The door control module (150) can control the opening or closing of at least one door (e.g., at least one door (300) of FIG. 1). The door control module (150) can include a motor drive unit (151) and a motor (152). For example, the door control module (150) can precisely control the movement of at least one door (300) depending on whether or not the at least one door (300) is open or the degree to which the at least one door (300) is opened.

[0072] The motor driving unit (151) can control the motor. For example, the motor driving unit (151) can activate or deactivate the motor (152). For example, the motor driving unit (151) can control the operating state of the motor (152) by supplying or cutting off power to the motor (152).

[0073] The motor (152) can open at least one door by rotating. For example, the motor (152) can be activated based on the control of the motor drive unit to open at least one door. For example, the motor (152) can be deactivated based on the control of the motor drive unit to close at least one door.

[0074] The sensor unit (160) may include a temperature sensor (161), a proximity sensor (162), a camera sensor (163), and a door open / close sensor (164).

[0075] The temperature sensor (161) can detect the temperature around the refrigerator (1) or inside the refrigerator (1). For example, the temperature sensor (161) may include a plurality of temperature sensors that detect the temperature inside the storage compartment (20). For example, the temperature sensor (161) may include a plurality of temperature sensors that detect the external temperature around the refrigerator (1).

[0076] For example, a plurality of temperature sensors may be installed in each of the plurality of storage rooms (20) to detect the temperature of each of the plurality of storage rooms (20) and output an electrical signal corresponding to the detected temperature to the control unit (190). Each of the plurality of temperature sensors may include a thermistor whose electrical resistance changes depending on the temperature.

[0077] The proximity sensor (162) may be a sensor that detects whether the distance to a person or object is within a predetermined distance by detecting a change in distance from the person or object. For example, the proximity sensor (162) may identify the approach of a user and detect the distance between the user and the refrigerator (1).

[0078] For example, the proximity sensor (162) may include at least one of an infrared sensor, an ultrasonic sensor, a capacitive sensor, and an inductive sensor.

[0079] The camera sensor (163) may be a sensor that converts light collected in a sensing area into an electrical signal to generate a digital image. For example, the camera sensor (163) may photograph objects around or inside the refrigerator (1) and generate a digital image.

[0080] For example, the camera sensor (163) may include at least one of a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, a CCD (Charge-Coupled Device) sensor, an IR camera, and an RGB camera.

[0081] As described above, the door open / close sensor (164) can output a preset judgment value indicating whether the door (300) is open or closed. For example, the door open / close sensor (164) can output 1 if the door (300) is open, and output 0 if the door (300) is closed.

[0082] The door open / close sensor (164) can also be implemented as a distance sensor, and if the distance between the door (300) and the main body is greater than or equal to a reference distance, the door (300) can be determined to be open, and if the distance is less than the reference distance, the door (300) can be determined to be closed.

[0083] However, the door open / close sensor (164) is not limited to this, and there is no limitation on its configuration as long as it can determine whether the door (300) is open and output a preset determination value.

[0084] The cooling unit (170) can supply cooled air to the storage room. Specifically, the cooling unit (170) can maintain the temperature of the storage room within a range specified by the user by utilizing the circulation of refrigerant in the refrigerant circuit.

[0085] The cooling unit (170) may include a compressor (171) that compresses a gaseous refrigerant, a condenser (172) that converts the compressed gaseous refrigerant into a liquid state, an expander (173) that decompresses the liquid refrigerant, and an evaporator (174) that converts the decompressed liquid refrigerant into a gaseous state. The cooling unit (170) may cool the air in the storage room by utilizing a phenomenon in which the liquid refrigerant absorbs heat energy of the surrounding air while converting into a gaseous state.

[0086] However, the cooling unit (170) is not limited to including a refrigerant circuit. For example, the cooling unit (170) may include a Peltier element utilizing the Peltier effect or a magnetic cooling material utilizing the magneto-caloric effect.

[0087] The communication unit (180) can exchange data with external devices such as a server device (e.g., server (5) of FIG. 17) and / or a user device (e.g., user device (4) of FIG. 17) and / or a display (200) and / or a cooking device (e.g., an oven or a microwave oven).

[0088] The communication unit (180) may include a wired communication module (182) that exchanges data with external devices via wire, and a wireless communication module (181) that exchanges data with external devices via wireless.

[0089] The wired communication module (182) can connect to a wired communication network and communicate with external devices through the wired communication network. For example, the wired communication module (182) can connect to a wired communication network through Ethernet (IEEE 802.3 technology standard) and receive data from external devices through the wired communication network.

[0090] The wireless communication module (181) can communicate wirelessly with a base station or an access point (AP), and can connect to a wired communication network via the base station or the access point. The wireless communication module (181) can also communicate with external devices connected to the wired communication network via the base station or the access point. For example, the wireless communication module (181) can wirelessly communicate with the access point (AP) using WiFi (IEEE 802.11 technology standard), or can communicate with the base station using CDMA, WCDMA, GSM, LET (Long Term Evolution), WiBro, etc. The wireless communication module (181) can also receive data from external devices via the base station or the access point. In addition, the wireless communication module (181) can directly communicate with external devices. For example, the wireless communication module (181) can wirelessly receive data from external devices using Wi-Fi, Bluetooth (IEEE 802.15.1 technology standard), ZigBee (IEEE 802.15.4 technology standard), etc.

[0091] In this way, the communication unit (180) can transmit or receive data with external devices, and in particular, can receive video data including video and / or audio from external devices, and output the received data to the control unit (190).

[0092] The control unit (190) may process user input and / or door open / close detection data and / or communication data, and control components included in the refrigerator (1) based on the data processing. According to one embodiment, the control unit (190) may include various processing circuits and / or multiple processors. For example, the term “processor” as used in the present disclosure, including the claims, may include various processing circuits including at least one processor, wherein one or more of the at least one processor may be configured to individually and / or collectively perform various functions described in the present disclosure in a distributed manner. As used herein, when "a processor," "at least one processor," and "one or more processors" are described as being configured to perform numerous functions, these terms encompass, for example, but are not limited to, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, and also situations where a single processor can perform all of the recited functions. Additionally, the at least one processor may comprise a combination of processors that perform the various recited / disclosed functions, for example, in a distributed manner. The at least one processor may execute program instructions to achieve or perform the various functions.

[0093] The control unit (190) may include a memory (192) including one or more storage media for storing / memorizing programs and / or data, and a processor (191) for processing user input and / or door opening / closing detection data and / or communication data according to the programs and / or data stored in the memory (192).

[0094] The memory (192) can store / remember programs and / or data. The program includes a plurality of commands combined to perform a specific function, and data can be processed and / or manipulated by the plurality of commands included in the program. In addition, the program and / or data can include a system program and / or system data directly related to the operation of the refrigerator (1), and an application program and / or application data that provide convenience to the user.

[0095] The memory (192) may include a non-volatile memory that stores a program and / or data for controlling the components included in the refrigerator (1) and a volatile memory that stores temporary data generated while controlling the components included in the refrigerator (1).

[0096] Non-volatile memory can store programs and / or data electrically, magnetically, or optically, for example. Non-volatile memory may include, for example, read-only memory (ROM) for long-term data storage and flash memory. Furthermore, non-volatile memory may include solid-state drives (SSDs), hard disk drives (HDDs), or optical disk drives (ODDs).

[0097] Volatile memory can load programs and / or data from non-volatile memory, for example, and electrically store programs and / or data. Volatile memory can include, for example, static random access memory (S-RAM), dynamic random access memory (DRAM), etc., for temporarily storing data.

[0098] This memory (192) can store / remember programs and data such as an operating system (OS), middleware, and applications, and can provide programs and data to the processor (191) in response to a request from the processor (191).

[0099] The processor (191) can process user input of the display (200) and / or detection data of the proximity sensor (162) and / or communication data of the communication unit (180) according to the program and / or data stored / stored in the memory (192). In addition, the processor (191) can generate a control signal for controlling the operation of the camera sensor (163), the display (200) and / or the communication unit (180) based on the data processing.

[0100] FIG. 4 is an exemplary diagram showing a door control module (150) and a sensor unit (160) of a refrigerator (1) according to one embodiment of the present disclosure.

[0101] Referring to FIG. 4, the refrigerator (1) may include a door control module (150) and a sensor unit (160). The door control module (150) and the sensor unit (160) may be formed at a predetermined location of the refrigerator (1) according to their respective functions.

[0102] The door control module (150) can control the opening or closing of at least one door. The door control module (150) can include a motor drive unit (151) and a motor (152). For example, the door control module (150) can precisely control the movement of at least one door (300) depending on whether or not the at least one door (300) is open or the degree to which the door is opened.

[0103] The motor driving unit (151) can control the motor. For example, the motor driving unit (151) can activate or deactivate the motor (152). For example, the motor driving unit (151) can control the operating state of the motor (152) by supplying or cutting off power to the motor (152).

[0104] The motor (152) can open at least one door by rotating. For example, the motor (152) can be activated based on the control of the motor drive unit to open at least one door. For example, the motor (152) can be deactivated based on the control of the motor drive unit to close at least one door.

[0105] As shown in FIG. 4, the door control module (150) may include a first door control module (150a), a second door control module (150b), a third door control module (150c), and a fourth door control module (150d) corresponding to at least one door (300). Each of the first door control module (150a) to the fourth door control module (150d) may include a motor driving unit (151) (e.g., motor driving units (151a, 151b, 151c, 151d)) and a motor (152) (e.g., motors (152a, 152b, 152c, 152d)) for controlling the opening and closing of at least one door (300).

[0106] The first door control module (150a) is formed on the upper frame adjacent to the left upper door (300a), thereby controlling the opening or closing of the left upper door (300a). For example, the first motor driving unit (151a) receives a door control signal from the processor (191), and activates or deactivates the first motor (152a) based on the door control signal, thereby controlling the opening or closing of the left upper door (300a).

[0107] The second door control module (150b) is formed on the upper frame adjacent to the right upper door (300b), thereby controlling the opening or closing of the right upper door (300b). For example, the second motor driving unit (151b) receives a door control signal from the processor (191), and activates or deactivates the second motor (152b) based on the door control signal, thereby controlling the opening or closing of the right upper door (300b).

[0108] The third door control module (150c) is formed on the lower frame adjacent to the left lower door (300c), thereby controlling the opening or closing of the left lower door (300c). For example, the third motor driving unit (151c) receives a door control signal from the processor (191), and activates or deactivates the third motor (152c) based on the door control signal, thereby controlling the opening or closing of the left lower door (300c).

[0109] The fourth door control module (150d) is formed on the lower frame adjacent to the right lower door (300d), thereby controlling the opening or closing of the right lower door (300d). For example, the fourth motor driving unit (151d) receives a door control signal from the processor (191), and activates or deactivates the fourth motor (152d) based on the door control signal, thereby controlling the opening or closing of the right lower door (300d).

[0110] The sensor unit (160) may include a temperature sensor (161), a proximity sensor (162), a camera sensor (163), and a door open / close sensor (164). In FIG. 4, a sensor group (160') including some components of the sensor unit (160) is illustrated. For example, the sensor group (160') may include a proximity sensor (162) and a camera sensor (163).

[0111] The proximity sensor (162) may be a sensor that detects whether the distance to a person or object is within a predetermined distance by detecting a change in distance from the person or object. For example, the proximity sensor (162) may identify the approach of a user and detect the distance between the user and the refrigerator (1).

[0112] The camera sensor (163) may be a sensor that converts light collected in a sensing area into an electrical signal to generate a digital image. For example, the camera sensor (163) may photograph objects around or inside the refrigerator (1) and generate a digital image.

[0113] In the present disclosure, the camera sensor (163) may include an IR camera (163a) and an RGB camera (163b). For example, the camera sensor (163) may capture objects around the refrigerator (1) and generate digital images using both the IR camera (163a) and the RGB camera (163b). In particular, the camera sensor (163) of the present disclosure may detect a user's hand by simultaneously using the IR camera (163a) and the RGB camera (163b), thereby recognizing the user's hand with high accuracy even in various environments.

[0114] For example, the proximity sensor (162) may be positioned independently as a separate sensor from the camera sensor (163). For example, the proximity sensor (162) and the camera sensor (163) may be positioned adjacently as a set. In FIG. 4, a case in which the proximity sensor (162) and the camera sensor (163) are positioned adjacently as a single sensor group (160') is illustrated.

[0115] In one embodiment, the camera sensor (163) may be formed at various locations in the refrigerator (1). Various arrangements of the camera sensor (163) in the refrigerator (1) are described below with reference to FIG. 5.

[0116] FIG. 5 is an exemplary diagram showing various arrangements of a camera sensor (163) according to one embodiment of the present disclosure.

[0117] Referring to FIG. 5, the camera sensor (163) may be formed at various locations on the refrigerator (1) in order to effectively capture objects around the refrigerator (1). For example, the camera sensor (163) may be formed between the upper door (300a, 300b) and the lower door (300c, 300d) of the refrigerator (1). For example, the camera sensor (163) may be formed on the upper frame of the refrigerator (1). For example, the camera sensor (163) may be formed between the left door (300a, 300c) and the right door (300b, 300d) of the refrigerator (1). For example, the camera sensor (163) may be formed on the front of at least one door. For example, the camera sensor (163) may be formed in a portion of the display.

[0118] As shown in (a) and (b) of FIG. 5, the camera sensor (163) may be formed between the upper door (300a, 300b) and the lower door (300c, 300d) of the refrigerator (1). For example, the camera sensor (163) may be formed in the center between the upper door (300a, 300b) and the lower door (300c, 300d) of the refrigerator (1). For example, one camera sensor (163) among the plurality of camera sensors (163) may be formed on the left side between the upper door (300a, 300b) and the lower door (300c, 300d) of the refrigerator (1), and the other camera sensor (163) may be formed on the right side between the upper door (300a, 300b) and the lower door (300c, 300d) of the refrigerator (1). When the camera sensor (163) is formed between the upper door (300a, 300b) and the lower door (300c, 300d) of the refrigerator (1), the camera sensor (163) can recognize and distinguish with high accuracy a hand facing the upper door (300a, 300b) and a hand facing the lower door (300c, 300d).

[0119] As shown in (c) and (d) of FIG. 5, the camera sensor (163) may be formed on the upper frame of the refrigerator (1). For example, the camera sensor (163) may be formed in the center of the upper frame of the refrigerator (1). For example, among the plurality of camera sensors (163), one camera sensor (163) may be formed on the left side of the upper frame of the refrigerator (1), and another camera sensor (163) may be formed on the right side of the upper frame of the refrigerator (1). When the camera sensor (163) is formed on the upper frame of the refrigerator (1), the camera sensor (163) can identify and recognize the user's hand toward the refrigerator (1) with high accuracy based on a wide viewing angle.

[0120] As shown in (e), (f), and (g) of FIG. 5, the camera sensor (163) may be formed between the left door (300a, 300c) and the right door (300b, 300d) of the refrigerator (1). For example, the camera sensor (163) may be formed between the left upper door (300a) and the right upper door (300b) of the refrigerator (1). For example, the camera sensor (163) may be formed between the left lower door (300c) and the right lower door (300d) of the refrigerator (1). For example, the camera sensor (163) may be formed between the left door (300a, 300c) and the right door (300b, 300d) of the refrigerator (1) so as to be connected to the upper frame of the refrigerator (1). When the camera sensor (163) is formed between the left door (300a, 300c) and the right door (300b, 300d) of the refrigerator (1), the camera sensor (163) can recognize and distinguish with high accuracy the hand facing the left door (300a, 300c) and the hand facing the right door (300b, 300d).

[0121] As shown in (h) and (i) of FIG. 5, the camera sensor (163) may be formed on the front of at least one door. For example, the at least one door may include at least one hole through which the camera sensor (163) can capture an object in front. For example, the at least one door may be made of glass through which the camera sensor (163) can capture an object in front, or may be partially made of glass or another transparent or semi-transparent material. When the camera sensor (163) is formed on the front of at least one door, the camera sensor (163) can identify and recognize the user's hand toward the refrigerator (1) with high accuracy based on an increase in sensing distance.

[0122] As shown in (j) of FIG. 5, the camera sensor (163) may be formed in a portion of a display (e.g., display (200) of FIG. 1). For example, the camera sensor (163) may be formed in a peripheral portion (e.g., non-display portion) of a display panel (e.g., display panel (220) of FIG. 1). For example, the camera sensor (163) may be an under-display camera (UDC) formed in a display portion of a display panel (e.g., display panel (220) of FIG. 1). When the camera sensor (163) is formed in a portion of a display, the refrigerator (1) may provide a user-friendly user interface based on the linkage between the camera sensor (163) and the display.

[0123] Meanwhile, the arrangement of the camera sensor (163) included in the refrigerator (1) according to one embodiment of the present disclosure is not limited to the positions shown in (a) to (j) of FIG. 5. In addition, although only the camera sensor (163) included in the refrigerator (1) is illustrated in FIG. 5, a proximity sensor (162) may also be arranged at the same position as the camera sensor (163) or at a position adjacent to the camera sensor (163).

[0124] Fig. 6 is a flowchart showing the operation of a refrigerator (1) according to one embodiment of the present disclosure.

[0125] Referring to FIG. 6, the refrigerator (1) of the present disclosure detects a user's hand by simultaneously using an IR camera (163a) and an RGB camera (163b), and clearly identifies structural information of the user's hand through preprocessing of the hand image, thereby automatically opening a door that matches the user's intention.

[0126] Specifically, the refrigerator (1) activates the camera sensor (163) (operation 610) based on the identification that the user is located within a first distance (D1) from the refrigerator (1) using the proximity sensor (162), obtains an input image through the camera sensor (163) based on the identification that the user is located within a second distance (D2) shorter than the first distance (D1) from the refrigerator (1) using the proximity sensor (162), identifies a hand image corresponding to the hand of the user from the input image (operation 630), performs preprocessing on the hand image (operation 640), determines a hand gesture of the user from the hand image using at least one artificial intelligence model (operation 650), and opens a door selected based on the first gesture based on the determination that the hand gesture is a first gesture for selecting a door to be opened (operation 660).

[0127] In one example, at operation 610, the refrigerator (1) may activate the camera sensor (163) based on the identification that the user is located within a first distance (D1) from the refrigerator (1) using the proximity sensor (162).

[0128] The proximity sensor (162) can detect whether the user is within a predetermined distance from the refrigerator (1) by detecting a change in the distance between the user and the refrigerator (1). The refrigerator (1) can activate the camera sensor (163) based on the identification that the user is located within a first distance (D1) from the refrigerator (1). When the camera sensor (163) is activated, the camera sensor (163) can capture a picture of the user within the first distance (D1).

[0129] For example, in operation 620, the refrigerator (1) may acquire an input image through the camera sensor based on the identification that the user is located within a second distance (D2) shorter than the first distance (D1) from the refrigerator (1) using the proximity sensor (162). For example, the refrigerator (1) may acquire the input image using the camera sensor (163). The refrigerator (1) may sense the user's hand using the camera sensor (163).

[0130] The camera sensor (163) can capture a user's hand within a second distance (D2) from the refrigerator (1). The camera sensor (163) can sense the user's hand by simultaneously using an IR camera (163a) and an RGB camera (163b). The IR camera (163a) may be a camera sensor (163) that detects an infrared spectrum to generate an input image. For example, the refrigerator (1) can capture a user's hand using the IR camera (163a) to generate a first input image. The RGB camera (163b) may be a camera sensor (163) that detects red, green, and blue, which are the primary colors of light, and generates an input image based on each color. For example, the refrigerator (1) can capture a user's hand using the RGB camera (163b) to generate a second input image.

[0131] In this way, the refrigerator (1) of the present disclosure can increase the hand recognition rate by sensing the user's hand using an IR camera (163a) and an RGB camera (163b) simultaneously.

[0132] According to an example, in operation 630, the refrigerator (1) can identify a hand image corresponding to the user's hand from an input image acquired through the camera sensor (163). For example, the refrigerator (1) can identify a hand image corresponding to the user's hand from a first input image acquired from an IR camera (163a). For example, the refrigerator (1) can identify a hand image corresponding to the user's hand from a second input image acquired from an RGB camera (163b).

[0133] In one embodiment, the refrigerator (1) can generate a third input image by synthesizing a first input image acquired from the IR camera (163a) and a second input image acquired from the RGB camera (163b). For example, the third input image can be an input image obtained by synthesizing the first input image and the second input image at the same ratio (e.g., 1:1). The refrigerator (1) can identify a hand image corresponding to a user's hand from the third input image.

[0134] In one embodiment, the third input image may be an input image synthesized by synthesizing the first input image and the second input image at different ratios. For example, the refrigerator (1) may synthesize the first input image and the second input image by applying a first weight to the first input image and a second weight to the second input image.

[0135] The first weighting may increase as the ambient illumination decreases. For example, since IR cameras can operate in environments with little light, the first weighting applied to the first input image may increase as the ambient illumination decreases.

[0136] The second weighting may increase as the ambient illumination increases. For example, since an RGB camera provides high resolution, the second weighting applied to the second input image may increase as the ambient illumination increases.

[0137] The refrigerator (1) can analyze the shape of the hand, the palm area, the finger length, the angle between the fingers, the distinctive features of the hand (e.g., wrinkles, creases, scars, joint positions, etc.) included in at least one of the first input image, the second input image, and the third input image. The refrigerator (1) can identify the hand image included in the input image by filtering the analyzed input image.

[0138] For example, in operation 640, the refrigerator (1) may perform preprocessing on a hand image. Before determining a hand gesture using an artificial intelligence model (as described in more detail herein), the refrigerator (1) may remove unnecessary noise from the hand image and emphasize important features of the hand image. For example, the preprocessing on the hand image may include at least one of noise removal from the hand image, emphasis on key features of the hand, standardization and normalization of the hand image, and shape transformation of the hand image.

[0139] In one embodiment, the refrigerator (1) can perform a first preprocessing operation to remove a background image excluding the hand image from the input image. By effectively removing all background elements excluding the hand image from the input image through the first preprocessing operation, the refrigerator (1) can clearly determine the hand gesture.

[0140] In one embodiment, the refrigerator (1) can perform a second preprocessing operation to emphasize hand wrinkles, hand joints, and fingers by performing contrast correction on the hand image. The refrigerator (1) can clearly determine hand gestures by generating an emphasis line (EL) representing the detailed structure of the user's hand through contrast correction on the hand image through the second preprocessing operation.

[0141] In one embodiment, the refrigerator (1) may perform a third preprocessing step to generate bounding lines or bounding boxes for the thickness, length, and orientation of the fingers. The refrigerator (1) can accurately determine hand gestures by determining the precise positions and orientations of the fingers, palm, and back of the hand in the hand image through the third preprocessing step.

[0142] In this way, the refrigerator (1) of the present disclosure can increase the accuracy of judgment of the hand gesture by performing preprocessing on the hand image.

[0143] For example, in operation 650, the refrigerator (1) can determine a user's hand gesture from a hand image using an artificial intelligence model (described in more detail in the present disclosure). For example, the refrigerator (1) can determine the hand gesture by inferring an intended gesture from the hand image using the artificial intelligence model that has learned a correlation between the hand image and the preset hand gesture.

[0144] An intention gesture may be a hand gesture implemented by a user through a hand movement or hand pose. For example, an intention gesture may be a gesture actually performed by a user to provide a preset hand gesture (e.g., a first gesture, a second gesture, a third gesture) to a refrigerator (1). The refrigerator (1) can infer an intention gesture from a hand image and determine a hand gesture (e.g., a first gesture, a second gesture, a third gesture) intended by the user from the intention gesture.

[0145] The AI ​​model can determine hand orientation based on hand wrinkles, knuckles, and thumb position. For example, the AI ​​model can determine whether fingers are extended or folded based on the angle between the fingers and the length of the fingers. For example, the AI ​​model can infer the intended gesture based on the hand orientation and whether the fingers are extended or folded.

[0146] The refrigerator (1) can determine the intention gesture based on the hand direction and the extended or folded fingers as one of a first gesture for selecting a door to be opened, a second gesture for agreeing to a door to be opened, and a third gesture for denying a door to be opened.

[0147] In this way, the refrigerator (1) of the present disclosure can increase the efficiency of automatic door opening by judging the hand gesture using the artificial intelligence model.

[0148] For example, in operation 660, the refrigerator (1) may open the door selected based on the first gesture, based on the hand gesture being determined to be the first gesture for selecting the door to be opened.

[0149] The refrigerator (1) can recognize a command to open at least one door from a first gesture. The refrigerator (1) can select at least one door by analyzing the meaning of the first gesture. The refrigerator (1) can open the selected door based on the first gesture. For example, the door control module (150) can automatically open the selected door using the rotational force of a motor.

[0150] In this way, the refrigerator (1) of the present disclosure can detect the user's hand with high accuracy even in various environments by simultaneously using the IR camera (163a) and the RGB camera (163b). In addition, the refrigerator (1) of the present disclosure can increase the accuracy of determining the user's hand gesture by clearly identifying the structural information of the user's hand through preprocessing of the hand image. Accordingly, the refrigerator (1) of the present disclosure can increase the convenience and efficiency of using the refrigerator (1) by accurately and automatically opening the door that matches the user's intention.

[0151] FIG. 7 is a drawing showing an operation of a proximity sensor (162) according to one embodiment of the present disclosure to sense the approach of a user (2) to a refrigerator (1).

[0152] Referring to FIG. 7, the refrigerator (1) can activate the camera sensor (163) based on the identification that the user (2) is located within a first distance (D1) from the refrigerator (1) using the proximity sensor (162).

[0153] The proximity sensor (162) can detect whether the user (2) is within a predetermined distance from the refrigerator (1) by detecting a change in the distance between the user (2) and the refrigerator (1). For example, the proximity sensor (162) can identify the approach of the user (2) and detect whether the user (2) is located within a first distance (D1) from the refrigerator (1). For example, the proximity sensor (162) can include at least one of an infrared sensor, an ultrasonic sensor, a capacitive sensor, and an inductive sensor.

[0154] The refrigerator (1) can activate the camera sensor (163) based on the identification that the user (2) is located within a first distance (D1) from the refrigerator (1). When the camera sensor (163) is activated, the camera sensor (163) can capture a photo of the user (2) within the first distance (D1). For example, the camera sensor (163) can generate an input image by capturing a part of the body of the user (2).

[0155] The refrigerator (1) can sense the hand of the user (2) based on the identification that the user (2) is located within a second distance (D2) shorter than the first distance (D1) from the refrigerator (1) using a proximity sensor (162). For example, the refrigerator (1) can sense the hand of the user (2) using a camera sensor (163).

[0156] The camera sensor (163) may be a sensor that converts light collected in a sensing area into an electrical signal to generate a digital image. For example, the camera sensor (163) may capture a user (2) within a first distance (D1) from the refrigerator (1). For example, the first distance (D1) may be substantially 0.8 m to substantially 1.2 m, but may be a different distance in other embodiments. For example, the camera sensor (163) may include at least one of a CMOS sensor, a CCD sensor, an IR camera (163a), and an RGB camera (163b).

[0157] The camera sensor (163) can capture the hand of the user (2) within a second distance (D2) from the refrigerator (1). The camera sensor (163) can sense the hand of the user (2) simultaneously using an IR camera (163a) and an RGB camera (163b). For example, the second distance (D2) may be substantially 0.2 m to substantially 0.4 m, but may be a different distance in other embodiments.

[0158] FIG. 8 is an exemplary diagram showing the configuration of a camera sensor (163) according to one embodiment of the present disclosure.

[0159] Referring to FIG. 8, the camera sensor (163) may include an IR camera (163a) and an RGB camera (163b). For example, the camera sensor (163) may capture a user approaching the refrigerator (1) using both the IR camera (163a) and the RGB camera (163b) and generate an input image. According to one embodiment, the camera sensor (163) may detect the user's hand (3) by simultaneously using the IR camera (163a) and the RGB camera (163b), thereby recognizing the user's hand (3) with high accuracy even in various environments.

[0160] The IR camera (163a) may be a camera sensor (163) that detects the infrared spectrum and generates an input image. Since the IR camera (163a) can operate even in environments with minimal light, it can identify and capture a user's hand (3) even in dark conditions such as at night. For example, a refrigerator (1) can capture a user's hand (3) using the IR camera (163a) to generate a first input image.

[0161] The RGB camera (163b) may be a camera sensor (163) that detects the primary colors of light, red, green, and blue, and generates an input image based on each color. The RGB camera (163b) provides high resolution and can clearly capture details of a user's hand (3). For example, a refrigerator (1) can capture a user's hand (3) using the RGB camera (163b) to generate a second input image.

[0162] The camera sensor (163) senses the user's hand (3) by simultaneously using an IR camera (163a) and an RGB camera (163b), so that in a high-light environment (e.g., bright daytime), the hand (3) can be sensed according to the high resolution of the RGB camera (163b), and in a low-light environment (e.g., dark or at night), the hand (3) can be sensed according to the infrared rays of the IR camera (163a).

[0163] FIG. 9 is an exemplary diagram showing an operation of a refrigerator (1) according to one embodiment of the present disclosure to identify a hand image from an input image.

[0164] Referring to FIG. 9, the refrigerator (1) can identify a hand image (901) corresponding to the user's hand (3) from an input image acquired through a camera sensor (163). For example, the refrigerator (1) can identify a hand image (901) corresponding to the user's hand (3) from a first input image acquired from an IR camera (163a). For example, the refrigerator (1) can identify a hand image (901) corresponding to the user's hand (3) from a second input image acquired from an RGB camera (163b).

[0165] In one embodiment, the refrigerator (1) can generate a third input image by synthesizing a first input image acquired from the IR camera (163a) and a second input image acquired from the RGB camera (163b). For example, the third input image can be an input image obtained by synthesizing the first input image and the second input image at the same ratio (e.g., 1:1). The refrigerator (1) can identify a hand image (901) corresponding to the user's hand (3) from the third input image.

[0166] In one embodiment, the third input image may be an input image synthesized by synthesizing the first input image and the second input image at different ratios. For example, the refrigerator (1) may synthesize the first input image and the second input image by applying a first weight to the first input image and a second weight to the second input image. For example, the first weight may increase as the ambient illuminance decreases. For example, the second weight may increase as the ambient illuminance increases.

[0167] Specifically, the refrigerator (1) can analyze each of the first input image, the second input image, and the third input image. For example, the refrigerator (1) can analyze the shape of the hand (3), the palm area, the finger length, the angle between the fingers, etc. included in at least one of the first input image, the second input image, and the third input image.

[0168] The refrigerator (1) can identify the hand image included in the input image by filtering the analyzed input image. For example, in a high-light environment (e.g., bright daylight), the refrigerator (1) can identify the hand image by assigning a greater weight to the first input image than to the second or third input image. For example, in a low-light environment (e.g., dark night), the refrigerator (1) can identify the hand image by assigning a greater weight to the second input image than to the first or third input image.

[0169] FIG. 10 is an exemplary diagram showing the first pretreatment of a refrigerator (1) according to one embodiment of the present disclosure.

[0170] Referring to FIG. 10, the refrigerator (1) can preprocess a hand image (901) to generate an image (1001). For example, preprocessing of the hand image (901) to generate the image (1001) may include at least one of noise removal from the hand image (901), emphasis on key features of the hand, standardization and normalization of the hand image (901), and shape transformation of the hand image (901).

[0171] In one embodiment, the refrigerator (1) can perform a first preprocessing operation that removes a background image from the input image (e.g., a hand image (901)) while leaving the hand in the image (e.g., image (1001)). By effectively removing background elements from the input image through the first preprocessing operation while leaving the hand in the image, the refrigerator (1) can clearly determine the hand gesture.

[0172] For example, the refrigerator (1) can leave a hand in the image while removing the background image from the input image using the image segmentation method. The refrigerator (1) can distinguish the hand and the background in the input image based on parameters such as color, texture, and boundary in the input image. The refrigerator (1) can remove the background image that differs from the hand image in the parameters such as color, texture, and boundary.

[0173] For example, a refrigerator (1) can remove a background image from an input image using a deep learning model such as a convolutional neural network (CNN) while leaving a hand in the image. The deep learning model can pre-learn the shape, size, and position of the hand. The deep learning model can pre-learn the surrounding environment (e.g., background) of the refrigerator (1) before the user approaches it. Based on the pre-learning, the deep learning model can remove the background image from the input image and extract only the hand image.

[0174] FIG. 11 is an exemplary diagram showing a second pretreatment of a refrigerator (1) according to one embodiment of the present disclosure.

[0175] Referring to FIG. 11, the refrigerator (1) can perform preprocessing on a hand image. For example, the preprocessing on the hand image can include at least one of noise removal from the hand image, emphasis on key features of the hand, standardization and normalization of the hand image, and shape transformation of the hand image.

[0176] In one embodiment, the refrigerator (1) can perform a second preprocessing operation to emphasize hand wrinkles, hand joints, and fingers by performing contrast correction on the hand image. The refrigerator (1) can clearly determine hand gestures by generating an emphasis line (EL) representing the detailed structure of the user's hand through contrast correction on the hand image through the second preprocessing operation.

[0177] For example, the refrigerator (1) can emphasize fine wrinkles and joints of a hand by adjusting the contrast of the hand image using histogram equalization. Based on histogram equalization, highlight lines (ELs) for fine wrinkles and joints of the hand can be created in the hand image.

[0178] For example, the refrigerator (1) can enhance the contrast of a specific portion of a hand image (e.g., a finger boundary, a finger joint, a palm wrinkle) using a local contrast enhancement method. Based on the local contrast enhancement, an emphasis line (EL) can be generated for at least one of the finger boundary, the finger joint, and the palm wrinkle in the hand image.

[0179] FIG. 12 is an exemplary diagram showing the third pretreatment of a refrigerator (1) according to one embodiment of the present disclosure.

[0180] Referring to FIG. 12, the refrigerator (1) can perform preprocessing on a hand image. For example, the preprocessing on the hand image can include at least one of noise removal from the hand image, emphasis on key features of the hand, standardization and normalization of the hand image, and shape transformation of the hand image.

[0181] In one embodiment, the refrigerator (1) may perform a third preprocessing step to generate a bounding line (BL) or a bounding box (BB) for the thickness, length, and direction of the finger. By determining the precise positions and directions of the fingers, palm, and back of the hand in the hand image through the third preprocessing step, the refrigerator (1) can clearly determine the hand gesture.

[0182] For example, the refrigerator (1) can detect fingers in a hand image using an image processing algorithm. The image processing algorithm can perform analysis on at least one of color, texture, and boundary of the hand image.

[0183] The refrigerator (1) can generate a bounding box (BB) by calculating the minimum rectangular area of ​​at least one finger. For example, the bounding box (BB) can represent finger information including the length and thickness of the finger.

[0184] The refrigerator (1) can generate a bounding line (BL) along the center line of at least one finger. For example, the bounding line (BL) can indicate the direction of at least one finger. For example, the bounding line (BL) can indicate the angle between at least two fingers.

[0185] FIG. 13 is a diagram illustrating a process in which an artificial intelligence model determines a user's hand gesture from a hand image according to one embodiment of the present disclosure.

[0186] Referring to FIG. 13, the refrigerator (1) can determine a user's hand gesture from a hand image (1301) using an artificial intelligence model. For example, the refrigerator (1) can determine the hand gesture by inferring an intended gesture from the hand image (1301) using the artificial intelligence model that has learned a correlation between the hand image and the preset hand gesture (e.g., AI inference (1302)).

[0187] An intention gesture may be a hand gesture implemented by a user through a hand movement or a hand pose. For example, an intention gesture may be a gesture actually performed by a user to provide a preset hand gesture (e.g., a first gesture (e.g., Hand Gesture 1 (1303)), a second gesture (e.g., Hand Gesture 2 (1304)), a third gesture (e.g., Hand Gesture 3 (1305))) to the refrigerator (1). The refrigerator (1) can infer an intention gesture from a hand image and determine a hand gesture (e.g., a first gesture, a second gesture, a third gesture) intended by the user to provide from the intention gesture.

[0188] Meanwhile, the artificial intelligence model used by the refrigerator (1) for AI inference (1302) of the intended gesture may be a single artificial intelligence model or may be implemented as multiple artificial intelligence models. The artificial intelligence model may be composed of a neural network (or artificial neural network) and may include a statistical learning algorithm that mimics biological neurons in machine learning and cognitive science. A neural network may refer to a model in general that has problem-solving capabilities by changing the binding strength of synapses through learning, in which artificial neurons (nodes) that form a network by combining synapses. The neurons of the neural network may include a combination of weights or biases. The neural network may include one or more layers composed of one or more neurons or nodes. For example, the device may include an input layer, a hidden layer, and an output layer. The neural network constituting the device can infer a desired result (output) from an arbitrary input (input) by changing the weights of neurons through learning.

[0189] At least one processor included in the refrigerator (1) can create a neural network, train a neural network, perform a calculation based on received input data, generate an information signal based on the result of the calculation, or retrain a neural network. The models of the neural network may include various types of models such as CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), RNN (RecuREnt Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, etc., but are not limited thereto. The processor may include one or more processors for performing calculations according to the models of the neural network. There are. For example, a neural network can include a deep neural network.

[0190] Neural networks include CNN (Convolutional Neural Network), RNN (RecuREnt Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated RecuREnt Unit), AE (Auto Encoder), VAE (Variational Auto Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics) Network), GAN (Generative Adversarial Network), LSM (Liquid State Machine), ELM (Extreme Learning It will be understood by those skilled in the art that any neural network may be included, including but not limited to, a Machine (Echo State Network), an ESN (Echo State Network), a DRN (Deep Residual Network), a DNC (Differentiable Neural Computer), an NTM (Neural Turning Machine), a CN (Capsule Network), a KN (Kohonen Network), and an AN (Attention Network).

[0191] According to an exemplary embodiment of the present disclosure, at least one processor included in the refrigerator (1) is a CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, etc., R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), RNN (RecuREnt Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT for natural language processing, SP-BERT, MRC / QA, Text Analysis, Dialog System, GPT-3, GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, ResNet Anomaly Detection for data intelligence, Various artificial intelligence structures and algorithms can be used, including but not limited to Prediction, Time-Series Forecasting, Optimization, Recommendation, and Data Creation.

[0192] In one embodiment, the artificial intelligence model can determine hand orientation based on hand wrinkles, hand joints, and thumb positions, determine extended or folded fingers based on the angle between fingers and finger lengths, and infer the intended gesture based on the hand orientation and the extended or folded fingers.

[0193] For example, the artificial intelligence model may include a first model that infers hand orientation. The first model may determine hand orientation based on hand wrinkles, knuckles, and thumb positions. For example, the refrigerator (1) may determine whether the user's hand included in the hand image is the back of the hand or the palm based on the hand wrinkles, knuckles, and thumb positions.

[0194] For example, the artificial intelligence model may include a second model that infers whether fingers are extended or folded. The second model may determine whether fingers are extended or folded based on the angle between the fingers and the length of the fingers. For example, the refrigerator (1) may determine which of the user's fingers included in the hand image are extended and which are folded based on the angle between the fingers and the length of the fingers.

[0195] For example, the artificial intelligence model may include a third model that infers an intention gesture. The third model may infer the intention gesture based on the hand orientation and the extended or folded fingers. For example, the refrigerator (1) may determine that the intention gesture is a predetermined hand gesture with the highest degree of consistency among the first gesture, the second gesture, and the third gesture.

[0196] Specifically, the refrigerator (1) can determine the intention gesture based on the hand direction and the extended or folded fingers as one of a first gesture for selecting a door to be opened (e.g., Hand Gesture 1 (1303)), a second gesture for agreeing to a door to be opened (e.g., Hand Gesture 2 (1304)), and a third gesture for denying a door to be opened (e.g., Hand Gesture 3 (1305)).

[0197] The refrigerator (1) can open a door selected based on the first gesture, based on the hand gesture being determined to be a first gesture (e.g., Hand Gesture 1 (1303)) for selecting a door to be opened. The refrigerator (1) can recognize a command to open at least one door from the first gesture. The refrigerator (1) can select at least one door by analyzing the meaning of the first gesture.

[0198] The refrigerator (1) can open a selected door based on the first gesture (e.g., Hand Gesture 1 (1303)). For example, the door control module (150) can automatically open the selected door using the rotational force of the motor. Depending on the embodiment, the refrigerator (1) can provide a visual or auditory notification indicating that the selected door has been opened.

[0199] FIG. 14 is a drawing showing an operation of a refrigerator (1) according to one embodiment of the present disclosure to determine a door to be opened based on the position of a user's hand (3).

[0200] Referring to FIG. 14, the refrigerator (1) divides the sensing area where the hand (3) is sensed into a plurality of sub-areas based on a virtual center point (VCP), and selects a door to be opened among at least one door based on the area where the hand (3) is located among the plurality of sub-areas.

[0201] The refrigerator (1) can set a virtual center point (VCP) in front of the camera sensor (163). The refrigerator (1) can divide the sensing area where the hand (3) is sensed into a plurality of sub-areas based on the number and arrangement of at least one door.

[0202] Each of the multiple sub-areas may correspond to at least one door. For example, if a refrigerator (1) includes four doors, the refrigerator (1) may divide the sensing area into four sub-areas.

[0203] For example, as shown in FIG. 14, when the user's hand (3) is located in the first sub-area, the refrigerator (1) can open the left upper door (300a) corresponding to the first sub-area. For example, when the user's hand (3) is located in the second sub-area, the refrigerator (1) can open the right upper door (300b) corresponding to the second sub-area. For example, when the user's hand (3) is located in the third sub-area, the refrigerator (1) can open the left lower door (300c) corresponding to the third sub-area. For example, when the user's hand (3) is located in the fourth sub-area, the refrigerator (1) can open the right lower door (300d) corresponding to the fourth sub-area.

[0204] Meanwhile, the door opening method illustrated in FIG. 14 (e.g., determining whether the door is open based on the position of the user's hand (3)) and the door opening method illustrated in FIG. 13 (e.g., determining whether the door is open based on a hand gesture determined by an artificial intelligence model) may be complementary. For example, the refrigerator (1) may determine whether the door is open based on either the door opening method illustrated in FIG. 14 or the door opening method illustrated in FIG. 13. For example, the refrigerator (1) may determine whether the door is open by combining the door opening method illustrated in FIG. 14 and the door opening method illustrated in FIG. 13.

[0205] FIG. 15 is a diagram illustrating a confirmation notification for a first gesture and a second gesture according to one embodiment of the present disclosure.

[0206] Referring to FIG. 15, the refrigerator (1) provides a confirmation notification for confirming a door to be opened based on the first gesture, and can open at least one door based on a second gesture for agreeing to the door to be opened.

[0207] The refrigerator (1) may provide a confirmation notification confirming the door to be opened based on the hand gesture detected from the sensor group (160') being determined to be the first gesture for selecting the door to be opened. For example, the refrigerator (1) may provide a visual notification (1501) (e.g., a message) or an auditory notification (e.g., voice output) confirming the door to be opened.

[0208] After providing a confirmation notification confirming the door to be opened, the refrigerator (1) can determine an additional hand gesture from the user. For example, the refrigerator (1) can determine the additional hand gesture from the user as either a second gesture confirming the door to be opened or a third gesture denying the door to be opened.

[0209] The refrigerator (1) can open the door confirmed by the user through the confirmation notification based on the hand gesture being determined to be a second gesture agreeing to the door to be opened. For example, the door control module (150) can automatically open the door confirmed by the user using the rotational force of the motor.

[0210] FIG. 16 is a diagram illustrating a confirmation notification for a first gesture and a third gesture according to one embodiment of the present disclosure.

[0211] Referring to FIG. 16, the refrigerator (1) can provide a first confirmation notification (1601) for confirming the door to be opened based on the first gesture, and can re-judge the user's hand gesture based on a third gesture for denying the door to be opened.

[0212] The refrigerator (1) may provide a first confirmation notification (1601) confirming the door to be opened based on the hand gesture detected from the sensor group (160') being determined to be a first gesture for selecting a door to be opened. For example, the refrigerator (1) may provide a visual notification (e.g., a message) or an auditory notification (e.g., a voice output) confirming the door to be opened.

[0213] After providing a first confirmation notification (1601) confirming the door to be opened, the refrigerator (1) can determine an additional hand gesture of the user. For example, the refrigerator (1) can determine the additional hand gesture of the user as either a second gesture agreeing to open the door or a third gesture denying the door to be opened.

[0214] The refrigerator (1) may re-evaluate the user's hand gesture based on the determination that the hand gesture is a third gesture that denies the door to be opened. For example, the refrigerator (1) may re-analyze the meaning of the first gesture and re-select at least one door. In this case, the refrigerator (1) may provide a second confirmation notification (1602) confirming the door to be opened.

[0215] After providing a second confirmation notification (1602) confirming the door to be opened, the refrigerator (1) may determine an additional hand gesture of the user. For example, the refrigerator (1) may determine the additional hand gesture of the user as either a second gesture agreeing to open the door or a third gesture denying the door to be opened.

[0216] The refrigerator (1) can open the door confirmed by the user through the confirmation notification based on the hand gesture being determined to be a second gesture agreeing to the door to be opened. For example, the door control module (150) can automatically open the door confirmed by the user using the rotational force of the motor.

[0217] FIG. 17 is a drawing showing a user device (4) and a server (5) connected to a refrigerator (1) according to one embodiment of the present disclosure.

[0218] Referring to FIG. 17, the refrigerator (1) can communicate with a user device (4) or a server (5). For example, the refrigerator (1) can communicate using any of various wired or wireless communication protocols, such as Ethernet, GSM (global system for mobile communications), EDGE (enhanced data GSM environment), CDMA (code division multiple access), TDMA (time division multiplexing access), LTE (long term evolution), LTE-A (LTE advance), NR (new radio), Wi-Fi, or Bluetooth. For example, the refrigerator (1) can communicate with the user device (4) or the server (5) based on a wired or wireless communication protocol.

[0219] According to one embodiment, the user device (4) may be a device capable of performing various computing functions, such as a communication function, a display function, or an output function (e.g., a voice or audio output function). For example, the user device (4) may be a TV, a wearable device (e.g., earbuds, a hearing aid, or a head-mounted display (HMD)), a mobile device (e.g., a smartphone or a mobile phone), a tablet, a personal computer (PC), a desktop computer, a notebook computer, a personal digital assistant (PDA), a laptop, a media player, an e-book reader, a digital broadcasting terminal, a navigation device, a kiosk, a digital camera, or a home appliance. Meanwhile, the user device (4) is not limited to the above-described devices and may be another type of electronic device.

[0220] According to one embodiment, the user device (4) can perform communication using any of various wired or wireless communication protocols, such as Ethernet, GSM, EDGE, CDMA, TDMA, LTE, LTE-A, NR, Wi-Fi, or Bluetooth. For example, the user device (4) can perform communication with the refrigerator (1) or the server (5) based on the wired or wireless communication protocol.

[0221] In one embodiment, the server (5) may be connected to the refrigerator (1) and / or the user device (4). For example, the server (5) may be a cloud server. In one example, the server (5) may receive sensing data or setting data from the refrigerator (1) and / or the user device (4). For example, the server (5) may store the sensing data or setting data or transmit it to the refrigerator (1) and / or the user device (4).

[0222] In one embodiment, a user can change the control action based on a hand gesture. For example, the user can change the first gesture that selects a door to be opened. For example, the user can change the second gesture that agrees to the door to be opened. For example, the user can change the third gesture that denies the door to be opened.

[0223] According to one example, the refrigerator (1) can change the control operation according to the hand gesture based on the user's settings through at least one application of the user device (4). For example, the user device (4) can provide an application that changes at least one of the first gesture, the second gesture, and the third gesture. The user can change at least one of the first gesture, the second gesture, and the third gesture using the application of the user device (4).

[0224] According to one example, the refrigerator (1) can change the control operation according to the hand gesture based on the user's settings for the hand image identified through the camera sensor (163). For example, the refrigerator (1) can provide a control operation change mode. In the control operation change mode, the user can change at least one of the first gesture, the second gesture, and the third gesture by inputting a hand image through the camera sensor (163).

[0225] In one embodiment, the refrigerator (1) stores the hand image in the server (5), learns the hand characteristics of the user from the hand image using the at least one artificial intelligence model, and identifies the hand image corresponding to the hand of the user from the input image based on the hand characteristics of the user.

[0226] For example, the refrigerator (1) can store the hand image in the server (5) by transmitting the hand image identified from the input image to the server (5). As the hand image is accumulated and stored in the server (5), a database of the user's hands can be created.

[0227] The refrigerator (1) can analyze at least one of the user's hand shape, size, wrinkles, and finger length from a database of the user's hands stored in the server (5). For example, the refrigerator (1) can learn unique hand characteristics that distinguish the user's hands from those of other users, such as the length ratio between the user's fingers, the location of moles or scars on the user's hand, and the shape of the user's fingernails, using at least one artificial intelligence model.

[0228] The refrigerator (1) can quickly and accurately identify a hand image from an input image of the user's hand based on the learned hand characteristics.

[0229] In one embodiment, the refrigerator (1) can determine a user characteristic based on at least one of the input image or the hand image, and transmit a notification to the user device (4) notifying access of a restricted user based on the user characteristic.

[0230] The refrigerator (1) can determine user characteristics from an input image acquired based on a camera sensor (163) or the hand image. For example, the refrigerator (1) can determine user characteristics by analyzing hand size, hand wrinkles, the ratio between the palm and fingers, hand color, etc. from the hand image. For example, the user characteristics may include at least one of the user's gender or age.

[0231] The refrigerator (1) may transmit a notification to the user device (4) notifying of the access of a restricted user based on user characteristics. For example, the refrigerator (1) may detect the access of a child user and transmit a notification to the user device (4) notifying of the access of the child user. For example, the refrigerator (1) may detect the access of an elderly user and transmit a notification to the user device (4) notifying of the access of the elderly user. In some embodiments, the refrigerator (1) may deactivate the automatic door opening function based on the user characteristics if the user is determined to be a restricted user.

[0232] According to one or more embodiments, AI inference (1302) may be performed by the refrigerator (1), the user device (4), and / or the server (5).

[0233] In this way, the refrigerator (1) of the present disclosure can detect the user's hand with high accuracy even in various environments by simultaneously using the IR camera (163a) and the RGB camera (163b). In addition, the refrigerator (1) of the present disclosure can increase the accuracy of determining the user's hand gesture by clearly identifying the structural information of the user's hand through preprocessing of the hand image. Therefore, the refrigerator (1) of the present disclosure can increase the convenience and efficiency of using the refrigerator (1) by accurately and automatically opening the door that matches the user's intention. However, since this has been described above, a redundant description thereof will be omitted.

[0234] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0235] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0236] According to embodiments of the present disclosure, a refrigerator may include at least one storage compartment, at least one door for opening and closing the at least one storage compartment, a door control module for controlling opening or closing of the at least one door, a sensor unit including a proximity sensor and a camera sensor, at least one processor, and a memory including one or more storage media for storing instructions, wherein the instructions, when individually or collectively executed by the at least one processor, may cause the refrigerator to perform operations. The above operations may include an operation of activating the camera sensor based on the identification that the user is located within a first distance from the refrigerator using the proximity sensor, an operation of acquiring an input image through the camera sensor based on the identification that the user is located within a second distance shorter than the first distance from the refrigerator using the proximity sensor, an operation of identifying a hand image corresponding to the user's hand from the input image, an operation of performing preprocessing on the hand image, an operation of determining a hand gesture of the user from the hand image, and an operation of controlling the door control module to open one of the at least one door selected based on the first gesture in response to determining that the hand gesture is a first gesture for selecting a door to be opened.

[0237] In one embodiment, the instructions, when executed individually or collectively by at least one processor, may cause the refrigerator to perform at least one of a first preprocessing step of removing a background image excluding the hand image from the input image, a second preprocessing step of performing contrast correction on the hand image to emphasize hand wrinkles, hand joints, and fingers, and a third preprocessing step of generating bounding lines or bounding boxes for thickness, length, and direction of the fingers.

[0238] In one embodiment, the camera sensor may include an infrared (IR) camera and a red-green-blue (RGB) camera. The instructions, when individually or collectively executed by at least one processor, may cause the refrigerator to synthesize a first input image acquired from the IR camera and a second input image acquired from the RGB camera to generate a third input image.

[0239] In one embodiment, the instructions, when individually or collectively executed by at least one processor, may cause the refrigerator to synthesize the first input image and the second input image by applying a first weight to the first input image and a second weight to the second input image. The first weight may increase as the ambient illuminance decreases. The second weight may increase as the ambient illuminance increases.

[0240] In one embodiment, the instructions, when executed individually or collectively by at least one processor, may cause the refrigerator to analyze the first input image, the second input image, and the third input image, respectively, and identify the hand image included in the input images by filtering the analyzed input images.

[0241] In one embodiment, the instructions, when executed individually or collectively by at least one processor, may cause the refrigerator to determine the hand gesture by inferring the intended gesture from the hand image using at least one artificial intelligence model that has learned a correlation between the hand image and the preset hand gesture.

[0242] In one embodiment, the artificial intelligence model may include a first model that infers hand orientation based on hand wrinkles, hand joints, and thumb positions, a second model that infers extended or folded fingers based on the angles between fingers and finger lengths, and a third model that infers the intended gesture based on the hand orientation and the extended or folded fingers.

[0243] In one embodiment, the instructions, when executed individually or collectively by at least one processor, may cause the refrigerator to divide the sensing area where the hand is sensed into a plurality of sub-areas based on a virtual center point, and to select a door to be opened among the at least one door based on the area where the hand is located among the plurality of sub-areas.

[0244] In one embodiment, the instructions, when executed individually or collectively by at least one processor, may cause the refrigerator to provide a notification confirming a door to be opened based on the first gesture, and to open the at least one door based on a second gesture agreeing to the door to be opened.

[0245] In one embodiment, the instructions, when executed individually or collectively by at least one processor, may cause the refrigerator to provide a notification confirming the door to be opened based on the first gesture, and to re-evaluate the hand gesture of the user based on a third gesture denying the door to be opened.

[0246] In one embodiment, the instructions, when executed individually or collectively by at least one processor, may cause the refrigerator to change a control action according to the hand gesture based on a user's settings via at least one application of the user device.

[0247] In one embodiment, the instructions, when executed individually or collectively by at least one processor, may cause the refrigerator to change a control operation according to the hand gesture based on the user's settings for the hand image identified through the camera sensor.

[0248] In one embodiment, the instructions, when individually or collectively executed by at least one processor, may cause the refrigerator to store the hand image on a server, learn hand characteristics of the user from the hand image using at least one artificial intelligence model, and identify the hand image corresponding to the hand of the user from the input image based on the hand characteristics of the user.

[0249] In one embodiment, the instructions, when executed individually or collectively by at least one processor, may cause the refrigerator to determine a user characteristic based on at least one of the input image or the hand image, and to transmit a notification to the user device notifying access of a restricted user based on the user characteristic.

[0250] In one embodiment, the instructions, when individually or collectively executed by at least one processor, may cause the refrigerator to sense the user's hand using an infrared (IR) camera and a red-green-blue (RGB) camera simultaneously, thereby increasing a hand recognition rate. The instructions, when individually or collectively executed by at least one processor, may cause the refrigerator to perform preprocessing on the hand image, thereby increasing the accuracy of judging the hand gesture. The instructions, when individually or collectively executed by at least one processor, may cause the refrigerator to judge the hand gesture using an artificial intelligence model, thereby increasing the efficiency of automatic door opening.

[0251] A method for controlling a refrigerator according to embodiments of the present disclosure may include an operation of activating a camera sensor based on identification that a user is located within a first distance from the refrigerator using a proximity sensor, an operation of acquiring an input image through the camera sensor based on identification that the user is located within a second distance shorter than the first distance from the refrigerator using the proximity sensor, an operation of identifying a hand image corresponding to a hand of the user from the input image, an operation of performing preprocessing on the hand image, an operation of determining a hand gesture of the user from the hand image, and an operation of opening one of the at least one door selected based on the gesture in response to determining that the hand gesture is a gesture for selecting a door to be opened.

[0252] In one embodiment, the operation of performing preprocessing on the hand image may include performing at least one of: a first preprocessing for removing a background image excluding the hand image from the input image; a second preprocessing for emphasizing hand wrinkles, hand joints, and fingers by performing contrast correction on the hand image; and a third preprocessing for generating a bounding line or bounding box for the thickness, length, and direction of the finger.

[0253] In one embodiment, the camera sensor may include an infrared (IR) camera and a red-green-blue (RGB) camera. The operation of identifying a hand image corresponding to the hand of the user may include an operation of generating a third input image by synthesizing a first input image acquired from the IR camera and a second input image acquired from the RGB camera.

[0254] In one embodiment, the operation of identifying a hand image corresponding to the hand of the user may include an operation of analyzing the first input image, the second input image, and the third input image, respectively, and an operation of identifying the hand image included in the input images by filtering the analyzed input images.

[0255] In one embodiment, the act of determining the user's hand gesture from the hand image may include an act of determining the hand gesture by inferring an intended gesture from the hand image using at least one artificial intelligence model that has learned a correlation between the hand image and the preset hand gesture.

[0256] In one embodiment, the artificial intelligence model may include a first model that infers hand orientation based on hand wrinkles, hand joints, and thumb positions, a second model that infers extended or folded fingers based on the angles between fingers and finger lengths, and a third model that infers the intended gesture based on the hand orientation and the extended or folded fingers.

Claims

1. In the refrigerator (1), At least one storage room (20); At least one door (300) for opening and closing the at least one storage room; A door control module (150) for controlling the opening or closing of at least one door; A sensor unit (160) including a proximity sensor and a camera sensor; At least one processor (191); and A memory comprising one or more storage media for storing instructions, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: Activating the camera sensor based on the identification that the user is located within a first distance from the refrigerator using the proximity sensor; Based on the identification that the user is located within a second distance shorter than the first distance from the refrigerator using the proximity sensor, an input image is acquired through the camera sensor, Identifying a hand image corresponding to the user's hand from the input image, Perform preprocessing on the above hand image, Determining the user's hand gesture from the above hand image, A refrigerator, wherein, in response to determining that the hand gesture is a first gesture for selecting a door to be opened, the door control module is controlled to open one of the at least one door selected based on the first gesture.

2. In paragraph 1, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: A first preprocessing step for removing a background image excluding the hand image from the input image; Second preprocessing to emphasize hand wrinkles, hand joints, and fingers by performing contrast correction on the above hand image; and Causing at least one of the third preprocessing to generate bounding lines or bounding boxes for the thickness, length, and direction of the finger; refrigerator.

3. In paragraph 1, The above camera sensor, Includes an IR (infrared) camera and an RGB (red-green-blue) camera, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: Causing a first input image acquired from the IR camera and a second input image acquired from the RGB camera to be synthesized to generate a third input image, refrigerator.

4. In paragraph 3, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: By applying a first weight to the first input image and a second weight to the second input image, the first input image and the second input image are synthesized, The above first weight increases as the ambient light level decreases, The above second weight increases as the ambient light level increases. refrigerator.

5. In paragraph 3, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: Analyzing the first input image, the second input image, and the third input image, respectively, By filtering the analyzed input image, causing the hand image included in the input image to be identified, refrigerator.

6. In paragraph 1, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: By inferring an intended gesture from the hand image using at least one artificial intelligence model that has learned a correlation between the hand image and the preset hand gesture, thereby causing the hand gesture to be judged, refrigerator.

7. In paragraph 6, At least one artificial intelligence model, A first model that infers hand orientation based on hand wrinkles, knuckles, and thumb positions; A second model that infers whether the fingers are extended or folded based on the angle between the fingers and the length of the fingers; and a third model for inferring the intention gesture based on the hand direction and the extended or folded fingers; refrigerator.

8. In paragraph 6, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: Based on the virtual center point, the sensing area where the hand is sensed is divided into multiple sub-areas, Causing to select a door to be opened among the at least one door based on the area in which the hand is located among the plurality of sub-areas; refrigerator.

9. In paragraph 1, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: Based on the first gesture above, a notification is provided to confirm the door to be opened, Causing the opening of at least one door based on a second gesture agreeing to the door to be opened; refrigerator.

10. In paragraph 1, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: Based on the first gesture above, a notification is provided to confirm the door to be opened, Causing the user's hand gesture to be re-evaluated based on the third gesture denying the door to be opened. refrigerator.

11. In paragraph 1, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: Causing to change the control action according to said hand gesture based on the user's settings through at least one application of the user device, refrigerator.

12. In paragraph 1, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: Causing to change the control operation according to the hand gesture based on the user's setting for the hand image identified through the camera sensor. refrigerator.

13. In paragraph 1, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: Save the above hand image on the server, Using at least one artificial intelligence model, learning the user's hand characteristics from the hand image, Based on the hand characteristics of the user, causing the hand image corresponding to the hand of the user to be identified from the input image, refrigerator.

14. In paragraph 1, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: Based on at least one of the input image or the hand image, determine the user characteristics, Based on the above user characteristics, causing a notification to be sent to the user device notifying the restricted user's access. refrigerator.

15. In paragraph 1, When the above instructions are individually or collectively executed by at least one processor, the refrigerator causes: By sensing the user's hand simultaneously using an IR (infrared) camera and an RGB (red-green-blue) camera, the hand recognition rate is increased. By performing preprocessing on the above hand image, the judgment accuracy of the hand gesture is increased, By judging the hand gesture using an artificial intelligence model, it causes the efficiency of automatic door opening to increase. refrigerator.

Images