Refrigerator and method for controlling same

Abstract

The present invention relates to a refrigerator and a method for controlling same. The refrigerator acquires an image by using a camera provided on an outer case of a main body, recognizes first and second objects from the acquired image, recognizes the orientation of the camera on the basis of the locations of the recognized first and second objects, adjusts the orientation of the camera on the basis of the recognized orientation of the camera, and displays, through a display unit, information about the recognized orientation of the camera.

Description

Refrigerator and its control method

[0001] The present invention relates to a refrigerator including a camera for acquiring images of food being stored in and shipped from a storage room, and a method for controlling the same.

[0002] A refrigerator is typically a device that stores food at low temperatures in a storage compartment enclosed by a door. Refrigerators cool the storage compartment using the cold air generated through heat exchange with the refrigerant circulating in the refrigeration cycle, thereby maintaining the food in the storage compartment in optimal condition.

[0003] Recent refrigerators incorporate a variety of convenient features to enhance user convenience. Examples of these convenient features include a display unit that uses a camera to capture images of food stored in the storage compartment and then generates and displays a list of items based on the captured images; a door with a transparent window that allows the interior of the storage compartment to be viewed without opening the door; and a door opening device that automatically opens the door based on user input.

[0004] When a camera is installed inside a storage room, there is a problem in that dew forms on the camera, making it difficult to accurately capture images of food, and even when an image of food is captured through the camera, information about the food cannot be accurately recognized from the captured image.

[0005] When the camera is installed on the outer surface of the main body, there was a problem in that the camera's position was changed by external force, making it impossible to accurately obtain images of food being received or delivered from the storage room.

[0006] One aspect of the disclosed invention provides a refrigerator and a control method thereof that recognizes first and second objects for recognizing the attitude of a camera in an image acquired by a camera and adjusts the attitude of the camera based on position information of the recognized first and second objects.

[0007] Another aspect of the disclosed invention provides a refrigerator and a control method thereof that acquires images in real time based on reception in a manual mode and recognizes and then displays camera attitude adjustment information based on the acquired images.

[0008] A refrigerator according to one aspect includes a main body including a storage compartment formed by an inner surface; a camera provided on the outer surface of the main body; a posture adjustment unit for adjusting a posture of the camera; and a processor for recognizing objects in an image acquired by the camera, recognizing an object corresponding to the inner surface among the recognized objects as a first object, recognizing a posture of the camera based on the recognized first object, and controlling the posture adjustment unit based on the recognized posture of the camera.

[0009] The storage compartment of a refrigerator according to one aspect includes a refrigerating compartment provided in the upper portion of the main body and a freezer compartment provided in the lower portion of the main body. The field of view of the camera of the refrigerator according to one aspect is the entrance of the refrigerating compartment. The internal structure recognized as the first object of the refrigerator according to one aspect includes the internal structure provided at the entrance of the refrigerating compartment.

[0010] A refrigerator according to one aspect includes first and second doors for opening or closing a refrigerator compartment, and third and fourth doors for opening or closing a freezer compartment. A camera of the refrigerator according to one aspect is provided in an area between a first area where the first door contacts and a second area where the second door contacts, among the areas of the external surface.

[0011] A processor of a refrigerator according to one aspect controls a camera when the first and second doors are open and the third and fourth doors are closed, recognizes an object corresponding to the upper surface of the third and fourth doors among the recognized objects as a second object, and controls a posture adjustment unit based on the recognized second object.

[0012] The refrigerator's posture adjustment unit according to one aspect includes a first adjustment unit that adjusts a first posture of the camera by rotating the camera around a first axis; a second adjustment unit that adjusts a second posture of the camera by rotating the camera around a second axis; and a third adjustment unit that adjusts a third posture of the camera by rotating the camera around a third axis.

[0013] The first and second axes of the refrigerator according to one aspect are axes that are perpendicular to the horizontal plane. The third axis is an axis that is perpendicular to the first and second axes and to the vertical plane.

[0014] A processor of a refrigerator according to one aspect recognizes a first position of a first object existing in a first border among the borders of an image acquired by a camera and a second position of the first object existing in a second border, recognizes a height of the first position and a height of the second position based on a third border among the borders, recognizes a difference value between the height of the first position and the height of the second position, and controls a first adjustment unit based on the recognized difference value and the reference value if the recognized difference value is greater than a reference value.

[0015] A processor of a refrigerator according to one aspect recognizes the position of the inner surface of the lower surface among the inner surfaces of the storage room in the first object, recognizes a height corresponding to the position of the inner surface of the lower surface recognized based on a third border, and controls a second adjustment unit based on the recognized height and the reference height when the recognized height is lower than a reference height.

[0016] The processor of the refrigerator according to one aspect recognizes the position of the second object based on the third border, and if the recognized position of the second object is outside the reference range, controls the third adjustment unit based on the position of the second object and the reference range.

[0017] The reference range of the refrigerator according to one aspect includes a first reference position on the first side based on the center position of the third frame and a second reference position on the second side based on the center position of the third frame.

[0018] According to one aspect, the refrigerator further includes a memory for storing information about a reference object. The processor of the refrigerator according to one aspect recognizes first and second objects among the recognized objects based on the information about the reference object.

[0019] According to one aspect, the processor of the refrigerator controls the second control unit when the first object is not recognized in the image acquired by the camera, and terminates the control of the second control unit when the first object is recognized by the control of the second control unit.

[0020] According to one aspect, the processor of the refrigerator controls the third control unit when the second object is not recognized in the image acquired by the camera, and terminates the control of the third control unit when the second object is recognized by the control of the third control unit.

[0021] The refrigerator according to one aspect further includes a display unit. The processor of the refrigerator according to one aspect controls the display unit to display information about the recognized camera position.

[0022] According to one aspect, the refrigerator further includes an input unit for receiving user input. The processor of the refrigerator according to one aspect deactivates control of the posture adjustment unit when manual mode input information is received through the input unit, and controls the display unit to display information about the posture of the camera recognized in real time.

[0023] The processor of the refrigerator according to one aspect recognizes information on food being received or delivered from the storage room based on the image acquired by the camera, if the recognized camera posture is the reference posture, and updates the list of food stored in the storage room based on the recognized food information.

[0024] A method of controlling a refrigerator according to another aspect comprises: obtaining an image using a camera provided on the outer surface of the main body, recognizing objects in the obtained image, recognizing an object corresponding to the inner surface of the main body among the recognized objects as a first object, recognizing a posture of the camera based on the recognized first object, adjusting the posture of the camera based on the recognized posture of the camera, and displaying information about the recognized posture of the camera through a display unit.

[0025] Obtaining an image using a camera includes acquiring an image when the first and second doors of the first storage room are open and the third and fourth doors of the second storage room are closed. Adjusting the attitude of the camera includes recognizing an object corresponding to the upper surface of the third and fourth doors among the recognized objects as a second object, and controlling an attitude adjustment unit based on the recognized second object.

[0026] Adjusting the attitude of the camera includes recognizing a first position of a first object existing in a first border among the borders of an image acquired by the camera and a second position of the first object existing in a second border, recognizing a height of the first position and a height of the second position based on a third border among the borders, recognizing a difference value between the height of the first position and the height of the second position, and rotating the camera about the first axis based on the recognized difference value and the reference value when the recognized difference value is greater than or equal to a reference value.

[0027] Adjusting the camera's pose includes recognizing the position of the inner image of the lower surface among the inner images of the first storage room in the first object, recognizing the height of the position of the inner image of the lower surface recognized based on the third border, and rotating the camera around the second axis based on the recognized height and the reference height if the recognized height is lower than the reference height. The second axis is perpendicular to the first axis and the horizontal plane.

[0028] Adjusting the camera's pose includes recognizing the position of the second object relative to the third frame, and if the recognized position of the second object is outside the reference range, rotating the camera relative to the third axis based on the position of the second object and the reference range. The third axis is perpendicular to the first and second axes and a vertical plane. The reference range includes the first reference position on the first side and the second reference position on the second side relative to the center position of the third frame.

[0029] According to the disclosed invention, the disclosed invention can stably acquire images of food being stored in and shipped from a refrigerator by adjusting the posture of a camera provided on the outer surface of the main body to a reference posture and then acquiring images of food being stored in and shipped from the refrigerator, and can accurately recognize information about the food through the acquired images of the food.

[0030] The disclosed invention can reduce the time and cost for camera maintenance because the camera's posture can be adjusted to a reference posture without user intervention through an automatic mode.

[0031] The disclosed invention can provide users with reliable information on food by improving the recognition accuracy of information on food stored in and shipped from a refrigerator.

[0032] The disclosed invention can display a list of food stored in a refrigerator through a display unit of the refrigerator, thereby allowing a user to easily and quickly recognize food stored in the refrigerator without opening the door, thereby improving user convenience and reducing energy consumption due to a reduction in the number of times the door is opened.

[0033] The disclosed invention can improve the safety of a refrigerator, enhance the quality and marketability of the refrigerator, and further secure the competitiveness of the refrigerator.

[0034] FIG. 1 is a front view showing a refrigerator with its door closed according to an embodiment of the present disclosure.

[0035] FIG. 2 is a perspective view showing an open door of a refrigerator according to an embodiment of the present disclosure.

[0036] FIG. 3 is an exemplary diagram of a camera and a posture control unit provided in a refrigerator according to an embodiment of the present disclosure.

[0037] Figure 4 is a control configuration diagram of a refrigerator according to an embodiment of the present disclosure.

[0038] FIG. 5 is an example diagram of conditions of a refrigerator when acquiring an image for adjusting the posture of a camera provided in the refrigerator according to an embodiment of the present disclosure.

[0039] FIG. 6 is an example of an image acquired by a camera provided in a refrigerator according to an embodiment of the present disclosure.

[0040] FIGS. 7A, 7B and 7C are exemplary views of the posture adjustment of the first axis of a camera provided in a refrigerator according to an embodiment of the present disclosure.

[0041] FIGS. 8A, 8B and 8C are exemplary views of the posture adjustment for the second axis of a camera provided in a refrigerator according to an embodiment of the present disclosure.

[0042] FIGS. 9a, 9b, 9c, 9d, and 9e are exemplary views of the third axis posture adjustment of a camera provided in a refrigerator according to an embodiment of the present disclosure.

[0043] Figure 10 is a control flowchart of a refrigerator according to an embodiment of the present disclosure.

[0044] FIG. 11 is a control flowchart for adjusting the posture of a camera for the first axis provided in a refrigerator according to an embodiment of the present disclosure.

[0045] FIG. 12 is a control flowchart for posture adjustment of a second axis of a camera provided in a refrigerator according to an embodiment of the present disclosure.

[0046] FIG. 13 is a control flowchart for posture adjustment of a third axis of a camera provided in a refrigerator according to an embodiment of the present disclosure.

[0047] It should be understood that the various embodiments of the present disclosure and the terminology used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, but rather to encompass various modifications, equivalents, or alternatives of the embodiments.

[0048] In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0049] The singular form of a noun corresponding to an item may include one or more of said items, unless the relevant context clearly indicates otherwise.

[0050] In this disclosure, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0051] The term "and / or" includes any combination of a plurality of related described elements or any one of a plurality of related described elements.

[0052] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish one component from another and do not qualify the components in any other respect (e.g., importance or order).

[0053] In addition, terms such as 'front', 'rear', 'top', 'bottom', 'side', 'left', 'right', 'upper', and 'lower' used in the present disclosure are defined based on the drawings, and the shape and position of each component are not limited by these terms.

[0054] Terms such as "include" or "have" are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the present disclosure, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0055] When a component is said to be “connected,” “coupled,” “supported,” or “in contact with” another component, this includes not only cases where the components are directly connected, coupled, supported, or in contact, but also cases where the components are indirectly connected, coupled, supported, or in contact through a third component.

[0056] When we say that a component is "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where there is another component between the two components.

[0057] A refrigerator according to one embodiment may include a body.

[0058] The "body" may include an inner case, an outer case disposed on the outside of the inner case, and an insulating material provided between the inner case and the outer case.

[0059] The "inner case" may include at least one of a case, a plate, a panel, or a liner forming a storage compartment. The inner case may be formed as a single body, or may be formed by assembling a plurality of plates. The "outer case" may form the outer appearance of the main body, and may be joined to the outer side of the inner case so that insulation is placed between the inner case and the outer case.

[0060] "Insulation" can insulate the interior and exterior of a storage room so that the temperature inside the storage room can be maintained at a set temperature without being affected by the external environment. In one embodiment, the insulation can include foam insulation. The foam insulation can be formed by injecting and foaming urethane foam, a mixture of polyurethane and a foaming agent, between the inner and outer layers.

[0061] In one embodiment, the insulation may include a vacuum insulation material in addition to the foam insulation, or the insulation may consist solely of the vacuum insulation material instead of the foam insulation. The vacuum insulation material may include a core material and an outer shell material that accommodates the core material and seals the interior under a vacuum or near-vacuum pressure. However, the insulation material is not limited to the foam insulation or vacuum insulation material described above, and may include various materials that can be used for insulation.

[0062] A "storage room" may include a space defined by an interior wall. The storage room may further include an interior wall defining a corresponding space. The storage room may store various items, such as food, medicine, and cosmetics, and the storage room may be configured to be open on at least one side for the entry and exit of items.

[0063] A refrigerator may include one or more storage compartments. When a refrigerator includes two or more storage compartments, each compartment may have a different purpose and be maintained at different temperatures. To achieve this, each storage compartment may be separated from the others by a partition wall containing insulation.

[0064] The storage room may be designed to maintain an appropriate temperature range depending on its intended use, and may include a "refrigerator," a "freezer," or a "variable temperature room," which are distinguished by their intended use and / or temperature range. A refrigerator may be maintained at a temperature appropriate for refrigerating items, and a freezer may be maintained at a temperature appropriate for freezing items. "Refrigeration" may mean cooling items to a temperature that does not freeze them, and for example, a refrigerator may be maintained at a temperature ranging from 0 degrees Celsius to +7 degrees Celsius. "Freezing" may mean cooling items to freeze them or keep them frozen, and for example, a freezer may be maintained at a temperature ranging from -20 degrees Celsius to -1 degree Celsius. A variable temperature room may be used as either a refrigerator or a freezer, at the user's option or not.

[0065] In addition to names such as "refrigerator," "freezer," and "variable temperature room," a storage room may also be called by various other names such as "vegetable room," "fresh room," "cooling room," and "ice room." The terms "refrigerator," "freezer," and "variable temperature room" used hereinafter should be understood to encompass storage rooms having corresponding uses and temperature ranges.

[0066] In one embodiment, the refrigerator may include at least one door configured to open and close an open side of a storage compartment. The door may be configured to open and close one or more storage compartments, or a single door may be configured to open and close multiple storage compartments. The door may be installed on the front of the main body in a pivotal or sliding manner.

[0067] The "door" may be configured to seal the storage compartment when the door is closed. The door may include insulation, similar to the body, to insulate the storage compartment when the door is closed.

[0068] According to one embodiment, the door may include a door outer panel forming the front of the door, a door inner panel forming the back of the door and facing the storage compartment, an upper cap, a lower cap, and door insulation provided on the interior of these.

[0069] The door inner panel may be provided with a gasket that seals the storage compartment by contacting the front of the body when the door is closed. The door inner panel may include a dyke that protrudes rearward to accommodate a door basket for storing items.

[0070] In one embodiment, the door may include a door body and a front panel detachably coupled to the front side of the door body and forming the front of the door. The door body may include a door outer panel forming the front of the door body, a door inner panel forming the rear of the door body and facing the storage compartment, an upper cap, a lower cap, and door insulation provided inside these.

[0071] Depending on the arrangement of the door and storage compartment, refrigerators can be classified into French door type, side-by-side type, bottom mounted freezer (BMF), top mounted freezer (TMF), or single-door refrigerator.

[0072] According to one embodiment, the refrigerator may include a cold air supply device configured to supply cold air to the storage compartment.

[0073] A "cold air supply device" may include a system of machines, devices, electronic devices and / or combinations thereof that can generate cold air and guide the cold air to cool a storage room.

[0074] In one embodiment, the cold air supply device can generate cold air through a refrigeration cycle that includes the processes of compression, condensation, expansion, and evaporation of a refrigerant. To this end, the cold air supply device can include a refrigeration cycle device having a compressor, a condenser, an expansion device, and an evaporator capable of driving the refrigeration cycle. In one embodiment, the cold air supply device can include a semiconductor, such as a thermoelectric element. The thermoelectric element can cool a storage compartment through heat generation and cooling through the Peltier effect.

[0075] According to one embodiment, the refrigerator may include a machine room in which at least some components belonging to the cold air supply device are arranged.

[0076] The "machine room" may be designed to be partitioned and insulated from the storage room to prevent heat generated by components placed within the machine room from being transferred to the storage room. The interior of the machine room may be configured to be in communication with the exterior of the main body to dissipate heat from components placed within the machine room.

[0077] In one embodiment, the refrigerator may include a dispenser provided on the door to provide water and / or ice. The dispenser may be provided on the door so that it is accessible to a user without having to open the door.

[0078] In one embodiment, a refrigerator may include an ice-making device configured to produce ice. The ice-making device may include an ice-making tray configured to store water, an ice-separating device configured to separate ice from the ice-making tray, and an ice bucket configured to store ice produced in the ice-making tray.

[0079] According to one embodiment, the refrigerator may include a control unit for controlling the refrigerator.

[0080] The "control unit" may include a memory that stores or memorizes a program and / or data for controlling the refrigerator, and a processor that outputs a control signal for controlling a cold air supply device, etc. according to the program and / or data memorized in the memory.

[0081] Memory stores or records various information, data, commands, programs, etc. necessary for the operation of the refrigerator. Memory can store temporary data generated during the generation of control signals for controlling components within the refrigerator. Memory may include at least one of volatile memory and non-volatile memory, or a combination thereof.

[0082] The processor controls the overall operation of the refrigerator. The processor can control the components of the refrigerator by executing programs stored in memory. The processor may include a separate NPU that performs the operations of an artificial intelligence model. The processor may also include a central processing unit (CPU), a graphics processing unit (GPU), or the like. The processor may generate control signals to control the operation of the cooling system. For example, the processor may receive temperature information about the storage compartment from a temperature sensor and generate a cooling control signal to control the operation of the cooling system based on the temperature information.

[0083] Additionally, the processor may process user input of the user interface and control the operation of the user interface based on programs and / or data stored / stored in the memory. The user interface may be provided using an input interface and an output interface. The processor may receive user input from the user interface. Additionally, the processor may transmit display control signals and image data to the user interface for displaying an image on the user interface in response to the user input.

[0084] The processor and memory may be provided as a single unit or separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one subprocessor. The memory may include one or more memories.

[0085] In one embodiment, a refrigerator may include a processor and memory that control all components within the refrigerator, and may include multiple processors and multiple memories that individually control the components within the refrigerator. For example, the refrigerator may include a processor and memory that control the operation of a cooling device based on the output of a temperature sensor. Additionally, the refrigerator may separately include a processor and memory that control the operation of a user interface based on user input.

[0086] The communication module can communicate with external devices, such as servers, mobile devices, and other home appliances, via a nearby access point (AP). The AP can connect the local area network (LAN) where the refrigerator or user device is connected to the wide area network (WAN) where the server is connected. The refrigerator or user device can then connect to the server via the WAN.

[0087] The input interface may include keys, a touchscreen, a microphone, etc. The input interface may receive user input and transmit it to the processor.

[0088] The output interface may include a display unit and a speaker, etc. The output interface may output various notifications, messages, information, etc. generated by the processor.

[0089]

[0090] Hereinafter, refrigerators according to various embodiments will be specifically described with reference to the attached drawings.

[0091] Fig. 1 is an exemplary drawing of the exterior of a refrigerator according to an embodiment, Fig. 2 is an exemplary drawing of the front of a refrigerator according to an embodiment, and Fig. 3 is an exemplary drawing of a camera and a posture adjustment unit provided in a refrigerator according to an embodiment.

[0092] As illustrated in FIG. 1, a refrigerator (1) may include a body (10) that forms the exterior of the refrigerator and has an opening, a door (20) provided on the body (10) and opening and closing the opening of the body (10), and a user interface (120) for interfacing with a user.

[0093] The main body (10) may include an outer body (11) forming the exterior of the refrigerator.

[0094] The outer case (11) may be formed to have the shape of a box with an open front. The outer case (11) may include the upper surface, lower surface, left surface, right surface, and rear surface of the refrigerator (1).

[0095] The main body (10) may include a top table (12) provided on the upper part of the main body (10). The top table (12) may be coupled to the upper surface of the outer body (11).

[0096] The top table (12) can cover various electrical components. A storage space for various electrical components can be provided on the inside of the top table (12). For example, a camera (100) and a posture adjustment unit (110, see FIG. 3), which will be described later, can be provided on the top table (12).

[0097] There may be one or more doors (20). The doors (20) may be provided rotatably on the main body (10).

[0098] This embodiment illustrates a refrigerator having four doors (21, 22, 23, 24) as an example.

[0099] A user interface (120) is provided on at least one of the doors and can receive user input and display operation information of the refrigerator.

[0100] The user interface (120) may include an input interface and an output interface. The user interface (120) may include a touch screen.

[0101] As shown in FIG. 2, the refrigerator may include a storage compartment (30) provided inside the main body (10).

[0102] The main body (10) may include an inner case (13) forming a storage room (30).

[0103] The inner case (13) may be of an open-fronted type. The inner case (13) may be provided on the inner side of the outer case (11). The inner wall of the inner case (13) may form the inner wall of the storage chamber (30).

[0104] An insulating material (not shown) that can prevent cold air from leaking from the storage room (30) may be provided between the outer case (11) and the inner case (13) of the main body (10).

[0105] The insulation can be foamed between the inner case (13) and the outer case (11), thereby bonding the inner case (13) and the outer case (11) to each other. The insulation can insulate the outer case (11) and the inner case (13) from each other. That is, the insulation can prevent heat exchange between the inside of the storage chamber (30) and the outside of the main body (10), thereby improving the cooling efficiency inside the storage chamber (30).

[0106] Insulation materials that can be used include urethane foam insulation, expanded polystyrene insulation, and vacuum insulation panels. However, the present invention is not limited to these, and the main body insulation can be composed of various materials.

[0107] There may be one or more storage rooms (30).

[0108] When there are multiple storage rooms, the multiple storage rooms can be divided by partitions. Storage rooms divided vertically can be divided by a first partition arranged horizontally within the storage room, and storage rooms divided left and right can be divided by a second partition arranged vertically within the storage room.

[0109] The storage room (30) may include a refrigerator that is maintained at approximately 0 to 5 degrees Celsius to refrigerate food, and may include a freezer that is maintained at approximately -30 to 0 degrees Celsius to freeze food.

[0110] The refrigerator compartment may be provided at the top of the main body, and there may be one or more refrigerator compartments. The freezer compartment may be provided at the bottom of the main body, and there may be one or more refrigerator compartments.

[0111] This embodiment describes a refrigerator having one refrigerating chamber (31) provided in the upper part of the main body and two freezing chambers (32, 33) provided in the lower part of the main body as an example.

[0112] The refrigerator and freezer of this embodiment can be partitioned by a first partition (14) arranged horizontally inside the main body, and the two freezer compartments can be partitioned by a second partition (15) arranged vertically inside the main body.

[0113] In this embodiment, the refrigerator may be the first storage room (31). One freezer may be the second storage room (32) or the first freezer. The other freezer may be the third storage room (33) or the second freezer.

[0114] The first, second, and third storage rooms (31, 32, 33) may be provided with at least one shelf (34) for placing food and at least one drawer (35) for storing food.

[0115] Each storage room (30) is provided with one or more doors (20) and can be opened or closed by one or more doors (20).

[0116] For example, the first storage room (31) may be provided with first and second doors (21, 22). The first storage room (31) may be opened by at least one of the first door (21) and the second door (22), and may be closed by the first and second doors (21, 22).

[0117] The first door (21) and the second door (22) can be arranged parallel to each other in the horizontal direction (X direction).

[0118] The first door (21) may be provided to open and close the left side of the first storage room (31), and the second door (22) may be provided to open and close the right side of the first storage room (31).

[0119] The freezer can be opened or closed by the third and fourth doors (23, 24). The third door (23) and the fourth door (24) can be arranged parallel to each other in the horizontal direction (X direction).

[0120] If the freezer is divided into two, a third door (23) may be provided in the first freezer (32), and a fourth door (24) may be provided in the second freezer (32).

[0121] The first freezer (32) can be opened or closed by the third door (23). The second freezer (33) can be opened or closed by the fourth door (24).

[0122] The third door (23) can be arranged parallel to the first door (21) in a vertical direction (Z).

[0123] The fourth door (24) can be arranged parallel to the second door (22) in a vertical direction (Z). In addition, the fourth door (24) can be arranged parallel to the third door (23) in a horizontal direction (X).

[0124] Each door (20) can be provided to be rotatable on the main body (10).

[0125] A refrigerator (1) may include a hinge connecting a main body (10) and a door (20). The hinge may be provided so that the door (20) can rotate relative to the main body (10). The hinge may be fixed to the main body (10). Specifically, the hinge may be coupled to an outer case (11).

[0126] For example, the refrigerator (1) may include a pair of upper door hinges that are coupled to the upper portion of the main body (10) and rotatably support the first door (21) and the second door (22), respectively. In addition, the refrigerator (1) may include a pair of lower door hinges that are coupled to the lower portion of the main body (10) and rotatably support the third door (23) and the fourth door (24), respectively. In addition, for example, the refrigerator (1) may include a pair of middle hinges that are coupled to the middle portion of the main body (10) (specifically, the first partition (14)) and rotatably support the first door (21), the second door (22), the third door (23), and the fourth door (24), respectively.

[0127] A door basket (25) for storing food may be provided on the inner surface of each door (20). A door gasket (26) may be provided on the inner surface of the door (20) to seal the gap between the door (20) and the main body (10) and prevent cold air from leaking from the storage compartment (30).

[0128] The outer surface of the door (20) may form a part of the exterior of the refrigerator (1). When the door (20) is closed, the outer surface of the door (20) may form at least a part of the front exterior of the refrigerator (1). When the door (20) is closed, the inner surface of the door (20) may face the interior of the storage compartment (30). The inner surface of the door (20) referred to herein means one side of the door (20) that faces the storage compartment (30) when the door (20) closes the storage compartment (30). In addition, the outer surface of the door (20) referred to herein means the other side opposite to the inner surface of the door (20) that faces the storage compartment (30) when the door (20) closes the storage compartment (30), and means the front of the door (20) that is visible when the refrigerator (1) is viewed from the front.

[0129] Each door (20) may be provided with a handle.

[0130] It is also possible to provide a door open / close sensor for detecting an open or closed state for each door (20).

[0131] One of the multiple doors may also be provided with a dispenser (not shown) for dispensing water or ice directly to the outside.

[0132] The refrigerator may further include a cooling device for cooling the storage compartment.

[0133] The cooling device can generate cold air using a cooling cycle and supply the generated cold air to a storage room (30). The cooling device can generate cold air using a cooling cycle that compresses, condenses, expands, and evaporates a refrigerant. For example, the cooling device can include a compressor, a condenser, an expansion valve, an evaporator, a blower fan, and the like. The cold air generated by the cooling device can be supplied to the storage room (30) through a cold air supply duct formed at the rear portion of the inner case (13).

[0134] A refrigerator (1) according to an embodiment of the present disclosure may be a direct-cooling refrigerator or a direct-cooling refrigerator.

[0135] The refrigerator (1) may further include a camera (100) that acquires images of food being stored in or shipped from at least one storage room (20).

[0136] The camera (100) may include a three-dimensional space recognition camera such as a TOF (Time Of Flight) camera, a stereo camera, etc.

[0137] The camera (100) may include an image sensor. The image sensor may include a CCD or CMOS image sensor.

[0138] The camera (100) can obtain images of the surroundings and interior (13) of the storage room.

[0139] The camera (100) can have a field of view of the inner chamber (13) provided at the entrance of the refrigerator (31).

[0140] The camera (100) can also have a field of view of the inner chamber (13) provided at the entrance of the refrigerator (31) and the freezer (32, 33).

[0141] The camera (100) can have a field of view of the upper surface of the inner case (13) provided at the entrance of the refrigerator (31) and the third and fourth doors (23, 24) of the freezer (32, 33).

[0142] The camera (100) can be placed in the center of the upper front of the main body (10).

[0143] The camera (100) is provided on the outer case (11) of the main body (10), and can be placed in the center of the front of the outer case (11).

[0144] The camera (100) may be placed at the center of the top table (12). The center of the top table (12) may include the center of the X-axis of the front of the main body of the refrigerator.

[0145] The camera (100) is installed on the outer surface (11) of the main body (10), and may be installed in an area between the first area (da1) where the first door (21) comes into contact and the second area (da2) where the second door (22) comes into contact among the areas of the outer surface (11).

[0146] As shown in FIG. 3, the camera (100) may be provided with a posture adjustment unit (110: 111, 112, 113) for adjusting the posture of the camera (100).

[0147] The posture adjustment unit (110) can be provided on the top table (12). The posture adjustment unit (110) can also be provided inside the top table (12).

[0148] The posture adjustment unit (110) may include a first adjustment unit (111) for adjusting the rotation of the camera (100), a second adjustment unit (112) for adjusting the up-down posture of the camera (100), and a third adjustment unit (113) for adjusting the left-right posture of the camera (100).

[0149] The first adjustment unit (111) can adjust the camera's posture based on the camera's first axis (vertical axis, y-axis). The first adjustment unit (111) can perform roll control. The first adjustment unit (111) can include a first motor.

[0150] The second adjustment unit (112) can adjust the camera's attitude based on the camera's second axis (horizontal axis, x-axis). The second adjustment unit (112) can perform pitch (or tilt) control. The second adjustment unit (112) can include a second motor.

[0151] The third adjustment unit (113) can adjust the camera's posture based on the camera's third axis (vertical axis, z-axis). The third adjustment unit (113) can perform yaw (or pan) control. The third adjustment unit (113) can include a second motor.

[0152] The first and second axes can be perpendicular to each other in the horizontal plane.

[0153] The first and third axes can be perpendicular to each other in a vertical plane.

[0154] The second and third axes can be perpendicular to each other in a vertical plane.

[0155] The configuration of the refrigerator (1) described above with reference to FIGS. 1 and 2 is merely an example for explaining a refrigerator according to the concept of the present disclosure, and the concept of the present disclosure is not limited thereto.

[0156] FIG. 4 is a control configuration diagram of a refrigerator according to an embodiment, which is described with reference to FIGS. 5, 6, 7a, 7b, 7c, 8a, 8b, 8c, 9a, 9b, 9c, 9d, and 9e.

[0157] FIG. 5 is an example diagram of conditions of a refrigerator when acquiring an image for adjusting the posture of a camera provided in a refrigerator according to an embodiment of the present disclosure, and FIG. 6 is an example diagram of an image acquired by a camera provided in a refrigerator according to an embodiment of the present disclosure.

[0158] FIGS. 7a, 7b, and 7c are exemplary views of posture adjustment for a first axis of a camera provided in a refrigerator according to an embodiment of the present disclosure, FIGS. 8a, 8b, and 8c are exemplary views of posture adjustment for a second axis of a camera provided in a refrigerator according to an embodiment of the present disclosure, and FIGS. 9a, 9b, 9c, 9d, and 9e are exemplary views of posture adjustment for a third axis of a camera provided in a refrigerator according to an embodiment of the present disclosure.

[0159] The refrigerator (1) may include a camera (100), a posture adjustment unit (110), a user interface (120), a processor (130), and a memory (131), and may further include an open / close detection unit (125).

[0160] The camera (100) can obtain an image of the entrance to the storage room (30).

[0161] The camera (100) has a field of view of the entrance of the first storage room (31) among the multiple storage rooms (30).

[0162] The camera (100) can obtain an image of the entrance of the first storage room (31) among the multiple storage rooms.

[0163] The camera (100) can be activated or deactivated in response to a control command from the processor (130).

[0164] The camera (100) can transmit the image acquired when activated to the processor (130).

[0165] The camera (100) can acquire images in real time in response to control commands from the processor (130) and transmit the images acquired in real time to the processor (130).

[0166] The posture adjustment unit (110) can adjust the posture of the camera (100) in response to a control command of the processor (130).

[0167] The attitude adjustment unit (110) can adjust the attitude of the camera (100) based on the first axis (vertical axis, y-axis) in response to the control command of the processor (130), can adjust the attitude of the camera (100) based on the second axis (vertical axis, z-axis) in response to the control command of the processor (130), and can adjust the attitude of the camera (100) based on the third axis (horizontal axis, x-axis) in response to the control command of the processor (130).

[0168] The attitude adjustment unit (110) may include a first adjustment unit (111) that adjusts the attitude of the camera (100) based on a first axis, a second adjustment unit (112) that adjusts the attitude of the camera (100) based on a second axis, and a third adjustment unit (113) that adjusts the attitude of the camera (100) based on a third axis.

[0169] The first adjusting unit (111) can adjust the left-right distortion of the image by adjusting the rotation of the camera (100). The first adjusting unit (111) can perform roll control. The first adjusting unit (111) can include a first motor.

[0170] The first adjusting unit (111) rotates by a first reference angle based on the control command of the processor (130), and can rotate in the forward or reverse direction.

[0171] The second adjustment unit (112) can adjust the vertical position of the camera (100). The second adjustment unit (112) can perform pitch (or tilt) control. The second adjustment unit (112) can include a second motor.

[0172] The second adjusting unit (112) rotates by a second reference angle based on the control command of the processor (130), and can rotate in the forward or reverse direction.

[0173] The third adjustment unit (113) can adjust the left and right posture of the camera (100). The third adjustment unit (113) can perform yaw (or pan) control. The third adjustment unit (113) can include a third motor.

[0174] The third adjustment unit (113) rotates by a third reference angle based on the control command of the processor (130), and can rotate in the forward or reverse direction.

[0175] The first, second and third reference angles may be the same or different.

[0176] The user interface (120) can receive user input and output information related to the operation of the refrigerator.

[0177] The user interface (120) may include an input interface (121) for receiving user input and an output interface (122) for outputting information related to the operation of the refrigerator.

[0178] The input interface (121) can receive user input and transmit it to the processor (130).

[0179] The input interface (121) can receive any one of the posture adjustment mode, automatic mode and manual mode and transmit information about the received mode to the processor.

[0180] The attitude adjustment mode is a mode for adjusting the attitude of the camera, and is a mode for acquiring images of the entrance to the storage compartment of the refrigerator.

[0181] Auto mode is a mode to automatically adjust the camera's posture.

[0182] Manual mode is a mode for manually adjusting the camera's pose based on user input.

[0183] The input interface (121) can also receive adjustment commands from the first, second, and third adjustment units (111, 112, 1113) to adjust the roll, pitch, and yaw of the camera (110) when performing manual mode.

[0184] The adjustment commands of the first, second, and third adjustment units (111, 112, 1113) may include information corresponding to the rotation angle of each motor.

[0185] The adjustment commands of the first, second, and third adjustment units (111, 112, 1113) may include input signals of each button corresponding to the first, second, and third adjustment units (111, 112, 1113).

[0186] The input interface (121) can receive an image acquisition command of the camera when performing the camera attitude adjustment mode.

[0187] The input interface (121) can receive a command to output a food list and a command to end output of the food list.

[0188] The input interface (121) may include hardware devices such as keys, buttons, switches, pedals, mouse, trackball, microphone, etc.

[0189] The input interface (121) may include a GUI (Graphical User Interface), i.e., a software device, such as a touch pad. The touch pad may be implemented as a touch screen panel (TSP) and may form a mutual layer structure with the display unit (122a).

[0190] The output interface (122) may include a display unit (122a) and a speaker (122b).

[0191] The display unit (122a) can display information related to the status or operation of the refrigerator (1) based on the control command of the processor (130) and can display information to guide the user's input.

[0192] The display unit (122a) can display information entered into the input interface (121).

[0193] The display unit (122a) can display guidance information for performing the camera attitude adjustment mode.

[0194] The display unit (122a) can display information about the recognized camera posture.

[0195] Information about the camera's attitude may include information on whether attitude adjustment is required for each axis, information on whether attitude adjustment is not required for each axis, and information on attitude adjustment for each axis. The attitude adjustment information for each axis may include adjustment values ​​for each axis. The adjustment values ​​for each axis of the camera may include adjustment values ​​for yaw, roll, and pitch.

[0196] The display unit (122a) can display information about the recognized camera posture in real time.

[0197] The display unit (122a) can display the current mode, for example, posture adjustment mode, automatic mode, and manual mode.

[0198] The display unit (122a) includes a plurality of seven segments.

[0199] The display unit (122a) may be provided as a liquid crystal display (LCD), a digital light processing (DLP) panel, a plasma display panel, an electroluminescence (EL) panel, an electrophoretic display (EPD) panel, an electrochromic display (ECD) panel, a light emitting diode (LED) panel, or an organic light emitting diode (OLED) panel, but is not limited thereto.

[0200] The speaker (122b) can output information related to the status or operation of the refrigerator (1) as a guide sound based on the control command of the processor (130), and can output information for guiding the user's input as a guide sound.

[0201] The speaker (122b) can output guidance information in voice for performing the camera's attitude adjustment mode.

[0202] The speaker (122b) can output information about the recognized camera posture as a voice.

[0203] The speaker (122b) can output information about the need for camera attitude adjustment as a notification sound, or output information about the need for camera attitude adjustment as a notification sound.

[0204] The speaker (122b) can also output the axis-specific adjustment values ​​as a voice when the camera's attitude needs to be adjusted.

[0205] The open / close detection unit (125) detects the state of the door. The state of the door may include an open state and a closed state.

[0206] The opening / closing detection unit (125) may include a first opening / closing sensor that detects the open and closed states of the first door (21), a second opening / closing sensor that detects the open and closed states of the second door (22), a third opening / closing sensor that detects the open and closed states of the third door (23), and a fourth opening / closing sensor that detects the open and closed states of the fourth door (24).

[0207] The first open / close sensor can transmit an open signal corresponding to the open state of the first door (21) and a close signal corresponding to the closed state of the first door (21) to the processor (130).

[0208] The second open / close sensor can transmit an open signal corresponding to the open state of the second door (22) and a close signal corresponding to the closed state of the second door (22) to the processor (130).

[0209] The third open / close sensor can transmit an open signal corresponding to the open state of the third door (23) and a close signal corresponding to the closed state of the third door (23) to the processor (130).

[0210] The fourth open / close sensor can transmit an open signal corresponding to the open state of the fourth door (24) and a close signal corresponding to the closed state of the fourth door (24) to the processor (130).

[0211] Each open / close sensor may include, but is not limited to, a micro switch, a limit switch, a magnetic switch, a reed switch, a toggle switch, a tact switch, etc.

[0212] Each open / close sensor may include, but is not limited to, a light sensor, an ultrasonic sensor, or an impact sensor.

[0213] The processor (130) controls the overall operation of the refrigerator (1).

[0214] The processor (130) can control the display unit (122a) to output the food list stored in the memory (131) based on receiving a command to output the food list from the input interface (121).

[0215] The processor (130) can terminate the display of the food list displayed on the display unit (122a) based on receiving a command to terminate the output of the food list from the input interface (121).

[0216] When the processor (130) receives an opening signal of the first door (21) from the opening / closing detection unit (125), it controls the activation of the camera (100) and can determine whether to bring in or take out food from the first storage room (31) based on the image received from the camera (100).

[0217] When the processor (130) receives an opening signal of the second door (22) from the opening / closing detection unit (125), it controls the activation of the camera (100) and can determine whether to bring in or take out food from the first storage room (31) based on the image received from the camera (100).

[0218] The image received from the camera (100) may be an image received in real time.

[0219] The image received from the camera (100) may be a video.

[0220] The image received from the camera (100) may include an image of the surrounding area of ​​the entrance to the first storage room (31).

[0221] The image received from the camera (100) may include an internal image provided at the entrance of the first storage room (31).

[0222] When it is determined that food is to be stored in the first storage room (31), the processor (130) recognizes information about the food stored in the first storage room (31) based on the received image and updates the food list stored in the memory (131) based on the information about the recognized food.

[0223] That is, the processor (130) can control the memory (131) to add information on food stored in the first storage room (31) to the food list and store the food list with the added food information.

[0224] When it is determined that food is to be shipped to the first storage room (31), the processor (130) recognizes information about food to be shipped to the first storage room (31) based on the received image and updates the food list stored in the memory (131) based on the information about the recognized food.

[0225] That is, the processor (130) can control the memory (131) to delete information about food shipped to the first storage room (31) from the food list and store the food list with the food information deleted.

[0226] The processor (130) controls the activation of the camera (100) when an opening signal of the third door (23) is received from the opening / closing detection unit (125), and can also determine whether to bring in or take out food from the second storage room (32) based on the image received from the camera (100).

[0227] When it is determined that food is to be stored in the second storage room (32), the processor (130) can recognize information about the food stored in the second storage room (32) based on the received image and update the food list stored in the memory (131) based on the information about the recognized food.

[0228] When it is determined that food is to be shipped to the second storage room (32), the processor (130) can recognize information about food to be shipped to the second storage room (32) based on the received image and update the food list stored in the memory (131) based on the information about the recognized food.

[0229] The processor (130) controls the activation of the camera (100) when an opening signal of the fourth door (22) is received from the opening / closing detection unit (125), and can also determine whether to bring in or take out food from the third storage room (33) based on the image received from the camera (100).

[0230] When it is determined that food is to be stored in the third storage room (33), the processor (130) can recognize information about the food stored in the third storage room (33) based on the received image and update the food list stored in the memory (131) based on the information about the recognized food.

[0231] When it is determined that food is to be shipped to the third storage room (33), the processor (130) can recognize information about food to be shipped to the third storage room (33) based on the received image and update the food list stored in the memory (131) based on the information about the recognized food.

[0232] The processor (130) can update the food list for each storage room based on the decision to receive and ship food for each storage room.

[0233] The processor (130) can periodically determine whether the attitude of the camera (100) needs to be adjusted and automatically adjust the attitude of the camera (100) in response to the need for attitude adjustment of the camera (100).

[0234] The processor (130) determines whether the state of the refrigerator satisfies the conditions for adjusting the position of the camera (100) based on the door-specific signal detected by the open / close detection unit (121), and if it is determined that the state of the refrigerator satisfies the conditions for adjusting the position of the camera, it can transmit an image acquisition command to the camera (100).

[0235] As shown in Fig. 5, the conditions for posture adjustment may include conditions in which the first and second doors (21, 22) are in an open or closed state and the third and fourth doors (23, 24) are in a closed state.

[0236] If the processor (130) determines that the state of the refrigerator satisfies the conditions for adjusting the camera's attitude, it can determine whether or not adjustment of the camera's attitude is necessary based on the image received from the camera (100).

[0237] The processor (130) can control the performance of the attitude adjustment mode based on what is determined to be the need for attitude adjustment of the camera (100).

[0238] The processor (130) can control the output of guidance information for posture adjustment of the camera (100) based on receiving a posture adjustment mode from the input interface (121).

[0239] For example, when the processor (130) receives a posture adjustment mode from the input interface (121), it is also possible to control the display unit (122a) to display guidance information requesting the opening and closing of the first and second doors (21, 22) and the closing of the third and fourth doors (23, 24).

[0240] The processor (130) can also control the speaker (122b) to output guidance information requesting the opening and closing of the first and second doors (21, 22) and the closing of the third and fourth doors (23, 24) in voice when the posture adjustment mode is received from the input interface (121).

[0241] The processor (130) controls the output of guidance information, and when an image acquisition command is received from the input interface (121), it controls the activation of the camera (100) and can also transmit the image acquisition command to the activated camera (100).

[0242] When the processor (130) performs the camera attitude adjustment mode, it recognizes the edges of the received image based on the image received from the camera (100).

[0243] As illustrated in FIG. 6, the border may include a left border (t1), a right border (t2), a lower border (t3), and an upper border (t4).

[0244] The processor (130) recognizes objects in the image received from the camera (100).

[0245] The objects may include the first and second doors (21, 22), the third and fourth doors (23, 24), the inner case of the first storage room (13), the door basket (25) and the door gasket (26).

[0246] The processor (130) recognizes the first object (a) and the second object (b: b1, b2) required for camera posture recognition among the recognized objects based on the information of the reference object stored in the memory (131).

[0247] Information of the reference object stored in the memory (131) may include information of the first object corresponding to the inner surface of the entrance of the first storage room and information of the second object corresponding to the upper surface of the third and fourth doors.

[0248] Information of the first object may include image information, size information, shape information, color information, etc. of the first object.

[0249] Information of the second object may include image information, size information, shape information, and color information of the second object.

[0250] This embodiment acquires an image of the interior of the first storage chamber using a camera installed at the upper center of the main body. Therefore, in the acquired image, the interior of the first storage chamber may have a crescent shape, and in the acquired image, the upper surfaces of the third and fourth doors (23, 24) may have a curved rectangular shape or a trapezoidal shape.

[0251] The processor (130) can recognize the positions of the left and right ends of the first object (a) within the image and control the first adjustment unit (111) based on the difference in the positions of the left and right ends of the first object (a).

[0252] As illustrated in FIG. 7a, the processor (130) can recognize a first object (a) existing at the left edge (t1) of the received image, recognize a first position (a1) of the first object (a), and recognize a first object (a) existing at the right edge (t2) of the received image, recognize a second position (a2) of the first object (a).

[0253] The first position (a1) of the first object (a) may include an upper position on the inner surface of the first side of the first storage room in the image.

[0254] The second position (a2) of the first object (a) may include an upper position on the inner surface of the second side of the first storage room in the image.

[0255] The first position (a1) of the first object (a) and the second position (a2) of the first object (a) may include pixel coordinate values ​​of the image.

[0256] The pixel coordinate values ​​of the first position (a1) and the second position (a2) may include the coordinate values ​​of the pixels with respect to the vertical axis of the image.

[0257] As illustrated in FIG. 7b, the processor (130) recognizes the height of the first position (a1) of the first object based on the lower edge (t3) of the image, recognizes the height of the second position (a2) of the first object based on the lower edge (t3) of the image, and can recognize a first difference value (d1) between the height of the recognized first position (a1) and the height of the recognized second position (a2).

[0258] If the recognized first difference value (d1) is greater than or equal to the reference value, the processor (130) can control the first adjustment unit (111) based on the recognized first difference value and the reference value.

[0259] As illustrated in FIG. 7c, the processor (130) can control the first adjustment unit (111) to rotate the camera around the first axis if the recognized first difference value (d1) is greater than or equal to a reference value. In other words, the processor (130) can control the roll of the camera (100).

[0260] The processor (130) can control the first adjustment unit (111) so that the recognized first difference value (d1) becomes less than the reference value. In this case, the processor (130) can recognize the first adjustment value (ad1) based on the recognized first difference value (d1) and the reference value (s1) and control the first adjustment unit (111) based on the recognized first adjustment value (ad1).

[0261] Recognizing the first adjustment value (ad1) may include recognizing a value (f1) resulting from a difference between the recognized first difference value (d1) and the reference value (s1), and recognizing the first adjustment value (ad1) based on the recognized value (f1).

[0262] Here, the first adjustment value may be equal to or greater than the recognized value.

[0263] f1 = |d1-s1|, ad1 >= f1

[0264] The processor (130) can also control the first adjustment unit (111) so that the recognized first difference value (d1) becomes zero (0).

[0265] The processor (130) can also control the first adjustment unit (111) so that the recognized first difference value (d1) reaches the reference value.

[0266] The processor (130) recognizes a pixel difference value between the first position and the second position based on the pixel coordinate value of the first position of the first object and the pixel coordinate value of the second position of the first object, and if the recognized pixel difference value is greater than or equal to a reference value, the first adjustment unit can be controlled based on the recognized difference value and the reference value.

[0267] Here, the reference value can include approximately 60 pixels.

[0268] If the recognized first difference value (d1) is less than the reference value, the processor (130) may determine that the camera's left-right misalignment has not occurred and may deactivate the control of the first adjustment unit (111). If the recognized first difference value (d1) is less than the reference value, the processor (130) may determine that the camera (110) is in a reference pose with respect to the first axis.

[0269] The processor (130) can recognize an area corresponding to the inner surface of the lower surface of the first storage room in the first object (a) and control the second adjustment unit (112) based on the recognized area.

[0270] As illustrated in FIG. 8a, the processor (130) can recognize an area (ac) corresponding to the inner surface of the lower surface of the first storage room in the first object, recognize the uppermost part of the recognized area (ac), and recognize the recognized uppermost part as a third position (a3).

[0271] The processor (130) can also recognize the location where the boundary line of the second object (b) meets the recognized area (ac) as the third location (a3).

[0272] The boundary line of the second object may be a line formed by the meeting of the third and fourth doors when the third and fourth doors are closed, and may be a line perpendicular to the line forming the area (ac) corresponding to the inner surface of the lower surface of the first storage room.

[0273] The third position (a3) ​​of the first object (a) may include pixel coordinate values ​​for the vertical axis of the image.

[0274] As illustrated in FIG. 8b, the processor (130) can recognize the height of the third position (a3) ​​of the first object based on the lower edge (t3) of the image, and recognize whether the height of the recognized third position (a3) ​​is less than the reference height (h1).

[0275] If the height of the recognized third position (a3) ​​is less than the reference height (h1), the processor (130) can control the second adjustment unit (112) based on the height of the recognized third position (a3) ​​and the reference height (h1).

[0276] As illustrated in FIG. 8c, the processor can control the second adjusting unit (112) to rotate the camera (100) around the second axis when the height of the recognized third position (a3) ​​is less than the reference height (h1). In other words, the processor (130) can control the pitch of the camera (100).

[0277] If the height of the recognized third position (a3) ​​is less than the reference height (h1), the processor (130) can control the second adjustment unit (112) so that the height of the recognized third position (a3) ​​is greater than or equal to the reference height (h1). In this case, if the height of the recognized third position (a3) ​​is less than the reference height (h1), the processor (130) can obtain a second difference value (d2) between the height of the recognized third position (a3) ​​and the reference height (h1), recognize a second adjustment value (ad2) based on the obtained second difference value (d2), and control the second adjustment unit (112) based on the recognized second adjustment value (ad2).

[0278] Here, the second adjustment value may be equal to or greater than the second difference value.

[0279] ad2 >= d1

[0280] The processor (130) can also control the second adjustment unit (112) so that the height of the third position (a3) ​​reaches the reference height (h1).

[0281] The processor (130) can also control the second adjustment unit so that the recognized second difference value (d2) becomes zero (0).

[0282] The processor (130) obtains a pixel value corresponding to the height of the third position based on the pixel coordinate value of the third position of the first object, obtains a reference coordinate value corresponding to the reference height, recognizes a second adjustment value based on a pixel difference value between the obtained pixel value corresponding to the height of the third position and the reference pixel value, and controls the second adjustment unit (112) based on the recognized second adjustment value.

[0283] Here, the pixel value corresponding to the reference height can contain approximately 20 pixels.

[0284] If the height of the recognized third position (a3) ​​is higher than the reference height (h1), the processor (130) can determine the vertical position of the camera as normal and deactivate the control of the second adjustment unit (112).

[0285] The processor (130) can determine that the camera (110) is in a reference pose with respect to the second axis if the height of the recognized third position (a3) ​​is greater than or equal to the reference height (h1).

[0286] When the processor (130) recognizes the attitude of the camera with respect to the second axis, if the area (ac) corresponding to the inner surface of the lower surface of the first storage room is not recognized in the image acquired by the camera (100), the processor (130) can control the second adjusting unit (112) to rotate the camera with respect to the second axis.

[0287] The processor (130) controls the rotation of the second motor so that the camera rotates downward about the second axis, and can control the rotation by a second reference angle.

[0288] The processor (130) can acquire an image through the camera at each point in time when the second motor rotates by the second reference angle, and can recognize an area (ac) corresponding to the inner surface of the lower surface of the first storage room from the image acquired by the camera.

[0289] The processor (130) can control the rotation of the second motor until an area (ac) corresponding to the inner surface of the lower surface of the first storage room is recognized in the image acquired by the camera.

[0290] As illustrated in FIG. 9a, the processor (130) recognizes the upper surface (b2) of the fourth door and the upper surface (b1) of the third door in the second object (b), recognizes the boundary line (bc) between the upper surface (b2) of the fourth door and the upper surface (b1) of the third door, recognizes the position of the recognized boundary line (bc), and controls the third adjustment unit (113) based on the position of the recognized boundary line (bc).

[0291] The boundary line (bc) can be the center of the third and fourth doors or the center of the body.

[0292] Recognizing the location of the boundary line (bc) may include recognizing the pixel coordinate values ​​of the boundary line (bc). The pixel coordinate values ​​of the boundary line (bc) may include pixel coordinate values ​​with respect to the horizontal axis of the image.

[0293] The processor (130) can recognize the reference range (rr) based on the lower edge (t3) of the image.

[0294] The processor (130) recognizes a reference range (rr) based on the horizontal axis of the image, and can recognize whether the position of the boundary line (bc) exists within the reference range (rr).

[0295] The reference range may include a first reference position (r1) to a second reference position (r2) on the horizontal axis, and may include a center position (r0) between the first reference position (r1) and the second reference position (r2).

[0296] As illustrated in FIG. 9b, the reference range (rr) may include a first pixel value (r1) on the first side of the horizontal axis from the center position (r0) to a second pixel value (r2) on the second side of the horizontal axis. For example, the center position (r0) may be the center position of the left and right sides of the image. The reference range (rr) may have a range of 26 pixels to the left and 62 pixels to the right from the center position of the image.

[0297] As illustrated in FIG. 9c, the processor (130) can control the third adjustment unit (113) to rotate the camera (100) around the third axis of the camera when the position of the boundary line (bc) deviates from the reference range (rr). In other words, the processor (130) can control the yaw of the camera (100).

[0298] The processor (130) can determine the left and right posture of the camera as normal if the position of the boundary line (bc) is within the reference range (rr) and deactivate the control of the third adjustment unit (113).

[0299] The processor (130) can determine that the camera (110) is in a reference pose with respect to the third axis if the position of the boundary line (bc) exists within the reference range (rr).

[0300] If the position of the boundary line (bc) is outside the reference range (rr), the processor (130) can control the third adjustment unit (113) so that the position of the boundary line (bc) is within the reference range (rr).

[0301] When the position of the boundary line (bc) is outside the reference range (rr), the processor (130) recognizes whether the position of the boundary line (bc) is located to the left or right of the center position (r0) within the image.

[0302] As illustrated in FIG. 9d, if the processor (130) determines that the position of the boundary line (bc) is located to the left of the center position (r0), the processor (130) can control the third adjustment unit (113) based on the first reference position (r1) and the position of the boundary line (bc).

[0303] In this case, the processor (130) can recognize a third difference value (d3) between the first reference position (r1) and the position of the boundary line (bc), recognize a third adjustment value (ad3) based on the recognized third difference value (d3), and control the third adjustment unit (113) based on the recognized third adjustment value (ad3).

[0304] The third adjustment value (ad3) may be equal to or greater than the third difference value (d3).

[0305] The processor (130) can also control the third adjusting unit (113) so that the position of the boundary line (bc) becomes the center position (r0).

[0306] The processor (130) can also control the third adjusting unit (113) so that the position of the boundary line (bc) becomes the first reference position (r1).

[0307] The processor (130) can also control the third adjusting unit (113) so that the position of the boundary line (bc) becomes the second reference position (r2).

[0308] The processor (130) can recognize a third difference value based on the pixel coordinate value of the boundary line (bc) and the pixel coordinate value of the first reference position, and can recognize a third adjustment value based on the recognized third difference value.

[0309] As illustrated in FIG. 9e, if the processor (130) determines that the position of the boundary line (bc) is located to the right of the center position (r0), the processor (130) can control the third adjustment unit (113) based on the second reference position (r2) and the position of the boundary line (bc).

[0310] In this case, the processor (130) can obtain a fourth difference value (d4) between the second reference position (r2) and the position of the boundary line (bc), recognize a third adjustment value (ad3) based on the obtained second difference value (d4), and control the third adjustment unit (113) based on the recognized third adjustment value (ad3).

[0311] The third adjustment value (ad3) may be equal to or greater than the fourth difference value (d4).

[0312] The processor (130) can also control the third adjusting unit (113) so that the position of the boundary line (bc) becomes the center position (r0).

[0313] The processor (130) can also control the third adjusting unit (113) so that the position of the boundary line (bc) becomes the second reference position (r2).

[0314] The processor (130) can also control the third adjusting unit (113) so that the position of the boundary line (bc) becomes the first reference position (r1).

[0315] The processor (130) can recognize a fourth difference value based on the pixel coordinate value of the boundary line (bc) and the pixel coordinate value of the second reference position, and can recognize a third adjustment value based on the recognized fourth difference value.

[0316] When the processor (130) recognizes the attitude of the camera with respect to the third axis, if the second object is not recognized in the image acquired by the camera, the processor (130) can control the third adjusting unit (113) to rotate the camera with respect to the third axis.

[0317] When the processor (130) controls the third adjustment unit (113), it rotates the third motor in the forward direction at a third reference angle, and when the rotation of the third motor is completed, it can determine whether the second object is recognized based on the image acquired by the camera (100).

[0318] If the second object is not recognized even after the third motor is rotated by the third reference angle, the processor (130) can control the third motor again to rotate the third motor by the third reference angle.

[0319] That is, the processor (130) can control the third motor to rotate at a third reference angle until the second object is recognized in the image acquired by the camera (100).

[0320] The processor (130) controls the rotation of the third motor in the reverse direction when it is determined that the second object is not recognized in the image after the third motor rotates to the maximum angle in the forward direction, and can control the rotation of the third motor by the third reference angle.

[0321] When the processor (130) controls the third motor to rotate in the reverse direction, it is possible to control the third motor to return to the initial angle at the time of starting the forward rotation, and then, when the third motor reaches the initial angle, control the third motor to rotate in the reverse direction by a third reference angle from the initial angle.

[0322] The processor (130) can control the rotation of the third motor in the reverse direction, recognize a second object in an image acquired by the camera (100), and control the rotation of the third motor until the second object is recognized in the image acquired by the camera (100).

[0323]

[0324] The processor (130) can control the display unit (122a) to display posture adjustment information when the manual mode is received from the input interface (121).

[0325] The processor (130) can control the camera so that images are acquired in real time during the execution of the manual mode, recognize camera attitude adjustment information based on images received from the camera, and control the display unit (122a) to display the recognized camera attitude adjustment information.

[0326] When the processor (130) recognizes posture adjustment information, it recognizes the first and second objects in the received image, recognizes the heights of the first, second, and third positions of the recognized first object, and determines whether posture adjustment to the first axis is necessary based on the height difference between the first and second positions, and if it determines that posture adjustment to the first axis is necessary, it recognizes the first adjustment value of the first adjustment unit (111) based on the height difference value and the reference value between the first and second positions, and controls the display unit (122a) to display the recognized first adjustment value as posture adjustment information.

[0327] When the processor (130) recognizes the posture adjustment information, it determines whether posture adjustment to the second axis is necessary based on the height of the third position and the reference height, and if it determines that posture adjustment to the second axis is necessary, it recognizes the second adjustment value of the second adjustment unit based on the height of the third position and the reference height, and controls the display unit (122a) to display the recognized second adjustment value as posture adjustment information.

[0328] When the processor (130) recognizes the posture adjustment information, it recognizes the position of the boundary line in the second object and determines whether posture adjustment to the third axis is necessary based on the position of the recognized boundary line and the reference range, and if it determines that posture adjustment to the third axis is necessary, it recognizes the third adjustment value of the third adjustment unit (113) based on the position of the recognized boundary line and the reference range, and controls the display unit (122a) to display the recognized third adjustment value as posture adjustment information.

[0329] When the processor (130) determines that the camera's posture is a reference posture during the execution of the manual mode, it can control the display unit (122a) to display information on the completion of the camera's posture adjustment.

[0330] When determining whether the pose of the camera is a reference pose, the processor (130) can determine whether the poses of all of the first, second, and third axes are reference poses.

[0331] When the processor (130) performs manual mode, the processor (130) can control the first motor to rotate by a first reference angle based on one reception of an adjustment command from the first adjustment unit (111) through the input interface (121).

[0332] When the processor (130) performs manual mode, the second motor can be controlled to rotate by a second reference angle based on one reception of an adjustment command from the second adjustment unit (112) through the input interface (121).

[0333] When the processor (130) performs manual mode, the processor (130) can control the third motor to rotate by a first reference angle based on one reception of an adjustment command from the third adjustment unit (113) through the input interface (121).

[0334] The processor (130) can control the display unit (122a) to acquire an image through the camera after controlling any one of the first, second, and third motors, recognize the camera's attitude adjustment information based on the acquired image, and display the recognized camera's attitude adjustment information.

[0335] The processor (130) may be implemented as a memory (not shown) that stores data regarding an algorithm for controlling the operation of components within the refrigerator or a program that reproduces the algorithm, and a processor (not shown) that performs the aforementioned operations using the data stored in the memory. In this case, the memory (131) and the processor (130) may each be implemented as separate chips. Alternatively, the memory (131) and the processor (130) may be implemented as a single chip.

[0336] The processor (130) can perform the above-described operation using data stored in the memory (131).

[0337] The processor (130) may include hardware such as a CPU or memory, and software such as a control program. For example, the processor (340) may include one or more processor chips that perform the aforementioned operations using an algorithm for controlling the operation of components within the refrigerator, at least one memory that stores program-type data, and data stored in the at least one memory, or may include one or more processing cores.

[0338] The processor (130) may include a separate NPU that performs the operation of the artificial intelligence model, and may include a graphics-only processor (GPU), etc.

[0339] The memory (131) can store information on a reference object. The information on the reference object can include image information necessary to recognize the first and second objects.

[0340] Information of the reference object stored in the memory (131) can be updated through learning.

[0341] The memory (131) can store reference information for a reference value, a reference height, and a reference range.

[0342] The memory (131) can store information about the first, second, and third reference angles.

[0343] The memory (131) can store identification information and location information of each storage room door.

[0344] The memory (131) can store identification information of an opening / closing sensor corresponding to each of a plurality of doors.

[0345] The memory (131) can store data for an algorithm for controlling the operation of components in the refrigerator or a program that reproduces the algorithm.

[0346] The memory (131) may be implemented as at least one of a non-volatile memory element such as a cache, a ROM (Read Only Memory), a PROM (Programmable ROM), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), and a flash memory, a volatile memory element such as a RAM (Random Access Memory), or a storage medium such as a hard disk drive (HDD) or a CD-ROM, but is not limited thereto.

[0347] The memory (131) may include one or more memory chips or one or more memory blocks.

[0348] At least one component may be added or deleted to correspond to the performance of the components of the refrigerator illustrated in Fig. 4. Furthermore, it will be readily apparent to those skilled in the art that the relative positions of the components may be altered to correspond to the performance or structure of the refrigerator.

[0349] Meanwhile, each component illustrated in FIG. 4 refers to software and / or hardware components such as a Field Programmable Gate Array (FPGA) and an Application Specific Integrated Circuit (ASIC).

[0350] Figure 10 is a control flowchart of a refrigerator according to an embodiment of the present disclosure.

[0351] When the refrigerator receives a posture adjustment mode from the input interface (121) (201), it acquires an image through the camera (100) (202).

[0352] When the refrigerator receives a posture adjustment mode from the input interface (121), it can output guidance information requesting the opening and closing of the first and second doors (21, 22) and the closing of the third and fourth doors (23, 24) through at least one of the display unit and the speaker.

[0353] After the refrigerator controls the output of guidance information, when an image acquisition command is received from the input interface (121), it controls the activation of the camera (100) and can also acquire an image through the activated camera (100).

[0354] The refrigerator can also automatically enter posture adjustment mode when it determines that the first and second doors are open and the third and fourth doors are closed.

[0355] The refrigerator recognizes the image borders based on the image acquired from the camera. Here, the image borders may include a left border (t1), a right border (t2), a lower border (t3), and an upper border (t4).

[0356] The refrigerator recognizes objects in images captured by the camera (100).

[0357] The objects may include the first and second doors (21, 22), the third and fourth doors (23, 24), the inner case of the first storage room (13), the door basket (25) and the door gasket (26).

[0358] The refrigerator recognizes the first object and the second object (203) required for camera posture recognition among the recognized objects based on the information of the reference object stored in the memory (131).

[0359] The refrigerator can recognize the positions of the first and second objects within the image (204), and identify whether camera posture adjustment is necessary based on the positions of the recognized first and second objects and reference information (205).

[0360] When the refrigerator identifies that the camera requires attitude adjustment, it identifies whether the attitude adjustment mode is automatic mode (206).

[0361] Identifying whether the posture adjustment mode is an automatic mode may include identifying whether input information for a manual mode has been received via the input interface (121).

[0362] That is, the refrigerator can identify the posture adjustment mode as the automatic mode based on the fact that input information of the manual mode is not received through the input interface (121), and can identify the posture adjustment mode as the manual mode based on the fact that input information of the manual mode is received through the input interface (121).

[0363] When the posture adjustment mode is identified as the automatic mode, the refrigerator can recognize posture adjustment information based on the positions and reference information of the first and second objects (207) and control the posture adjustment unit based on the recognized posture adjustment information (208).

[0364] When the posture adjustment mode is identified as manual mode, the refrigerator can recognize posture adjustment information based on the positions and reference information of the first and second objects, and output the recognized posture adjustment information through the display unit (209).

[0365] When the refrigerator is in manual mode, guidance information for adjusting the camera's position may be displayed. The guidance information may include information guiding the user to manually adjust the camera's position.

[0366] The refrigerator controls the camera (100) to acquire images in real time while performing manual mode, recognizes camera posture adjustment information in real time based on the images acquired by the camera (100), and displays the camera posture adjustment information recognized in real time through the display unit (122a).

[0367] When the refrigerator recognizes posture adjustment information while performing manual mode, it recognizes the first and second objects in the received image, recognizes the heights of the first, second and third positions of the recognized first object, and determines whether posture adjustment to the first axis is necessary based on the height difference between the first and second positions, and if it determines that posture adjustment to the first axis is necessary, it recognizes the first adjustment value of the first adjustment unit (111) based on the height difference value and the reference value between the first and second positions, and can display the recognized first adjustment value as posture adjustment information through the display unit (122a).

[0368] When the refrigerator recognizes posture adjustment information while performing manual mode, it determines whether posture adjustment to the second axis is necessary based on the height of the third position and the reference height, and if it determines that posture adjustment to the second axis is necessary, it recognizes the second adjustment value of the second adjustment unit based on the height of the third position and the reference height, and can display the recognized second adjustment value as posture adjustment information through the display unit (122a).

[0369] When the refrigerator recognizes posture adjustment information while performing manual mode, it recognizes the position of the boundary line in the second object and determines whether posture adjustment to the third axis is necessary based on the position of the recognized boundary line and the reference range, and if it determines that posture adjustment to the third axis is necessary, it recognizes the third adjustment value of the third adjustment unit (113) based on the position of the recognized boundary line and the reference range, and can display the recognized third adjustment value as posture adjustment information through the display unit (122a).

[0370] When the refrigerator determines that the postures of all of the first, second, and third axes of the camera are the reference postures during the manual mode, it can display information on the completion of the posture adjustment of the camera through the display unit (122a).

[0371] FIG. 11 is a control flowchart for adjusting the posture of the first axis of a camera provided in a refrigerator according to an embodiment of the present disclosure (see FIGS. 7a, 7b, and 7c).

[0372] The refrigerator obtains images of the entrance to the storage room through a camera (211).

[0373] The refrigerator recognizes objects in the acquired image and recognizes the first object required for camera posture adjustment among the recognized objects (212).

[0374] The refrigerator can recognize a first object (a) existing on the left edge (t1) of the image, recognize a first position (a1) of the first object (a), and recognize a first object (a) existing on the right edge (t2) of the received image, recognize a second position (a2) of the first object (a) (213).

[0375] The first position (a1) of the first object (a) may include an upper position on the inner surface of the first side of the first storage room in the image.

[0376] The second position (a2) of the first object (a) may include an upper position on the inner surface of the second side of the first storage room in the image.

[0377] The refrigerator can recognize the height of the first position (a1) of the first object based on the lower edge (t3) of the image, recognize the height of the second position (a2) of the first object based on the lower edge (t3) of the image, and recognize the first difference value (d1) between the height of the recognized first position (a1) and the height of the recognized second position (a2) (214).

[0378] The refrigerator can determine whether the image is distorted left and right by comparing the recognized first difference value (d1) with the reference value (215) (216). Here, the distortion of the image left and right may be the distortion of the camera left and right.

[0379] If the first difference value (d1) is greater than or equal to the reference value, the refrigerator determines that the camera has been misaligned left and right, and if the first difference value (d1) is less than or equal to the reference value, the camera's position with respect to the first axis can be determined to be the reference position.

[0380] When the refrigerator determines that the camera is misaligned left and right, it recognizes the first adjustment value based on the recognized first difference value and the reference value (217), and controls the first adjustment unit (111) based on the recognized first adjustment value, thereby controlling the roll of the camera (218).

[0381] For example, the refrigerator can control the first adjustment unit (111) so that the recognized first difference value (d1) becomes less than the reference value.

[0382] As another example, the refrigerator may include recognizing a value (f1) based on the difference between the recognized first difference value (d1) and the reference value (s1) and recognizing a first adjustment value (ad1) based on the recognized value (f1). Here, the first adjustment value may be equal to or greater than the recognized value (f1).

[0383] As another example, the refrigerator can also control the first adjustment unit (111) so that the recognized first difference value (d1) becomes zero (0).

[0384] As another example, the refrigerator can also control the first adjustment unit (111) so that the recognized first difference value (d1) reaches the reference value.

[0385] Here, the reference value can include approximately 60 pixels.

[0386] The refrigerator can disable the control of the first adjustment unit (111) if it is determined that no left-right distortion of the camera has occurred.

[0387] FIG. 12 is a control flowchart for posture adjustment of a second axis of a camera provided in a refrigerator according to an embodiment of the present disclosure (see FIGS. 8a, 8b, and 8c).

[0388] The refrigerator obtains images of the entrance to the storage room through a camera (241).

[0389] The refrigerator recognizes objects in the acquired image and recognizes the first object required for camera posture adjustment among the recognized objects (242).

[0390] The refrigerator can recognize an area (ac) corresponding to the inner surface of the lower surface of the first storage room in the first object, recognize the top of the recognized area (ac), and recognize the recognized top as a third position (a3) ​​(243).

[0391] The refrigerator can determine whether it has successfully recognized the area (ac) corresponding to the inner surface of the lower surface of the first storage compartment in the first object or the third location (a3) ​​(244).

[0392] If the refrigerator determines that it has failed to recognize the area (ac) or the third position (a3), it rotates the second motor to control the pitch of the camera (245).

[0393] The refrigerator can acquire an image of the entrance to the storage room through the camera while performing pitch control of the camera and determine whether an area (ac) or a third position (a3) ​​is recognized in the acquired image.

[0394] The refrigerator can control the pitch of the camera until an area (ac) or a third position (a3) ​​is recognized in the image acquired by the camera.

[0395] When the refrigerator controls the pitch of the camera, it rotates the second motor by a second reference angle, and each time it rotates by the second reference angle, it can recognize an area (ac) or a third position (a3) ​​in the image acquired by the camera.

[0396] If it is determined that the refrigerator has successfully recognized the area (ac) and the third position (a3), the height of the third position (a3) ​​of the first object can be recognized based on the lower border (t3) of the image (246).

[0397] The third position (a3) ​​of the first object (a) may include pixel coordinate values ​​for the vertical axis of the image.

[0398] The refrigerator compares the height of the recognized third position (a3) ​​with the reference height (h1) (247), and can determine whether the image is upside-down based on the comparison result (248).

[0399] Comparing the height of the recognized third position (a3) ​​with the reference height (h1) may include recognizing whether the height of the recognized third position (a3) ​​is less than the reference height (h1).

[0400] The vertical distortion of the image may include the vertical distortion of the camera.

[0401] The refrigerator can determine that the camera's vertical displacement has occurred if the height of the recognized third position (a3) ​​is less than the reference height (h1), and can determine that the camera's posture with respect to the second axis is the reference posture if the height of the recognized third position (a3) ​​is greater than the reference height (h1).

[0402] When it is determined that the camera is misaligned up and down, the refrigerator recognizes a second adjustment value based on the height of the recognized third position (a3) ​​and the reference height (h1) (249), and controls the second adjustment unit (112) based on the recognized second adjustment value, thereby controlling the pitch of the camera (250).

[0403] The refrigerator can control the second adjustment unit (112) to rotate the camera (100) around the second axis when the height of the recognized third position (a3) ​​is less than the reference height (h1).

[0404] If the height of the recognized third position (a3) ​​is less than the reference height (h1), the refrigerator can control the second adjustment unit (112) so that the height of the recognized third position (a3) ​​is greater than or equal to the reference height (h1).

[0405] For example, if the height of the recognized third position (a3) ​​is less than the reference height (h1), the refrigerator can obtain a second difference value (d2) between the height of the recognized third position (a3) ​​and the reference height (h1), recognize a second adjustment value (ad2) based on the obtained second difference value (d2), and control the second adjustment unit (112) based on the recognized second adjustment value (ad2).

[0406] Here, the second adjustment value may be equal to or greater than the second difference value.

[0407] As another example, the refrigerator can also control the second adjustment unit (112) so that the height of the third position (a3) ​​reaches the reference height (h1).

[0408] As another example, the refrigerator can also control the second adjustment unit so that the recognized second difference value (d2) becomes zero (0).

[0409] Here, the pixel value corresponding to the reference height can contain approximately 20 pixels.

[0410] If the height of the recognized third position (a3) ​​is higher than the reference height (h1), the refrigerator can determine the vertical position of the camera as normal and deactivate the control of the second adjustment unit (112).

[0411] FIG. 13 is a control flowchart for posture adjustment of a third axis of a camera provided in a refrigerator according to an embodiment of the present disclosure (see FIGS. 9a, 9b, 9c, 9d, and 9e).

[0412] The refrigerator obtains an image of the entrance to the storage room through a camera (100) (221).

[0413] The refrigerator recognizes objects in the acquired image and recognizes a second object among the recognized objects that is required for adjusting the camera's posture (222).

[0414] The refrigerator can recognize the upper surface (b2) of the fourth door and the upper surface (b1) of the third door in the second object (b) and can recognize the boundary line (bc) between the upper surface (b2) of the fourth door and the upper surface (b1) of the third door (223).

[0415] The refrigerator can determine whether the recognition of the boundary (bc) has been successful (224).

[0416] The refrigerator can control the camera's yaw if it is determined that the recognition of the boundary line (bc) has failed (225).

[0417] Controlling the yaw of the camera may include rotating a third motor by a third reference angle.

[0418] The refrigerator can acquire images using the camera whenever the third motor rotates at a third reference angle and determine whether boundary recognition was successful in the acquired images.

[0419] The refrigerator can control the rotation of the third motor until it successfully recognizes the boundary line.

[0420] If the refrigerator determines that it has successfully recognized the boundary line, it can recognize the location of the recognized boundary line (bc).

[0421] The boundary line (bc) can be the center of the third and fourth doors or the center of the body.

[0422] Recognizing the location of the boundary line (bc) may include recognizing the pixel coordinate values ​​of the boundary line (bc). The pixel coordinate values ​​of the boundary line (bc) may include pixel coordinate values ​​with respect to the horizontal axis of the image.

[0423] The refrigerator can recognize the reference range (rr) based on the lower edge (t3) of the image (226).

[0424] The refrigerator recognizes the reference range (rr) based on the horizontal axis of the image, and can recognize whether the position of the boundary line (bc) exists within the reference range (rr).

[0425] The reference range may include a first reference position (r1) to a second reference position (r2) on the horizontal axis, and may include a center position (r0) between the first reference position (r1) and the second reference position (r2).

[0426] The reference range (rr) may include a first pixel value (r1) on the first side of the horizontal axis from the center position (r0) to a second pixel value (r2) on the second side of the horizontal axis. For example, the center position (r0) may be the center position of the left and right sides of the image. The reference range (rr) may have a range of 26 pixels to the left and 62 pixels to the right from the center position of the image.

[0427] The refrigerator can determine whether the image is aligned left and right by comparing the position of the boundary line (bc) with the reference range (rr) (228). The left and right alignment of the image may include the left and right alignment of the camera.

[0428] The refrigerator may determine that the left and right of the camera are not aligned if it determines that the position of the boundary line (bc) is outside the reference range (rr), and may determine that the left and right of the camera are aligned if it determines that the position of the boundary line (bc) is within the reference range (rr).

[0429] The refrigerator can determine the left and right attitude of the camera as normal if the position of the boundary line (bc) is within the reference range (rr), thereby deactivating the control of the third adjustment unit (113).

[0430] The refrigerator can determine that the camera (110) is in a reference pose with respect to the third axis if the position of the boundary line (bc) is within the reference range (rr).

[0431] If the refrigerator determines that the position of the boundary line (bc) is outside the reference range (rr), it recognizes a third adjustment value based on the position of the boundary line (bc) and the reference range (rr) (229), and controls the third adjustment unit (113) based on the recognized third adjustment value, thereby controlling the yaw of the camera (230).

[0432] The recognition of the third adjustment value and the control configuration of the third adjustment unit are described in more detail.

[0433] The refrigerator recognizes whether the position of the boundary line (bc) is to the left or right of the center position (r0) within the image when the position of the boundary line (bc) is outside the reference range (rr).

[0434] If the refrigerator determines that the position of the boundary line (bc) is to the left of the center position (r0), the third adjustment unit (113) can be controlled based on the first reference position (r1) and the position of the boundary line (bc).

[0435] In this case, the refrigerator can recognize the third difference value (d3) between the first reference position (r1) and the position of the boundary line (bc), recognize the third adjustment value (ad3) based on the recognized third difference value (d3), and control the third adjustment unit (113) based on the recognized third adjustment value (ad3).

[0436] The third adjustment value (ad3) may be equal to or greater than the third difference value (d3).

[0437] The refrigerator can also control the third adjustment unit (113) so that the position of the boundary line (bc) becomes the center position (r0).

[0438] The refrigerator can also control the third adjustment unit (113) so that the position of the boundary line (bc) becomes the first reference position (r1).

[0439] The refrigerator can also control the third adjustment unit (113) so that the position of the boundary line (bc) becomes the second reference position (r2).

[0440] The refrigerator can recognize a third difference value based on the pixel coordinate value of the boundary line (bc) and the pixel coordinate value of the first reference position, and can recognize a third adjustment value based on the recognized third difference value.

[0441] If the refrigerator determines that the position of the boundary line (bc) is to the right of the center position (r0), the refrigerator can control the third adjustment unit (113) based on the second reference position (r2) and the position of the boundary line (bc).

[0442] In this case, the refrigerator can obtain the fourth difference value (d4) between the second reference position (r2) and the position of the boundary line (bc), recognize the third adjustment value (ad3) based on the obtained second difference value (d4), and control the third adjustment unit (113) based on the recognized third adjustment value (ad3).

[0443] The third adjustment value (ad3) may be equal to or greater than the fourth difference value (d4).

[0444] The refrigerator can also control the third adjustment unit (113) so that the position of the boundary line (bc) becomes the center position (r0).

[0445] The refrigerator can also control the third adjustment unit (113) so that the position of the boundary line (bc) becomes the second reference position (r2).

[0446] The refrigerator can also control the third adjustment unit (113) so that the position of the boundary line (bc) becomes the first reference position (r1).

[0447] The refrigerator can recognize the fourth difference value based on the pixel coordinate value of the boundary line (bc) and the pixel coordinate value of the second reference position, and can recognize the third adjustment value based on the recognized fourth difference value.

[0448] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0449] Computer-readable storage media include all types of storage media that store instructions that can be deciphered by a computer. Examples include read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0450] The disclosed embodiments have been described with reference to the attached drawings as described above. Those skilled in the art will understand that the present invention can be implemented in forms other than the disclosed embodiments without altering the technical spirit or essential characteristics of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. A body including a storage room formed by an internal injury; A camera provided on the outer surface of the above body; A posture adjustment unit for adjusting the posture of the above camera; and A refrigerator including a processor that recognizes objects in an image acquired by the camera, recognizes an object corresponding to the internal injury among the recognized objects as a first object, recognizes the posture of the camera based on the recognized first object, and controls the posture adjustment unit based on the recognized posture of the camera.

2. In paragraph 1, The above storage room includes a refrigerator provided in the upper part of the main body and a freezer provided in the lower part of the main body, The field of view of the above camera is the entrance to the refrigerator, The inner surface recognized as the first object is a refrigerator including an inner surface provided at the entrance of the refrigerator compartment.

3. In paragraph 1, First and second doors for opening or closing the refrigerator compartment; and Including a third and fourth door for opening or closing the above freezer, The above camera is provided in an area between the first area where the first door comes into contact and the second area where the second door comes into contact among the above trauma areas, A refrigerator in which the processor controls the camera when the first and second doors are open and the third and fourth doors are closed, recognizes an object corresponding to the upper surface of the third and fourth doors among the recognized objects as a second object, and controls the posture adjustment unit based on the recognized second object.

4. In paragraph 3, The above-mentioned posture adjustment unit includes a first adjustment unit that adjusts a first posture of the camera by rotating the camera around a first axis; a second adjustment unit that adjusts a second posture of the camera by rotating the camera around a second axis; and a third adjustment unit that adjusts a third posture of the camera by rotating the camera around a third axis. The first and second axes are axes that are perpendicular to each other with respect to the horizontal plane, The third axis is an axis that is perpendicular to the first axis and the second axis with respect to the vertical plane, The above processor recognizes a first position of a first object existing in a first border among the borders of an image acquired by the camera and a second position of the first object existing in a second border, recognizes a height of the first position and a height of the second position based on a third border among the borders, recognizes a difference value between the height of the first position and the height of the second position, and controls the first adjustment unit based on the recognized difference value and the reference value when the recognized difference value is greater than a reference value.

5. In the fourth paragraph, the processor, In the first object above, the position of the inner surface of the lower surface among the inner surfaces of the storage room is recognized, A refrigerator that recognizes a height corresponding to the position of the inner surface of the recognized lower surface based on the third border, and controls the second adjustment unit based on the recognized height and the reference height when the recognized height is less than the reference height.

6. In the fourth paragraph, the processor, Recognize the position of the second object based on the third border, and if the recognized position of the second object is outside the reference range, control the third adjustment unit based on the position of the second object and the reference range. The above reference range is a refrigerator including a first reference position on the first side and a second reference position on the second side based on the center position of the third frame.

7. In paragraph 3, It further includes memory for storing information about the reference object, A refrigerator in which the processor recognizes the first and second objects among the recognized objects based on information about the reference object.

8. In the 6th paragraph, the processor, A refrigerator that controls the second control unit when the first object is not recognized in the image acquired by the camera, and ends the control of the second control unit when the first object is recognized by the control of the second control unit.

9. In the 6th paragraph, the processor, A refrigerator that controls the third control unit when the second object is not recognized in the image acquired by the camera, and ends the control of the third control unit when the second object is recognized by the control of the third control unit.

10. In paragraph 1, Including more display parts, A refrigerator in which the processor controls the display unit to display information about the recognized camera posture.

11. In paragraph 1, Further comprising an input unit for receiving user input, A refrigerator in which the processor, when input information in manual mode is received through the input unit, disables control of the posture adjustment unit and controls the display unit to display information on the recognized camera posture in real time.

12. In the first paragraph, the processor, A refrigerator that recognizes information on food being stored in or shipped from the storage room based on an image acquired by the camera, if the recognized camera posture is a reference posture, and updates a list of food stored in the storage room based on the recognized food information.

13. Obtain images using the camera installed on the body’s exterior, Recognize objects in the acquired image, Among the above recognized objects, the object corresponding to the inner surface of the main body is recognized as the first object, Recognize the pose of the camera based on the recognized first object, Adjusting the pose of the camera based on the recognized pose of the camera, A control method for a refrigerator that displays information about the position of the recognized camera through a display unit.

14. In paragraph 13, Obtaining an image using the above camera includes obtaining an image when the first and second doors of the first storage room are open and the third and fourth doors of the second storage room are closed. A method for controlling a refrigerator, wherein adjusting the posture of the camera comprises recognizing an object corresponding to the upper surface of the third and fourth doors among the recognized objects as a second object and controlling the posture adjustment unit based on the recognized second object.

15. In paragraph 14, adjusting the attitude of the camera is as follows: Recognize a first position of a first object existing in a first frame and a second position of the first object existing in a second frame among the frames of an image acquired by the camera, recognize a height of the first position and a height of the second position based on a third frame among the frames, recognize a difference value between the height of the first position and the height of the second position, and if the recognized difference value is greater than a reference value, rotate the camera about a first axis based on the recognized difference value and the reference value. In the first object, the position of the inner surface of the first storage room is recognized, and the height of the position of the inner surface of the recognized lower surface is recognized based on the third border, and if the recognized height is less than the reference height, the camera is rotated based on the recognized height and the reference height about the second axis. Recognize the position of the second object based on the third border, and if the recognized position of the second object is outside the reference range, rotate the camera around the third axis based on the position of the second object and the reference range, The second axis is an axis perpendicular to the first axis and the horizontal plane, The third axis is an axis perpendicular to the first axis and the second axis and a vertical plane, The above reference range is a control method for a refrigerator including a first reference position on the first side and a second reference position on the second side based on the center position of the third frame.

Images