Home appliance including camera, and operating method thereof

Abstract

Provided are a home appliance including a camera, and an operating method thereof. This home appliance comprises: a camera; a memory for storing at least one instruction; and at least one processor. The at least one processor executes the at least one instruction to: control the camera to be turned on, on the basis of a user trigger; store a primary correction value by performing primary automatic correction through the camera according to a light environment around the home appliance; and obtain a captured image obtained by applying secondary automatic correction of the camera, according to a door opening event of the home appliance. The secondary automatic correction is performed on the basis of the primary correction value.

Description

Home appliance including a camera and its operating method

[0001] Embodiments of the present disclosure relate to a home appliance including a camera and a method of operating the home appliance.

[0002] Recently, components such as cameras have been added to home appliances, enabling the monitoring of objects entering and leaving the appliance. For example, camera-captured video can be used to check information about food entering a refrigerator or monitor the cooking process in an oven.

[0003] Images captured by home appliances can be processed for object recognition using artificial intelligence models. However, if the captured image's brightness is unclear or the colors are inconsistent, accurate object recognition can be difficult. Accordingly, cameras embedded in home appliances can adjust the captured image's brightness for clarity or colors for consistency. Performing object recognition using corrected captured images can improve the accuracy of object recognition.

[0004] However, the longer it takes for the camera to capture an object being input or output from a home appliance and to adjust the brightness or color of the captured image, the more difficult it is for the user to monitor the object being input or output from the home appliance in real time, which may reduce user convenience.

[0005] A home appliance according to one embodiment includes a camera, a memory storing at least one instruction, and at least one processor.

[0006] At least one processor according to one embodiment controls power on of the camera based on a user trigger by executing the at least one instruction.

[0007] At least one processor according to one embodiment performs a first automatic correction corresponding to the light environment around the home appliance through the camera, thereby storing the first correction value.

[0008] At least one processor according to one embodiment obtains a captured image with secondary automatic correction applied to the camera in response to a door opening event of the home appliance.

[0009] The above secondary automatic correction is characterized in that it is performed based on the above primary correction value.

[0010] A method for controlling a home appliance including a camera according to one embodiment includes the steps of: controlling the power of the camera to be turned on based on a user trigger; performing a first automatic correction in response to a light environment around the home appliance through the camera, thereby storing a first correction value; and obtaining a captured image to which a second automatic correction of the camera is applied based on a door opening event of the home appliance. The second automatic correction is characterized in that it is performed based on the first correction value.

[0011] FIG. 1 is a drawing for explaining an outline of an object recognition operation of a home appliance according to one embodiment of the present disclosure.

[0012] FIG. 2A is a diagram for explaining an outline of an object recognition operation of a home appliance according to one embodiment of the present disclosure.

[0013] FIG. 2b is a diagram for explaining an outline of an object recognition operation of a home appliance according to one embodiment of the present disclosure.

[0014] FIG. 2c is a diagram for explaining an outline of an object recognition operation of a home appliance according to one embodiment of the present disclosure.

[0015] FIG. 2d is a diagram for explaining an outline of an object recognition operation of a home appliance according to one embodiment of the present disclosure.

[0016] FIG. 3 is an example of a captured image corrected through a camera according to one embodiment of the present disclosure.

[0017] FIG. 4 is a block diagram of a home appliance according to one embodiment of the present disclosure.

[0018] FIG. 5A is a flowchart illustrating an operation of a home appliance according to one embodiment of the present disclosure to acquire a captured image through a camera.

[0019] FIG. 5b is a flowchart illustrating an operation of a home appliance according to one embodiment of the present disclosure to acquire a captured image through a camera.

[0020] FIG. 5c is a flowchart illustrating an operation of a home appliance according to one embodiment of the present disclosure to acquire a captured image through a camera.

[0021] FIG. 6 is a drawing showing a structure of a home appliance including a sensor according to one embodiment of the present disclosure.

[0022] FIG. 7 is a flowchart illustrating an operation of a home appliance according to one embodiment of the present disclosure to obtain a user trigger through a sensor.

[0023] FIG. 8 is a flowchart illustrating an operation of a home appliance according to one embodiment of the present disclosure to obtain a user trigger through a sensor.

[0024] FIG. 9 is a diagram showing external devices registered to a server with the same account as a home appliance according to one embodiment of the present disclosure.

[0025] FIG. 10 is a flowchart illustrating an operation of a home appliance according to one embodiment of the present disclosure to obtain a user trigger through an in-home human body detection event.

[0026] FIG. 11 is a flowchart for determining whether to enter a power saving mode based on a usage pattern of a home appliance according to one embodiment of the present disclosure.

[0027] FIG. 12 is a flowchart for explaining an operation of performing automatic correction on a captured image based on a change in the light environment surrounding a home appliance according to one embodiment of the present disclosure.

[0028] FIG. 13 is a graph showing exposure values ​​obtained through automatic exposure of a camera according to one embodiment of the present disclosure.

[0029] FIG. 14 is a flowchart illustrating an operation for acquiring a light environment change event around a home appliance according to one embodiment of the present disclosure.

[0030] FIG. 15 is an exemplary drawing showing a light environment surrounding a home appliance according to one embodiment of the present disclosure.

[0031] FIG. 16 is a flowchart illustrating an operation for acquiring a light environment change event surrounding a home appliance according to one embodiment of the present disclosure.

[0032] FIG. 17 is a diagram illustrating an operation of a home appliance according to one embodiment of the present disclosure to obtain an adjustment value using an inference model.

[0033] FIG. 18 is a detailed block diagram of a home appliance according to one embodiment of the present disclosure.

[0034] The terms used in this disclosure are selected from widely used, current terms, taking into account the functions of one embodiment of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments of the disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of the disclosure.

[0035] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0036] In this disclosure, the term "and / or" includes a combination of a plurality of described elements or any element among the plurality of described elements. In this disclosure, terms such as "first," "second," or "first" or "second" may be used simply to distinguish the corresponding element from other corresponding elements and do not limit the corresponding elements in any other respect (e.g., importance or order).

[0037] Throughout this disclosure, whenever a part is said to "include" a certain component, this does not mean that it excludes other components, but rather that it may include other components, unless specifically stated otherwise.

[0038] In addition, terms such as "... unit", "module", etc. described in the present disclosure mean a unit that processes at least one function or operation, and "... unit", "module" may be implemented by hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), or software, or may be implemented by a combination of hardware and software. The term "~ unit" used in one embodiment of the present disclosure is not limited to software or hardware. The "~ unit" described in the present disclosure may be configured to be in an addressable storage medium and may be configured to reproduce one or more processors. In one embodiment of the present disclosure, the "~ unit" may include components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided by a particular component or a particular "part" may be combined to reduce the number of components or separated into additional components. Furthermore, in one embodiment, a "part" may include one or more processors.

[0039] In one embodiment of the present disclosure, each block of the flowchart diagrams and combinations of the flowchart diagrams can be performed by computer program instructions. The computer program instructions can be installed on a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment. Instructions executed by the processor of the computer or other programmable data processing equipment can generate means for performing the functions described in the flowchart block(s). The computer program instructions can also be stored in a computer-accessible or computer-readable memory that can direct a computer or other programmable data processing equipment to implement the functions in a particular manner. The instructions stored in the computer-accessible or computer-readable memory can also produce an article of manufacture that includes instruction means for performing the functions described in the flowchart block(s). The computer program instructions can also be installed on a computer or other programmable data processing equipment.

[0040] Additionally, each block in the flowchart diagram may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specified logical function(s). In one embodiment of the present disclosure, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may be executed substantially simultaneously or, depending on the function, may be executed in reverse order.

[0041] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, one embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted to clearly describe one embodiment of the present disclosure, and similar parts are designated with similar drawing reference numerals throughout the present disclosure.

[0042] Additionally, in this specification, 'image' may correspond to a still image, a moving image composed of multiple consecutive still images (or frames), or a video.

[0043] FIG. 1 is a drawing for explaining an outline of an object recognition operation of a home appliance according to one embodiment of the present disclosure.

[0044] Referring to FIG. 1, a home appliance (100) according to one embodiment of the present disclosure may include a camera (130) for photographing objects entering and exiting the home appliance (100). The home appliance (100) may further include at least one processor (130) and a memory (120) as illustrated in FIG. 4, which will be described later. The home appliance (100) may further include at least one sensor (140, 150) as illustrated in FIG. 6, which will be described later. In the present disclosure, the home appliance (100) is exemplified as a refrigerator, but is not limited thereto. For example, the home appliance (100) may include various home appliances including a camera.

[0045] The camera (130) illustrated in FIG. 1 is installed to capture images of the interior or exterior of the home appliance (100). For example, the camera (130) may be installed on one side of the upper portion of the home appliance (100). For example, the camera (130) may be installed inside the storage compartment of the home appliance (100). For example, the camera (130) may be installed between the doors (101) of the home appliance.

[0046] The camera (130) illustrated in FIG. 1 can capture still images and / or video of the interior or exterior of a home appliance (100). The camera (130) may be, but is not limited to, a wide-angle camera, a miniature camera, or a pinhole camera.

[0047] According to one embodiment of the present disclosure, a home appliance (100) can capture an image of the inside or outside of the home appliance (100) through a camera (130). For example, the camera (130) can capture an image when a door (101) of the home appliance (100) is opened. The home appliance (100) can perform image recognition using the captured image (10) acquired through the camera (130). By performing image recognition, the home appliance (100) can acquire at least one object information present in the captured image, for example, food information (20).

[0048] A camera (130) according to an embodiment of the present disclosure may have an auto correction function. For example, the auto correction function may include at least one of an auto exposure (AE) function, an auto white balance (AWB) function, and an auto focus (AF) function. The camera (130) may perform at least one of auto exposure, auto white balance, and auto focus, thereby allowing a user to easily obtain high-quality images without separate operations. Each of the auto exposure operation, auto white balance operation, and auto focus operation of the camera (130) may be referred to as being performed by an image signal processor (ISP) included in the camera (130).

[0049] For example, the camera (130) can perform an automatic exposure operation. For example, the camera (130) can perform an automatic exposure operation when the door (101) of the home appliance (100) is opened.

[0050] The automatic exposure of the camera (130) is a function to obtain high-quality images by maintaining an appropriate exposure value according to changes in light entering through the lens included in the camera (130). For example, the automatic exposure of the camera (130) is a function that automatically adjusts the shutter speed, etc. to adjust the exposure time even when the ambient light of the camera (130) changes, thereby maintaining a constant brightness of light received by the image sensor included in the camera (130).

[0051] For example, the camera (130) can perform an automatic white balance operation. For example, the camera (130) can perform an automatic white balance operation when the door (101) of the home appliance (100) is opened.

[0052] The automatic white balance of the camera (130) is a function that corrects the color of a white object to appear white by adjusting the color balance by correcting the color of the reflected light when taking a picture with the camera. When taking a picture with the camera, there is a difference in the expression of color depending on the light. If the color temperature is low, the captured image appears reddish, and if the color temperature is high, the captured image appears blueish. For example, under incandescent light, the subject appears reddish, under fluorescent light, the subject appears greenish, and under sunlight, the color temperature may appear different depending on the time of day. In this way, the camera (130) can perform white balance correction to correct the change in color depending on the light source. For example, the camera (130) can determine the current color temperature according to the automatic white balance, and correct each component of the R, G, and B signals of the captured image based on the determined color temperature, thereby maintaining an appropriate white balance value.

[0053] For example, the camera (130) can perform an autofocus operation. The autofocus of the camera (130) is a function that automatically adjusts the focus so that the subject is in focus when taking a picture. Below, the autoexposure operation and the autowhite balance operation are described as examples of the autocompensation operation performed by the camera (130), but the same can be applied to the autofocus operation.

[0054] According to one embodiment of the present disclosure, a home appliance (100) may turn on a camera (130) upon obtaining a user trigger. The camera (130) may be turned on in advance before a door opening event of the home appliance (100) occurs. The home appliance (100) may transmit a shooting request signal to the camera (130) upon occurrence of the door opening event. The camera (130) may remain powered on based on the user trigger and then perform video capturing when a door opening event of the home appliance (100) occurs. The camera (130) may perform an automatic exposure operation and an automatic white balance operation to obtain a captured image in which brightness, color, color temperature, etc. are appropriately corrected. The home appliance (100) may obtain information on at least one object present in the captured image by applying the captured image in which brightness, color, color temperature, etc. are appropriately corrected to an image recognition model.

[0055] In one embodiment of the present disclosure, the home appliance (100) can shorten the time required to acquire a captured image by controlling the power of the camera (130) to be turned on in advance before a door opening event occurs.

[0056] Hereinafter, with reference to other drawings, an operation of a home appliance (100) according to one embodiment of the present disclosure to obtain a user trigger for applying power to a camera (130) before a door opening event occurs will be described.

[0057] FIG. 2A is a diagram for explaining an outline of an object recognition operation of a home appliance according to one embodiment of the present disclosure.

[0058] Referring to FIG. 2A, a home appliance (100) according to one embodiment of the present disclosure can perform automatic correction via a camera (130) based on changes in the ambient light environment. For example, the camera (130) can perform at least one automatic correction operation, such as automatic exposure, automatic white balance, and automatic focus operation, by reflecting changes in the ambient light environment.

[0059] According to one embodiment of the present disclosure, a home appliance (100) may request a camera (130) to capture an image before a door opening event occurs when the light environment around the home appliance (100) changes. The camera (130) may perform automatic correction before the door opening event occurs. The automatic correction performed by the camera (130) before the door opening event may be referred to as 'primary automatic correction' or 'pre-auto correction'. According to one embodiment of the present disclosure, a home appliance (100) may store a primary correction value according to the primary automatic correction of the camera (130).

[0060] For example, when there is a change in the brightness of the light around the home appliance (100), the camera (130) can maintain an appropriate exposure value by correcting the exposure time based on the change in brightness according to the automatic exposure operation. When the automatic exposure operation becomes stable, the camera (130) can provide the exposure value to the home appliance (100) when the automatic exposure operation is in a stable state. The home appliance (100) can store the first exposure value according to the first automatic exposure in the memory. The first exposure value can be an exposure value that is stabilized in response to the brightness of the light around the home appliance (100) before the door opening event occurs.

[0061] For example, when there is a change in the color temperature of the light around the home appliance (100), the camera (130) can maintain the white balance by correcting the color temperature based on the change in brightness according to the automatic white balance operation. When the automatic white balance operation is stabilized, the camera (130) can provide the home appliance (100) with a value related to the white balance when in a stabilized state. For example, the value related to the white balance can include a color temperature value. The home appliance (100) can store the primary white balance according to the primary automatic white balance in the memory. The primary white balance can be a white balance that is stabilized in response to the brightness of the light around the home appliance (100) before the door opening event occurs.

[0062] According to one embodiment of the present disclosure, a home appliance (100) may perform a first automatic correction in response to the surrounding light environment, and then request the camera (130) to capture an image when a door opening event occurs. The camera (130) may perform automatic correction-processed image capture when the door (101) of the home appliance (100) is opened. The automatic correction performed by the camera (130) after the door opening event occurs may be referred to as a "second automatic correction." According to one embodiment of the present disclosure, a home appliance (100) may perform a second automatic correction using the first correction value stored according to the first automatic correction of the camera (130).

[0063] For example, when a door opening event occurs, the light environment surrounding the home appliance (100) may change. For example, when a door opening event occurs, the lighting (not shown) included in the home appliance (100) may be turned on, making the surroundings brighter.

[0064] Since the camera (130) has been stabilized in the light environment surrounding the home appliance (100) through the first automatic correction, it can perform a second automatic correction operation to stabilize in response to the light change due to the door opening event. The camera (130) can perform the second automatic correction taking into account the changed light environment.

[0065] The camera (130) can quickly converge to the target correction value by performing the secondary automatic correction using the primary correction value. For example, the home appliance (100) provides the primary correction value stored in the memory to the camera (130), thereby allowing the camera (130) to use the primary correction value as the initial value (or setting value) of the secondary automatic correction. Alternatively, the camera (130) can store the previously acquired primary correction value unless it is initialized after the primary automatic correction, and thus can use the primary correction value as the initial value (or setting value) of the secondary automatic correction. Here, the camera (130) can be initialized as the power of the camera (130) is turned on and off, but is not limited to the above-described example.

[0066] For example, the camera (130) can perform a secondary automatic exposure operation based on the stored primary exposure value. The camera (130) can quickly converge to the target exposure value according to the automatic exposure operation. In addition, the home appliance (100) can perform a secondary automatic white balance operation based on the stored primary automatic white balance. The camera (130) can quickly converge to the target white balance according to the automatic white balance operation.

[0067] The home appliance (100) performs secondary automatic correction using the primary correction value as the initial value, thereby more quickly obtaining a captured image in which brightness, color, color temperature, etc. are appropriately corrected in response to the surrounding light environment.

[0068] A home appliance (100) can obtain at least one piece of object information present in a captured image (10) by applying the captured image (10) whose brightness, color, color temperature, etc. have been appropriately corrected to an image recognition model. For example, the at least one piece of object information may include food information (20).

[0069] In one embodiment of the present disclosure, a home appliance (100) performs primary automatic correction according to changes in the ambient light environment and acquires a captured image based on the primary correction value, thereby shortening the time required to acquire a captured image.

[0070] Meanwhile, the home appliance (100) according to one embodiment illustrated in FIG. 2a is exemplified as acquiring and storing a primary correction value using a camera (130) that is turned on according to a user trigger, but is not limited thereto. An example of the home appliance (100) acquiring and storing a primary correction value is described in more detail in FIGS. 2b, 2c, and 2d.

[0071] FIG. 2b is a diagram for explaining an outline of an object recognition operation of a home appliance according to one embodiment of the present disclosure. FIG. 2c is a diagram for explaining an outline of an object recognition operation of a home appliance according to one embodiment of the present disclosure.

[0072] Referring to FIGS. 2B and 2C, a home appliance (100) according to one embodiment can periodically detect changes in the ambient light environment, power on the camera (130), and obtain a primary correction value. The home appliance (100) can store the primary correction value in memory.

[0073] For example, the home appliance (100) can detect changes in the ambient light environment through at least one sensor. Alternatively, for example, the home appliance (100) can acquire an event indicating a change in the ambient light environment through an external device.

[0074] For example, the home appliance (100) may power the camera (130) based on changes in the ambient light environment. The camera (130) may perform primary automatic correction in response to the ambient light environment. The home appliance (100) may obtain and store the primary correction value corresponding to the ambient light environment from the camera (130).

[0075] According to one embodiment, the home appliance (100) may provide a power-off signal to the camera (130) after acquiring the first correction value. The camera (130) may be powered off after performing the first automatic correction. The home appliance (100) may repeatedly store the first correction value based on changes in the ambient light environment before a user trigger or door opening event occurs.

[0076] According to one embodiment, a home appliance (100) can perform video capture through a camera (130) based on a door opening event. The home appliance (100) can perform secondary automatic correction using the primary correction value stored according to the primary automatic correction of the camera (130).

[0077] For example, referring to FIG. 2B, the home appliance (100) can control the camera (130) to turn on in response to a door opening event. The home appliance (100) can provide the pre-stored primary correction value to the initialized camera (130). The camera (130) can perform secondary automatic correction using the pre-stored primary correction value. The home appliance (100) can acquire a captured image with the secondary automatic correction applied through the camera (130).

[0078] Alternatively, for example, referring to FIG. 2c, the home appliance (100) may control the camera (130) to turn on according to a user trigger. The home appliance (100) may also control the camera (130) to turn on in advance before the door opening event so that the camera (130) takes a video immediately after the door opening event occurs.

[0079] FIG. 2d is a diagram for explaining an outline of an object recognition operation of a home appliance according to one embodiment of the present disclosure.

[0080] Referring to FIG. 2D, a home appliance (100) according to one embodiment can periodically detect a change in the ambient light environment and control the camera (130) to turn on the power. The home appliance (100) can obtain a first primary correction value (A) corresponding to the first ambient light environment. The home appliance (100) can store the first primary correction value (A). After obtaining the first primary correction value (A), the home appliance (100) can provide a power-off signal to the camera (130). In the present disclosure, the primary correction value obtained through the camera (130) that is turned on according to a change in the ambient light environment is expressed as a 'first primary correction value' or 'A'.

[0081] According to one embodiment, a home appliance (100) can control the power of a camera (130) to be turned on according to a user trigger. The home appliance (100) can obtain a second primary correction value (B) in response to a second ambient light environment. The home appliance (100) can store the second primary correction value (B). After obtaining the second primary correction value (B), the home appliance (100) can request the camera (130) to capture an image based on a door opening event. In the present disclosure, the primary correction value obtained through the camera (130) turned on according to the user trigger is expressed as a 'second primary correction value' or 'B'.

[0082] In the present disclosure, the first ambient light environment and the second ambient light environment may be different environments. For example, the ambient light environment may vary depending on changes in sunlight over time, changes in the lighting environment around the home appliance (100), changes in light sensor values, etc. However, this is not limited to the first ambient light environment and the second ambient light environment may be the same environment. Accordingly, the first primary correction value and the second primary correction value may be different or the same.

[0083] According to one embodiment, a home appliance (100) may compare previously stored primary correction values ​​based on a door opening event and provide the results to a camera (130). For example, the home appliance (100) may compare a first primary correction value (A) and a second primary correction value (B). The home appliance (100) may set a primary correction value closer to a target correction value among the first primary correction value (A) and the second primary correction value (B) as an initial value of the camera (130). The camera (130) may perform secondary automatic correction based on at least one of the first primary correction value (A) and the second primary correction value (B). By performing secondary automatic correction based on at least one of the first primary correction value (A) and the second primary correction value (B), the camera (130) may quickly converge on the target correction value.

[0084] Hereinafter, with reference to other drawings, an operation of a home appliance (100) according to one embodiment of the present disclosure performing primary automatic correction according to a change in the ambient light environment and performing secondary automatic correction based on the primary correction value will be described.

[0085] FIG. 3 is an example of a captured image corrected through a camera according to one embodiment of the present disclosure.

[0086] Referring to FIG. 3, the home appliance (100) can acquire captured images (310, 320, 330) corresponding to a plurality of frames through the camera (130). The first captured image (310) may be the first frame captured through the camera (130). As the images move from the first captured image (310), the second captured image (320), and the third captured image (330), the images may converge to an optimal correction value through automatic correction. The optimal correction value may refer to an exposure value or a value related to white balance that is appropriate for recognizing an object using an image recognition function based on images acquired by the camera (130).

[0087] For example, when image recognition is performed using the third captured image (330), object recognition accuracy can be improved compared to when image recognition is performed using the first captured image (310) or the second captured image (320).

[0088] For example, the camera (130) can select an appropriate illuminance and exposure time according to the automatic exposure operation to consistently maintain the desired brightness and color level. For example, a state in which the automatic exposure operation is stabilized may mean a state in which an image is acquired that is distributed in the middle value when the histogram of the brightness distribution of the image acquired by the camera (130) is expressed from 0 to 255. The exposure value in the stabilized state may correspond to the middle value of the histogram of the brightness distribution. The camera (130) can adjust at least one of the shutter speed, the aperture value, the exposure time, the exposure value, or the gain according to the automatic exposure operation.

[0089] For example, a captured image that has not been automatically exposed means that the exposure time is long (Overexposed) or short (Underexposed), and may have an exposure value close to 0 or 255 in the histogram of the brightness distribution.

[0090] For example, the camera (130) can maintain a desired color temperature according to the automatic white balance operation. For example, when the B (Blue) component of the light irradiated on the subject is strong, the camera (130) can correct the white balance of the image so that the sensitivity to the B component of the light is low. Conversely, when the color temperature of the light irradiated on the subject is low and the R (Red) light is strong, the electronic device (100) can correct the white balance of the image so that the sensitivity to the R component of the light is low.

[0091] For example, a captured image that has not been automatically white balanced may mean that the color temperature of the light irradiating the subject is high or low.

[0092] FIG. 4 is a block diagram of a home appliance according to one embodiment of the present disclosure.

[0093] Referring to FIG. 4, the home appliance (100) may include a processor (110), memory (120), and a camera (130).

[0094] The processor (110) controls the overall operation of the home appliance (100). The processor (110) may be implemented with one or more processors. The processor (110) may execute instructions or commands stored in the memory (120) to perform a predetermined operation. In addition, the processor (110) controls the operation of components provided in the home appliance (100). The processor (110) may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Processing Unit), or a combination thereof.

[0095] The processor (110) may be implemented as a main processor (not shown) and a sub processor (not shown) operating in a sleep mode.

[0096] The processor (110) may be connected to the camera (130) via a USB (Universal Serial Bus) communication method, but is not limited thereto. The home appliance (100) may also be described as including a communication interface for transmitting and receiving data with the camera (130).

[0097] The processor (110) acts as a USB host, supplies power to the bus, and can detect connected USB devices. The processor (110) can initiate USB communication with the camera (130) by providing a power-on signal to the camera (130) via USB communication. Upon identifying that the connected USB device is a camera (130), the processor (110) can register camera information in the kernel. The processor (110) can request the camera (130) to capture an image through a camera application and acquire a frame corresponding to the captured image.

[0098] The camera (130) can capture images of the interior or exterior of the home appliance (100) as described in FIGS. 1 to 2D . The camera (130) can include at least one of an automatic compensation function, such as automatic exposure, automatic white balance, and automatic focus. The camera (130) can be powered on based on a user trigger. Upon receiving a shooting request signal from the processor (110), the camera (130) can perform shooting while reflecting the automatic compensation.

[0099] The memory (120) stores various information, data, commands, programs, etc. required for the operation of the home appliance (100). The memory (120) may include at least one of volatile memory or non-volatile memory, or a combination thereof. The memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk. In addition, the memory (120) may correspond to a web storage or cloud server that performs a storage function on the Internet.

[0100] According to one embodiment, the processor (110) controls the power of the camera (130) to be turned on based on a user trigger by executing at least one instruction stored in the memory (120).

[0101] According to one embodiment, a processor (110) may acquire a user trigger. The user trigger may include various information for turning on the camera (130) and preparing for shooting before a door opening event of the home appliance (100) occurs.

[0102] For example, the user trigger may include information that the user is in proximity to the home appliance (100). For example, the processor (110) may obtain a user trigger corresponding to the user's proximity to the home appliance (100) through at least one sensor. For example, the processor (110) may control the camera (130) to turn on when it receives information regarding the user's proximity to the home appliance (100) through the proximity sensor. The proximity sensor may detect the user's proximity to the home appliance (100). When the user approaches the home appliance (100), this may be viewed as a user trigger for turning on the camera (130).

[0103] For example, the user trigger may include information on whether the user touches the handle located on the door of the home appliance (100). For example, the processor (110) may obtain the presence of a user's touch or a predetermined touch pressure through a handle sensor located on the door of the home appliance (100). Based on the user trigger corresponding to the presence of the user's touch or the predetermined touch pressure obtained through the handle sensor, the processor (110) may determine that it is a user trigger and control the camera (130) to turn on the power.

[0104] For example, the user trigger may include information according to a human body detection event in a home, including a home appliance (100). For example, the processor (110) may obtain the user trigger based on a human body detection event obtained from an external device connected to the same account as the home appliance (100). For example, the server may manage an external device connected to the same account as the home appliance (100). When the server detects the presence of a person in the home through the external device (i.e., a human body detection event occurs), the server may transmit a camera (130) power-on signal to the home appliance (100) in response to the human body detection event through the communication module. The home appliance (100) may determine the camera (130) power-on signal received from the server as a user trigger and control the camera (130) to turn on.

[0105] According to one embodiment, the processor (110) may provide a power-on signal to the camera (130) based on a user trigger. For example, the processor (110) may provide the power-on signal to the camera (130) via USB communication.

[0106] According to one embodiment, a processor (110) performs a first automatic correction in response to the light environment surrounding the home appliance (100) through a camera (130) by executing at least one instruction stored in a memory (120). The processor (110) may store a first correction value according to the first automatic correction. For example, the first automatic correction may be an automatic correction performed before a door opening event of the home appliance (100) occurs.

[0107] For example, the light environment around the home appliance (100) may include at least one of the lighting status of an external device located around the home appliance (100), the amount of sunlight by time zone, and the illuminance around the home appliance (100).

[0108] For example, the processor (110) can acquire a light environment change event around the home appliance (100). The light environment change event can include an on / off change of the lighting of an external device located around the home appliance (100), a change in the amount of sunlight by time zone, and a change in the illuminance around the home appliance (100).

[0109] For example, the processor (110) can obtain a lighting condition change event around the home appliance (100) through the server. For example, the processor (110) can obtain a lighting environment change event around the home appliance (100) in response to an event in which at least one light in the same space as the home appliance (100) is turned on or off. For example, the processor (110) can obtain a lighting environment change event around the home appliance (100) in response to at least one of a first lighting change event in which all lights in the space adjacent to the home appliance (100) are turned off and a second lighting change event in which all lights are changed from off to on at least one light.

[0110] For example, the processor (110) can identify a light environment change event based on changes in sunlight amount by predetermined time zone. For example, the processor (110) can identify changes in sunlight amount at sunrise, noon (12 o'clock), sunset, and midnight (00 o'clock), and perform a primary automatic correction based on the sunlight amount by time zone.

[0111] According to one embodiment, the processor (110) may transmit a shooting request signal to the camera (130) upon acquiring a change event in the surrounding light environment. The camera (130) may perform a primary automatic correction in response to the shooting request signal. For example, the camera (130) may correct exposure and / or white balance in response to the surrounding light environment (lighting conditions, amount of sunlight, illuminance, etc.). The camera (130) may store a primary correction value corresponding to the surrounding light environment. The camera (130) may provide the primary correction value to the processor (110). The processor (110) may store the primary correction value corresponding to the surrounding light environment of the camera (130) in the memory (120).

[0112] According to one embodiment, the processor (110) may acquire a captured image to which automatic correction of the camera (130) is applied in response to a door opening event of the home appliance (100) by executing at least one instruction stored in the memory (120). For example, the automatic correction may be a secondary automatic correction performed after the door opening event of the home appliance (100) occurs.

[0113] For example, the processor (110) may transmit a shooting request signal to the camera (130) in response to a door opening event of the home appliance (100). The camera (130) may perform secondary automatic correction in response to the shooting request signal. The camera (130) may perform secondary automatic correction by using the first corrected value in response to the surrounding light environment (lighting condition, amount of sunlight, illuminance, etc.) as an initial value (or setting value). Since the camera (130) uses the first correction value adapted to the light environment around the home appliance (100) as the initial value, it may quickly converge on a target correction value in response to a change in the surrounding light environment due to the door opening.

[0114] According to one embodiment, the processor (110) may obtain object information about at least one object recognized from a captured image by executing at least one instruction stored in the memory (120). The captured image subjected to secondary automatic correction processing may have values ​​related to exposure values ​​or white balance appropriate for object recognition. For example, pixels of the captured image to which the secondary correction value has been applied may have a greater middle gray distribution than pixels of the captured image to which the primary automatic correction value has been applied.

[0115] In the present disclosure, the captured image is exemplified as being subjected to image recognition processing in a home appliance (100), but is not limited thereto. For example, the home appliance (100) may transmit the captured image to a server, the captured image may be subjected to image recognition processing using an image recognition model installed in the server, and the home appliance (100) may also receive object information from the server.

[0116] FIG. 5A is a flowchart illustrating an operation of a home appliance according to one embodiment of the present disclosure to acquire a captured image through a camera.

[0117] Referring to FIG. 5a, in operation 510, an appliance (100) according to one embodiment can control the power of a camera (130) to be turned on based on a user trigger.

[0118] The home appliance (100) can acquire a user trigger. The user trigger may include various information for turning on the camera (130) and preparing for shooting before a door opening event of the home appliance (100) occurs. For example, the user trigger may include information that a user is approaching the home appliance (100) (see FIGS. 6 and 7). For example, the user trigger may include information on whether the user touches a handle located on the door of the home appliance (100) (see FIGS. 6 and 8). For example, the user trigger may include information on detecting the presence of a person in the home, including the home appliance (100), (i.e., occurrence of a human body detection event) (see FIGS. 9 and 10).

[0119] The home appliance (100) may provide a power-on signal to the camera (130) based on a user trigger. For example, the home appliance (100) may provide a power-on signal to the camera (130) via USB communication. The camera (130) may be powered on before a door opening event occurs.

[0120] In operation 520, the home appliance (100) according to one embodiment may perform a first automatic correction in response to the light environment around the home appliance (100) through a camera (130), thereby storing the first correction value. For example, the first automatic correction may be a pre-automatic correction performed before a door opening event of the home appliance (100) occurs.

[0121] For example, the light environment around the home appliance (100) may include at least one of the lighting status of an external device located around the home appliance (100), the amount of sunlight by time zone, and the illuminance around the home appliance (100).

[0122] The home appliance (100) can acquire a light environment change event around the home appliance (100). The light environment change event may include an on / off change of the lighting of an external device located around the home appliance (100), a change in the amount of sunlight by time zone, and a change in the illuminance around the home appliance (100).

[0123] For example, the home appliance (100) can obtain a light environment change event around the home appliance (100) through the server. For example, when the server obtains an event in which lighting in the same space as the home appliance (100) or an adjacent space is turned on or off, the server can provide the home appliance (100) with a light environment change event around the home appliance (100) (see FIGS. 14 and 15).

[0124] For example, the home appliance (100) can identify a light environment change event based on changes in sunlight amount by predetermined time zone (see FIG. 16).

[0125] For example, the home appliance (100) can obtain a primary correction value using an exposure inference model or a white balance inference model. The home appliance (100) can obtain a stabilized exposure value and white balance corresponding to the surrounding light environment by inputting information about the light environment around the home appliance (100) into the exposure inference model or the white balance inference model (see FIG. 17). The value output by the exposure inference model or the white balance inference model can be used as the primary correction value. In this case, the primary automatic correction operation of the camera (130) may be omitted.

[0126] The home appliance (100) may transmit a shooting request signal to the camera (130) as it acquires a surrounding light environment change event. The camera (130) may perform primary automatic correction in response to the shooting request signal. The camera (130) may provide the primary corrected value in response to the surrounding light environment to the processor (110). The processor (110) may store the primary corrected value in response to the surrounding light environment of the camera (130) in the memory (120).

[0127] For example, the camera (130) can compensate for exposure and / or white balance in response to the surrounding light environment (lighting conditions, sunlight, illuminance, etc.). For example, the camera (130) can obtain an appropriate exposure value for the surrounding light environment by performing automatic exposure based on the surrounding light environment. The camera (130) can obtain an appropriate white balance, etc. for the surrounding light environment by performing automatic white balance based on the surrounding light environment. The camera (130) can transmit the pre-exposure value and the pre-white balance to the processor (110), and the processor (110) can store the pre-exposure value and the pre-white balance.

[0128] In operation 530, the home appliance (100) according to one embodiment can acquire a captured image to which secondary automatic correction of the camera (130) is applied in response to a door opening event of the home appliance (100). For example, the automatic correction may be an automatic correction performed after the door opening event of the home appliance (100) occurs.

[0129] The home appliance (100) can acquire a door opening event. The processor (110) can transmit a shooting request signal to the camera (130) based on the door opening event. The camera (130) can perform automatic correction in response to the shooting request signal. The camera (130) can perform automatic correction using a first-calibrated value as an initial value in response to the surrounding light environment (lighting conditions, amount of sunlight, illuminance, etc.).

[0130] For example, the pixels of the captured image with the secondary correction applied may have a greater middle gray distribution than the pixels of the captured image with the primary correction applied. For example, the correction value applied to the captured image (the secondary correction value) based on the door opening event may be closer to the target correction value than the pre-corrected correction value (the primary correction value). For example, the secondary exposure value applied to the captured image based on the door opening event may be longer or shorter than the primary exposure value. For example, the color temperature set for the secondary white balance applied to the captured image based on the door opening event may be higher or lower than the color temperature set for the primary corrected white balance.

[0131] For example, when a door opening event occurs, a light element due to the lighting of the home appliance (100) may be added to the home appliance (100). Since the camera (130) uses a pre-compensation value adapted to the light environment around the home appliance (100) as an initial value, the camera (130) can quickly converge to a target compensation value that additionally takes into account the light element due to the lighting of the home appliance (100).

[0132] A home appliance (100) according to one embodiment of the present disclosure can perform image capturing immediately when a door opening event occurs by controlling the power of a camera (130) to be turned on based on a user trigger.

[0133] According to one embodiment of the present disclosure, a home appliance (100) can perform a primary automatic correction of a camera (130) based on changes in the surrounding light environment and store the primary correction value. The home appliance (100) can perform a secondary automatic correction based on a door opening event. Since the camera (130) performs the secondary automatic correction using the previously stored primary correction value as an initial value, it can quickly converge on a target correction value required for accurate image recognition processing.

[0134] FIG. 5b is a flowchart illustrating the operation of a home appliance acquiring a captured image through a camera according to one embodiment of the present disclosure. In FIG. 5b, any details overlapping with those in FIG. 5a are omitted.

[0135] Referring to FIG. 5b, in operation 511, the home appliance (100) according to one embodiment can control the power of the camera (130) to be turned on as it detects a change in the light environment around the home appliance (100).

[0136] For example, the home appliance (100) can detect changes in the ambient light environment through at least one sensor. Alternatively, for example, the home appliance (100) can acquire an event indicating a change in the ambient light environment through an external device.

[0137] In operation 521, the home appliance (100) according to one embodiment may perform a first automatic correction in response to the light environment around the home appliance (100) through the camera (130), thereby storing a first correction value. The first correction value may be stored in the memory (120). The home appliance (100) may store the first correction value and turn off the camera (130).

[0138] According to one embodiment, the home appliance (100) can periodically perform operations 511 and 521.

[0139] In operation 531, the home appliance (100) according to one embodiment can obtain a captured image with secondary automatic correction applied to the camera (130) according to a door opening event of the home appliance (100).

[0140] For example, the home appliance (100) can control the power of the camera (130) to be turned on in response to a door opening event. The home appliance (100) can provide the pre-stored primary correction value to the initialized camera (130). The camera (130) can perform secondary automatic correction using the pre-stored primary correction value. The home appliance (100) can obtain a captured image with the secondary automatic correction applied through the camera (130).

[0141] Alternatively, for example, the home appliance (100) may control the camera (130) to turn on according to a user trigger. The home appliance (100) may also control the camera (130) to turn on in advance before the door opening event so that the camera (130) takes a video immediately after the door opening event occurs.

[0142] FIG. 5C is a flowchart illustrating an operation of a home appliance acquiring a captured image through a camera according to one embodiment of the present disclosure. In FIG. 5C, any content overlapping with FIG. 5A or FIG. 5B is omitted.

[0143] Referring to FIG. 5c, in operation 512, the home appliance (100) according to one embodiment can control the power of the camera (130) to be turned on as it detects a change in the light environment around the home appliance (100).

[0144] In operation 522, the home appliance (100) according to one embodiment may perform a first automatic correction in response to a first light environment around the home appliance (100) through the camera (130), thereby storing a first primary correction value. The home appliance (100) may turn off the power of the camera (130).

[0145] In operation 532, the home appliance (100) according to one embodiment can control the power of the camera (130) to be turned on based on a user trigger.

[0146] In operation 542, the home appliance (100) according to one embodiment may perform a first automatic correction in response to a second light environment around the home appliance (100) through a camera (130), thereby storing a second first correction value.

[0147] The first ambient light environment and the second ambient light environment may be different environments or the same environments. Accordingly, the first primary correction value and the second primary correction value may be different or the same.

[0148] In operation 552, the home appliance (100) according to one embodiment can obtain a captured image to which secondary automatic correction of the camera (130) is applied using at least one of the first primary correction value and the second secondary correction value, based on a door opening event of the home appliance (100). The home appliance (100) can set at least one of the first primary correction value and the second secondary correction value as an initial value of the camera (130). The camera (130) can perform secondary automatic correction based on at least one of the first primary correction value and the second secondary correction value.

[0149] FIG. 6 is a drawing showing a structure of a home appliance including a sensor according to one embodiment of the present disclosure.

[0150] Referring to FIG. 6, a home appliance (100) according to one embodiment may further include at least one sensor (140, 150).

[0151] For example, referring to 601, the home appliance (100) may include a proximity sensor (140). The proximity sensor (140) may detect a surrounding object or the degree of proximity to an object. For example, the proximity sensor (140) may include a PIR (Passive infrared sensor) sensor that uses infrared to identify an object within the sensor's field of view. For example, when the PIR sensor is installed at the top of the home appliance (100), the field of view may be determined according to the angle at which the PIR sensor is installed. For example, the proximity sensor (140) may be located between the doors (101a, 101b) of the home appliance (100), but is not limited thereto. For example, the proximity sensor (140) may be located close to the camera (130), but is not limited thereto.

[0152] The home appliance (100) can determine whether a user is close to the home appliance (100) through a proximity sensor (140). The proximity sensor (140) can detect a user close to the home appliance (100) or detect the degree of proximity of the user.

[0153] For example, referring to 602, the home appliance (100) may include a handle sensor (150). The handle sensor (150) may detect whether a user touches or applies pressure to the door (101a, 101b) or the handle of the door (101a, 101b). For example, the handle sensor (150) may be located on the handle of the door (101a, 101b).

[0154] The home appliance (100) can determine whether the user has touched the handle of the home appliance (100) through the handle sensor (150). The home appliance (100) can determine that a touch has occurred if the touch pressure obtained through the handle sensor (150) is greater than a predetermined value.

[0155] FIG. 7 is a flowchart illustrating an operation of a home appliance according to one embodiment of the present disclosure to obtain a user trigger through a sensor.

[0156] Referring to FIG. 7, in operation 710, the home appliance (100) can obtain a sensing value through a proximity sensor (140). For example, the proximity sensor (140) can detect an object around the home appliance (100) or the degree of proximity to an object, and transmit the sensing value to the processor (110).

[0157] In operation 720, the home appliance (100) can determine whether the sensing value is greater than or equal to a threshold value. If the processor (110) determines that the sensing value is greater than or equal to the threshold value, the processor (110) can determine that the user is close to the home appliance (100). If the processor (110) determines that the sensing value is less than the threshold value, the processor (110) can determine that the user is not close to the home appliance (100).

[0158] In operation 730, the home appliance (100) can control the camera (130) to turn on when the sensing value is greater than or equal to a threshold value. The processor (110) can identify information that a user is approaching the home appliance (100) as a user trigger and control the camera (130) to turn on. Based on information acquired through the proximity sensor (140), the home appliance (100) can start preparing the camera (130) in advance when the user is located in front of the home appliance (100).

[0159] In operation 740, the home appliance (100) can identify whether a door opening event has occurred. For example, the home appliance (100) can identify whether a door opening event has occurred through a door opening detection sensor present in the home appliance (100), but is not limited thereto.

[0160] In operation 750, if it is determined that a door opening event has not occurred and a predetermined time has elapsed, the home appliance (100) can control the camera (130) to be turned off according to operation 780. Alternatively, if it is determined that a door opening event has not occurred and a predetermined time has not elapsed, the home appliance (100) can re-identify whether a door opening event has occurred.

[0161] In operation 760, the home appliance (100) can obtain a captured image with automatic correction applied by the camera (130) when a door opening event occurs. The processor (110) can provide a capture request signal to the camera (130) based on the door opening event. The camera (130) can transmit the captured image with automatic correction applied to the processor (110). Here, the captured image can be an image to which both the first automatic correction and the second automatic correction are applied, or an image to which only the second automatic correction is applied.

[0162] In operation 770, the home appliance (100) can obtain at least one object information from a captured image through image recognition processing. For example, the image recognition processing can be performed through an artificial intelligence model. The artificial intelligence model can input the captured image and output at least one object information included in the captured image. The artificial intelligence model can correspond to a neural network model trained using various food-related data as training data. The captured image can have an exposure value or white balance-related value appropriate for object recognition through image recognition processing.

[0163] In operation 780, the home appliance (100) can control the camera to turn off as it acquires object information.

[0164] FIG. 8 is a flowchart illustrating an operation of a home appliance according to one embodiment of the present disclosure to acquire a user trigger via a sensor. In FIG. 8, any content overlapping with that in FIG. 7 is briefly explained.

[0165] Referring to FIG. 8, in operation 810, the home appliance (100) can obtain whether the door handle of the home appliance (100) has been touched or a touch pressure value through the handle sensor (150). For example, when a user touches the door handle, the handle sensor (150) can transmit information on whether the user has touched the door handle or a touch pressure value to the processor (110).

[0166] In operation 820, the home appliance (100) can identify whether the touch pressure is greater than or equal to a threshold value. If the processor (110) determines that the touch pressure is greater than or equal to the threshold value, the processor (110) can identify the touch as a user trigger, assuming that the user is about to open the door of the home appliance (100). If the processor (110) determines that the touch pressure is less than the threshold value, the processor (110) can not identify it as a user trigger. Alternatively, the home appliance (100) can identify whether a touch exists, and if so, identify it as a user trigger.

[0167] In operation 830, the home appliance (100) can control the camera (130) to turn on when the touch pressure is above a threshold or when a touch is present. Based on information acquired through the handle sensor (150), the home appliance (100) can start preparing the camera (130) in advance when the user holds the door.

[0168] In operation 840, the home appliance (100) can identify whether a door opening event has occurred.

[0169] In operation 850, if it is determined that a door opening event has not occurred and a predetermined time has elapsed, the home appliance (100) can control the camera (130) to be turned off according to operation 880.

[0170] In operation 860, the home appliance (100) can obtain a captured image with automatic correction applied to the camera (130) as a door opening event occurs.

[0171] In operation 870, the home appliance (100) can obtain at least one object information from the captured image through image recognition processing.

[0172] In operation 880, the home appliance (100) can control the camera to turn off as it acquires object information.

[0173] FIG. 9 is a diagram showing external devices registered to a server with the same account as a home appliance according to one embodiment of the present disclosure.

[0174] Referring to FIG. 9, a home appliance (100) can connect to a network via an AP device (not shown) in the home and communicate with a server (200) or an external device. The home appliance (100) may include a communication module. For example, the home appliance (100) may indirectly communicate with an external device via the server (200) in a D2S (Device to Server) manner. Alternatively, for example, the home appliance (100) may directly communicate with an external device in a D2D (Device to Device) manner.

[0175] External devices may correspond to various types of electronic devices equipped with communication capabilities. For example, the external device may take the form of a mobile device. Additionally, the external device may correspond to a smartphone, a wearable device, a tablet PC, an oven, a smart cooking appliance, a refrigerator, a smart home hub, and the like. In FIG. 9, the external devices (310, 320, 330) are exemplified as including an oven (310), a robot vacuum cleaner (320), and a user terminal (330).

[0176] According to one embodiment of the present disclosure, an external device (310, 320, 330) may be connected to a server (200) using the same user identification information (e.g., login information, account information) as the home appliance (100).

[0177] The server (200) may include a communication module for communicating with an external device. The server (200) may also communicate with a home appliance (100) or an external device (310, 320, 330) through the communication module. The server (200) may update the software of the home appliance (100) based on communication with the home appliance (100) or transmit updated software related to the home appliance (100) based on communication with an external device (310, 320, 330).

[0178] Hereinafter, with reference to FIG. 10, an operation of an external device (310, 320, 330) located in a home such as a home appliance (100) to determine whether a user is present in the home and, if a user is present, to provide a user trigger to the home appliance (100) so that the home appliance (100) turns on the camera in advance and prepares is described.

[0179] In one embodiment of the present disclosure, an external device (310, 320, 330) can identify whether a human body is present around the external device (310, 320, 330). When the external device (310, 320, 330) identifies a human body detection event, the external device (310, 320, 330) can transmit information about the human body detection event to the server (200). Based on the information about the human body detection event, the server (200) can identify that a user is present in the home where the external device (310, 320, 330) is located.

[0180] FIG. 10 is a flowchart illustrating an operation of a home appliance according to one embodiment of the present disclosure to obtain a user trigger through an in-home human body detection event.

[0181] Referring to FIG. 10, in operation 1010, an external device (300) may transmit information regarding a human body detection event to a server (200). The server (200) may receive information regarding the human body detection event from the external device (300). The server (200) may determine that there is a person in the house through the human body detection event.

[0182] The external device (300) is connected to the same account as the home appliance (100) and can be located in the home of the home appliance (100).

[0183] For example, the external device (300) can identify the occurrence of a human detection event corresponding to GPS (Global Positioning System) location information, a touch input to the external device (300), information obtained from a human detection sensor on an indoor door, etc. The human detection event is not limited to the examples described above.

[0184] For example, if the GPS location information of the external device (300) received from the external device (300) indicates that the external device (300) is inside the house, the server (200) can determine that there is a person inside the house. For example, if there is a touch input on the external device (300) inside the house, the server (200) can determine that there is a person inside the house. For example, if the server (200) detects a human body through a human body detection sensor of the house entrance door, the server (200) can determine that there is a person inside the house.

[0185] Or, for example, if there is no human body detection in the house for a period of time longer than a standard time, the server (200) may determine that there is no person in the house.

[0186] In operation 1020, the server (200) may determine whether there is a home appliance (100) including a camera around the external device (300) upon receiving information about a human body detection event from the external device (300).

[0187] For example, the server (200) can receive information related to the external device (300) that transmitted the human body detection event, for example, the server (200) can receive device information, location information, IP information, etc. of the external device (300). Information related to devices registered to the same account can be stored in the database of the server (200). Based on the information related to the external device (300), the server (200) can determine whether there is a home appliance (100) including a camera around the external device (300). For example, when the server (200) receives a human body detection event from an oven located in the kitchen, the server (200) can determine whether there is a home appliance (100) including a camera in the kitchen based on the location information of the oven.

[0188] In operation 1030, the server (200) can identify device information of a home appliance (100) including a camera.

[0189] For example, the server (200) can identify device information of the home appliance (100) when it determines that there is a home appliance (100) including a camera among the home appliances located around the external device (300). The server (200) can identify device information of the home appliance (100) using device information registered to the same account stored in the database. For example, the server (200) can identify device information of the home appliance (100) including a camera located in the kitchen, which is the same location as the oven.

[0190] In operation 1040, the server (200) can transmit a trigger signal to the home appliance (100) corresponding to the identified device information. The home appliance (100) can receive the trigger signal from the server (200).

[0191] In operation 1050, the home appliance (100) can control the camera to turn on upon receiving a trigger signal from the server (200). The home appliance (100) can determine the trigger signal received from the server (200) in response to a human body detection event as a user trigger and initiate the camera's preparation operation.

[0192] The home appliance (100) can control the camera to turn on based on a human body detection event acquired from an external device (300) connected to the same account as the home appliance (100) via a communication module. A trigger signal acquired from the server (200) can correspond to a user trigger for camera preparation.

[0193] The home appliance (100) can quickly perform video recording through a camera that is turned on when a door opening event occurs, as in operation 740 of FIG. 7.

[0194] FIG. 11 is a flowchart for determining whether to enter a power saving mode based on a usage pattern of a home appliance according to one embodiment of the present disclosure.

[0195] Referring to FIG. 11, in operation 1110, the home appliance (100) can identify whether a door opening event has occurred.

[0196] In operation 1120, the home appliance (100) can check the time information when a door opening event occurs. When a user uses the home appliance (100), a door opening event of the home appliance (100) may occur. Accordingly, the time information when the door opening event occurs may correspond to information regarding the time zone during which the user uses the home appliance (100).

[0197] In operation 1130, the home appliance (100) can acquire a learned usage pattern learning model using information about the time at which a door-opening event occurred. The home appliance (100) can acquire a usage pattern learning model that learns the usage pattern of the home appliance (100) using time zones or frequency of use when the user frequently uses the home appliance (100). The home appliance (100) can train the usage pattern learning model using information about the time at which a door-opening event occurred as learning data. Here, the usage pattern learning model may be an artificial intelligence model.

[0198] AI models can be processed by dedicated AI processors, which are hardware architectures specifically designed for AI model processing. Training an AI model can mean creating a mathematical model that can infer optimal results by appropriately adjusting weights based on input training data.

[0199] The usage pattern learning model can infer the user's usage pattern of the home appliance (100), such as frequently used time zones or frequency of use.

[0200] The home appliance (100) can store a learned usage pattern learning model in memory (120) by using the time information when a door opening event occurred as learning data.

[0201] In operation 1140, the home appliance (100) can identify whether the current time is a time zone in which the home appliance (100) is frequently used by inputting current time information into the usage pattern learning model. When the usage pattern learning model receives current time information, it can output a result regarding whether the current time information is a time zone in which the home appliance (100) is frequently used.

[0202] At operation 1150, the home appliance (100) can identify whether the current time corresponds to a time zone in which the home appliance (100) is frequently used.

[0203] At operation 1160, the home appliance (100) may enter a power saving mode for the home appliance (100) as the current time does not correspond to a time zone in which the home appliance (100) is frequently used.

[0204] In operation 1170, the home appliance (100) may not enter a power saving mode for the home appliance (100) as the current time corresponds to a time zone in which the home appliance (100) is frequently used. The home appliance (100) may operate in a normal mode.

[0205] According to one embodiment, a home appliance (100) may enter a power-saving mode (or low-power mode) to reduce power consumption when the home appliance (100) is not in use. The home appliance (100) operating in power-saving mode may wake up from the power-saving mode and enter a normal mode upon obtaining a user trigger. After entering the normal mode, the home appliance (100) may provide a power-on signal to the camera (130). Accordingly, when the home appliance (100) obtains a user trigger in power-saving mode, it may take some time to turn on the power.

[0206] According to one embodiment, a home appliance (100) can acquire a usage pattern learning model that learns the usage pattern of the home appliance (100) by using the time zone or frequency of use when a user frequently uses the home appliance (100). The home appliance (100) can use the usage pattern learning model to determine whether the home appliance (100) will enter a power saving mode.

[0207] According to one embodiment, a home appliance (100) can omit the operation of waking up from power-saving mode according to a user trigger by performing a normal mode when the current time is a time zone in which the home appliance (100) is frequently used. By omitting the operation of waking up from power-saving mode according to a user trigger, the preparation time for turning on the camera can be reduced.

[0208] FIG. 12 is a flowchart illustrating an operation for automatically correcting a captured image based on changes in the ambient light environment of a home appliance according to one embodiment of the present disclosure. FIG. 13 is a graph representing exposure values ​​acquired through automatic exposure of a camera according to one embodiment of the present disclosure.

[0209] Referring to FIG. 12, in operation 1210, the home appliance (100) can obtain a light environment change event around the home appliance (100).

[0210] Events of changes in the light environment surrounding a home appliance (100) may include turning on / off the lighting of external devices located around the home appliance (100), changes in the amount of sunlight by time zone, and changes in the illuminance surrounding the home appliance (100). These are described in FIGS. 14 to 17.

[0211] In operation 1220, the home appliance (100) can perform primary automatic correction by operating the camera (130) as it acquires an ambient light environment change event.

[0212] For example, the home appliance (100) may transmit a shooting request signal to the camera (130) upon acquiring an ambient light environment change event. The camera (130) may perform primary automatic correction in response to the shooting request signal. For example, the camera (130) may correct exposure and / or white balance in response to the ambient light environment (lighting conditions, amount of sunlight, illuminance, etc.).

[0213] For example, the camera (130) can obtain a primary exposure value by performing automatic exposure in response to the surrounding light environment. For example, the camera (130) can obtain a primary white balance by performing automatic white balance in response to the surrounding light environment. The camera (130) can store a primary correction value including a primary exposure value and a primary white balance in response to the surrounding light environment.

[0214] When the automatic correction is stabilized, the camera (130) can provide the processor (110) with a primary correction value corresponding to the stabilized state. For example, when the automatic exposure operation is stabilized, the camera (130) can provide the processor (110) with an exposure value when the automatic exposure operation is in a stabilized state. When the automatic white balance operation is stabilized, the camera (130) can provide the processor (110) with a value related to the white balance when the automatic white balance operation is in a stabilized state.

[0215] In operation 1230, the home appliance (100) may store a primary correction value corresponding to the surrounding light environment within the home appliance (100). For example, the processor (110) may store the primary correction value obtained from the camera (130) in the memory (120).

[0216] In operation 1240, the home appliance (100) can identify whether a door opening event has occurred.

[0217] In operation 1250, the home appliance (100) can perform secondary automatic correction using the primary correction value through the camera (130) according to the door opening event of the home appliance (100).

[0218] For example, the home appliance (100) can transmit a shooting request signal to the camera (130) according to a door opening event of the home appliance (100).

[0219] In one embodiment, the camera (130) may be powered on or powered off before a door open event occurs. For example, the camera (130) may perform a primary automatic calibration and remain powered on. For example, if the camera (130) is powered off, the home appliance (100) may control the camera (130) to turn on by providing a power-on signal to the camera (130). The camera (130) may be powered on in response to a door open event. The camera (130) may also be powered on in response to a user trigger.

[0220] The camera (130) may perform secondary automatic correction upon receiving a shooting request signal. For example, the camera (130) may correct exposure and / or white balance in response to the surrounding lighting environment (lighting conditions, sunlight, illuminance, lighting conditions of the home appliance, etc.). For example, when a door opening event occurs, the lighting included in the home appliance (100) may be turned on, making the surroundings brighter, thus changing the surrounding lighting environment. The camera (130) may perform secondary automatic correction by taking into account changes in the surrounding lighting environment.

[0221] For example, the camera (130) can perform secondary automatic correction using the primary correction value.

[0222] The camera (130) can obtain a primary correction value corresponding to a stabilized state in the light environment surrounding the home appliance (100). For example, the processor (110) can set the primary correction value as the initial value (or setting value) of the secondary automatic correction by providing the primary correction value stored in the memory (120) to the camera (130).

[0223] Alternatively, for example, the camera (130) may store the previously acquired primary correction value unless it is initialized after the primary automatic correction, and thus may use the primary correction value as the initial value (or setting value) of the secondary automatic correction. For example, the camera (130) may not be initialized if it is kept powered on after performing the primary automatic correction.

[0224] For example, the camera (130) can obtain a secondary exposure value by performing a secondary automatic exposure using a primary exposure value that has been stabilized in response to the surrounding light environment. For example, the camera (130) can obtain a secondary white balance by performing a secondary automatic white balance using a secondary white balance that has been stabilized in response to the surrounding light environment.

[0225] In operation 1260, the home appliance (100) can acquire a captured image with a secondary correction value applied through the camera (130). The camera (130) performs secondary automatic correction using the previously stored primary correction value as an initial value, so that it can quickly converge on an optimal correction value. The optimal correction value refers to a correction value that can secure the accuracy of image recognition processing. The home appliance (100) can acquire object information about at least one object subjected to image recognition processing using the captured image.

[0226] For example, the graph in Fig. 13 shows exposure values ​​according to automatic exposure. The x-axis of the graph represents the exposure value, which can be expressed from 0 to 255. The larger the x-axis, the brighter the image can be. The y-axis of the graph can represent the amount of pixels in the image. The more the pixel distribution of the image is in the middle gray corresponding to the middle value of the x-axis, the more it can mean that the image has a stable exposure value.

[0227] The graph shows the initial value (1310) set in the camera (130) before auto exposure, the first exposure value (1320) corresponding to the stabilized state according to the first auto exposure, and the second exposure value (1330) corresponding to the stabilized state according to the second auto exposure. The second exposure value (1330) may be an optimal exposure value or a target exposure value. The exposure value of the initial value (1310) is exemplified as being close to 0, but is not limited thereto. Since the first exposure value (1320) corresponds to the exposure value that has been stabilized primarily in response to the ambient light environment, the distribution of middle gray may be greater than that of the image to which the initial value (1310) is applied. Since the second exposure value (1330) corresponds to the exposure value that has been stabilized using the first exposure value (1320), the distribution of middle gray may be greater than that of the image to which the first exposure value (1320) is applied.

[0228] The camera (130) can converge to the optimal exposure value more quickly when performing automatic exposure using the primary exposure value (1320) than when performing automatic exposure using the initial value (1310). The home appliance (100) can quickly acquire a captured image by storing the primary exposure value (1320) and using it in the secondary exposure operation.

[0229] Meanwhile, while the graph illustrates the second exposure value as being shorter than the first exposure, the second exposure value can be longer than the first exposure value. The magnitude of the first and second exposure values ​​can vary depending on the brightness or darkness of the surrounding lighting environment. Additionally, the color temperature set for the second white balance can be higher or lower than the color temperature set for the first corrected white balance.

[0230] FIG. 14 is a flowchart illustrating an operation for acquiring a change event of a light environment surrounding a home appliance according to one embodiment of the present disclosure. FIG. 15 is an exemplary diagram illustrating a light environment surrounding a home appliance according to one embodiment of the present disclosure.

[0231] Referring to FIG. 14, in operation 1410, the server (200) can track the lighting status around the home appliance (100). The lighting can represent a lighting device.

[0232] For example, in operation 1412, the server (200) may identify a lighting change event in which at least one light within the same space as the home appliance (100) is turned on or off.

[0233] For example, in operation 1414, the server (200) may identify a first event in which the entire lighting state within the home appliance (100) and the adjacent space is turned OFF, or a second event in which the entire lighting state is changed from OFF to at least one lighting state ON.

[0234] Referring to the layout diagram (1500) of FIG. 15, the home appliance (100) may be located in an indoor space of a house (e.g., a kitchen, a living room, a study, a bedroom, etc.). The surroundings of the home appliance (100) may refer to the same space as the home appliance (100) and / or an adjacent space. The same space as the home appliance (100) refers to the same space as the home appliance (100), and the adjacent space to the home appliance (100) may refer to the space closest to the space where the home appliance (100) is located or one or more spaces separated by a predetermined distance. For example, when the home appliance (100) is located in the kitchen, the same space (1510) may refer to the kitchen. For example, when the home appliance (100) is located in the kitchen, the adjacent space (1520) may refer to a living room adjacent to the kitchen.

[0235] For example, the server (200) can check the status of lights A and B existing in the same space (1510) as the home appliance (100). The server (200) can identify that a lighting change event has occurred when at least one of lights A and B is turned on or off.

[0236] For example, the server (200) can check the status of lights C, D, and E existing in the adjacent space (1520). If lights C, D, and E are all off, the server (200) can identify that a first lighting change event has occurred. If lights C, D, and E are all off, and one light, for example, light C, is turned on, the server (200) can identify that a second lighting change event has occurred. If lights D and E are turned on when only light C is on, the server (200) can determine that no lighting change event has occurred.

[0237] A change in the lighting status around a home appliance (100) can have a significant impact on the lighting environment around the home appliance (100). Lighting existing in a space (1520) adjacent to the home appliance (100) can have a lesser impact on the lighting environment change of the home appliance (100) than lighting existing in the same space (1510) as the home appliance (100). Therefore, the server (200) can identify a lighting change event by considering all status changes of individual lights in the case of lights existing in the same space (1510). In the case of lights existing in an adjacent space (1520), the server (200) can identify a lighting change event by considering a case where all lights are off or one or more lights are turned on while all lights are off.

[0238] In operation 1420, the server (200) can transmit information about a change in the light environment around the home appliance (100) to the home appliance (100).

[0239] The home appliance (100) can receive information about a light environment change event from the server (200) according to operation 1210 of FIG. 12.

[0240] For example, the home appliance (100) can acquire a light environment change event in response to a lighting change event in which at least one light in the same space as the home appliance (100) is turned on or off.

[0241] For example, the home appliance (100) can acquire a light environment change event in response to at least one of a first lighting change event in which all lighting in the home appliance (100) and an adjacent space is turned off and a second lighting change event in which all lighting is changed from off to at least one lighting is turned on.

[0242] FIG. 16 is a flowchart illustrating an operation for acquiring a light environment change event surrounding a home appliance according to one embodiment of the present disclosure.

[0243] Referring to FIG. 16, in operation 1610, the home appliance (100) can check the current time zone.

[0244] At step 1620, the appliance (100) can identify whether the current time zone corresponds to sunrise, noon (12 o'clock), sunset, or midnight (00 o'clock). Changes in sunlight intensity depending on the time zone can have a significant impact on the lighting environment around the appliance (100).

[0245] The home appliance (100) can identify changes in the amount of sunlight at predetermined time periods (e.g., sunrise, noon (12 o'clock), sunset, and midnight (00 o'clock)). The home appliance (100) can acquire an event of a change in the light environment around the home appliance (100) in response to changes in the amount of sunlight at predetermined time periods according to operation 1210 of FIG. 12.

[0246] In one embodiment, the predetermined time zones are exemplified as sunrise time, noon (12 o'clock), sunset time, and midnight (00 o'clock), but are not limited thereto.

[0247] FIG. 17 is a diagram illustrating an operation of a home appliance according to one embodiment of the present disclosure to obtain an adjustment value using an inference model.

[0248] Referring to FIG. 17, a home appliance (100) according to one embodiment can infer exposure values / white balance using an exposure / white balance inference model (1700).

[0249] Here, the exposure / white balance inference model (1700) may be an artificial intelligence model. The artificial intelligence model may be processed by an artificial intelligence-dedicated processor designed with a hardware structure specialized for processing artificial intelligence models. The artificial intelligence model may be created through learning. Such learning may be performed in the device itself (e.g., the home appliance (100)) on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server (200) and / or system. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0250] The learning data (1710) of the exposure / white balance inference model (1700) according to one embodiment may be automatically corrected exposure values ​​and white balance values ​​acquired through the camera (130). The learning data (1710) may vary depending on various variables acquired by the home appliance (100). For example, the exposure value according to automatic exposure and the white balance according to automatic white balance may vary depending on at least one of the amount of sunlight acquired by the home appliance (100), the lighting conditions inside the house, and the illuminance sensor value inside the house. The amount of sunlight acquired by the home appliance (100) may correspond to the data acquisition time. The exposure / white balance inference model (1700) may use the exposure value and white balance automatically corrected differently depending on at least one of the amount of sunlight acquired by the home appliance (100), the lighting conditions inside the house, and the illuminance sensor value inside the house, as the learning data (1710). The exposure / white balance inference model (1700) can be trained by appropriately changing weights based on input learning data (1710).

[0251] An exposure / white balance inference model (1700) according to one embodiment can input an ambient light environment, such as the amount of sunlight or the lighting conditions inside a home (1720), and output an optimal exposure value and white balance value corresponding to the ambient light environment (1730). For example, a home appliance (100) can input the amount of sunlight or the lighting conditions inside a home into the exposure / white balance inference model (1700). The exposure / white balance inference model (1700) can output an exposure / white balance value corresponding to a stable state of an automatic correction operation based on the current amount of sunlight or the lighting conditions inside a home. The home appliance (100) can obtain a stabilized exposure / white balance value corresponding to the current amount of sunlight or the lighting conditions inside a home through the exposure / white balance inference model (1700).

[0252] For example, the home appliance (100) can obtain an appropriate exposure / white balance value to be set as the initial value of the camera (130) in response to the surrounding light environment. For example, the home appliance (100) can set the exposure / white balance value obtained through the exposure / white balance inference model (1700) as the initial value (1740) of the automatic correction operation of the camera (130). The camera (130) can initiate the automatic exposure or automatic white balance operation at a value most similar to the current light environment. Accordingly, the camera (130) can quickly converge to the target value.

[0253] In one embodiment, the home appliance (100) can use the output value of the exposure / white balance inference model (1700) as the primary correction value. For example, the home appliance (100) can obtain a stabilized exposure / white balance value corresponding to the light environment around the home appliance (100) by inputting information about the light environment around the home appliance (100) into the exposure / white balance inference model (1700). The home appliance (100) can perform secondary automatic correction of the camera (130) using the exposure / white balance value obtained from the exposure / white balance inference model (1700) as the initial value. In this case, the primary automatic correction operation of the camera (130) may be omitted.

[0254] Alternatively, in one embodiment, the home appliance (100) may perform a primary automatic correction via the camera (130) and a secondary automatic correction to approximate the output value of the exposure / white balance inference model (1700).

[0255] FIG. 18 is a detailed block diagram of a home appliance according to one embodiment of the present disclosure.

[0256] A home appliance (100) according to one embodiment of the present disclosure may correspond to a home appliance (1800). A home appliance (1800) according to one embodiment of the present disclosure includes a sensor (1810), an output interface (1820), an input interface (1830), a memory (1840), a communication module (1850), a home appliance function module (1860), a camera (1870), a power module (1880), and a processor (1890). The home appliance (1800) may be configured with various combinations of the components illustrated in FIG. 18, and not all of the components illustrated in FIG. 18 are essential components.

[0257] The home appliance (1800) of FIG. 18 corresponds to the home appliance (100) described in FIG. 4, the memory (1840) corresponds to the memory (120) described in FIG. 4, the processor (1890) corresponds to the processor (110) described in FIG. 4, and the camera (1870) corresponds to the camera (130) described in FIG. 4.

[0258] The sensor (1810) may include various types of sensors, for example, the sensor (1810) may include various types of sensors such as an image sensor, an infrared sensor, an ultrasonic sensor, a lidar sensor, a human body detection sensor, a motion detection sensor, a proximity sensor, a handle sensor, and an illuminance sensor. Since the function of each sensor can be intuitively inferred from its name by those skilled in the art, a detailed description thereof will be omitted.

[0259] The output interface (1820) may include a display (1821), a speaker (1822), etc. The output interface (1820) outputs various notifications, messages, information, etc. generated by the processor (1890).

[0260] The input interface (1830) may include keys (1831), a touch screen (1832), etc. The input interface (1830) receives user input and transmits it to the processor (1890).

[0261] The communication module (1850) may include at least one or a combination of a short-range communication module (1852) or a long-range communication module (1854). The communication module (1850) may include at least one antenna for wirelessly communicating with another device.

[0262] The short-range wireless communication module (1852) may include, but is not limited to, a Bluetooth communication module, a BLE (Bluetooth Low Energy) communication module, a near field communication module, a WLAN (Wi-Fi) communication module, a Zigbee communication module, an infrared (IrDA, infrared Data Association) communication module, a WFD (Wi-Fi Direct) communication module, a UWB (ultrawideband) communication module, an Ant+ communication module, a microwave (uWave) communication module, etc.

[0263] The remote communication module (1854) may include a communication module that performs various types of remote communication and may include a mobile communication unit. The mobile communication unit transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signals may include various types of data according to the transmission and reception of voice call signals, video call signals, or text / multimedia messages.

[0264] The home appliance function module (1860) includes an operation module that performs the original function of the home appliance (1800). The power module (1880) is connected to a power source and supplies power to the home appliance (1800).

[0265] The processor (1890) controls the overall operation of the home appliance (1800). The processor (1890) can control components of the home appliance (1800) by executing a program stored in the memory (1840).

[0266] According to one embodiment of the present disclosure, the processor (1890) may include a separate NPU that performs the operation of an artificial intelligence model. In addition, the processor (1890) may include a central processing unit (CPU), a graphics processor (GPU; Graphic Processing Unit), etc.

[0267] In one embodiment, at least one processor controls the power of the camera to be turned on based on a user trigger by executing the at least one instruction. The at least one processor performs a first automatic correction in response to a light environment around the home appliance through the camera, thereby storing a first correction value. The at least one processor obtains a captured image to which a second automatic correction of the camera is applied based on a door opening event of the home appliance. The second automatic correction is characterized in that it is performed based on the first correction value.

[0268] At least one processor according to one embodiment can control the camera to turn on when information regarding a user's proximity to the home appliance is received through the proximity sensor by executing the at least one instruction.

[0269] At least one processor according to one embodiment can control the camera to be turned on by executing the at least one instruction, upon obtaining a user's touch on the handle or a touch pressure greater than a predetermined value through the handle sensor.

[0270] At least one processor according to one embodiment can control the power of the camera to be turned on based on a human body detection event acquired from an external device connected to the same account as the home appliance through the communication module by executing the at least one instruction.

[0271] According to one embodiment, at least one processor may acquire a learned usage pattern learning model by using time information when a door opening event of the home appliance occurs as learning data by executing the at least one instruction. The at least one processor may identify whether the current time is a time period in which the home appliance is frequently used by inputting current time information into the usage pattern learning model. The at least one processor may not enter a power saving mode for the home appliance when the current time corresponds to a time period in which the home appliance is frequently used.

[0272] According to one embodiment, at least one processor may transmit a shooting request signal to the camera by acquiring a light environment event around the home appliance by executing the at least one instruction. The at least one processor may acquire the primary correction value corresponding to the stabilization state of the automatic correction operation from the camera. The at least one processor may store the primary correction value.

[0273] The pixels of the captured image to which the secondary correction value according to the secondary automatic correction is applied may be characterized by having a greater middle gray distribution than the pixels of the captured image to which the primary automatic correction value is applied.

[0274] According to one embodiment, at least one processor may obtain a light environment change event around the home appliance by executing the at least one instruction. The at least one processor may obtain a light environment change event around the home appliance in response to an event in which at least one light within the same space as the home appliance is turned on or off, and may obtain a light environment change event around the home appliance in response to at least one of a first event in which all lights within a space adjacent to the home appliance are turned off and a second event in which all lights are turned off and at least one light is changed to turn on.

[0275] At least one processor according to one embodiment can acquire a light environment change event around the home appliance in response to a change in sunlight amount by predetermined time zone.

[0276] According to one embodiment, at least one processor may execute the at least one instruction to input information about the light environment surrounding the home appliance into an exposure / white balance inference model, thereby obtaining a stabilized exposure / white balance value corresponding to the light environment surrounding the home appliance. The secondary automatic correction may be characterized in that it is performed based on the exposure / white balance value.

[0277] According to one embodiment, at least one processor may control the camera to turn on when a change in the lighting environment around the home appliance is detected by executing at least one instruction. The processor may perform a first automatic correction in response to the lighting environment around the home appliance through the camera, and store the first correction value. The processor may acquire a captured image with the second automatic correction applied by the camera in response to a door opening event of the home appliance.

[0278] According to one embodiment, at least one processor may control to turn on the camera by executing at least one instruction, upon detecting a change in the light environment around the home appliance. The processor may perform a first automatic correction in response to the first light environment around the home appliance through the camera, thereby storing a first primary correction value. The processor may control to turn on the camera based on a user trigger. The processor may perform a first automatic correction in response to the second light environment around the home appliance through the camera, thereby storing a second primary correction value. The processor may obtain a captured image to which the second automatic correction of the camera is applied, using at least one of the first primary correction value and the second secondary correction value, in response to a door opening event of the home appliance.

[0279] A method for controlling a home appliance including a camera according to one embodiment includes the steps of: controlling the power of the camera to be turned on based on a user trigger; performing a first automatic correction in response to a light environment around the home appliance through the camera, thereby storing a first correction value; and obtaining a captured image to which a second automatic correction of the camera is applied based on a door opening event of the home appliance. The second automatic correction is characterized in that it is performed based on the first correction value.

[0280] According to one embodiment of the present disclosure, a camera can be turned on based on a user trigger prior to a door-opening event occurring in a home appliance. The home appliance can quickly capture video using a camera that is pre-turned on upon the door-opening event.

[0281] According to one embodiment of the present disclosure, automatic correction can be performed in advance based on the lighting environment surrounding the home appliance before a door-opening event occurs, and the correction value can be stored. The home appliance can perform automatic correction using the previously stored correction value in response to the door-opening event, thereby quickly converging to a target value.

[0282] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0283] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. In home appliances (100), Camera (130); A memory (120) storing at least one instruction; and comprising at least one processor (110), The at least one processor (110) executes the at least one instruction, Based on a user trigger, the camera (130) is controlled to be powered on, By performing the first automatic correction in response to the light environment around the home appliance (100) through the above camera (130), the first correction value is stored, According to the door opening event of the above home appliance (100), a captured image with the second automatic correction applied of the above camera (130) is obtained, A home appliance (100), characterized in that the above-mentioned second automatic correction is performed based on the above-mentioned first correction value.

2. In paragraph 1, The above home appliance (100) further includes a proximity sensor (140), The at least one processor (110) executes the at least one instruction, A home appliance (100) that controls power to be turned on by the camera (130) when information regarding a user's proximity to the home appliance (100) is received through the proximity sensor (140).

3. In paragraph 1 or 2, The above home appliance (100) further includes a handle sensor (150) located on the door of the above home appliance (100). The at least one processor (110) executes the at least one instruction, A home appliance (100) that controls the power of the camera (130) to be turned on when the user touches the handle or a touch pressure greater than a predetermined value is obtained through the handle sensor (150).

4. In any one of paragraphs 1 to 3, The above home appliance (100) further includes a communication module (2350), The at least one processor (110) executes the at least one instruction, A home appliance (100) that controls the power of the camera (130) to be turned on according to a human body detection event acquired from an external device connected to the same account as the home appliance (100) through the communication module (2350).

5. In any one of paragraphs 1 to 4, The at least one processor (110) executes the at least one instruction, A learning model for learned usage patterns is obtained by using the time information of the door opening event of the above home appliance (100) as learning data, By inputting current time information into the above usage pattern learning model, it is identified whether the current time is a time zone in which home appliances (100) are frequently used, A home appliance (100) that does not enter a power saving mode for the home appliance (100) because the current time corresponds to a time zone in which the home appliance (100) is frequently used.

6. In any one of paragraphs 1 to 5, The at least one processor (110) executes the at least one instruction, As the light environment event around the above home appliance (100) is acquired, a shooting request signal is transmitted to the camera (130), Obtain the first correction value corresponding to the stabilization state of the automatic correction operation from the above camera (130), A home appliance (100) storing the above first correction value.

7. In any one of paragraphs 1 to 6, A home appliance (100), characterized in that the pixels of the photographed image to which the second correction value according to the second automatic correction is applied have a greater middle gray distribution than the pixels of the photographed image to which the first correction value is applied.

8. In any one of paragraphs 1 to 7, The at least one processor (110) executes the at least one instruction, Obtaining a light environment change event around the above home appliance (100), Acquire a light environment change event around the home appliance (100) in response to an event in which at least one light in the same space as the home appliance (100) is turned on or off, A home appliance (100) that acquires a light environment change event around the home appliance (100) in response to at least one of a first event in which all lights in the home appliance (100) and an adjacent space are turned OFF and a second event in which at least one light changes from OFF to ON.

9. In any one of paragraphs 1 to 8, The at least one processor (110) executes the at least one instruction, Obtaining a light environment change event around the above home appliance (100), A home appliance (100) that acquires a light environment change event around the home appliance (100) in response to changes in sunlight amount at predetermined time periods.

10. In any one of paragraphs 1 to 9, The at least one processor (110) executes the at least one instruction, By inputting information about the light environment around the home appliance (100) into the exposure / white balance inference model (1700), a stabilized exposure / white balance value corresponding to the light environment around the home appliance (100) is obtained, A home appliance (100), characterized in that the above secondary automatic correction is performed based on the above exposure / white balance value.

11. A method for controlling a home appliance (100) including a camera (130), Step (510) of controlling power on of the camera (130) based on a user trigger; Step (520) of storing the first correction value by performing the first automatic correction in response to the light environment around the home appliance (100) through the above camera (130); In accordance with the door opening event of the above home appliance (100), a step (530) of obtaining a captured image with the second automatic correction applied to the camera (130) is included. A method, characterized in that the second automatic correction is performed based on the first correction value.

12. In paragraph 11, Based on the above user trigger, the step of controlling power on of the camera (130) is as follows: A method comprising a step of controlling power to be turned on of the camera (130) when information regarding a user's proximity to the home appliance (100) is received through a proximity sensor (140).

13. In clause 11 or 12, Based on the above user trigger, the step of controlling power on of the camera (130) is as follows: A method comprising a step of controlling the power of the camera (130) to be turned on when the user touches the handle or a touch pressure greater than a predetermined value is obtained through the handle sensor (150).

14. In any one of paragraphs 11 to 13, Based on the above user trigger, the step of controlling power on of the camera (130) is as follows: A method comprising a step of controlling the power of the camera (130) to be turned on according to a human body detection event acquired from an external device connected to the same account as the home appliance (100) through a communication module (2350).

15. In any one of paragraphs 11 to 14, The above method, A step of obtaining a learned usage pattern learning model by using the time information at which a door opening event of the above home appliance (100) occurred as learning data; A step of identifying whether the current time is a time zone in which the home appliance (100) is frequently used by inputting current time information into the above-mentioned usage pattern learning model; and A method further comprising a step of not entering a power saving mode for the home appliance (100) when the current time corresponds to a time zone in which the home appliance (100) is frequently used.

Images