Home appliances and their control methods

Abstract

A home appliance and its control method are disclosed. The disclosed home appliance performs a first appliance action on an object in space that corresponds to the shape of a human body. Then, objects that are not the user are removed from the object list, and a second appliance action is performed only on the user. At this time, the first and second appliance actions are not performed simultaneously and are distinct from each other.

Description

Technical Field

[0001] This invention relates to a household appliance and its control method. Background Technology

[0002] An air conditioner is a device that regulates indoor temperature and purifies indoor air by expelling cool or warm air into the room to create a comfortable indoor environment. By installing air conditioners, people can enjoy a more comfortable indoor environment.

[0003] An air conditioner includes: an indoor unit, installed indoors and including a heat exchanger; and an outdoor unit, installed outdoors and including a compressor and a heat exchanger, supplying refrigerant to the indoor unit. The air conditioner is controlled by both indoor and outdoor units. At least one indoor unit can be connected to an outdoor unit, operating in either cooling or heating mode depending on the required operating conditions.

[0004] On the other hand, in order to adjust the direction of the airflow into the room, an airflow adjustment device is installed at the air outlet of the air conditioner. Users can change the airflow direction by operating the airflow setting button located on the remote control or other similar device.

[0005] Existing air conditioners adjust the airflow direction manually as described above, but users who move frequently indoors need to change the airflow direction each time, which may cause inconvenience.

[0006] Therefore, in recent years, technologies have been developed to control wind speed or air current based on the location of users (i.e., indoor occupants) in an indoor space.

[0007] In the prior art 1 (KR 10-2018-0071031, Air conditioner and control method thereof), images are acquired by a camera, and the areas where indoor occupants are located in an indoor space divided into multiple areas are identified based on the acquired images. The location identification results of indoor occupants are used as input to data learned by machine learning and living areas are distinguished among the multiple areas. Airflow is controlled based on the distinguished living areas.

[0008] In prior art 2 (KR 10-2019-0118337, Air conditioner, cloud server and operation method based on artificial intelligence for selective regional operation), the centralized air supply area is determined by the distance and direction of indoor occupants sensed in the air supply area of ​​the air conditioner. Areas other than the centralized air supply area are classified as non-centralized air supply areas. For centralized air supply areas, the air conditioner is instructed to operate in a centralized operation mode. If the operation is completed in the centralized operation mode, the control is to operate in a different operation mode than the centralized operation mode for both centralized and non-centralized air supply areas.

[0009] However, due to the poor accuracy of indoor human (human) position sensing in the existing technology described above, there is a problem of ineffective airflow control. In particular, indoor spaces may contain picture frames, display devices (for example, TVs), etc., which may lead to the problem of human body shape displayed in photos on picture frames or on display devices being mistaken for users, resulting in a decrease in the accuracy of human body sensing. Therefore, a technology that can more accurately sense the human body is needed. Summary of the Invention

[0010] The problem that the invention aims to solve

[0011] The purpose of this invention is to provide a home appliance device and its control method that can perform home appliance actions based on user location by more accurately sensing the presence of users (indoor occupants) in the space.

[0012] The purpose of this invention is to provide a home appliance and its control method that can clearly distinguish between a user existing in space and a human body displayed in a picture frame or display device.

[0013] The purpose of this invention is to provide a home appliance and its control method that can eliminate the inconvenience of using home appliances for users.

[0014] The purpose of this invention is not limited to the purposes mentioned above. Other purposes and advantages of this invention not mentioned can be understood through the following description, and will become clearer through the embodiments of this invention.

[0015] Technical solutions to the problem

[0016] In a home appliance and its control method according to an embodiment of the present invention, a first home appliance action is performed on an object in space that corresponds to the shape of a human body. Objects that are not the user are excluded from the list of objects, and a second home appliance action, different from the first, is performed only on the user. In this case, the first and second home appliance actions are not performed simultaneously and are distinct from each other.

[0017] Specifically, in a home appliance and its control method according to an embodiment of the present invention, in a first time interval, one or more first regions of interest, including objects corresponding to the shape of the human body, are detected, and a first home appliance action is performed based on the first regions of interest. In a second time interval, similar objects to the user are removed from one or more first regions of interest, one or more second regions of interest are detected, and a second home appliance action is performed based on the second regions of interest.

[0018] In addition, in a home appliance and its control method according to an embodiment of the present invention, the user existing in the space and the human body displayed in the picture frame or display device are distinguished based on the cumulative difference image, so as to simply and effectively distinguish the user from similar objects.

[0019] A home appliance device according to an embodiment of the present invention includes: a memory storing at least one instruction; a processor executing the at least one instruction; and an action unit controlled by the processor to perform home appliance actions. The processor controls the action unit to perform a first home appliance action, which is based on a first image in the image and performed on an object in the space corresponding to the shape of a human body. The processor controls the action unit to perform a second home appliance action different from the first home appliance action, which is based on a second image in the image after the first image and performed only on a user in the space of the object.

[0020] At this time, the processor can control the action unit to detect, based on the first image, one or more first regions of interest including the object in a plurality of regions constituting the space, and to perform the first appliance action based on one or more first regions of interest. The processor can also control the action unit to detect, based on the second image, one or more second regions of interest where the object is the user in one or more first regions of interest, and to perform the second appliance action based on one or more second regions of interest.

[0021] Additionally, the object includes the user and user-similar objects, which can be photos or pictures of a human body included in a picture frame and photos or pictures of a human body displayed on a display device.

[0022] Additionally, the processor can generate a cumulative difference image based on the second images acquired sequentially by the camera over a preset time period, and can detect one or more second regions of interest based on the cumulative difference image.

[0023] In addition, the cumulative difference image may correspond to the average image of the plurality of difference images, and the difference image may be an image corresponding to the difference between the image at a first time point and the image at a second time point, which is a time point before the first time point.

[0024] In addition, the difference image may be an image corresponding to the difference between the grayscale image of the image at the first time point and the grayscale image of the image at the second time point.

[0025] In addition, the processor can detect more than one second region of interest by comparing the pixel value of the first region of interest with the preset threshold pixel value.

[0026] In addition, the critical pixel value includes a first critical pixel value and a second critical pixel value. If the pixel value of a certain first region of interest in more than one first region of interest is greater than the first critical pixel value and less than the second critical pixel value, the processor can detect the certain first region of interest as the second region of interest.

[0027] Furthermore, if the pixel value of one of the first regions of interest is less than the first critical pixel value or greater than the second critical pixel value in one or more first regions of interest, the processor may not detect the first region of interest as the second region of interest.

[0028] Furthermore, if the pixel value of a certain first area of ​​interest is less than the first critical pixel value, the processor can determine that the object included in the certain first area of ​​interest is a photograph or picture of a human body included in a frame; if the pixel value of a certain first area of ​​interest is greater than the second critical pixel value, the processor can determine that the object included in the certain first area of ​​interest is a photograph or picture displayed on the display device.

[0029] Additionally, the processor detects quadrilateral shapes in the accumulated difference image and can further detect the second region of interest based on the detected quadrilateral shapes. In this case, the critical pixel values ​​include a second critical pixel value and a third critical pixel value. If one of the first regions of interest is a quadrilateral shape, and the pixel value inside the quadrilateral-shaped first region of interest is greater than the second critical pixel value, and the difference between the pixel value inside and outside the quadrilateral-shaped first region of interest is greater than the third critical pixel value, the processor may not detect the quadrilateral-shaped first region of interest as the second region of interest.

[0030] Additionally, the processor can use the first image as input to data learned using machine learning, and detect one or more of the first regions of interest.

[0031] Another embodiment of the present invention provides a method for controlling a home appliance, comprising: receiving an image of a space acquired by a camera; performing a first home appliance action based on a first image in the image, targeting an object in the space that corresponds to the shape of a human body; and performing a second home appliance action, different from the first home appliance action, targeting only a user in the space within the object, based on a second image following the first image in the image.

[0032] Invention Effects

[0033] According to the present invention, it is possible to distinguish whether a human body present in an image is a user (indoor person) or a human body displayed in a picture frame or display device by analyzing the pixel values ​​of the image, thereby enabling more accurate sensing of the user.

[0034] Furthermore, according to the present invention, by more accurately sensing the user within the image, it is possible to provide the user with greater ease of use of home appliances.

[0035] Furthermore, according to the present invention, by accurately sensing the presence of a user in the space, it is possible to prevent errors that occur when providing appliance operation based on the user's location.

[0036] In the following description of specific embodiments, the specific effects of the present invention will be described together with the effects described above. Attached Figure Description

[0037] Figure 1 This is a simplified conceptual diagram illustrating an embodiment of an air conditioner according to the present invention.

[0038] Figure 2 This is a block diagram illustrating the control relationship between the main components of an air conditioner according to an embodiment of the present invention.

[0039] Figures 3 to 7 This is a flowchart illustrating a control method for an air conditioner according to an embodiment of the present invention.

[0040] Figure 8 This is an image illustrating an indoor space according to an embodiment of the present invention.

[0041] Figure 9 and Figure 10 This is a diagram used to illustrate the concept of an artificial neural network (ANN).

[0042] Figure 11 This is a diagram illustrating an example of a difference image of the present invention.

[0043] Figure 12 This is a diagram illustrating an example of a cumulative difference image according to an embodiment of the present invention.

[0044] Figure 13This is a diagram used to illustrate the concept of detecting a second region of interest according to an embodiment of the present invention. Detailed Implementation

[0045] The foregoing objects, features, and advantages will now be described in detail with reference to the accompanying drawings, thereby enabling those skilled in the art to readily implement the technical concept of the present invention. In describing the present invention, detailed descriptions of relevant known technologies will be omitted if they are deemed to obscure the essence of the invention. Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals denote the same or similar constituent elements.

[0046] Although terms like "first," "second," etc., are used to describe multiple constituent elements, these constituent elements are clearly not limited by these terms. These terms are only used to distinguish one constituent element from other constituent elements, and unless otherwise specified, a first constituent element can also be a second constituent element.

[0047] Furthermore, when a constituent element is described as being "connected", "combined", or "linked" with other constituent elements, it should be understood that the constituent elements can be directly connected or linked to each other, or other constituent elements can be "set" between each constituent element, or each constituent element can be "connected", "combined", or "linked" through other constituent elements.

[0048] Throughout this specification, unless otherwise stated, each constituent element may be singular or plural. Unless explicitly stated in the context, the singular expression used herein includes the plural expression. In this application, terms such as “constituting” or “comprising” should not be construed as necessarily including all the various constituent elements or steps described in the specification, but rather as potentially excluding some constituent elements or steps, and potentially including additional constituent elements or steps.

[0049] On the other hand, the present invention can be applied to air conditioners that regulate indoor temperature and purify indoor air. However, the present invention is not limited thereto; it can be applied to all home appliances that perform specific actions based on images acquired by a camera, such as home robots (for example, robotic vacuum cleaners), ovens, refrigerators, water purifiers, dishwashers, washing machines, and air purifiers.

[0050] Here, the image-based control of home appliance actions described in this invention may include on / off control of home appliances, opening and closing control of doors, control of display devices or lighting devices installed on home appliances, and control of the operating range, intensity, and frequency of home appliances.

[0051] For ease of explanation, the household appliances will be limited to air conditioners in the following description of several embodiments of the present invention.

[0052] Figure 1 This is a simplified conceptual diagram illustrating an embodiment of an air conditioner according to the present invention. Figure 2 This is a block diagram illustrating the control relationship between the main components of an air conditioner according to an embodiment of the present invention.

[0053] According to one embodiment of the present invention, the air conditioner 100 illustrates a floor-standing air conditioner. However, the present invention is not limited thereto; in addition to floor-standing air conditioners, the present invention can also be applied to wall-mounted air conditioners, ceiling-mounted air conditioners, and other types of air conditioners.

[0054] Reference Figure 1 and Figure 2 An air conditioner 100 according to an embodiment of the present invention includes a camera 110, a memory 120, a processor 130, a communication unit 140, and an action unit 150.

[0055] Camera 110 acquires images of the space. Camera 110 may be attached to the exterior surface (for example, the front) of the indoor unit of air conditioner 100. The space may be an indoor space, but the invention is not limited thereto.

[0056] On the other hand, an external camera can be installed inside the space. In this case, the image of the space received from the external camera via the communication unit 140 can be received by the air conditioner 100, and the camera 110 may not be included in the air conditioner 100. Hereinafter, for ease of explanation, it will be assumed that the camera 110 is installed in the air conditioner 100.

[0057] The memory 120 may be a volatile and / or non-volatile memory, storing instructions or data related to at least one other component of the air conditioner 100. Additionally, the memory 120 may store images used to control the air conditioner 100. In this case, the images include images acquired by the camera 110 and images processed from the acquired images.

[0058] Processor 130 may include one or more of a central processing unit, an application processor, or a communication processor. Processor 130 may perform calculations or processing of instructions and data relating to control and / or communication of at least one other component of air conditioner 100. In particular, processor 130 receives images acquired from camera 110 and controls actuator 150 based on the received images, thereby controlling airflow (i.e., airflow).

[0059] Multiple modules can be configured within the processor 130.

[0060] Here, a module can refer to the functional and structural combination of software used to implement the technical ideas of this invention. For example, a module can refer to a logical unit of defined code and resources used to execute the defined code.

[0061] As an example, the plurality of modules may include a human body recognition module 131, a region recognition module 132, a signal generation module 133, and a motion control module 134.

[0062] The human body recognition module 131 can detect the presence and location of users (indoor people) in the space based on the images acquired by the camera 110.

[0063] The region recognition module 132 detects one or more regions of interest among a plurality of regions in the space based on the image acquired by the camera 110 and the user's location recognition result. Here, the region of interest corresponds to the living area, which is the area in the space where the user lives.

[0064] The region identification module 132 detects only one or more regions of interest that include the user, removing regions of interest that include objects similar to the user. That is, the region identification module 132 performs the action of determining the region of interest, excluding objects similar to the user and including only the user within the determined region of interest. In other words, "determining the region of interest" corresponds to "detecting the first region of interest that includes the user."

[0065] The signal generation module 133 generates control signals for the action of the control unit 150 based on a defined region of interest.

[0066] The motion control module 134 controls the motion unit 150 based on the control signal generated in the signal generation module 133.

[0067] The operation of the processor 130 that controls the air conditioner 100 will be explained in more detail below.

[0068] The communication unit 140 can communicate with a designated server. In this case, the communication unit 140 may include a mobile communication module, a short-range communication module, etc.

[0069] A mobile communication module is a module that transmits and receives wireless signals on a mobile communication network built according to technical standards or communication methods used for mobile communication. For example, a mobile communication network can be built based on GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), and LTE-A (Long Term Evolution-Advanced).

[0070] The short-range communication module is used for short-range communication and may include at least one of the following technologies: Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus).

[0071] The actuator 150 is a device that performs air conditioning operations and related auxiliary operations. As an example, the actuator 150 may include a fan motor, an airflow adjustment device, a display device, and a lighting device.

[0072] As mentioned above, the action unit 150 can be controlled by the processor 130. The control of the home appliance actions performed by the action unit 150 may include the on and off control of the air conditioner 100, the action control of the display device or lighting device installed on the air conditioner 100, the operating range of the air conditioner 100 (for example, the wind direction and angle), the operating intensity and the number of times it operates, etc.

[0073] Reference Figure 1 The following is a brief explanation of the operation of air conditioner 100.

[0074] The air conditioner 100 may have an intake port (not shown) for drawing in indoor air and an outlet port 190 for expelling the air drawn in through the intake port back into the room.

[0075] The outlet 190 can be equipped with airflow adjustment devices such as louvers (not shown) and blades (not shown) that open and close the outlet 190 and adjust the direction of the discharged air. Under the control of the processor 130, a motor or the like is driven, thereby switching the angle and direction of the airflow adjustment devices. Thus, the airflow direction discharged through the outlet 190 can be adjusted. Furthermore, the airflow direction and range can be adjusted by controlling the operation of the airflow adjustment devices such as louvers and blades.

[0076] For example, the blades are fixed in a certain direction, thus allowing air to be delivered in that direction. Alternatively, the direction of the blades can be continuously varied within a set range (i.e., oscillation), allowing the airflow direction to continuously change. Furthermore, the range of airflow can be adjusted by regulating the angle range of the oscillation motion of the blades.

[0077] On the other hand, inside the indoor unit of the air conditioner 100, there is a fan (not shown) for controlling the flow of indoor air drawn in through the intake port and discharged into the indoor space through the exhaust port 190. The fan is rotated by a fan motor, and the operation of the fan motor is controlled by the processor 130.

[0078] Therefore, the processor 130 can control the direction of airflow (airflow) emitted from the air conditioner 100 by controlling airflow adjustment devices such as louvers and blades, and the fan motor. The processor 130 can control the amount and speed of airflow by controlling the speed of the fan motor, and control the direction of airflow by controlling airflow adjustment devices such as louvers and blades. Furthermore, the processor 130 can control the direction and speed of airflow to rapidly cool or heat specific areas.

[0079] On the other hand, an air conditioner 100 according to one embodiment of the present invention is a device that performs air conditioning operations based on the user's location. As an example, the air conditioner 100 can perform air conditioning operations such as centrally delivering airflow to areas with many users in the space or indirectly delivering airflow to said areas.

[0080] To execute this user-location-based air conditioning action, the user's position within the space needs to be accurately sensed. However, the space may contain items such as picture frames with photos hanging on them or TVs displaying human shapes; these items could be mistakenly sensed as the presence of a user. This missensing could prevent accurate user-based airflow control, potentially causing inconvenience to the user.

[0081] Therefore, an object of an air conditioner 100 according to an embodiment of the present invention is to accurately perform user-based airflow control by accurately sensing a user existing in a human-shaped object in space.

[0082] Hereinafter, with reference to the accompanying drawings, a control method for an air conditioner 100 according to an embodiment of the present invention will be described in detail.

[0083] Figures 3 to 7 This is a flowchart illustrating a control method for an air conditioner 100 according to an embodiment of the present invention.

[0084] At this time, it is assumed that the control method of the air conditioner 100 is executed with the processor 130 as the center, and the camera 110 acquires images of the space in real time or sequentially.

[0085] The following details the process of performing each step.

[0086] First, refer to Figure 3 In step S10, the processor 130 detects the position of the object in space based on the first image acquired by the camera 110.

[0087] Here, the number of first images can be more than one. Furthermore, object recognition includes identifying whether an object exists within an image. Additionally, an object can have a shape corresponding to the shape of a human body. That is, an object can be a user (indoor occupant) existing in an indoor space, or an object existing in an indoor space but not having the shape of a user's human body. Hereinafter, objects that are not users but have the shape of a human body are defined as "user-similar objects."

[0088] According to one embodiment of the present invention, the user-similar object may be a photograph or picture of a human body included inside a picture frame, or a photograph or picture of a human body displayed on a display device (for example, a TV, a monitor, a smartphone, a tablet PC, etc.).

[0089] Next, in step S20, the processor 130 detects one or more first regions of interest among a plurality of regions constituting the space based on the object's position detection results.

[0090] Here, the first region of concern is the region among the plurality of regions constituting the space that is the object of the appliance operation (for example, air conditioning operation) performed by the air conditioner 100, and is the region that includes an image of an object corresponding to the shape of a human body. That is, each first region of concern may include one object.

[0091] Figure 8 This is an image illustrating an indoor space according to an embodiment of the present invention.

[0092] Reference Figure 8 The interior space is a living room, and an air conditioner 100 is installed inside the interior space. Images of the interior space can be acquired by a camera 110 installed on the air conditioner 100.

[0093] at this time, Figure 8 Image (a) shows an image of a user and a TV in a living room. In this case, the processor 130 can detect the regions of the image where the user is located and the regions of the image where the TV is located as the first regions of interest.

[0094] Figure 8 (b) shows an image of a picture frame located in a living room. In this case, the processor 130 can detect the region of the image where the picture frame is located as the first region of interest.

[0095] According to an embodiment of the present invention, steps S10 and S20 utilize an algorithm model based on an artificial neural network to detect the position of the object, and can detect more than one first region of interest.

[0096] Artificial intelligence (AI) is a field of computer engineering and information technology that studies methods that enable computers to think, learn, and self-inspire in the same way that human intelligence can. It refers to the ability of computers to imitate human intelligent behavior.

[0097] Artificial intelligence does not exist in itself, but rather is directly or indirectly connected to many other areas of computer science. Especially in modern times, attempts to introduce elements of artificial intelligence into various fields of information technology in an effort to use it to solve problems in those fields are very active.

[0098] Machine learning is a field of artificial intelligence that is a research area that gives computers the ability to learn without explicit programs.

[0099] Deep learning, a type of machine learning, is a technique that uses data to learn deeply in multiple stages. Deep learning can represent a collection of machine learning algorithms; the higher the stage, the more core data it can extract from a complex dataset.

[0100] Figure 9 and Figure 10 This is a diagram used to illustrate the concept of an artificial neural network (ANN).

[0101] Reference Figure 9 Artificial neural networks can include input layers, hidden layers, and output layers. Each layer consists of multiple nodes, and each layer is connected to the next. Multiple nodes in adjacent layers can be connected to each other by weights.

[0102] Reference Figure 10 The computer (machine) discovers the prescribed pattern from the loaded input data 1010 and forms a feature map. The computer extracts intermediate-level features 1030 and higher-level features 1040 from the lower-level features 1020, thereby recognizing the object and outputting its result 1050.

[0103] The closer an artificial neural network is to the next layer in the sequence, the more abstract it can be to a higher-level feature.

[0104] Reference Figure 8 and Figure 9 Each node can act based on the activation model, and the output value corresponding to the input value can be determined based on the activation model.

[0105] Any node, for example, the output value of lower-level feature 1020, can be input to the next level connected to that node, for example, the node of intermediate-level feature 1030. The node of the next level, for example, the node of intermediate-level feature 1030, can receive a plurality of values ​​output from a plurality of nodes of lower-level feature 1020.

[0106] At this point, the input value of each node can be a value derived from the output values ​​of nodes in previous layers, with the weights applied. Weights can refer to the connection strength between nodes. Furthermore, the deep learning process can also be viewed as a process of finding appropriate weights.

[0107] On the other hand, any node, for example, the output value of intermediate-level feature 1030, can be input to the next layer connected to that node, for example, the node of upper-level feature 1040. The node of the next layer, for example, the node of upper-level feature 1040, can receive a plurality of values ​​output from a plurality of nodes of intermediate-level feature 1030.

[0108] Artificial neural networks can extract feature information corresponding to each level by utilizing learned layers at each level. Artificial neural networks abstract features sequentially, allowing them to identify specific objects using feature information from the highest level.

[0109] For example, if we observe the process of deep learning-based face recognition, the computer distinguishes between bright and dark pixels in the input image based on pixel brightness. After distinguishing simple shapes such as borders and edges, it can distinguish more complex shapes and objects. Ultimately, the computer can master the definition of the human face.

[0110] The deep learning architecture of this invention can utilize various known architectures. For example, the deep learning architecture of this invention can be CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), DBN (Deep Belief Network), etc.

[0111] RNNs are widely used in natural language processing and are an effective structure for processing time-series data that changes over time. They can be stacked in real time to form an artificial neural network structure.

[0112] A Deep Learning Network (DBN) is a deep learning structure constructed by stacking multiple layers of Restricted Boltzmann Machines (RBMs), a deep learning technique. By repeatedly learning from RBMs to form a predetermined number of layers, a DBN with that corresponding number of layers can be constructed.

[0113] CNNs are widely used, especially in the field of object recognition. They are models that mimic the functions of the human brain. These models are based on the assumption that when a person recognizes an object, they extract the basic features of the object and then perform complex calculations in the brain, and recognize the object based on the results.

[0114] On the other hand, artificial neural networks can learn by adjusting the weights of the connections between nodes to produce the desired output for a given input. Furthermore, artificial neural networks can continuously update weight values ​​through learning. Additionally, methods such as backpropagation can be used in the learning process of artificial neural networks.

[0115] In summary, the processor 130 of one embodiment of the present invention can utilize the various deep learning structures described above to detect the positions of more than one object and to detect more than one region of interest. That is, the processor 130 of one embodiment of the present invention uses the first image acquired by the camera 110 as input data for an artificial neural network pre-learned using machine learning, thereby detecting the positions of more than one object in space and more than one first region of interest. For example, the present invention is not limited to this; CNN structures commonly used in in-image object recognition can be utilized.

[0116] Figure 4 This is a flowchart showing the detailed steps of step S20.

[0117] In step S21, the processor 130 receives human body sensing data including the location identification result of the object.

[0118] In step S22, the processor 130 accumulates the received human body sensing data.

[0119] In step S23, the processor 130 determines whether the accumulated human body sensing data has reached a predetermined number M or more by counting the accumulated data.

[0120] When the accumulated data reaches a specified number M or more, in step S24, the processor 130 generates a histogram.

[0121] In step S25, the processor 130 uses the generated histogram as input data for an artificial neural network algorithm based on machine learning, thereby distinguishing multiple regions into living regions and non-living regions.

[0122] Here, the living area corresponds to the first region of interest, and step S25 corresponds to the step of detecting more than one first region of interest. Machine learning can utilize techniques such as SVM (Support Vector Machine) and Adaboost (iterative algorithm), and more preferably, deep learning techniques can be used.

[0123] That is, the processor 130 generates histograms for a plurality of regions, and uses the generated histograms as input data for an artificial neural network that has been learned using machine learning, thereby distinguishing between residential areas and non-residential areas. On the other hand, the processor 130 can further subdivide the residential areas, distinguishing them into permanent residential areas and mobile residential areas.

[0124] More specifically, an air conditioner 100 according to an embodiment of the present invention can distinguish a plurality of areas of a space into living areas and non-living areas based on the frequency of identifying objects (including users), i.e., the number of times an object is sensed per unit time or the time a person is sensed.

[0125] For example, the processor 130 can distinguish and classify the separation intervals in a plurality of regions where an object is sensed at least once and the separation intervals where no object is sensed at all, and can classify the regions where an object is sensed as living regions and the regions where no object is sensed as non-living regions.

[0126] At this point, the criteria for distinguishing between living and non-living areas are not limited to whether indoor people are detected in the area. Obviously, areas where the number of human body detections or the duration of detections are too few to be considered as living areas can also serve as the criteria for distinguishing between living and non-living areas.

[0127] According to an embodiment, the various areas that are divided into living areas can be further divided into permanent living areas and mobile living areas based on the number of times or the duration of continuous sensing of objects.

[0128] For example, the processor 130 classifies the various areas classified as living areas into one of permanent living areas and mobile living areas based on the number of times or the time of continuous sensing of objects. Thus, the living areas can be further divided into permanent living areas and mobile living areas, i.e., two subdivided areas.

[0129] On the other hand, steps S26 and S27 are steps that are performed selectively.

[0130] To further improve the accuracy of distinguishing living areas, in step S26, the processor 130 repeatedly performs the distinction between living and non-living areas. In step S27, the processor 130 integrates multiple distinction results and, based on the integrated results, finally classifies the multiple areas constituting the space into living areas and non-living areas, etc. That is, when the number of living area distinction results accumulates to a predetermined number, the final result is derived, thereby ensuring the reliability of the living area identification result and eliminating transient errors in non-living areas caused by human body sensing errors.

[0131] Refer again Figure 3 In step S30, the processor 130 controls the action unit 150 to perform a first appliance action based on one or more first areas of concern.

[0132] Here, the operation of the home appliance can be an operation related to air conditioning. For example, the operating range, intensity, frequency, and control of the display device or lighting device installed on the air conditioner 100.

[0133] That is, steps S20 and S30 are steps of detecting objects corresponding to the shape of a human body in space based on the first image, and controlling the motion unit 150 to perform the first appliance action with the detected object as the target.

[0134] At this point, the object corresponding to the shape of the human body could be the user or a user-like object. Before performing step S40 as described below, the processor 130 cannot distinguish between the user and a user-like object. Therefore, the processor 130 can perform the first appliance action on all regions of first interest.

[0135] Next, in step S40, the processor 130 detects that the object in one or more first regions of interest is a user's second region of interest based on the second image received after the first image. Here, the number of second images can be multiple.

[0136] That is, step S40 can be a step of removing the first region of interest that includes a user-similar object from one or more first regions of interest detected in step S30, and only detecting the first region of interest that includes the user. In other words, step S40 is a step of determining one or more second regions of interest that are the first regions of interest that include the user from one or more first regions of interest.

[0137] According to an embodiment of the present invention, in step S40, the processor 130 can detect one or more second regions of interest from one or more first regions of interest based on the cumulative difference image of a plurality of second images acquired sequentially.

[0138] Furthermore, in step S50, the processor 130 controls the action unit 150 to perform a second appliance action based on one or more second areas of interest.

[0139] Steps S40 and S50 are steps to select a specific user from an object existing in space based on the second image, and to control the action unit 150 to perform a second appliance action on the specific user.

[0140] Here, the first appliance action and the second appliance action are different from each other. That is, the air conditioner 100 of the present invention is a device that performs air conditioning operations based on the user's location, so the appliance action performed on all first areas of concern (the first appliance action) and the appliance action performed on a portion of the first areas of concern (i.e., the second areas of concern) (the second appliance action) are different from each other.

[0141] In summary, during the first time interval corresponding to steps S20 and S30, the user and similar objects have not yet been distinguished, so the control action unit 150 performs the first appliance action on all objects corresponding to the shape of the human body. Then, during the second time interval corresponding to steps S40 and S50, the user and similar objects are distinguished by analyzing the second image, so the control action unit 150 performs the second appliance action only on the user.

[0142] The following is for reference Figures 5 to 7The process of step S40 will be explained in further detail.

[0143] Figure 5 This is a flowchart illustrating the detailed steps of step S40. At this point, step S40 is performed based on the sequentially acquired second image. In step S41, the processor 130 generates a difference image.

[0144] Here, the difference image is the image corresponding to the difference between the image at the first time point and the image at the second time point preceding the first time point. In this case, the first time point can be the current time point, and the second time point can be a previous time point. That is, step S41 is the step of generating the difference image between the image at the current time point (current image frame) and the image at the previous time point (previous image frame).

[0145] According to one embodiment of the present invention, the difference image may be an image corresponding to the difference between the grayscale image of the image at a first time point and the grayscale image of the image at a second time point. Here, the grayscale image is an image in which the average value of each of the color values ​​R, G, and B is used as the pixel value. In this case, the difference image can be represented by the following mathematical formula 1.

[0146] [Mathematical Expression 1]

[0147] D(x,y)=|I(x,y)-T(x,y)|

[0148] Here, D(x, y) represents the difference image, I(x, y) represents the grayscale image of the image at the first time point, T(x, y) represents the grayscale image of the image at the second time point, and (x, y) represents a specific coordinate within the image.

[0149] Figure 11 This is a diagram illustrating an example of a difference image of the present invention.

[0150] Reference Figure 11 In the difference image, the whiter the pixels are displayed, the greater the difference between the images at the first time point and the images at the second time point. Therefore, the objects corresponding to the two regions displayed in white are moving objects, and thus these two regions correspond to the first region of interest.

[0151] Refer again Figure 5 In step S42, the processor 130 generates a cumulative difference image by accumulating the difference images.

[0152] Here, the cumulative difference image can correspond to the average image of two or more difference images. As an example, the current cumulative difference image can be generated by averaging the previously generated cumulative difference image and the currently generated difference image. The cumulative difference image can be stored in memory 120.

[0153] In step S43, the processor 130 determines whether there are more than n difference images constituting the cumulative difference image. Here, n corresponds to more than two positive numbers.

[0154] If there are two or more difference images less than n, then steps S41 and S42 are executed again. Conversely, if there are two or more difference images greater than n, then the cumulative difference image is determined.

[0155] That is, steps S41 to S43 correspond to the step of generating a cumulative difference image by sequentially acquiring a plurality of second images in camera 110 based on a preset time period. Here, the second images correspond to images acquired after the first image. By using a large number of images to generate the cumulative difference image, the accuracy of user sensing can be improved.

[0156] Figure 12 This is a diagram illustrating an example of a cumulative difference image according to an embodiment of the present invention.

[0157] Figure 12 The cumulative difference image of (a) is relative to Figure 8 The cumulative difference image of image (a), Figure 12 The cumulative difference image of (b) is relative to Figure 8 The cumulative difference image of image (b).

[0158] Refer again Figure 5 In step S44, the processor 130 detects one or more second regions of interest from one or more first regions of interest by comparing the pixel values ​​of one or more first regions of interest in the cumulative difference image with a preset threshold pixel value.

[0159] Here, the critical pixel values ​​include a first critical pixel value, a second critical pixel value, and a third critical pixel value. The first critical pixel value can be a low pixel value, while the second and third critical pixel values ​​can be high pixel values. The first to third critical pixel values ​​can be determined experimentally. Furthermore, the comparison of pixel values ​​can be performed on a per-pixel basis or based on the average of the pixel values ​​existing in the region.

[0160] Figure 6 This is a flowchart illustrating an example of the detailed steps of step S44.

[0161] In step S61, the processor 130 determines whether the pixel value of a certain first region of interest is above the first critical pixel value.

[0162] If the pixel value of a certain first region of interest is less than the first critical pixel value, then in step S62, the processor 130 determines that a certain first region of interest is a first region of interest including the frame.

[0163] Conversely, if the pixel value of a certain first region of interest is above the first critical pixel value, then in step S63, the processor 130 determines whether the pixel value of a certain first region of interest is above the second critical pixel value.

[0164] At this time, if the pixel value of a certain first region of concern is less than the second critical pixel value, then in step S64, the processor 130 determines that a certain first region of concern includes the user's first region of concern, i.e., the second region of concern.

[0165] Conversely, if the pixel value of a certain first region of interest is above the second critical pixel value, the processor 130 determines that the certain first region of interest includes the display device.

[0166] In summary, if the pixel value of a certain first region of interest is greater than a first threshold pixel value and less than a second threshold pixel value, the processor 130 can detect that first region of interest as a second region of interest. A more detailed explanation follows.

[0167] Reference Figure 11 and Figure 12 In a poor image, the whiter the area, the more the object within that area moves; conversely, the blacker the area, the less the object within that area moves.

[0168] The object included in the first region of interest, which has a pixel value less than the first critical pixel value, moves almost no direction. At this time, the photograph or image included in the frame is stationary and therefore does not move. Therefore, the first region of interest with a low pixel value can correspond to the frame or the image or photograph included in the frame.

[0169] The object included in the first region of concern, which has a pixel value greater than a first critical pixel value, exhibits significant movement. Therefore, the first region of concern with a high pixel value can be either the first region of concern including the user (i.e., the second region of concern) or the first region of concern including the display device.

[0170] On the other hand, the movement of an object including the first area of ​​interest of the display device can be greater than the movement of an object including the first area of ​​interest of the user. That is, due to screen switching and other reasons, the human body displayed on the display device is constantly moving more, and therefore moves more than a user existing in an indoor space. Therefore, the first area of ​​interest having a pixel value greater than the second threshold pixel value can correspond to an image or photograph displayed on the display device.

[0171] Therefore, if the pixel value of a certain first region of interest is less than a first threshold pixel value, the processor 130 determines that the certain first region of interest is a photograph or image of a person included in a frame, and can remove the first region of interest corresponding to the frame from the region of interest group. Furthermore, if the pixel value of a certain first region of interest is greater than a second threshold pixel value, the processor 130 determines that the certain first region of interest is a photograph or image displayed on the display device, and can remove the first region of interest corresponding to the display device from the region of interest group. In this case, "removing a certain first region of interest from the region of interest group" corresponds to "determining that a certain first region of interest is not a second region of interest."

[0172] Figure 13 The concept of identifying the first region of concern has been summarized.

[0173] On the other hand, according to another embodiment of the present invention, in step S44, the processor 130 can detect a quadrilateral shape in the cumulative difference image and can detect a second region of interest based on the detected quadrilateral shape.

[0174] Figure 7 This is a flowchart illustrating another example of the detailed steps of step S44.

[0175] In step S71, the processor 130 determines whether a certain first region of interest is quadrilateral in shape.

[0176] If a region of primary concern is not quadrilateral, proceed to steps S72 through S75. Conversely, if a region of primary concern is quadrilateral, proceed to steps S76 and S77.

[0177] Specifically, if a certain first region of interest is not a quadrilateral shape, in step S72, the processor 130 determines whether the pixel value of a certain first region of interest is above the first critical pixel value.

[0178] At this time, if the pixel value of a certain first region of interest is less than the first critical pixel value, in step S73, the processor 130 determines that the certain first region of interest includes the frame.

[0179] In addition, if the pixel value of a certain first region of interest is above the first critical pixel value, in step S74, the processor 130 determines whether the pixel value of a certain first region of interest is above the second critical pixel value.

[0180] If the pixel value of a certain first region of interest is less than the second critical pixel value, then in step S75, the processor 130 determines that a certain first region of interest includes the user's first region of interest, i.e., the second region of interest.

[0181] Conversely, if a pixel value in a first region of interest is above a second critical pixel value, the processor 130 returns to step S71. In this case, it is determined again whether a first region of interest is a second region of interest based on the updated cumulative difference image.

[0182] On the other hand, if a certain first region of interest is quadrilateral in shape, in step S76, the processor 130 determines whether the difference between the pixel values ​​inside and outside a certain first region of interest is greater than or equal to the third critical pixel value.

[0183] If the difference between the inner and outer pixel values ​​of a certain first area of ​​concern is greater than or equal to the third critical pixel value, then in step S77, the processor 130 determines that the certain first area of ​​concern includes the first area of ​​concern of the display device.

[0184] Conversely, if the difference between the pixel values ​​inside and outside a first region of interest is less than a third critical pixel value, the processor 130 returns to step S71. In this case, the processor re-determines whether a first region of interest is a second region of interest based on the updated cumulative difference image.

[0185] The above content will be explained in further detail below.

[0186] Reference Figure 12 The user and the picture frame exist in a non-amorphous form in the cumulative difference image, while the display device exists in a morphous quadrilateral shape in the cumulative difference image.

[0187] Additionally, refer to Figure 12 The user and display device have movement and form in the cumulative difference image (i.e., they are displayed as bright colors in the cumulative difference image), while the frame does not have movement and form in the cumulative difference image (i.e., it is displayed as dark colors in the cumulative difference image).

[0188] Additionally, refer to Figure 12 In (a), the cumulative difference image gradually becomes blurred due to less user movement, while the object displayed on the display device gradually becomes clearer due to screen switching, etc.

[0189] Therefore, if a certain first region of interest in the cumulative difference image is quadrilateral in shape and the pixel value of a certain first region of interest is above the second critical pixel value, the processor 130 can determine that a certain first region of interest is a photograph or picture displayed on the display device.

[0190] Furthermore, in the cumulative difference image, the difference between the pixel values ​​inside and outside the quadrilateral shape is significant. Therefore, if a first region of interest in the cumulative difference image is a quadrilateral shape, and the pixel values ​​inside the quadrilateral shape are greater than a second threshold pixel value, and the difference between the pixel values ​​inside and outside the quadrilateral shape is greater than a third threshold pixel value, the processor 130 can determine that the first region of interest is a photograph or image displayed on the display device. In this case, the pixels outside the quadrilateral shape's first region of interest can be pixels adjacent to the quadrilateral shape's edge.

[0191] In summary, step S40 is a process of determining whether an object within one or more regions of interest is a user based on the movement type of the region of interest within the image. That is, step S40 is a step that retains the region of interest containing the user within the regions of interest contained in the cumulative difference image, and removes the region of interest containing objects similar to the user. Here, the movement type can be determined based on the pixel values ​​and critical pixel values ​​of the region of interest in the cumulative difference image, and can be further determined based on the quadrilateral shape detection results.

[0192] In this scenario, if the movement type of the first area of ​​concern is "a first movement type with almost no movement," the processor 130 can determine that the first area of ​​concern is a photograph or image of a person included in a frame, and can remove the first area of ​​concern corresponding to the frame. Alternatively, if the movement type of the first area of ​​concern is "a second movement type with significant movement," the processor 130 can determine that the first area of ​​concern is a photograph or image displayed on the display device, and can remove the first area of ​​concern corresponding to the display device. Furthermore, if the movement type of the area of ​​concern is "a third movement type with appropriate movement," the processor 130 determines that the first area of ​​concern is the user's first area of ​​concern, i.e., the second area of ​​concern, and can retain the user's first area of ​​concern.

[0193] On the other hand, according to an embodiment of the present invention, in step S50, the processor 130 can control the action unit 150 to generate airflow by distinguishing between the living area (i.e., the second area of ​​concern) and the unliving area. At this time, the processor 130 can control the intensity of the airflow by controlling the fan motor, and can control the direction and range of the airflow by controlling the airflow direction adjustment device such as blades and louvers provided at the outlet 190.

[0194] For example, the air conditioner 100 can operate in a centralized cooling and heating mode for centrally transmitting airflow to the permanent living area in the living area, a direct cooling and heating mode for transmitting airflow to both the permanent living area and the mobile living area in the living area, and an indirect cooling and heating mode for preventing the direct transmission of airflow to the permanent living area in the living area.

[0195] In summary, the air conditioner 100 of one embodiment of the present invention can detect a first area of ​​concern (i.e., a second area of ​​concern) including the user based on the movement type of the area of ​​concern. Specifically, the air conditioner 100 of one embodiment of the present invention detects one or more first areas of concern including objects corresponding to the shape of a human body, removes first areas of concern including objects similar to the user based on the movement type of the one or more first areas of concern, and detects only the first areas of concern including the user (i.e., the second area of ​​concern). At this time, the air conditioner 100 can be controlled based on the area of ​​concern including the user, thereby realizing user-based appliance operation.

[0196] In addition, the air conditioner 100 of one embodiment of the present invention can distinguish whether the human body in the image is a user (indoor person) or a human body displayed in a picture frame or display device by analyzing the cumulative difference image of the sequentially acquired images, thereby more accurately sensing the user.

[0197] Furthermore, the air conditioner 100 of one embodiment of the present invention can provide users with greater ease of use of home appliances by more accurately sensing users within an image. Additionally, the air conditioner 100 of one embodiment of the present invention can prevent errors that occur when providing home appliance operation based on user location by accurately sensing users present in the space.

[0198] Furthermore, embodiments of the present invention can be implemented in the form of program commands executable by various computer means and recorded in a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., individually or in combination. The program commands recorded on the medium may be configured specifically for the present invention or may be program commands known to computer software practitioners. Examples of computer-readable recording media include: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks; and hardware devices specifically configured to store and execute program commands, such as read-only memory (ROM), random access memory (RAM), and flash memory. Examples of program commands include: machine language code created by a compiler and high-level language code executable by a computer using an interpreter. To perform the operations of an embodiment of the present invention, the aforementioned hardware device may be configured to operate as one or more software modules, or vice versa.

[0199] As described above, the present invention has been illustrated through specific details of its constituent elements, defined embodiments, and accompanying drawings. However, this is merely provided to aid in a comprehensive understanding of the invention. The invention is not limited to the above embodiments, and those skilled in the art can make various modifications and variations based on the foregoing description. Therefore, the concept of the invention is not limited to the above embodiments, and the following claims and their equivalents or variations fall within the scope of the present invention.

Claims

1. A household appliance, wherein, include: Memory, storing at least one instruction; The processor executes the at least one instruction; as well as The actuator, controlled by the processor, performs the actions of the home appliance. The processor receives spatial images acquired by the camera. The processor controls the action unit to execute a first home appliance action, which is based on a first image in the image and is executed on an object in the space that corresponds to the shape of a human body. The processor controls the action unit to execute a second home appliance action that is different from the first home appliance action. The second home appliance action is based on a second image following the first image in the image, and is executed only for the user within the space of the object. The processor controls the action unit to detect, based on the first image, one or more first regions of interest comprising the object among a plurality of regions constituting the space, and to perform the first appliance action based on one or more first regions of interest. The processor controls the action unit to generate a cumulative difference image based on the second images sequentially acquired by the camera during a preset time period, and to detect, based on the cumulative difference image, one or more first regions of interest where the object is one or more second regions of interest of the user, and to perform the second appliance action based on one or more second regions of interest. The processor detects one or more second regions of interest by comparing the pixel values ​​of the first region of interest with a preset threshold pixel value in the cumulative difference image.

2. The household appliance according to claim 1, wherein, The objects include the user and user-similar objects. The user-similar object includes photographs or pictures of a human body in a frame and photographs or pictures of a human body displayed on a display device.

3. The household appliance according to claim 1, wherein, The cumulative difference image corresponds to the average image of the complex difference images. The difference image is the image corresponding to the difference between the image at a first time point and the image at a second time point, which is a time point prior to the first time point.

4. The household appliance according to claim 3, wherein, The difference image is the image corresponding to the difference between the grayscale image of the image at the first time point and the grayscale image of the image at the second time point.

5. The household appliance according to claim 1, wherein, The critical pixel values ​​include a first critical pixel value and a second critical pixel value. If the pixel value of one of the first regions of interest is greater than the first critical pixel value and less than the second critical pixel value, the processor detects the first region of interest as the second region of interest.

6. The household appliance according to claim 1, wherein, The critical pixel values ​​include a first critical pixel value and a second critical pixel value. If the pixel value of one of the first regions of interest is less than the first critical pixel value or greater than the second critical pixel value, the processor will not detect the first region of interest as the second region of interest.

7. The household appliance according to claim 6, wherein, If the pixel value of a certain first region of interest is less than the first critical pixel value, the processor will determine that the object included in the certain first region of interest is a photograph or image of a human body included in a frame. If the pixel value of a certain first region of interest is greater than the second critical pixel value, the processor determines that the object included in the certain first region of interest is a photograph or picture to be displayed on the display device.

8. The household appliance according to claim 1, wherein, The processor detects quadrilateral shapes in the cumulative difference image and further detects the second region of interest based on the detected quadrilateral shapes.

9. The household appliance according to claim 8, wherein, The critical pixel values ​​include a second critical pixel value and a third critical pixel value. If, in one or more first regions of interest, a first region of interest is quadrilateral in shape, and the pixel value inside the quadrilateral first region of interest is greater than the second critical pixel value, and the difference between the pixel value inside the quadrilateral first region of interest and the pixel value outside the quadrilateral first region of interest is greater than the third critical pixel value, the processor will not detect the first region of interest as the second region of interest.

10. The household appliance according to claim 1, wherein, The processor uses the first image as input to data learned using machine learning and detects one or more of the first regions of interest.

11. The household appliance according to claim 1, wherein, It also includes the camera; The camera is mounted on the exterior of the home appliance.

12. A method for controlling a home appliance, the home appliance comprising a processor, wherein, The control method includes: The steps for receiving spatial images acquired by the camera; Based on the first image in the image, the step of performing a first home appliance action on an object in the space that corresponds to the shape of a human body; and Based on the second image following the first image in the image, the steps of performing a second home appliance action, different from the first home appliance action, are performed only on the user within the space of the object. The step of performing the first appliance action includes detecting, based on the first image, one or more first regions of interest comprising the object among a plurality of regions constituting the space, and performing the first appliance action based on one or more first regions of interest. The steps of performing the second appliance action include generating a cumulative difference image based on the second images sequentially acquired by the camera during a preset time period, detecting, based on the cumulative difference image, that the object in one or more first regions of interest is one or more second regions of interest of the user, and performing the second appliance action based on one or more second regions of interest. Furthermore, in the step of performing the second appliance operation, one or more second regions of interest are detected by comparing the pixel values ​​of the first region of interest with a preset threshold pixel value in the accumulated difference image.

Images