Object recognition method for refrigerator, server and refrigerator

Abstract

An object recognition method for a refrigerator is disclosed. The disclosed object recognition method for a refrigerator includes the following steps: acquiring a captured image of a storage compartment of the refrigerator; when a change in the captured image is confirmed compared to a previously stored image, checking a change in an imaging direction of an image capture device that has captured an image of the storage compartment; and when the imaging direction is maintained, performing an object recognition operation of the captured image.

Description

Technical Field

[0001] The present disclosure relates to a refrigerator, a server, and an object recognition method thereof, and more particularly, to a refrigerator, a server, and an object recognition method of a refrigerator for recognizing a change in an image capturing direction by comparing an original image of the refrigerator with a currently captured image. Background Art

[0002] A refrigerator may refer to an electronic device (or household appliance) for keeping edible and drinkable food refrigerated or frozen by using a refrigeration cycle of a refrigerant. In addition to food, a refrigerator may also store medicines, alcoholic beverages, or cosmetics.

[0003] In recent years, in order to manage food stored in a refrigerator, a method for capturing an image of a refrigerator storage compartment and performing object recognition on the stored food by using the captured image has been applied. To this end, the refrigerator can capture an image of the storage compartment by using a camera attached to the front surface of the door.

[0004] However, since the door is opened and closed many times by the user, impact may be applied to the door, which may cause a change in the image capturing direction of the camera attached to the front surface of the door. When the camera captures an image of the storage room while the image capturing direction is changed, additional object recognition may be performed by recognizing a significant change between the original image and the currently captured image, although there is no actual change in the food stored in the storage room. In this case, the above-mentioned additional object recognition has a problem of unnecessary cost increase, which is attributed to the cost caused by resource usage when performing object recognition. Summary of the invention

[0005] Technical issues

[0006] An object of the present disclosure is to provide a refrigerator, a server, and an object recognition method of a refrigerator which recognizes a change in an image capturing direction by comparing an original image of the refrigerator with a currently captured image.

[0007] Technical Solution

[0008] According to one aspect of the present disclosure, there is provided an object recognition method for a refrigerator, the method comprising: acquiring a captured image by capturing a storage room of the refrigerator; recognizing a change in an image capturing direction of an image capture device that photographed the storage room based on changes in the captured image compared to a pre-stored original image; and performing an object recognition operation on the captured image based on the image capturing direction being maintained.

[0009] According to another aspect of the present disclosure, a refrigerator is provided, comprising: a storage chamber configured to store food; an image capture device configured to capture an image of the storage chamber; a memory configured to store an original image previously captured by the image capture device; and a processor configured to acquire a captured image by capturing an image of the storage chamber via controlling the capture device, identify whether an image capture direction of the image capture device is changed based on a change in the captured image compared to the original image, and perform an object recognition operation on the captured image based on the image capture direction being maintained.

[0010] According to another aspect of the present disclosure, a server is provided, comprising: a memory configured to store an original image previously received from a refrigerator; a communication device configured to receive a captured image obtained by photographing a storage chamber of the refrigerator from the refrigerator, identify changes in the captured image compared to the original image based on identifying whether an image capture direction of the captured image has changed, and perform an object recognition operation on the captured image based on the image capture direction being maintained.

[0011] Effects of the Invention BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram for illustrating a specific configuration of a refrigerator according to an embodiment,

[0013] Figure 2 is a block diagram showing a specific configuration of a refrigerator according to an embodiment,

[0014] Figure 3 is a block diagram showing a specific configuration of a server according to an embodiment,

[0015] Figure 4 is a block diagram showing a specific configuration of a server according to an embodiment,

[0016] Figure 5 is a diagram showing an example in which a change in an image capturing direction occurs according to an embodiment,

[0017] Figures 6 to 9 is a diagram showing an example of a method for identifying a change in an image capturing direction using a plurality of feature points in an image according to an embodiment,

[0018] Fig.10 is a diagram showing an example of a method for recognizing a change in an image capturing direction using an artificial intelligence model according to an embodiment,

[0019] Fig.11 is a flowchart for illustrating an object recognition method according to an embodiment,

[0020] Fig.12 is a flowchart for specifically illustrating an object recognition method according to an embodiment,

[0021] Fig.13 is a sequence diagram for illustrating an object recognition method using a refrigerator and a server of the first embodiment, and

[0022] Fig.14 is a sequence diagram for illustrating an object recognition method using a refrigerator and a server according to a second embodiment. DETAILED DESCRIPTION

[0023] Detailed Description of Exemplary Embodiments

[0024] After briefly explaining terms used in the present disclosure, the present disclosure will be described in more detail below.

[0025] Considering the functions in the present disclosure, the terms used in the embodiments of the present disclosure are selected as general terms that are used as widely as possible, but these terms may change according to the intention of those skilled in the art, precedents, the emergence of new technologies, etc. In addition, in a certain case, there may also be arbitrarily selected terms, in which case the meaning will be described in the description of the present disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the terms themselves and the content throughout the present disclosure rather than based on the simple names of the terms.

[0026] The embodiments of the present disclosure may be variously changed and include various embodiments, and specific embodiments will be shown in the drawings and described in detail in the specification. However, it should be understood that this is not intended to limit the scope of the specific embodiments, and all modifications, equivalents and / or alternative forms included in the disclosed spirit and technical scope are included. In describing the present disclosure, when it is determined that the detailed description may unnecessarily obscure the main points of the present disclosure, the detailed description of the related art may be omitted.

[0027] The terms "first", "second", etc. may be used to describe various elements, but these elements may not be limited by the terms. The terms are only used to distinguish one element from another.

[0028] Unless otherwise specifically specified, singular expressions may encompass plural expressions. It will be understood that terms such as "including" or "consisting of" are used herein to indicate the presence of features, quantities, steps, operations, elements, parts, or combinations thereof, without excluding the presence or possibility of adding one or more of other features, quantities, steps, operations, elements, parts, or combinations thereof.

[0029] Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail to enable those skilled in the art to easily practice the embodiments. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the accompanying drawings, parts not related to the description are omitted in order to clearly describe the present disclosure.

[0030] Hereinafter, the present disclosure will be described in more detail with reference to the accompanying drawings.

[0031] Figure 1 is a block diagram for illustrating a specific configuration of a refrigerator according to an embodiment.

[0032] Reference Figure 1 , the refrigerator 100 may include a storage chamber 110 , an image capturing device 120 , a memory 130 , and a processor 140 .

[0033] The storage chamber 110 may be a place for storing food arranged inside the refrigerator 100. The storage chamber 110 may be used as a refrigerating chamber maintained at about 0 to 5 degrees Celsius to cool and store food and a freezing chamber maintained at about minus 30 to 0 degrees Celsius to freeze and store food.

[0034] The storage chamber 110 may have an open front surface so that food is put in and taken out, and its open front surface may be opened and closed by a door (not shown). A shelf for placing food may be provided in the storage chamber 110 .

[0035] The image capture device 120 may acquire a captured image of a predetermined area corresponding to the image capture direction. Specifically, the image capture device 120 may be disposed on a side surface of the door facing the storage room 110 to have an image capture direction toward the storage room 110. In addition, the image capture device 120 may acquire a captured image of a predetermined area in the storage room 110 corresponding to the image capture direction.

[0036] The image capture direction here may be the direction faced by the lens of the image capture device 120. The image formed by the lens may change according to the change of the image capture direction. Therefore, when the image capture direction changes, the predetermined area corresponding thereto may also change. The predetermined area here may refer to a unique detection area that can be detected by the image capture device 120 through the lens.

[0037] The number of image capture devices 120 may be more than one, and the plurality of image capture devices may acquire captured images of a plurality of predetermined areas respectively corresponding to a plurality of image capture directions. Specifically, the image capture devices 120 may be arranged at predetermined intervals on the one side surface of the door facing the storage room 110, and may acquire captured images of the plurality of predetermined areas in the storage room 110 respectively corresponding to the plurality of image capture directions.

[0038] When a predetermined event occurs, the image capture device 120 may acquire a capture image in a predetermined area. The predetermined event here may correspond to an opening event of the refrigerator 100, a predetermined cycle arrival event, or a door closing event of the refrigerator 100, and is not limited to the above examples.

[0039] The memory 130 may store various pieces of data for general operation of the refrigerator 100, such as a program for processing or controlling the processor 140. Specifically, the memory 130 may store a plurality of application programs running in the refrigerator 100 and data and instructions for operation of the refrigerator 100.

[0040] In addition, the memory 130 can be accessed by the processor 140, and data reading, recording, editing, deletion or updating by the processor 140 can be performed. In addition to the storage medium in the refrigerator 100, the memory 130 can also be implemented as a removable disk including an external storage medium or a USB memory, a network server via a network, etc.

[0041] The memory 130 may store an image captured by the image capturing device 120. Specifically, the memory 130 may store an image acquired by capturing the interior of the storage room 110 by the image capturing device 120.

[0042] The processor 140 may perform control for each element in the refrigerator 100. Specifically, an instruction regarding a specific function is received, and the processor 140 may control the operation of the elements related to the corresponding function.

[0043] The processor 140 may perform object recognition on the food contained in the storage chamber 110 of the refrigerator. Specifically, the processor 140 may perform object recognition using a captured image of the storage chamber 100 captured by the image capture device 120. For example, the processor 140 may analyze the captured image to recognize the type of food contained in the storage chamber 110, the amount of food, the storage time point, expiration date information, etc.

[0044] However, a considerable amount of operations is required if the processor 140 performs object recognition for all captured images captured by the image capture device 120. Therefore, the processor 140 may determine whether it is necessary to perform object recognition for each captured image before performing object recognition for the current captured image.

[0045] In the related art, the processor 140 compares the images captured in sequence, and when it is recognized that a change region is generated in the captured image, the processor determines that a change of the object has occurred and determines to perform object recognition. In addition, when it is not recognized that no change region is generated, the processor 140 recognizes that no change of the object has occurred and determines not to perform object recognition.

[0046] However, the user may apply an impact to the image capture device 120 by opening or closing the door of the refrigerator multiple times, and the image capture direction of the image capture device 120 may be slightly changed accordingly. In this case, although the food contained in the storage chamber 110 does not actually change, a change area may be generated in the captured image due to the change in the image capture direction.

[0047] In other words, there is a problem in that although the result of object recognition with respect to each image captured in sequence is calculated in the same manner, the processor 140 determines that it is necessary to perform object recognition.

[0048] For example, refer to Figure 5 , an example in which a change in the image capturing direction has occurred can be identified. Specifically, referring to Figure 5 (a) and Figure 5 From the image (b), it can be recognized that pineapples, cans, grapes, and apples are contained in the storage room 110, and there is no actual change in the quantity or position of each object. Figure 5 (a) Object recognition results and Figure 5 The object recognition result of (b) can be the same.

[0049] However, it can be identified Figure 5 (b) The image capture direction is Figure 5 The image capture direction of (a) is slightly different. For example, Figure 5 In (b), it is recognized that the right edge of the bowl with apples is not captured. If the image capture direction is changed as described above, when Figure 5 Each pixel of the image (a) is Figure 5 (b) of the image corresponding to each pixel, the change between the values ​​occurs. Therefore, Figure 5 (a) and Figure 5 (b) are determined to be different from each other, thereby performing unnecessary object recognition.

[0050] To solve such a problem, the processor 140 may determine whether to perform object recognition by additionally identifying whether the image capturing direction changes. Specifically, the processor 140 may determine whether to perform object recognition by additionally identifying whether the object capturing the image moves and whether the image capturing direction changes.

[0051] More specifically, the processor 140 can analyze the following situations corresponding to each captured image: the object moves and the image capturing direction changes (①), the object moves but the image capturing direction does not change (②), the object does not move but the image capturing direction changes (③), and neither the object moves nor the image capturing direction changes (④).

[0052] In addition, if the captured image corresponds to the cases ① and ② among the above-mentioned multiple cases, the processor 140 may determine to perform object recognition because the object movement occurs. If the captured image corresponds to the case ④, the processor 140 may determine not to perform object recognition because no change occurs in the captured image. In addition, if the captured image corresponds to the case ③, the processor 140 may determine not to perform object recognition because no substantial movement of the object occurs.

[0053] In the related art, although the captured image corresponds to case ③, the processor 140 performs object recognition, and in the present disclosure, if the captured image corresponds to case ③, the processor 140 may determine not to perform object recognition, thereby exhibiting an effect of reducing operations due to additional object recognition. Specific operations of the processor 140 regarding determining whether it is necessary to perform object recognition will be described below.

[0054] First, the processor 140 may acquire a capture image by controlling the image capture device 120. Specifically, the processor 140 may acquire a capture image by controlling the image capture device 120 to capture an image of the interior of the storage room 110. The processor 140 may store the acquired capture image in the memory 130.

[0055] In addition, the processor 140 can identify whether a change area of ​​the captured image is generated by comparing the acquired captured image with the original image. The original image here may refer to a captured image acquired by previously capturing an image of the inside of the storage room 110 by the image capture device 120. In addition, the original image may be the most recently acquired image among the previously acquired multiple captured images.

[0056] Specifically, the processor 140 may detect a pixel value change region by comparing each pixel value of the original image with each pixel value of the captured image corresponding thereto to identify whether a change region of the captured image is generated.

[0057] If no pixel value change area is detected, this may mean that the original image is the same image as the captured image. In other words, this means that this situation corresponds to the situation (④) among the above four situations where no object movement or change in image capture direction occurs, and therefore, the processor 140 may determine not to perform object recognition on the captured image.

[0058] On the other hand, if a pixel value change area is detected, the processor 140 may identify the size of the detected area and determine whether to perform object recognition or identify whether the image capturing direction is additionally changed according to whether the size of the detected area exceeds a predetermined size.

[0059] Specifically, if the size of the change region does not exceed the predetermined size, this means that the change region is generated only in a local area from the captured image. If a change in the image capture direction occurs, the change region of the captured image is generated over the entire area rather than in the local area. Therefore, if the size of the change region does not exceed the predetermined size, the processor 140 can determine that the image capture direction has not changed, and determine that the generated change region is attributed to the movement of the object.

[0060] In other words, if the size of the change area does not exceed the predetermined size, the pixel value change area of ​​the captured image is caused by the actual movement of the object, and therefore, the captured image may correspond to the case (②) in which the object moves but the image capture direction does not change among the above four types of cases. Therefore, the processor 140 may determine to perform object recognition on the captured image.

[0061] As described above, if the size of the change region does not exceed a predetermined size, the processor 140 may determine to perform object recognition without recognizing a change in the image capturing direction, thereby exhibiting an effect of reducing operations.

[0062] On the other hand, if the size of the change area exceeds a predetermined size, not only an actual change of the object but also a change in the image capturing direction may have occurred, and thus, the processor 140 may additionally identify whether the image capturing direction has changed.

[0063] If it is recognized that the image capturing direction has not changed, the pixel value change area of ​​the captured image is caused by the actual movement of the object, and therefore, this may correspond to a case where a change has actually occurred in the food contained in the storage chamber 110. In other words, the captured image corresponds to a case (②) in which the object has moved but the image capturing direction has not changed among the above four types of cases, and therefore, the processor 140 may determine to perform object recognition on the captured image.

[0064] On the other hand, if the image capturing direction is changed, the processor 140 may determine to perform object recognition according to whether the object moves. Specifically, the processor 140 may determine to perform object recognition when the object moves and the image capturing direction does not change (①).

[0065] On the other hand, the situation (③) in which no object movement occurs but a change in the image capturing direction occurs means that the pixel value change area of ​​the captured image is not caused by an actual change in the food contained in the storage chamber 110, but is caused by a change in the image capturing direction, and therefore, the processor 140 can determine not to perform object recognition.

[0066] Meanwhile, a detailed description on recognizing whether a change in the image capturing direction and movement of the object has occurred will be referred to later. Figures 6 to 10 to describe.

[0067] The processor 140 may store the captured image in the memory 130 by replacing the original image stored in the memory 130 , and may use the captured image as a comparison target when determining whether to perform object recognition on an image captured later.

[0068] In addition, when it is determined to perform object recognition, the processor 140 may perform object recognition using the captured image. The processor 140 may display the object recognition result to the user via the display.

[0069] On the other hand, if it is determined that object recognition is not performed, the processor 140 can reuse the result of the object recognition previously performed with the original image. In other words, when it is determined that no change between the captured image and the original image has occurred or it is determined that a change in the image capturing direction has occurred, the processor 140 can use the object recognition result on the original image.

[0070] At the same time, in showing and describing Figure 1 In the embodiment, the image capture device is shown and described as being provided in the refrigerator, but in the embodiment, the image capture device may be provided as a device separate from the refrigerator. In this case, the image capture device may be implemented by the following method: capturing an image of the storage compartment of the refrigerator by attaching it to the refrigerator, sending the captured image to the refrigerator via wired or wireless communication, and performing object recognition using the captured image received by the refrigerator. In addition, the image capture device may store all original images and captured images obtained by photographing the storage compartment of the refrigerator, and may directly identify whether the image capture direction of the captured image has changed. The image capture device may perform an object recognition operation based on the result of the change in the image capture direction. In addition to the image capture device, the above operations may also be performed on other external devices.

[0071] In the above, only a brief configuration of configuring a refrigerator is illustrated and described, but various elements may be additionally provided in the embodiment. This will be referred to below. Figure 2 to describe.

[0072] Figure 2 is a block diagram showing a detailed configuration of a refrigerator according to an embodiment.

[0073] Reference Figure 2 , a refrigerator 100 according to an embodiment of the present disclosure may be configured with a storage chamber 110 , an image capturing device 120 , a memory 130 , a processor 140 , a communication device 150 , an input device 160 , and a display 170 .

[0074] The storage room 110, the image capture device 120, the memory 130 and the processor 140 perform the same Figure 1 The elements have the same function, so overlapping descriptions will not be repeated.

[0075] The communication device 150 can be connected to an external device (not shown) to send and receive various pieces of data to and from the external device. Specifically, in addition to connection with the external device via a local area network (LAN) or the Internet, the communication device 150 can also be connected via a universal serial bus (USB) port or a wireless communication (e.g., Wi-Fi 802.11a / b / g / n, near field communication (NFC) or Bluetooth) port. The external device can be a PC, a laptop, a smart phone, a server, etc.

[0076] The communication device 150 may transmit the image captured by the image capture device 120 to an external device. Specifically, the communication device 150 may transmit a captured image of a predetermined area in the storage room 110 to an external device. The external device may perform object recognition using the received captured image and transmit the object recognition result to the communication device 150 again.

[0077] In this case, the processor 140 may display the object recognition result received from the external device to the user via the display 170 without directly performing object recognition on the captured image. Meanwhile, a detailed description of the operation of acquiring the object recognition result through communication with the external device will be referred to later. Fig.13 and Fig.14 to describe.

[0078] The input device 160 may include a plurality of function keys for setting or selecting various functions supported by the refrigerator 100 by the user. Thereby, the user may input various operation commands regarding the refrigerator 100. For example, the user may input a command regarding the cooling intensity of the refrigerator via the input device 160.

[0079] In addition, the input device 160 may receive information about the food contained in the storage chamber 110 as user feedback about the object recognition result. For example, if the user puts newly purchased grapes into the storage chamber 110, the grapes may be newly identified as the object recognition result, and the user may input information related to the food, such as expiration date information about the newly identified grapes.

[0080] The display 170 may display various information provided from the refrigerator 100. Specifically, the display 170 may display an operating state of the refrigerator 100 or display a user interface window for a function or option selection selected by a user. The display 170 may be implemented in the form of a touch screen for simultaneously performing the function of the input device 160.

[0081] In addition, the display 170 may display the result of object recognition performed by the processor 140 or the result of object recognition of an external device received via the communication device 150. For example, information about the food name, food category, initial accommodation time point, accommodation location, etc. of grapes accommodated in the storage chamber 110 may be displayed.

[0082] In the related art, even when the image capturing direction of the image capturing device is changed, the refrigerator performs object recognition for the captured image. In this case, although the food contained in the storage room does not actually change, there is a problem that a large number of operations are required due to the additional object recognition of the refrigerator caused by the change in the captured image.

[0083] In contrast, as described above, the refrigerator according to the embodiment can identify whether the image capturing direction has changed by comparing the captured image and the original image, and when it is identified that only a change in the image capturing direction has occurred, the refrigerator can determine not to perform additional object recognition, thereby exhibiting the effect of reducing operations.

[0084] Figure 3 is a block diagram showing a specific configuration of a server according to an embodiment.

[0085] Reference Figure 3 , the server 200 may include a memory 210 , a communication device 220 , and a processor 230 .

[0086] The memory 210 may store various pieces of data for general operation of the server 200, such as a program for processing or control of the processor 230. Specifically, the memory 210 may store a plurality of application programs running in the server 200 and data and instructions for operation of the server 200.

[0087] In addition, the memory 210 can be accessed by the processor 230, and data reading, recording, editing, deleting or updating by the processor 230 can be performed. In addition to the storage medium in the server 200, the memory 210 can also be implemented as a removable disk including an external storage medium or a USB memory, a network server via a network, etc.

[0088] In addition, the memory 210 may store a captured image received via the communication device 220. Specifically, the memory 210 may store a captured image of a predetermined area in the storage chamber received from the refrigerator 100 via the communication device 220.

[0089] The communication device 220 can be connected to an external device (not shown) to send various pieces of data to the external device and receive various pieces of data from the external device. Specifically, in addition to the connection with the external device via a local area network (LAN) or the Internet, the communication device 220 can also be connected via a universal serial bus (USB) port or a wireless communication (e.g., Wi-Fi 802.11a / b / g / n, near field communication (NFC) or Bluetooth) port. The external device can be a refrigerator, a PC, a laptop, a smart phone, a server, etc.

[0090] In addition, the communication device 220 may receive a captured image of a predetermined area in the storage room from the refrigerator 100 .

[0091] The processor 230 may perform control of each element in the server 200. Specifically, when a command regarding a specific function is received, the processor 230 may control the operation of the elements related to the corresponding function.

[0092] In addition, the processor 230 may use the received captured image to perform object recognition on the food contained in the refrigerator 100. However, when the processor 230 performs object recognition for all received captured images, a considerable amount of operation is required. Therefore, the processor 230 may determine whether it is necessary to perform object recognition before performing object recognition for the currently received captured image.

[0093] Specifically, the processor 230 may determine whether to perform object recognition by recognizing both movement of an object capturing an image and a change in an image capturing direction.

[0094] More specifically, the processor 230 can respectively select the following situations corresponding to each captured image: the object moves and the image capturing direction changes (①), the object moves but the image capturing direction does not change (②), the object does not move but the image capturing direction changes (③), and neither the object moves nor the image capturing direction changes (④).

[0095] In addition, if the captured image corresponds to the cases ① and ② among the above-mentioned multiple cases, the processor 230 may determine to perform object recognition because object movement occurs. If the captured image corresponds to the case ④, the processor 230 may determine not to perform object recognition because no change occurs in the captured image. In addition, if the captured image corresponds to the case ③, the processor 230 may determine not to perform object recognition because no substantial movement of the object occurs. The specific operation of the processor 230 regarding determining whether it is necessary to perform object recognition will be described below.

[0096] First, the processor 230 may acquire a captured image of the refrigerator 100 via the communication device 220. The processor 230 may store the acquired captured image in the memory 210.

[0097] In addition, the processor 230 can identify whether a change in the captured image is detected by comparing the acquired captured image with the original image. The original image here may refer to a captured image acquired by capturing an image of the interior of the refrigerator 100 received in advance via the communication device 220. In addition, the original image may be the most recently acquired image among the plurality of previously received captured images.

[0098] Specifically, the processor 230 may detect a pixel value change region by comparing each pixel value of the original image with each pixel value of the captured image corresponding thereto, thereby identifying whether a change of the captured image has occurred.

[0099] If no pixel value change area is detected, this means that this situation corresponds to the situation (④) among the above four situations where no object movement and no change in image capture direction occurs. Therefore, the processor 230 can determine not to perform object recognition on the captured image.

[0100] On the other hand, if a pixel value change area is detected, the processor 230 may identify the size of the detected area and determine whether to perform object recognition or identify whether the image capturing direction is additionally changed according to whether the size of the detected area exceeds a predetermined size.

[0101] Specifically, if the size of the change region does not exceed the predetermined size, this means that the change region is generated only in a local area from the captured image. If a change in the image capture direction occurs, the change region of the captured image is generated over the entire area rather than in the local area. Therefore, if the size of the change region does not exceed the predetermined size, the processor 230 can determine that there is no change in the image capture direction, and determine that the generated change region is attributed to the movement of the object.

[0102] In other words, if the size of the change area does not exceed the predetermined size, the pixel value change area of ​​the captured image is caused by the actual movement of the object, and therefore, the captured image may correspond to the case (②) in which the object moves but the image capture direction does not change among the above four cases. Therefore, the processor 230 may determine to perform object recognition on the captured image.

[0103] As described above, if the size of the change region does not exceed a predetermined size, the processor 230 may determine to perform object recognition without recognizing a change in the image capturing direction, thereby exhibiting an effect of reducing operations.

[0104] On the other hand, if the size of the change area exceeds a predetermined size, not only an actual change of the object but also a change in the image capturing direction may have occurred, and thus, the processor 230 may additionally identify whether the image capturing direction has changed.

[0105] If it is recognized that the image capturing direction has not changed, the pixel value change area of ​​the captured image is caused by the actual movement of the object, and therefore, this may correspond to a case where a change has actually occurred in the food contained in the refrigerator 100. In other words, the captured image corresponds to a case (②) in which the object has moved but the image capturing direction has not changed among the above four types of cases, and therefore, the processor 230 may determine to perform object recognition on the captured image.

[0106] On the other hand, if the image capturing direction is changed, the processor 230 may determine to perform object recognition according to whether the object moves. Specifically, the processor 230 may determine to perform object recognition when both the object moves and the image capturing direction changes (①).

[0107] On the other hand, the situation (③) in which no object movement occurs but a change in the image capturing direction occurs means that the pixel value change area of ​​the captured image is not caused by an actual change in the food contained in the refrigerator 100, but is caused by a change in the image capturing direction, and therefore, the processor 230 can determine not to perform object recognition.

[0108] Meanwhile, a detailed description of recognizing whether a change in the image capturing direction and movement of the object has occurred will be referred to later. Figures 6 to 10 to describe.

[0109] In this case, the processor 230 may store the captured image in the memory 210 by replacing the original image stored in the memory 210, and may use the captured image as a comparison target when determining whether to perform object recognition on a captured image received later.

[0110] In addition, when it is determined to perform object recognition, the processor 230 may perform object recognition using the captured image. The processor 230 may transmit the object recognition result to the refrigerator 100 via the communication device 220.

[0111] On the other hand, if it is determined that object recognition is not performed, the processor 230 may reuse the result of the object recognition previously performed with the original image. In other words, when it is determined that no change between the captured image and the original image has occurred or it is determined that a change in the image capturing direction has occurred, the processor 230 may send the object recognition result about the original image to the refrigerator 100 via the communication device 220.

[0112] At the same time, in showing and describing Figure 3 When receiving the captured image from the refrigerator, it is shown and described, but in an embodiment, the captured image may be received from a separate image capture device that photographs the refrigerator.

[0113] In the above, only a brief configuration of the configuration server is illustrated and described, but various elements may be additionally provided in the implementation. This will be referred to below. Figure 4 to describe.

[0114] Figure 4 is a block diagram showing a specific configuration of a server according to an embodiment.

[0115] Reference Figure 4 , the server 200 may be configured with a memory 210 , a communication device 220 , a processor 230 , an input device 240 , and a display 250 .

[0116] The memory 210, the communication device 220 and the processor 230 execute Figure 3 The elements have the same function, so overlapping descriptions will not be repeated.

[0117] The input device 240 may include a plurality of function keys for setting or selecting various functions supported by the server 200. Thus, the user may input various operation commands regarding the server 200.

[0118] In addition, the input device 240 may receive an input through a menu displayed on the display 250. In addition, the input device 240 may be implemented as a plurality of buttons, a keyboard, a mouse, etc., and may also be implemented as a touch screen for simultaneously performing functions of the display 250 which will be described later.

[0119] The display 250 may display various messages provided from the server 200. For example, the display 250 may display a user interface window for selecting various functions provided by the server 200. The display 250 may be a monitor such as an LCD, a CRT, an OLED, etc., or may be implemented as a touch screen for simultaneously performing the functions of the input device 240 which will be described later.

[0120] In the related art, even when the image capturing direction of the received captured image is changed, the server performs object recognition on the captured image. In this case, although the food contained in the refrigerator does not actually change, there is a problem that a large number of operations are required due to the additional object recognition of the server caused by the change in the captured image.

[0121] In contrast, as described above, the server according to the embodiment can identify whether the image capturing direction has changed by comparing the captured image and the original image, and when it is identified that only a change in the image capturing direction has occurred, the server can determine not to perform additional object recognition, thereby exhibiting the effect of reducing operations.

[0122] Figures 6 to 9 is a diagram illustrating an example of a method for recognizing a change in an image capturing direction using a plurality of feature points according to an embodiment.

[0123] The processor 140 or 230 of the refrigerator or server can identify whether the image capturing direction is changed based on the position change of the plurality of feature points between the original image and the captured image according to the optical flow method. Specific operations related thereto will be described below.

[0124] Figure 6 is a diagram illustrating an example of positional changes of a plurality of feature points in an image according to an embodiment. Figure 6 (a) corresponds to the original image, Figure 6 (b) corresponds to the captured image.

[0125] First, the processor 140 or 230 may identify a plurality of feature points in the original image and detect a plurality of feature points corresponding thereto from the captured image.

[0126] Specifically, refer to Figure 6 In (a), it can be identified that a sector-shaped pizza image is located in the original image, and three black dots are displayed at the edge of the sector.

[0127] Reference Figure 6 In (b), it can be recognized that the position of the sector-shaped pizza piece moves to the upper right side in the captured image, and the three black dots on the edge of the sector also move to the upper right side.

[0128] The points shown here may refer to feature points of an object in an image, which are points such as corners or endpoints of a segment having features for distinguishing from a background or another object. Feature points may be configured with two-dimensional coordinates, and feature points of a captured image as a three-dimensional image may be configured with three-dimensional coordinates.

[0129] In other words, processor 140 or 230 can Figure 6 Identify multiple feature points from the original image (a) and Figure 6 A plurality of feature points corresponding to a plurality of feature points of the original image are detected in the captured image of (b).

[0130] Meanwhile, the processor 140 or 230 may recognize a plurality of feature points from the original image, but may not observe a plurality of feature points corresponding to the plurality of feature points of the original image from the captured image. This may correspond to a case where it is difficult to observe a plurality of feature points corresponding to the plurality of feature points of the original image from the captured image according to the movement of the object that can be recognized from the original image.

[0131] Therefore, when it is difficult to identify a plurality of feature points corresponding to a plurality of feature points of the original image from the captured image, the processor 140 or 230 may determine that a change has occurred with respect to the object and determine that the image capturing direction has not changed.

[0132] Specifically, the processor 140 or 230 can identify the number of multiple feature points of the original image and the number of multiple feature points of the captured image, and if the change rate of the number of feature points exceeds a preset threshold rate, the processor 140 or 230 can determine that the image capture direction has not changed.

[0133] For example, if 10 feature points are identified from the original image, but only 5 feature points are identified from the captured image, the processor may determine that the object corresponding to the unidentified 5 feature points moves and that the image capture direction has not changed.

[0134] If a plurality of feature points corresponding to the plurality of feature points of the original image are detected from the captured image, the processor 140 or 230 can distinguish whether the object in the image moves or the image capturing direction changes by comparing the moving distance and moving direction of each of the plurality of feature points of the original image and the plurality of feature points of the captured image. This will be referred to below. Figures 7 to 9 to describe.

[0135] Figures 7 to 9 is a diagram showing an example of moving distances and moving directions of a plurality of feature points in a captured image according to an embodiment. Here, the direction of the arrow may correspond to the moving direction of each feature point, and the length of the arrow may correspond to the moving distance of each feature point.

[0136] Reference Figure 7 , it can be recognized that all arrows of the plurality of feature points of the captured image have the same direction and the same length. In other words, it can be recognized that the plurality of feature points of the captured image move along the same moving direction and the same moving distance.

[0137] In this case, the processor 140 or 230 may determine that the captured image may correspond to a case where no object movement occurs but a change in the image capturing direction occurs (③). Therefore, the processor 140 or 230 may determine not to perform object recognition.

[0138] Reference Figure 8, it can be recognized that the arrow in the square area shown with a dotted line from the captured image is pointing to the lower right direction, while the arrows in the remaining areas are pointing to the upper right direction. In this case, the processor 140 or 230 can determine that the object corresponding to the feature point in the square area with a dotted line is moved and the image capture direction is maintained.

[0139] In other words, the processor 140 or 230 may determine that the captured image corresponds to a situation where the object movement occurs and the image capturing direction changes (①). Therefore, the processor 140 or 230 may determine to perform object recognition.

[0140] Reference Fig. 9 , it can be recognized that the moving directions and moving distances of the plurality of feature points of the captured image may be different from each other. In this case, the processor 140 or 230 may determine that the object is moved and the image capturing direction is maintained.

[0141] In other words, the processor 140 or 230 may determine that the captured image may correspond to a situation where the object movement occurs but the image capturing direction is not changed (②). Therefore, the processor 140 or 230 may determine to perform object recognition.

[0142] As described above, the processor 140 or 230 can identify whether the object movement or the image capture direction change occurs by comparing the moving distance and the moving direction of each feature point as described above. At the same time, in order to compare the moving distance and the moving direction, the processor 140 or 230 can calculate the coordinate change amount of each of the multiple feature points of the captured image by comparing the pixel coordinate values ​​between each of the multiple points of the original image and the multiple feature points of the captured image.

[0143] Furthermore, the processor 140 or 230 may calculate a moving distance and a moving direction of each of a plurality of feature points of the captured image by using the calculated coordinate change amount, and identify whether the image capturing direction is changed based on the calculated moving distance and moving direction.

[0144] If the coordinate change calculated here is (x, y), the moving distance can be And the moving direction can be calculated as the tilt Alternatively, the direction of movement can be calculated as an angle

[0145] Specifically, when the calculated moving distances and moving directions of the plurality of feature points of the captured image coincide within a predetermined error range, the processor 140 or 230 may determine that the image capturing direction is changed.

[0146] On the other hand, when the moving distance or the moving direction of at least one of the plurality of feature points of the captured image has a difference exceeding a predetermined error range, the processor 140 or 230 may determine that the image capturing direction is maintained.

[0147] Furthermore, the predetermined error range may refer to a range that is included in the error range when an image is captured by the image capturing device, although there is a certain difference in the moving distance and the moving direction of each of the plurality of feature points.

[0148] As described above, the processor 140 or 230 may identify whether the image capturing direction is changed based on the position change of the plurality of feature points between the original image and the captured image.

[0149] Fig.10 is a diagram illustrating an example of a method for recognizing a change in an image capturing direction using an artificial intelligence model according to an embodiment.

[0150] In addition to the method using multiple feature points, the processor 140 or 230 of the refrigerator or server can also recognize whether the image capture direction changes by using an artificial intelligence model. Specific operations related to this will be described below.

[0151] Reference Fig.10 , the processor 1000 may include at least one of a learning unit 1010 and an acquisition unit 1020. Fig.10 The processor 1000 may correspond to the processor 140 of the refrigerator 100, the processor 230 of the server 200, or a processor of a data learning server (not shown).

[0152] The learning unit 1010 may generate or train a model for identifying changes in the image capture direction. Specifically, the learning unit 1010 may use the collected learning data to generate an artificial intelligence model for identifying changes in the image capture direction.

[0153] The artificial intelligence model here can be configured with machine learning and basic technologies using machine learning. Machine learning can refer to, for example, algorithmic technologies for self-classification / self-training features of input data, and basic technologies can refer to, for example, technologies that use machine learning algorithms such as deep learning to simulate human brain functions such as recognition or judgment, and include technical fields such as language understanding, visual understanding, reasoning / prediction, knowledge expression, and operational control.

[0154] In addition, the artificial intelligence model may correspond to a convolutional neural network (CNN), a recurrent neural network (RNN), a convolutional recurrent neural network (CRNN), etc., and is not limited to the above examples.

[0155] The learning data here may correspond to a set of images of the same object captured by moving slightly upward, downward, leftward or rightward, or a set of images of the same image slightly cut in the upward, downward, leftward or rightward direction. Meanwhile, the learning data is not limited to the above examples.

[0156] In other words, the learning unit 1010 can generate or train a model for recognizing a change in an image capturing direction by using a set of images with a changed image capturing direction, which is learning data, as input data.

[0157] The acquisition unit 1020 may recognize a change in the image capturing device by using the original image and the captured image as input data for training a model.

[0158] As described above, the processor 1000 may generate an artificial intelligence model for identifying a change in an image capturing direction through the learning unit 1010 , and identify a change in an image capturing direction of a currently captured image by using the artificial intelligence model through the acquisition unit 1020 .

[0159] Meanwhile, the learning unit 1010 and the acquisition unit 1020 may be installed on one electronic device such as a server, or may be installed on separate electronic devices, respectively. For example, the learning unit 1010 may be installed on a data learning server, and the acquisition unit 1020 may be installed on the server 200 or the refrigerator 100. In this case, the model information constructed by the learning unit 1010 may be provided to the acquisition unit 1020 by using wired or wireless communication.

[0160] Fig.11 is a flowchart for illustrating an object recognition method according to an embodiment.

[0161] Reference Fig.11 , the object recognition method according to an embodiment of the present disclosure may be executed by the processor 140 of the refrigerator 100 or the processor 230 of the server 200 .

[0162] First, an image acquired by photographing the interior of the storage compartment is acquired (S1110). Specifically, the processor of the refrigerator may acquire a captured image using an image capture device, and the processor of the server may receive the captured image from the refrigerator via a communication device to acquire the captured image.

[0163] When a change in the captured image compared to the original image is identified, a change in the image capture direction may be identified (S1120). The original image here may refer to a captured image previously acquired by the refrigerator using an image capture device to capture an image of the storage room, or may refer to a captured image previously acquired by the server by receiving an image of the storage room from the refrigerator. In addition, the original image may be a most recently acquired image among a plurality of previously acquired captured images.

[0164] Specifically, the pixel value change area may be detected by comparing each pixel value of the original image with each pixel value of the captured image corresponding thereto to identify whether the captured image is changed.

[0165] Whether the image capturing direction is changed may be identified by using a plurality of feature points in the original image and a plurality of feature points of the captured image corresponding thereto.

[0166] Specifically, a plurality of feature points may be identified from the original image, and a plurality of feature points corresponding thereto may be identified from the captured image. If some of the plurality of feature points are not observed from the captured image due to movement of the object identified in the original image, it may be determined that the image capturing direction is maintained.

[0167] More specifically, if the rate of change between the number of feature points of the original image and the number of feature points of the captured image exceeds a predetermined threshold rate, it may be determined that the image capturing direction is maintained.

[0168] In addition, the change in the image capture direction can be identified by comparing the moving distance and moving direction of each of the multiple feature points. Specifically, multiple feature points in the original image and multiple feature points corresponding thereto in the captured image can be detected, and the pixel coordinate values ​​between each of the multiple feature points can be compared to calculate the coordinate change amount of each of the multiple feature points of the captured image.

[0169] Furthermore, the moving distance and moving direction of each of the plurality of feature points of the captured image may be calculated by using the calculated coordinate change amount, and whether the image capturing direction is changed may be recognized based on the calculated moving distance and moving direction.

[0170] More specifically, when the calculated moving distances and moving directions of the plurality of feature points of the captured image coincide within a predetermined error range, it can be determined that the image capturing direction has changed. On the other hand, when the moving distance or moving direction of at least one of the plurality of feature points of the captured image has a difference exceeding a predetermined error range, it can be determined that the image capturing direction has been maintained.

[0171] In addition, the change of the image capturing direction can be identified by using an artificial intelligence model. Specifically, it is possible to identify whether the image capturing direction has changed by inputting the original image and the captured image into the artificial intelligence model.

[0172] The artificial intelligence model here may refer to a model that is trained using a set of images of the same object captured by moving slightly upward, downward, left or right, or a set of images of the same image slightly cut in the upward, downward, left or right direction as learning data to recognize whether the image capturing direction has changed.

[0173] In addition, the type of artificial intelligence model may correspond to a convolutional neural network (CNN), a recurrent neural network (RNN), a convolutional recurrent neural network (CRNN), etc., and is not limited to the above examples.

[0174] In addition, when the image capturing direction is maintained, object recognition may be performed on the captured image (S1130). On the other hand, when the image capturing direction is changed, object recognition may not be performed on the captured image, and the captured image may be stored by replacing the original image. In addition, the result of the object recognition performed on the original image may be reused.

[0175] Therefore, the object recognition method of the present disclosure can identify whether the image capturing direction has changed by comparing the captured image with the original image, and when it is identified that only the image capturing direction has changed, determine not to perform additional object recognition, thereby showing an effect of reducing operations. Fig.11 The object recognition method in Figure 1 or Figure 2 On refrigerators with Figure 3 or Figure 4 The method is executed on a server having a configuration of and may be executed on a refrigerator or server having other configurations.

[0176] Furthermore, the above-mentioned object recognition method may be implemented as at least one execution program for executing the above-mentioned object recognition method, and such execution program may be stored in a non-transitory computer-readable medium.

[0177] A non-transitory computer-readable medium is not a medium that stores data for a short period of time, such as a register, a cache, or a memory, but may refer to a medium that stores data semi-permanently and can be read by a machine. Specifically, the above-mentioned various applications or programs may be stored and provided to a non-transitory computer-readable medium, such as a CD, a DVD, a hard disk drive, a Blu-ray disc, a USB, a memory card, and a ROM.

[0178] Fig.12 is a flowchart for specifically illustrating an object recognition method according to an embodiment.

[0179] Reference Fig.12 , a captured image of the storage room of the refrigerator can be first obtained (S1210). Specifically, the refrigerator can obtain an image obtained by photographing the storage room through an image capture device, and the server can receive the captured image of the storage room from the refrigerator via a communication device.

[0180] In addition, the change of the captured image compared with the original image can be identified (S1220). Specifically, the pixel value change area can be detected by comparing each pixel value of the original image with each pixel value of the captured image corresponding thereto to identify whether the captured image has changed.

[0181] When no change region of the captured image is detected (S1220-N), it can be determined that object recognition is not performed on the captured image. On the other hand, when a change region of the captured image is detected (S1220-Y), it can be identified that the size of the change region exceeds a predetermined size (S1230).

[0182] When the size of the change occurrence area does not exceed the predetermined size (S1230-N), object recognition may be performed with respect to the captured image (S1250).

[0183] On the other hand, when the size of the change occurrence area exceeds a predetermined size (S1230-Y), a change in the image capture direction of the captured image can be identified (S1240). The operation of identifying whether the image capture direction has changed is specifically described in the above part related to the operation of the processor of the refrigerator or server, so the overlapping description will not be repeated.

[0184] When a change in the image capturing direction occurs (S1240-Y), it may be determined not to perform object recognition on the captured image. When a change in the image capturing direction does not occur (S1240-N), object recognition may be performed on the captured image (S1250).

[0185] Therefore, in the object recognition method of the present disclosure, it is possible to determine whether to perform the operation of recognizing the change in the image capturing direction according to the size of the change area of ​​the captured image. Therefore, although the image capturing direction is obviously maintained, the effect of preventing the operation of recognizing the change in the image capturing direction is still exhibited. Fig.12 The object recognition method in Figure 1 or Figure 2 On refrigerators with Figure 3 or Figure 4 The method is executed on a server having a configuration of and may be executed on a refrigerator or server having other configurations.

[0186] Furthermore, the above-mentioned object recognition method may be implemented as at least one execution program for executing the above-mentioned object recognition method, and such execution program may be stored in a non-transitory computer-readable medium.

[0187] Fig.13 is a sequence diagram for illustrating an object recognition method using a refrigerator and a server according to the first embodiment.

[0188] First, the server 200 may store an original image acquired by photographing the storage room of the refrigerator 100 (S1310). Specifically, the server 200 may receive and store a captured image of the storage room interior previously captured from the refrigerator 100 via communication with the refrigerator 100.

[0189] In addition, the refrigerator 100 may photograph the storage room using an image capture device and acquire a captured image (S1320). The refrigerator 100 may transmit the captured image of the storage room to the server 200 (S1330).

[0190] Furthermore, the server 200 may identify whether a change occurs in the received captured image compared to the original image (S1340).

[0191] When a change occurrence area is not detected, the server 200 may not perform additional object recognition and transmit the result of the object recognition performed on the original image to the refrigerator 100 .

[0192] On the other hand, when the change occurrence area is detected, the server 200 can identify whether the size of the change occurrence area exceeds a predetermined size (S1350). When the size of the change occurrence area does not exceed the predetermined size, object recognition can be performed on the captured image. On the other hand, when the size of the change occurrence area exceeds the predetermined size, a change in the image capture direction can be identified (S1360).

[0193] When the image capturing direction is maintained, object recognition may be performed on the captured image (S1370). On the other hand, when the image capturing direction is not maintained, the result of the object recognition performed on the original image may be transmitted to the refrigerator 100.

[0194] In addition, when object recognition is performed on the captured image, the server 200 can send the object recognition result of the captured image to the refrigerator 100, and when object recognition is not performed on the captured image, the server can send the object recognition result of the original image to the refrigerator 100 (S1380).

[0195] The refrigerator 100 may display the object information to the user by using the received object recognition result (S1390). Specifically, the refrigerator 100 may display the object information to the user via a display.

[0196] Therefore, in the object recognition method of the present disclosure, object recognition information can be obtained without directly performing object recognition by the refrigerator, thereby showing an effect of providing information to the user. Fig.13 The object recognition method in Figure 1 or Figure 2 On refrigerators with Figure 3 or Figure 4 The method is executed on a server having a configuration of and may be executed on a refrigerator or server having other configurations.

[0197] Furthermore, the above-mentioned object recognition method may be implemented as at least one execution program for executing the above-mentioned object recognition method, and such execution program may be stored in a non-transitory computer-readable medium.

[0198] Fig.14 is a sequence diagram for illustrating an object recognition method using a refrigerator and a server according to a second embodiment.

[0199] First, the refrigerator 100 may store an original image acquired by photographing the storage room (S1410). The refrigerator 100 may photograph the storage room using an image capture device and acquire a captured image (S1420).

[0200] In addition, the refrigerator 100 may identify whether a change occurs in the captured image compared to the original image (S1430).

[0201] When a change occurrence area is not detected, the refrigerator 100 may transmit the original image as object recognition target information to the server 200 without recognizing whether an image capturing direction of the captured image is changed.

[0202] On the other hand, when the change occurrence area is detected, the refrigerator 100 can identify whether the size of the change occurrence area exceeds a predetermined size (S1440). When the size of the change occurrence area does not exceed the predetermined size, the captured image can be sent to the server 200 as object recognition target information. On the other hand, when the size of the change occurrence area exceeds the predetermined size, a change in the image capture direction can be identified (S1450).

[0203] When the image capturing direction is maintained, the captured image may be transmitted as object recognition target information to the server 200. On the other hand, when the image capturing direction is not maintained, the original image may be transmitted as object recognition target information to the server 200 (S1460).

[0204] In addition, the server 200 may perform object recognition according to the received object recognition target (S1470). For example, when the received object recognition target is a captured image, object recognition may be performed on the captured image, and when the received object recognition target is an original image, the result of the object recognition previously performed on the original image may be recognized without performing additional object recognition.

[0205] In addition, the server 200 may transmit an object recognition result corresponding to the object recognition target to the refrigerator 100 (S1480).

[0206] The refrigerator 100 may display the object information to the user by using the received object recognition result (S1490). Specifically, the refrigerator 100 may display the object information to the user via a display.

[0207] Therefore, in the object recognition method of the present disclosure, the refrigerator can determine the object recognition target by recognizing whether the image capturing direction has changed, and send the determined object recognition target to the server, thereby showing an effect of reducing the operation of the server because the server does not have to recognize the change in the image capturing direction. Fig.14 The object recognition method in Figure 1 or Figure 2 On refrigerators with Figure 3 or Figure 4 The method is executed on a server having a configuration of and may be executed on a refrigerator or server having other configurations.

[0208] Furthermore, the above-mentioned object recognition method may be implemented as at least one execution program for executing the above-mentioned object recognition method, and such execution program may be stored in a non-transitory computer-readable medium.

[0209] Although the preferred embodiments of the present disclosure are shown and described, the present disclosure is not limited to the aforementioned specific embodiments, and it is obvious that various modifications can be made by a person skilled in the art of the present disclosure without departing from the gist of the present disclosure as claimed in the appended claims. In addition, it is intended that such modifications will not be interpreted independently of the technical ideas or prospects of the present disclosure.

[0210] Industrial Applicability

[0211] Sequence Listing Free Text

Claims

1. An object recognition method for a refrigerator having an image capture device, the method comprising: acquiring, by an image capturing device, a captured image of a storage compartment of the refrigerator; identifying whether an image capturing direction of an image capturing device by which the captured image of the storage compartment was captured has changed based on a change in the captured image compared to the stored original image being identified; as well as Based on determining that the image capture direction has not changed, an object recognition operation is performed based on the captured image to obtain information associated with items in the storage compartment of the refrigerator.

2. The object recognition method according to claim 1, further comprising: Based on the change in the image capture direction being identified, the captured image is stored by replacing the original image.

3. The object recognition method according to claim 1, wherein identifying whether the image capture direction has changed comprises: Detecting areas of pixel value change by comparing each pixel between the original image and the captured image; as well as Based on the area of ​​the pixel value change region being larger than a predetermined size, it is recognized that the image capturing direction has been changed.

4. The object recognition method according to claim 3, wherein identifying whether the image capture direction has changed comprises: detecting, from the captured image, a plurality of second feature points corresponding to the plurality of first feature points in the original image; as well as A moving distance and a moving direction of the plurality of second feature points are calculated by comparing pixel coordinate values ​​of each of the plurality of first feature points and the plurality of second feature points, and a change in the image capturing direction is recognized based on the calculated moving distance and moving direction.

5. The object recognition method according to claim 4, wherein recognizing a change in the image capturing direction based on the moving distance and the moving direction comprises: determining that the image capturing direction is changed based on that the moving distances and moving directions of the plurality of second feature points coincide within a predetermined error range; as well as Based on the fact that a movement distance or a movement direction of at least one second feature point among the plurality of second feature points is different from the movement distances or the movement directions of the remaining second feature points beyond a predetermined error range, it is determined that the image capturing direction is maintained.

6. The object recognition method according to claim 4, wherein identifying whether the image capture direction has changed comprises: Based on a change rate between the number of the plurality of second feature points and the number of the plurality of first feature points exceeding a predetermined threshold rate, it is determined that the image capturing direction is maintained.

7. The object recognition method of claim 1 , wherein identifying whether the image capture direction has changed comprises identifying the change in the image capture direction by identifying the change in the image capture direction using an artificial intelligence model set, and The artificial intelligence model is at least one of a convolutional neural network (CNN), a recurrent neural network (RNN) and a convolutional recurrent neural network (CRNN).

8. A refrigerator, comprising: a storage room configured to store items; an image capture device configured to capture an image of the storage compartment; a memory configured to store an original image previously captured by an image capture device; as well as Processor, configured as: acquiring a captured image by controlling an image capturing device to capture an image of the storage room, Based on the change of the captured image compared to the original image being identified, identifying whether the image capturing direction of the image capturing device is changed, and Based on determining that the image capture direction has not changed, an object recognition operation is performed based on the captured image to obtain information associated with items in the storage compartment of the refrigerator. 9 . The refrigerator according to claim 8 , wherein the processor is configured to store the captured image in the memory by replacing the original image based on the recognition that a change in the image capturing direction is identified.

10. The refrigerator according to claim 8, wherein the processor is configured to: detecting a pixel value change region by comparing pixels of each of the original image and the captured image; and Based on the area of ​​the pixel value change region being larger than a predetermined size, a change in the image capturing direction is recognized.

11. The refrigerator according to claim 10, wherein the processor is configured to: A plurality of second feature points corresponding to a plurality of first feature points in an original image are detected from a captured image, a moving distance and a moving direction of the plurality of second feature points are calculated by comparing the pixel coordinate values ​​of the plurality of first feature points and each of the plurality of second feature points, and a change in the image capturing direction is identified based on the calculated moving distance and moving direction.

12. The refrigerator according to claim 11, wherein the processor is configured to: determining that the image capturing direction is changed based on that the moving distances and moving directions of the plurality of second feature points coincide within a predetermined error range; and Based on the fact that a movement distance or a movement direction of at least one second feature point among the plurality of second feature points is different from the movement distances or the movement directions of the remaining second feature points beyond a predetermined error range, it is determined that the image capturing direction is maintained.

13. The refrigerator according to claim 11, wherein the processor is configured to: Based on a change rate between the number of the plurality of second feature points and the number of the plurality of first feature points exceeding a predetermined threshold rate, it is determined that the image capturing direction is maintained.

14. The refrigerator according to claim 8, wherein the processor is configured to recognize a change in the image capturing direction by recognizing the change in the image capturing direction using an artificial intelligence model set, and The artificial intelligence model is at least one of a convolutional neural network (CNN), a recurrent neural network (RNN) and a convolutional recurrent neural network (CRNN).

15. A server, comprising: a memory configured to store an original image acquired by capturing an image of a storage compartment of the refrigerator; a communication device configured to receive, from the refrigerator, a captured image obtained by capturing a storage compartment of the refrigerator; as well as Based on the confirmation that the captured image has changed compared to the original image, identifying whether an image capturing direction of the captured image has changed, and Based on determining that the image capture direction has not changed, an object recognition operation is performed based on the captured image to obtain information associated with items in the storage compartment of the refrigerator.

Images