Refrigerator

Abstract

To provide a refrigerator that properly controls foods in the storage room.SOLUTION: A refrigerator comprises a housing having a storage room, a door closing the opening of the housing, a photography part provided outside the housing and photographing the storage room and the outside of the storage room, and a control part letting the photography part take photographs, and the control part informs an information terminal of a user that a food is left outside as it is on condition that the food is taken out of the refrigerator and still recognized outside the storage room even after the door is closed and no person is detected.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present invention relates to a refrigerator. [Background technology]

[0002] As a technology for detecting food in a refrigerator, for example, Patent Document 1 discloses a technology (Example 2) that uses an RFID reader antenna and a camera together as a food sensor to detect food in a refrigerator equipped with a storage compartment, a food sensor that detects food, and a communication module that transmits information about the food detected by the food sensor to the outside of the refrigerator. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7213784 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1, for example, claims that it is possible to manage food that has been placed outside the cabinet, but it is not possible to determine whether food that has been placed outside the cabinet has been left behind or not, and there is room for more appropriate photography using a camera. [Means for solving the problem]

[0005] The refrigerator according to the present disclosure includes a housing having a storage compartment, a door closing an opening of the housing, an image capturing unit provided on the outside of the housing for capturing images of the storage compartment and the outside of the storage compartment, and a control unit for causing the image capturing unit to capture images, and the control unit is characterized in that, when food is removed from the storage compartment based on the captured images and the door is closed, the removed food is recognized outside the storage compartment and no person is detected, the control unit notifies a user's information terminal that the food has been left behind. Other aspects of the present invention will be described in the embodiments described later. [Brief description of the drawings]

[0006] [Figure 1] FIG. 2 is a front view of the refrigerator according to the embodiment. [Diagram 2] FIG. 2 is a side view of the refrigerator according to the embodiment. [Diagram 3] FIG. 2 is a front view of the refrigerator according to the embodiment with the left and right refrigerator compartment doors open. [Figure 4] FIG. 2 is a plan view of the refrigerator according to the embodiment with the left and right refrigerator compartment doors open. [Diagram 5] 1 is a perspective view of a camera unit of a refrigerator according to an embodiment when viewed from diagonally below. [Figure 6] FIG. 2 is a control block diagram of the refrigerator and the camera unit according to the embodiment. [Figure 7] 13 is an example of an image captured by a camera in an inside mode according to the embodiment. [Figure 8] 11 is a flowchart of a left-behind detection process by a control unit according to the embodiment. [Figure 9] FIG. 2 is a plan view of the living area as seen from an outside camera according to the embodiment. [Figure 10] FIG. 1 is a plan view of the living area in a shed-out state as seen from an outside camera in accordance with an embodiment. [Figure 11] FIG. 11 is a plan view of image recognition of the living area as seen from an outside camera according to the embodiment. [Figure 12] FIG. 11 is a plan view showing a left-behind notification state of the living area as seen from an outside camera according to the embodiment. [Figure 13] 10 is a time chart showing an example of a misplacement notification according to the embodiment; [Figure 14] 11 is an example of a captured image for detecting misplacement according to the embodiment. [Figure 15] 13 is another example of a captured image for detecting misplacement according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings. Fig. 1 is a front view of a refrigerator 100 according to an embodiment. The refrigerator 100 is an appliance for cooling food and the like, and includes a housing 1, doors such as refrigerator compartment doors 211, 212, and a camera unit 3 (photographing section). The housing 1 has a plurality of storage compartments therein. In the example of Fig. 1, the storage compartments of the refrigerator 100 include, from the top, a refrigerator compartment 21, an ice-making compartment 22 and an upper freezer compartment 23 arranged on the left and right, a vegetable compartment 24, and a lower freezer compartment 25.

[0008] The housing 1 is configured such that a space between an outer box 11 made of a steel plate and an inner box (not shown) made of resin is filled with a heat insulating material (not shown) such as urethane foam. A plurality of openings 10 (see Figs. 2 and 4) corresponding to each chamber are provided on the front side (front surface side) of the housing 1. For example, when food or the like is placed through the opening 10 of the refrigerator compartment 21, the refrigerator compartment doors 211, 212 are opened. When the refrigerator compartment doors 211, 212 are closed, the opening 10 of the refrigerator compartment 21 is closed. In this way, the refrigerator compartment doors 211, 212 (doors) have a function of closing the opening 10 of the housing 1. The same applies to the other doors.

[0009] Refrigerator 100 is equipped with left and right refrigerator compartment doors 211, 212 as French style doors which form refrigerator compartment 21 together with housing 1. Left refrigerator compartment door 211 (left door) is rotatable about the axis of hinge 211a (see FIG. 4) at the left end. The same is true for right refrigerator compartment door 212 (right door). Refrigerator 100 is also equipped with ice-making compartment door 221, upper freezer compartment door 231, as well as vegetable compartment door 241 and lower freezer compartment door 251 as drawer-type doors.

[0010] Refrigerator compartment 21 is provided with a plurality of shelves 213 (see FIG. 3) that divides refrigerator compartment 21 into predetermined sections. A plurality of door pockets 211c (see FIG. 3) for storing food and the like are provided on the inside of left refrigerator compartment door 211 (similar to right refrigerator compartment door 212). Ice-making compartment 22 is provided with an ice-making compartment container (not shown) that is pulled out together with ice-making compartment door 221. Similarly, upper freezer compartment 23 is provided with an upper freezer compartment container (not shown), vegetable compartment 24 is provided with a vegetable compartment container (not shown), and lower freezer compartment 25 is provided with a lower freezer compartment container (not shown).

[0011] Although not shown, the refrigerator 100 also includes a compressor, a radiator (condenser), a capillary tube (throttle mechanism), and an evaporator. A refrigerant circulates through the compressor, the radiator, the capillary tube, and the evaporator in this order, and the air in the storage compartment is cooled by heat exchange with the refrigerant flowing through the evaporator. The camera unit 3 shown in Fig. 1 captures an image of the storage compartment, etc. of the refrigerator 100, and is installed on the top surface of the housing 1.

[0012] Fig. 2 is a side view of refrigerator 100. As shown in Fig. 2, camera unit 3 (imaging section) includes main body section 31 and support section 32. Main body section 31 has a function of capturing images of a storage compartment, etc. Support section 32 supports main body section 31 and is installed on the top surface of housing 1.

[0013] A lens 31a (see also FIG. 5) is provided near the front end of the main body 31. The lens 31a is an optical element that refracts and focuses light. For example, a fisheye lens is used as the lens 31a. When the refrigerator compartment doors 211, 212 (see FIG. 1) are opened, the lens 31a faces downward so that the camera unit 3 can capture an image of the refrigerator compartment 21 (see FIG. 1).

[0014] As shown in FIG. 2, lens 31a is located forward of the front end of housing 1 (opening 10 of housing 1). More preferably, lens 31a is located further forward of the front surface of refrigerator compartment doors 211, 212 (see FIG. 1) in the closed state. This makes it easier for refrigerator compartment 21 to be within the field of view of lens 31a when refrigerator compartment doors 211, 212 are opened, for example. It is also possible for camera unit 3 to capture images of vegetable compartment 24, etc. (see FIG. 1) when a drawer-type door such as vegetable compartment door 241 is open. In this way, lens 31a is provided in a position that allows each storage compartment to be viewed from above.

[0015] 3 is a front view of refrigerator 100 with the left and right refrigerator compartment doors 211, 212 open. As described above, the inner panel 211b of the left refrigerator compartment door 211 is provided with a plurality of door pockets 211c for storing food and the like (the same is true for the right refrigerator compartment door 212). When the left and right refrigerator compartment doors 211, 212 are opened, the food and the like in the refrigerator compartment 21 and the door pockets 211c, 212c are arranged to be viewed from above within the field of view of lens 31a of camera unit 3.

[0016] Rotating partition 211d shown in Fig. 3 is a partition for preventing cold air from leaking from the gap between refrigerator compartment doors 211, 212. In the example of Fig. 3, rotating partition 211d is provided at the end opposite to the axis of hinge 211a (see Fig. 4) in left refrigerator compartment door 211, and is configured to rotate in a predetermined manner when refrigerator compartment door 211 is opened or closed.

[0017] Furthermore, when manually photographing the refrigerator compartment 21 using the camera unit 3, the left and right refrigerator compartment doors 211, 212 are provided with one photographing button 7 pressed by the user. In the example of Fig. 3, the left refrigerator compartment door 211 (door) has the photographing button 7 provided at the bottom of the surface opposite to the hinge 211a (see Fig. 4) of this refrigerator compartment door 211 (similar to the right refrigerator compartment door 212). From another perspective, when the left and right refrigerator compartment doors 211, 212 are closed (see Fig. 1), the photographing button 7 is provided on the surface where one of the refrigerator compartment doors 211, 212 faces the other (near the rotating partition 211d in Fig. 3).

[0018] Fig. 4 is a plan view of refrigerator 100 with left and right refrigerator compartment doors 211, 212 open. In the example of Fig. 4, main body 31 of camera unit 3 is disposed slightly left of the center of housing 1 of refrigerator 100 in the left-right direction. To explain in more detail, main body 31 of camera unit 3 is disposed directly above the joint between refrigerator compartment doors 211, 212 in the closed state (see also Fig. 1). By disposing main body 31 in this manner, for example, when refrigerator compartment doors 211, 212 are opened, food etc. in door pockets 211c, 212c are more likely to come into the field of view of camera unit 3.

[0019] Furthermore, when using refrigerator 100, the user is often conscious of the seam (boundary) of refrigerator compartment doors 211, 212. Therefore, by arranging main body 31 directly above the seam of refrigerator compartment doors 211, 212, the user can be prevented from feeling uncomfortable compared to when main body 31 is positioned off the seam in the left-right direction.

[0020] FIG. 5 is a perspective view of the camera unit 3 (photographing section) when viewed from diagonally below. The main body section 31 of the camera unit 3 shown in FIG. 5 includes the lens 31a (see also FIG. 2) described above, as well as a case 31b and a camera LED 31c (Light Emitting Diode: light emitting section 34b (see FIG. 6)). In the example of FIG. 5, the case 31b of the main body section 31 is generally rectangular parallelepiped elongated in the front-rear direction. A circular hole (not shown) is provided on the lower surface near the front end of the case 31b, and the lens 31a is exposed through this hole. The camera LED 31c is provided on the rear side (rear side) of the lens 31a.

[0021] The camera LED 31c is a light source that irradiates light onto the refrigerator compartment 21 (see FIG. 3) and door pockets 211c, 212c (see FIG. 3) so that the camera unit 3 can photograph the refrigerator compartment 21 and the like under appropriate brightness. The camera LED 31c (illumination unit) is provided on the outside of the housing 1 (see FIG. 2) of the refrigerator 100. In this embodiment, the camera unit 3 photographs the refrigerator compartment 21 and the like (see FIG. 3) with the interior light of the refrigerator 100 turned off and the camera LED 31c turned on.

[0022] As shown in Fig. 5, by providing lens 31a in front of camera LED 31c, reflected light from refrigerator compartment 21, etc. (see Fig. 3) can easily enter lens 31a when refrigerator compartment doors 211, 212 are open. Therefore, food stored in refrigerator compartment 21, etc. can be properly photographed. Note that camera LED 31c is preferably positioned forward of the front end (opening 10 of housing 1) of housing 1 (see Fig. 2) of refrigerator 100. This makes it easier for light emitted from camera LED 31c to enter refrigerator compartment 21, etc., and ultimately makes it easier to obtain clear photographing results.

[0023] Fig. 6 is a control block diagram of the refrigerator and the camera unit. As shown in Fig. 6, in addition to the camera unit 3, the refrigerator 100 is equipped with a control unit 101, an operation unit 102, an opening / closing detection unit 103 (door sensor), an inside temperature detection unit 104 (inside temperature sensor), an outside temperature detection unit 105 (outside temperature sensor), etc. The outside temperature sensor is disposed at the hinge 211a in Fig. 4.

[0024] Open / close detection unit 103 (door sensor) outputs a predetermined signal indicating the open / closed state of refrigerator compartment doors 211, 212 to control unit 101. Although one door sensor is shown in FIG. 6, in reality, a door sensor corresponding to left refrigerator compartment door 211 (see FIG. 1) and another door sensor corresponding to right refrigerator compartment door 212 (see FIG. 1) are provided. For example, when left refrigerator compartment door 211 is opened, an open signal is output from the door sensor corresponding to refrigerator compartment door 211 to control unit 8. Door sensors are also provided on ice-making compartment door 221 (see FIG. 1) and upper freezer compartment door 231, as well as vegetable compartment door 241 and lower freezer compartment door 251. Opening and closing of the doors is not limited to detection by a specific means, and can also be detected by an image captured by camera unit 3.

[0025] The control unit 101 is, for example, a microcomputer, and although not shown, is configured to include electronic circuits such as a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and various interfaces. Programs stored in the ROM are read out and expanded in the RAM, and the CPU executes various processes. The control unit 101 has an inside mode for monitoring the inside of the fridge and an outside mode for monitoring the outside of the fridge.

[0026] The camera unit 3 has a lens 31a (see also FIG. 5) and a camera LED 31c (see also FIG. 5, light-emitting unit 34b), as well as a camera / wireless communication control unit 33, an imaging unit 34a, a sound generation unit 34c, an image recognition unit 35 that recognizes items (food) based on the captured image, a human detection unit 36 ​​that detects a human body (person) based on the captured image, an inside / outside camera mode switching unit 37 that switches between inside and outside the fridge mode, and a timer unit 38 that measures various predetermined times. The imaging unit 34a photoelectrically converts the light incident via the lens 31a to generate captured image data.

[0027] The camera / wireless communication control unit 33 is an integrated circuit designed to integrate circuits having the functions of a microcomputer and other applied functions on one chip and function in cooperation with each other. The camera / wireless communication control unit 33 communicates with the control unit 101 in a predetermined manner and outputs a shooting command to the imaging unit 34a. As a result, captured image data is input from the imaging unit 34a to the camera / wireless communication control unit 33. Note that the camera / wireless communication control unit 33 is an example of an integrated circuit, and other types of integrated circuits may be used.

[0028] The captured image data output from the camera / wireless communication control unit 33 is transmitted to the router 51. The router 51 is a communication relay device, and transmits the received captured image data to the server 53 via the network 52. The server 53 performs a predetermined process on the captured image data and stores it in association with the identification information of the refrigerator 100. Furthermore, when the server 53 receives a request signal for an image of the interior of the refrigerator 100 from the user's information terminal 54, it transmits the captured image data to the information terminal 54. As such an information terminal 54, a mobile phone, a smartphone, a tablet, a wearable terminal, etc. may be used.

[0029] FIG. 7 is an example of an image captured by the camera in the inside mode. FIG. 7 is an example of the result of capturing images by the camera unit without image processing. Note that in FIG. 7, the left and right are reversed with respect to FIG. 3 since the upper side of the page is the front and the lower side is the rear. In the example of FIG. 7, both the left and right refrigerator compartment doors 211, 212 are open, and in addition to the refrigerator compartment 21, door pocket 211c of the left refrigerator compartment door 211 and door pocket 212c of the right refrigerator compartment door 212 are also captured. In particular, a fisheye lens is used as lens 31a of camera unit 3 (see FIG. 5), so that images can be captured with a wide range of angle of view.

[0030] (Detection of items left behind) The method for detecting misplacement will be described with reference to FIGS. 8 is a flowchart of the left-behind detection process S800 of the control unit 101. The control unit 101 sets the outside mode to ON (step S801), and determines whether or not a human body is detected around the refrigerator 100 (step S802). If a human body is not detected (step S802, No), the process returns to step S802, and if a human body is detected (step S802, Yes), the process proceeds to step S803.

[0031] In the outside-fridge mode, the camera is operated to detect a human body, but the camera may be operated when a human body is detected by a human presence sensor such as a pyroelectric sensor. The outside-fridge mode may be activated when a human body is detected by a separately provided human presence sensor.

[0032] FIG. 9 is a plan view of the living area as seen from the outside camera. FIG. 9 shows living areas 96, 97 as seen from the camera unit 3. The refrigerator is usually placed in the kitchen, and the living area 96 can be said to be the kitchen periphery. The living area 97 is an area outside the living area 96, and is outside the kitchen periphery. The human detection area θ1 is in the range from the direction 91 to the direction 92. The food misplacement detection area θ2 is in the range from the direction 93 to the direction 94. The range of the misplacement detection area θ2 is narrower than the range of the human detection area θ1 because the range in which food is placed is taken into consideration. In the example of FIG. 9, for example, a system kitchen is placed in front of the refrigerator 100, and the system kitchen is composed of a stove, a sink, a cooking counter, etc. In this case, the location where food is placed may be limited to the stainless steel auxiliary plate (sink) placed on the sink and the cooking counter. Therefore, the range of the misplacement detection area θ2 is determined based on the positional relationship with the refrigerator 100 in the kitchen.

[0033] Returning to FIG. 8, in step S803, the control unit 101 judges whether or not the door of the refrigerator 100 has been opened. If the door has not been opened (step S803, No), the process returns to step S802. If the door has been opened (step S803, Yes), the control unit 101 turns on the inside-fridge mode (step S804). Then, the control unit 101 judges whether or not the door has been closed (step S805), and if the door has not been closed (step S805, No), the process returns to step S805. If the door has been closed (step S805, Yes), the control unit 101 sets the outside-fridge mode to ON (step S806), and proceeds to step S807.

[0034] Fig. 10 is a plan view of the living area in an unloaded state as seen from the outside camera. When human body B approaches the refrigerator 100, the camera unit 3 starts detecting the human body (person), and when human body B (characteristic parts recorded by image recognition) faces the front of the refrigerator 100 and opens the door (either a French door or a pull-out door), the open / close detection unit 103 detects the door being open and switches the outside mode to the inside mode. When human body B removes food F1 from inside the refrigerator, the inside camera records the food F1 as an image recognition. When the open / close detection unit 103 of the refrigerator 100 detects a predetermined time has passed after the close detection, or when it detects that human body B is moving away (far away) without facing the front of the refrigerator, it switches the inside mode to the outside mode.

[0035] Returning to Fig. 8, in step S807, control unit 101 determines whether or not the food has been taken out, and if the food has not been taken out (step S807, No), the process returns to step S802, and if the food has been taken out (step S807, Yes), the process proceeds to step S808. In step S808, control unit 101 starts a timer for first predetermined time T1. Note that the start time of first predetermined time T1 differs depending on whether the food is in, for example, refrigerator compartment 21, upper freezer compartment 23, or lower freezer compartment 25 shown in Fig. 1. Details will be described later with reference to Fig. 13.

[0036] For example, in FIG. 10, when human body B moves away, control unit 101 acquires temperature information inside and outside the refrigerator, estimates the temperature range in which food F1 should be stored, and starts counting the timer.

[0037] After step S808, the control unit 101 determines whether the outgoing food can be detected (whether it can be detected outside the warehouse) (step S809). If the outgoing food cannot be detected (step S809, No), the control unit 101 returns to step S802. If the outgoing food can be detected (step S809, Yes), the control unit 101 determines whether the first predetermined time T1 has elapsed (step S810). If the first predetermined time T1 has not elapsed (step S810, No), the control unit 101 returns to step S809. If the first predetermined time T1 has elapsed (step S810, Yes), the control unit 101 proceeds to step S811.

[0038] Fig. 11 is a plan view of image recognition of the living area as seen from an outside camera. The camera unit 3 detects whether food F1 and human body B are present in the living area (around the kitchen) using the human detection unit 36, and performs image sampling of food F1 at predetermined time intervals (starts detection of left behind) using image recognition. The predetermined time interval can be set on the smartphone app side. For example, every 30 minutes, every hour, 3 hours, 6 hours, etc. The control unit 101 determines whether the first predetermined time T1 has elapsed.

[0039] 8, in step S811, control unit 101 determines whether or not a human body is present, and if a human body is present (step S811, No), returns to step S802, and if no human body is present (step S811, Yes), notifies registered user's information terminal 54 that the food has been left behind (step S812). Then, control unit 101 starts a timer for second predetermined time T2 (step S813), determines whether or not the delivered food can be detected (step S814), and if the delivered food cannot be detected (step S814, No), returns to step S802, and if the delivered food can be detected (step S814, Yes), determines whether or not second predetermined time T2 has elapsed (step S815). If the second predetermined time T2 has not elapsed (step S815, No), the control unit 101 returns to step S814; if the second predetermined time T2 has elapsed (step S815, Yes), the control unit 101 sounds the buzzer of the sound generation unit 34c (see FIG. 6) and notifies the information terminal 54 of the registered user (step S816), and ends the series of processes.

[0040] Fig. 12 is a plan view showing the notification state of the living area as seen from the outside camera. Fig. 12 shows the state where the food F1 has been image-recognized as left behind in the living area 96. When the human detection unit 36 ​​of the camera unit 3 of the refrigerator 100 determines that there is no human body in the living area (around the kitchen) after the first predetermined time T1, which is the temperature range in which the food F1 should be stored, has elapsed, the camera unit 3 of the refrigerator 100 immediately notifies the information terminal 54 (e.g., a smartphone) whether or not the food F1 needs to be returned to the refrigerator. If there is no change (change in position) in the food F1 (if the food F1 remains there) even after the second predetermined time T2 has elapsed after the notification that the food F1 has been left behind, the buzzer of the outside camera of the refrigerator 100 is sounded.

[0041] If the camera outside the refrigerator determines that the food has been left behind after the third predetermined time T3 has elapsed because the camera's field of view in the living area (around the kitchen) has become dark (=the lights in the room have been turned off), it may notify the paired user's information terminal 54 or emit an alarm from the refrigerator itself. When the food F1 is put into the refrigerator, the detection of food being left behind (sampling, timer) is canceled.

[0042] Fig. 13 is a time chart showing an example of the notification of leaving something behind. Fig. 13 shows, from the top, a time chart of the human detection unit 36 ​​(a human detection unit that detects a human body based on a captured image), the inside / outside camera mode switching unit 37, the image recognition unit 35, the temperature range to be stored and timer unit 38, and the open / close detection unit 103. The human detection unit 36, the inside / outside camera mode switching unit 37, the image recognition unit 35, and the open / close detection unit 103 switch between the H (High) side and the L (Low) side, with the H side meaning that a human body is detected, outside the cabinet mode, food is detected, and the door is open, respectively.

[0043] In Fig. 13, when the door is opened after a human body is detected, the mode is switched to the inside of the cabinet mode, and when the door is closed, the mode is switched to the outside of the cabinet mode. After the door is closed, the image recognition unit 35 detects whether food is detected at predetermined time intervals.

[0044] The temperature range in which the food should be stored is monitored by the temperature difference between the outside temperature and the inside temperature (outside temperature-inside temperature), and when the temperature difference becomes approximately zero, for example, a timer for a first predetermined time T1 is started.

[0045] The refrigerator compartment 21 shown in FIG. 1 is a refrigerator storage compartment whose interior temperature is set to, for example, about 4°C on average in the refrigerator temperature range (0°C or higher). The upper freezer compartment 23 and the lower freezer compartment 25 are freezer storage compartments whose interior temperature is set to, for example, about -18°C on average in the freezer temperature range (less than 0°C). The vegetable compartment 24 is a refrigerator storage compartment whose interior temperature is set to, for example, about 6°C on average in the refrigerator temperature range, and is a refrigerator storage compartment that prevents food from drying out. The temperatures of the storage compartments are different. For example, if the room temperature is 26°C and food is stored in the lower freezer compartment 25, the temperature difference when the food is taken out is as much as 44°C. Also, the temperature difference when the food is stored in the refrigerator compartment 21 is 22°C. For this reason, the control unit 101 predicts the time at which the removed food will reach the outside temperature (predetermined temperature) based on the outside temperature at which the food was stored and the inside temperature (see the curve indicated by the thick solid line sloping downward to the right in FIG. 13).

[0046] An additional explanation is provided for the downward sloping curve (line) in the graph of the temperature range in which food should be stored in Figure 13. The temperature of food F1 that has just been taken out is the same as the temperature inside the cabinet, so the temperature difference is outside the cabinet temperature - inside the cabinet temperature (i.e. the temperature difference is large when food F1 is taken out). If food F1 is then left outside the cabinet, the temperature of food F1 will rise and the difference with the outside cabinet temperature will become smaller. For this reason, the curve slopes downward to the right, as shown in Figure 13.

[0047] For example, if food F1 is a frozen food and the user defrosts it by natural thawing or the like, the downward sloping curve is set so that the count of the first predetermined time starts at the time (around the time) when the defrosting of the frozen food that was taken out and left behind is completed. Here, when a user takes a frozen food out of the freezer and defrosts it, it may be left behind even after the defrosting is completed. In the example of FIG. 13, the count of the first predetermined time starts around the time when the food is defrosted based on the downward sloping curve, and it is determined whether the food has been left behind, and if it has been left behind, a notice of the food being left behind is given to the user. Incidentally, in the example of FIG. 13, the human detection unit detects a human body until halfway through the count of the first predetermined time, but the person is absent in the latter half of the count.

[0048] The tendency of this curve is that the lower the temperature inside the cabinet where the food is placed, the longer the time until the first specified time starts to count. In other words, frozen food that has been in the freezer will have a longer time until the count starts than refrigerated food that has been in the refrigerator. Also, the lower the temperature outside the cabinet, the longer the time until the first specified time starts to count. Incidentally, for frozen foods with a high moisture content, a curve with flat parts along the way (a curve with a longer time until the count starts) is estimated due to the latent heat of fusion (endotherm) when the ice inside melts.

[0049] The first predetermined time may be set according to the type of food F1. For example, in the case of sashimi, it is preferable to set the first predetermined time to a shorter time.

[0050] When the human detection unit 36 ​​determines that there is no human body in the living area (around the kitchen) after the first predetermined time T1 has elapsed, the control unit 101 notifies the user's information terminal 54 whether it is necessary to immediately return the food to the refrigerator (notification of leaving it behind). If there is no change in the food (if it is still there) even after the second predetermined time T2 has elapsed after the notification of leaving it behind, the buzzer of the camera outside the refrigerator is activated.

[0051] The control unit 101 has an inside mode for monitoring the inside of the fridge and an outside mode for monitoring the outside of the fridge, and the control unit 101 may switch to the outside mode when the door is closed, and switch to the inside mode when the door is opened, and determine whether the removed food has been placed back into the storage room based on the captured image, before notifying the user's information terminal 54 that food has been left behind.

[0052] Fig. 14 is an example of a captured image for misplacement detection. In the image, food F1, which is the subject of misplacement detection and has been detected by image recognition after a first and second predetermined time has passed, is captured in front of the refrigerator. In addition, no human body is seen around the food F1.

[0053] Fig. 15 is another example of a captured image for misplacement detection. In the image, the area in front of the refrigerator is dark, and food F1 that is the subject of a misplacement notification and has been detected as having been left behind for a third predetermined time by image recognition is captured. In addition, no human body is seen around the food F1.

[0054] When the camera field of view of the living area (around the kitchen) becomes dark (= the lights in the room are turned off) and the refrigerator exterior camera (camera unit 3) determines that the user has been left behind after a third predetermined time T3 has elapsed, it may send a notification to the information terminal 54 of the paired user or emit an alarm from the refrigerator itself.

[0055] The refrigerator 100 etc. described in the embodiment are merely examples, and the present invention can be applied to refrigerators of other types. For example, the present invention can be applied to a refrigerator having a single-door refrigerator compartment door (not shown).

[0056] The refrigerator 100 according to this embodiment has the following features. (1) The refrigerator 100 includes a housing 1 having a storage compartment, a door that closes an opening of the housing 1, an image capturing unit (e.g., camera unit 3) that is provided on the outside of the housing 1 and captures images of the storage compartment and the outside of the storage compartment, and a control unit 101 that causes the image capturing unit to capture images. When food is removed from the storage compartment based on the captured image and the door is closed, and the removed food is recognized outside the storage compartment and no person is detected, the control unit 101 notifies the user's information terminal 54 that the food has been left behind. This makes it possible to determine whether the food placed outside the refrigerator is left behind or not, and to notify the user that the food has been left behind. In this embodiment, the control unit 101 has been described as processing the notification of the food being left behind, but this is not limited to this. For example, the camera / wireless communication control unit 33 may perform the notification of the food being left behind.

[0057] (2) In the refrigerator 100 of (1) above, when no person is detected after the first predetermined time T1 has elapsed since the door was closed, the control unit 101 notifies the user's information terminal 54 that food has been left behind. This allows the user to be informed that food has been left behind.

[0058] (3) In the refrigerator 100 of (2) above, the photographing unit has an alarm buzzer, and the control unit 101, after notifying the user's information terminal 54, further sounds the buzzer and notifies the information terminal 54 again if the removed food is recognized outside the storage compartment after a second specified time has elapsed.

[0059] (4) In the refrigerator 100 of (2) above, the first predetermined time is a predicted time that the removed food can be stored outside the storage compartment (see FIG. 13).

[0060] (5) The refrigerator 100 of (2) above, wherein the refrigerator 100 has an internal temperature sensor (e.g., internal temperature detection unit 104) and an external temperature sensor (e.g., external temperature detection unit 105), and the control unit 101 predicts the start time of the first specified time based on the external temperature and the internal temperature at the time of leaving the refrigerator (see FIG. 13).

[0061] (6) In the refrigerator 100 of (1) above, the control unit 101 notifies the user's information terminal 54 that food has been left behind after the door is closed, the field of view of the photographing unit becomes dark, and a third predetermined time has passed (see FIG. 15).

[0062] (7) In the refrigerator 100 of (6) above, the photographing unit has an alarm buzzer, and the control unit 101, after notifying the user's information terminal, further sounds the buzzer and notifies the information terminal 54 again if the removed food is recognized outside the storage compartment after a second predetermined time has elapsed.

[0063] (8) In the refrigerator 100 of (1) above, the control unit 101 has an inside mode for monitoring the inside of the refrigerator and an outside mode for monitoring the outside of the refrigerator, and the control unit 101 switches to the outside mode when the door is closed, and switches to the inside mode when the door is opened before notifying the user's information terminal 54 that the food has been left behind, and determines whether the removed food has been put back into the storage compartment based on the captured image. This makes it possible to determine whether the food has been returned to the refrigerator before the first predetermined time has elapsed in FIG. 13.

[0064] In addition, each embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to having all of the configurations described. In addition, it is possible to add, delete, or replace part of the configuration of an embodiment with other configurations. Furthermore, the above-mentioned mechanisms and configurations are those considered necessary for the explanation, and do not necessarily show all mechanisms and configurations of the product. [Explanation of symbols]

[0065] 1 Case 3 Camera unit (photography unit, outside camera) 7 Shooting button 10 aperture 21 Refrigerator (storage room) 22 Ice room (storage room) 23 Upper freezer (storage room) 24 Vegetable room (storage room) 25 Lower freezer (storage room) 31c Camera LED (lighting unit) 33 Camera / wireless communication control unit (control unit) 34a Imaging section 34b Light-emitting part 34c Pronunciation section 35 Image Recognition Unit 36 Human detection unit 37 Inside and outside camera mode switching section 38 Timer section 51 Router 52 Network 53 Server 54 Information terminals (mobile terminals) 100 Refrigerator 101 Control section 102 Operation section 103 Open / close detector 104 Internal temperature detection unit (internal temperature sensor) 105 Outside temperature detection unit (outside temperature sensor) 211 Refrigerator door (left door) 212 Refrigerator door (right door) 221 Ice room door 231 Upper freezer door (door) 241 Vegetable compartment door (door) 251 Lower freezer door (door) T1 First scheduled time T2 Second specified time T3 Third Scheduled Time

Claims

1. a housing having a storage chamber; a door that closes the opening of the housing; an imaging unit provided on the outside of the housing and configured to image the storage chamber and the outside of the storage chamber; A control unit that causes the imaging unit to perform imaging, The control unit of the refrigerator notifies a user's information terminal that food has been left behind when food is removed from the storage compartment based on the captured image, the door is closed, the removed food is recognized outside the storage compartment, and no person is detected.

2. The control unit notifies a user's information terminal of a forgotten item when a first predetermined time has elapsed since the door was closed and no person is detected.

2. The refrigerator according to claim 1 .

3. The photographing unit has an alarm buzzer, The control unit, after notifying the information terminal of the user, activates the buzzer and notifies the information terminal again when the removed food is recognized outside the storage room after a second predetermined time has elapsed.

3. The refrigerator according to claim 2.

4. 3. The refrigerator according to claim 2, wherein the first predetermined time is a predicted time for which the removed food can be stored outside the storage compartment.

5. The refrigerator has an internal temperature sensor and an external temperature sensor, The control unit predicts a start time of the first predetermined time based on an outside temperature and an inside temperature at the time of leaving the storage unit.

3. The refrigerator according to claim 2.

6. The control unit notifies a user's information terminal that the food has been left behind after the door is closed, the field of view of the photographing unit is dark, and a third predetermined time has elapsed.

2. The refrigerator according to claim 1 .

7. The photographing unit has an alarm buzzer, The control unit, after notifying the information terminal of the user, activates the buzzer and notifies the information terminal again when the removed food is recognized outside the storage room after a second predetermined time has elapsed.

7. The refrigerator according to claim 6.

8. The control unit has an inside mode for monitoring the inside of the fridge and an outside mode for monitoring the outside of the fridge, The control unit switches to the outside storage mode when the door is closed, switches to the inside storage mode when the door is opened, and determines whether the removed food has been put back into the storage compartment based on the captured image before notifying a user's information terminal that the food has been left behind.

2. The refrigerator according to claim 1 .

Images