storage

Abstract

To ensure that the status of the storage facility can be properly obtained. [Solution] The storage unit 100 is provided with a housing on which a door is arranged, and a brightness change detection unit 40 having multiple pixels and detecting brightness changes of each pixel in a first field of view that includes the range in which the door opens and closes. The first field of view includes multiple first regions arranged along the trajectory when the door opens and closes, and it is even more preferable to further include a door state determination unit 83 that outputs a signal indicating that the door has opened when the brightness change detection unit 40 detects brightness changes in the multiple first regions in order from the closed side of the door, and outputs a signal indicating that the door has closed when it detects brightness changes in the multiple first regions in order from the open side of the door.

Description

Technical Field

[0001] The present invention relates to a storage.

Background Art

[0002] As the background art in this technical field, the abstract of Patent Document 1 below states that, "……The refrigerator 100 includes a housing having a refrigerating chamber, a refrigerating chamber door for closing the opening of the housing, a door pocket installed on the inner side of the refrigerating chamber door, and a camera unit 3 installed outside the housing for photographing at least the refrigerating chamber. When an item stored in the door pocket or the door pocket is captured in a first region in the captured image of the camera unit 3, a control unit 8 is provided to prevent the captured image from being displayed on the user's mobile terminal." Also, the abstract of Patent Document 2 below states that, "…The photographing device includes a first camera for photographing a first storage chamber of the storage, and a first selection unit for selecting, from a moving image captured by the first camera, a first captured image in which the first storage chamber with the door in an open state is captured, based on the rotation angle of a rotary door for opening and closing the opening of the first storage chamber." It is described as such. The descriptions of these documents are incorporated as part of the specification of the present application.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in the above-described technology, there is a desire to more appropriately acquire the state of the storage.

Means for Solving the Problems

[0005] To solve the above problems, the storage unit of the present invention is characterized by comprising a housing on which a door is arranged, and a brightness change detection unit having a plurality of pixels, which detects changes in brightness of each pixel in a first field of view that includes the range in which the door opens and closes. [Brief explanation of the drawing]

[0006] [Figure 1] This is a front view of a refrigerator according to the first embodiment. [Figure 2] This is a side view of a refrigerator. [Figure 3] This is a front view of the refrigerator with both the left and right refrigerator compartment doors open. [Figure 4] This is a plan view of the refrigerator with both the left and right refrigerator compartment doors open. [Figure 5] This is a perspective view of the camera unit as seen from a diagonal angle below. [Figure 6] This figure shows an example of the field of view obtained by EVS and a color camera. [Figure 7] This is a system configuration diagram for a refrigerator. [Figure 8] This figure shows an example of the state in the field of view of EVS. [Figure 9] This figure shows an example of the state transition around the lower freezer compartment drawer door. [Figure 10] This figure shows other examples of state transitions around the lower freezer compartment drawer door. [Figure 11] This figure shows yet another example of the state transition around the lower freezer compartment drawer door. [Figure 12] This figure shows other examples of conditions in the field of view of EVS. [Figure 13] This figure shows yet another example of a state in the field of view of EVS. [Figure 14] This figure shows an example of the peripheral vision in EVS. [Figure 15] This figure shows examples of captured image data and processed image data. [Figure 16] This figure shows other examples of captured image data and processed image data. [Figure 17]It is a schematic diagram of the main part of the camera unit in the second embodiment. [Figure 18] It is a block diagram of a computer.

Embodiments for Carrying Out the Invention

[0007] [Overview of the Embodiment] In the above-mentioned Patent Document 1, there is no particular description regarding the acquisition timing of the image provided to the user and the correspondence to the pull-out door without a door pocket. Also, when applying the technology of the above-mentioned Patent Document 2, it is considered that the acquisition timing of the image provided to the user can be determined by providing a distance measurement sensor for each door. However, in this technology, since it is necessary to provide a distance measurement sensor for each door, there is a problem that the configuration becomes complicated and the cost increases.

[0008] By the way, a color camera is generally applied to the camera that captures the image provided to the user, and the color camera includes, for example, an image sensor in the RGB color space. By keeping this color camera always on, it is considered possible to determine the acquisition timing of the image provided to the user. However, since this type of color camera has high power consumption, keeping it always on leads to energy loss.

[0009] [[ID=二十一]]Therefore, in the embodiments described later, by utilizing an EVS (Event-based Vision Sensor), it is possible to appropriately acquire the image provided to the user with a simple configuration. Since the power consumption of the EVS is sufficiently small, even if it is always on, the energy loss is small. And based on the sensing result of the EVS, the power of the color camera is turned on, or the color camera is released from the standby state. [[ID=二十二]] [[ID=二十三]]

[0010] [[ID=二十四]] [[ID=二十五]][First Embodiment][[ID=二十六]] [[ID=二十七]]〈Overall Configuration of the Refrigerator〉[[ID=二十八]] [[ID=二十九]]FIG. 1 is a front view of a refrigerator 100 according to the first embodiment. The refrigerator 100 (storage compartment) is a device for cooling articles such as food, and includes, in addition to the housing 101, each door such as the refrigerator doors 211 and 212, and the camera unit 3 (imaging unit). The housing 101 has a plurality of storage compartments inside. In the example of FIG. 1, as the storage compartments of the refrigerator 100, in order from the top, there are provided a refrigerator compartment 121, an ice-making compartment 122 and an upper freezer compartment 123 arranged side by side left and right, a vegetable compartment 124, and a lower freezer compartment 125.

[0011] The housing 101 has a structure in which a heat insulating material (not shown) such as foamed urethane is filled between an outer box 111 made of a steel plate and an inner box (not shown) made of resin. A plurality of openings 110 (see FIGS. 2 and 4) corresponding to each compartment are provided on the front side (front surface side) of the housing 101. For example, when food or the like is put through the opening 110 of the refrigerator compartment 121, the refrigerator doors 211 and 212 are opened. Further, when the refrigerator doors 211 and 212 are closed, the opening 110 of the refrigerator compartment 121 is in a closed state. Thus, the refrigerator doors 211 and 212 have functions such as closing the opening 110 of the housing 101. The same applies to the drawer doors described later.

[0012] The refrigerator 100 includes the left and right refrigerator doors 211 and 212 as French doors that form the refrigerator compartment 121 together with the housing 101. The left refrigerator door 211 (left door) is rotatable about the axis of the hinge 211a (rotating shaft) at the left end shown in FIG. 4. Similarly, the right refrigerator door 212 (right door) is rotatable about the axis of the hinge 212a (rotating shaft) at the left end shown in FIG. 4. Further, the refrigerator 100 includes an ice-making compartment drawer door 221 (door, drawer door), an upper freezer compartment drawer door 231 (door, drawer door), a vegetable compartment drawer door 241 (door, drawer door), and a lower freezer compartment drawer door 251 (door, drawer door) in order to close the openings 110 of the ice-making compartment 122, the upper freezer compartment 123, the vegetable compartment 124, and the lower freezer compartment 125. Hereinafter, these may be referred to as "drawer doors 221, 231, 241, 251".

[0013] The refrigerator compartment 121 is provided with multiple shelves 213 (see Figure 3) that divide the refrigerator compartment 121 into predetermined sections. The left refrigerator compartment door 211 has multiple door pockets 211c (see Figure 3) on the inside for storing food and other items (the same applies to the right refrigerator compartment door 212). The ice-making compartment 122 is provided with an ice-making container 223 that is pulled out together with the ice-making drawer door 221. Similarly, the upper freezer compartment 123 is provided with an upper freezer container 233, the vegetable compartment 124 is provided with a vegetable container 243, and the lower freezer compartment 125 is provided with a lower freezer container 253. These containers are sometimes referred to as "containers 223, 233, 243, and 253."

[0014] Although not shown in the diagram, the refrigerator 100 is equipped with a compressor, a heat sink (condenser), a capillary tube (throttling mechanism), and an evaporator. The refrigerant circulates sequentially through the compressor, heat sink, capillary tube, and evaporator, and the air in the storage compartment is cooled by heat exchange with the refrigerant flowing through the evaporator. The camera unit 3 shown in Figure 1 is used to image the storage compartment of the refrigerator 100 and is installed on the top surface of the housing 101.

[0015] Figure 2 is a side view of refrigerator 100. As shown in Figure 2, the camera unit 3 comprises a main body 31 and a support part 32. The main body 31 has the function of imaging a storage room or the like. The support part 32 supports the main body 31 and is installed on the upper surface of the housing 101. In this embodiment, the camera unit 3 is fixed integrally with the housing 101, but the camera unit 3 and the housing 101 may be separate. In this case, the camera unit 3 is removablely mounted on the top surface of the housing 101 or the like.

[0016] An optical unit 300 is provided near the front end of the main body 31. The optical unit 300 is equipped with optical elements such as lenses that refract and focus light. Such an optical unit 300 includes, for example, a fisheye lens. The optical unit 300 is positioned to face downwards so that the camera unit 3 can image the refrigerator compartment 121 (see Figure 1) when the refrigerator compartment doors 211, 212 (see Figure 1) are opened.

[0017] As shown in Figure 2, the optical unit 300 is positioned in front of the front end of the housing 101 (the opening 110 of the housing 101). More preferably, the optical unit 300 is positioned even further in front of the front surface of the closed refrigerator compartment doors 211, 212 (see Figure 1). This makes it easier for the refrigerator compartment 121 to come into the field of view of the optical unit 300 when, for example, the refrigerator compartment doors 211, 212 are opened. It is also possible to image the vegetable compartment 124, etc. (see Figure 1) with the camera unit 3 when the drawer door, such as the vegetable compartment drawer door 241, is open. In this way, the optical unit 300 is positioned so that each storage compartment can be viewed from above.

[0018] Figure 3 is a front view of refrigerator 100 with the left and right refrigerator compartment doors 211 and 212 open. As described above, the inner panels 211b and 212b of the left and right refrigerator compartment doors 211 and 212 are provided with multiple door pockets 211c and 212c for storing food and other items. When the left and right refrigerator compartment doors 211 and 212 are opened, the food and other items in the refrigerator compartment 121 and the door pockets 211c and 212c are brought into the field of view of the optical unit 300 of the camera unit 3. As described above, the refrigerator compartment 121 is divided vertically by multiple shelves 213, and a chilled compartment container 263 is provided at the bottom. The temperature inside the chilled compartment container 263, i.e., the chilled compartment, is lower than the temperature inside the refrigerator compartment 121.

[0019] The rotating partition 211d shown in Figure 3 is a partition designed to suppress cold air leakage from the gap between the refrigerator doors 211 and 212. In the example shown in Figure 3, the rotating partition 211d is provided on the left refrigerator door 211 at the end opposite to the axis of the hinge 211a (see Figure 4), and rotates to a predetermined extent as the refrigerator door 211 is opened and closed.

[0020] Furthermore, when manually photographing the refrigerator compartment 121 using the camera unit 3, a shutter button 7 is provided on each of the left and right refrigerator compartment doors 211 and 212, which is pressed by the user. In the example in Figure 3, on the left refrigerator compartment door 211, the shutter button 7 is located at the bottom of the side opposite the hinge 211a (see Figure 4) of the refrigerator compartment door 211 (the same applies to the right refrigerator compartment door 212). To explain from another perspective, when the left and right refrigerator compartment doors 211 and 212 are closed (see Figure 1), the shutter button 7 is located on the side of one of the refrigerator compartment doors 211 and 212 that faces the other (near the rotating partition 211d in Figure 3). Additionally, the drawer doors 221, 231, 241, and 251 also have shutter buttons 7 located near the top.

[0021] A "manual shot" is taken when at least one of these multiple shooting buttons 7 is pressed by the user. A "automatic shot" is taken when the camera control unit 80 (see Figure 8) takes a shot in accordance with the open / closed state of the refrigerator doors 211, 212 and the drawer doors 221, 231, 241, 251, etc. It is preferable to perform an "automatic shot" after a predetermined waiting period has elapsed since the previous shot (automatic or manual shot).

[0022] Figure 4 is a plan view of refrigerator 100 with the left and right refrigerator compartment doors 211 and 212 open. In the example shown in Figure 4, the main body 31 of the camera unit 3 is positioned slightly to the left of the center of the refrigerator casing 101 in the left-right direction. More specifically, the main body 31 of the camera unit 3 is positioned directly above the joint of the closed refrigerator doors 211 and 212 (see also Figure 1). By positioning the main body 31 in this way, for example, when the refrigerator doors 211 and 212 are opened, food items in the door pockets 211c and 212c are more easily brought into the field of view of the camera unit 3.

[0023] Furthermore, when using the refrigerator 100, users are often aware of the seam (boundary) between the refrigerator doors 211 and 212. Therefore, by positioning the main body 31 directly above the seam between the refrigerator doors 211 and 212, the user's discomfort can be reduced compared to when the left-right position of the main body 31 is off-center from the seam.

[0024] Figure 5 is a perspective view of the camera unit 3 when viewed from diagonally below. The main body 31 of the camera unit 3 shown in Figure 5 includes, in addition to the optical unit 300 (see also Figure 2), a case 31b and a camera LED 31c (Light Emitting Diode). The optical unit 300 also includes lenses 42 and 52. These lenses 42 and 52 are applied to the EVS 40 (luminance change detection unit) and color camera 50, which will be described later in Figure 7.

[0025] In the example shown in Figure 5, the case 31b of the main body 31 is generally elongated in the front-to-back direction and has a rectangular parallelepiped shape. A pair of circular holes (not shown) are provided on the lower surface near the front end of the case 31b, and the lens 42 (first lens) and lens 52 (second lens) are exposed through these holes. Here, the lens 42 for the EVS is located in front of the lens 52 for the color camera. In other words, the lens 52 for the color camera is located closer to the housing 101 (see Figure 4) and closer to the hinges 211a and 212a compared to the lens 42 for the EVS. This allows the lens 52 for the color camera to be brought closer to the food inside the refrigerator 100, enabling the acquisition of clearer images of the food.

[0026] Furthermore, a camera LED 31c is provided on the rear (back) side of the optical unit 300. The camera LED 31c is a light source that illuminates the refrigerator compartment 121 (see Figure 3) and door pockets 211c, 212c (see Figure 3) with appropriate brightness so that the camera unit 3 can photograph the refrigerator compartment 121 (see Figure 3) and other door pockets 211c, 212c (see Figure 3). The camera LED 31c is also provided on the front side of the casing 101 (see Figure 2) of the refrigerator 100. In this embodiment, the camera unit 3 photographs the refrigerator compartment 121 (see Figure 3) and other door pockets (see Figure 3) with the interior light 4 (see Figure 7) of the refrigerator 100 turned off and the camera LED 31c turned on.

[0027] As shown in Figure 5, by providing the optical unit 300 in front of the camera LED 31c, reflected light from the refrigerator compartment 121, etc. (see Figure 3) is more easily incident on the optical unit 300 when the refrigerator doors 211, 212, etc. are open. Therefore, food stored in the refrigerator compartment 121, etc. can be properly photographed. It is preferable that the camera LED 31c be positioned in front of the front end of the casing 101 of the refrigerator 100 (see Figure 2) (opening 110 of the casing 101). This makes it easier for the light emitted from the camera LED 31c to incident on the refrigerator compartment 121, etc., and consequently, easier to obtain clear photographic results. The camera LED 31c can also function as a welcome light when a user approaches the refrigerator 100.

[0028] Figure 6 shows an example of the field of view VS40 and VS50 obtained by the EVS40 and color camera 50. As shown in the diagram, the fields of view VS40 (first field of view) and VS50 (second field of view) overlap to some extent, but the center of the field of view VS50 is located slightly forward of the center of the field of view VS40. This is because, as shown in Figure 5, the lens 42 for the EVS is positioned in front of the lens 52 for the color camera. Note that "field of view" in the case of the EVS40 means "range in which brightness changes are detected," and in the case of the color camera 50 it means "shooting range."

[0029] Since the color camera 50 is primarily used to photograph the interior of the refrigerator 100, there is little need to photograph areas beyond the movable range of the refrigerator doors 211, 212 and the drawer doors 221, 231, 241, 251 (see Figure 1), such as area A10 in the figure. On the other hand, since the EVS 40 can also be used as a sensor to detect the proximity of a user, for example, it is preferable to secure a wider field of view VS40 on the front side of the refrigerator 100, as shown in Figure 6. Both the field of view VS40 and VS50 include the opening and closing surfaces where the refrigerator doors 211, 212 and the drawer doors 221, 231, 241, 251 move open and closed.

[0030] Figure 7 is a system configuration diagram of refrigerator 100. As shown in Figure 7, the refrigerator 100 includes a camera unit 3, an interior light 4, a buzzer 5, a shooting button 7, a power supply unit 8, a door sensor 10, a drawer door sensor 12, and a refrigerator body control unit 90. The camera unit 3 includes an EVS 40, a color camera 50, and a camera control unit 80. Here, the EVS 40 includes a lens 42, an EVS sensor 44, and a control unit 46. The color camera 50 also includes a lens 52, an image sensor 54, and a control unit 56.

[0031] The EVS sensor 44 has multiple pixels (not shown) that detect brightness changes within a range including the field of view VS40 (see Figure 6) described above. When the EVS sensor 44 detects a brightness change in any of its pixels, it supplies an EVS signal S44 to the control unit 46, which includes the identification information of that pixel and the direction of the brightness change (brightness increase or brightness decrease). The control unit 46 supplies this EVS signal S44 to the camera control unit 80.

[0032] Furthermore, when the control unit 46 detects from the EVS signal S44 that an opening operation has been initiated on the refrigerator doors 211, 212 or the drawer doors 221, 231, 241, 251, it sets the color camera 50 to an operating state. "Setting to an operating state" means, for example, turning on the power of the color camera 50. Also, if the color camera 50 is in a standby state that does not output moving image information S54, the control unit 46 cancels this standby state.

[0033] Furthermore, the image sensor 54 is equipped with multiple pixels (not shown) that detect the brightness of each color, R (red), G (green), and B (blue), within a range that includes the field of view VS50 (see Figure 6) described above. Based on the brightness information of these pixels, the image sensor 54 supplies video information S54 at a predetermined frame rate to the control unit 56. The control unit 56 supplies this video information S54 to the camera control unit 80. Also, when instructed by the camera control unit 80, the control unit 56 supplies the video information S54 to the server 153 via the wireless LAN unit 31g.

[0034] Thus, one of the features of this embodiment is the use of the EVS40 and the color camera 50 in combination. In order to detect the movement of drawer doors 221, 231, 241, 251, etc. using the moving image information S54 from the color camera 50, it is generally necessary to distinguish between the container and the floor. However, with the EVS40, the movement of drawer doors, etc. can be detected without distinguishing between the container and the floor.

[0035] The interior light 4 is a light source that illuminates the inside of the refrigerator 100 and is installed inside the housing 101. The buzzer 5 is sounded when the interior light 4 is turned off, for example, when taking a picture with the camera unit 3, and is installed in a predetermined location on the housing 101. The shooting button 7 (see also Figure 3) is a button that is pressed by the user when taking a picture manually using the camera unit 3, as described above.

[0036] The door sensor 10 detects the open / closed state of the refrigerator doors 211 and 212 and supplies an open / closed signal indicating the respective state to the refrigerator main unit control unit 90. The door sensor 10 is a sensor that detects if the refrigerator doors 211 and 212 are even slightly open, in order to warn the user if they have forgotten to close the doors, etc. On the other hand, the camera control unit 80 determines whether the opening angle of the refrigerator doors 211 and 212 is greater than or equal to a predetermined angle suitable for shooting, based on the detection result of the EVS 40 (details will be described later).

[0037] Furthermore, the drawer door sensor 12 detects the open / closed state of the drawer doors 221, 231, 241, and 251, and supplies an open / closed signal indicating the respective open / closed state to the refrigerator main unit control unit 90. Similar to the door sensor 10, the drawer door sensor 12 is a sensor that detects if the drawer doors 221, 231, 241, and 251 are even slightly open, and considers that to be the "open state". Regarding the drawer doors 221, 231, 241, and 251, the camera control unit 80 determines, based on the detection result of the EVS 40, whether the extension distance is greater than or equal to a predetermined length suitable for shooting (details will be described later).

[0038] In addition to performing the "automatic shooting" described above, the camera control unit 80 also performs "manual shooting" when the shooting button 7 is pressed. The camera control unit 80 includes a proximity detection unit 81, an exposure adjustment unit 82, a door state detection unit 83, a shooting control unit 85, and an item recognition unit 86. The camera control unit 80 communicates with the refrigerator body control unit 90 and inputs and outputs various signals.

[0039] The proximity detection unit 81 detects when a user approaches the refrigerator 100. The exposure adjustment unit 82 adjusts the exposure of the camera unit 3. The door state determination unit 83 determines, based on the EVS signal S44, whether each of the refrigerator doors 211, 212 and the drawer doors 221, 231, 241, 251 is in a state suitable for shooting, i.e., a state suitable for shooting. The shooting control unit 85 performs various shooting controls. Specifically, with respect to manual shooting, when the shooting button 7 is pressed, the shooting control unit 85 outputs a shooting command to the control unit 56 for the color camera.

[0040] Furthermore, regarding automatic shooting, the shooting control unit 85 outputs a shooting command to the control unit 56 for the color camera when a predetermined waiting time has elapsed since the previous shooting (automatic or manual shooting) and the door state determination unit 83 has detected a shooting recommendation state. In both automatic and manual shooting, when the control unit 56 receives a shooting command, it outputs the contents of the frame of the moving image information S54 at that time as the captured image data GS to the wireless LAN unit 31g.

[0041] The wireless LAN unit 31g transmits the supplied captured image data GS to the router 151. The router 151 is a communication relay device and transmits the captured image data GS received from the wireless LAN unit 31g to the server 153 via the network 152. The server 153 performs predetermined processing on the captured image data GS and stores the resulting processed image data GM associated with the identification information of the refrigerator 100. Furthermore, if the server 153 receives a request signal for an image of the inside of the refrigerator 100 from the user's mobile terminal 154, it transmits the processed image data GM to the mobile terminal 154. Such a mobile terminal 154 can be a mobile phone, smartphone, tablet, wearable device, etc.

[0042] Figure 8 shows an example of the state in the field of view VS40 of EVS40. The illustrated field of view VS40 corresponds to the image acquired through a fisheye lens applied as lens 52. In Figure 8, the lower freezer compartment drawer door 251 is in the open position. During the process of the lower freezer compartment drawer door 251 changing from a closed state to an open state as shown in the figure, a change in brightness appears in the left and right regions A20 in the figure. Therefore, this change in brightness is also reflected in the EVS signal S44 (see Figure 7).

[0043] Figure 9 shows an example of state transitions around the lower freezer compartment drawer door 251. As the lower freezer compartment drawer door 251 is pulled out, the position of the lower freezer compartment drawer door 251 transitions from state ST11, ST12, and ST13 relative to the housing 101. Regions A21 (first region), A22 (first region), and A23 (first region, open state corresponding region) shown in Figure 9 are obtained by dividing region A20 shown in Figure 8 along the front-to-back direction. As the state of the lower freezer compartment drawer door 251 changes in the order of state ST11, ST12, and ST13, a change in brightness occurs in the order of region A21, A22, and A23.

[0044] The door state determination unit 83 (see Figure 7) determines that the lower freezer compartment drawer door 251 is in an open state, i.e., a state where photography is recommended, when a brightness change occurs in region A23 after a brightness change occurs in regions A21 and A22 in that order, and outputs a signal indicating that it is in an open state. Conversely, the door state determination unit 83 determines that the lower freezer compartment drawer door 251 is in a closed state when a brightness change occurs in region A21 after a brightness change occurs in region A23 and A22 in that order, and outputs a signal indicating that it is in a closed state.

[0045] When a predetermined waiting time has elapsed since the last shooting (automatic or manual shooting) and the lower freezer compartment drawer door 251 is in a state where shooting is recommended, the shooting control unit 85 outputs a shooting command to the color camera control unit 56. As a result, the frames of the moving image information S54 at the timing of state ST13 are supplied to the server 153 as captured image data GS.

[0046] Although the state when the vegetable compartment drawer door 241 (see Figure 1) is pulled out is not shown, the door state determination unit 83 and the shooting control unit 85 perform the same operations for the vegetable compartment drawer door 241. As a result, the frames of the moving image information S54 in the open state of the vegetable compartment drawer door 241 are supplied to the server 153 as captured image data GS.

[0047] Figure 10 shows another example of state transitions around the lower freezer compartment drawer door 251. In state ST31 shown in Figure 10, the lower freezer compartment drawer door 251 is in the same state as state ST13 in Figure 9. Furthermore, in state ST31, the user 200 is in close proximity to the lower freezer compartment container 253. However, the user 200 is away from the storage area A51 (second area) of the lower freezer compartment container 253. In this case, if the lower freezer compartment container 253 is stationary, no change in brightness occurs in the storage area A51.

[0048] On the other hand, in state ST32, user 200 is overlapping the storage area A51 from the previous state. In this case, a change in brightness occurs in storage area A51. Subsequently, when user 200 pulls the lower freezer drawer door 251 towards him, changes in brightness occur in areas A21, A22, and A23 in that order. Thus, when a change in brightness appears in storage area A51 and then in area A23, it is assumed that user 200 is overlapping storage area A51. Therefore, in this case, the shooting control unit 85 lowers the priority of the captured image data GS compared to the case where no change in brightness appeared in storage area A51.

[0049] More specifically, the shooting control unit 85 has a function to determine the "priority" for generating processed image data GM from the captured image data GS. The shooting control unit 85 sets the priority to "high" in state ST31 and to "low" in state ST32. When the captured image data GS is "low," the shooting control unit 85 shortens the waiting time allowed for acquiring the next captured image data GS compared to when the priority is "high."

[0050] For example, suppose that after the captured image data GS, in which user 200 overlaps storage area A51, is sent to the server 153, state ST31 occurs within a short time. Even in this case, the shooting control unit 85 causes the server 153 to send the captured image data GS in state ST31 again. As a result, the latest captured image data GS stored in the server 153 and the mobile terminal 154 will be one taken in the more favorable state ST31.

[0051] Figure 11 shows yet another example of the state transition around the lower freezer compartment drawer door 251. State ST21 shown in Figure 11 is the same state as state ST13 in Figure 9 for the lower freezer drawer door 251. Here, food 20 is stored in the lower freezer container 253. When the user removes the food 20 from the lower freezer container 253, the state of the lower freezer container 253 becomes as shown in state ST22. When the state of the lower freezer container 253 transitions from state ST21 to state ST22, a change in brightness occurs in region A36.

[0052] As a result, the item recognition unit 86 (see Figure 7) detects that some change has occurred in area A36 based on the EVS signal S44. Furthermore, the item recognition unit 86 recognizes the contents of the removed food item 20 based on the video information S54. Conversely, if food item 20 is newly placed in the lower freezer container 253, the item recognition unit 86 recognizes the contents of the stored food item 20 in the same manner.

[0053] In other words, when the item recognition unit 86 detects a change in brightness in area A23 using the EVS 40, and also detects a change in brightness in area A36, which is closer to the housing 101 than area A23, it performs item recognition based on the image captured by the color camera 50. Such processing to detect changes in state can also be performed using only the video information S54. However, as in this embodiment, the processing load can be reduced by using the EVS signal S44 in combination.

[0054] Figure 12 shows another example of the view VS40 of EVS40. Specifically, in Figure 12, the refrigerator doors 211 and 212 are open. When the refrigerator door 212 changes from a closed state to an open state, brightness changes occur in the order of regions A31, A32, A33, and A34 (the first region). The door state determination unit 83 (see Figure 7) determines that the refrigerator door 212 is in an open state, i.e., a state where photography is recommended, when brightness changes occur in region A34 after brightness changes have occurred in the order of regions A31, A32, and A33, and outputs a signal indicating that it is in an open state. Conversely, when brightness changes occur in region A31 after brightness changes have occurred in the order of regions A34, A33, and A32, the unit determines that the refrigerator door 212 is in a closed state and outputs a signal indicating that it is in a closed state. The shooting control unit 85 controls the shooting of the refrigerator door 212 when the refrigerator door 212 is in a state where photography is recommended and predetermined shooting conditions are met.

[0055] Furthermore, when the refrigerator door 212 is open, the item recognition unit 86 performs item recognition for the removed or stored food item if a change in brightness occurs in an area closer to the housing 101 than area A34 (for example, area A52). The operation of this item recognition unit 86 is the same as that described earlier in Figure 11. In addition, although the above example was for the refrigerator door 212, the door state determination unit 83, the shooting control unit 85, and the item recognition unit 86 perform similar control for the refrigerator door 211.

[0056] Figure 13 shows yet another example of the field of view VS40 of EVS40. Specifically, in Figure 13, as in Figure 12, the refrigerator doors 211 and 212 are in the open position. Here, when the chilled compartment container 263 (see Figure 3) inside the refrigerator compartment 121 changes from a closed state to an open state, a change in brightness occurs in the left and right regions A42. As a result, the door state determination unit 83 determines that the chilled compartment container 263 is in a state where photography is recommended. Therefore, the photography control unit 85 performs photography control for the chilled compartment container 263 when the predetermined photography conditions are met.

[0057] Furthermore, when the upper freezer compartment drawer door 231 (see Figure 1) changes from a closed state to an open state, a change in brightness occurs in the left and right regions A44. As a result, the door state determination unit 83 determines that the upper freezer compartment drawer door 231 is in a state where photography is recommended. Therefore, the photography control unit 85 performs photography control for the upper freezer compartment drawer door 231 when the predetermined photography conditions are met.

[0058] Furthermore, when the ice-making compartment drawer door 221 (see Figure 1) changes from a closed state to an open state, a change in brightness occurs in the left and right regions A46. As a result, the door state determination unit 83 determines that the ice-making compartment drawer door 221 is in a state where photography is recommended. Therefore, the photography control unit 85 performs photography control for the ice-making compartment drawer door 221 when the predetermined photography conditions are met.

[0059] Figure 14 shows an example of the peripheral state of the field of view VS40 of the EVS40. As described above, the camera LED 31c (see Figure 5) provided on the camera unit 3 can function as a welcome light when a user approaches the refrigerator 100. Therefore, the proximity determination unit 81 of this embodiment lights up the camera LED 31c when the user 200 enters the area A62 (predetermined area) shown in Figure 14.

[0060] More specifically, the proximity detection unit 81 monitors the EVS signal S44 and, after a change in brightness occurs at the periphery of the field of view VS40, determines whether the area of ​​brightness change has moved to area A62 in front of the refrigerator 100 and whether a change in brightness has occurred over an area greater than a predetermined area within area A62. If the determination result is positive, the proximity detection unit 81 considers that the user 200 has entered area A62 and lights up the camera LED 31c. The "predetermined area" may be, for example, approximately the area of ​​area A64 shown in Figure 14.

[0061] As described above, according to this embodiment, as shown in Figures 8 to 14, a single EVS 40 can detect the open / closed state of various doors (refrigerator doors 211, 212, drawer doors 221, 231, 241, 251), the status of the user 200, and so on.

[0062] Figure 15 shows examples of captured image data GS and processed image data GM (GS1, GM1). Note that in the captured image data GS1 of Figure 15, the top of the page is considered the front and the bottom of the page is considered the rear, so the left and right are reversed compared to Figure 3. Captured image data GS1 was taken with all drawer doors 221, 231, 241, 251 (see Figure 1) closed and both the left and right refrigerator doors 211, 212 open.

[0063] In the illustrated captured image data GS1, in addition to the refrigerator compartment 21, the door pocket 211c of the left refrigerator compartment door 211 and the door pocket 212c of the right refrigerator compartment door 212 are also captured. However, in the captured image data GS1, which has not undergone image processing, as shown in Figure 15, straight edges appear curved, and even for edges of the same length, the number of pixels in the captured image decreases as they move further away from the camera unit 3.

[0064] Furthermore, the processed image data GM1 is an example of the result of image processing performed by the server 153 (see Figure 8) on the captured image data GS1. The processed image data GM1 is an image that looks as if the refrigerator compartment 21 and the left and right refrigerator compartment doors 211 and 212 are viewed from the front. When such processed image data GM1 is displayed on the mobile terminal 154 (see Figure 8), the user can see at a glance what kind of food is stored in the refrigerator compartment 21, etc.

[0065] The captured image data GS1 does not include images of containers 223, 233, 243, and 253 (see Figure 1). However, if one of the drawer doors were opened during the capture, the corresponding container's image would be included within the area AR1 shown by the dashed line in the captured image data GS1.

[0066] Figure 16 shows other examples (GS2, GM2) of captured image data GS and processed image data GM. Captured image data GS2 was taken with both the left and right refrigerator compartment doors 211, 212 closed, and the other drawer doors 221, 231, 251 (see Figure 1) closed, with the vegetable compartment drawer door 241 in the open position. In the illustrated captured image data GS2, parts of the refrigerator compartment doors 211, 212, as well as the ice maker drawer door 221 and the upper freezer compartment drawer door 231 are also captured.

[0067] Furthermore, the processed image data GM2 is an example of the result of performing image processing on the captured image data GS2 using the server 153 (see Figure 8). The processed image data GM2 is an image of the vegetable compartment container 243 as if viewed from the front. By displaying such processed image data GM2 on the mobile terminal 154 (see Figure 8), the user can quickly understand what kind of food is stored in the vegetable compartment container 243.

[0068] [Second Embodiment] Next, a refrigerator according to the second embodiment will be described. The refrigerator of the second embodiment is the same as the refrigerator 100 of the first embodiment (see Figure 1), except for the configuration of the camera unit 3. In the description of each embodiment, the same reference numerals are used for parts corresponding to parts of the other embodiments described above, and their descriptions may be omitted. Figure 17 is a schematic diagram of the main parts of the camera unit 3 in the second embodiment. The camera unit 3 in the second embodiment includes a lens 302, a reflector 304 (reflector part), an EVS sensor 44, and an image sensor 54.

[0069] In the camera unit 3 according to the first embodiment, separate lenses 42 and 52 were applied to the EVS 40 and the color camera 50. In the second embodiment, a common lens 302 is applied instead of these lenses 42 and 52. In the normal state ST41, light incident on the lens 302 is reflected by the reflector 304 and incident on the EVS sensor 44.

[0070] Here, let's assume that the door state determination unit 83 (see Figure 7) detects, based on the EVS signal S44, that the refrigerator doors 211, 212 or the drawer doors 221, 231, 241, 251 have entered a state where photography is recommended. Then, the door state determination unit 83 moves the reflector 304 as shown in state ST42. As a result, the light incident on the lens 302 travels in a straight line and is incident on the image sensor 54. The configuration and operation of the refrigerator according to the second embodiment, other than those described above, are the same as those of the first embodiment (Figures 1 to 16).

[0071] In the example shown in Figure 17, the reflector 304 directs the light incident on the lens 302 to either the EVS sensor 44 or the image sensor 54. However, by using a half-mirror for the reflector 304, it is possible to direct light to both the EVS sensor 44 and the image sensor 54 while keeping the reflector 304 fixed in position ST41. In Figure 17, the positions of the EVS sensor 44 and the image sensor 54 may be swapped.

[0072] [Computer Configuration] Figure 18 is a block diagram of the computer 980. The control units 46 and 56, camera control unit 80, refrigerator body control unit 90, server 153, and mobile terminal 154 shown in Figure 7 each consist of one or more computers 980 as shown in Figure 18. In Figure 18, the computer 980 includes a CPU (Central Processing Unit) 981, a storage unit 982, a communication port 983, an input / output port 984, and a DSP (Digital Signal Processor) 985.

[0073] Here, the memory unit 982 includes RAM 982a and ROM 982b. The communication port 983 is connected to the communication circuit 986. The input / output port 984 is connected to the input / output device 987. ROM 982b stores control programs and various data executed by the CPU 981 and DSP 985. The CPU 981 and DSP 985 realize various functions by executing these control programs. The internal structure of the camera control unit 80 shown earlier in Figure 7 is a representation of functions realized by application programs and the like as blocks.

[0074] [Differentiation] The present invention is not limited to the embodiments described above, and various modifications are possible. The embodiments described above are illustrative examples provided to facilitate understanding of the present invention, and are not necessarily limited to those comprising all the described configurations. Furthermore, it is possible to replace parts of the configuration of one embodiment with those of another embodiment, and to add configurations from other embodiments to the configuration of one embodiment. It is also possible to delete parts of the configuration of each embodiment, or to add or replace other configurations. In addition, the control lines and information lines shown in the figures are those considered necessary for explanation, and do not necessarily represent all control lines and information lines required in the product. In practice, it can be assumed that almost all configurations are interconnected. Possible modifications to the above embodiments are as follows, for example.

[0075] (1) In the above embodiment, the item recognition unit 86 is provided in the camera control unit 80 inside the refrigerator 100, but the item recognition unit 86 may be provided in the server 153.

[0076] (2) The hardware of the control units 46, 56, 80, server 153, and mobile terminal 154 (see Figure 7) in each of the above embodiments can be implemented using a general-purpose computer. Therefore, programs that perform the various processes described above may be stored in a storage medium (a computer-readable recording medium on which the program is recorded) or distributed via a transmission line.

[0077] (3) The storage facility of the present invention is not limited to refrigerators, and storage facilities include warming cabinets, temperature- and humidity-controlled cabinets, and other facilities that do not have a function to control temperature or humidity.

[0078] (4) Although the various processes described above were explained as software processes using a program in the above embodiment, some or all of them may be replaced with hardware processes using an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), etc.

[0079] [Effects of the Embodiment] As described above, according to the embodiment described above, the storage unit (100) comprises a housing 101 on which doors (211, 212, 221, 231, 241, 251) are arranged, and a brightness change detection unit (40) having a plurality of pixels and detecting brightness changes of each pixel in a first field of view (VS40) that includes the range in which the doors (211, 212, 221, 231, 241, 251) open and close and move. By providing the brightness change detection unit (40) in this way, the state of the storage unit can be appropriately acquired.

[0080] Furthermore, the first field of view (VS40) includes multiple first regions (A21-A23, A31-A34) arranged along the trajectory of the doors (211, 212, 221, 231, 241, 251) as they open and close, and the brightness change detection unit (40) detects brightness changes in the multiple first regions (A21-A23, A31-A34) in order from the closed side of the doors (211, 212, 221, 231, 241, 251). It is even more preferable to further include a door state determination unit 83 that outputs a signal indicating that a door (211, 212, 221, 231, 241, 251) is open, and outputs a signal indicating that a door (211, 212, 221, 231, 241, 251) is closed when it detects a change in brightness in a plurality of first regions (A21~A23, A31~A34) in order from the open side of the door (211, 212, 221, 231, 241, 251). This allows for proper recognition of the open / closed state of the door and more accurate acquisition of the storage facility's state.

[0081] Furthermore, the storage room (100) is further equipped with a color camera 50 having a second field of view (VS50) which is at least partially common with the first field of view (VS40), and the door includes a retractable door (221, 231, 241, 251) that can reciprocate in the front-rear direction, and the multiple first regions (A21~A23) are arranged along the front-rear direction on the left and right ends of the retractable doors (221, 231, 241, 251), with the direction perpendicular to the front-rear direction being the left-right direction, and has a function to determine the priority for processing images captured by the color camera 50 to generate processed image data GM. It is even more preferable to further include an imaging control unit 85 having the following function: to detect a change in brightness in the open state corresponding region (A23) among the multiple first regions (A21~A23) that corresponds to the open state of the drawer doors (221, 231, 241, 251), to determine that the drawer doors (221, 231, 241, 251) have been opened, and to lower the priority of detecting a change in brightness in the second region (A51) sandwiched between the left and right first regions (A21~A23) before detecting the change in brightness in the open state corresponding region (A23), compared to the case where no change in brightness in the second region (A51) was detected. This allows for lowering the priority, for example, when a person is in the second region (A51), and enables more appropriate acquisition of the storage unit's state.

[0082] Furthermore, it is even more preferable that the storage unit (100) further includes a proximity determination unit 81 that determines that a person has approached the storage unit (100) when the brightness change area moves from the periphery of the first field of view (VS40) to a predetermined area (A62) in front of the storage unit (100). This makes it possible to appropriately detect when a person has approached the storage unit (100).

[0083] Furthermore, it is even more preferable that the storage room (100) further includes a color camera 50 having a second field of view (VS50) that is at least partially common with the first field of view (VS40). This allows the color camera 50 to acquire images of the items more appropriately in accordance with the detection results of the brightness change detection unit (40).

[0084] Furthermore, the door includes retractable doors (221, 231, 241, 251) that can move back and forth in the front-to-back direction, the brightness change detection unit (40) is equipped with a first lens (42), and the color camera 50 is more preferably equipped with a second lens (52) positioned closer to the housing 101 than the first lens (42). This allows the color camera 50 to acquire images of the objects more appropriately.

[0085] Furthermore, the door includes a revolving door (211, 212) that rotates around a rotation axis (211a, 212a), the brightness change detection unit (40) is equipped with a first lens (42), and the color camera 50 is more preferably equipped with a second lens (52) positioned closer to the rotation axis (211a, 212a) than the first lens (42). This allows the color camera 50 to acquire images of the object more appropriately.

[0086] Furthermore, as in the second embodiment, the brightness change detection unit (40) comprises a lens 302, a reflecting unit (304) that transmits a portion of the light incident from the lens 302 and reflects the rest, and a brightness change sensor (44). The color camera 50 comprises an image sensor 54, and it is even more preferable that the brightness change sensor (44) is positioned in either the transmission direction or the reflection direction of the reflecting unit (304), and the image sensor 54 is positioned in the other direction of the transmission direction or the reflection direction of the reflecting unit (304). This allows a common lens 302 to be applied to the color camera 50 and the brightness change detection unit (40).

[0087] Furthermore, the first field of view (VS40) includes a plurality of first regions (A21~A23, A31~A34) arranged along the trajectory when the doors (211, 212, 221, 231, 241, 251) open and close. It is even more preferable to further include an item recognition unit 86 that performs item recognition for the removed or stored item based on the image capture results of the color camera 50. This allows the item recognition unit 86 to perform item recognition for the removed or stored item. [Explanation of Symbols]

[0088] 40 EVS (Brightness Change Detection Unit) 42 Lens (First Lens) 44 EVS sensor (brightness change sensor) 50 Color Camera 52 Lens (Second Lens) 54 Image Sensors 81 Proximity detection unit 83 Door state determination unit 85 Imaging Control Unit 86 Item Recognition Unit 100 Refrigerator (storage) 101 cabinets 211,212 Refrigerator door (door, revolving door) 211a, 212a Hinge (rotation axis) 221 Ice maker compartment drawer door (door, drawer door) 231 Upper freezer compartment drawer door (door, drawer door) 241 Vegetable compartment drawer door (door, drawer door) 251 Lower freezer compartment drawer door (door, drawer door) 302 Lens 304 Reflector (reflector) GM processed image data Areas A21, A22, A31-A34 (First Area) A23 Region (First region, Open state corresponding region) A51 Storage area (second area) A62 area (specified area) VS40 Field of View (First Field of View) VS50 Field of View (Second Field of View)

Claims

1. A cabinet with doors, The system includes a brightness change detection unit having multiple pixels and detecting brightness changes of each pixel in a first field of view that includes the range in which the door opens and closes. A storage facility characterized by the following features.

2. The first field of view includes a plurality of first regions arranged along the trajectory of the door as it opens and closes, The door state determination unit further comprises a door state determination unit that outputs a signal indicating the door is open when the brightness change detection unit detects brightness changes in a plurality of first regions in order from the closed side of the door, and outputs a signal indicating the door is closed when the brightness change detection unit detects brightness changes in a plurality of first regions in order from the open side of the door. The storage facility according to feature 1.

3. The system further comprises a color camera having a second field of view that is at least partially common to the first field of view, The aforementioned door includes a retractable door that can move back and forth in the front-to-back direction. The multiple first regions are arranged along the front-to-back direction, with the direction perpendicular to the front-to-back direction being the left-to-right direction, on both the left and right ends of the drawer door. The shooting control unit further includes a function to determine the priority for processing images captured by the color camera to generate processed image data, and a function to detect a change in brightness in an open state corresponding region among a plurality of first regions that corresponds to the open state of the drawer door, determine that the drawer door is open, and, if a change in brightness in a second region sandwiched between the left and right first regions is detected before the change in brightness in the open state corresponding region is detected, lower the priority compared to the case where no change in brightness in the second region is detected. The storage facility according to feature 2.

4. The system further includes a proximity determination unit that determines that a person has approached the storage unit when the brightness change area moves from the periphery of the first field of view to a predetermined area in front of the storage unit. The storage facility according to feature 1.

5. The system further comprises a color camera having a second field of view that is at least partially common to the first field of view. The storage facility according to feature 1.

6. The aforementioned door includes a retractable door that can move back and forth in the front-to-back direction. The brightness change detection unit includes a first lens, The color camera includes a second lens positioned closer to the housing than the first lens. The storage facility according to feature 5.

7. The aforementioned door includes a revolving door that rotates around a pivot axis, The brightness change detection unit includes a first lens, The color camera includes a second lens positioned closer to the axis of rotation than the first lens. The storage facility according to feature 5.

8. The brightness change detection unit comprises a lens, a reflecting unit that transmits a portion of the light incident from the lens and reflects the remainder, and a brightness change sensor. The aforementioned color camera is equipped with an image sensor, The brightness change sensor is positioned in either the transmission direction or the reflection direction of the reflective portion. The image sensor is positioned in the transmission direction or the other direction of reflection of the reflective portion. The storage facility according to feature 5.

9. The first field of view includes a plurality of first regions arranged along the trajectory of the door as it opens and closes, The system further includes an item recognition unit that, when the brightness change detection unit detects a brightness change in the open state corresponding area of ​​the first region that corresponds to the open state of the door, and also detects a brightness change in a second region closer to the housing than the open state corresponding area, performs item recognition of the removed or stored item based on the image capture results of the color camera. The storage facility according to feature 5.

Images