Imaging system

Abstract

This system provides an imaging system that ensures a large amount of information is contained in a single image. [Solution] The imaging system is applied to a storage facility comprising a box-shaped body having a storage area and an open front, and an openable and closable door arranged to open and close the opening of the box-shaped body. The system comprises an imaging unit capable of imaging such that the storage area is included in the field of view when the door is open, and a removal processing unit 33 that determines a removal range from within the storage area captured in the first image data captured by the imaging unit, and overwrites the removal range of the first image data with a second image data captured in the vicinity before the imaging time of the first image data. The removal processing unit 33 determines the area with large pixel changes as the removal range between two image data captured at two nearby times in substantially the same field of view.

Description

Technical Field

[0001] The present invention relates to an imaging system.

Background Art

[0002] Patent Document 1 discloses a photographing unit that photographs a moving image when viewing the opening of a storage from above, and by performing image processing on the moving image, detects whether still objects other than the storage are reflected in a first area of the moving image. A first detection unit, a second detection unit that detects whether a user's hand or arm located in front of the opening is reflected in a second area of the moving image by performing image processing on the moving image, and a second area. A determination unit that determines that the still object is an object to be subject to inventory management on the condition that it is detected that a still object other than the storage is reflected in the first area during a period when the user's hand or arm is not reflected in the second area is described.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 discloses detecting the presence or absence of movement of an object by photographing a first reference image obtained at the detection of the opening of a door, at the time of initial setting, or at any time while the door is open, and using the difference in pixel values included in this and the frame at the current time (see paragraphs 0041 to 0046, FIGS. 3 to 6).

[0005] When considering a function that uses camera images of the food storage area to display information about food inventory to the user, it is desirable to improve user experience by capturing the entire food storage area in as few images as possible, preferably just one, to ensure high overview. However, human bodies such as faces and hands, or other obstructions, may appear in the camera's field of view and obscure the food. It is desirable to create images that remove such obstructions that temporarily hide the food, thereby improving the amount of information that can be confirmed in a single image.

[0006] Patent Document 1 estimates whether a user intentionally placed food in a specific area P, that is, whether they intend to use the food for inventory management (paragraph 0065), and does not attempt to secure a large amount of information in a single image, so there is room for improvement. [Means for solving the problem]

[0007] The imaging system according to this disclosure is applied to a storage facility comprising a box-shaped body having a storage area and an open front, and an openable and closable door arranged to open and close the opening of the box-shaped body, and comprises an imaging unit capable of imaging such that the storage area is included in the field of view when the door is open, and a removal processing unit that determines a removal range from within the storage area captured in first image data captured by the imaging unit, and overwrites the removal range of the first image data with second image data captured in the vicinity before the time the first image data was captured. Other aspects of the present invention will be described in the embodiments described below. [Brief explanation of the drawing]

[0008] [Figure 1] This is a front view of a refrigerator having an imaging system according to an embodiment. [Figure 2] This is a front view of the refrigerator according to the embodiment, with the left and right refrigerator compartment doors open. [Figure 3] This is a plan view of the refrigerator according to the embodiment, with the left and right refrigerator compartment doors open. [Figure 4]This is a cross-sectional view of the camera unit according to the embodiment. [Figure 5] This is a system configuration diagram of a refrigerator including an imaging system according to the embodiment. [Figure 6] This is a flowchart showing the processing of the removal unit according to the embodiment. [Figure 7] This figure shows the relationship between the captured image and the output image according to the embodiment (part 1). [Figure 8] This figure shows the relationship between the captured image and the output image according to the embodiment (part 2). [Figure 9] This figure shows the relationship between the captured image and the output image according to the embodiment (part 3). [Figure 10] This figure shows the method for setting up an area according to the embodiment (part 1). [Figure 11] This figure shows the method for setting up the area according to the embodiment (part 2). [Figure 12] This figure shows the method for setting up an area according to the embodiment (part 3). [Figure 13] This is a schematic side view of the refrigerator according to the embodiment, with each door closed. [Figure 14] This is a schematic side view of the lower freezer compartment door of the refrigerator according to the embodiment, in a fully extended state. [Figure 15] This is an example of the image capture result from the camera unit of the refrigerator according to the embodiment, before any image processing has been performed. [Figure 16] This figure shows the immovable area of ​​a drawer container, etc., according to the embodiment. [Modes for carrying out the invention]

[0009] Embodiments for carrying out the present invention will be described in detail with reference to the drawings as appropriate. Figure 1 is a front view of a refrigerator 100 having an imaging system according to an embodiment. The refrigerator 100 (storage unit) shown in Figure 1 is a device for storing food and other items at low temperatures, and includes a housing 1, various doors such as refrigerator doors 211 and 212, and a camera unit 3. The housing 1 is constructed by filling a vacuum insulation material or foamed urethane insulation material (not shown) between a steel plate outer box 11 and a resin inner box 12 (see Figure 3). The housing 1 is also provided with multiple storage compartments. In the example in Figure 1, the multiple storage compartments are, from top to bottom, a refrigerator compartment 21, ice-making compartments 22 and an upper freezer compartment 23 arranged on the left and right, a lower freezer compartment 24, and a vegetable compartment 25.

[0010] Furthermore, the refrigerator 100 is equipped with multiple doors corresponding to each storage compartment, including refrigerator compartment doors 211 and 212, an ice maker door 221, an upper freezer compartment door 231, a lower freezer compartment door 241, and a vegetable compartment door 251. The refrigerator compartment doors 211 and 212 are a pair of left and right French-style doors that form the refrigerator compartment 21 by closing the opening on the front side of the housing 1. The left refrigerator compartment door 211 is rotatable around the axis of the hinge 211a (see Figure 3) at the left end. The same applies to the right refrigerator compartment door 212.

[0011] The remaining doors—the ice maker door 221, the upper freezer door 231, the lower freezer door 241, and the vegetable compartment door 251—are all retractable doors. These retractable doors are positioned lower than the refrigerator doors 211 and 212. In other words, the French-style refrigerator doors 211 and 212 are positioned higher than the multiple retractable doors. This means that the refrigerator doors 211 and 212 are closer to the camera unit 3 than the retractable doors.

[0012] Behind the rear side (inner side) of the vegetable compartment door 251, a vegetable compartment container 252 (see FIG. 14) is installed. The vegetable compartment door 251 and the vegetable compartment container 252 are integrally pulled out (the same applies to other pull-out doors). Note that the "storage compartment" corresponding to the pull-out door is defined as the internal space of the container that is pulled out integrally with the pull-out door. For example, the vegetable compartment 25 is the internal space of the vegetable compartment container 252 (see FIG. 14) that is pulled out integrally with the vegetable compartment door 251.

[0013] Although not shown in the figure, the refrigerator 100 includes a compressor, a radiator (condenser), a capillary tube (throttling mechanism), and a cooler (evaporator). The refrigerant circulates sequentially through the compressor, radiator, capillary tube, and cooler, and the air in each storage compartment is cooled by heat exchange with the refrigerant flowing through the cooler. The camera unit 3 shown in FIG. 1 is for photographing the storage compartment when a predetermined door is opened, and is provided on the top surface of the housing 1.

[0014] FIG. 2 is a front view of the refrigerator 100 according to the embodiment with the left and right refrigerator compartment doors 211 and 212 opened. As shown in FIG. 2, the refrigerator compartment 21 is provided with a plurality of shelf boards 213 that partition the refrigerator compartment 21 into a predetermined configuration. The inner plate 211b of the left refrigerator compartment door 211 is provided with a plurality of door pockets 211c for storing food and the like (the same applies to the right refrigerator compartment door 212). When the left and right refrigerator compartment doors 211 and 212 are opened, for example, the food and the like in the refrigerator compartment 21 and the door pockets 211c and 212c are panoramically within the field of view of the lens 31a of the camera unit 3.

[0015] When the storage room is automatically photographed using the camera unit 3, if it is detected that the door has been opened, images are taken at predetermined intervals and output as multiple frames (see Figure 7). Alternatively, for example, when the storage room is manually photographed using the camera unit 3, a shutter button 7, which is pressed by the user, may be provided on both the left and right refrigerator doors 211 and 212. In the example in Figure 3, the shutter button 7 is provided on the lower part of the side of the left refrigerator door 211 opposite the axis of the hinge 211a (see Figure 4). Similarly, the shutter button 7 is provided in the same position on the right refrigerator door 212.

[0016] Figure 3 is a plan view of the refrigerator 100 according to the embodiment, with the left and right refrigerator compartment doors 211 and 212 open. As shown in Figure 3, hinges 211a and 212a are provided on the top surface of the housing 1. The left hinge 211a pivotally supports the refrigerator compartment door 211 so that it can rotate freely (the right hinge 212a does the same). The main body 31 of the camera unit 3 extends in a long, narrow shape in the front-to-back direction. The left-to-right position of the main body 31 may be directly above the joint of the refrigerator compartment doors 211 and 212 (see Figure 1), or it may be at another predetermined position. Furthermore, the camera unit 3 may be configured to be detachable.

[0017] Figure 4 is a cross-sectional view of the camera unit 3 according to the embodiment. The main body 31 of the camera unit 3 includes, in addition to the lens 31a, case 31b, and cover 31c mentioned above, an image sensor 31d, an LED substrate 31e, a camera LED 31f, a buzzer 31g, and a camera control substrate 31h. The image sensor 31d is an element that converts light incident through the lens 31a into photoelectric data and generates captured image data. For example, a CCD sensor (Charge Coupled Device) or a CMOS sensor (Complementary Metal Oxide Semiconductor) can be used as such an image sensor 31d.

[0018] The LED board 31e is a board on which the camera LED 31f and buzzer 31g are installed. The camera LED 31f is a light source that lights up when the camera unit 3 takes a picture. This ensures that, for example, when the vegetable compartment door 251 (see Figure 1) is opened, the vegetable compartment 25 (see Figure 1) is photographed with appropriate brightness. The buzzer 31g emits a predetermined sound when the camera unit 3 takes a picture. The camera control board 31h is a board on which the camera microcontroller 311h is mounted, and may be installed, for example, at the rear or front of the case 31b, or it may be configured so that all components (lens 31a, image sensor 31d, camera LED 31f, buzzer 31g, camera microcontroller 311h, etc.) are integrated onto the camera control board 31h, or a wireless LAN unit 4 may be placed inside the camera unit 3. Alternatively, the camera microcontroller 311h may be replaced with a camera / communication control SoC 31k.

[0019] Figure 5 is a system configuration diagram of a refrigerator 100 including an imaging system 200 according to an embodiment. As shown in Figure 5, the refrigerator 100 includes a camera unit 3, a wireless LAN unit 4, a door sensor 5, a control panel 6, a shooting button 7, and a control unit 8. In addition to the aforementioned lens 31a, image sensor 31d, camera LED 31f, and buzzer 31g, the camera unit 3 includes a camera / communication control SoC 31k.

[0020] The camera / communication control SoC31k is an integrated circuit designed to integrate microcontroller functions and other application functions onto a single chip and to function in coordination. The camera / communication control SoC31k communicates with the control unit 8 as specified and outputs a capture command to the image sensor 31d. As a result, captured image data is input from the image sensor 31d to the camera / communication control SoC31k.

[0021] The camera / communication control SoC 31k has a removal processing unit 33, which is described in detail in Figures 6 to 12. The removal processing unit 33 determines a removal range from within the storage area captured in the first image data captured by the imaging unit, and overwrites the removal range of the first image data with second image data captured in the vicinity before the time the first image data was captured (overwriting the removal range). The removal range can be determined based on the change (difference) in pixel values ​​between the pixels constituting the first image data and the pixels constituting the second image corresponding to those pixels.

[0022] The wireless LAN unit 4 is a device that transmits and receives data with the server 63 via the network 62, etc. The wireless LAN unit 4 transmits the output image data acquired by the camera / communication control SoC 31k and processed by the removal processing unit 33 to the router 61. The router 61 is a communication relay device and transmits the output image data received from the wireless LAN unit 4 to the server 63 via the network 62. The server 63 performs predetermined processing on the output image data and stores it in association with the identification information of the refrigerator 100. In addition, if the server 63 receives a request signal for an image of the inside of the refrigerator 100 from a mobile terminal 64 that has been paired via the control panel 6, it transmits the output image data to the mobile terminal 64. Examples of such mobile terminals 64 include mobile phones, smartphones, tablets, and wearable devices.

[0023] The door sensor 5 is a sensor that detects the opening and closing of each door of the refrigerator 100. Although Figure 7 shows one door sensor 5, in reality, at least one door sensor 5 is provided for each door (see Figure 13). A predetermined signal indicating the open or closed state of the door is output from the door sensor 5 to the control unit 8. By providing a door sensor 5 for each door in this way, it is possible to associate the output image with the storage compartment.

[0024] The control panel 6 displays the status of the refrigerator 100 and accepts operations such as changing settings, and is located, for example, on the surface of the refrigerator doors 211 and 212 (see Figure 1). The shooting button 7 (see also Figure 2) is a button pressed by the user when manually taking a picture using the camera unit 3, as described above.

[0025] The control unit 8 is, for example, a microcontroller, and although not shown in the diagram, it is composed of electronic circuits including a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), and various interfaces. It reads the program stored in the ROM, loads it into the RAM, and the CPU executes various processes.

[0026] The imaging system 200 is applied to a storage facility that includes a box-shaped body (housing 1) with a storage area and an open front, and openable and closable doors (for example, refrigerator doors 211, 212, lower freezer door 241, vegetable compartment door 251) arranged to open and close the opening of the box-shaped body. The system includes an imaging unit (for example, an image sensor 31d) capable of imaging so that the storage area is included in the field of view when the door is open, and a removal processing unit 33 that determines a removal range from within the storage area captured in the first image data captured by the imaging unit, and overwrites the removal range of the first image data with a second image data captured in the vicinity before the time the first image data was captured. The storage area includes, for example, a refrigerator compartment 21, a lower freezer compartment 24, and a vegetable compartment 25. The second image data captured in the vicinity before the time the first image data was captured is an image (output image) from which an obstruction that temporarily hides food has been removed. Furthermore, the time zone immediately preceding the set time can be appropriately set within a range that effectively removes any obstructions that might temporarily conceal the food.

[0027] Figure 6 is a flowchart showing the process S60 of the removal processing unit 33 according to the embodiment. The camera / communication control SoC 31k takes a picture at predetermined intervals when the door is open and stores each picture as a frame. The removal processing unit 33 determines whether it is the first frame (1st frame) after the start of imaging, in other words, whether it is the second frame or later (process S61). If it is the first frame (process S61, No), the removal processing unit 33 uses the image captured in the current frame as the output image of the original frame (process S62). The output image refers to an image that may be provided to the user.

[0028] On the other hand, if the image is from the second frame onwards (processing S61, Yes), the removal processing unit 33 creates an output image for the current frame (processing S63). Specifically, areas (pixels) in the current frame's captured image that have changed significantly from the previous frame's captured image are retained as the removal range, and the previous frame's output image is used. Other areas (pixels) are updated to reflect the current frame's captured image.

[0029] The removal processing unit 33 then determines whether it is the final frame while the door is open (process S64). If it is the final frame (process S64, Yes), it terminates the series of processes. If it is not the final frame (process S64, No), it returns to process S61.

[0030] In processing S63, the removal processing unit 33 determines the removal range as an area where pixel changes are large between two image data (imported image data) captured at two neighboring time points within substantially the same field of view. That is, the difference between the pixel value of a pixel in a frame captured in the vicinity before the imaging time and the pixel value of a pixel in a frame captured at the imaging time is calculated. If this difference is greater than or equal to a predetermined value, it is determined to be a pixel with large changes.

[0031] Regarding process S60, a specific example will be explained with reference to Figures 7 to 12. Figure 7 shows the relationship between the captured image and the output image according to the embodiment (part 1). Figure 8 shows the relationship between the captured image and the output image according to the embodiment (part 2). Figure 9 shows the relationship between the captured image and the output image according to the embodiment (part 3). Figure 8 shows frames 1 to 3, Figure 8 shows frames 4 and 5, and Figure 9 shows frames 6 and 7. The upper part of each figure shows the captured image, and the lower part shows the output image. In each figure, parts other than the storage section corresponding to the refrigerator compartment 21 (see Figure 2) and the door storage section corresponding to the door pocket 211c (see Figure 2) are masked in black to make the stored items clear. In terms of imaging time, frame 1 is captured first, followed by frame 2, then frame 3, and so on, in a chronological order of imaging.

[0032] Frame 1: The captured image shows the storage unit and the door storage compartments on both sides. The output image for the first frame is the captured image of the current frame (frame 1). Frame 2: Since there were no areas with large pixel changes, the output image for the second frame will be the image captured in the current frame (frame 2).

[0033] 3 frames: In the captured image, an area with large pixel changes is identified because a hand appears. In the output image of 3 frames, the area with large pixel changes is retained from the output image of 2 frames, while the other areas are retained from the captured image of the current frame. That is, in a comparison between the captured image data of the previous frame, 2 frames (second image data), and the captured image data of the current frame, 3 frames (first image data), the area with large pixel changes in 3 frames (first image data) is determined as the removal range. That area (removal range) in the captured image of 3 frames is overwritten by the corresponding part (area) of the output image (or captured image) of 2 frames, and becomes the output image of 3 frames.

[0034] 4 frames: In the captured image, areas where the hand moves and pixel changes are large are identified. In the output image of 4 frames, the areas with large pixel changes are retained from the output image of 3 frames, while the other areas are retained from the captured image of the current frame. That is, in a comparison between the captured image of 3 frames (second image data) in Figure 7, which is the previous frame, and the captured image of 4 frames (first image data) in Figure 8, which is the current frame, there are two areas with large pixel changes in 4 frames, as indicated by the dashed lines.

[0035] First, in comparing the image of the current frame with the image of the previous frame (comparing the images themselves), as shown in Figure 8, there are two areas of removal with large pixel changes, and these two areas of removal are overwritten by the output image of the previous frame. In other words, the two areas designated as removal areas in the image of frame 4 retain the portion of the output image of frame 3. After overwriting, this becomes the output image of frame 4.

[0036] Frame 5: In this Frame 5, the hand moves even more than in Frame 4. In this captured image, as with Frame 4, areas where the hand moves and pixel changes are large are identified. Incidentally, when comparing captured images, there are two areas with large pixel changes, and these are designated as the removal range. In the output image of Frame 5, as with Frame 4, the areas with large pixel changes are retained from the output image of Frame 4, while the other areas are retained from the captured image of the current frame.

[0037] Frame 6: In the captured image, areas with large pixel changes are identified because the hand has disappeared. When compared with the captured image of the previous frame, there is only one area with large pixel changes. Similarly, in the output image of Frame 6, the area with large pixel changes is retained from the output image of Frame 5, while the other areas are retained from the captured image of the current frame. Frame 7: Since there were no areas with large pixel changes, the output image for the 7th frame will be the image captured in the current frame.

[0038] Figures 7 to 9 illustrate the appearance, movement, and disappearance of a hand (arm), but the same principle applies when an object is removed by the hand. For example (though not shown in the illustration), if an object in the storage room is grasped by the hand in frame 5 of Figure 8, and both the object and the hand disappear in frame 9, the area where the object disappeared is determined to be an area with a large pixel change. Then, the processing S63 in Figure 6 is performed. At this time, areas without objects are temporarily processed to appear as if objects exist in order to maintain the output image of the previous frame. However, if it is determined that there is no area with a large pixel change in the next frame, the image will be captured for the current frame, resulting in an image without the erased object, so no problem arises. Furthermore, although the explanation here uses the appearance of a hand, a key feature of this embodiment is that, unlike Patent Document 1, the image processing is performed based on whether or not there is a pixel change, rather than determining whether or not there is a hand.

[0039] Furthermore, if the imaging unit is a thermal camera capable of detecting the temperature of an object, the removal processing unit 33 may determine objects within the human body temperature range as the removal area. This ensures that the human hand or arm area is reliably determined as the removal area.

[0040] (How to set the area) Figure 10 shows a method for setting an area according to the embodiment (part 1). Figure 11 shows a method for setting an area according to the embodiment (part 2). Figure 12 shows a method for setting an area according to the embodiment (part 3). In Figure 6, an area with large pixel changes is set (i.e., determined as the removal range), but in addition to the process of setting an area on a pixel-by-pixel basis, as shown in Figures 10 to 12, the area may also be set to include pixels surrounding pixels (i.e., determined as the removal range) with large pixel changes.

[0041] In Figure 10, setting areas 10A and 10B represent the case where a pixel with large pixel variation is set to "1" for the surrounding pixels. In Figure 11, setting areas 11A and 11B represent the case where the surrounding pixels are set to "2" for a pixel (pixel) with large pixel variation. In Figure 12, the setting areas 12A and 12B represent cases where multiple pixels are pre-defined as areas, and the area containing pixels with significant changes is designated as the area with change.

[0042] As shown in Figure 2, the refrigerator 100 has a storage area consisting of a stationary area (excluding the door pockets, 21) that does not move even when the door is opened and closed, and a movable area (door pockets, 22-25) that moves with the door.

[0043] Therefore, at least a portion of the immovable area may be set as a processing area to be processed by the removal processing unit 33, and at least a portion of the area into which the movable area can enter may be set as a non-processing area not to be processed by the removal processing unit 33. This makes it possible to shorten the processing time. The processing area refers to the area where the process of determining the removal range is performed.

[0044] (Door sensor location) Figure 13 is a schematic side view of the refrigerator 100 according to this embodiment with each door closed. As shown in Figure 13, the refrigerator 100 is equipped with multiple door sensors 5, including a door sensor 52 for the right refrigerator compartment door 212 (see Figure 1), a door sensor 53 for the upper freezer compartment door 231, a door sensor 54 for the lower freezer compartment door 241, and a door sensor 55 for the vegetable compartment door 251. In addition, although not shown in Figure 13, door sensors 5 are also installed on the left refrigerator compartment door 211 (see Figure 1) and the ice maker door 221 (see Figure 1).

[0045] For example, the door sensor 52 for the right-side refrigerator door 211 includes a magnet 52a embedded in the refrigerator door 212 and a Hall element 52b located in the housing 1 opposite the magnet 52a. A predetermined signal corresponding to the open / closed state of the refrigerator door 212 is output from the Hall element 52b to the control unit 8 (see Figure 5). The remaining door sensors 5 have a similar configuration.

[0046] As shown in Figure 13, each door of the refrigerator 100 overlaps in height with other predetermined doors when closed. Specifically, for example, the right refrigerator door 212 (see also Figure 1) overlaps in height with the upper freezer door 231, the lower freezer door 241, and the vegetable compartment door 251 when closed. Also, for example, the vegetable compartment door 251 overlaps in height with the left and right refrigerator doors 211 and 212 (see Figure 1), as well as the ice maker door 221 (see Figure 1), the upper freezer door 231, and the lower freezer door 241 when closed.

[0047] Figure 14 is a schematic side view of the vegetable compartment door 251 according to the embodiment in a fully extended state. For example, the door sensor 55 of the vegetable compartment door 251 outputs an open signal to the control unit 8 (see Figure 5) even when the vegetable compartment door 251 is only slightly open (the same applies to the remaining door sensors 5). Then, for example, if the duration of a predetermined door remaining open reaches a predetermined value, the buzzer 31g (see Figure 5) sounds. This prevents forgetting to close the door and suppresses the reduction in cooling capacity and increase in power consumption due to cold air leakage.

[0048] The imaging system 200 (see Figure 5) is equipped with a distance measuring sensor located near the imaging unit, and the removal processing unit 33 may determine the removal range for objects detected by the distance measuring sensor to be closer than the distance to the storage area. This ensures that the removal range for objects can be reliably determined.

[0049] The imaging system 200 may be equipped with a notification unit (buzzer 31g, camera LED 31f) that activates when the imaging unit detects that there is a removal area during imaging. This allows the user to be aware that the removal processing unit 33 is functioning during imaging.

[0050] The imaging system 200 includes a door opening / closing detection unit (for example, recognition by a Hall sensor or camera recognition by a marker) that detects the opening and closing of a door, and the image data captured while the door opening / closing detection unit detects that the door is open is processed by the removal processing unit 33. This allows for the rapid creation of an output image to be presented to the user.

[0051] (In the case of drawer containers) Figure 15 shows the results of a camera unit 3 of a refrigerator 100 according to an embodiment, as an example before image processing is performed. In the example in Figure 15, the vegetable compartment door 251 is open and the remaining doors are closed when the image is taken. As mentioned above, a fisheye lens is used as the lens 31a of the camera unit 3 (see Figure 5), so it is possible to capture images with a wide field of view. However, as shown in Figure 15, when no image processing is performed, straight edges appear curved in the captured image, and even for edges of the same length, the number of pixels in the captured image decreases the further away from the camera unit 3 they are.

[0052] Figure 16 shows the immovable region of the drawer container. Figure 16 is an example of an unfolded image after image processing, which is the result of a photograph taken by the camera unit 3. By performing predetermined image processing in the camera / communication control SoC 31k (see Figure 5), the image is converted into an unfolded image of the vegetable compartment 25 (inside the vegetable compartment container 252) as seen from above, as shown in Figure 16. In Figure 16, the vegetable compartment door 251 and the frame of the vegetable compartment container 252 are the immovable regions.

[0053] When such an unfolded image is displayed on the mobile terminal 64 (see Figure 5), the user can see at a glance what kind of food is stored in the vegetable compartment 25. Although not specifically shown in Figure 16, predetermined characters or images indicating that it is the "vegetable compartment" may be associated with the output image and displayed on the mobile terminal 64. This makes it easier for the user to recognize that it is an output image of the vegetable compartment 25. Incidentally, the control unit 8 can recognize which storage compartment the image was taken from because each door is equipped with a door sensor 5.

[0054] The imaging system 200 (see Figure 5) includes a drawer container with an open top (for example, a vegetable compartment container 252) and a drawer door (for example, a vegetable compartment door 251) that allows the drawer container to be freely pulled out. At least a portion of the area where the drawer container is located when the drawer door is fully open may be set as a stationary area. This has the effect of improving the positional accuracy when performing the image processing in process S60.

[0055] Furthermore, the imaging system 200 includes a drawer door opening / closing detection unit (for example, a door sensor 55) that detects the opening and closing of the drawer door, and at least a portion of the area where the drawer container is located when the drawer door is fully open may be set as a stationary area when the drawer door opening / closing detection unit detects that it is open. This has the effect of improving the positional accuracy when performing the image processing in process S60.

[0056] (Door storage compartment, door pocket) Figures 7 to 9 illustrate the processing of the removal processing unit 33 for the storage section of the refrigerator compartment 21. Figures 13 to 16 illustrate the processing of the removal processing unit 33 for the drawer containers of the lower freezer compartment 24 and the vegetable compartment 25. However, it is not limited to these. The processing of the removal processing unit 33 of this embodiment can also be applied to the door storage section (door pocket). Since the door storage section is a movable area, it is not appropriate to apply the processing of the removal processing unit 33 while it is moving. However, as shown in Figure 7, if the door storage section does not move between imaging frames, the processing of the removal processing unit 33 may be executed. This prevents human bodies such as faces and hands or other obstructions from appearing in the camera's field of view and obscuring the food, as shown in Figure 7. By creating an image with such temporarily obscuring obstructions removed, the amount of information that can be confirmed in a single image, including the door storage section, can be improved.

[0057] As described above, the imaging system according to this embodiment is applied to a storage facility that includes a box-shaped body (housing 1) having a storage area and an open front, and an openable and closable door (e.g., refrigerator doors 211, 212) arranged to open and close the opening of the box-shaped body. The system comprises an imaging unit (e.g., image sensor 31d) capable of imaging so that the storage area is included in the field of view when the door is open, and a removal processing unit 33 that determines a removal range from within the storage area captured in the first image data captured by the imaging unit, and overwrites the removal range of the first image data with a second image data captured in the vicinity before the time the first image data was captured. This makes it possible to create an image from which an obstruction that temporarily hides food has been removed, thereby improving the amount of information that can be confirmed in a single image.

[0058] In this embodiment, a case in which a fisheye lens is used as the lens 31a of the camera unit 3 (see Figure 7) has been described, but the embodiment is not limited to this. For example, a general-purpose lens other than a fisheye lens may be used as the lens 31a of the camera unit 3.

[0059] Furthermore, the refrigerator 100 etc. described in each embodiment are merely examples, and the embodiments can be applied to other types of refrigerators. For example, the embodiments can be applied to refrigerators equipped with a single-opening refrigerator door (not shown) or portable refrigerators (not shown).

[0060] Furthermore, the "storage facilities" to which each embodiment can be applied are not limited to refrigerators. In this case, the enclosure of the "storage facility" may have a configuration that does not particularly distinguish between an outer box and an inner box. Also, each embodiment can be applied to freezers and storage rooms as well as "storage facilities".

[0061] Furthermore, the "storage facilities" to which each embodiment can be applied can also be used for storage facilities that have functions other than refrigeration and freezing (such as warming facilities, drying facilities, temperature-controlled storage facilities, and humidity-controlled storage facilities).

[0062] Furthermore, the programs executed by the control unit 8 (see Figure 5) and the server 63 (see Figure 5) can be provided via a communication line, or they can be written to a recording medium such as a CD-ROM and distributed.

[0063] Furthermore, each embodiment is described in detail to clearly illustrate the present invention and is not necessarily limited to having all the configurations described. In addition, it is possible to add, delete, or replace some of the configurations in the embodiments with other configurations. Furthermore, the mechanisms and configurations described above are those deemed necessary for explanatory purposes and do not necessarily represent all of the mechanisms and configurations shown in the actual product. [Explanation of Symbols]

[0064] 1 Housing (box body) 3 Camera Unit 4 Wireless LAN Unit 5, 52, 53, 54, 55 Door Sensor 6. Control Panel 7. Shoot button 8 Control Unit 21. Refrigerated room (storage area, first storage area) 22 Ice-making room (storage area, second storage area) 23. Upper freezer compartment (storage area, second storage area) 24. Lower freezer compartment (storage area, second storage area) 25. Vegetable compartment (storage area, second storage area) 31a Lens (imaging unit) 31d Image sensor (imaging unit) 31k Camera / Communication Control SoC 31f Camera LED (Lighting Unit) 31g buzzer 33 Removal Processing Unit 52, 53, 54, 55 Door Sensor 61 Routers 62 Networks 63 servers 64 Mobile devices 211,212 Refrigerator door (door, French door) 211c, 212c Door pocket (storage area, movable area) 221 Ice maker door (door, pull-out door, retractable door) 231 Upper freezer door (door, pull-out door, drawer door) 241 Lower freezer door (door, pull-out door, drawer door) 251 Vegetable compartment door (door, drawer-type door, pull-out door) 252 Vegetable compartment container 100 Refrigerator (storage) 200 Imaging Systems

Claims

1. Applicable to a storage facility comprising a box-shaped body having a storage area and an open front, and an openable and closable door arranged to open and close the opening of the box-shaped body, the imaging unit is capable of imaging such that the storage area is included in the field of view when the door is open, A removal processing unit determines a removal range from within the storage area captured in the first image data captured by the imaging unit, and overwrites the removal range of the first image data with second image data captured in the vicinity of the time before the first image data was captured. An imaging system equipped with the following features.

2. The removal processing unit determines the removal range as the region with large pixel changes between two image data captured at two close points in time within substantially the same field of view. The imaging system according to feature 1.

3. The storage area includes a stationary area that does not move even when the door is opened or closed, and a movable area that moves together with the door. At least a portion of the immovable region is set as a processing area where the removal processing unit performs processing. At least a portion of the area into which the movable area can enter is set as a non-processing area where processing by the removal processing unit is not performed. The imaging system according to feature 2.

4. A drawer container with an opening at the top, The drawer container is equipped with a drawer door that allows the drawer to be pulled out freely, At least a portion of the area where the drawer container is located when the drawer door is fully open is set as the immovable area. The imaging system according to claim 3.

5. It includes a drawer door opening / closing detection unit that detects the opening and closing of the aforementioned drawer door, When the drawer door is fully open, at least a portion of the area where the drawer container is located is set as the immovable area when the drawer door opening / closing detection unit detects that it is open. The imaging system according to feature 4.

6. The imaging unit is capable of detecting the temperature of an object. The removal processing unit determines objects within the human body temperature range as the removal range. The imaging system according to feature 1.

7. It is equipped with a distance measuring sensor located near the imaging unit, The removal processing unit determines the removal range to include objects detected by the distance measuring sensor as being closer than the distance to the storage area. The imaging system according to feature 1.

8. The imaging unit includes a notification unit that activates when it detects that there is a removal area during imaging. The imaging system according to feature 1.

9. It includes a door opening / closing detection unit that detects the opening and closing of the aforementioned door, The image data captured while the door opening / closing detection unit detects that the door is open is processed by the removal processing unit. The imaging system according to feature 1.

Images