Imaging system and imaging device

Abstract

To provide an imaging system that can appropriately output image data in a refrigerator (storage chamber).SOLUTION: An imaging system comprises: a refrigerator comprising a storage chamber comprising an opening in a front part, and a rotary type door for opening / closing the opening, and provided with a door housing part; and an imaging device arranged outside the storage chamber, and comprising a lens and an illumination part located in front of an edge surface of the opening. A light blocking wall for blocking direct light is provided between the illumination part and the lens. The imaging device is arranged in such a manner that the door housing part can be included in a portion of a peripheral edge of an imaging range. The illumination part is arranged closer to the side of the door housing part than the lens.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to an imaging system and an imaging device. [Background technology]

[0002] A refrigerator has been proposed that uses a camera to capture images of the interior of the refrigerator for the purpose of inventory management inside the refrigerator. According to the refrigerator described in Patent Document 1, the refrigerator "includes a plurality of storage compartments partitioned into at least one capture range, a plurality of interior cameras that capture at least one of the capture ranges, or at least one interior camera that captures the plurality of capture ranges, and a control device that stores images of the plurality of capture ranges in a storage device and communicates with an operation display device that displays the images." The operation display device also describes that "when an operation is performed to instruct the movement of the image inside the refrigerator compartment by touching and moving a finger to the left or right (a so-called swipe operation or flick operation performed on a touch panel of a smartphone or tablet terminal), the control device displays an image of the door pocket side of the door that is located in the opposite direction to the refrigerator compartment in an observation state in which the user observes the interior of the refrigerator compartment from the front side of the refrigerator, in conjunction with the image inside the refrigerator compartment." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6296908 Summary of the Invention [Problem to be solved by the invention]

[0004] The refrigerator described in Patent Document 1 requires a user to perform a swipe or flick operation on the operation display device in order to simultaneously view the interior of the refrigerator and the door pocket. [Means for solving the problem]

[0005] The imaging system of the present invention, which has been made in consideration of the above circumstances, comprises a refrigerator including a storage compartment with an opening at the front and a revolving door that opens and closes the opening and is provided with a door storage section, and an imaging device that is arranged outside the storage compartment and has a lens and an illumination section located forward of an edge surface of the opening, a light-shielding wall that blocks direct light is provided between the illumination section and the lens, the imaging device is arranged so that the door storage section can be included in a part of the periphery of an imaging range, and the illumination section is arranged on the door storage section side of the lens. Other aspects of the present invention will be described in the embodiments described later. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a front view showing a refrigerator according to a first embodiment. [Figure 2A] FIG. 1 is a side view showing a refrigerator according to a first embodiment. [Figure 2B] FIG. 2 is a side view showing a case where the imaging device according to the first embodiment is stored. [Diagram 3] FIG. 1 is a front view showing a state in which the door of a refrigerator according to a first embodiment is open. [Figure 4] FIG. 1 is a top view showing a state in which the door of a refrigerator according to a first embodiment is open. [Figure 5A] 1 is a front view showing an imaging device according to a first embodiment. [Figure 5B] 1 is a side view showing an imaging device according to a first embodiment. [Figure 5C] 1 is a side cross-sectional view showing an imaging device according to a first embodiment. [Figure 6] 1 is a perspective view showing an imaging device according to a first embodiment. [Figure 7] FIG. 1 is a configuration diagram showing an imaging system for a refrigerator according to a first embodiment. [Figure 8] 5 is a flowchart showing a process when manual shooting settings are made according to the first embodiment. [Figure 9] 5 is a flowchart showing a process when automatic shooting setting is performed according to the first embodiment. [Figure 10]4 is a flowchart showing image processing according to the first embodiment. [Figure 11] 13 is an image showing the interior of the fridge before image processing. [Figure 12] 11 is an image showing the cutting process of each storage section. [Figure 13] 13 is an image showing the distortion correction process for each storage section. [Figure 14] 13 is an image showing a rotation process of each storage section. [Figure 15] 11 is an image showing keystone correction processing for each image. [Figure 16] 11 is an image showing a process of combining images. [Figure 17] FIG. 11 is a side view showing the state in which the drawer part of the refrigerator according to the second embodiment is pulled out. [Figure 18] 13 is an image showing a cutout process of a drawer section of a refrigerator according to a second embodiment. [Figure 19] 13 is an image showing enlargement processing of a drawer section of a refrigerator according to a second embodiment. [Figure 20] 13 is an image showing a synthesis process of each image including a pull-out portion according to the second embodiment. [Figure 21] 13 is another example of an image showing a synthesis process of each image including a drawer portion according to the second embodiment. [Figure 22A] FIG. 2 is an explanatory diagram showing a normal state of the imaging device. [Figure 22B] FIG. 11 is an explanatory diagram showing a state in which the imaging device is in the middle of being stored. [Figure 22C] FIG. 2 is an explanatory diagram showing a state in which the imaging device is stored. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The following describes in detail the mode (embodiment) for carrying out the present invention with reference to the drawings as appropriate. The order of the control steps of the operation flows and the like disclosed in the embodiment can be interchanged or executed simultaneously as long as there is no mutual contradiction. <<First embodiment>> FIG. 1 is a front view showing the refrigerator 100 according to the first embodiment. FIG. 2A is a side view showing the refrigerator 100 according to the first embodiment. FIG. 2B is a side view showing the refrigerator 100 when the imaging device 30 according to the first embodiment is stored. FIG. 3 is a front view showing the refrigerator 100 according to the first embodiment with the door open. FIG. 4 is a top view showing the refrigerator 100 according to the first embodiment with the door open. In the following description, a six-door refrigerator 100 will be taken as an example, but is not limited to six doors.

[0008] As shown in FIG. 1, the insulated box (box 10) of the refrigerator 100 has storage compartments in the order of refrigerator compartment 1, ice-making compartment 2, upper freezer compartment 3, vegetable compartment 4, and lower freezer compartment 5, which are arranged on the left and right sides in this order from above. The refrigerator 100 has doors that open and close the openings of each storage compartment. These doors are rotating refrigerator compartment doors 1a and 1b divided into left and right for opening and closing the opening of the refrigerator compartment 1, and pull-out ice-making compartment door 2a, upper freezer compartment door 3a, vegetable compartment door 4a, and lower freezer compartment door 5a for opening and closing the openings of the ice-making compartment 2, upper freezer compartment 3, vegetable compartment 4, and lower freezer compartment 5, respectively. The positions of the vegetable compartment 4 and the lower freezer compartment 5 may be changed in the vertical arrangement. The ice-making compartment 2, upper freezer compartment 3, and lower freezer compartment 5 are collectively referred to as the freezer compartment 7.

[0009] Door hinges (reference numerals omitted) are provided at the top and bottom of the refrigerator compartment 1 to fix the refrigerator compartment doors 1a, 1b to the refrigerator 100. As shown in Fig. 3 and Fig. 4, when the refrigerator compartment doors 1a, 1b are opened and closed, door storage sections 11a (door storage section a), 11b (door storage section b) are provided.

[0010] The refrigerator compartment 1 is a refrigerator storage compartment whose interior temperature is kept within the refrigeration temperature range (0°C or higher), for example, at about 4°C on average. The freezer compartment 7 is a refrigerator storage compartment whose interior temperature is kept within the freezing temperature range (less than 0°C), for example, at about -18°C on average. The vegetable compartment 4 is a refrigerator storage compartment whose interior temperature is kept within the refrigeration temperature range, for example, at about 6°C on average, which prevents food from drying out.

[0011] In the refrigerator 100, the imaging device 30 for photographing the interior is disposed in the approximate center of the upper outer part of the storage compartment. That is, it is provided in the approximate center above the box body (insulated box body) of the refrigerator 100. When the box body door is closed, the lens 34 (see Figs. 5A and 5B) of the imaging device 30 is located forward of the opening edge surface 10a of the storage compartment, preferably forward of the refrigerator compartment doors 1a, 1b (doors). That is, the imaging device 30 is provided outside the storage compartment, and can capture an image of the interior of the refrigerator including the door storage sections 11a, 11b from a bird's-eye view.

[0012] Moreover, the imaging device 30 has a mechanism for sliding back and forth (a mechanism for sliding in a direction perpendicular to the opening of the storage compartment), and can move back and forth across the front door of the refrigerator 100. For example, as shown in FIG. 2B, since it can move behind the front door or behind the edge of the opening of the storage compartment, it does not affect the depth dimension during packaging. Furthermore, the number of refrigerators that can be loaded onto a transport truck is not reduced, and even after installation, the imaging device 30 can be stored and the imaging range can be concealed when not in use. By concealing it, the configuration can be designed with consideration given to users who do not like the imaging range of the camera always being within their living space.

[0013] The refrigerator compartment door 1b is provided with an image capture button 39 (an image capture command receiving unit that receives an image capture command from a user) that is used when manually capturing an image with the imaging device 30. The image capture button 39 may be provided on the refrigerator compartment door 1a or in another area of ​​the box or door.

[0014] Fig. 5A is a front view showing the imaging device 30 according to the first embodiment. Fig. 5B is a side view showing the imaging device 30 according to the first embodiment. Fig. 5C is a side cross-sectional view (cross-sectional view taken along line AA in Fig. 5A) showing the imaging device 30 according to the first embodiment. Fig. 6 is a perspective view showing the imaging device 30 according to the first embodiment.

[0015] The imaging device 30 is composed of a case 31 and a cover 32, and has a camera 33 (imaging unit) and a fisheye lens 34 in front of the case 31. The lens 34 is surrounded by a light-shielding wall 35, as shown in Fig. 6. A plurality of illumination units 38 are provided near the light-shielding wall 35.

[0016] As shown in Fig. 5C, a control SoC 41 and a wireless LAN unit 42 are stored inside the case 31. The control SoC 41 (system on a chip) is an integrated circuit that is designed to integrate functions such as a processor core and other functions of a general microcontroller, as well as functions for application purposes, on a single chip of the integrated circuit, and to function in cooperation with each other as a system. Note that the control SoC 41 is an example of an integrated circuit, and may be replaced by other integrated circuits, etc.

[0017] An imaging device 30 provided outside the storage compartment captures images of the door storage compartments 11a and 11b and the interior of the storage compartment from a bird's-eye view.

[0018] The features of the imaging device 30 will now be described further. (1) The camera 33 used in the imaging device 30 can capture images at a wide angle of 180 degrees or more, and has a fisheye lens, so that the entire interior of the fridge can be viewed from above when the door is opened. (2) The camera lens 34 of the imaging device 30 is positioned in front of the front surface of the refrigerator door, so that the state inside the refrigerator can be photographed even when only the vegetable compartment 4 and the drawer section 6 of the lower freezer compartment 5 are open.

[0019] (3) The imaging device 30 is equipped with an illumination unit 38, and by using the exterior illumination unit 38 that can be adjusted to an illumination suitable for shooting, images with high visibility can be obtained. The illumination provided inside the refrigerator is relatively high in illumination, so when an image is shot with a camera, the stored items are strongly reflected, and the captured image is likely to be overexposed. For this reason, in this embodiment, when shooting, the interior illumination is turned off and the exterior illumination unit 38 is turned on. In addition, the illumination of the illumination unit 38 is directional so as to illuminate the entire interior of the refrigerator, the door storage units 11a and 11b when the door is opened, and the second storage compartment (for example, the vegetable compartment 4 and the lower freezer compartment 5). (4) The imaging device 30 has the light-shielding wall 35, which prevents direct light from the lighting unit 38 from entering the shooting range, making it possible to obtain an image with high visibility and reduced reflections and overexposure. (5) By providing the light-shielding wall 35 between the lighting unit 38 of the imaging device 30 and the lens 34 to block direct light, the lighting can be installed as close as possible to the lens 34, making it possible to fit the device into a compact external shape and also preventing the lens 34 from being scratched.

[0020] (6) Imaging device 30 has a mechanism for sliding back and forth and can move back and forth across the front door of refrigerator 100, so it does not affect the depth dimension during packaging. In addition, the number of refrigerators that can be loaded onto a transport truck is not reduced, and even after installation, imaging device 30 can be stored to hide the imaging range when not in use. (7) The imaging device 30 is equipped with a communication unit (wireless LAN unit 42), and image data transmitted from the communication unit is supplied to a server 54 (see FIG. 7) via, for example, a wireless router in the home (e.g., a home router 53 (see FIG. 7)). Details will be described later. (8) The imaging device 30 has a communication circuit that is connected to the refrigerator body for power supply and image capture command transmission. The refrigerator body receives power from a commercial power source or the like. Power for the imaging device 30 can be received from the commercial power source via the refrigerator body. The imaging device 30 may be driven by a rechargeable battery.

[0021] 7 is a configuration diagram showing an imaging system of refrigerator 100 according to the first embodiment. Door sensors (not shown) are also provided on the door side to detect the open / closed states of refrigerator doors 1a and 1b, ice-making door 2a, upper freezer door 3a, vegetable door 4a, and lower freezer door 5a.

[0022] A main microcomputer 52 (processing device, control device, control section) equipped with a CPU (Central Processing Unit), which is part of the control device, memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory), an interface circuit, etc. is disposed on the top of the refrigerator 100. The main microcomputer 52 controls an imaging system that captures images of the interior of the refrigerator based on information from the door sensor or the image capture button 39. The main microcomputer 52 also supplies power to the imaging device 30.

[0023] When automatic shooting is set, the main microcomputer 52 sends a shooting command to the control SoC 41 when it detects a door opening angle or more that is a predetermined angle based on information from the door sensor, such as the door opening angle. The control SoC 41 issues a shooting command to the camera 33 and acquires unprocessed image data from the camera 33. The control SoC 41 performs image processing on the unprocessed image data, for example, as shown in FIG. 10. On the other hand, when manual shooting is set, the main microcomputer 52 sends a shooting command to the control SoC 41 when it detects that the shooting button 39 has been operated. Subsequent processing is the same as when automatic shooting is set.

[0024] As described above, the imaging device 30 is provided with the wireless LAN unit 42 that can be connected to the home router 53. By providing this wireless LAN unit 42, the interior image data processed by the control SoC 41 can be transmitted to the server 54 via the home router 53. The server 54 transmits the interior image data in response to a request to display an interior image from a mobile device such as a smartphone 55 or a personal computer owned by the user. This enables the user to grasp the status of the interior of the fridge.

[0025] Fig. 8 is a flow chart showing the process when manual photography is set according to the first embodiment. The description will be made with reference to Fig. 7 as appropriate. When the setting from the setting device (not shown) is the camera photography setting (camera setting ON) (S31), the main microcomputer 52 judges whether the door is open (door open?) using the door sensor (S32), and if the door is open (S32, Yes), proceeds to S33. On the other hand, if the door is not open (S32, No), the main microcomputer 52 returns to S32.

[0026] In S33, the main microcomputer 52 determines whether the shooting button 39 has been operated (pressed), and if the shooting button 39 has been pressed (S33, Yes), proceeds to S34, and if the shooting button 39 has not been pressed (S33, No), returns to S32.

[0027] In S34, the main microcomputer 52 commands the control SoC 41 to take an image, and turns off the interior lights and turns on the exterior lights (lighting unit 38) (S35). The control SoC 41 commands the camera 33 to take an image (S36). The main microcomputer 52 then turns on the interior lights and turns off the exterior lights (lighting unit 38) (S37).

[0028] The camera 33 transmits the unprocessed image data to the control SoC 41 (S38), and the control SoC 41 performs an interior image correction process (image processing) (S39). Details of the image processing will be described later with reference to FIG.

[0029] The control SoC 41 performs image output processing (image update) on the processed image data via the wireless LAN unit 42 and the home router 53 to the server 54 (S40). When the image is updated, the server 54 notifies the user's mobile device or the like (S41). The above is the process when manual shooting is set.

[0030] Fig. 9 is a flow chart showing the process when automatic photography is set according to the first embodiment. The description will be made with reference to Fig. 7 as appropriate. Note that the description of the same process as Fig. 8 will be omitted. When the setting from the setting device (not shown) is the camera automatic photography setting (camera automatic setting ON) (S31A), the main microcomputer 52 judges whether the door is open (door open?) using the door sensor (S32), and if the door is open (S32, Yes), proceeds to S33A. On the other hand, if the door is not open (S32, No), the main microcomputer 52 returns to S32.

[0031] In S33A, the main microcomputer 52 determines whether the door opening angle is equal to or greater than a predetermined angle (e.g., door angle 85 degrees or greater), and if the door opening angle is equal to or greater than the predetermined angle (S33A, Yes), the process proceeds to S34, and if the door opening angle is less than the predetermined angle (S33A, No), the process returns to S32. S34 to S41 are the same as in Fig. 13. Note that in S33A, in the case of a double-door refrigerator compartment 1, if the opening angle of at least one of the refrigerator compartment doors 1a, 1b is equal to the predetermined angle (S33A, Yes), the process proceeds to S34. The above is the process when automatic shooting is set.

[0032] The features of the imaging system of the refrigerator 100 will now be described further. (1A) The refrigerator has a detection unit (e.g., main microcomputer 52) that detects the opening angle of the door, and can automatically take a photograph when the detection unit determines that the angle is equal to or greater than a predetermined angle (e.g., 85 degrees). (2A) When photographing, the interior lighting is turned off and the lighting of the imaging device 30 is turned on. This not only adjusts the illuminance to an appropriate level for photographing, but also has the effect of informing the user when it is time to photograph. (3A) The refrigerator 100 has a detection unit (for example, installed in the drawer section 6) that detects when the drawer distance reaches a certain distance or more. The detection unit detects the operation of the drawer and automatically takes a photograph, thereby keeping the imaging range of the vegetable compartment and freezer compartment roughly constant.

[0033] (4A) The refrigerator door has a button for photographing (e.g., photographing button 39). When the user presses the button for photographing, the interior lighting is turned off and the lighting of the imaging device 30 is turned on, after which a photograph is taken. This makes it easy to recognize the timing at which the photograph was taken. (5A) The captured image undergoes image correction processing, and the distorted image captured by the wide-angle camera is corrected to provide an image that is close to the user's line of sight. (6A) An application service is provided that allows a user to check the images via a mobile phone or the like, and a detection unit is provided on the door or the drawer to automatically determine which storage room the captured image belongs to.

[0034] As described above, the refrigerator 100 of this embodiment includes a storage compartment (e.g., refrigerator compartment 1) with an opening at the front, a door (e.g., refrigerator compartment door 1a, 1b) with a rotatable storage section (e.g., door storage section 11a, 11b) provided at the front of the storage compartment, a door opening angle detection means for detecting the opening angle of the door, a lighting device for illuminating the storage compartment when the door opening detection means detects that the door is open, an image capture device (e.g., image capture device 30) provided at approximately the center of the upper outer part of the storage compartment, the image capture device having a wide-angle fisheye lens (e.g., lens 34), an illumination device (e.g., illumination section 38) that lights up when capturing an image, and a communication means (e.g., wireless LAN unit 42) for transmitting captured image data. The refrigerator 100 includes a control device (e.g., main microcomputer 52) for issuing an image capture command to the image capture device when a predetermined opening angle is detected by the door opening angle detection means.

[0035] With the door open, an image captured by the imaging device can include not only the storage compartment but the entire door storage section in the capture range in one captured image due to the action of the wide-angle fisheye lens, as shown in Fig. 11. Furthermore, the vegetable compartment (e.g., vegetable compartment 4) and freezer compartment (e.g., lower freezer compartment 5) of the drawer storage section can also be included in this capture range, eliminating the need to provide multiple imaging devices for each storage compartment as in Patent Document 1, and eliminating the need for complex and expensive drive devices and drive controls to change the capture range.

[0036] Although the image captured by the imaging device is sufficient to grasp the state of the stored items inside the refrigerator, the image is curved due to the action of the fisheye lens, and objects closer to the imaging device are more distorted. Also, since the image is taken from above the refrigerator, it is positioned differently from the view from the front of the refrigerator that users normally observe, so there is room for improvement in terms of visibility.

[0037] According to this embodiment, by performing the following image processing (S39), it is possible to provide an interior image in a state close to the positional relationship of the field of view from the front when a user normally observes the refrigerator, by converting a distorted and curved image captured by the imaging device 30 having a fisheye lens with a wide angle of view.

[0038] FIG. 10 is a flow chart showing image processing (S39) according to the first embodiment. Images of each process will be described with reference to FIG. 11 to FIG. 16. FIG. 11 is an image showing the interior of the storage unit (storage unit c as inside the refrigerator compartment 1, door storage unit a as door storage unit 11a, and door storage unit b as door storage unit 11b) before image processing. FIG. 12 is an image showing the cropping process of each storage unit. FIG. 13 is an image showing the distortion correction process of each storage unit. FIG. 14 is an image showing the rotation process of each storage unit. FIG. 15 is an image showing the keystone correction process of each image. FIG. 16 is an image showing the synthesis process of each image.

[0039] The main microcomputer 52 receives the captured image (see FIG. 11) (S391) and cuts out the image into a sector shape (cutout process) so as to encompass the range of each storage compartment (S392). Specifically, the main microcomputer 52 cuts out an image of the interior of the refrigerator compartment 1 and images of each of the door storage compartments 11a and 11b, for example, cuts out the image into a sector shape (see FIG. 12).

[0040] Next, the main microcomputer 52 performs distortion correction processing (distortion correction process) on the images of each storage section curved due to the action of the fisheye lens (S393). Specifically, the main microcomputer 52 performs distortion correction processing on the internal image of the refrigerator compartment 1 and the curved images of the door storage sections 11a and 11b, as shown in Fig. 13. This distortion correction processing can use various known correction algorithms, but for example, the distortion can be corrected by executing a calibration process in which parameters are set in advance (for example, before the refrigerator is shipped).

[0041] Next, the main microcomputer 52 performs an image rotation process to rotate the image of each storage section that has been subjected to the distortion correction process so that the image is viewed from the front, which is the observation state of the user (S394). Specifically, the main microcomputer 52 arranges the door storage section 11a on the left side and the door storage section 11b on the right side so that the image is viewed from the front, which is the observation state of the user, as shown in FIG.

[0042] Next, the main microcomputer 52 performs distortion correction processing (keystone correction processing) to convert the image that is distorted trapezoidally in the depth direction into a rectangle (S395). Specifically, the main microcomputer 52 performs keystone correction processing on the image inside the refrigerator compartment 1 and the images of the door storage sections 11a and 11b, as shown in Fig. 15. For example, by executing a perspective projection transformation and its complementary processing such as the nearest neighbor method or the Lanczos method, the image of the storage section that is roughly trapezoidal or sector-shaped can be converted into a roughly rectangular shape.

[0043] Next, the main microcomputer 52 performs image synthesis processing on the images created in S395 to create one image (S396). This image is displayed on an application of the smartphone 55 (see FIG. 7). Note that the embodiment shows only one example, and each process may be arbitrarily changed.

[0044] The method for processing image data in a refrigerator according to the first embodiment is a method for processing image data in a refrigerator that processes image data captured by an imaging device equipped with a wide-angle lens that includes multiple food storage areas in its imaging range, and is characterized by including and executing a cutout process (S392) for cutting out multiple parts from the image data captured by the imaging device 30, a synthesis process (S396) for synthesizing the images cut out by the cutout process, and an output process (S40) for outputting the image synthesized by the synthesis process as one image data. The cutout process can remove at least a part of the area that is not a food storage area from the input captured image (FIG. 11), for example an area in which no food is captured and only the wall surface of the refrigerator or the floor surface outside the refrigerator is captured. By synthesizing the images of the cut out food storage areas, the ratio of the food storage area displayed in the synthesized image (FIG. 16) can be increased relative to the entire display screen of the smartphone 55 on which the images are displayed. In this embodiment, the cutout process is executed on one image captured by one imaging device 30, but two or more imaging devices may be prepared to capture two or more images, and the portions including the food storage areas included in each image data may be cut out.

[0045] In this way, the present embodiment is characterized in that the ratio of the area occupied by the food storage area to the entire image data is larger in the image after the synthesis process than in the image data taken with a wide-angle lens. For example, comparing Figure 11 with Figure 16, the area in which the food is captured is larger, making it easier for the user to see.

[0046] In this embodiment, the distance between the multiple parts cut out in the cutout process is shorter after the synthesis process. For example, comparing FIG. 12 with FIG. 16, the distance between the door storage section a, the storage section c, and the door storage section b shown in FIG. 16 is smaller than the distance between the door storage section a, the storage section c, and the door storage section b shown in FIG. 12, making it easier for the user to see. In addition, since multiple storage areas that are separated from each other in real distance, such as the door storage section and the storage section, and the two door storage sections, are output as one image, it is possible to make the whole image easier to see without having the user perform a swipe or flick operation to switch the displayed image, as in Patent Document 1. In particular, the two door storage sections are opened by rotating in opposite directions around door hinges arranged at different positions, so that the actual distance between them is relatively large in the refrigerator.

[0047] In this embodiment, the refrigerator compartment 1 has been described as a storage compartment with double doors, but the present invention can also be applied to a storage compartment with a single door.

[0048] <<Second embodiment>> In the first embodiment, the image synthesis process of the refrigerator compartment 1 (first storage compartment) has been described, but is not limited thereto. In the second embodiment, the image synthesis process including the refrigerator compartment 1 (first storage compartment) and the vegetable compartment 4 etc. (second storage compartment) will be described.

[0049] Fig. 17 is a side view showing the state in which the drawer section 6 of the refrigerator 100 according to the second embodiment is pulled out. Fig. 18 is an image showing a cropping process of the drawer section 6 of the refrigerator according to the second embodiment. Fig. 19 is an image showing a zooming process of the drawer section 6 of the refrigerator according to the second embodiment. The imaging device 30 for photographing the interior can capture an image of the inside of the drawer section 6. Therefore, the storage state can be ascertained from the image of the drawer section 6.

[0050] FIG. 20 is an image showing a synthesis process of each image including the drawer section 6 according to the second embodiment. In FIG. 20, compared to FIG. 16, an image of the drawer section 6 (second storage compartment d) is also synthesized. This allows the user to grasp the storage status including the refrigerator compartment 1 (first storage compartment) and the vegetable compartment 4 etc. (second storage compartment) at a glance. Note that FIG. 16 is an image synthesized from the captured images shown in FIG. 12.

[0051] That is, refrigerator 100 is characterized by having at least a first storage compartment that opens like double doors and a second storage compartment that opens in the front-to-rear direction, and the cutting process is performed on the first storage compartment and the second storage compartment. Also, refrigerator 100 may have at least a first storage compartment that opens like double doors and a second storage compartment that opens in the front-to-rear direction, the first storage compartment may have a door storage section, and the cutting process may be performed on the interior of the first storage compartment, the door storage section, and the second storage compartment. The cutting process can be performed on the second storage compartment when it is detected that the second storage compartment is open.

[0052] Fig. 21 is another example of an image showing a synthesis process of each image including the drawer section 6 according to the second embodiment. Since the display screen of the smartphone 55 on which the image is expected to be displayed is often rectangular, the positions of the door storage sections a and b can be moved from the side of the storage cabinet c so that the synthesized image becomes rectangular as a whole, as shown in Fig. 21. In the example of Fig. 21, the door storage sections a and b are moved to the opposite side of the second storage cabinet d with the storage cabinet c at the center.

[0053] That is, the cutting process is performed with at least one door storage section, the interior of the first storage compartment, and the second storage compartment as food storage areas. In the combining process, the at least one door storage section and the second storage compartment are arranged to be located on opposite sides of the interior of the first storage compartment.

[0054] <<Modifications of the Imaging Device>> In the first embodiment, the imaging device 30 has been described as having a slide mechanism, but the present invention is not limited to this. In a modified example, a folding mechanism will be described.

[0055] Fig. 22A is an explanatory diagram showing a normal state of the imaging device 30. Fig. 22B is an explanatory diagram showing a state of the imaging device 30 in the middle of being stored. Fig. 22C is an explanatory diagram showing a stored state of the imaging device 30.

[0056] The refrigerator 100 (see FIG. 1) includes a box body forming a storage compartment with an opening at the front, an imaging device 30 provided outside the storage compartment, and a support part 45 (hinge part) that supports the imaging device 30 movably at least in a direction perpendicular to the opening.

[0057] As shown in FIGS. 22A to 22C , the tip portion having camera 33 of imaging device 30 is configured to be able to be folded in half from support part 45, and since it does not affect the depth dimension at the time of packaging, the number of refrigerators that can be loaded onto a transport truck is not reduced. Even after installation, when imaging device 30 is not in use, it can be stored to hide the imaging range.

[0058] The imaging device 30 can also revolve (move in a circular path around the hinge; revolve around an axis) in the direction of looking into the interior of the fridge. Since the box is present in the circular path, the angle at which it can revolve is less than 360° when observed from a direction perpendicular to the revolution plane. In this case, a drive unit capable of applying a force in the revolution direction to the imaging device 30 may be attached, and the imaging device 30 may be controlled to have an angle at which the interior of the fridge or other compartment can be viewed during imaging. For example, the imaging device 30 can be positioned within the movement path of the door during imaging. This allows the interior of the fridge to be imaged more effectively, and since the imaging device 30 can be returned to outside the movement path of the door after imaging is completed, or revolved to the rear of the edge surface of the front opening of the door or storage compartment for storage, it is characterized in that the arm connecting the support part 45 (hinge part) and the camera can be easily lengthened.

[0059] The present invention is not limited to the first and second embodiments described above, and includes various modified examples. For example, the above-mentioned embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described above. In addition, it is possible to add, delete, or replace some of the configurations of the above-mentioned embodiments with other configurations. For example, the photographing button 39 in FIG. 3 may be provided on the refrigerator compartment door 1a or on the top surface of the drawer section 6. In addition, a plurality of imaging devices used for photographing may be prepared, and image data of the food storage area included in each photographed image may be used to synthesize into one image.

[0060] According to the refrigerator 100 of this embodiment, curved images captured by an imaging device having a fisheye lens (wide-angle lens) installed outside the storage compartment are distortion-corrected, cropped, and synthesized to output a single image, thereby providing an image of the interior of the refrigerator that is close to the state observed by the user while minimizing image switching operations on the display operation device.

[0061] An imaging device having a wide-angle lens such as a fisheye lens installed outside the storage compartment captures an image of the interior of the storage compartment including the door storage compartment from above, corrects distortion of the curved image, cuts out and processes each storage compartment, synthesizes the processed images, and processes these images to provide a single image of the situation inside the refrigerator close to the user's line of sight, making it easy to recognize the situation inside the refrigerator. In addition, since there is no need to install multiple imaging devices or imaging devices with complex variable control, it is advantageous in terms of cost, and it can solve problems such as lens scratches and obstruction of the shooting range due to stored items being close to the lens. Users can check the latest situation inside the refrigerator from images even when they are out, which can lead to a reduction in unnecessary shopping and food waste. [Explanation of symbols]

[0062] 1 Refrigerator compartment (first storage compartment) 1a,1b Refrigerator door 2 Ice making room (second storage room) 2a Ice maker door 3 Upper freezer compartment (second storage compartment) 3a Upper freezer door 4 Vegetable compartment (second storage compartment) 4a Vegetable compartment door 5 Lower freezer compartment (second storage compartment) 5a Lower freezer door 6 Drawer section 10 box body 10a Opening edge surface 11a, 11b Door storage section 30 Imaging device 31 Case part 32 Lid 33 Camera (imaging unit) 34 Lens (wide-angle lens) 35 Blackout Wall 38 Lighting Department 39 Shooting button (shooting command reception section) 41 Control SoC (output section) 42 Wireless LAN unit 45 Support part (hinge part) 51 Door sensor 52 Main microcomputer (processing unit) 53 Home Router 54 Server 55 Smartphone 100 Refrigerator

Claims

1. A storage chamber having an opening at the front; a revolving door that opens and closes the opening and has a door storage section; an imaging device disposed outside the storage chamber and having a lens and an illumination unit located forward of an edge surface of the opening; a light shielding wall for blocking direct light is provided between the illumination unit and the lens; The imaging device is arranged so as to be able to include the door storage section in a part of a periphery of an imaging range, The illumination unit is disposed on the door storage unit side of the imaging system relative to the lens.

2. A storage chamber having an opening at the front; a revolving door that opens and closes the opening and has a door storage section; an imaging device disposed outside the storage chamber and having a lens and an illumination unit located forward of an edge surface of the opening; a light shielding wall for blocking direct light is provided between the illumination unit and the lens; the imaging device is disposed so as to be able to include the interior of the storage chamber in a part of a periphery of an imaging range; The illumination unit is disposed on the inner side of the storage chamber relative to the lens.

3. A storage chamber having an opening at the front; a revolving door that opens and closes the opening and has a door storage section, The imaging device is disposed outside the storage chamber and has a lens and an illumination unit located forward of an edge surface of the opening, a light shielding wall for blocking direct light is provided between the illumination unit and the lens; The door storage section as a food storage area is arranged so as to be included in a part of the periphery of the imaging range, The illumination unit is an imaging device disposed on the door storage unit side of the lens.

4. A storage chamber having an opening at the front; a revolving door that opens and closes the opening and has a door storage section, The imaging device is disposed outside the storage chamber and has a lens and an illumination unit located forward of an edge surface of the opening, a light shielding wall for blocking direct light is provided between the illumination unit and the lens; The imaging range is arranged so as to include the interior of the storage chamber as a food storage area in a part of the periphery of the imaging range, The illumination unit is an imaging device disposed on the inner side of the storage chamber relative to the lens.

Images