Imaging system and imaging device

Abstract

To provide an imaging system that can appropriately output image data in a refrigerator (storage chamber).SOLUTION: An imaging system comprises: a refrigerator comprising a rotary type door provided with a door housing part, and a drawer part arranged below the door housing part, and capable of being drawn by a drawer type door; and an imaging device arranged on the upper side of the door housing part. The imaging device can take one image in which the drawn drawer part is shown approximately at a center, and the door housing part is shown.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to an imaging system and an imaging device.

Background Art

[0002] In a refrigerator, for the purpose of inventory management in the refrigerator, etc., there has been proposed a device that photographs the inside of the refrigerator with a camera. According to the refrigerator described in Patent Document 1, "it includes a plurality of storage rooms partitioned into at least one photographing range, a plurality of in-cabinet cameras that photograph at least one of the photographing ranges, or at least one in-cabinet camera that photographs a plurality of photographing ranges, a control device that stores images of the plurality of photographing ranges in a storage device and communicates with an operation display device that displays the images." And in the operation display device, "when an operation for instructing the movement of an image inside the refrigerator is performed by touching a finger and moving it left and right (a so-called swipe operation or flick operation performed on the touch panel of a smartphone or tablet terminal), the control device displays an image on the door pocket side of the door located in the opposite direction to the counter-instruction direction inside the refrigerator, lined up with the image inside the refrigerator, in the observation state where the user observes the inside of the refrigerator from the front side of the refrigerator."

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to simultaneously view the inside of the refrigerator and the door pocket of the door, the refrigerator described in Patent Document 1 requires the user to perform a swipe operation or a flick operation on the operation display device.

Means for Solving the Problems

[0005] The imaging system of the present invention made in consideration of the above circumstances includes a rotary door provided with a door storage part, a refrigerator including a drawer part arranged below the door storage part and being drawable by a drawer-type door, and an imaging device arranged above the door storage part. The imaging device is characterized in that it can capture an image in which the drawn-out drawer part is imaged at approximately the center and the door storage part is also imaged. Other aspects of the present invention will be described in the embodiments described later.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22A

Figure 22B

Figure 22C

Embodiments for Carrying Out the Invention

[0007] Embodiments (modes for carrying out) of the present invention will be described in detail with appropriate reference to the drawings. The order of the control steps such as the operation flow disclosed in the embodiments can be interchanged or executed simultaneously as long as they do not conflict with each other. <<First Embodiment>> FIG. 1 is a front view showing the refrigerator 100 according to the first embodiment. FIG. 2A is a side view showing the refrigerator 100 according to the first embodiment. FIG. 2B is a side view when the imaging device 30 according to the first embodiment is housed. FIG. 3 is a front view showing a state where the door of the refrigerator 100 according to the first embodiment is opened. FIG. 4 is a top view showing a state where the door of the refrigerator 100 according to the first embodiment is opened. In the following description, the six-door refrigerator 100 will be described as an example, but it is not limited to six doors.

[0008] As shown in FIG. 1, the heat-insulating box body (box body 10) of the refrigerator 100 has storage compartments in the order of the refrigerating compartment 1 from above, the ice-making compartment 2 provided side by side on the left and right, the upper freezing compartment 3, the vegetable compartment 4, and the lower freezing compartment 5. The refrigerator 100 is provided with doors for opening and closing the openings of the respective storage compartments. These doors are the rotary refrigerating compartment doors 1a and 1b divided into left and right for opening and closing the opening of the refrigerating compartment 1, and the drawer-type ice-making compartment door 2a, upper freezing compartment door 3a, vegetable compartment door 4a, and lower freezing compartment door 5a for opening and closing the openings of the ice-making compartment 2, upper freezing compartment 3, vegetable compartment 4, and lower freezing compartment 5, respectively. Note that the positions of the vegetable compartment 4 and the lower freezing compartment 5 may be changed to a vertical arrangement. Also, the ice-making compartment 2, the upper freezing compartment 3, and the lower freezing compartment 5 are collectively referred to as the freezing compartment 7.

[0009] In order to fix the refrigerating compartment doors 1a and 1b to the refrigerator 100, door hinges (reference numerals omitted) are provided at the upper and lower parts of the refrigerating compartment 1. As shown in FIGS. 3 and 4, when the refrigerating compartment doors 1a and 1b are opened and closed, door storage portions 11a (door storage portion a) and 11b (door storage portion b) are provided.

[0010] The refrigerating compartment 1 is a refrigerated storage compartment in which the inside of the compartment is set to a refrigerating temperature range (0°C or higher), for example, an average of about 4°C. The freezing compartment 7 is a frozen storage compartment in which the inside of the compartment is set to a freezing temperature range (less than 0°C), for example, an average of about -18°C. The vegetable compartment 4 is a refrigerated storage compartment in which the inside of the compartment is set to a refrigerating temperature range, for example, an average of about 6°C, and is a refrigerated storage compartment that suppresses drying of food.

[0011] In the refrigerator 100, an imaging device 30 for photographing the interior of the refrigerator is disposed at substantially the center of the upper portion outside the storage chamber. That is, it is provided at substantially the center above the box body (heat-insulated box body) of the refrigerator 100. When the door of the box body is closed, the lens 34 (see FIGS. 5A and 5B) of the imaging device 30 is located in front of the opening edge surface 10a of the storage chamber, preferably in front of the refrigerator doors 1a and 1b (doors). That is, by providing the imaging device 30 outside the storage chamber, the situation inside the refrigerator including the door storage portions 11a and 11b can be photographed from above.

[0012] Further, the imaging device 30 has a mechanism that slides back and forth (a mechanism that slides in a direction perpendicular to the storage chamber opening), and can move back and forth across the front surface of the door of the refrigerator 100. For example, as shown in FIG. 2B, since it can move behind the front surface of the door or behind the edge surface of the storage chamber opening, it does not affect the depth dimension during packaging. Also, the number of refrigerators loaded on the transport truck does not decrease, and even after installation, when the imaging device 30 is not used, it can be stored to hide the imaging range. By hiding the imaging range, a configuration can be adopted that takes into consideration users who do not preferably think that the imaging area of the camera always enters their living area.

[0013] On the refrigerator door 1b, a photographing button 39 (a photographing command receiving unit that receives a photographing command from a user) used when manually photographing by the imaging device 30 is provided. The photographing button 39 may be provided in other areas of the refrigerator door 1a, the box body, or the door.

[0014] FIG. 5A is a front view showing the imaging device 30 according to the first embodiment. FIG. 5B is a side view showing the imaging device 30 according to the first embodiment. FIG. 5C is a side cross-sectional view (sectional view taken along line AA in FIG. 5A) showing the imaging device 30 according to the first embodiment. FIG. 6 is a perspective view showing the imaging device 30 according to the first embodiment.

[0015] The imaging device 30 is composed of a case part 31 and a lid part 32. In front of the case part 31, there are a camera 33 (imaging part) and a lens 34 which is a fish-eye lens. As shown in FIG. 6, the lens 34 is surrounded by a light-shielding wall 35. In the vicinity of the light-shielding wall 35, there are a plurality of lighting parts 38.

[0016] Inside the case part 31, as shown in FIG. 5C, a control SoC 41 and a wireless LAN unit 42 are stored. The control SoC 41 (System on a chip) is an integrated circuit designed to integrate functions such as a processor core and general functions of a microcontroller, as well as application-specific functions, on a single chip of an integrated circuit and function as a system in cooperation. Note that the control SoC 41 is an example of an integrated circuit, and other integrated circuits or the like can be used as an alternative.

[0017] The imaging device 30 provided outside the storage chamber images the door storage parts 11a and 11b and the inside of the storage chamber in an overlooking manner.

[0018] The features of the imaging device 30 will be further described. (1) The camera 33 used in the imaging device 30 can capture images with a wide angle of 180 degrees or more. By having a fish-eye lens, when the door is opened, the entire interior of the storage can be visually recognized in an overlooking manner. (2) Since the lens 34 for the camera of the imaging device 30 is located in front of the front surface of the refrigerator door, the state inside the storage can be captured even when only the drawers 6 of the vegetable compartment 4 and the lower freezer compartment 5 are opened.

[0019] (3) The imaging device 30 includes an illumination unit 38. By using the external illumination unit 38 that can be adjusted to an illuminance suitable for shooting, an image with high visibility can be obtained when shooting. Since the illumination provided inside the refrigerator has a relatively high illuminance, when shooting with a camera, the reflection of the stored items is strong, and the captured image is likely to be overexposed. Therefore, in this embodiment, during shooting, the internal refrigerator illumination is turned off, and the external illumination unit 38 is turned on for shooting. Also, the illumination of the illumination unit 38 is made directional so as to irradiate the entire interior of the refrigerator, the door storage sections 11a and 11b when the door is opened, and the second storage chambers (for example, the vegetable chamber 4 and the lower freezer 5). (4) By having a light-shielding wall 35, the imaging device 30 can prevent direct light from the illumination unit 38 from entering the shooting range, thereby obtaining an image with high visibility that suppresses reflection and overexposure. (5) By providing a light-shielding wall 35 that blocks direct light between the illumination unit 38 and the lens 34 of the imaging device 30, the installation position of the illumination can be brought as close as possible to the lens 34. Therefore, it can be housed in a compact outer shape, and furthermore, the effect of preventing damage to the lens 34 can also be obtained.

[0020] (6) The imaging device 30 has a mechanism that slides back and forth, and can move back and forth across the front surface of the refrigerator door 100. Therefore, it does not affect the depth dimension during packaging. Also, the number of refrigerators loaded on the transport truck does not decrease, and even after installation, when the imaging device 30 is not in use, it can be stored to conceal the shooting range. (7) The imaging device 30 includes a communication unit (wireless LAN unit 42). The image data transmitted from the communication unit is supplied to a server 54 (see FIG. 7), for example, via a home wireless router (for example, the home router 53 (see FIG. 7)). Details will be described later. (8) The imaging device 30 has a communication circuit that transmits power supply and shooting commands connected to the refrigerator body side. The refrigerator body receives power supply from a commercial power supply or the like. The power supply of the imaging device 30 can be received from the commercial power supply via the refrigerator body. Note that the imaging device 30 may be driven by a rechargeable battery.

[0021] FIG. 7 is a configuration diagram showing an imaging system of the refrigerator 100 according to the first embodiment. As sensors on the door side, door sensors (not shown) for detecting the opening and closing states of the refrigerator doors 1a and 1b, the ice-making chamber door 2a, the upper freezer door 3a, the vegetable chamber door 4a, and the lower freezer door 5a are also provided.

[0022] On the upper part of the refrigerator 100, a main microcomputer 52 (processing device, control device, control unit) equipped with a CPU (Central Processing Unit) which is a part of the control device, memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and an interface circuit, etc. is arranged. The main microcomputer 52 controls the imaging system for imaging inside the cabinet based on information from the door sensor or the shooting button 39. Also, the main microcomputer 52 supplies power to the imaging device 30.

[0023] When the automatic shooting is set, the main microcomputer 52 transmits a shooting command to the control SoC 41 when it detects an angle of a certain degree or more based on information such as the opening angle of the door from the door sensor. The control SoC 41 gives a shooting instruction to the camera 33 and acquires unprocessed image data from the camera 33. The control SoC 41 performs image processing on the unprocessed image data, for example, as shown in FIG. 10. On the other hand, when the manual shooting is set, when the main microcomputer 52 detects that the shooting button 39 has been operated, it transmits a shooting command to the control SoC 41. The subsequent processing is the same as that in the case of automatic shooting setting.

[0024] As described above, the imaging device 30 is provided with a wireless LAN unit 42 that can be connected to the home router 53. By providing this wireless LAN unit 42, the in-cabinet image data processed by the control SoC 41 can be transmitted to the server 54 via the home router 53. The server 54 transmits the in-cabinet image data in response to a request for displaying the in-cabinet image from a mobile device such as a smartphone 55 owned by the user or a personal computer. Thereby, the user can grasp the situation inside the cabinet.

[0025] FIG. 8 is a flowchart showing the processing at the time of manual shooting setting according to the first embodiment. This will be described with reference to FIG. 7 as appropriate. When the setting from a setting device (not shown) is for camera shooting setting (camera setting ON), the main microcomputer 52 determines whether there is an opening of the door by the door sensor (is the door open?) (S31). If there is an opening of the door (S32, Yes), the process proceeds to S33. On the other hand, if there is no opening of the door (S32, No), the main microcomputer 52 returns to S32.

[0026] In S33, the main microcomputer 52 determines whether there is an operation (pressing) of the shooting button 39. If the shooting button 39 is pressed (S33, Yes), the process proceeds to S34. If the shooting button 39 is not pressed (S33, No), the process returns to S32.

[0027] In S34, the main microcomputer 52 gives a shooting command to the control SoC 41, and turns off the lighting in the warehouse and turns on the outdoor lighting (lighting unit 38) (S35). The control SoC 41 gives a shooting command to the camera 33 to take a picture (S36). Then, the main microcomputer 52 turns on the lighting in the warehouse and turns off the outdoor lighting (lighting unit 38) (S37).

[0028] The camera 33 transmits unprocessed image data to the control SoC 41 (S38), and the control SoC 41 performs in-warehouse image correction processing (image processing) (S39). Details of the image processing will be described later with reference to FIG. 10.

[0029] The control SoC 41 performs image output processing (image update) of the processed image data to the server 54 via the wireless LAN unit 42 and the home router 53 (S40). When the image is updated, the server 54 notifies the user's mobile device or the like (S41). The above is the processing at the time of manual shooting setting.

[0030] FIG. 9 is a flowchart showing the processing at the time of automatic shooting setting according to the first embodiment. It will be described with reference to FIG. 7 as appropriate. Note that the description of the same processing as in FIG. 8 is omitted. When the setting from a setting device (not shown) is for camera automatic shooting setting (camera automatic setting ON), the main microcomputer 52 determines whether there is an opening of the door by the door sensor (Is the door open?) (S31A, S32). If there is an opening of the door (S32, Yes), the process proceeds to S33A. On the other hand, if there is no opening of the door (S32, No), the main microcomputer 52 returns to S32.

[0031] In S33A, the main microcomputer 52 determines whether the opening angle of the door is equal to or greater than a predetermined angle (for example, 85 degrees or more). If the opening angle of the door is equal to or greater than the predetermined angle (S33A, Yes), the process proceeds to S34. If the opening angle of the door is less than the predetermined angle (S33A, No), the process returns to S32. S34 to S41 are the same as in FIG. 13. Note that in S33A, in the case of the double - door refrigerator 1, if the opening angle of at least one of the refrigerator doors 1a and 1b is equal to the predetermined angle (S33A, Yes), the process proceeds to S34. The above is the processing at the time of automatic shooting setting.

[0032] The features of the imaging system of the refrigerator 100 will be further described. (1A) It has a detection unit (for example, the main microcomputer 52) that detects the opening angle of the refrigerator door, and can automatically perform shooting when the detection unit determines that the angle is equal to or greater than a predetermined angle (for example, 85 degrees). (2A) When shooting, the interior lighting is turned off and the lighting of the imaging device 30 is turned on, which not only adjusts the illuminance suitable for shooting but also has the effect of informing the user of the shooting timing. (3A) It has a detection unit (for example, installed in the drawer unit 6) that detects when the pulling - out distance of the refrigerator 100 reaches a certain level or more. By detecting the pulling - out operation by the detection unit and automatically performing shooting, the imaging range of the vegetable compartment and the freezer compartment can be kept substantially constant.

[0033] (4A) The refrigerator door has a shooting button (e.g., the shooting button 39) for photography. When the user presses the shooting button, the interior lighting of the refrigerator turns off, and after the lighting of the imaging device 30 turns on, photography is performed. Therefore, it is easy to recognize the timing of photography. (5A) The captured image is subjected to image correction processing, and the distorted image by the wide-angle camera is corrected, so that an image close to the user's line of sight can be provided. (6A) It has an application service that allows the user to view the image via a mobile phone or the like. The image can be automatically determined by the detection unit provided on the door (door) or the drawer as to which storage compartment the captured image is of.

[0034] As described above, the refrigerator 100 of the present embodiment has a storage compartment (e.g., the refrigerating compartment 1) with an opening at the front, a door (e.g., the refrigerator compartment doors 1a, 1b) having a rotatable storage part (e.g., the door storage parts 11a, 11b) provided on the front surface of the storage compartment, a door opening angle detection means for detecting the opening angle of the door, an illumination device for illuminating the storage compartment when the door is detected to be opened by the door opening detection means, an imaging device (e.g., the imaging device 30) provided at approximately the center of the upper part outside the storage compartment, the imaging device having a fish-eye lens (e.g., the lens 34) with a wide-angle field of view, an illumination device (e.g., the illumination unit 38) that lights up during photography, and a communication means (e.g., the wireless LAN unit 42) for transmitting the captured image data. The refrigerator 100 has a control device (e.g., the main microcomputer 52) that issues a shooting command to the imaging device when a predetermined opening angle is detected by the door opening angle detection means.

[0035] In the state where the door is open, the image captured by the imaging device can, as shown in FIG. 11, by the action of the fish-eye lens with a wide-angle field of view, in one captured image, cover not only the storage compartment but also the entire door storage part within the shooting range. Furthermore, it is possible to include the vegetable compartment (e.g., the vegetable compartment 4) and the freezer compartment (e.g., the lower freezer compartment 5) of the drawer storage part within this shooting range. There is no need to provide a plurality of imaging devices in each storage compartment as in Patent Document 1, nor is there a need for a complicated and expensive drive device and drive control for changing the shooting range.

[0036] The image captured by the imaging device is sufficient to grasp the state of the stored items in the storage as it is. However, due to the effect of the fish-eye lens, the image is curved, and objects closer to the imaging device are more distorted. Also, since it is a visual field image from above the refrigerator, there is room for improvement in terms of visibility as it is different from the positional relationship arranged in the visual field from the front of the refrigerator that the user usually observes.

[0037] According to the present embodiment, by performing the following image processing (S39), it is possible to provide an in-storage image in a state close to the positional relationship of the visual field from the front where the user usually observes the refrigerator for a distorted and curved image captured by the imaging device 30 having a wide-angle fish-eye lens.

[0038] FIG. 10 is a flowchart showing the image processing (S39) according to the first embodiment. The images of each process will be described with reference to FIGS. 11 to 16. FIG. 11 is an image showing the inside of the storage (the storage c as the inside of the refrigerating chamber 1, the door storage part a as the door storage part 11a, and the door storage part b as the door storage part 11b) before the image processing. FIG. 12 is an image showing the cut-out process of each storage part. FIG. 13 is an image showing the distortion correction process of each storage part. FIG. 14 is an image showing the rotation process of each storage part. FIG. 15 is an image showing the trapezoid correction process of each image. FIG. 16 is an image showing the composition process of each image.

[0039] The main microcomputer 52 inputs the above-described captured image (see FIG. 11) (S391), and performs a cut-out process (cut-out process) in a fan shape so as to respectively include the ranges of each storage part (S392). Specifically, as shown in FIG. 12, the main microcomputer 52 performs a process of cutting out the internal image of the refrigerating chamber 1 and the images of the door storage parts 11a and 11b, for example, cutting them out in a fan shape (see FIG. 12).

[0040] Next, the main microcomputer 52 performs distortion correction processing (distortion correction) on the images of each storage unit curved by the action of the fisheye lens (S393). Specifically, as shown in FIG. 13, the main microcomputer 52 performs distortion correction processing on the internal image of the refrigerator compartment 1 and the curved images of the door storage units 11a and 11b, respectively. As this distortion correction processing, various known correction algorithms can be used. For example, calibration processing with parameters set in advance (e.g., before the refrigerator is shipped) can be executed to correct the distortion.

[0041] Next, the main microcomputer 52 performs image rotation processing to rotate the images of the respective storage units that have undergone distortion correction so as to have a field of view from the front, which is the observation state of the user (S394). Specifically, as shown in FIG. 14, the main microcomputer 52 arranges the door storage unit 11a on the left side and the door storage unit 11b on the right side so as to have a field of view from the front, which is the observation state of the user.

[0042] Next, the main microcomputer 52 performs trapezoidal distortion correction processing (trapezoid correction processing) to correct the distortion of the image distorted in a trapezoidal shape in the depth direction into a rectangle (S395). Specifically, as shown in FIG. 15, the main microcomputer 52 performs trapezoid correction processing on the internal image of the refrigerator compartment 1 and the images of the door storage units 11a and 11b. For example, by executing perspective projection transformation and its complementary processing such as the nearest neighbor method and the lanczos method, the images of the storage units that were generally trapezoidal or fan-shaped can be converted into a rectangle.

[0043] Next, the main microcomputer 52 performs image composition processing on the images created in S395 to create a single image (S396). This image is displayed on the application of the smartphone 55 (see FIG. 7). Note that the embodiment shows an example, and each step may be appropriately rearranged.

[0044] The method for processing image data in the refrigerator according to the first embodiment is a method for processing image data in the refrigerator that processes the image data captured by an imaging device equipped with a wide-angle lens that includes a plurality of food storage areas in the imaging range. The method includes a cutting process (S392) for cutting out a plurality of portions from the image data captured by the imaging device 30, a synthesizing process (S396) for synthesizing the images cut out in the cutting process, and an output process (S40) for outputting the image synthesized in the synthesizing process as one piece of image data. By the cutting process, at least a part of an area that is not a food storage area, for example, an area where only the wall surface of the refrigerator or the floor surface outside the refrigerator is shown without food being shown, can be removed from the input captured image (FIG. 11). By synthesizing the images of the cut-out food storage areas, the ratio of the food storage area displayed in the synthesized image (FIG. 16) to the entire display screen of, for example, the smartphone 55 on which this image is displayed can be increased. In this embodiment, the cutting process is executed on one image captured by one imaging device 30, but two or more imaging devices may be prepared, two or more images may be captured, and the portions including the food storage areas included in the respective image data may be cut out respectively.

[0045] Thus, in this embodiment, it is characteristic that the ratio of the area occupied by the food storage area in the entire image data is larger in the image after the synthesizing process than in the image data captured by the wide-angle lens. For example, when comparing FIG. 11 and FIG. 16, the area where food is shown becomes larger, making it easier for the user to view.

[0046] In this embodiment, it is characteristic that the separation distance between a plurality of parts cut out by the cutting process becomes shorter after the synthesis process. For example, when comparing FIG. 12 and FIG. 16, the separation distance between the door storage part a, the storage c, and the door storage part b shown in FIG. 16 is smaller than the separation distance between the door storage part a, the storage c, and the door storage part b shown in FIG. 12, which can be made easier for the user to view. Also, since a plurality of storage areas that are separated from each other in actual distance, such as a door storage part and a storage, or two door storage parts, are output as one image, as in Patent Document 1, it is possible to make the whole easier to view without causing the user to perform a swipe operation or a flick operation to switch the displayed image. In particular, since the two door storage parts rotate and open in opposite directions with the door hinges arranged at different positions as the rotation axes, the separation in actual distance is relatively large in the refrigerator.

[0047] In this embodiment, the refrigerating chamber 1 has been described by taking the example of a double - opening storage chamber, but it can also be applied to the case of a single - opening storage chamber.

[0048] <<Second Embodiment>> In the first embodiment, the image synthesis process of the refrigerating chamber 1 (the first storage chamber) has been described, but it is not necessarily limited to this. In the second embodiment, the image synthesis process including the refrigerating chamber 1 (the first storage chamber) and the vegetable chamber 4 etc. (the second storage chamber) will be described.

[0049] FIG. 17 is a side view showing the state where the drawer part 6 of the refrigerator 100 according to the second embodiment is pulled out. FIG. 18 is an image showing the cutting process of the drawer part 6 of the refrigerator according to the second embodiment. FIG. 19 is an image showing the enlargement process of the drawer part 6 of the refrigerator according to the second embodiment. The imaging device 30 for photographing the interior of the refrigerator can image the inside of the drawer part 6. Therefore, the storage state can be grasped from the image of the drawer part 6.

[0050] FIG. 20 is an image showing the composite processing of each image including the drawer portion 6 according to the second embodiment. In FIG. 20, compared with FIG. 16, the image of the drawer portion 6 (the second storage d) is also composite. Thereby, the user can grasp the storage state including the refrigerator compartment 1 (the first storage compartment) and the vegetable compartment 4 etc. (the second storage compartment) at once. Note that FIG. 16 is an image obtained by composite processing from the captured image shown in FIG. 12.

[0051] That is, the refrigerator 100 has at least a first storage compartment that opens like a double door and a second storage compartment that opens in the front-rear direction, and the cut-out process is characterized by being the first storage compartment and the second storage compartment. Further, the refrigerator 100 has at least a first storage compartment that opens like a double door and a second storage compartment that opens in the front-rear direction. The first storage compartment has a door storage portion, and the cut-out process may be the interior of the first storage compartment, the door storage portion, and the second storage compartment. The cut-out process can perform the cut-out of the second storage compartment when the opening of the second storage compartment is detected.

[0052] FIG. 21 is another example of an image showing the composite processing of each image including the drawer portion 6 according to the second embodiment. Since the display screen of the smartphone 55 on which the image is assumed to be displayed is often rectangular, as shown in FIG. 21, the positions of the door storage portions a and b can also be moved from the side of the storage c so that the composite image becomes rectangular as a whole. In the example of FIG. 21, the door storage portions a and b are moved to the side opposite to the second storage d with the storage c as the center.

[0053] That is, the cut-out process is executed with at least one door storage portion, the interior of the first storage compartment, and the second storage compartment as food storage areas respectively. In the composite processing, at least one door storage portion and the second storage compartment are arranged so as to be located on opposite sides across the interior of the first storage compartment.

[0054] <<Modification Example of Imaging Device>> In the first embodiment, it has been described that the imaging device 30 has a slide mechanism, but it is not necessarily limited thereto. In the modification example, a folding mechanism will be described.

[0055] FIG. 22A is an explanatory diagram showing the normal state of the imaging device 30. FIG. 22B is an explanatory diagram showing the state of the imaging device 30 during storage. FIG. 22C is an explanatory diagram showing the state of the imaging device 30 when stored.

[0056] The refrigerator 100 (see FIG. 1) includes a box body that forms a storage chamber with an open front, an imaging device 30 provided outside the storage chamber, and a support portion 45 (hinge portion) that supports the imaging device 30 so as to be movable in a direction at least perpendicular to the opening.

[0057] As shown in FIGS. 22A to 22C, the tip portion having the camera 33 of the imaging device 30 is configured to be foldable in two from the support portion 45, and since it does not affect the depth dimension during packaging, the number of refrigerators loaded on the transport truck does not decrease. Even after installation, when the imaging device 30 is not used, it can be stored and the imaging range can be hidden.

[0058] Further, the imaging device 30 can also revolve (move in a circular locus around the hinge portion; revolve around an axis) in the direction of looking into the interior of the refrigerator. Since the box body exists in the circular locus, the angle of revolution possible is less than 360° when observed from a direction perpendicular to the revolution plane. In this case, a drive portion that can apply a force in the revolution direction may be attached to the imaging device 30 and controlled so as to be at an angle of looking into the interior of the refrigerator compartment or the like during imaging. For example, during imaging, the imaging device 30 can be positioned within the movement locus of the door. By doing so, the interior of the refrigerator can be imaged more effectively. Further, after imaging is completed, it can be returned outside the movement locus of the door, or revolved to a position behind the edge surface of the front opening of the door or the storage chamber and stored. Therefore, it is characterized in that it is easy to lengthen the arm connecting the support portion 45 (hinge portion) and the camera.

[0059] Note that the present invention is not limited to the above-described first and second embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Also, it is possible to add, delete, or replace a part of the configuration of the above-described embodiments with other configurations. For example, the shooting button 39 in FIG. 3 may be provided on the refrigerator door 1a or may be provided on the upper surface of the drawer portion 6. Further, a plurality of imaging devices used for shooting may be prepared, and the image data of the food storage area included in the images shot respectively may be used for compositing processing into one image.

[0060] According to the refrigerator 100 of the present embodiment, by distorting, trimming, and compositing a curved image captured by an imaging device having a fisheye lens (wide-angle lens) provided outside the storage chamber, and outputting one image, it is possible to provide an in-cabinet image close to the user's observation state while minimizing the image switching operation on the display operation device.

[0061] An imaging device having a wide-angle lens such as a fisheye lens provided outside the storage chamber captures an image of the inside of the storage chamber including the door storage portion from above in a bird's-eye view, performs distortion correction of the curved image, trimming of each storage portion, compositing of the processed images, and these image processes, so that the situation inside the refrigerator can be provided to the user's line of sight in a state close to the user's line of sight with one image, and it becomes easy to recognize the situation inside the cabinet. Also, since it is not necessary to provide a large number of imaging devices or an imaging device having variable and complex controls, it is advantageous in terms of cost, and problems such as damage to the lens and obstruction of the shooting range due to the stored items approaching the lens can be solved. The user can also check the latest situation inside the refrigerator with an image even when away from home, which can lead to reduction of unnecessary shopping and food loss.

Explanation of Reference Numerals

[0062] 1 Refrigerator compartment (first storage chamber) 1a, 1b Refrigerator compartment doors 2 Ice-making chamber (second storage chamber) 2a Ice-making chamber door 3 Upper Refrigerating Chamber (Second Storage Chamber) 3a Door of Upper Refrigerating Chamber 4 Vegetable Chamber (Second Storage Chamber) 4a Door of Vegetable Chamber 5 Lower Refrigerating Chamber (Second Storage Chamber) 5a Door of Lower Refrigerating Chamber 6 Drawer 10 Cabinet 10a Opening Edge Surface 11a, 11b Door Storage 30 Imaging Device 31 Case 32 Cover 33 Camera (Imaging Unit) 34 Lens (Wide - angle Lens) 35 Light - shielding Wall 38 Lighting Unit 39 Shooting Button (Shooting Command Reception Unit) 41 Control SoC (Output Unit) 42 Wireless LAN Unit 45 Support Unit (Hinge Unit) 51 Door Sensor 52 Main Microcomputer (Processing Device) 53 In - home Router 54 Server 55 Smartphone 100 Refrigerator

Claims

1. A refrigerator comprising: a storage compartment with an opening at the front; a revolving door that opens and closes the opening, has a door storage section disposed therein, and has a rotation axis on either the left or right side of the storage compartment when viewed from the front; an imaging device having a lens and an illumination unit located at an outer upper portion of the storage chamber and forward of an edge surface of the opening, the imaging device captures an image including the door storage section and the interior of the storage compartment when the door is open, An imaging system in which the illumination unit is disposed at one of the positions on the left and right sides of the lens.

2. A refrigerator comprising: a storage compartment with an opening at the front; a revolving door that opens and closes the opening, has a door storage section disposed therein, and has a vertical axis of rotation; an imaging device having a lens and an illumination unit located at an outer upper portion of the storage chamber and forward of an edge surface of the opening, the imaging device captures an image including the door storage section and the interior of the storage compartment when the door is open, The illumination unit is disposed in a position closer to the rotation axis of the door than the lens.

3. In the imaging system according to claim 1 or claim 2, The revolving door has a left door with a left door storage section and a rotation axis on the left side of the storage chamber when viewed from the front, and a right door with a right door storage section and a rotation axis on the right side of the storage chamber when viewed from the front, The illumination units are disposed on both the left and right sides of the lens, The imaging device is an imaging system that captures an image including the left door storage compartment when the left door is open, the right door storage compartment when the right door is open, and the inside of the storage compartment.

4. In the imaging system according to claim 1 or claim 2, The imaging system, wherein the illumination unit is also disposed behind the lens.

5. In the imaging system according to claim 1 or claim 2, An imaging system having a light-shielding wall between the lens and the illumination unit.

6. An imaging device disposed on the upper outer side of a storage compartment of a refrigerator, the storage compartment having an opening at the front, a door storage section for opening and closing the opening, and a rotary door having a rotation axis on either the left or right side of the storage compartment when viewed from the front, a lens and a lighting unit located at an outer upper portion of the storage chamber and forward of an edge surface of the opening; taking an image including the door storage section and the inside of the storage compartment when the door is open; The imaging device, wherein the illumination unit is disposed at one of the positions on the left and right sides of the lens.

7. An imaging device disposed on the upper outer side of a storage compartment of a refrigerator, the storage compartment having an opening at the front, a door storage section for opening and closing the opening, and a rotary door with a vertical axis of rotation, a lens and a lighting unit located at an outer upper portion of the storage chamber and forward of an edge surface of the opening; taking an image including the door storage section and the inside of the storage compartment when the door is open; The illumination unit is an imaging device disposed at a position closer to the rotation axis of the door than the lens.

8. In the imaging device according to claim 6 or claim 7, the refrigerator has, as the revolving door, a left door having a left door storage section and a rotation axis on the left side of the storage chamber in a front view, and a right door having a right door storage section and a rotation axis on the right side of the storage chamber in a front view, The illumination units are disposed on both the left and right sides of the lens, The imaging device is an imaging device that captures an image including the left door storage compartment when the left door is open, the right door storage compartment when the right door is open, and the inside of the storage compartment.

9. In the imaging device according to claim 6 or claim 7, The imaging device, wherein the illumination unit is also disposed behind the lens.

10. In the imaging device according to claim 6 or claim 7, An imaging device having a light-shielding wall between the lens and the illumination unit.

Images