depot

Abstract

The image data processing method in the refrigerator of the present application is an image data processing method in a refrigerator that processes image data captured by one or more than two image pickup devices that include a total of a plurality of food storage areas in a captured range, and includes and performs the following processes: a cutting process (S392) that cuts a plurality of portions that respectively include food storage areas from one or more than two image data captured by the image pickup devices; a synthesis process (S396) that generates one image data using the plurality of portions; and an output process that outputs one image data synthesized by the synthesis process.

Description

[0001] (This application is a divisional application of an application with the application date of December 1, 2020, the application number of 202080018909.3, and the invention name of "Image data processing method in refrigerators and refrigerators") TECHNICAL FIELD

[0002] The present application relates to an image data processing method in refrigerators and a refrigerator. BACKGROUND

[0003] In a refrigerator, for the purpose of inventory management and the like in the refrigerator, a scheme of photographing the inside of the refrigerator by using a camera is proposed. According to the refrigerator described in Patent Literature 1, "comprises: a plurality of storage compartments divided into at least one photographing range; a plurality of in-store cameras photographing at least one photographing range, or at least one in-store camera photographing a plurality of photographing ranges; and a control device that stores images of a plurality of photographing ranges in a storage device and communicates with an operation display device that displays images". Further, it is described that, in the operation display device, "when an operation of indicating movement of an image in the storage compartment is performed by moving left and right by a finger touch (so-called swipe operation, flip operation implemented on a touch panel of a smartphone, tablet terminal), the control device displays an image on the door shelf side of the door located in the opposite direction of the storage compartment from the observation state in which the user observes the inside of the storage compartment from the front side of the refrigerator, in connection with the image in the storage compartment".

[0004] PRIOR ART DOCUMENTS

[0005] PATENT LITERATURE

[0006] Patent Literature 1: Japanese Patent No. 6296908 SUMMARY

[0007] PROBLEMS TO BE SOLVED BY THE INVENTION

[0008] The refrigerator described in Patent Literature 1 requires the user to perform a swipe operation, a flip operation in the operation display device in order to observe the inside of the storage compartment and the door shelf of the door at the same time.

[0009] METHOD FOR SOLVING THE PROBLEMS

[0010] The image data processing method in a refrigerator according to the present application, which is achieved in view of the above, is an image data processing method in a refrigerator that processes image data captured by one or two or more image capturing devices that contain a total of a plurality of food storage areas in a capturing range, characterized by containing and executing: a cutting process (e.g., S392) that cuts a plurality of portions each containing a food storage area from one or two or more image data captured by the image capturing devices; a synthesizing process (e.g., S396) that generates one image data using the plurality of portions; and an output process (e.g., S40) that outputs one image data synthesized in the synthesizing process. Other modes of the present application are described in the embodiments described later. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a front view showing the refrigerator of the first embodiment.

[0012] Figure 2A is a side view showing the refrigerator of the first embodiment.

[0013] Figure 2B is a side view in a case where the image capturing device of the first embodiment is stored.

[0014] Figure 3 is a front view showing a state where the door of the refrigerator of the first embodiment is opened.

[0015] Figure 4 is a plan view showing a state where the door of the refrigerator of the first embodiment is opened.

[0016] Figure 5A is a front view showing the image capturing device of the first embodiment.

[0017] Figure 5B is a side view showing the image capturing device of the first embodiment.

[0018] Figure 5C is a side sectional view showing the image capturing device of the first embodiment.

[0019] Figure 6 is a perspective view showing the image capturing device of the first embodiment.

[0020] Figure 7 is a structure diagram showing the image capturing system of the refrigerator of the first embodiment.

[0021] Figure 8 is a flowchart showing the process at the time of manual capturing setting of the first embodiment.

[0022] Figure 9 is a flowchart showing the process at the time of automatic capturing setting of the first embodiment.

[0023] Figure 10 is a flowchart showing the image processing of the first embodiment.

[0024] Figure 11 is an image showing the inside of the library before the image processing.

[0025] Figure 12 is an image showing the cropping processing of each storage section.

[0026] Figure 13 is an image showing the distortion correction processing of each storage section.

[0027] Figure 14 is an image showing the rotation processing of each storage section.

[0028] Figure 15 is an image showing the keystone correction processing of each image.

[0029] Figure 16 is an image showing the synthesis processing of each image.

[0030] Figure 17 is a side view showing a state in which the pull-out section of the refrigerator of the second embodiment is pulled out.

[0031] Figure 18 is an image showing the cropping processing of the pull-out section of the refrigerator of the second embodiment.

[0032] Figure 19 is an image showing the enlargement processing of the pull-out section of the refrigerator of the second embodiment.

[0033] Figure 20 is an image showing the synthesis processing of each image including the pull-out section of the second embodiment.

[0034] Figure 21 is another example of an image showing the synthesis processing of each image including the pull-out section of the second embodiment.

[0035] Figure 22A is an explanatory diagram showing a state of the imaging device at normal time.

[0036] Figure 22B is an explanatory diagram showing a state of the imaging device at midway of the storage.

[0037] Figure 22C is an explanatory diagram showing a state of the imaging device at the storage. DETAILED DESCRIPTION

[0038] A mode (embodiment) for carrying out the present application will be explained in detail with appropriate reference to the drawings. The order of control steps of the flow of actions and the like disclosed in the embodiment can be changed in order or executed simultaneously as long as they do not contradict each other.

[0039] <<First Embodiment>>

[0040] Figure 1 is a front view showing the refrigerator 100 of the first embodiment. Figure 2A is a side view showing the refrigerator 100 of the first embodiment. Figure 2B is a side view in a case where the camera 30 of the first embodiment is housed. Figure 3 is a front view showing a state where the door of the refrigerator 100 of the first embodiment is opened. Figure 4 is a plan view showing a state where the door of the refrigerator 100 of the first embodiment is opened. Further, in the following description, a 6-door refrigerator 100 will be described as an example, but is not limited to the 6-door refrigerator.

[0041] As shown in Figure 1 , the refrigerator 100 has storage compartments in the order of the refrigerating compartment 1, the ice-making compartment 2 and the upper freezer compartment 3 arranged side by side to the left and right, the vegetable compartment 4, and the lower freezer compartment 5 from the top. The refrigerator 100 includes doors that open and close the openings of the respective storage compartments. These doors are the refrigerating compartment doors la, lb that open and close the opening of the refrigerating compartment 1, and the ice-making compartment door 2a, the upper freezer compartment door 3a, the vegetable compartment door 4a, and the lower freezer compartment door 5a that open and close the openings of the ice-making compartment 2, the upper freezer compartment 3, the vegetable compartment 4, and the lower freezer compartment 5, respectively. In addition, the positions of the vegetable compartment 4 and the lower freezer compartment 5 can be changed to be arranged vertically. In addition, the ice-making compartment 2, the upper freezer compartment 3, and the lower freezer compartment 5 are collectively referred to as the freezer compartment 7.

[0042] In order to fix the refrigerating compartment doors la, lb to the refrigerator 100, door hinges (omitted from the drawing) are provided at the upper and lower portions of the refrigerating compartment 1. As shown in Figure 3 , Figure 4 , when the refrigerating compartment doors la, lb are opened and closed, door storage portions 11a (door storage portion a), 11b (door storage portion b) are provided.

[0043] The refrigerating compartment 1 is a refrigerating storage compartment in which the inside of the compartment is set to a refrigerating temperature zone (0°C or higher), for example, an average of about 4°C. The freezer compartment 7 is a freezing storage compartment in which the inside of the compartment is set to a freezing temperature zone (less than 0°C), for example, an average of about -18°C. The vegetable compartment 4 is a refrigerating storage compartment in which the inside of the compartment is set to a refrigerating temperature zone, for example, an average of about 6°C, and is a refrigerating storage compartment in which drying of food is suppressed.

[0044] A camera device 30 for filming the interior of the cold storage 100 is installed approximately at the center of the upper outer part of the storage compartment. Specifically, it is located approximately at the center of the upper part of the cold storage 100's enclosure (insulated enclosure). With the enclosure door closed, the lens 34 of the camera device 30 (see reference...) Figure 5A , Figure 5B The camera is positioned forward of the opening edge 10a of the storage compartment, and preferably forward of the refrigerator doors 1a and 1b. That is, by placing the camera 30 outside the storage compartment, it can take a bird's-eye view of the interior of the storage compartment, including the door storage sections 11a and 11b.

[0045] Additionally, the camera device 30 has a mechanism for sliding back and forth (a mechanism that slides in a direction perpendicular to the opening of the storage compartment), enabling it to move back and forth across the front surface of the door of the cold storage 100. For example, as Figure 2B As shown, since it can be moved to a position further back than the front surface of the door or the edge of the storage compartment opening, the depth dimension is not affected during packaging. Furthermore, the capacity of the refrigerated storage to transport trucks is not reduced, and even after installation, the camera device 30 can be stored away to obscure the shooting range when not in use. This obscuring also caters to user needs who do not wish for the camera's shooting area to always be within their living space.

[0046] A shooting button 39 (a shooting instruction receiving unit that receives shooting instructions from the user) is provided on the refrigerator door 1b for manual shooting using the camera device 30. The shooting button 39 may also be provided on the refrigerator door 1a, the cabinet, or other areas of the door.

[0047] Figure 5A This is a front view of the camera device 30 according to the first embodiment. Figure 5B This is a side view of the camera device 30 according to the first embodiment. Figure 5C This is a side cross-sectional view showing the camera device 30 of the first embodiment. Figure 5A (AA section diagram). Figure 6 This is a perspective view of the camera device 30 according to the first embodiment.

[0048] The camera device 30 consists of a housing 31 and a cover 32. A camera 33 (image-taking unit) and a lens 34 serving as a fisheye lens are located in front of the housing 31. Figure 6 As shown, the lens 34 is surrounded by a light-shielding wall 35. Multiple illumination sections 38 are located near the light-shielding wall 35.

[0049] like Figure 5CAs shown, a control SoC 41 and a wireless LAN unit 42 are housed inside the housing 31. The control SoC 41 (System on a chip) is an integrated circuit designed to function as a system in cooperation with functions such as application-purpose functions in addition to functions possessed by a general microcontroller typified by a processor core on one chip of the integrated circuit. Note that the control SoC 41 is an example of an integrated circuit, and other integrated circuits or the like can be used instead.

[0050] The camera 30 provided outside the storage compartment takes a bird's-eye view of the door storage portions 11a, 11b and the inside of the storage compartment.

[0051] The features of the camera 30 are further described.

[0052] (1) The camera 33 used by the camera 30 can take a wide-angle of 180 degrees or more, and by having a fisheye lens, the entire interior of the refrigerator can be visually recognized in a bird's-eye view when the door is opened.

[0053] (2) The lens 34 for the camera of the camera 30 is located at a position closer to the front surface of the door of the refrigerator than the door storage portions 11a, 11b, and thus the state of the interior of the refrigerator can be captured even when only the vegetable compartment 4 and the lower freezer compartment 5 are opened.

[0054] (3) The camera 30 includes an illuminating portion 38, and by capturing an image using the illuminating portion 38 outside the refrigerator, which can be adjusted to an appropriate illuminance for capturing, an image with high visual recognition can be obtained. The illuminance of the illumination provided inside the refrigerator is relatively high, and thus when the camera is used, the reflection of the contents is strong, and the captured image is likely to be a blown out highlight image. Therefore, in the present embodiment, when capturing an image, the illumination inside the refrigerator is turned off, and the illuminating portion 38 outside the refrigerator is turned on to capture an image. Note that the illumination of the illuminating portion 38 has directivity to illuminate the entire interior of the refrigerator, the door storage portions 11a, 11b, and the second storage compartments (e.g., the vegetable compartment 4 and the lower freezer compartment 5) when the door is opened.

[0055] (4) The camera 30 has a light-shielding wall 35, and thus the direct light of the illuminating portion 38 does not enter the capturing range, and thus an image with high visual recognition with suppressed reflection and blown out highlights can be obtained.

[0056] (5) By providing the light-shielding wall 35 that shields direct light between the illuminating portion 38 and the lens 34 of the camera 30, the position of the illumination can be made as close to the lens 34 as possible, and thus a compact outer shape can be obtained, and further, an effect of preventing damage to the lens 34 can be obtained.

[0057] (6) The imaging device 30 has a mechanism that slides front and back, and is able to move front and back across the front surface of the door of the refrigerator 100, so it does not affect the depth dimension when packaging. In addition, the number of refrigerators loaded relative to a transport truck is not reduced, and even after being set, it is able to be stored when the imaging device 30 is not in use, blocking the range of the shot.

[0058] (7) The imaging device 30 includes a communication unit (wireless LAN unit 42), and image data transmitted from the communication unit is supplied to the server 54 (refer to Figure 7 ) via a wireless router in the home, such as the home wireless router 53 (refer to Figure 7 ), for example. Details will be described later.

[0059] (8) The power supply of the imaging device 30 and the communication circuit that transmits the shooting instruction are connected to the refrigerator main body side. The refrigerator main body receives power supply from a power frequency power supply or the like. The power supply of the imaging device 30 is able to be obtained from the power frequency power supply via the refrigerator main body. In addition, it is also possible to drive the imaging device 30 with a rechargeable battery.

[0060] Figure 7 is a configuration diagram of the imaging system of the refrigerator 100 of the first embodiment. As a sensor on the door (door) side, a door sensor (not shown) that detects the opening and closing state of the refrigerating chamber doors la, lb, the ice making chamber door 2a, the upper freezing chamber door 3a, the vegetable chamber door 4a, and the lower freezing chamber door 5a, respectively, and the like are also provided.

[0061] A main microcomputer 52 (processing device, control device, control section) that loads a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like that are a part of the control device, an interface circuit, and the like is disposed on the upper portion of the refrigerator 100. The main microcomputer 52 performs control of the imaging system that photographs the inside of the refrigerator based on information from the door sensor or the shooting button 39. In addition, the main microcomputer 52 also performs power supply to the imaging device 30.

[0062] The main microcomputer 52, when a predetermined angle or more is detected based on information such as the opening angle of the door from the door sensor at the time of automatic shooting setting, transmits a shooting instruction to the control SoC 41. The control SoC 41 issues a shooting instruction to the camera 33, and acquires unprocessed image data from the camera 33. For example, as shown in Figure 10 , the control SoC 41 performs image processing on the unprocessed image data. On the other hand, the main microcomputer 52, when it is detected that the shooting button 39 is operated at the time of manual shooting setting, transmits a shooting instruction to the control SoC 41. The processing thereafter is the same as at the time of automatic shooting setting.

[0063] As described above, the camera 30 is provided with the wireless LAN unit 42 capable of connecting with the in-house router 53. By providing the wireless LAN unit 42, the in-house image data processed in the control SoC 41 can be transmitted to the server 54 via the in-house router 53. The server 54 transmits the in-house image data in accordance with an in-house image display request from a mobile device or a personal computer, etc. owned by the user, such as a smartphone 55. Thus, the user can grasp the situation in the warehouse.

[0064] Figure 8 is a flowchart showing the process at the time of the manual photographing setting of the first embodiment. Appropriate reference is made to Figure 7 is made. The host microcomputer 52 determines whether the setting from the setting device (not shown) is the camera photographing setting (camera setting is ON) (S31), and determines whether the door is opened using the door sensor (is the door opened?) (S32). When the door is opened (S32, YES), the process proceeds to S33. When the door is not opened (S32, NO), the host microcomputer 52 returns to S32.

[0065] In S33, the host microcomputer 52 determines whether there is an operation (pressing) of the photographing button 39. When there is the pressing of the photographing button 39 (S33, YES), the process proceeds to S34. When there is no pressing of the photographing button 39 (S33, NO), the process returns to S32.

[0066] In S34, the host microcomputer 52 issues a photographing instruction to the control SoC 41, and turns off the in-warehouse lighting and turns on the out-of-warehouse lighting (lighting unit 38) (S35). The control SoC 41 issues a photographing instruction to the camera 33 to perform photographing (S36). Then, the host microcomputer 52 turns on the in-warehouse lighting and turns off the out-of-warehouse lighting (lighting unit 38) (S37).

[0067] The camera 33 transmits the unprocessed image data to the control SoC 41 (S38), and the control SoC 41 performs in-warehouse image correction processing (image processing) (S39). Details of the image processing are described in Figure 10 later.

[0068] The control SoC 41 performs image output processing (image update) to the server 54 via the wireless LAN unit 42 and the in-house router 53 (S40). The server 54 notifies the user's mobile device or the like after the image is updated (S41).

[0069] The above is the process at the time of the manual photographing setting.

[0070] Figure 9is a flowchart showing the process at the time of automatic photographing setting of the first embodiment. Appropriate reference is made to Figure 7 The description will be given. Further, the description of the same process as Figure 8 will be omitted. The host microcomputer 52, in the case where the setting from the setting device (not shown) is the camera automatic photographing setting (camera automatic setting is effective) (S31A), judges whether the door is opened or not (the door is opened?) using the door sensor (S32), and in the case where the door is opened (S32, Yes), proceeds to S33A. On the other hand, in the case where the door is not opened (S32, No), the host microcomputer 52 returns to S32.

[0071] In S33A, the host microcomputer 52 judges whether the opening angle of the door is the prescribed angle or more (for example, the door angle is 85 degrees or more), and in the case where the opening angle of the door is the prescribed angle or more (S33A, Yes), proceeds to S34, and in the case where the opening angle of the door is less than the prescribed angle (S33A, No), returns to S32. S34 to S41 are the same as Figure 13 Further, in S33A, in the case of the cold storage room 1 of the double-door type, in the case where the opening angle of at least either one of the cold storage room doors la, lb is the prescribed angle (S33A, Yes), proceeds to S34.

[0072] The above is the process at the time of automatic photographing setting.

[0073] The features of the photographing system of the cold storage room 100 will be further described.

[0074] (1A) Having a detection section (for example, the host microcomputer 52) which detects the opening angle of the door of the cold storage room, in the case where the detection section judges that the opening angle is the prescribed angle (for example, 85 degrees) or more, the photographing can be automatically performed.

[0075] (2A) In the case where the photographing is performed, the in-room illumination is turned off and the illumination of the camera 30 is turned on, whereby not only the illumination is adjusted to be suitable for the photographing, but also the effect of notifying the user of the timing of the photographing is obtained.

[0076] (3A) Having a detection section (for example, provided to the pull-out section 6) which detects that the pull-out distance of the cold storage room 100 becomes the prescribed value or more, by detecting the operation of the pull-out using the detection section and automatically performing the photographing, the photographing range of the vegetable room and the freezing room can be maintained to be substantially constant.

[0077] (4A) In the case where the door of the cold storage room has a photographing button (for example, the photographing button 39), in the case where the user presses the photographing button, the in-room illumination is turned off and the photographing is performed after the illumination of the camera 30 is turned on, and thus the timing of the photographing can be easily recognized.

[0078] (5A) The captured images are processed for image correction. By using a wide-angle camera to correct distorted images, images that are close to the user's line of sight can be provided.

[0079] (6A) An application service that allows users to view the image via a mobile phone or the like, and the image can be automatically determined by a detection unit installed on the door and the pull-out section to determine which storage room the captured image belongs to.

[0080] As described above, the cold storage 100 of this embodiment includes: a storage compartment with a front opening (e.g., cold storage compartment 1); a door (e.g., cold storage compartment door 1a, 1b) provided on the front surface of the storage compartment and having a rotatable storage portion (e.g., door storage portions 11a, 11b); a door opening angle detection unit for detecting the opening angle of the door; a lighting device that illuminates the storage compartment by detecting the door opening status using the door opening detection unit; a camera device (e.g., camera device 30) provided approximately at the center of the upper part of the outer side of the storage compartment; a fisheye lens (e.g., lens 34) of the camera device having a wide-angle field of view; a lighting device (e.g., lighting unit 38) that illuminates during shooting; and a communication unit (e.g., wireless LAN unit 42) for transmitting the captured image data. The cold storage 100 includes a control device (e.g., a main microcomputer 52) that issues a shooting command to the camera device when a predetermined opening angle is detected by the door opening angle detection unit.

[0081] With the door open, the image captured by the camera device is as follows: Figure 11 As shown, due to the wide-angle fisheye lens, not only can the storage room be included in the shooting range in a single image, but the entire door storage unit can also be included in the shooting range. Moreover, the vegetable compartment (e.g., vegetable compartment 4) and the freezer compartment (e.g., lower freezer compartment 5) of the pull-out storage unit can also be included in the shooting range, eliminating the need for multiple camera devices in each storage compartment as required by Patent Document 1, and eliminating the need for complex and expensive drive devices and drive controls for changing the shooting range.

[0082] Even if the images captured by the camera device remain unchanged, they are sufficient to show the condition of the items stored in the cold storage. However, due to the effect of the fisheye lens, the image is distorted, and the distortion of objects close to the camera device is greater. In addition, since the image is a view from above the cold storage, its positional relationship differs from the view of the cold storage from the front, which is normally observed by users, leaving room for improvement in terms of visual recognition.

[0083] According to the present embodiment, by performing the following image processing (S39), the distorted, curved image captured by the imaging device 30 having the wide-angle field of view of the fisheye lens can be provided as an image of the inside of the storage compartment in a state close to the positional relationship of the field of view observed from the front of the user who normally observes the refrigerator.

[0084] Figure 10 is a flowchart showing the image processing (S39) of the first embodiment. Referring to Figure 11 to Figure 16 The images of each processing are described. Figure 11 is an image of the inside of the storage compartment (as the storage compartment c inside the refrigerating chamber 1, as the door storage compartment a of the door storage compartment 11a, as the door storage compartment b of the door storage compartment 11b) before the image processing. Figure 12 is an image showing the cropping processing of each storage compartment. Figure 13 is an image showing the distortion correction processing of each storage compartment. Figure 14 is an image showing the rotation processing of each storage compartment. Figure 15 is an image showing the keystone correction processing of each image. Figure 16 is an image showing the synthesis processing of each image.

[0085] The main microcomputer 52 inputs the captured image described above (refer to Figure 11 ) (S391), and crops (cropping processing) into a fan shape so as to include the range of each storage compartment (S392). Specifically, as shown in Figure 12 , the main microcomputer 52 performs the processing of cropping the image of the inside of the refrigerating chamber 1, the images of the door storage compartments 11a, 11b, for example, into a fan shape (refer to Figure 12 ).

[0086] Next, the main microcomputer 52 performs the distortion correction processing (distortion correction processing) of the image of each storage compartment which is curved due to the effect of the fisheye lens (S393). Specifically, as shown in Figure 13 , the main microcomputer 52 performs the distortion correction processing of the curved images of the inside of the refrigerating chamber 1, the door storage compartments 11a, 11b, respectively. As this distortion correction processing, various known correction algorithms can be used, and for example, a calibration processing in which parameters are set in advance (for example, before the refrigerating chamber is shipped) can be performed to correct the distortion.

[0087] Next, the main microcomputer 52 performs the image rotation processing which rotates the images of each storage compartment after the distortion correction processing to the orientation of the field of view observed from the front, that is, the observation state of the user (S394). Specifically, as shown in Figure 14 , the main microcomputer 52 arranges the door storage compartment 11a on the left side in the field of view observed from the front, that is, the observation state of the user, and similarly, the door storage compartment 11b on the right side.

[0088] Next, the main microcomputer 52 performs distortion correction processing (trapezoidal correction processing) on ​​the image that is distorted into a trapezoid in the depth direction to make it rectangular (S395). Specifically, as shown in... Figure 15 As shown, the main microcomputer 52 performs trapezoidal correction processing on the images of the interior of the refrigerator compartment 1 and the images of the door storage sections 11a and 11b. For example, by performing processes such as the nearest neighbor method and the Lanczos method as perspective projection transformation and their supplementary processing, the images of the roughly trapezoidal or fan-shaped storage sections can be transformed into roughly rectangular images.

[0089] Next, the main microcomputer 52 performs image compositing processing on the image generated in S395, generating a single image (S396). This image is then displayed on the smartphone 55 (see reference). Figure 7 On the application. The implementation method is shown as an example, and the order of each step can be adjusted as appropriate.

[0090] The image data processing method for a cold storage facility according to the first embodiment is a method for processing image data captured by a camera device having a wide-angle lens that includes multiple food storage areas within its shooting range. The method is characterized by including and performing the following processes: a cropping process (S392), cropping multiple portions from the image data captured by the camera device 30; a compositing process (S396), compositing the images cropped by the cropping process; and an output process (S40), outputting the composited image as a single image data unit. Through the cropping process, it is possible to process the input captured image (…). Figure 11 The image excludes at least a portion of areas that are not food storage areas, such as areas where only the walls of the cold storage or the ground outside the storage area are photographed, without any food being captured. The images of the extracted food storage areas are then composited to obtain the final image. Figure 16 For the overall display screen of a smartphone 55 displaying the image, the proportion of the food storage area displayed can be increased. In this embodiment, the cropping process is performed on one image captured by one camera device 30, but it is also possible to prepare two or more camera devices, capture two or more images, and crop the portion containing the food storage area contained in each image data.

[0091] In this way, the characteristic of this embodiment is that the proportion of the food storage area to the total area of ​​the image data is larger in the synthesized image compared to the image data captured using a wide-angle lens. For example, if comparing... Figure 11 and Figure 16 This makes the area displaying the food larger, making it easier for users to observe.

[0092] The characteristic of this embodiment is that the spacing between the multiple parts cut out by the cutting process is shortened after the synthesis process. For example, if... Figure 12 and Figure 16 If compared, then Figure 16 The distances between the door storage section a, storage compartment c, and door storage section b shown are... Figure 12 The small spacing between the door storage section a, storage compartment c, and door storage section b makes them easy for users to observe. Furthermore, since multiple storage areas that are actually spaced apart are output as a single image, as in Patent Document 1, users do not need to perform swipe or tap operations to switch images, allowing for easy overall observation. In particular, the two door storage sections open by rotating in opposite directions around hinges located at different positions, resulting in a relatively large actual spacing within the cold storage compartment.

[0093] In this embodiment, the refrigerator compartment 1 is described as an example of a double-opening storage compartment, but it can also be applied to the case of a single-opening storage compartment.

[0094] <<Second Implementation>>

[0095] In the first embodiment, image compositing processing for the cold storage compartment 1 (first storage compartment) was described, but the embodiment is not limited thereto. In the second embodiment, image compositing processing including the cold storage compartment 1 (first storage compartment) and the vegetable compartment 4, etc. (second storage compartment) was described.

[0096] Figure 17 This is a side view showing the state in which the pull-out section 6 of the cold storage 100 of the second embodiment is pulled out. Figure 18 This is an image showing the cutting process of the pull-out section 6 of the cold storage in the second embodiment. Figure 19 This is an enlarged image showing the pull-out section 6 of the cold storage unit according to the second embodiment. The camera device 30 for photographing the interior of the cold storage unit can capture images of the pull-out section 6. Therefore, the storage state can be determined based on the image of the pull-out section 6.

[0097] Figure 20 This is an image showing the composite processing of images including the pull-out section 6 in the second embodiment. Figure 20 In, with Figure 16 In comparison, the image of pull-out section 6 (second storage compartment d) is also composited. This allows the user to simultaneously view the storage status of compartments including refrigerator compartment 1 (first storage compartment) and vegetable compartment 4 (second storage compartment). Furthermore, Figure 16 From Figure 12 The image shown is a composite image.

[0098] That is, the refrigeratory 100 is characterized by having at least the first storage compartment of the split type and the second storage compartment opened in the front-rear direction, and the cutting process is performed on the first storage compartment and the second storage compartment. In addition, the refrigeratory 100 can have at least the first storage compartment of the split type and the second storage compartment opened in the front-rear direction, and the cutting process is performed on the inside of the first storage compartment, the door storage part, and the second storage compartment. The cutting process can be performed on the second storage compartment when the opening of the second storage compartment is detected.

[0099] Figure 21 is another example of an image indicating the synthesis process of each image including the pull-out part 6 of the second embodiment. Since it is assumed that the display screen of the smartphone 55, which displays the image, is mostly rectangular, as shown in Figure 21 , it is also possible to move the positions of the door storage parts a and b from the side of the storage compartment c so that the entire synthesized image becomes rectangular. In the example of Figure 21 , the door storage parts a and b are moved to the side opposite to the second storage compartment d with the storage compartment c as the center.

[0100] That is, the cutting process is performed on the at least one door storage part, the inside of the first storage compartment, and the second storage compartment as food storage areas, respectively. In the synthesis process, the at least one door storage part and the second storage compartment are arranged to be located on opposite sides of each other with the inside of the first storage compartment therebetween.

[0101] << Modified example of the camera >>

[0102] In the first embodiment, the case where the camera 30 has a sliding mechanism is described, but it is not limited thereto. In the modified example, a folding mechanism is described.

[0103] Figure 22A is an explanatory diagram indicating the state of the camera 30 at normal times. Figure 22B is an explanatory diagram indicating the state of the camera 30 during storage. Figure 22C is an explanatory diagram indicating the state of the camera 30 during storage.

[0104] The refrigeratory 100 (see Figure 1 ) includes a case that forms a storage compartment with an opening in front, a camera 30 provided outside the storage compartment, and a support part 45 (hinge part) that supports the camera 30 so as to be movable with respect to at least a direction perpendicular to the opening.

[0105] As shown in Figure 22A to Figure 22CAs shown, the front end portion of the camera 30 having the camera 33 is configured to be foldable (two-folded) from the support portion 45, and does not affect the depth dimension at the time of packaging, so that the number of refrigerators loaded relative to a transport truck is not reduced, and even after installation, the camera 30 can be stored and the shooting range can be blocked when the camera 30 is not used.

[0106] In addition, the camera 30 can also revolve (move in a circular trajectory around the hinge portion; revolve around the axis) in the direction of observing the inside of the refrigerator. Since there is a box in the circular trajectory, the angle of revolution when viewed from the direction orthogonal to the revolving plane is less than 360°. At this time, a driving portion that can impart a force in the revolving direction can also be installed on the camera 30, and the angle of viewing the inside of the refrigerator such as the refrigerating chamber at the time of shooting can be controlled. For example, at the time of shooting, the camera 30 can be positioned within the moving trajectory of the door. By adopting such a manner, the inside of the refrigerator can be more effectively shot, and after shooting is completed, it can be restored to outside the moving trajectory of the door, or revolved to a position behind the edge surface of the front opening of the door and the storage chamber and stored, and the arm connecting the support portion 45 (hinge portion) and the camera can be easily lengthened.

[0107] In addition, the present application is not limited to the above-described first and second embodiments, and includes various modifications. For example, the above-described embodiments are described in detail in order to easily understand the present application, and are not limited to necessarily including all the structures described. In addition, for a part of the structure of the above-described embodiments, addition, deletion, and substitution of other structures can be performed. For example, Figure 3 The shooting button 39 of the camera 30 can be provided on the refrigerating chamber door la, or can be provided on the upper surface of the drawer 6. In addition, a plurality of cameras for shooting can be prepared, and image data of the food storage area included in the images shot by the respective cameras can be synthesized into one image.

[0108] According to the refrigerator 100 of the present embodiment, a curved image shot by a camera having a fish-eye lens (wide-angle lens) provided on the outside of the storage chamber is subjected to distortion correction, cutting processing, and synthesis, and one image is output, whereby the image switching operation on the display operation device can be minimized, and a refrigerator interior image close to the observation state of the user can be provided.

[0109] A camera device having a wide-angle lens such as a fish-eye lens provided outside a storage compartment, photographs the inside of the storage compartment including a door storage portion from above, performs distortion correction of a curved image, cut processing of each storage portion, and image synthesis after the processing, and thus provides the state inside the refrigerator with one image in a state close to the user's line of sight, and the recognition of the state inside the refrigerator becomes easy. In addition, since many camera devices or camera devices with variable and complex controls are not required, the cost is advantageous, and problems such as damage to the lens and obstruction of the photographing range due to the storage close to the lens can be solved. The user can also confirm the latest state inside the refrigerator at the destination by the image, and unnecessary shopping and food loss can be reduced.

[0110] BRIEF DESCRIPTION OF DRAWINGS

[0111] 1 refrigerating compartment (1st storage compartment)

[0112] 1a, 1b refrigerating compartment door

[0113] 2 ice making compartment (2nd storage compartment)

[0114] 2a ice making compartment door

[0115] 3 upper freezing compartment (2nd storage compartment)

[0116] 3a upper freezing compartment door

[0117] 4 vegetable compartment (2nd storage compartment)

[0118] 4a vegetable compartment door

[0119] 5 lower freezing compartment (2nd storage compartment)

[0120] 5a lower freezing compartment door

[0121] 6 drawer portion

[0122] 10 case

[0123] 10a opening edge surface

[0124] 11a, 11b door storage portion

[0125] 30 camera device

[0126] 31 housing portion

[0127] 32 cover portion

[0128] 33 camera (image pickup portion)

[0129] 34 lens (wide-angle lens)

[0130] 35 light shielding wall

[0131] 38 illumination portion

[0132] 39 photographing button (photographing instruction accepting section)

[0133] 41 control SoC (output section)

[0134] 42 wireless LAN unit

[0135] 45 support section (hinge section)

[0136] 51 door sensor

[0137] 52 main microcomputer (processing device)

[0138] 53 home router

[0139] 54 server

[0140] 55 smartphone

[0141] 100 refrigerator

Claims

1. A storage library characterized by, Comprise: a storage chamber provided with an opening in front; a rotary door provided with a door housing for opening and closing the opening; and a camera including a lens having a wide-angle field of view, disposed outside the storage chamber, the camera captures image data in a state in which the door is open, the camera and the lens are disposed in such a manner that the storage chamber and the door housing are included in one of the image data.

2. The storage cabinet according to claim 1, wherein: the camera is capable of moving in a revolution direction to the front and rear of the opening.

3. The storage cabinet according to claim 1, wherein: a lighting device is disposed inside the storage chamber, the lighting device is turned off when the camera captures an image.

4. The storage cabinet according to claim 1, wherein: the camera has a lighting portion, the lighting portion is turned on when the camera captures an image.

5. The storage cabinet according to claim 1, wherein: the rotary door and the door housing are disposed on both side surfaces of the storage chamber, the lens is a fisheye lens, the camera and the lens are disposed in such a manner that both side surfaces of the storage chamber and two door housings are included in one of the image data. ​

Images