Image management system, image management server, image management method, and image display control system manufacturing method
The system uses rotating partitions and door sensors to dynamically update image capture and display areas based on door openings, addressing incomplete imaging in refrigerators with two doors, ensuring comprehensive interior representation.
Patent Information
- Application Number
- JP2022025674
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-02-22
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2042-02-22
AI Technical Summary
When a refrigerator with two doors is photographed, if only one door is open, the interior of the closed door is not captured, leading to incomplete image representation.
The system employs rotating partitions and door sensors to dynamically update the image capture and display areas based on the opening and closing of both doors, ensuring complete coverage of the refrigerator interior.
Ensures comprehensive imaging and display of the refrigerator contents regardless of which door is open, providing a complete and coherent view of the interior.
Smart Images

Figure 0007595039000001 
Figure 0007595039000002 
Figure 0007595039000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an image management system, an image management server, an image management method, and a method for producing an image display control system. [Background technology]
[0002] In recent years, a service has been provided that allows users to view the interior of a refrigerator on a mobile terminal by uploading images of the interior of the refrigerator to a server and processing the uploaded images. Patent Document 1 describes an invention that allows many locations to be photographed with a small number of cameras. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2003-42626 A Summary of the Invention [Problem to be solved by the invention]
[0004] In the case of a refrigerator with left and right doors, there are cases where items are taken in and out by opening only the left door or only the right door. In this case, if the condition is set to shoot the interior with both doors open, the interior cannot be photographed if only one door is open and food is taken in and out. If the condition is set to shoot the interior with one door open, the other closed door will not be photographed. [Means for solving the problem]
[0005] In order to solve the above-mentioned problems, the image display control system of the present invention comprises: Equipped with a rotating partition, First door to open and close the storage area and opening and closing the storage area.A storage unit including a second door, a door sensor that detects whether the first door and the second door are open or closed, and an imaging unit; an imaging control means that causes the imaging unit to capture an image in response to detection information from the door sensor; and a storage means that stores an image including the storage area to be displayed on a display unit. When the first door and the second door are open, the image data captured by the imaging unit is updated using a range that is opened by the first door, a range that is opened by the second door, and a range that is opened by the rotating partition, and the second door is closed, First door When the The first door of the storage area for an image to be displayed on the display unit Opened in updating the range using image data captured by the imaging unit; The first door is closed, The second door When the The second door of the storage area for the image to be displayed on the display unit Opened in and an image update means for updating the range using image data captured by the imaging section.
[0006] The image display control server of the present invention comprises: Equipped with a rotating partition, The first door that opens and closes the storage area and Opening and closing the storage area A receiving means for receiving detection information from a door sensor that detects whether the second door is open or closed and image data of the storage area; and a storage means for storing an image including the storage area to be displayed on a display unit. When the first door and the second door are open, the storage area is updated using the range of the first door, the range of the second door, and the range of the rotating partition among the image data of the storage area, and when the second door is closed, First door When the The first door of the storage area for an image to be displayed on the display unit Opened in Range 、 updating said storage area using captured image data; The first door is closed, The second door When the The second door of the storage area for the image to be displayed on the display unit Opened in Range 、 and an image updating means for updating the storage area using photographed image data.
[0007] The image display control method of the present invention includes the steps of: Equipped with a rotating partition, First door to open and close the storage area and opening and closing the storage area. 1. An image display control method for a storage unit including a second door, a door sensor that detects whether the first door and the second door are open or closed, and an imaging unit, the method comprising the steps of: causing the imaging unit to capture an image in response to detection information from the door sensor; and causing a display unit to display an image including the storage area, the steps being performed by one or more calculation units; a step of updating, when the first door and the second door are open, an image to be displayed on the display unit using an area of the image captured by the imaging unit that is opened by the first door, an area of the image captured by the imaging unit that is opened by the second door, and an area of the image captured by the rotating partition; Among the images to be displayed on the display unit, The second door is closed, First door When the The first door of the storage area Opened in a step of updating the range using image data captured by the imaging unit; an image to be displayed on the display unit, The first door is closed, The second door When the The second door of the storage area Opened in The range is updated using image data captured by the imaging unit.
[0008] A method for producing an image display control system according to the present invention includes the steps of: Equipped with a rotating partition, First door to open and close the storage area and opening and closing the storage area. a storage unit including a second door, a door sensor that detects whether the first door and the second door are open or closed, and an imaging unit; an imaging control means that causes the imaging unit to capture an image in response to detection information from the door sensor; and a display control means that displays an image including the storage area on a display unit; When the first door and the second door are open, the image data captured by the imaging unit is updated using a range that is opened by the first door, a range that is opened by the second door, and a range that is opened by the rotating partition, and the second door is closed, First door When the The first door of the storage area for an image to be displayed on the display unit Opened in updating the range using image data captured by the imaging unit; The second door is closed, First door When the The second door of the storage area for the image to be displayed on the display unit Opened in and an image update means for updating the range using image data captured by the imaging unit, the method for producing an image display control system including the step of providing a terminal with a program for executing the display control means.
[0009] Other means will be described in the description of the embodiment of the invention. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a block diagram of a refrigerator according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram of two types of servers. [Diagram 3] FIG. 2 is a block diagram of a mobile terminal. [Figure 4] FIG. 2 is a logical block diagram of the image management system. [Diagram 5] FIG. [Figure 6] FIG. [Figure 7] FIG. 2 is a front view of the refrigerator with the left and right doors of the refrigerator compartment open. [Figure 8] FIG. 2 is a plan view of the refrigerator with the left and right doors of the refrigerator compartment open. [Figure 9] This is a perspective view of the camera unit of the refrigerator when viewed diagonally from below. [Figure 10] FIG. 11 is a sequence diagram of a first image processing process. [Figure 11] FIG. 11 is a detailed sequence diagram of the image synthesis process. [Figure 12] 13 is a flowchart of an image synthesis method according to a door sensor state. [Figure 13] FIG. 13 is a diagram showing an unprocessed image captured by a refrigerator camera. [Figure 14] FIG. 13 is a diagram showing an example of a processed image. [Figure 15] FIG. 13 is a diagram showing an example of a processed image in a state where the right door is closed. [Figure 16] FIG. 13 is a diagram showing an example of a processed image in a state where the left door is closed. [Figure 17] FIG. 13 shows a refrigerator contents screen. [Figure 18] FIG. 13 is a diagram showing an adjustment screen. [Figure 19] FIG. 13 is a diagram showing a preview adjustment screen. [Figure 20] FIG. 13 is a diagram showing a preview adjustment screen during adjustment. [Figure 21] FIG. 13 is a diagram showing a dialog box for discarding adjustment contents. [Figure 22] FIG. 11 is a sequence diagram of an image processing process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the image management system of this embodiment, a refrigerator 1 in Fig. 1, servers 2a and 2b and storage servers 6a and 6b in Fig. 2, and a mobile terminal 3 in Fig. 3 cooperate with each other to provide services to users.
[0012] The configuration of a refrigerator 1 will be described with reference to FIG. The refrigerator 1 has, as its hardware configuration, a camera unit 10, a calculation unit 112, a ROM (Read Only Memory) 113, a RAM (Random Access Memory) 114, a communication unit 115, and a camera LED (Light Emitting Diode) 100. The refrigerator 1 further has a left door 12, a right door 13, a door sensor 117, and a refrigerator compartment 116, all of which are shown in FIG.
[0013] The calculation unit 112 is, for example, a CPU (Central Processing Unit) and controls the refrigerator 1. The RAM 114 serves as a working area for the calculation unit 112. The RAM 114 is also a non-temporary or temporary recording medium that stores various programs and data, and stores a control program (not shown). The calculation unit 112 executes the control program to embody an imaging control means that causes the camera unit 10 to capture an image.
[0014] The camera unit 10 is an imaging unit that captures images of the refrigerator compartment 116, and is, for example, a fisheye camera or a panoramic camera. The camera unit 10 is attached to the upper side of the refrigerator compartment 116. The camera LED 100 is an indicator that lights up when the camera unit 10 is capturing an image. The left door 12 and the right door 13 are a first door and a second door that open and close the refrigerator compartment 116, which is a storage area, and the open and closed states of the doors are detected separately by a door sensor 117. When a user opens the left door 12 and / or the right door 13, the camera unit 10 operates to capture an image of the refrigerator compartment 116. The refrigerator compartment 116 is a storage room that stores items, typically foods.
[0015] The communication unit 115 is connected to a network and transmits and receives data to and from the server 2a, which is an external device.
[0016] The configurations of the servers 2a, 2b and the storage servers 6a, 6b will be described with reference to Fig. 2. The servers 2a, 2b and the storage servers 6a, 6b each include a CPU 21, a ROM 22, a RAM 23, a storage unit 24, and a communication unit 25.
[0017] The CPU 21 controls the servers 2a and 2b. The ROM 22 is a non-volatile memory, and stores, for example, a BIOS (Basic I / O System). The RAM 23 serves as a working area for the CPU 21. The RAM 23 is a non-transient or temporary recording medium that stores various programs and data, and stores a program (not shown).
[0018] The storage unit 24 is a non-temporary or temporary recording medium for storing various programs and data, and is composed of, for example, an HDD (Hard Disk Drive), a flash memory, etc. It is preferable that the storage units 24 of the storage servers 6a and 6b have a large capacity and are accessible at high speed. The communication unit 25 is connected to a network and transmits and receives data to and from external devices.
[0019] The configuration of the mobile terminal 3 will be described with reference to Fig. 3. The mobile terminal 3 includes a CPU 31, a ROM 32, a RAM 33, a storage unit 34, a communication unit 35, and a touch panel display 36.
[0020] The CPU 31 controls the mobile terminal 3. The ROM 32 is a non-volatile memory, and stores, for example, a BIOS. The RAM 33 serves as a working area for the CPU 31. The RAM 33 is a non-transient or temporary recording medium that stores various programs and data, and stores a program (not shown).
[0021] Storage unit 34 is a non-transient or temporary recording medium that stores various programs and data, and is constituted by, for example, a flash memory. That is, a refrigerator management program is installed in storage unit 34. CPU 31 executes the refrigerator management program to embody a display control unit that displays an image including refrigerator compartment 116 on touch panel display 36, and a presetting means that accepts adjustment instructions for the angle, range, and center position of the image as the display mode of the image and sets them in advance.
[0022] The communication unit 35 is connected to a network and transmits and receives data between the communication unit 35 and an external device. The touch panel display 36 is configured by stacking a touch panel on a flat display. The touch panel display 36 is a display unit that displays characters, images, or figures, and also an input unit that detects user operations.
[0023] FIG. 4 is a logical block diagram of the image management system. This image management system includes an imaging control means 81 , a receiving means 82 , an image updating means 83 , a storage means 84 , a display control means 85 , a presetting means 86 , and a display unit 87 .
[0024] The imaging control means 81 causes the camera unit 10 to capture an image in response to the detection information of the door sensor 117 . The receiving means 82 receives detection information from the door sensor 117 that distinguishes between the opening and closing of the left door 12 and the right door 13 that open and close the refrigerator compartment 116, and image data capturing an image of the refrigerator compartment 116. The receiving means 82 is embodied by the CPU 21 of the server 2a executing a program.
[0025] The image update means 83 updates the range of the first door side of refrigerator compartment 116 in the image to be displayed on display unit 87 using image data captured by camera unit 10 in response to the opening of the left door 12, and updates the range of the right door 13 side of refrigerator compartment 116 in the image to be displayed on display unit 87 in response to the opening of the right door 13 using image data captured by camera unit 10. The image update means 83 is embodied by the CPU 21 of server 2b executing a program.
[0026] The storage means 84 stores an image including the refrigerator compartment 116 to be displayed on the display unit 87. The storage means 84 is realized by the CPU 21 of the storage server 6b executing a program.
[0027] The display control means 85 displays an image including the refrigerator compartment 116 on the display unit 87 . The presetting means 86 accepts and presets at least one of an angle, a range and a center position of the image as a display mode for displaying the image including the refrigerator compartment 116. The display control means 85 causes the display unit 87 to display the image in a mode that reflects the adjustment instruction instructed by the presetting means 86. The display control means 85 and the presetting means 86 are embodied by the CPU 31 of the mobile terminal 3 executing a refrigerator management program.
[0028] FIG. 5 is a front view of the refrigerator 1. Refrigerator 1 is an appliance for cooling food and the like, and includes a left door 12 and a right door 13 for refrigerator compartment 116, and a camera unit 10. Refrigerator 1 has multiple storage compartments inside its housing. The storage compartments of refrigerator 1 include, from the top, refrigerator compartment 116, ice-making compartment 14 and upper freezer compartment 15 lined up on the left and right, as well as a vegetable compartment 16 and a lower freezer compartment 17.
[0029] The housing of refrigerator 1 is configured such that a heat insulating material (not shown) such as urethane foam is filled between an outer box made of steel plate and an inner box (not shown) made of resin. A plurality of openings corresponding to each compartment are provided on the front side (front side) of the housing of refrigerator 1. For example, when food or the like is placed through the opening of refrigerator compartment 116, left door 12 and right door 13 are opened. When left door 12 and right door 13 are closed, the opening of refrigerator compartment 116 is blocked. In this way, left door 12 and right door 13 have a function of closing the opening of refrigerator compartment 116. The same applies to each of the other doors.
[0030] Refrigerator 1 is equipped with a left door 12 and a right door 13 as French-style doors which form refrigerating compartment 116 together with the housing. Left door 12 and right door 13 correspond to a first door and a second door which open and close the storage area. Left door 12 is rotatable about the axis of hinge 121 (see FIG. 8) on the left end. The same is true for right door 13. Refrigerator 1 is also equipped with drawer-type doors, such as an ice-making compartment door and an upper freezer compartment door, as well as a vegetable compartment door and a lower freezer compartment door.
[0031] Refrigerating compartment 116 is provided with a plurality of shelves that divides refrigerator compartment 116 into predetermined sections. A plurality of door pockets 123 (see FIG. 7) for storing food and the like are provided on the inside of left door 12. The same is true for left door 12. Ice-making compartment 14 is provided with an ice-making compartment container (not shown) that is pulled out together with the ice-making compartment door. Similarly, upper freezer compartment 15 is provided with an upper freezer container (not shown), vegetable compartment 16 is provided with a vegetable container (not shown), and lower freezer compartment 17 is provided with a lower freezer container (not shown).
[0032] Although not shown, the refrigerator 1 also includes a compressor, a radiator (condenser), a capillary tube (throttle mechanism), and an evaporator. A refrigerant circulates through the compressor, the radiator, the capillary tube, and the evaporator in this order, and the air in the storage compartment is cooled by heat exchange with the refrigerant flowing through the evaporator. The camera unit 10 shown in Fig. 4 is for capturing images of the storage compartment and the like of the refrigerator 1, and is installed on the top surface of the housing.
[0033] FIG. 6 is a side view of the refrigerator 1. As shown in Fig. 6, the camera unit 10 includes a main body 101 and a support 102. The main body 101 has a function of capturing images of a storage compartment, etc. The support 102 supports the main body 101 and is installed on the top surface of the housing of the refrigerator 1.
[0034] A lens 103 (see also FIG. 6) is provided near the front end of main body 101. Lens 103 is an optical element that refracts and focuses light. For example, a fisheye lens is used as such lens 103. When left door 12 and / or right door 13 (see FIG. 5) are opened, lens 103 faces downward so that camera unit 10 can capture an image of refrigerator compartment 116 (see FIG. 5). Camera unit 10 is attached in a position where it may capture an image of the living space including the living space outside refrigerator compartment 116.
[0035] As shown in FIG. 6, lens 103 is located forward of the front end of the housing (opening of the housing). More preferably, lens 103 is located further forward than the front of left door 12 and right door 13 (see FIG. 5) in the closed state. This makes it easier for refrigerator compartment 116 to be within the field of view of lens 103 when left door 12 and right door 13 of refrigerator compartment 116 are opened, for example. It is also possible to capture images of vegetable compartment 16 and the like (see FIG. 5) with a drawer-type door such as the vegetable compartment door open. In this way, lens 103 is provided in a position that allows each storage compartment to be viewed from above.
[0036] FIG. 7 is a front view of refrigerator 1 with left door 12 and right door 13 open. As described above, the inner panel 122 of the left door 12 is provided with a plurality of door pockets 123 for storing food and the like (the same is true for the right door 13). When the left door 12 and the right door 13 are opened, the food and the like in the refrigerator compartment 116 and the door pockets 123, 133 are arranged to be viewed from above within the field of view of the lens 103 of the camera unit 10.
[0037] The rotating partition 124 shown in Fig. 7 is a partition for preventing cool air from leaking from the gap between the left door 12 and the right door 13. In the example of Fig. 7, the rotating partition 124 is provided at the end of the left door 12 opposite to the axis of the hinge 121 (see Fig. 8), and is designed to rotate in a predetermined manner as the left door 12 is opened or closed. The camera unit 10 is disposed within an imaginary extension area in the longitudinal direction of the rotating partition 124 when the left door 12 is in a closed state.
[0038] Furthermore, when manually photographing the refrigerator compartment 116 using the camera unit 10, a photographing button 125 to be pressed by the user is provided on the left door 12, and a photographing button 135 is provided on the right door 13. In the example of FIG. 7, the photographing button 135 is provided on the lower part of the surface of the left door 12 opposite the hinge 121 (see FIG. 8) of the left door 12 (similar to the right door 13). From another perspective, when the left door 12 and the right door 13 are closed (see FIG. 5), the photographing button 125 is provided on the surface where one of the left door 12 and the right door 13 faces the other (near the rotating partition 124 in FIG. 7).
[0039] Shooting performed when at least one of the shooting buttons 125, 135 is pressed by the user is called "manual shooting." Shooting performed by the calculation unit 112 based on the elapsed time from the timing when the left door 12 and the right door 13 start to open is called "automatic shooting."
[0040] FIG. 8 is a plan view of refrigerator 1 with left door 12 and right door 13 open. In the example of Fig. 8, the main body 101 of the camera unit 10 is disposed slightly to the left of the center of the housing of the refrigerator 1 in the left-right direction. To explain in more detail, the main body 101 of the camera unit 10 is disposed directly above the joint between the left door 12 and the right door 13 in the closed state (see also Fig. 5). By disposing the main body 101 in this manner, for example, when the left door 12 and the right door 13 are opened, food etc. in the door pockets 123, 133 are more likely to come into the field of view of the camera unit 10.
[0041] Furthermore, when using refrigerator 1, the user is often conscious of the seam (boundary) between left door 12 and right door 13. Therefore, by locating main body 101 directly above the seam between left door 12 and right door 13, the user is less likely to feel uncomfortable compared to when main body 101 is not positioned at the seam in the left-right direction.
[0042] FIG. 9 is a perspective view of the camera unit 10 as viewed obliquely from below. The main body 101 of the camera unit 10 shown in Fig. 9 includes the lens 103 (see also Fig. 6) described above, as well as a case 31b and a camera LED 100 (Light Emitting Diode: illumination unit). In the example of Fig. 6, the case of the main body 101 is generally a rectangular parallelepiped that is elongated in the front-rear direction. A circular hole (not shown) is provided on the underside near the front end of the case of the main body 101, and the lens 103 is exposed through this hole. The camera LED 100 is provided on the rear side (rear side) of the lens 103.
[0043] The camera LED 100 is a light source that irradiates light onto the refrigerator compartment 116 (see FIG. 7) and door pockets 123, 133 (see FIG. 7) so that the camera unit 10 can photograph the refrigerator compartment 116 and the like under appropriate brightness. The camera LED 100 (illumination unit) is provided on the outside of the housing of the refrigerator 1 (see FIG. 5). Details will be described later, but in this embodiment, the camera unit 10 photographs the refrigerator compartment 116 and the like (see FIG. 7) with the interior light of the refrigerator 1 turned off and the camera LED 100 turned on.
[0044] As shown in Fig. 9, by providing lens 103 in front of camera LED 100, reflected light from refrigeration compartment 116 and the like (see Fig. 5) can easily enter lens 31a when left door 12 and right door 13 are open. Therefore, food stored in refrigeration compartment 116 and the like can be properly photographed. Note that it is preferable that camera LED 100 is positioned forward of the front end (opening of the housing) of the refrigerator 1 housing (see Fig. 6). This makes it easier for light emitted from camera LED 100 to enter refrigeration compartment 116 and the like, which in turn makes it easier to obtain clear photographing results.
[0045] The refrigerator 1 is a storage unit including a refrigerating chamber 116 which is a storage area for storing items, and a camera unit 10 which captures an image of the area containing the items including the refrigerating chamber 116.
[0046] FIG. 10 is a sequence diagram of the image processing according to the first embodiment. When the user opens the door of refrigerator 1, calculation unit 112 detects that the door has been opened by door sensor 117. As a result, calculation unit 112 captures an image of refrigeration compartment 116 in which food is stored by camera unit 10 (step S10). In other words, calculation unit 112 embodies an imaging control means that causes camera unit 10 to capture an image in accordance with the detection information of door sensor 117.
[0047] The calculation unit 112 uploads the captured camera image of the refrigerator compartment 116 to the server 2a (step S11). This camera image is an image before any processing is performed that can be viewed by the user.
[0048] The CPU 21 of the server 2a is an unprocessed image acquisition means for acquiring an unprocessed image captured by the camera unit 10. The server 2a transmits the unprocessed camera image to the storage server 6a (step S12).
[0049] When the storage server 6a notifies the server 2b of the event (step S13), the storage server 6a stores the unprocessed camera image (step S14). The storage server 6a functions as a first storage unit that stores the unprocessed image. Then, when the storage server 6a transmits the unprocessed camera image to the server 2b (step S15), the storage server 6a deletes the unprocessed camera image stored in itself (step S16). That is, the CPU 21 of the storage server 6a functions as a deletion unit that deletes the unprocessed camera image.
[0050] The server 2b corrects the distortion of the camera image before processing due to the lens (step S17), and processes the image into a front view image by keystone correction (step S18). The server 2b then requests the existing processed image from the storage server 6b (step S19), and receives the processed image from the storage server 6b (step S20).
[0051] When the server 2b combines the existing processed image with the keystone corrected image (step S21), it transmits the combined processed image to the storage server 6b to be stored (step S22). The storage server 6b stores the processed image received from the server 2b (step S23). The storage server 6b functions as a storage means for storing an image including the refrigerator compartment 116 to be displayed on the touch panel display 36 of the mobile terminal 3.
[0052] Then, in response to a request from the mobile terminal 3, the storage server 6b transmits the processed image to the mobile terminal 3 (step S24). Upon receiving the processed image from the storage server 6b, the mobile terminal 3 displays the processed image on the touch panel display 36. This allows the customer to view the image of the refrigerator compartment 116 on the mobile terminal 3.
[0053] The mobile terminal 3 does not access the storage server 6a, and therefore does not know the IP address of the storage server 6a. This makes it impossible for the mobile terminal 3 to know how to access the storage server 6a, and therefore prevents the customer or the customer's living environment from being seen by others unintentionally in the unedited image.
[0054] The storage server 6b deletes the edited image after a predetermined period of time has elapsed since the storage server 6b saved the edited image. In other words, the storage server 6a deletes the unedited image that it saved before the storage server 6b deletes the edited image. This prevents the customer who appears in the unedited image and the customer's living environment from being seen by others unintentionally.
[0055] 11 is a detailed sequence diagram of the image synthesis process. Here, the process between the storage server 6a, the server 2b, and the storage server 6b will be described in detail. The server 2b has two functional parts: a door part processing part 26 that processes images of the door part, and an interior processing part 27 that processes images of the inside of the refrigerator compartment 116.
[0056] The storage server 6a transmits the model of the refrigerator 1 and angle information of the left door 12 and the right door 13 to the server 2b (step S50). The door section processing unit 26 cuts out the unprocessed image according to the angle information (step S51), performs synthesis processing (step S52), and generates a processed image. In this synthesis processing, the door section processing unit 26 detects a human body or the like captured in the image and performs mosaic processing or masking processing, and performs mosaic processing or masking processing on the living space other than the inside of the door section. The door section processing unit 26 then transmits the processed image of the door section to the storage server 6b (step S53).
[0057] In parallel with these processes, the interior processing unit 27 determines whether the left door 12 and the right door 13 are open (step S54). If the left door 12 and the right door 13 are open (Yes), the interior processing unit 27 transmits the processed interior image to the storage server 6b (step S55). If either the left door 12 or the right door 13 is not open (No), the interior processing unit 27 cuts out the interior image corresponding to the open door (step S56) and performs a synthesis process (step S57) to generate a processed image. In this synthesis process, the interior processing unit 27 detects a human body or the like that is captured in the image and performs a mosaic process or a masking process, and performs a mosaic process or a masking process on the living space other than the inside of the door part. The interior processing unit 27 then transmits the processed image of the door part to the storage server 6b (step S58).
[0058] When the storage server 6b receives the processed image, it stores the image (step S59).
[0059] FIG. 12 is a flow chart of a method for synthesizing an image according to the state of the door sensor. The interior processing unit 27 receives the captured image of the refrigeration compartment 116 of the refrigerator 1 and the detection information of the door sensor 117 (Step S70). Then, the interior processing unit 27 determines the detection state of the door sensor 117 (Step S71).
[0060] If only the left door 12 is open, the interior processing unit 27 updates the range of the left door side in the processed image (step S72) and proceeds to step S75. That is, the interior processing unit 27 updates the range of the left door 12 side of the refrigerator compartment 116 in the image to be displayed on the mobile terminal 3 using image data captured by the camera unit 10 in response to the opening of the left door 12. The left door 12 is equipped with a rotating partition 124. The interior processing unit 27 may update the range of the left door 12 side of the refrigerator compartment 116 and the range of the rotating partition 124 in the image to be displayed on the mobile terminal 3 by using image data captured by the camera unit 10 in response to the opening of the left door 12.
[0061] If only the right door 13 is open, the interior processing unit 27 updates the range of the right door side in the processed image (step S73) and proceeds to step S75. That is, the interior processing unit 27 updates the range of the right door 13 side in the refrigerator compartment 116 in the image to be displayed on the mobile terminal 3 using image data captured by the camera unit 10 in response to the opening of the right door 13.
[0062] If both the left door 12 and the right door 13 are open, the interior processing unit 27 updates the ranges of the right door 13 side, the left door 12 side, and the rotating partition 124 side in the processed image (step S74), and proceeds to step S75. That is, the interior processing unit 27 updates the ranges of the left door 12 side and the right door 13 side in the refrigerator compartment 116 in the image to be displayed on the mobile terminal 3, using the image data captured by the camera unit 10, in accordance with whether the left door 12 and the right door 13 are open.
[0063] In step S75, the interior processing unit 27 transmits the processed image of the refrigeration compartment 116 of the refrigerator 1 to the storage server 6b to store it, and ends the process in Fig. 12. This allows the interior processing unit 27 to create an appropriate image of the interior when capturing an image of the inside of a refrigerator with two doors and making it available to customers.
[0064] Fig. 13 shows an unprocessed image captured by the camera of refrigerator 1. The lower half of the image shows the refrigerator compartment 116. The left side of the image shows the inside of the right door 13. The right side of the image shows the inside of the left door 12. And the upper half of the image shows the customer's living environment. This part will be unnecessary when viewed from the processed image.
[0065] FIG. 14 is a diagram showing an example of the processed image. Server 2b corrects lens distortion of the unprocessed image taken by camera unit 10 of refrigerator 1 and processes it into a front view image by performing keystone correction. At this time, if server 2b detects a face in the image, it performs mosaic processing on the area in which the face appears.
[0066] The pixelization process performed by this system is a process that reduces the amount of information in at least a part of the area other than the product area. This prevents the customer from being seen by others unintentionally. Note that this face detection process and pixelization process are performed by server 2b, but server 2a or refrigerator 1 may also perform them on the unprocessed image.
[0067] 14 is a processed image cut out from an area capturing an image of the inside of the left door 12. The interior image 72 is a processed image cut out from an area capturing an image of the refrigerator compartment 116. The interior image 72 is divided into an area 721, an area 722, and an area 723. In the fridge interior image 72, areas 721 and 722 are hidden when the left door 12 is closed. Areas 722 and 723 are hidden when the right door 13 is closed. Area 722 is an area corresponding to the rotating partition. The right door inside image 73 is a processed image cut out from an area where the inside of the right door 13 is photographed.
[0068] 15 is a diagram showing an example of an edited image 74 in a state where the right door 13 is closed. The right side of the edited image 74 is blocked by the right door 13, and an area 721 of the refrigerator compartment 116 is shown on the left side. At this time, the server 2b combines the left area 721 with the previous captured image of the refrigerator compartment 116.
[0069] 16 is a diagram showing an example of processed image 75 in a state where left door 12 is closed. The left side of processed image 75 is blocked by left door 12, and an area 723 of refrigerator compartment 116 is shown on the right side. At this time, server 2b combines right-side area 723 with the previous captured image of refrigerator compartment 116.
[0070] The following Fig. 17 to Fig. 21 show the procedure for setting the preview display. These figures show operation screens operated by the user on the touch panel display 36 of the mobile terminal 3. That is, the CPU 31 of the mobile terminal 3 executes the refrigerator management program installed in the mobile terminal 3, thereby displaying these operation screens.
[0071] FIG. 17 is a diagram showing a refrigerator contents screen 51 displayed on the touch panel display 36 of the mobile terminal 3. As shown in FIG. Refrigerator contents screen 51 displays interior image 511 and interior selection button 512. Interior image 511 shows foods and the like stored in refrigeration compartment 116. In other words, CPU 31 executes a refrigerator management program (not shown) to embody a display control means for displaying an image including refrigeration compartment 116 on touch panel display 36.
[0072] The interior selection button 512 selects one of the images of the refrigerator compartment 116, the image of the inside of the left door 12, and the image of the inside of the right door 13. When the inside of the refrigerator image 511 is tapped, the screen transitions to a pre-adjustment screen 52 shown in FIG.
[0073] FIG. 18 is a diagram showing the pre-adjustment screen 52. As shown in FIG. On the pre-adjustment screen 52, an inside-fridge image 521 is displayed, below which a delete button 522, a preview adjustment button 523, and a download button 524 are arranged.
[0074] The delete button 522 is a button for deleting the fridge interior image 521 from the storage server 6b. The preview adjustment button 523 is a button for transitioning to the preview adjustment screen 53 shown in Fig. 19. The download button 524 is a button for downloading the fridge interior image 521 from the storage server 6b.
[0075] FIG. 19 is a diagram showing the preview adjustment screen 53. As shown in FIG. The preview adjustment screen 53 includes a "restore to factory settings" button 532, an interior image 531, a slide 533, a cancel button 534, and an overwrite save button 535. Around the interior image 531, an image is faintly visible through the image.
[0076] The "restore to factory settings" button 532 is a button for returning the adjustment parameters to the factory settings without any adjustments. The "restore to factory settings" button 532 is inactive when no adjustments have been made. The interior image 531 is an image of the interior of the fridge as seen from the front, and is displayed together with a guide for checking horizontal and vertical orientation. The slide 533 is for adjusting the rotation angle of the image. The rotation angle of the image can also be adjusted by the user touching the interior image 531 with two fingers and rotating it.
[0077] The cancel button 534 is a button for returning to the screen before pressing the preview adjustment button 523. The overwrite save button 535 is a button for overwriting the adjustment parameters being set. When the overwrite save button 535 is tapped, the adjustment parameters being set are overwritten, and the screen transitions to the pre-adjustment screen 52, and the previous values cannot be restored.
[0078] FIG. 20 is a diagram showing the preview adjustment screen 53 during adjustment. This preview adjustment screen 53 is obtained by touching the image with two fingers and rotating it. The adjusted rotation angle is 2 degrees, and "2°" is displayed on slide 533. In addition, it is possible to adjust the display range by pinching in, to adjust the display range by pinching out, to adjust the center position by swiping, and the like. When cancel button 534 is tapped after such adjustment, a transition is made to discard dialog 54 shown in FIG. 21. That is, CPU 31 executes a refrigerator management program (not shown) to embody a presetting means for accepting an adjustment instruction for at least any one of the angle, range, and center position of the image and setting it in advance as a display mode for displaying an image including refrigerator compartment 116.
[0079] FIG. 21 is a diagram showing the adjustment content discard dialog 54. As shown in FIG. The cancel button 541 is a button for canceling the discarding of the adjustment contents and returning to the previous preview adjustment screen 53. The OK button 542 is a button for discarding the adjustment contents and returning to the unadjusted preview adjustment screen 53.
[0080] FIG. 22 is a sequence diagram of the image processing process according to the second embodiment. When the user opens the door of the refrigerator 1, the calculation unit 112 detects that the door has been opened by the door sensor 117. This causes the calculation unit 112 to capture an image of the refrigerator compartment 116 in which food is stored by the camera unit 10 (step S30). This camera image is an image before any processing is performed that can be viewed by the user.
[0081] The calculation unit 112 stores the unprocessed camera image (step S31). The calculation unit 112 corrects distortion of the unprocessed camera image caused by the lens (step S32), and processes the unprocessed camera image into a front-view image by keystone correction (step S33). The calculation unit 112 then deletes the unprocessed camera image (step S34). The calculation unit 112 requests the existing processed image from the storage server 6b (step S35), and receives the processed image from the storage server 6b (step S36).
[0082] When the calculation unit 112 combines the existing processed image and the keystone corrected image (step S37), it transmits the combined processed image to the server 2b (step S38). The server 2b transmits the processed image to the storage server 6b to store it (step S39). The storage server 6b stores the processed image received from the server 2b (step S40). Then, the storage server 6b transmits the processed image to the mobile terminal 3 in response to a request from the mobile terminal 3 (step S41). This allows the customer to view the image of the refrigerator compartment 116 on the mobile terminal 3.
[0083] The storage server 6b deletes the edited image after a predetermined period of time has elapsed since the storage server 6b saved the edited image. In other words, the storage server 6a deletes the unedited image that it saved before the storage server 6b deletes the edited image. This prevents the customer who appears in the unedited image and the customer's living environment from being seen by others unintentionally.
[0084] (Modification) The present invention is not limited to the above-described embodiment, and includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. It is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.
[0085] The above configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware such as an integrated circuit. The above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or a recording medium such as a flash memory card or a DVD (Digital Versatile Disk).
[0086] In each embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it may be considered that almost all components are connected to each other. As modified examples of the present invention, for example, the following (a) to (c) are available.
[0087] (a) The configuration of the server and storage server is arbitrary and is not limited to the configurations shown in the embodiments. (b) The present invention is not limited to viewing the refrigeration compartment of a refrigerator, but may be applied to viewing the storage area of any storage cabinet. (c) The image of the left area and the image of the right area of the refrigerator compartment 116 do not necessarily have to be synthesized by a server or a refrigerator. For example, a mobile terminal may synthesize both images. [Explanation of symbols]
[0088] 1. Refrigerator 100 Camera LED 10 Camera unit 112 Arithmetic section 113 ROM 114 RAM 115 Communications Department 116 Refrigerator 117 Door Sensor 12 Left door 124 Rotating partition 13 Right door 14 Ice Room 15 Upper freezer 16 Vegetable compartment 17 Lower freezer 2a Server 2b Server 21 CPU 22 ROM 23 RAM 24 Memory section 25 Communications Department 26 Door processing section 27 Internal Processing Section 3. Mobile devices 31 CPU 32 ROM 33 RAM 34 Storage section 35 Communications Department 36 Touch Panel Display 51 Fridge contents screen 511 Interior image 512 Indoor selection button 52 Screen before adjustment 521 Interior image 522 Delete button 523 Preview adjustment button 524 Download button 53 Preview adjustment screen 531 Interior image 532 "Factory Reset" button 533 Slides 534 Cancel button 535 Save overwrite button 54 Discard Dialog 541 Cancel button 542 OK button 6a Storage Server 6b Storage Server 71 Left door inside image 72 Interior image 721 area 722 areas 723 areas 73 Right door inside image 81 Imaging control means 82 Receiving means 83 Image update method 84 Memory means 85 Display control means 86 Pre-setting measures 87 Display section
Claims
1. A first door having a rotating partition for opening and closing a storage area; A second door for opening and closing the storage area; a door sensor that distinguishes between opening and closing of the first door and the second door; A storage unit including an imaging unit; an image capturing control means for causing the image capturing unit to capture an image in response to detection information from the door sensor; a storage means for storing an image including the storage area to be displayed on a display unit; an image updating means for updating, when the first door and the second door are open, a range opened by the first door, a range opened by the second door, and a range opened by the rotating partition among the image data captured by the imaging unit, updating, when the second door is closed and the first door is open, a range opened by the first door among the storage area of the image to be displayed on the display unit, using the image data captured by the imaging unit, and updating, when the first door is closed and the second door is open, a range opened by the second door among the storage area of the image to be displayed on the display unit, using the image data captured by the imaging unit; An image display control system comprising:
2. The imaging unit further includes an illumination unit provided on the outside of the housing.
2. The image display control system according to claim 1.
3. The imaging unit is disposed within a virtual extension area in the longitudinal direction of the rotating partition.
3. The image display control system according to claim 1, wherein the image display control system includes: a display unit that displays an image on the display unit;
4. A display control means for displaying an image including the storage area on a display unit; a preset means for receiving an adjustment instruction for any one of an angle, a range, and a center position of the image and presetting the angle, a range, and a center position of the image as a display mode for displaying the image including the storage area; The display control means causes the display unit to display an image in a manner that reflects the adjustment instruction instructed by the presetting means.
2. The image display control system according to claim 1.
5. A receiving means for receiving detection information from a door sensor that is equipped with a rotating partition and that detects the opening and closing of a first door that opens and closes a storage area and a second door that opens and closes said storage area, and image data of said storage area; a storage means for storing an image including the storage area to be displayed on a display unit; an image updating means for updating, when the first door and the second door are open, the range opened by the first door, the range opened by the second door, and the range opened by the rotating partition among the image data of the photographed storage area, when the second door is closed and the first door is open, updating the range opened by the first door among the storage area of the image to be displayed on the display unit, using the image data of the photographed storage area, and updating, when the first door is closed and the second door is open, the range opened by the second door among the storage area of the image to be displayed on the display unit, using the image data of the photographed storage area; An image display control server comprising:
6. A first door having a rotating partition for opening and closing a storage area; A second door for opening and closing the storage area; a door sensor that distinguishes between opening and closing of the first door and the second door; An image display control method for a storage unit including an imaging unit, One or more computing units, a step of causing the imaging unit to capture an image in response to detection information from the door sensor; A step of displaying an image including the storage area on a display unit; a step of updating, when the first door and the second door are open, an image to be displayed on the display unit using an area of the image captured by the imaging unit that is opened by the first door, an area of the image captured by the imaging unit that is opened by the second door, and an area of the image captured by the rotating partition; a step of updating, when the second door is closed and the first door is open, a range of the storage area that is opened by the first door, among the images to be displayed on the display unit, using image data captured by the imaging unit; a step of updating, among the images to be displayed on the display unit, a range of the storage area that is opened by the second door when the first door is closed and the second door is open, using image data captured by the imaging unit; 2. An image display control method comprising:
7. A first door having a rotating partition for opening and closing a storage area; A second door for opening and closing the storage area; a door sensor that distinguishes between opening and closing of the first door and the second door; A storage unit including an imaging unit; an image capturing control means for causing the image capturing unit to capture an image in response to detection information from the door sensor; A display control means for displaying an image including the storage area on a display unit; an image updating means for updating, when the first door and the second door are open, a range opened by the first door, a range opened by the second door, and a range opened by the rotating partition among the image data captured by the imaging unit, updating, when the second door is closed and the first door is open, a range opened by the first door among the storage area of the image to be displayed on the display unit, using the image data captured by the imaging unit, and updating, when the second door is closed and the first door is open, a range opened by the second door among the storage area of the image to be displayed on the display unit, using the image data captured by the imaging unit; A method for producing an image display control system comprising: A method for producing an image display control system, comprising the step of providing a terminal with a program for executing said display control means.
Citation Information
Patent Citations
Refrigerator
JP2003042626A
Refrigerator and refrigerator system
JP2014196846A
Refrigerator, image management system in refrigerator and program
JP2017106671A
Refrigerator, camera device, refrigerator door pocket, communication terminal, home appliance network system, and interior image display program
JP2018173272A
Refrigerator and control method for the same
US20160123657A1
