Image management system, image management system production method, and image management method

Abstract

To conceal a customer and dwelling environment of the customer, when an image obtained by photographing inside a refrigerator of the customer is browsed.SOLUTION: An image management system includes a camera unit 10 for photographing an image where a storage object and dwelling environment of a user are imaged, a storage server for storing the image, and a portable terminal, and provides browsing service of the image photographed by the camera unit 10, out of the images stored in the storage server, to a user registered in association with an individual number of the storage object or the camera unit 10. The image management system can exhibit functions of registration means for registering the user in association with the individual number, registered user authority deletion means capable of deleting browsing authority of the registered user associated with the individual number, according to an instruction of the user registered associated with the individual number, and display means of displaying a warning screen including that the registered user can browse an image corresponding to the individual number, on a portable terminal of the user which is newly associated with the individual number by the registration means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image management system, a method for producing an image management system, and an image management method. [Background technology]

[0002] In recent years, a service has been provided that allows users to view the interior of a refrigerator on a mobile terminal by uploading images of the interior of the refrigerator to a server and processing the uploaded images. Patent Document 1 describes an invention that makes it possible to capture images of many locations using a small number of cameras. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-42626 Summary of the Invention [Problem to be solved by the invention]

[0004] Images taken of the inside of a customer's refrigerator must be handled with care from the perspective of protecting personal information, as there is a risk that the customer or their living environment may be unintentionally seen by others. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the image management system of the present invention includes a storage unit, an imaging unit that is attached to the storage unit and captures images that show the storage unit and the user's living environment, a storage server that stores the images captured by the imaging unit, and a mobile terminal, and provides a viewing service for images captured by the imaging unit corresponding to the individual number of the storage unit or the imaging unit to a user that has been registered in association with the individual number of the storage unit or the imaging unit, among images stored in the storage server, and the image management system comprises: registration means that registers a user in association with the individual number of the storage unit or the imaging unit in response to an instruction from the user via the mobile terminal; registered user authority deletion means that can delete the viewing authority of a registered user that is associated with the individual number in response to an instruction from the user that has been registered in association with the individual number via the mobile terminal; and display means that displays a warning screen on the mobile terminal of a user that has been newly associated with the individual number by the registration means, the warning screen including a message that the registered user can view images corresponding to the individual number. characterized in that it has . The image management system of the present invention includes a storage unit, an imaging unit that is attached to the storage unit and captures images that show the storage unit and the user's living environment, a storage server that stores images captured by the imaging unit, and a mobile terminal, and provides a viewing service for images captured by the imaging unit corresponding to the individual number of the storage unit or the imaging unit to a user that has been registered in association with the individual number of the storage unit or the imaging unit, among images stored in the storage server, and includes a registration means that registers a user in association with the individual number of the storage unit or the imaging unit in response to an instruction from the user via the mobile terminal, a registered user authority deletion means that can delete the viewing authority of a registered user that is associated with the individual number in response to an instruction from the user that has been registered in association with the individual number via the mobile terminal, and a display means that displays a warning screen on the mobile terminal of a user that has been newly associated with the individual number by the registration means, the warning screen including a message indicating that the registered user can be deleted. characterized in that it has .

[0006] The method for producing an image management system of the present invention includes a storage unit, an imaging unit that captures images of the storage unit and a user's living environment when attached to the storage unit, a storage server that stores the images captured by the imaging unit, and a mobile terminal, and provides a viewing service for images captured by the imaging unit corresponding to an individual number of the storage unit or the imaging unit from among the images stored in the storage server to a user who is registered in association with the individual number of the storage unit or the imaging unit, a procedure for transmitting an instruction via the mobile terminal of the user in order to register the user in association with the individual number of the storage unit or the imaging unit; and a procedure for transmitting an instruction via the mobile terminal of the user registered in association with the individual number in order to enable deletion of the viewing authority of the registered user in association with the individual number. A warning screen including a message that the registered user can view the image corresponding to the individual number and / or a message that the registered user can be deleted is displayed on the mobile terminal of the user who is newly linked to the individual number. procedure , and causing the mobile terminal to execute and installing a management program for the mobile terminal.

[0007] The image management method of the present invention includes a storage unit, an imaging unit that captures images of the storage unit and a user's living environment when attached to the storage unit, a storage server that stores the images captured by the imaging unit, and a mobile terminal, and provides a viewing service for images captured by the imaging unit corresponding to the individual number of the storage unit or the imaging unit among the images stored in the storage server to a user who is registered in association with the individual number of the storage unit or the imaging unit, by transmitting the image to the mobile terminal. but A registration for registering a user in association with the individual number of the storage or the imaging unit. process and a registered user authority deletion function that can delete the viewing authority of the registered user linked to the individual number. process and a display for displaying a warning screen including a message that the registered user can view the image corresponding to the individual number and / or a message that the registered user can be deleted. Execute the process.

[0008] Other means will be described in the detailed description of the invention. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram of a refrigerator according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram of two types of servers. [Figure 3] FIG. 2 is a block diagram of a mobile terminal. [Figure 4] FIG. 1 is a logical block diagram of an image management system. [Figure 5] FIG. [Figure 6] FIG. [Figure 7] FIG. 2 is a front view of the refrigerator with the left and right doors of the refrigerator compartment open. [Figure 8] FIG. 2 is a plan view of the refrigerator with the left and right doors of the refrigerator compartment open. [Figure 9] This is a perspective view of the camera unit of the refrigerator when viewed from diagonally below. [Figure 10] FIG. 10 is a sequence diagram of a first image processing process. [Figure 11] FIG. 10 is a diagram showing an unprocessed image taken by a camera in a refrigerator. [Figure 12] FIG. 10 is a diagram showing an unprocessed image in which unnecessary portions are masked. [Figure 13] FIG. 10 is a diagram showing an example of a processed image. [Figure 14] FIG. 10 is a diagram showing a selection screen displayed on the mobile terminal. [Figure 15] FIG. 10 is a diagram showing a warning screen. [Figure 16] FIG. 10 is a diagram showing a registration continuation user selection screen. [Figure 17] FIG. 10 illustrates a confirmation dialog box. [Figure 18] FIG. 10 is a diagram showing a transfer warning screen. [Figure 19] FIG. 10 is a diagram showing a transfer warning screen. [Figure 20] FIG. 10 is a diagram showing an image utilization warning screen. [Figure 21] FIG. 10 is a diagram showing a home screen. [Figure 22] FIG. 10 is a diagram showing a refrigerator contents screen. [Figure 23] FIG. 10 is a sequence diagram of an image processing process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In this embodiment, the image management system provides services to users through cooperation between refrigerator 1 (Fig. 1), servers 2a and 2b and storage servers 6a and 6b (Fig. 2), and mobile terminal 3 (Fig. 3). Different IP addresses are assigned to refrigerator 1, servers 2a and 2b, storage servers 6a and 6b, and mobile terminal 3.

[0011] The configuration of a refrigerator 1 will be described with reference to FIG. The hardware configuration of refrigerator 1 includes camera unit 10, calculation unit 112, ROM (Read Only Memory) 113, RAM (Random Access Memory) 114, communication unit 115, and camera LED (Light Emitting Diode) 100. Refrigerator 1 further includes left door 12, right door 13, door sensor 117, and refrigerator compartment 116 shown in FIG.

[0012] The calculation unit 112 is, for example, a CPU (Central Processing Unit) and controls the refrigerator 1. The RAM 114 serves as a work area for the calculation unit 112. The RAM 114 is also a non-temporary or temporary recording medium that stores various programs and data, and stores programs (not shown).

[0013] Camera unit 10 is an imaging unit that captures images of refrigerator compartment 116, and is, for example, a fisheye camera or a panoramic camera. Camera unit 10 is attached to the upper side of refrigerator compartment 116. Camera LED 100 is an indicator that lights up when camera unit 10 is capturing an image. Left door 12 and right door 13 are doors provided in refrigerator compartment 116, and their open / closed states are detected by door sensor 117. When a user opens left door 12 and / or right door 13, camera unit 10 operates to capture an image of refrigerator compartment 116. Refrigerator compartment 116 is a storage compartment for storing items, such as food.

[0014] The communication unit 115 is connected to a network and transmits and receives data to and from the server 2a, which is an external device.

[0015] The configurations of the servers 2a and 2b and the storage servers 6a and 6b will be described with reference to Fig. 2. The servers 2a and 2b and the storage servers 6a and 6b each include a CPU 21, a ROM 22, a RAM 23, a storage unit 24, and a communication unit 25.

[0016] The CPU 21 controls the servers 2a and 2b. The ROM 22 is a non-volatile memory and stores, for example, a BIOS (Basic I / O System). The RAM 23 serves as a work area for the CPU 21. The RAM 23 is a non-transitory or temporary recording medium that stores various programs and data, and stores programs (not shown).

[0017] The storage unit 24 is a non-transitory or temporary recording medium for storing various programs and data, and is configured, for example, with an HDD (Hard Disk Drive), flash memory, etc. It is desirable that the storage unit 24 of the storage servers 6a, 6b has a large capacity and is accessible at high speed. The communication unit 25 is connected to a network and transmits and receives data to and from external devices.

[0018] The configuration of the mobile terminal 3 will be described with reference to Fig. 3. The mobile terminal 3 includes a CPU 31, a ROM 32, a RAM 33, a storage unit 34, a communication unit 35, and a touch panel display 36.

[0019] The CPU 31 performs overall control of the mobile terminal 3. The ROM 32 is a non-volatile memory and stores, for example, a BIOS. The RAM 33 serves as a work area for the CPU 31. The RAM 33 is also a non-transitory or temporary recording medium that stores various programs and data, and stores a program (not shown).

[0020] The storage unit 34 is a non-transitory or temporary recording medium that stores various programs and data, and is composed of, for example, a flash memory. The communication unit 35 connects to a network and transmits and receives data to and from external devices. The touch panel display 36 is composed of a touch panel layered on a flat display. The touch panel display 36 is a display means that displays characters, images, or figures, and also an input means that detects user operations.

[0021] FIG. 4 is a logical block diagram of the image management system. The unprocessed image acquisition means 81 acquires an unprocessed image captured by the camera unit 10. The unprocessed image acquisition means 81 is realized by the CPU 21 of the server 2a executing a program. The first storage means 82 stores the unprocessed image and corresponds to the storage server 6a.

[0022] The processed image creation means 84 creates a processed image by performing processing on the pre-processing image to delete or reduce the amount of information on at least a portion of the area other than the item area. This processed image creation means 84 creates a processed image by deleting or reducing the amount of information on the pre-processing image, including a predetermined area. Then, if a person's body part is captured in any of the pre-processing images, the processed image creation means 84 creates a processed image by deleting or reducing the amount of information on the person's body part. This processed image creation means 84 is embodied by the CPU 21 of the server 2b executing a program.

[0023] The deletion means 83 deletes the before-processing image after creating this after-processing image. The deletion means 83 deletes the before-processing image at least before deleting the after-processing image. This is realized by the CPU 21 of the server 2a executing a program.

[0024] The processed image is stored in the second storage means 85. This second storage means 85 corresponds to the storage server 6a. The registration means 86 registers users who can view the processed images by linking them to the individual number of the storage or imaging unit. When registering a new user to be linked to an individual number, the registered user information presentation means 87 presents information about already registered users to this user.

[0025] When registering a new user to be linked to this individual number, the registered user authority deletion means 88 allows the user to select an already registered user and deletes the viewing authority of the selected registered user. The registration means 86, the registered user information presentation means 87, and the registered user authority deletion means 88 are realized by the CPU 31 of the mobile terminal 3 executing a program.

[0026] FIG. 5 is a front view of the refrigerator 1. Refrigerator 1 is an appliance for cooling food and the like, and includes left and right doors 12 and 13 for refrigerator compartment 116, and camera unit 10. Refrigerator 1 has multiple storage compartments inside its housing. The storage compartments of refrigerator 1 are, from top to bottom, refrigerator compartment 116, ice-making compartment 14 and upper freezer compartment 15 lined up on the left and right, as well as vegetable compartment 16 and lower freezer compartment 17.

[0027] The housing of refrigerator 1 is configured such that an outer box made of steel plate and an inner box (not shown) made of resin are separated by a heat insulating material (not shown) such as urethane foam. The front side (front face) of the housing of refrigerator 1 is provided with a plurality of openings corresponding to the respective compartments. For example, when food or the like is placed through the opening of refrigerator compartment 116, left door 12 and right door 13 are opened. When left door 12 and right door 13 are closed, the opening of refrigerator compartment 116 is blocked. In this way, left door 12 and right door 13 have the function of closing the opening of refrigerator compartment 116. The same applies to the other doors.

[0028] Refrigerator 1 is equipped with left door 12 and right door 13 as French doors that, together with the housing, form refrigerating compartment 116. Left door 12 is rotatable around the axis of hinge 121 (see FIG. 8) at the left end. The same is true for right door 13. Refrigerator 1 is also equipped with drawer-type doors, such as an ice compartment door and an upper freezer compartment door, as well as a vegetable compartment door and a lower freezer compartment door.

[0029] Refrigerating compartment 116 is provided with a plurality of shelves that divide it into predetermined sections. The interior of left door 12 is provided with a plurality of door pockets 123 (see FIG. 7) for storing food and the like. The same is true for left door 12. Ice making compartment 14 is provided with an ice making compartment container (not shown) that is pulled out together with the ice making compartment door. Similarly, upper freezer compartment 15 is provided with an upper freezer container (not shown), vegetable compartment 16 is provided with a vegetable container (not shown), and lower freezer compartment 17 is provided with a lower freezer container (not shown).

[0030] Although not shown, refrigerator 1 also includes a compressor, a radiator (condenser), a capillary tube (throttle mechanism), and an evaporator. Refrigerant circulates through the compressor, radiator, capillary tube, and evaporator in this order, and the air in the storage compartment is cooled by heat exchange with the refrigerant flowing through the evaporator. Camera unit 10 shown in Fig. 5 is used to capture images of the storage compartment and other areas of refrigerator 1, and is installed on the top surface of the housing.

[0031] FIG. 6 is a side view of the refrigerator 1. As shown in Fig. 6, the camera unit 10 includes a main body 101 and a support 102. The main body 101 has a function of capturing images of a storage compartment, etc. The support 102 supports the main body 101 and is installed on the top surface of the housing of the refrigerator 1.

[0032] A lens 103 (see also FIG. 6) is provided near the front end of main body 101. Lens 103 is an optical element that refracts and focuses light. For example, a fisheye lens is used as such lens 103. Lens 103 faces downward so that camera unit 10 can capture an image of refrigerator compartment 116 (see FIG. 7) when left door 12 and / or right door 13 (see FIG. 5) are opened. Camera unit 10 is attached in a position where it may capture an image of the living space range, including the living space outside refrigerator compartment 116.

[0033] As shown in FIG. 6, lens 103 is located forward of the front end of the housing (the opening of the housing). More preferably, lens 103 is located further forward than the front surfaces of left door 12 and right door 13 (see FIG. 5) in the closed state. This makes it easier for refrigerator compartment 116 to be within the field of view of lens 103, for example, when left door 12 and right door 13 of refrigerator compartment 116 are opened. Note that it is also possible for camera unit 10 to capture an image of vegetable compartment 16 or the like (see FIG. 5) when a drawer-type door such as a vegetable compartment door is open. In this way, lens 103 is provided in a position that allows for a view down on each storage compartment from above.

[0034] FIG. 7 is a front view of the refrigerator 1 with the left door 12 and the right door 13 open. As described above, the inner panel 122 of the left door 12 is provided with a plurality of door pockets 123 for storing food and the like (the same is true for the right door 13). When the left door 12 and the right door 13 are opened, the food and the like in the refrigerator compartment 116 and the door pockets 123, 133 are viewed from above within the field of view of the lens 103 of the camera unit 10.

[0035] The rotating partition 124 shown in Fig. 7 is a partition for preventing cool air from leaking from the gap between the left door 12 and the right door 13. In the example of Fig. 7, the rotating partition 124 is provided at the end of the left door 12 opposite to the axis of the hinge 121 (see Fig. 8), and is designed to rotate in a predetermined manner as the left door 12 is opened or closed.

[0036] Furthermore, when manually photographing refrigerator compartment 116 using camera unit 10, photographing button 125 to be pressed by the user is provided on left door 12, and photographing button 135 is provided on right door 13. In the example of FIG. 7, photographing button 135 is provided on the lower part of the surface of left door 12 opposite hinge 121 (see FIG. 8) of left door 12 (similar to right door 13). From another perspective, when left door 12 and right door 13 are closed (see FIG. 5), photographing button 125 is provided on the surface of one of left door 12 and right door 13 facing the other (near rotating partition 124 in FIG. 7).

[0037] Shooting performed when at least one of the shooting buttons 125, 135 is pressed by the user is called "manual shooting." Shooting performed by the calculation unit 112 based on the elapsed time from the timing when the left door 12 and the right door 13 start to open is called "automatic shooting."

[0038] FIG. 8 is a plan view of the refrigerator 1 with the left door 12 and the right door 13 open. In the example of Fig. 8, the main body 101 of the camera unit 10 is disposed slightly to the left of the center of the housing of the refrigerator 1 in the left-right direction. To explain in more detail, the main body 101 of the camera unit 10 is disposed directly above the joint between the left door 12 and the right door 13 when they are closed (see also Figs. 5 and 6). By disposing the main body 101 in this manner, for example, when the left door 12 and the right door 13 are opened, food and the like in the door pockets 123, 133 (see Fig. 7) are more likely to come into the field of view of the camera unit 10.

[0039] Furthermore, when using refrigerator 1, the user is often conscious of the seam (boundary) between left door 12 and right door 13. Therefore, by locating main body 101 directly above the seam between left door 12 and right door 13, the user is less likely to feel uncomfortable compared to when main body 101 is positioned off the seam in the left-right direction.

[0040] FIG. 8 is a perspective view of the camera unit 10 when viewed from diagonally below. The main body 101 of the camera unit 10 shown in FIG. 8 includes the lens 103 (see also FIG. 6) described above, as well as a case 31b and a camera LED 100 (Light Emitting Diode: illumination unit). In the example of FIG. 6, the case of the main body 101 is generally a rectangular parallelepiped that is elongated in the front-to-rear direction. A circular hole (not shown) is provided on the underside near the front end of the case of the main body 101, and the lens 103 is exposed through this hole. The camera LED 100 is provided on the rear side (rear side) of the lens 103.

[0041] Camera LED 100 is a light source that irradiates refrigerator compartment 116 (see FIG. 7) and door pockets 123, 133 (see FIG. 7) with light so that camera unit 10 can photograph them under appropriate brightness. Camera LED 100 (illumination unit) is provided on the outside of the housing of refrigerator 1 (see FIG. 6). Details will be described later, but in this embodiment, camera unit 10 photographs refrigerator compartment 116 (see FIG. 7) with the interior light of refrigerator 1 turned off and camera LED 100 turned on.

[0042] As shown in FIG. 9, by providing lens 103 in front of camera LED 100, reflected light from refrigeration compartment 116 and the like (see FIG. 5) can easily enter lens 31a when left door 12 and right door 13 are open. This makes it possible to properly photograph food stored in refrigeration compartment 116 and the like. It is preferable that camera LED 100 be positioned forward of the front end (opening of the housing) of refrigerator 1 (see FIG. 6). This makes it easier for light emitted from camera LED 100 to enter refrigeration compartment 116 and the like, which in turn makes it easier to obtain clear photographic results.

[0043] The refrigerator 1 is a storage cabinet that includes a refrigerating compartment 116, which is a storage area for storing items, and a camera unit 10 that captures an image of the area containing the items including the refrigerating compartment 116.

[0044] FIG. 10 is a sequence diagram of the image processing according to the first embodiment. When a user opens the door of refrigerator 1, calculation unit 112 detects that the door has been opened using door sensor 117. This causes calculation unit 112 to capture an image of refrigeration compartment 116, where food is stored, using camera unit 10 (step S10). Calculation unit 112 then uploads the captured camera image of refrigeration compartment 116 to server 2a (step S11). This camera image is an unprocessed image that can be viewed by the user.

[0045] The CPU 21 of the server 2a functions as an unprocessed image acquisition unit that acquires an unprocessed image captured by the camera unit 10. The server 2a transmits this unprocessed camera image to the storage server 6a (step S12).

[0046] When the storage server 6a notifies the server 2b of the event (step S13), the storage server 6a stores the unprocessed camera image (step S14). The storage server 6a functions as a first storage means for storing the unprocessed image. Then, when the storage server 6a transmits the unprocessed camera image to the server 2b (step S15), it deletes the unprocessed camera image stored in itself (step S16). That is, the CPU 21 of the storage server 6a functions as a deletion unit that deletes the unprocessed camera image.

[0047] The server 2b corrects distortion caused by the lens in the unprocessed camera image (step S17), processes the image into a front-view image by keystone correction (step S18), and then requests the existing processed image from the storage server 6b (step S19) and receives the processed image from the storage server 6b (step S20).

[0048] Server 2b combines the existing processed image with the keystone corrected image (step S21), and then transmits the combined processed image to storage server 6b for storage (step S22). CPU 21 of server 2b functions as processed image creation means that performs processing on the unprocessed image to delete or reduce the amount of information on at least a portion of the area other than the item area, and creates a processed image.

[0049] The storage server 6b stores the processed image received from the server 2b (step S23). The storage server 6b functions as a second storage means for storing the processed image. Then, in response to a request from the mobile terminal 3, the storage server 6b transmits the processed image to the mobile terminal 3 (step S24). This allows the customer to view the image of the refrigerator compartment 116 on the mobile terminal 3.

[0050] The storage server 6a stores the unprocessed image, and the storage server 6b stores the synthesized processed image, making it easier to manage each image. The mobile terminal 3 can easily obtain the synthesized processed image by simply accessing the storage server 6b.

[0051] The mobile terminal 3 does not access the storage server 6a, and therefore does not know the IP address of the storage server 6a. This prevents the mobile terminal 3 from knowing how to access the storage server 6a, and therefore prevents customers who appear in unedited images or their living environment from being unintentionally seen by others.

[0052] The storage server 6b deletes the edited image after a predetermined period of time has elapsed since storing the edited image. In other words, the storage server 6a deletes the unedited image that it stored before the storage server 6b deletes the edited image. This prevents customers who appear in the unedited image and their living environment from being unintentionally seen by others.

[0053] FIG. 11 shows an unprocessed image taken by the camera unit 10 of the refrigerator 1. The lower half of the image shows the refrigerator compartment 116. The left side of the image shows the inside of the left door 12. The right side of the image shows the inside of the right door 13. The upper half of the image shows the customer's living environment. These parts are unnecessary when viewed from the processed image.

[0054] FIG. 12 shows the image before processing, with unnecessary parts filled in. This system masks unnecessary parts of the fisheye image. The masking process performed by this system is a processing process that removes at least a part of the area other than the area of ​​the item. This prevents the customer and their living environment from being unintentionally seen by others when viewing images of the inside of the customer's refrigerator. The masking process of the unnecessary parts is executed by the server 2b, but may be executed by the server 2a or the refrigerator 1.

[0055] FIG. 13 is a diagram showing an example of the processed image. Server 2b corrects lens distortion and performs keystone correction on the unprocessed image taken by camera unit 10 of refrigerator 1 to process it into a front-view image. If server 2b detects a face in the image, it performs mosaic processing on the area where the face appears.

[0056] The pixelation process performed by this system reduces the amount of information in at least some areas other than the product area. This prevents the customer from unintentionally viewing images of the inside of the refrigerator when the images are viewed by others. The face detection process and pixelation process are performed by server 2b, but may also be performed by server 2a or refrigerator 1 on the unprocessed image.

[0057] 14 to 20 show a series of images for the customer to confirm precautions regarding the handling of camera images when pairing the mobile terminal 3 with the refrigerator 1. These screens are displayed when the CPU 31 of the mobile terminal 3 executes the refrigerator management program installed on the mobile terminal 3.

[0058] FIG. 14 is a diagram showing a new / used item selection screen 41 displayed on the mobile terminal 3. The selection screen 41 includes a new purchase button 411 , a others button 412 , and a back button 419 .

[0059] The new purchase button 411 is for users who have made a new purchase. When this button is tapped, the mobile terminal 3 pairs with the refrigerator 1 and then registers the user.

[0060] The Other button 412 is a button for those who have purchased a second-hand refrigerator 1. When this button is tapped, the mobile terminal 3 transitions to a warning screen 42 shown in FIG. The back button 419 is a button for transitioning to the previous screen before this selection screen 41.

[0061] FIG. 15 is a diagram showing the warning screen 42. The warning screen 42 is a screen that warns about points to be careful about when using a second-hand refrigerator 1. The confirmation button 421 is a button for ending the confirmation on the warning screen 42. When the confirmation button 421 is tapped, the mobile terminal 3 pairs with the refrigerator 1 and then performs user registration. The back button 429 is a button for transitioning to the previous screen before transitioning to the warning screen 42.

[0062] By registering the user, the identification number (individual number) of the refrigerator 1 is linked to the mobile terminal 3. Note that the user registration is not limited to the identification number (individual number) of the refrigerator 1, and the individual number of the camera unit 10 may also be linked to the mobile terminal 3.

[0063] After user registration, if there is already a registered user, the screen transitions to a continued registration user selection screen 43 in Fig. 16. If there is no already registered user after user registration, the screen transitions to a transfer warning screen 45 in Fig. 18.

[0064] FIG. 16 is a diagram showing the registration continuation user selection screen 43. As shown in FIG. The registration continuation user selection screen 43 is a screen for selecting a user whose registration to the refrigerator 1 will continue. Check boxes 431 to 433 are check boxes for selecting a user whose registration will continue.

[0065] The decision button 435 is a button for deciding after selecting a user for continuing registration. By tapping the decision button 435, the mobile terminal 3 transitions to the confirmation dialog box 44 in Fig. 17. The back button 439 is a button for transitioning to the previous screen, which transitions to the registration continuation user selection screen 43.

[0066] FIG. 17 is a diagram showing the confirmation dialog box 44. The confirmation dialog box 44 is a dialog for confirming the cancellation of registration of a user who will not continue to be registered.

[0067] The Yes button 441 is a button for deleting the registration of users who have not been selected. The No button 442 is a button for not deleting the registration of the non-selected user. When the Yes button 441 or the No button 442 is tapped, the screen transitions to the transfer notice screen 45 of FIG.

[0068] FIG. 18 is a diagram showing the transfer notice screen 45. Transfer warning screen 45 is a screen that displays warnings to be observed when transferring this refrigerator 1. Confirm button 451 is a button that indicates that the warnings have been confirmed. The back button 459 is a button for transitioning to the previous screen before the transfer notice screen 45 .

[0069] FIG. 19 is a diagram showing the camera usage warning screen 46. The camera usage warning screen 46 is a screen for alerting customers about taking pictures with the camera unit 10, which is the camera of the refrigerator 1. When the customer swipes this screen and reaches the bottom, a confirmation button for transitioning to the next image usage warning screen 47 is displayed. The back button 469 is a button for transitioning to the previous screen before the camera usage warning screen 46 .

[0070] FIG. 20 is a diagram showing the image utilization warning screen 47. The image utilization notice screen 47 is a screen for obtaining the customer's consent regarding utilization of images taken by the camera unit 10, which is the camera of the refrigerator 1. The consent button 471 is a button for the customer to consent to the use of the image. When the consent button 471 is tapped, a series of settings are completed and the screen transitions to the home screen 50. The back button 479 is a button for transitioning to the previous screen before the image utilization warning screen 47 .

[0071] FIG. 21 is a diagram showing the home screen 50. The home screen 50 is the screen that is displayed after starting this refrigerator management program. Check refrigerator contents button 501 is a button for viewing the contents stored in the customer's refrigerator 1. When the check refrigerator contents button 501 is tapped, the mobile terminal 3 transitions to a refrigerator contents screen 51 shown in FIG. The usage concierge button 502 is a button that provides an explanation on how to use the refrigerator 1.

[0072] FIG. 22 is a diagram showing the refrigerator contents screen 51. The refrigerator contents screen 51 is a screen for viewing a front view image of the refrigerator compartment 116 of this refrigerator 1. The room selection button 512 is a button for selecting whether to display the refrigerator room 116, the inside of the left door 12, or the inside of the right door 13. The back button 519 is a button for transitioning to the home screen 50, which is the screen immediately before the refrigerator contents screen 51.

[0073] Second Embodiment In the second embodiment, lens correction processing, keystone correction processing, etc. are performed by the calculation unit 112 of the refrigerator 1, not by the server 2b. Since there is no need to transmit unprocessed camera images over the network, it is possible to prevent the customer or their living environment from being unintentionally seen by others.

[0074] FIG. 23 is a sequence diagram of the image processing according to the second embodiment. When the user opens the door of refrigerator 1, calculation unit 112 detects that the door has been opened using door sensor 117. This causes calculation unit 112 to capture an image of refrigeration compartment 116, where food is stored, using camera unit 10 (step S30). This camera image is an unprocessed image that can be viewed by the user.

[0075] The calculation unit 112 stores the unprocessed camera image (step S31). The calculation unit 112 corrects distortion caused by the lens in the unprocessed camera image (step S32) and processes the image into a front-view image using keystone correction (step S33). The calculation unit 112 then deletes the unprocessed camera image (step S34). The calculation unit 112 requests an existing processed image from the storage server 6b (step S35) and receives the processed image from the storage server 6b (step S36).

[0076] After arithmetic unit 112 combines the existing processed image with the keystone corrected image (step S37), it transmits the combined processed image to server 2b (step S38). Server 2b transmits the processed image to storage server 6b for storage (step S39). Storage server 6b stores the processed image received from server 2b (step S40). Then, storage server 6b transmits the processed image to mobile terminal 3 in response to a request from mobile terminal 3 (step S41). This allows the customer to view the image of refrigerator compartment 116 on mobile terminal 3.

[0077] The storage server 6b deletes the edited image after a predetermined period of time has elapsed since storing the edited image. In other words, the storage server 6a deletes the unedited image that it stored before the storage server 6b deletes the edited image. This prevents customers who appear in the unedited image and their living environment from being unintentionally seen by others.

[0078] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. It is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0079] The above-described configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware such as an integrated circuit. The above-described configurations, functions, etc. may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or on a storage medium such as a flash memory card or a DVD (Digital Versatile Disk).

[0080] In each embodiment, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are interconnected. As modified examples of the present invention, for example, the following (a) to (d) are available.

[0081] (a) The pixelation process to prevent the customer and their living environment from being unintentionally seen by others may be performed by either the refrigerator 1, the server 2a, or the server 2b. (b) The masking process to prevent the customer and their living environment from being unintentionally seen by others may be performed by either the refrigerator 1, the server 2a, or the server 2b. (c) Either the mosaic processing or the masking processing may be performed, or both may be performed.

[0082] (d) The present invention is not limited to browsing the refrigerating compartment of a refrigerator, but may be applied to browsing the storage area of ​​any storage cabinet. [Explanation of symbols]

[0083] 1 refrigerator (storage room) 100 Camera LEDs 10 Camera Unit 112 Arithmetic section 113 ROM 114 RAM 115 Communications Department 116 Refrigerator (storage area) 117 Door Sensor 12 Left door 124 Rotating partition 13 Right door 14 Ice maker 15 Upper freezer compartment 16 Vegetable compartment 17 Lower freezer 2a Server (means for acquiring and deleting unprocessed images) 2b Server (processed image creation method) 21 CPU 22 ROM 23 RAM 24 Memory section 25 Communications Department 3. Mobile devices 31 CPU 32 ROM 33 RAM 34 Storage section 35 Communications Department 36 Touch Panel Display 41 Selection screen 411 New purchase button 412 Other button 419 Back button 42 Warning Screen 429 Back button 43 Registration continuation user selection screen 431 Checkbox 432 Checkbox 433 Checkbox 435 Decision button 439 Back button 44 Confirmation Dialog Box 441 Yes button 442 No button 45 Transfer notice screen 451 Confirm button 459 Back button 46 Camera usage warning screen 469 Back button 47 Image usage warning screen 479 Back button 50 Home Screen 501 Refrigerator contents check button 502 How to use concierge button 51 Refrigerator contents screen 512 Indoor selection button 6a Storage server (first storage means) 6b Storage server (second storage means) 81 Means for obtaining unprocessed images 82 1st storage means 83 Removal Method 84 Post-processing image creation method 85 Second storage means 86 Registration Method 87 Registered user information presentation means 88 Registered user authority deletion method

Claims

1. The system includes a storage cabinet, an imaging unit that is attached to the storage cabinet and captures images of the storage cabinet and the user's living environment, a storage server that stores the images captured by the imaging unit, and a mobile terminal, An image management system that provides a viewing service of images captured by the imaging unit corresponding to an individual number of the storage facility or the imaging unit to a user who is registered in association with the individual number of the storage facility or the imaging unit, among images stored in the storage server, a registration means for registering a user in association with an individual number of the storage unit or the imaging unit in response to an instruction from the user via the mobile terminal; a registered user authority deletion means capable of deleting the viewing authority of a registered user linked to the individual number in response to an instruction via the mobile terminal of the user registered in association with the individual number; a display means for displaying, on the mobile terminal of the user newly linked to the individual number by the registration means, a warning screen including a message indicating that the registered user can view images corresponding to the individual number; 1. An image management system comprising:

2. The system includes a storage cabinet, an imaging unit that is attached to the storage cabinet and captures images of the storage cabinet and the user's living environment, a storage server that stores the images captured by the imaging unit, and a mobile terminal, An image management system that provides a viewing service of images captured by the imaging unit corresponding to an individual number of the storage facility or the imaging unit to a user who is registered in association with the individual number of the storage facility or the imaging unit, among images stored in the storage server, a registration means for registering a user in association with an individual number of the storage unit or the imaging unit in response to an instruction from the user via the mobile terminal; a registered user authority deletion means capable of deleting the viewing authority of a registered user linked to the individual number in response to an instruction via the mobile terminal of the user registered in association with the individual number; a display means for displaying a warning screen, including a message indicating that the registered user can be deleted, on the mobile terminal of the user who has been newly linked to the individual number by the registration means; 1. An image management system comprising:

3. 3. The image management system according to claim 1, An image management system that displays on the mobile terminal, for a user who has been newly linked to the individual number by the registration means, a continued registration user selection screen that displays other users who have already been registered to the individual number to which the user has been registered.

4. 4. The image management system according to claim 3, The image management system displays the registration continuation user selection screen after displaying the warning screen on the mobile terminal.

5. 3. The image management system according to claim 1, An image management system that displays a transfer warning screen on the mobile terminal, which includes a request to erase the user's registration information linked to the individual number to which the user is linked at the time of transfer.

6. The system includes a storage cabinet, an imaging unit that is attached to the storage cabinet and captures images of the storage cabinet and the user's living environment, a storage server that stores the images captured by the imaging unit, and a mobile terminal, A method for producing an image management system that provides a viewing service for images captured by the imaging unit corresponding to an individual number of the storage facility or the imaging unit to a user who is registered in association with the individual number of the storage facility or the imaging unit, among images stored in the storage server, a step of transmitting an instruction via the mobile terminal of the user to register the user in association with the individual number of the storage unit or the imaging unit; a step of transmitting an instruction via the mobile terminal of a user registered in association with the individual number to enable deletion of the viewing authority of the registered user linked to the individual number; a step of displaying a warning screen on the mobile terminal of the user newly linked to the individual number, the warning screen including a message that the registered user can view images corresponding to the individual number and / or a message that the registered user can be deleted; a step of installing, in the mobile terminal, a management program for causing the mobile terminal to execute the above-mentioned steps;

7. The system includes a storage cabinet, an imaging unit that is attached to the storage cabinet and captures images of the storage cabinet and the user's living environment, a storage server that stores the images captured by the imaging unit, and a mobile terminal, an image management system that provides a viewing service of images captured by the imaging unit corresponding to the individual number to a user who is registered in association with the individual number of the storage facility or the imaging unit, among images stored in the storage server; The mobile terminal, a registration process for registering a user in association with an individual number of the storage unit or the imaging unit; a registered user authority deletion process capable of deleting the viewing authority of a registered user linked to the individual number; and a display process for displaying a warning screen including a message that the registered user can view the image corresponding to the individual number and / or a message that the registered user can be deleted.

Images