Image management system, production method of image management system, and image management method

Abstract

To conceal a customer and dwelling environment of the customer, when an image obtained by photographing inside a refrigerator of the customer is browsed.SOLUTION: An image management system includes a camera unit 10 for photographing an image where a storage object and dwelling environment of a user are imaged, a storage server for storing the image, and a portable terminal, and provides browsing service of the image photographed by the camera unit 10, out of the images stored in the storage server, to a user registered in association with an individual number of the storage object or the camera unit 10. The image management system can exhibit functions of registration means for registering the user in association with the individual number, registered user authority deletion means capable of deleting browsing authority of the registered user associated with the individual number, according to an instruction of the user registered associated with the individual number, and display means of displaying a warning screen including that the registered user can browse an image corresponding to the individual number, on a portable terminal of the user which is newly associated with the individual number by the registration means.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image management system, a method for producing an image management system, and an image management method. [Background technology]

[0002] In recent years, a service has been provided that allows users to view the interior of a refrigerator on a mobile terminal by uploading images of the interior of the refrigerator to a server and processing the uploaded images. Patent Document 1 describes an invention that allows many locations to be photographed with a small number of cameras. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2003-42626 A Summary of the Invention [Problem to be solved by the invention]

[0004] Images taken of the inside of a customer's refrigerator must be handled with care from the perspective of protecting personal information, as there is a risk that the customer or their living environment may be unintentionally seen by others. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the image management system of the present invention includes a storage unit, an imaging unit that captures images of the storage unit and the user's living environment while attached to the storage unit, a storage server that stores images captured by the imaging unit, and a mobile terminal, and provides a viewing service for images captured by the imaging unit corresponding to the individual number of the storage unit or the imaging unit, among the images stored in the storage server, to a user who has been registered in association with the individual number of the storage unit or the imaging unit, and is capable of performing the functions of: registration means that registers a user in association with the individual number of the storage unit or the imaging unit in response to an instruction from the user via the mobile terminal; registered user authority deletion means that is capable of deleting the viewing authority of a registered user linked to the individual number in response to an instruction from the user registered in association with the individual number via the mobile terminal; and display means that displays a warning screen on the mobile terminal of a user newly linked to the individual number by the registration means, including a message indicating that the registered user can view images corresponding to the individual number. The image management system of the present invention includes a storage unit, an imaging unit that captures images of the storage unit and the user's living environment when attached to the storage unit, a storage server that stores images captured by the imaging unit, and a mobile terminal, and provides a viewing service for images captured by the imaging unit corresponding to the individual number of the storage unit or the imaging unit, among the images stored in the storage server, to a user who has been registered in association with the individual number of the storage unit or the imaging unit, and is capable of performing the functions of: registration means that registers a user in association with the individual number of the storage unit or the imaging unit in response to an instruction provided by the user via the mobile terminal; registered user authority deletion means that is capable of deleting the viewing authority of a registered user linked to the individual number in response to an instruction provided by the user registered in association with the individual number via the mobile terminal; and display means that displays a warning screen on the mobile terminal of a user newly linked to the individual number by the registration means, including a message indicating that the registered user can be deleted.

[0006] The production method of the image management system of the present invention includes a storage unit, an imaging unit that is attached to the storage unit and captures images that show the storage unit and a user's living environment, a storage server that stores images captured by the imaging unit, and a mobile terminal, and provides a viewing service for an image captured by the imaging unit corresponding to the individual number of the storage unit or the imaging unit, among the images stored in the storage server, to a user who is registered in association with the individual number of the storage unit or the imaging unit, and includes a step of installing a management program to function as part of the image management system that can function as: a registration means that registers a user in association with the individual number of the storage unit or the imaging unit in response to an instruction from the user via the mobile terminal; a registered user authority deletion means that is capable of deleting the viewing authority of a registered user linked to the individual number in response to an instruction from the user registered in association with the individual number via the mobile terminal; and a display means that displays a warning screen on the mobile terminal of a user newly linked to the individual number by the registration means, the warning screen including a message indicating that the registered user can view images corresponding to the individual number and / or a message indicating that the registered user can be deleted.

[0007] The image management method of the present invention includes an image management system that includes a storage unit, an imaging unit that is attached to the storage unit and captures images that show the storage unit and the user's living environment, a storage server that stores the images captured by the imaging unit, and a mobile terminal, and provides a viewing service for images captured by the imaging unit corresponding to the individual number of the storage unit or the imaging unit, among the images stored in the storage server, to a user who is registered in association with the individual number of the storage unit or the imaging unit.The mobile terminal is operated as a registration means that registers a user by linking the user to the individual number of the storage unit or the imaging unit, a registered user authority deletion means that can delete the viewing authority of a registered user linked to the individual number, and a display means that displays a warning screen indicating that the registered user can view images corresponding to the individual number and / or that the registered user can be deleted.

[0008] Other means will be described in the description of the embodiment of the invention. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of a refrigerator according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram of two types of servers. [Diagram 3] FIG. 2 is a block diagram of a mobile terminal. [Figure 4] FIG. 2 is a logical block diagram of the image management system. [Diagram 5] FIG. [Figure 6] FIG. [Figure 7] FIG. 2 is a front view of the refrigerator with the left and right doors of the refrigerator compartment open. [Figure 8] FIG. 2 is a plan view of the refrigerator with the left and right doors of the refrigerator compartment open. [Figure 9] This is a perspective view of the camera unit of the refrigerator when viewed diagonally from below. [Figure 10] FIG. 11 is a sequence diagram of a first image processing process. [Figure 11] FIG. 13 is a diagram showing an unprocessed image captured by a refrigerator camera. [Figure 12] FIG. 13 is a diagram showing an unprocessed image in which unnecessary parts are masked. [Figure 13] FIG. 13 is a diagram showing an example of a processed image. [Figure 14] FIG. 13 is a diagram showing a selection screen displayed on the mobile terminal. [Figure 15] FIG. 13 illustrates a warning screen. [Figure 16] FIG. 13 is a diagram showing a registration continuation user selection screen. [Figure 17] FIG. 13 illustrates a confirmation dialog box. [Figure 18] FIG. 13 shows a screen for cautions when transferring. [Figure 19] FIG. 13 shows a screen for cautions when transferring. [Figure 20] FIG. 13 is a diagram showing an image usage warning screen. [Figure 21] FIG. 2 is a diagram showing a home screen. [Figure 22] FIG. 13 shows a refrigerator contents screen. [Diagram 23] FIG. 11 is a sequence diagram of an image processing process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, the image management system provides services to users by linking the refrigerator 1 in Fig. 1, the servers 2a, 2b and storage servers 6a, 6b in Fig. 2, and the mobile terminal 3 in Fig. 3. The refrigerator 1, the servers 2a, 2b, the storage servers 6a, 6b, and the mobile terminal 3 are each assigned a different IP address.

[0011] The configuration of a refrigerator 1 will be described with reference to FIG. The refrigerator 1 has, as its hardware configuration, a camera unit 10, a calculation unit 112, a ROM (Read Only Memory) 113, a RAM (Random Access Memory) 114, a communication unit 115, and a camera LED (Light Emitting Diode) 100. The refrigerator 1 further has a left door 12, a right door 13, a door sensor 117, and a refrigerator compartment 116, all of which are shown in FIG.

[0012] The calculation unit 112 is, for example, a CPU (Central Processing Unit) and controls the refrigerator 1. The RAM 114 serves as a work area for the calculation unit 112. The RAM 114 is also a non-temporary or temporary recording medium that stores various programs and data, and stores a program (not shown).

[0013] The camera unit 10 is an imaging unit that captures images of the refrigerator compartment 116, and is, for example, a fisheye camera or a panoramic camera. The camera unit 10 is attached to the upper side of the refrigerator compartment 116. The camera LED 100 is an indicator that lights up when the camera unit 10 is capturing an image. The left door 12 and the right door 13 are doors provided in the refrigerator compartment 116, and the open / closed state is detected by a door sensor 117. When a user opens the left door 12 and / or the right door 13, the camera unit 10 operates to capture an image of the refrigerator compartment 116. The refrigerator compartment 116 is a storage unit that stores items, typically foods.

[0014] The communication unit 115 is connected to a network and transmits and receives data to and from the server 2a, which is an external device.

[0015] The configurations of the servers 2a, 2b and the storage servers 6a, 6b will be described with reference to Fig. 2. The servers 2a, 2b and the storage servers 6a, 6b each include a CPU 21, a ROM 22, a RAM 23, a storage unit 24, and a communication unit 25.

[0016] The CPU 21 controls the servers 2a and 2b. The ROM 22 is a non-volatile memory, and stores, for example, a BIOS (Basic I / O System). The RAM 23 serves as a working area for the CPU 21. The RAM 23 is a non-transient or temporary recording medium that stores various programs and data, and stores a program (not shown).

[0017] The storage unit 24 is a non-temporary or temporary recording medium for storing various programs and data, and is composed of, for example, an HDD (Hard Disk Drive), a flash memory, etc. It is preferable that the storage units 24 of the storage servers 6a and 6b have a large capacity and are accessible at high speed. The communication unit 25 is connected to a network and transmits and receives data to and from external devices.

[0018] The configuration of the mobile terminal 3 will be described with reference to Fig. 3. The mobile terminal 3 includes a CPU 31, a ROM 32, a RAM 33, a storage unit 34, a communication unit 35, and a touch panel display 36.

[0019] The CPU 31 controls the mobile terminal 3. The ROM 32 is a non-volatile memory, and stores, for example, a BIOS. The RAM 33 serves as a working area for the CPU 31. The RAM 33 is a non-transient or temporary recording medium that stores various programs and data, and stores a program (not shown).

[0020] The storage unit 34 is a non-transient or temporary recording medium that stores various programs and data, and is composed of, for example, a flash memory. The communication unit 35 connects to a network and transmits and receives data to and from external devices. The touch panel display 36 is composed of a touch panel layered on a flat display. The touch panel display 36 is a display means that displays characters, images, or figures, and is also an input means that detects user operations.

[0021] FIG. 4 is a logical block diagram of the image management system. The unprocessed image acquisition means 81 acquires an unprocessed image captured by the camera unit 10. The unprocessed image acquisition means 81 is realized by the CPU 21 of the server 2a executing a program. The first storage means 82 stores the unprocessed image. This first storage means 82 corresponds to the storage server 6a.

[0022] The processed image creation means 84 executes processing on the pre-processing image to delete or reduce the amount of information on at least a portion of the area other than the item area, and creates a processed image. This processed image creation means 84 deletes or reduces the amount of information on the pre-processing image, including a predetermined area, and creates a processed image. And when a person's body part is captured in any of the pre-processing images, the processed image creation means 84 deletes or reduces the amount of information on the person's body part and creates a processed image. This processed image creation means 84 is embodied by the CPU 21 of the server 2b executing a program.

[0023] The deletion means 83 deletes the pre-processed image after the post-processed image is created. The deletion means 83 deletes the pre-processed image at least before the post-processed image is deleted. This is realized by the CPU 21 of the server 2a executing a program.

[0024] The second storage means 85 stores the processed image. This second storage means 85 corresponds to the storage server 6a. The registration means 86 registers users who can view the processed image by linking them to the individual number of the repository or the imaging unit. The registered user information presentation means 87 presents information about already registered users to a new user who is to be linked to an individual number.

[0025] When registering a new user to be linked to this individual number, the registered user authority deleting means 88 has the user select an already registered user and deletes the browsing authority of the selected registered user. The registration means 86, the registered user information presenting means 87, and the registered user authority deleting means 88 are embodied by the CPU 31 of the mobile terminal 3 executing a program.

[0026] FIG. 5 is a front view of the refrigerator 1. Refrigerator 1 is an appliance for cooling food and the like, and includes a left door 12 and a right door 13 for refrigerator compartment 116, and a camera unit 10. Refrigerator 1 has multiple storage compartments inside its housing. The storage compartments of refrigerator 1 include, from the top, refrigerator compartment 116, ice-making compartment 14 and upper freezer compartment 15 lined up on the left and right, as well as a vegetable compartment 16 and a lower freezer compartment 17.

[0027] The housing of refrigerator 1 is configured such that a heat insulating material (not shown) such as urethane foam is filled between an outer box made of steel plate and an inner box (not shown) made of resin. A plurality of openings corresponding to each compartment are provided on the front side (front side) of the housing of refrigerator 1. For example, when food or the like is placed through the opening of refrigerator compartment 116, left door 12 and right door 13 are opened. When left door 12 and right door 13 are closed, the opening of refrigerator compartment 116 is blocked. In this way, left door 12 and right door 13 have a function of closing the opening of refrigerator compartment 116. The same applies to each of the other doors.

[0028] Refrigerator 1 is equipped with left door 12 and right door 13 as French-style doors which, together with the housing, form refrigerating compartment 116. Left door 12 is rotatable about the axis of hinge 121 (see FIG. 8) at the left end. The same is true for right door 13. Refrigerator 1 is also equipped with drawer-type doors, such as an ice-making compartment door and an upper freezer compartment door, as well as a vegetable compartment door and a lower freezer compartment door.

[0029] Refrigerating compartment 116 is provided with a plurality of shelves that divides refrigerator compartment 116 into predetermined sections. A plurality of door pockets 123 (see FIG. 7) for storing food and the like are provided on the inside of left door 12. The same is true for left door 12. Ice-making compartment 14 is provided with an ice-making compartment container (not shown) that is pulled out together with the ice-making compartment door. Similarly, upper freezer compartment 15 is provided with an upper freezer container (not shown), vegetable compartment 16 is provided with a vegetable container (not shown), and lower freezer compartment 17 is provided with a lower freezer container (not shown).

[0030] Although not shown, the refrigerator 1 also includes a compressor, a radiator (condenser), a capillary tube (throttle mechanism), and an evaporator. A refrigerant circulates through the compressor, the radiator, the capillary tube, and the evaporator in this order, and the air in the storage compartment is cooled by heat exchange with the refrigerant flowing through the evaporator. The camera unit 10 shown in Fig. 5 is for capturing images of the storage compartment and the like of the refrigerator 1, and is installed on the top surface of the housing.

[0031] FIG. 6 is a side view of the refrigerator 1. As shown in Fig. 6, the camera unit 10 includes a main body 101 and a support 102. The main body 101 has a function of capturing images of a storage compartment, etc. The support 102 supports the main body 101 and is installed on the top surface of the housing of the refrigerator 1.

[0032] A lens 103 (see also FIG. 6) is provided near the front end of main body 101. Lens 103 is an optical element that refracts and focuses light. For example, a fisheye lens is used as such lens 103. When left door 12 and / or right door 13 (see FIG. 5) are opened, lens 103 faces downward so that camera unit 10 can capture an image of refrigerator compartment 116 (see FIG. 7). Camera unit 10 is attached in a position where it may capture an image of the living space including the living space outside refrigerator compartment 116.

[0033] As shown in FIG. 6, lens 103 is located forward of the front end of the housing (opening of the housing). More preferably, lens 103 is located further forward than the front of left door 12 and right door 13 (see FIG. 5) in the closed state. This makes it easier for refrigerator compartment 116 to be within the field of view of lens 103 when left door 12 and right door 13 of refrigerator compartment 116 are opened, for example. It is also possible to capture images of vegetable compartment 16 and the like (see FIG. 5) with a drawer-type door such as the vegetable compartment door open. In this way, lens 103 is provided in a position that allows each storage compartment to be viewed from above.

[0034] FIG. 7 is a front view of refrigerator 1 with left door 12 and right door 13 open. As described above, the inner panel 122 of the left door 12 is provided with a plurality of door pockets 123 for storing food and the like (the same is true for the right door 13). When the left door 12 and the right door 13 are opened, the food and the like in the refrigerator compartment 116 and the door pockets 123, 133 are arranged to be viewed from above within the field of view of the lens 103 of the camera unit 10.

[0035] The rotating partition 124 shown in Fig. 7 is a partition for preventing cool air from leaking from the gap between the left door 12 and the right door 13. In the example of Fig. 7, the rotating partition 124 is provided at the end of the left door 12 opposite to the axis of the hinge 121 (see Fig. 8), and is designed to rotate in a predetermined manner when the left door 12 is opened or closed.

[0036] Furthermore, when manually photographing the refrigerator compartment 116 using the camera unit 10, a photographing button 125 to be pressed by the user is provided on the left door 12, and a photographing button 135 is provided on the right door 13. In the example of FIG. 7, the photographing button 135 is provided on the lower part of the surface of the left door 12 opposite the hinge 121 (see FIG. 8) of the left door 12 (similar to the right door 13). From another perspective, when the left door 12 and the right door 13 are closed (see FIG. 5), the photographing button 125 is provided on the surface where one of the left door 12 and the right door 13 faces the other (near the rotating partition 124 in FIG. 7).

[0037] Shooting performed when at least one of the shooting buttons 125, 135 is pressed by the user is called "manual shooting." Shooting performed by the calculation unit 112 based on the elapsed time from the timing when the left door 12 and the right door 13 start to open is called "automatic shooting."

[0038] FIG. 8 is a plan view of refrigerator 1 with left door 12 and right door 13 open. In the example of Fig. 8, the main body 101 of the camera unit 10 is disposed slightly to the left of the center of the housing of the refrigerator 1 in the left-right direction. To explain in more detail, the main body 101 of the camera unit 10 is disposed directly above the joint between the left door 12 and the right door 13 in the closed state (see also Figs. 5 and 6). By disposing the main body 101 in this manner, for example, when the left door 12 and the right door 13 are opened, food etc. in the door pockets 123, 133 (see Fig. 7) are more likely to come into the field of view of the camera unit 10.

[0039] Furthermore, when using refrigerator 1, the user is often conscious of the seam (boundary) between left door 12 and right door 13. Therefore, by locating main body 101 directly above the seam between left door 12 and right door 13, the user is less likely to feel uncomfortable compared to when main body 101 is not positioned at the seam in the left-right direction.

[0040] FIG. 8 is a perspective view of the camera unit 10 when viewed obliquely from below. The main body 101 of the camera unit 10 shown in Fig. 8 includes the lens 103 (see also Fig. 6) described above, as well as a case 31b and a camera LED 100 (Light Emitting Diode: illumination unit). In the example of Fig. 6, the case of the main body 101 is generally a rectangular parallelepiped that is elongated in the front-rear direction. A circular hole (not shown) is provided on the underside near the front end of the case of the main body 101, and the lens 103 is exposed through this hole. The camera LED 100 is provided on the rear side (rear side) of the lens 103.

[0041] The camera LED 100 is a light source that irradiates light onto the refrigerator compartment 116 (see FIG. 7) and door pockets 123, 133 (see FIG. 7) so that the camera unit 10 can photograph the refrigerator compartment 116 and the like under appropriate brightness. The camera LED 100 (illumination unit) is provided on the outside of the housing of the refrigerator 1 (see FIG. 6). Although details will be described later, in this embodiment, the camera unit 10 photographs the refrigerator compartment 116 and the like (see FIG. 7) with the interior light of the refrigerator 1 turned off and the camera LED 100 turned on.

[0042] As shown in Fig. 9, by providing lens 103 in front of camera LED 100, reflected light from refrigeration compartment 116 and the like (see Fig. 5) can easily enter lens 31a when left door 12 and right door 13 are open. Therefore, food stored in refrigeration compartment 116 and the like can be properly photographed. Note that it is preferable that camera LED 100 is positioned forward of the front end (opening of the housing) of the refrigerator 1 housing (see Fig. 6). This makes it easier for light emitted from camera LED 100 to enter refrigeration compartment 116 and the like, which in turn makes it easier to obtain clear photographing results.

[0043] The refrigerator 1 is a storage unit including a refrigerating chamber 116 which is a storage area for storing items, and a camera unit 10 which captures an image of the area containing the items including the refrigerating chamber 116.

[0044] FIG. 10 is a sequence diagram of the image processing according to the first embodiment. When the user opens the door of the refrigerator 1, the calculation unit 112 detects that the door has been opened by the door sensor 117. This causes the calculation unit 112 to capture an image of the refrigerator compartment 116 in which food is stored, using the camera unit 10 (step S10). The calculation unit 112 uploads the captured camera image of the refrigerator compartment 116 to the server 2a (step S11). This camera image is an image before any processing that can be viewed by the user.

[0045] The CPU 21 of the server 2a functions as an unprocessed image acquiring means for acquiring an unprocessed image captured by the camera unit 10. The server 2a transmits this unprocessed camera image to the storage server 6a (step S12).

[0046] When the storage server 6a notifies the server 2b of the event (step S13), the storage server 6a stores the unprocessed camera image (step S14). The storage server 6a functions as a first storage unit that stores the unprocessed image. Then, when the storage server 6a transmits the unprocessed camera image to the server 2b (step S15), the storage server 6a deletes the unprocessed camera image stored in itself (step S16). That is, the CPU 21 of the storage server 6a functions as a deletion unit that deletes the unprocessed camera image.

[0047] The server 2b corrects the distortion of the camera image before processing due to the lens (step S17), and processes the image into a front view image by keystone correction (step S18). The server 2b then requests the existing processed image from the storage server 6b (step S19), and receives the processed image from the storage server 6b (step S20).

[0048] When the server 2b combines the existing processed image with the keystone corrected image (step S21), it transmits the combined processed image to the storage server 6b for storage (step S22). The CPU 21 of the server 2b functions as a processed image creating means for creating a processed image by executing a process on the unprocessed image to delete at least a part of the area other than the item area or to reduce the amount of information.

[0049] The storage server 6b stores the processed image received from the server 2b (step S23). The storage server 6b functions as a second storage unit that stores the processed image. Then, in response to a request from the mobile terminal 3, the storage server 6b transmits the processed image to the mobile terminal 3 (step S24). This allows the customer to view the image of the refrigerator compartment 116 on the mobile terminal 3.

[0050] The storage server 6a stores the unprocessed image, and the storage server 6b stores the synthesized processed image, making it easier to manage each image. The mobile terminal 3 can easily obtain the synthesized processed image by simply accessing the storage server 6b.

[0051] The mobile terminal 3 does not access the storage server 6a, and therefore does not know the IP address of the storage server 6a. This makes it impossible to know how to access the storage server 6a from the mobile terminal 3, and therefore prevents the customer or the customer's living environment from being seen unintentionally by others in the unedited image.

[0052] The storage server 6b deletes the edited image after a predetermined period of time has elapsed since the storage server 6b saved the edited image. In other words, the storage server 6a deletes the unedited image that it saved before the storage server 6b deletes the edited image. This prevents the customer who appears in the unedited image and the customer's living environment from being seen by others unintentionally.

[0053] Fig. 11 shows an image taken by the camera unit 10 of the refrigerator 1 before processing. The lower half of the image shows the refrigerator compartment 116. The left side of the image shows the inside of the left door 12. The right side of the image shows the inside of the right door 13. And the upper half of the image shows the customer's living environment. This part will be unnecessary when viewed from the processed image.

[0054] FIG. 12 shows the image before processing in which the unnecessary parts have been painted out. The system performs a masking process on unnecessary parts of the fisheye image. The masking process performed by the system is a processing process that deletes at least a part of the area other than the area of ​​the item. This prevents the customer and their living environment from being unintentionally seen by others when viewing an image of the inside of the customer's refrigerator. The masking process of the unnecessary parts is executed by the server 2b, but may be executed by the server 2a or the refrigerator 1.

[0055] FIG. 13 is a diagram showing an example of the processed image. Server 2b corrects lens distortion of the unprocessed image taken by camera unit 10 of refrigerator 1 and processes it into a front view image by performing keystone correction. At this time, if server 2b detects a face in the image, it performs mosaic processing on the area in which the face appears.

[0056] The pixelization process performed by this system is a process that reduces the amount of information in at least a part of the area other than the area of ​​the item. This prevents the customer from unintentionally letting others see the image of the inside of the refrigerator when the customer is viewing it. Note that the face detection process and pixelization process are performed by server 2b, but server 2a or refrigerator 1 may also perform them on the image before processing.

[0057] 14 to 20 show a series of images for confirming with the customer important points regarding the handling of camera images when pairing the mobile terminal 3 with the refrigerator 1. These screens are displayed when the CPU 31 of the mobile terminal 3 executes the refrigerator management program installed in the mobile terminal 3.

[0058] FIG. 14 is a diagram showing a new / used item selection screen 41 displayed on the mobile terminal 3. As shown in FIG. The selection screen 41 includes a new purchase button 411 , an other button 412 , and a back button 419 .

[0059] The new purchase button 411 is for users who have made a new purchase. When this button is tapped, the mobile terminal 3 pairs with the refrigerator 1 and then registers the user.

[0060] The Other button 412 is a button for those who have purchased a second-hand refrigerator 1. When this button is tapped, the mobile terminal 3 transitions to a warning screen 42 shown in FIG. The back button 419 is a button for transitioning to the previous screen before this selection screen 41 .

[0061] FIG. 15 is a diagram showing the warning screen 42. As shown in FIG. The warning screen 42 is a screen that warns about points to be careful about when the refrigerator 1 is a second-hand item. The confirmation button 421 is a button for ending the confirmation on this warning screen 42. When this confirmation button 421 is tapped, the mobile terminal 3 pairs with the refrigerator 1 and then performs user registration. The back button 429 is a button for transitioning to the previous screen before the warning screen 42 .

[0062] Through user registration, the identification number (individual number) of the refrigerator 1 is linked to this mobile terminal 3. Note that user registration may also be performed by linking the individual number of the camera unit 10 to the mobile terminal 3, rather than being limited to the identification number (individual number) of the refrigerator 1.

[0063] After the user registration, if there is already a registered user, the screen transitions to a continued registration user selection screen 43 in Fig. 16. If there is no already registered user after the user registration, the screen transitions to a transfer warning screen 45 in Fig. 18.

[0064] FIG. 16 is a diagram showing the registration continuation user selection screen 43. As shown in FIG. The continuously registered user selection screen 43 is a screen for selecting a user whose registration to the refrigerator 1 is to be continued. Check boxes 431 to 433 are check boxes for selecting a user whose registration is to be continued.

[0065] The decision button 435 is a button for deciding a user to continue registration after the user is selected. By tapping the decision button 435, the mobile terminal 3 transitions to the confirmation dialog box 44 in Fig. 17. The back button 439 is a button for transitioning to the previous screen, which is the registration continuation user selection screen 43.

[0066] FIG. 17 is a diagram showing the confirmation dialog box 44. The confirmation dialog box 44 is a dialog for confirming the deregistration of a user whose registration will not continue.

[0067] The Yes button 441 is a button for deleting the registration of users who have not been selected. The No button 442 is a button for not deleting the registration of the non-selected user. When the Yes button 441 or the No button 442 is tapped, the screen transitions to the transfer notice screen 45 in FIG.

[0068] FIG. 18 is a diagram showing the transfer warning screen 45. The transfer warning screen 45 is a screen that displays warnings to be observed when transferring this refrigerator 1. The confirmation button 451 is a button that indicates that the warnings have been confirmed. The back button 459 is a button for transitioning to the previous screen, which is the transfer warning screen 45 .

[0069] FIG. 19 is a diagram showing the camera usage warning screen 46. The camera usage warning screen 46 is a screen for alerting the customer to the risk of taking pictures with the camera unit 10, which is the camera of the refrigerator 1. When the customer swipes this screen and reaches the bottom, a confirmation button for transitioning to the next image usage warning screen 47 is displayed. The back button 469 is a button for transitioning to the previous screen before the camera usage warning screen 46 .

[0070] FIG. 20 is a diagram showing the image usage warning screen 47. The image utilization notice screen 47 is a screen for obtaining the customer's consent regarding utilization of images taken by the camera unit 10 which is the camera of the refrigerator 1. The consent button 471 is a button for the customer to consent to the use of the image. When the consent button 471 is tapped, a series of settings is completed and the screen transitions to the home screen 50. The back button 479 is a button for transitioning to the previous screen before the image utilization warning screen 47 .

[0071] FIG. 21 is a diagram showing the home screen 50. As shown in FIG. The home screen 50 is a screen that is displayed after the refrigerator management program is started. Check refrigerator contents button 501 is a button for viewing the contents stored in the customer's refrigerator 1. When check refrigerator contents button 501 is tapped, mobile terminal 3 transitions to refrigerator contents screen 51 of FIG. The usage concierge button 502 is a button that provides an explanation of how to use the refrigerator 1.

[0072] FIG. 22 is a diagram showing the refrigerator contents screen 51. The refrigerator contents screen 51 is a screen for viewing a front view image of the refrigerator compartment 116 of the refrigerator 1. The room selection button 512 is a button for selecting whether to display the refrigerator room 116, the inside of the left door 12, or the inside of the right door 13. The back button 519 is a button for transitioning to the home screen 50, which is the screen immediately before transitioning to the refrigerator contents screen 51.

[0073] Second Embodiment In the second embodiment, lens correction processing, keystone correction processing, etc. are performed by the calculation unit 112 of the refrigerator 1, not by the server 2b. Since there is no need to transmit unprocessed camera images over the network, it is possible to prevent the customer or their living environment from being unintentionally seen by others.

[0074] FIG. 23 is a sequence diagram of the image processing according to the second embodiment. When the user opens the door of the refrigerator 1, the calculation unit 112 detects that the door has been opened by the door sensor 117. This causes the calculation unit 112 to capture an image of the refrigerator compartment 116 in which food is stored by the camera unit 10 (step S30). This camera image is an image before any processing is performed that can be viewed by the user.

[0075] The calculation unit 112 stores the unprocessed camera image (step S31). The calculation unit 112 corrects distortion of the unprocessed camera image caused by the lens (step S32), and processes the unprocessed camera image into a front-view image by keystone correction (step S33). The calculation unit 112 then deletes the unprocessed camera image (step S34). The calculation unit 112 requests the existing processed image from the storage server 6b (step S35), and receives the processed image from the storage server 6b (step S36).

[0076] When the calculation unit 112 combines the existing processed image and the keystone corrected image (step S37), it transmits the combined processed image to the server 2b (step S38). The server 2b transmits the processed image to the storage server 6b to store it (step S39). The storage server 6b stores the processed image received from the server 2b (step S40). Then, the storage server 6b transmits the processed image to the mobile terminal 3 in response to a request from the mobile terminal 3 (step S41). This allows the customer to view the image of the refrigerator compartment 116 on the mobile terminal 3.

[0077] The storage server 6b deletes the edited image after a predetermined period of time has elapsed since the storage server 6b saved the edited image. In other words, the storage server 6a deletes the unedited image that it saved before the storage server 6b deletes the edited image. This prevents the customer who appears in the unedited image and the customer's living environment from being seen by others unintentionally.

[0078] The present invention is not limited to the above-described embodiment, and includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. It is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0079] The above configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware such as an integrated circuit. The above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or a recording medium such as a flash memory card or a DVD (Digital Versatile Disk).

[0080] In each embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it may be considered that almost all components are connected to each other. As modified examples of the present invention, for example, the following (a) to (d) are available.

[0081] (a) The pixelation process to prevent the customer and his / her living environment from being seen unintentionally by others may be performed by either the refrigerator 1, the server 2a or the server 2b. (b) The masking process for preventing the customer and the customer's living environment from being unintentionally seen by others may be executed by the refrigerator 1, the server 2a, or the server 2b. (c) Either the mosaic processing or the masking processing may be performed, or both may be performed.

[0082] (d) The present invention is not limited to viewing the refrigeration compartment of a refrigerator, but may be applied to viewing the storage area of ​​any storage unit. [Explanation of symbols]

[0083] 1 Refrigerator (storage unit) 100 Camera LED 10 Camera unit 112 Arithmetic section 113 ROM 114 RAM 115 Communications Department 116 Refrigerator (storage area) 117 Door Sensor 12 Left door 124 Rotating partition 13 Right door 14 Ice Room 15 Upper freezer 16 Vegetable compartment 17 Lower freezer 2a Server (means for acquiring and deleting unprocessed images) 2b Server (processed image creation method) 21 CPU 22 ROM 23 RAM 24 Memory section 25 Communications Department 3. Mobile devices 31 CPU 32 ROM 33 RAM 34 Storage section 35 Communications Department 36 Touch Panel Display 41 Selection Screen 411 New purchase button 412 Other buttons 419 Back button 42 Warning Screen 429 Back button 43 Registration continuation user selection screen 431 Checkbox 432 Checkbox 433 Checkbox 435 Decision button 439 Back button 44 Confirmation Dialog Box 441 Yes button 442 No button 45 Transfer notice screen 451 Confirm button 459 Back button 46 Camera usage warning screen 469 Back button 47 Image usage warning screen 479 Back button 50 Home Screen 501 Fridge contents check button 502 How to use concierge button 51 Fridge contents screen 512 Indoor selection button 6a Storage server (first storage means) 6b Storage server (second storage means) 81 Means for obtaining unprocessed images 82 1st storage means 83 Removal Method 84 Post-processing image creation method 85 Second storage means 86 Registration Method 87 Registered user information presentation means 88 Registered user authority deletion method

Claims

1. The system includes a storage unit, an imaging unit that is attached to the storage unit and captures an image of the storage unit and a user's living environment, a storage server that stores the image captured by the imaging unit, and a mobile terminal, An image management system that provides a viewing service of an image captured by the imaging unit corresponding to an individual number of the storage facility or the imaging unit to a user who is registered in association with the individual number of the storage facility or the imaging unit, among images stored in the storage server, A registration means for registering a user in association with an individual number of the storage unit or the imaging unit in response to an instruction from the user via the mobile terminal; a registered user authority deleting means capable of deleting the browsing authority of a registered user linked to the individual number in response to an instruction via the mobile terminal of the user registered in association with the individual number; a display means for displaying, on the mobile terminal of the user newly associated with the individual number by the registration means, a warning screen including a message indicating that the registered user can view an image corresponding to the individual number; An image management system that can function as follows.

2. The system includes a storage unit, an imaging unit that is attached to the storage unit and captures an image of the storage unit and a user's living environment, a storage server that stores the image captured by the imaging unit, and a mobile terminal, An image management system that provides a viewing service of an image captured by the imaging unit corresponding to an individual number of the storage facility or the imaging unit to a user who is registered in association with the individual number of the storage facility or the imaging unit, among images stored in the storage server, A registration means for registering a user in association with an individual number of the storage unit or the imaging unit in response to an instruction from the user via the mobile terminal; a registered user authority deleting means capable of deleting the browsing authority of a registered user linked to the individual number in response to an instruction via the mobile terminal of the user registered in association with the individual number; a display means for displaying a warning screen including a message indicating that the registered user can be deleted on the mobile terminal of the user who has been newly linked to the individual number by the registration means; An image management system that can function as follows.

3. 3. The image management system according to claim 1, The image management system displays on the mobile terminal, for a user newly linked to the individual number by the registration means, a continued registration user selection screen that displays other users who are already registered to the individual number to which the user is registered.

4. 4. The image management system according to claim 3, The image management system displays the registration continuation user selection screen after displaying the warning screen on the mobile terminal.

5. 3. The image management system according to claim 1, An image management system that displays a transfer warning screen on the mobile terminal, which includes a request that the user's registration information linked to the individual number to which the user is linked be erased at the time of transfer.

6. The system includes a storage unit, an imaging unit that is attached to the storage unit and captures an image of the storage unit and a user's living environment, a storage server that stores the image captured by the imaging unit, and a mobile terminal, A method for producing an image management system for providing a viewing service of an image captured by the imaging unit corresponding to an individual number of the storage facility or the imaging unit to a user who is registered in association with the individual number of the storage facility or the imaging unit, among images stored in the storage server, the method comprising the steps of: A registration means for registering a user in association with an individual number of the storage unit or the imaging unit in response to an instruction from the user via the mobile terminal; a registered user authority deleting means capable of deleting the browsing authority of a registered user linked to the individual number in response to an instruction via the mobile terminal of the user registered in association with the individual number; a display means for displaying, on the mobile terminal of the user newly linked to the individual number by the registration means, a warning screen including a message indicating that the registered user can view images corresponding to the individual number and / or a message indicating that the registered user can be deleted; and a step of installing a management program on the mobile terminal to function as part of the image management system capable of fulfilling the function as a management program for the mobile terminal.

7. The system includes a storage unit, an imaging unit that is attached to the storage unit and captures an image of the storage unit and a user's living environment, a storage server that stores the image captured by the imaging unit, and a mobile terminal, an image management system that provides a viewing service of an image captured by the imaging unit corresponding to an individual number of the storage facility or the imaging unit to a user who is registered in association with the individual number of the storage facility or the imaging unit among the images stored in the storage server, The mobile terminal, A registration means for registering a user in association with an individual number of the storage unit or the imaging unit; a registered user authority deleting means capable of deleting the browsing authority of a registered user linked to the individual number; The image management method further comprises the steps of: operating the image management device as display means for displaying a warning screen including a message indicating that the registered user can view the image corresponding to the individual number and / or a message indicating that the registered user can be deleted.

Images