Image management system, image management server, image management method, and method for producing an image management system

Abstract

The present application provides an image management system, an image management server, an image management method, and a method of producing an image management system. When viewing an image obtained by photographing inside a customer's refrigerator, the customer and the customer's living environment are hidden. The image management system includes a refrigerator (1) including a refrigerating chamber (116) for storing food and a camera unit (10) for photographing an article range including the refrigerating chamber (116); a server (2a) for acquiring a pre-processed image photographed by the camera unit (10); a storage server (6a) for storing the pre-processed image; a server (2b) for performing a masking process or a mosaic process on the pre-processed image to generate a post-processed image; a storage server (6b) for storing the post-processed image; and a deletion unit for deleting the pre-processed image at least before the post-processed image is deleted.

Description

Technical Field

[0001] This invention relates to an image management system, an image management server, an image management method, and a method for producing an image management system. Background Technology

[0002] In recent years, by uploading images of the inside of a refrigerator to a server and processing them, a service has been developed that allows users to view the contents of the refrigerator on a portable device. Patent Document 1 describes an invention that can capture images of more locations with a small number of cameras.

[0003] Images obtained by photographing the inside of a customer's refrigerator must be handled carefully from the perspective of protecting personal information. This is because the customer or their living environment may be unintentionally seen by others.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2003-42626 Summary of the Invention

[0005] To address the aforementioned issues, the image management system of the present invention comprises: a storage area having a storage region for storing items and a camera unit for capturing images of an area containing the items within the storage region; a pre-processing image acquisition unit for acquiring a pre-processing image captured by the camera unit; a first storage unit for storing the pre-processing image; a post-processing image generation unit for performing processing on the pre-processing image, such as deleting at least a portion of the area outside the item range or reducing the amount of information, to generate a post-processing image; a second storage unit for storing the post-processing image; and a deletion unit for deleting the pre-processing image after the post-processing image is generated.

[0006] The image management server of the present invention includes: a pre-processing image acquisition unit, which acquires a pre-processing image obtained by a shooting unit capturing an area of ​​items including a storage area; a first storage unit, which stores the pre-processing image; a post-processing image generation unit, which performs processing on the pre-processing image to delete at least a portion of the area outside the item area or reduce the amount of information, thereby generating a post-processing image; a second storage unit, which stores the post-processing image; and a deletion unit, which deletes the pre-processing image at least before deleting the post-processing image.

[0007] In the production image management system method of the present invention, the image management system comprises: a storage warehouse having a storage area for storing items and a camera unit for capturing images of an area containing the storage area; a pre-processing image acquisition unit for acquiring a pre-processing image captured by the camera unit; a first storage unit for storing the pre-processing image; a post-processing image generation unit for performing processing on the pre-processing image, such as deleting at least a portion of the area outside the item range or reducing the amount of information, to generate a post-processing image; a second storage unit for storing the post-processing image; and a deletion unit for deleting the pre-processing image after the post-processing image is generated. The production method of the image management system provides a control program for controlling the camera unit to the storage warehouse or a terminal used by a user.

[0008] Other means are described in the embodiments used to carry out the invention. Attached Figure Description

[0009] Figure 1 This is a block diagram of the refrigerator according to this embodiment.

[0010] Figure 2 This is a block diagram of two types of servers.

[0011] Figure 3 This is a block diagram of a portable terminal.

[0012] Figure 4 This is a logical block diagram of the image management system.

[0013] Figure 5 This is the front view of the refrigerator.

[0014] Figure 6 This is a side view of the refrigerator.

[0015] Figure 7 This is a front view of the refrigerator's left and right doors with the refrigerator compartment open.

[0016] Figure 8 This is a top view of the refrigerator's left and right doors with the refrigerator compartment open.

[0017] Figure 9 This is a 3D view of the refrigerator's camera unit taken from a slightly downward angle.

[0018] Figure 10 This is a sequence diagram of the first image processing step.

[0019] Figure 11 This refers to an image taken by the refrigerator's camera before processing.

[0020] Figure 12 This indicates that the pre-processed image will not require partial masking.

[0021] Figure 13 This represents an example of a processed image.

[0022] Figure 14 This refers to the selection screen displayed on the portable terminal.

[0023] Figure 15 The screen displays a warning message.

[0024] Figure 16 The user can then select a screen to continue the registration process.

[0025] Figure 17 This indicates a confirmation dialog box.

[0026] Figure 18 Pay attention to the image when indicating a transfer.

[0027] Figure 19 Pay attention to the image when indicating a transfer.

[0028] Figure 20 The image is used to draw attention to the scene.

[0029] Figure 21 This refers to the homepage screen.

[0030] Figure 22 The screen shows the contents of the refrigerator.

[0031] Figure 23 This is a sequence diagram of image processing according to the second embodiment. Detailed Implementation

[0032] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. In the image management system of this embodiment, Figure 1 Refrigerator 1 Figure 2 Servers 2a and 2b, and storage servers 6a and 6b Figure 3 The portable terminal 3 collaborates to provide services to the user. Refrigerator 1, servers 2a and 2b, storage servers 6a and 6b, and portable terminal 3 are each assigned a different IP address.

[0033] Reference Figure 1 The structure of refrigerator 1 will be described.

[0034] As a hardware structure, refrigerator 1 includes a camera unit 10, an arithmetic unit 112, a ROM (Read Only Memory) 113, a RAM (Random Access Memory) 114, a communication unit 115, and a camera LED (Light Emitting Diode) 100. Refrigerator 1 also possesses... Figure 1The left door 12, right door 13, door sensor 117, and refrigerator compartment 116 are shown.

[0035] The arithmetic unit 112 is, for example, a CPU (Central Processing Unit) that controls the refrigerator 1. RAM 114 is the operating area of ​​the arithmetic unit 112. Additionally, RAM 114 is a non-temporary or temporary recording medium that stores various programs and data, including programs not shown.

[0036] Camera unit 10 is a camera unit for photographing the refrigerator compartment 116, such as a fisheye camera or a panoramic camera. Camera unit 10 is mounted on the upper side of the refrigerator compartment 116. Camera LED 100 is an indicator that illuminates when camera unit 10 is taking pictures. Left door 12 and right door 13 are doors provided in the refrigerator compartment 116, and their opening and closing status is detected by door sensor 117. When the user opens left door 12 and / or right door 13, camera unit 10 operates to photograph the refrigerator compartment 116. Refrigerator compartment 116 is a storage compartment for storing items, primarily food.

[0037] The communication unit 115 is connected to the network and sends and receives data with external devices, namely server 2a.

[0038] Reference Figure 2 The structures of servers 2a and 2b and storage servers 6a and 6b are described. Servers 2a and 2b and storage servers 6a and 6b each have a CPU 21, ROM 22, RAM 23, storage unit 24, and communication unit 25.

[0039] CPU21 is used to control servers 2a and 2b. ROM22 is non-volatile memory, for example, storing the BIOS (Basic I / O system). RAM23 is the operating area of ​​CPU21. Additionally, RAM23 is a non-temporary or temporary recording medium for storing various programs and data, including programs not shown in the diagram.

[0040] Storage unit 24 is a non-temporary or temporary recording medium for storing various programs and data, such as an HDD (Hard Disk Drive) or flash memory. Furthermore, the storage unit 24 of storage servers 6a and 6b is preferably a high-capacity storage unit capable of high-speed access. Communication unit 25 is connected to a network and sends and receives data with external devices.

[0041] Reference Figure 3 The structure of the portable terminal 3 will be described. The portable terminal 3 includes a CPU 31, a ROM 32, a RAM 33, a storage unit 34, a communication unit 35, and a touch screen display 36.

[0042] CPU 31 provides comprehensive control over portable terminal 3. ROM 32 is non-volatile memory, such as storing the BIOS. RAM 33 serves as the operating area for CPU 31. Additionally, RAM 33 is a non-temporary or temporary recording medium for storing various programs and data, including programs not shown in the diagram.

[0043] Storage unit 34 is a non-temporary or temporary recording medium for storing various programs and data, such as flash memory. Communication unit 35 connects to a network and sends and receives data with external devices. Touchscreen display 36 is constructed by overlaying a touchscreen onto a flat display. Touchscreen display 36 is a display unit for displaying text, images, or graphics, and also an input unit for detecting user operations.

[0044] Figure 4 This is a logical block diagram of the image management system.

[0045] The pre-processing image acquisition unit 81 acquires the pre-processing image captured by the camera unit 10. This pre-processing image acquisition unit 81 is implemented by the CPU 21 of the server 2a executing a program.

[0046] The first storage unit 82 stores the image before processing. This first storage unit 82 is equivalent to the storage server 6a.

[0047] The post-processing image generation unit 84 performs processing on the pre-processing image, deleting at least a portion of the area outside the object's range or reducing the information content, to generate a post-processing image. Specifically, the post-processing image generation unit 84 deletes or reduces the information content within a predetermined range of the pre-processing image to generate a post-processing image. Furthermore, if any pre-processing image contains a human body portion, the post-processing image generation unit 84 deletes or reduces the information content within that human body portion to generate a post-processing image. This post-processing image generation unit 84 is implemented by a program executed by the CPU 21 of the server 2b.

[0048] The deletion unit 83 deletes the image before processing after the processed image is generated. Then, the deletion unit 83 deletes the image before processing at least before deleting the processed image. This is implemented by the CPU 21 of server 2a executing a program.

[0049] The second storage unit 85 stores the processed image. This second storage unit 85 is equivalent to the storage server 6a.

[0050] Registration unit 86 registers users who can view processed images by associating them with the individual ID of the storage or shooting unit. Registered user information prompting unit 87 prompts a newly registered user with information related to already registered users when the user is newly registered with an individual ID.

[0051] When a new user is registered and associated with the individual ID, the registered user permission deletion unit 88 prompts the new user to select an already registered user and deletes the viewing permissions of the selected registered user. The registration unit 86, the registered user information prompt unit 87, and the registered user permission deletion unit 88 are implemented by a program executed by the CPU 31 of the portable terminal 3.

[0052] Figure 5 This is the front view of refrigerator 1.

[0053] Refrigerator 1 is a device for cooling food and other items, and includes a left door 12, a right door 13, and a camera unit 10 for a refrigerator compartment 116. Refrigerator 1 has multiple storage compartments inside its body. The storage compartments of refrigerator 1 are, from top to bottom, a refrigerator compartment 116, an ice-making compartment 14 arranged side-by-side, an upper freezer compartment 15, a vegetable compartment 16, and a lower freezer compartment 17.

[0054] The refrigerator 1 has a structure in which an outer steel sheet casing and an inner resin casing (not shown) are filled with an insulating material such as polyurethane foam (not shown). Multiple openings corresponding to each compartment are provided on the front side (front side) of the refrigerator 1. For example, when food is placed into the refrigerator compartment 116 through the opening, the left door 12 and right door 13 are opened. Conversely, when the left door 12 and right door 13 are closed, the opening of the refrigerator compartment 116 is sealed. Thus, the left door 12 and right door 13 have the function of sealing the opening of the refrigerator compartment 116. The same applies to the other doors.

[0055] As a French door forming the refrigerator compartment 116 together with the cabinet, the refrigerator 1 has a left door 12 and a right door 13. The left door 12 can be hinged at the left end by a hinge 121 (see reference). Figure 8 The refrigerator rotates around the axis of the refrigerator. The same applies to the right door 13. In addition, as drawer-type doors, the refrigerator 1 has an ice maker door, an upper freezer door, a vegetable compartment door, and a lower freezer door.

[0056] The refrigerator compartment 116 is provided with multiple partitions that divide the refrigerator compartment 116 in a predetermined manner. Multiple door shelves 123 for storing food and other items are provided inside the left door 12 (see reference). Figure 7 The same applies to the left door 12. An ice-making container (not shown) is provided in the ice-making compartment 14 and is pulled out integrally with the ice-making compartment door. Similarly, an upper freezer container (not shown) is provided in the upper freezer compartment 15, a vegetable compartment container (not shown) is provided in the vegetable compartment 16, and a lower freezer container (not shown) is provided in the lower freezer compartment 17.

[0057] Additionally, although not shown, refrigerator 1 includes a compressor, a radiator (condenser), a capillary tube (throttling mechanism), and an evaporator. Moreover, the refrigerant circulates sequentially through the compressor, radiator, capillary tube, and evaporator, cooling the air in the storage compartment through heat exchange with the refrigerant flowing in the evaporator. Figure 5 The camera unit 10 shown is installed on the upper surface of the refrigerator body to take pictures of the storage compartment of the refrigerator 1, etc.

[0058] Figure 6 This is a side view of refrigerator 1.

[0059] like Figure 6 As shown, the camera unit 10 includes a main body 101 and a support 102. The main body 101 has the function of taking pictures of the storage compartment, etc. The support 102 supports the main body 101 and is provided on the upper surface of the refrigerator body 1.

[0060] A lens 103 is provided near the front end of the main body 101 (see also...). Figure 6 Lens 103 is an optical element that refracts and focuses light. A fisheye lens is, for example, used as such a lens 103. And, in the left door 12 and / or the right door 13 (see...) Figure 5 When the camera is open, lens 103 faces downwards, allowing camera unit 10 to photograph the refrigerator compartment 116 (see reference). Figure 7 The camera unit 10 is installed in a location that may capture images of the living space area, including the living space other than the cold storage room 116.

[0061] like Figure 6 As shown, lens 103 is located further forward than the front end of the housing (the opening of the housing). More preferably, lens 103 is located further forward than the left door 12 and right door 13 in the closed state (see reference). Figure 5 The front surface of the refrigerator compartment 116 is positioned further forward. Therefore, for example, when the left door 12 and right door 13 of the refrigerator compartment 116 are open, the refrigerator compartment 116 easily enters the field of view of the lens 103. Furthermore, even with drawer-type doors such as the vegetable compartment door open, the camera unit 10 can also photograph the vegetable compartment 16, etc. (see reference). Figure 5 Thus, lenses 103 were installed at positions for viewing each storage room from above.

[0062] Figure 7 This is the front view of refrigerator 1 with its left door 12 and right door 13 open.

[0063] As described above, multiple shelves 123 for storing food and the like are provided on the inner panel 122 of the left door 12 (the same applies to the right door 13). Furthermore, when the left door 12 and the right door 13 are opened, the food and the like in the refrigerator compartment 116, the shelves 123 and 133 are viewed from above and enter the field of view of the lens 103 of the camera unit 10.

[0064] Figure 7 The rotating partition 124 shown is a partition used to suppress the leakage of cold air from the gap between the left door 12 and the right door 13. Figure 7 In the example, in the left door 12 with hinge 121 (see reference) Figure 8 A rotating separator 124 is provided at the end opposite to the axis of the left door 12, which rotates in a predetermined manner as the left door 12 is opened and closed.

[0065] Additionally, when manually photographing the refrigerator compartment 116 using the camera unit 10, a shutter button 125 pressed by the user is located on the left door 12, and a shutter button 135 is located on the right door 13. Figure 7 In the example, in the left door 12, the hinge 121 of the left door 12 (see reference) Figure 8 A shutter button 125 is located on the lower part of the opposite side (the same applies to the right door 13). From another perspective, when the left door 12 and right door 13 are closed (see...),... Figure 5 Below, on the surface opposite to one of the left door 12 and the right door 13 ( Figure 7 The shooting button 125 is set near the rotating separator 124.

[0066] The shooting performed when the user presses at least one of these shooting buttons 125 and 135 is called "manual shooting". In addition, the shooting performed by the calculation unit 112 based on the elapsed time since the timer started to open from the left door 12 and the right door 13 is called "automatic shooting".

[0067] Figure 8 This is a top view of the refrigerator 1 with its left door 12 and right door 13 open.

[0068] exist Figure 8 In the example, the main body 101 of the camera unit 10 is positioned slightly to the left of the center of the refrigerator 1 in the left-right direction. To explain in more detail, the main body 101 of the camera unit 10 is positioned directly above the seam between the left door 12 and the right door 13 when closed (see also...). Figure 5 and Figure 6 With the main body 101 configured in this way, for example when the left door 12 and the right door 13 are open, the door frames 123 and 133 (see reference) Figure 7 Food and other items can easily enter the field of view of the camera unit 10.

[0069] Furthermore, when using refrigerator 1, most users will be aware of the seam (boundary) between the left door 12 and the right door 13. Therefore, by arranging the main body 101 directly above the seam between the left door 12 and the right door 13, the user's sense of disharmony can be suppressed compared to the case where the position of the main body 101 in the left and right directions is offset from the seam.

[0070] Figure 9 This is a stereoscopic view of camera unit 10 viewed from a slightly downward angle.

[0071] Figure 9 The main body 101 of the camera unit 10 shown includes the lens 103 described above (see also...). Figure 6 In addition to the housing 31b, it also features a camera LED100 (Light Emitting Diode). Figure 6 In this example, the outer casing of the main body 101 is generally a long, narrow cuboid. A circular hole (not shown in the accompanying drawings) is provided on the lower surface near the front end of the outer casing of the main body 101, through which the lens 103 protrudes. In addition, a camera LED 100 is provided on the rear (inner) side of the lens 103.

[0072] The camera LED 100 is a light source that illuminates the refrigerator compartment 116, enabling the camera unit 10 to capture images of the refrigerator compartment 116 (see reference) with adequate brightness. Figure 7 ), gantry 123, 133 (refer to) Figure 7 Additionally, the camera LED100 (illumination unit) is installed in the refrigerator 1's casing (see reference). Figure 6 The outside of the refrigerator compartment 116. Details will be described later. In this embodiment, with the interior light of the refrigerator 1 off and the camera LED 100 on, the camera unit 10 takes pictures of the refrigerator compartment 116, etc. (see reference). Figure 7 ).

[0073] like Figure 9 As shown, by providing a lens 103 in front of the camera LED 100, when the left door 12 and right door 13 are open, light from the refrigerator compartment 116, etc. (see reference) Figure 5 The reflected light from the refrigerator easily enters the lens 31a. Therefore, it is possible to properly photograph food stored in the refrigerator compartment 116, etc. Furthermore, it is preferable that the camera LED 100 is located in a position relative to the refrigerator compartment 1 (see reference). Figure 6 The front end (the opening of the cabinet) is further forward. As a result, the light from the camera LED100 can easily enter the refrigerator compartment 116, making it easier to obtain clear shooting results.

[0074] The refrigerator 1 is a storage compartment with a refrigerator compartment 116 as a storage area for storing items and a camera unit 10 for taking pictures of the range of items including the refrigerator compartment 116.

[0075] Figure 10 This is a sequence diagram of image processing in the first embodiment.

[0076] When the user opens the door of refrigerator 1, the processing unit 112 detects that the door has been opened via the door sensor 117. Then, the processing unit 112 takes a picture of the refrigerator compartment 116 containing food via the camera unit 10 (step S10). The processing unit 112 uploads the captured camera image of the refrigerator compartment 116 to the server 2a (step S11). This camera image is the image before processing, allowing the user to view the image.

[0077] The CPU 21 of server 2a functions as a pre-processing image acquisition unit, which acquires the pre-processing image captured by camera unit 10. Server 2a sends the pre-processing camera image to storage server 6a (step S12).

[0078] When storage server 6a notifies server 2b of the event (step S13), it saves the camera image before processing (step S14). Storage server 6a functions as the first storage unit for storing the image before processing.

[0079] Then, when storage server 6a sends the unprocessed camera image to server 2b (step S15), it deletes the unprocessed camera image it has stored (step S16). That is, the CPU 21 of storage server 6a functions as a deletion unit for deleting the unprocessed camera image.

[0080] Server 2b corrects the distortion caused by the lens in the unprocessed camera image (step S17), and processes it into a front-view image through trapezoidal correction (step S18). Then, server 2b requests the existing processed image from storage server 6b (step S19) and receives the processed image from storage server 6b (step S20).

[0081] When server 2b combines the existing processed image with the trapezoidal-corrected image (step S21), it sends the combined processed image to storage server 6b for storage (step S22). The CPU 21 of server 2b functions as a processed image generation unit that performs processing on the image before processing, such as deleting at least a portion of the area outside the item range or reducing the amount of information, to generate the processed image.

[0082] Storage server 6b stores the processed image received from server 2b (step S23). Storage server 6b functions as a second storage unit for storing the processed image.

[0083] Then, storage server 6b sends the processed image to portable terminal 3 according to the request from portable terminal 3 (step S24). Thus, the customer can view the image of the refrigerator 116 through portable terminal 3.

[0084] Storage server 6a stores the image before processing, and storage server 6b stores the composited image after processing, thus facilitating the management of each image. The portable terminal 3 can easily obtain the composited image by simply accessing storage server 6b.

[0085] Portable terminal 3 does not access storage server 6a, and therefore does not know the IP address of storage server 6a. Thus, it does not know the method of access from portable terminal 3 to storage server 6a, thereby preventing the unintentional viewing by others of customers or their living environment in images taken before processing.

[0086] Storage server 6b deletes the processed image after a predetermined period of time has elapsed since it saved it. That is, before storage server 6b deletes the processed image, storage server 6a deletes the unprocessed image it has saved. This prevents unintentional viewing by others of customers or their living environment captured in the unprocessed image.

[0087] Figure 11 This image shows the pre-processing state, captured by the camera unit 10 of refrigerator 1. The lower half of the image shows the refrigerator compartment 116. The left side of the image shows the inside of the left door 12. The right side of the image shows the inside of the right door 13. Furthermore, the upper half of the image captures the customer's living environment. This portion is unnecessary when viewing the processed image.

[0088] Figure 12 This refers to the image before processing, after the unwanted parts have been covered up.

[0089] This system masks unwanted portions of fisheye images. The masking process involves deleting at least a portion of the area outside the object's view. Therefore, when viewing images taken inside a customer's refrigerator, it prevents others from unintentionally seeing the customer and their living environment.

[0090] In addition, the masking process for the unnecessary part is performed by server 2b, but it can also be performed by server 2a or refrigerator 1.

[0091] Figure 13 This represents an example of a processed image.

[0092] Server 2b corrects lens distortion and performs trapezoidal correction on the unprocessed image captured by camera unit 10 of refrigerator 1 to process it into a front-view image. At this time, when server 2b detects a face in the image, it performs mosaic processing on the area where the face was captured.

[0093] The mosaic processing performed by this system reduces the amount of information in at least a portion of the area outside the object's surface. This prevents unintentional exposure of the customer to others when viewing images taken inside the customer's refrigerator. While the face detection and mosaic processing are performed by server 2b, they can also be performed by server 2a or refrigerator 1 on the unprocessed image.

[0094] Figures 14 to 20 This refers to a series of screens used to confirm with the customer important information regarding the processing of camera images when the portable terminal 3 is paired with the refrigerator 1. These screens are displayed by the refrigerator management program installed on the portable terminal 3, which is executed by the CPU 31 of the portable terminal 3.

[0095] Figure 14 This refers to the new / used selection screen 41 displayed on the portable terminal 3.

[0096] The selection screen 41 includes a New Purchase button 411, an Other button 412, and a Back button 419.

[0097] The "New Purchase" button 411 is for newly purchased users. When this button is clicked, the portable terminal 3 will register the user after pairing with the refrigerator 1.

[0098] Other button 412 is for people who bought refrigerator 1 secondhand. When this button is clicked, the portable terminal 3 switches to... Figure 15 Warning screen 42.

[0099] The back button 419 is a button that takes you to the previous screen before the selected screen 41.

[0100] Figure 15 Warning screen 42.

[0101] Warning screen 42 is a screen that warns you of the precautions to take when refrigerator 1 is second-hand.

[0102] The confirmation button 421 is the button to confirm the end of the warning screen 42. When the confirmation button 421 is clicked, the portable terminal 3 performs user registration after pairing with the refrigerator 1.

[0103] The back button 429 is the button that takes you to the previous screen before the warning screen 42.

[0104] User registration associates the identification number (individual number) of refrigerator 1 with the portable terminal 3. Furthermore, user registration can also be performed by associating the individual number of camera unit 10 with the portable terminal 3, not limited to the identification number (individual number) of refrigerator 1.

[0105] Then, after user registration, if there are already registered users, the process is transferred to... Figure 16 The user registration process continues on screen 43. After registration, if no user is already registered, the user is redirected to screen 43. Figure 18 When transferring, please pay attention to screen 45.

[0106] Figure 16 The user selects screen 43 to continue registration.

[0107] The "Continue Registration" screen 43 is for selecting users to continue registering with refrigerator 1. Checkboxes 431 to 433 are for selecting users to continue registering.

[0108] The "Confirm" button 435 is used to confirm the selection after a user has chosen to continue registration. By clicking the "Confirm" button 435, the portable terminal 3 switches to... Figure 17 The confirmation dialog box 44. The return button 439 is the button that redirects to the previous screen to continue the registration user selection screen 43.

[0109] Figure 17 This indicates a confirmation dialog box (44).

[0110] Confirmation dialog box 44 is used to confirm the cancellation of registration for users who do not wish to continue registering.

[0111] The "Yes" button 441 is used to cancel the registration of unselected users.

[0112] The "No" button 442 is a button that does not cancel the registration of unselected users. When the "Yes" button 441 or the "No" button 442 is clicked, the user is redirected to... Figure 14 When transferring, please pay attention to screen 45.

[0113] Figure 18 When indicating a transfer, please pay attention to screen 45.

[0114] When transferring the refrigerator, please note the following on screen 45. The confirmation button 451 indicates that the transfer of the refrigerator is being processed.

[0115] The return button 459 is the button to go to the previous screen when the transfer is performed (attention screen 45).

[0116] Figure 19 The image shows screen 46, indicating the precautions for using the camera.

[0117] The camera uses attention screen 46 to draw the customer's attention to the camera unit 10 of the refrigerator 1 taking pictures. When the screen is swiped to the bottom, a confirmation button on attention screen 47 is displayed to move on to the next image.

[0118] The back button 469 is the button that moves to the previous screen when the camera is viewed using the attention screen 46.

[0119] Figure 20 The image is shown using screen 47.

[0120] Image utilization note screen 47 is the utilization of images captured by the camera unit 10 of the refrigerator 1 to obtain the image promised by the customer.

[0121] The "Agree" button 471 is used by the customer to agree to the use of the image. When the "Agree" button 471 is clicked, a series of settings are completed and the user is redirected to the main screen 50.

[0122] The return button 479 is the button that redirects to the previous screen when the image is viewed using the attention screen 47.

[0123] Figure 21 This indicates that the homepage screen is 50.

[0124] The homepage screen 50 is the screen displayed after the refrigerator management program is started.

[0125] The refrigerator contents check button 501 is used to view the contents of the customer's refrigerator 1. When the refrigerator contents check button 501 is pressed, the portable terminal 3 is transferred to... Figure 17 The contents of the refrigerator (screen 51).

[0126] The "Instructions for Use" button 502 is used to explain how to use the refrigerator 1.

[0127] Figure 22 Screen 51 shows the contents of the refrigerator.

[0128] The refrigerator's content screen 51 is a screen used to view a front view of the refrigerator compartment 116 of the refrigerator 1.

[0129] The indoor selection button 512 is used to select which of the following is displayed: the inside of the refrigerator compartment 116, the inside of the left door 12, and the inside of the right door 13.

[0130] The back button 519 is the button that redirects to the previous home screen 50, which is the screen that redirects to the refrigerator's content screen 51.

[0131] Second Implementation Method

[0132] In the second embodiment, lens correction processing, trapezoidal correction processing, etc., are performed by the arithmetic unit 112 of the refrigerator 1, not by the server 2b. Since the camera image before processing is not sent to the network, it is possible to prevent the customer or the customer's living environment from being unintentionally seen by others.

[0133] Figure 23 This is a sequence diagram of image processing according to the second embodiment.

[0134] When the user opens the door of refrigerator 1, the arithmetic unit 112 detects that the door has been opened via the door sensor 117. Then, the arithmetic unit 112 takes a picture of the refrigerator compartment 116 containing food via the camera unit 10 (step S30). This camera image is a pre-processed image that the user can view.

[0135] The arithmetic unit 112 saves the camera image before processing (step S31). The arithmetic unit 112 corrects the distortion caused by the lens in the camera image before processing (step S32), and processes it into a front-view image through trapezoidal correction (step S33). Then, the arithmetic unit 112 deletes the camera image before processing (step S34). The arithmetic unit 112 requests the existing processed image from the storage server 6b (step S35), and receives the processed image from the storage server 6b (step S36).

[0136] When the processing unit 112 combines the existing processed image with the trapezoidal-corrected image (step S37), it sends the combined processed image to the server 2b (step S38). The server 2b sends the processed image to the storage server 6b for storage (step S39). The storage server 6b saves the processed image received from the server 2b (step S40). Then, according to a request from the portable terminal 3, the storage server 6b sends the processed image to the portable terminal 3 (step S41). Thus, the customer can view the image of the refrigerator 116 through the portable terminal 3.

[0137] Storage server 6b deletes the processed image after a predetermined period has elapsed since it was saved. That is, before storage server 6b deletes the processed image, storage server 6a deletes the unprocessed image it has saved. This prevents others from unintentionally seeing customers or their living environment captured in the unprocessed image.

[0138] This invention is not limited to the embodiments described above, but includes various modifications. For example, the embodiments described above are detailed for the purpose of easily understanding and illustrating the invention, and are not limited to having all the structures described. It is possible to replace a part of the structure of a certain embodiment with the structure of another embodiment, and it is also possible to add the structure of another embodiment to the structure of a certain embodiment. In addition, for a part of the structure of each embodiment, it is also possible to add, delete, or replace other structures.

[0139] For example, some or all of the above-mentioned structures, functions, processing units, and processing modules can be implemented using hardware such as integrated circuits. Alternatively, the above-mentioned structures and functions can be implemented in software by having a processor interpret and execute the programs used to implement each function. The programs, tables, files, and other information used to implement each function can be stored in recording devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as flash memory cards and DVDs (Digital Versatile Disks).

[0140] In each embodiment, control lines and information lines refer to the control lines and information lines required for the description, but not necessarily all control lines and information lines are represented on the product. In fact, it can be considered that almost all the structures are interconnected.

[0141] As variations of the present invention, there are, for example, variations as shown in (a) to (d).

[0142] (a) The mosaic treatment used to prevent customers and their living environment from being unintentionally seen by others can be performed by any one of refrigerator 1, server 2a and server 2b.

[0143] (b) The covering process used to prevent customers and their living environment from being unintentionally seen by others can be performed by any one of refrigerator 1, server 2a and server 2b.

[0144] (c) Mosaic processing and masking processing can be applied to either one or both.

[0145] (d) This invention is not limited to reading in the refrigerator compartment, but can also be applied to reading in any storage area of ​​a storage room.

[0146] Explanation of reference numerals in the attached figures

[0147] 1. Refrigerator (Storage Unit)

[0148] 100 camera LED

[0149] 10 camera units

[0150] 112 Computing Department

[0151] 113ROM

[0152] 114 RAM

[0153] 115 Department of Communications

[0154] 116 Refrigerator compartment (storage area)

[0155] 117-gate sensor

[0156] 12 Left Gate

[0157] 124 rotating separator

[0158] 13 Right Gate

[0159] 14 Ice-making room

[0160] 15 Upper Freezer Compartment

[0161] 16 Vegetable Room

[0162] 17 Lower Freezer Compartment

[0163] 2a Server (Image Acquisition Unit and Deletion Unit Before Processing)

[0164] 2b server (processed image generation unit)

[0165] 21 CPU

[0166] 22ROM

[0167] 23 RAM

[0168] 24 Storage Department

[0169] 25Ministry of Communications

[0170] 3 portable terminals

[0171] 31 CPU

[0172] 32ROM

[0173] 33RAM

[0174] 34 Storage Unit

[0175] 35 Ministry of Communications

[0176] 36 Touchscreen Display

[0177] 41 Select Screen

[0178] 411 Newly Purchased Button

[0179] 412 Other Buttons

[0180] 419 Back button

[0181] 42 warning screen

[0182] 429 Back button

[0183] 43. Registration continues with user selection screen.

[0184] 431 Checkbox

[0185] 432 checkbox

[0186] 433 checkbox

[0187] 435 Decision Button

[0188] 439 Back button

[0189] 44 Confirmation Dialog Box

[0190] 441 "Yes" button

[0191] 442 "No" button

[0192] 45. Transfer notification screen

[0193] 451 Confirm Button

[0194] 459 Back button

[0195] 46. ​​Camera Usage Notes

[0196] 469 Back button

[0197] 47. Image Utilization and Attention to the Picture

[0198] 479 Back button

[0199] 50 Homepage Screens

[0200] Refrigerator 501 Contents Check Button

[0201] 502 Usage Guide (Manager Button)

[0202] Contents of 51 Refrigerator

[0203] 512 Indoor Selection Button

[0204] 6a Storage Server (First Storage Unit)

[0205] 6b storage server (second storage unit)

[0206] 81 Image Acquisition Unit Before Processing

[0207] 82 First storage unit

[0208] 83 Deletion Unit

[0209] 84 Processed Image Generation Unit

[0210] 85 Second Storage Unit

[0211] 86 registration units

[0212] 87 Registered User Information Notification Unit

[0213] 88 Registered users have permission to delete this unit.

Claims

1. An image management system, characterized in that, have: A storage room, which has a storage area for storing items and a camera unit for taking pictures of the range of items including the storage area; The pre-processing image acquisition unit acquires the pre-processing image captured by the imaging unit; The first storage unit stores the image before processing; The post-processing image generation unit, when a face is detected in the pre-processing image, performs processing on the pre-processing image to reduce the amount of information in at least a portion of the area outside the object range, i.e., the area where the face was captured, to generate a post-processing image; The second storage unit stores the processed image; as well as The deletion unit deletes the unprocessed image after the processed image is generated.

2. The image management system according to claim 1, characterized in that, The camera unit is installed in a location that may capture images of the living space, including the living space outside the storage compartment. The deletion unit deletes the unprocessed image at least before deleting the processed image.

3. The image management system according to claim 1, characterized in that, The image management system also has: The registration unit registers users who can view the processed images by associating them with the individual ID of the storage unit or the camera unit; and The registered user information prompting unit prompts the user with information related to the already registered user when a new user is registered and associated with the individual ID.

4. The image management system according to claim 3, characterized in that, The image management system also includes a registered user permission deletion unit. When a new user is registered and associated with the individual ID, the registered user permission deletion unit allows the new user to select an already registered user to delete the viewing permission of the selected registered user.

5. The image management system according to claim 1, characterized in that, The IP address assigned to the first storage unit is different from the IP address assigned to the second storage unit.

6. An image management server, characterized in that, have: The pre-processing image acquisition unit acquires a pre-processing image obtained by the imaging unit from photographing the storage area containing the stored items. The first storage unit stores the image before processing; The post-processing image generation unit, when a face is detected in the pre-processing image, performs processing on the pre-processing image to reduce the amount of information in at least a portion of the area outside the object range, i.e., the area where the face was captured, to generate a post-processing image; The second storage unit stores the processed image; as well as A deletion unit that deletes the unprocessed image at least before deleting the processed image.

7. An image management method, characterized in that, One or more arithmetic units perform the following steps: The step of obtaining a pre-processing image by photographing the area of ​​the storage room containing the items, which is then captured by the photography department; The step of storing the image before processing; When a face is detected in the pre-processing image, the pre-processing image is processed to reduce the amount of information in at least a portion of the area outside the object range, i.e., the area where the face was captured, in order to generate a post-processing image. The step of storing the processed image; and The step of deleting the unprocessed image at least before deleting the processed image.

8. A method for producing an image management system, This image management system has the following features: A storage room, which has a storage area for storing items and a camera unit for taking pictures of the range of items including the storage area; The pre-processing image acquisition unit acquires the pre-processing image captured by the imaging unit; The first storage unit stores the image before processing; The post-processing image generation unit, when a face is detected in the pre-processing image, performs processing on the pre-processing image to reduce the amount of information in at least a portion of the area outside the object range, i.e., the area where the face was captured, to generate a post-processing image; The second storage unit stores the processed image; as well as The deletion unit deletes the unprocessed image after the processed image is generated. Its features are, The image management system is produced by providing a control program for controlling the camera unit to the storage unit or the terminal used by the user.

Images