Refrigerator management device, refrigerator management method, and refrigerator management program

Abstract

The invention provides a refrigerator management device, a refrigerator management method, and a refrigerator management program, which can easily generate a list of food materials stored in a refrigerator. A server constituting a refrigerator management device is characterized by being provided with: an image acquisition unit (80) for acquiring an image of a storage compartment of a refrigerator; a food material recognition unit (81) that recognizes each food material from the image; a range specifying unit (82) capable of specifying a partial range of the image; and a control unit (84) that updates the food material list (85) on the basis of information of food materials included in the range acquired by the range specification unit (82) among the food materials recognized by the food material recognition unit (81).

Description

Technical Field

[0001] The present invention relates to a cold storage management device, a cold storage management method, and a cold storage management program. Background Art

[0002] As the background art of this technical field, the abstract of Patent Document 1 describes the following: "In the cold storage management device 20, there is provided a storage chamber image storage control unit 938 that stores, in the image storage unit 912, an image obtained from the imaging unit 26 that captures an image as an in-storage-chamber image associated with the storage chambers 61 to 64 designated by the storage chamber designation unit 936."

[0003] In addition, the abstract of Patent Document 2 describes the following: "It functions as an imaging control unit 111, an extraction unit 112, and a display control unit 113. Among them, the imaging control unit 111 generates an image of foodstuffs, that is, a foodstuff image PC, at regular intervals; the extraction unit 112 extracts, each time the imaging unit 13 generates a foodstuff image PC, candidates for the foodstuff name FN corresponding to the foodstuffs, that is, foodstuff name candidates CFN, based on the foodstuff image PC; the display control unit 113 causes the foodstuff image PC to be displayed on the LCD 121 each time the imaging unit 13 generates a foodstuff image PC, and causes the foodstuff name candidates CFN to be displayed as the foodstuff name FN on the LCD 121 when a predetermined condition is satisfied."

[0004] Patent Document 3 describes an invention of a cold storage system that can display, together with an image of the cold storage before the change, an image indicating the position of the change when the contents stored in the cold storage have changed.

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2021-117572

[0008] Patent Document 2: Japanese Unexamined Patent Application Publication No. 2022-81963

[0009] Patent Document 3: Japanese Unexamined Patent Application Publication No. 2020-106257 Summary of the Invention

[0010] Technical Problem to be Solved by the Invention

[0011] According to the invention described in Patent Document 1, it is possible to easily generate a list of foodstuffs stored in a cold storage. However, the device of this invention does not have an image recognition function, so it requires complicated operations for the user. In addition, if only the image of the storage chamber is simply recognized, there is also a problem that manual operations must be performed simultaneously when the reliability of image recognition is low.

[0012] In addition, according to the invention described in Patent Document 2, when registering multiple food ingredients that appear within a certain range, it is necessary to select them by touching each one individually, and there is room for improvement in operability.

[0013] According to the invention described in Patent Document 3, when the number or area of regions where there are differences between the new image and the previous image in the refrigerator reaches or exceeds a specified value, the differences are emphasized and displayed or the difference information is cropped and displayed, thereby enabling detection of food ingredients that have been left untouched for a long time. In the invention described in Patent Document 3, since both the new and old images for comparison show the entire interior of the storage, the processing efficiency is low. In addition, in the invention described in Patent Document 3, when a food ingredient that has been taken out once is re-stored in another location, it may be counted as a difference, and there is room for improvement.

[0014] Therefore, the problem of the present invention is to be able to easily generate a list of food ingredients stored in a refrigerator.

[0015] Technical solutions for solving the problem

[0016] To solve the above problems, a refrigerator management device according to the present invention is characterized by including: an image acquisition unit that acquires an image of a storage room of a refrigerator; a food ingredient recognition unit that recognizes each food ingredient from the image; a range specification unit that can specify a partial range of the image; and a control unit that updates a food ingredient list based on information of food ingredients included in the range obtained by the range specification unit among the food ingredients recognized by the food ingredient recognition unit.

[0017] A refrigerator management method according to the present invention is characterized by including: a step of acquiring an image of a storage room of a refrigerator by an image acquisition unit; a step of recognizing each food ingredient from the image by a food ingredient recognition unit; a step of being able to specify a partial range of the image by a range specification unit; and a step of updating a food ingredient list by a control unit based on information of food ingredients included in the range specified by the range specification unit among the food ingredients recognized by the food ingredient recognition unit.

[0018] A refrigerator management program according to the present invention causes a computer to execute the following steps: a step of acquiring an image of a storage room of a refrigerator; a step of recognizing each food ingredient from the image; a step of being able to specify a partial range of the image; and a step of updating a food ingredient list based on information of food ingredients included in the range among the food ingredients recognized from the image.

[0019] Other technical solutions are described in the detailed implementation section.

[0020] Advantages of the invention

[0021] According to the present invention, a list of food ingredients stored in a refrigerator can be easily generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a structural diagram of a refrigerator management system.

[0023] Figure 2 is a module diagram of a refrigerator

[0024] Figure 3 is a module diagram of two types of servers.

[0025] Figure 4 is a module diagram of a portable terminal.

[0026] Figure 5 is a logical module diagram of an image management system.

[0027] Figure 6 is a front view of a refrigerator.

[0028] Figure 7 is a side view of a refrigerator.

[0029] Figure 8 is a front view of a refrigerator with the left and right doors of the refrigerator compartment opened.

[0030] Figure 9 is a top view of a refrigerator with the left and right doors of the refrigerator compartment opened.

[0031] Figure 10 is a perspective view of a refrigerator camera when viewed obliquely from below upwards.

[0032] Figure 11 is a diagram showing a pre-processing image taken by a refrigerator camera.

[0033] Figure 12 is a diagram showing the pre-processing image after masking unnecessary parts.

[0034] Figure 13 is a diagram showing an example of a post-processing image.

[0035] Figure 14 is a flowchart of a process for updating a food ingredient list.

[0036] Figure 15 is the food ingredient list / upper screen of the "Refrigerator Butler Application".

[0037] Figure 16 is a screen for selecting a listing method.

[0038] Figure 17 is a screen for selecting the range of a storage room image.

[0039] Figure 18 It is a temporary list screen of food ingredients.

[0040] Figure 19 It is the updated food ingredient list / upper screen.

[0041] Figure 20 It is the food ingredient list / upper screen.

[0042] Figure 21 It is the list method selection screen.

[0043] Figure 22 It is the home screen.

[0044] Figure 23 It is the internal item inspection screen of the refrigerator.

[0045] Figure 24 It is the internal item inspection screen of the refrigerator with a specified range.

[0046] Figure 25 It is a temporary list screen of food ingredients.

[0047] Figure 26 It is the updated food ingredient list / upper screen.

[0048] Figure 27 It is the display location selection screen.

[0049] Figure 28 It is the range selection screen.

[0050] Figure 29 It is a diagram showing the tap action of the text part of the temporary list screen.

[0051] Figure 30 It is a diagram showing the swipe action of the temporary list screen.

[0052] Figure 31 It is the imported image information screen.

[0053] Figure 32 It is a flowchart of the judgment process for staying food ingredients.

[0054] Figure 33 It is an example of a timing diagram for detecting long-term staying items.

[0055] Figure 34 It is another example of a timing diagram for detecting long-term staying items.

[0056] Figure 35 It is a diagram showing the push notification dialog box for staying food ingredients.

[0057] Figure 36 It is the home screen of the "Refrigerator Butler Application".

[0058] Figure 37 It is a temporary list screen of food ingredients.

[0059] Figure 38 It is an imported image information screen.

[0060] Explanation of reference numerals

[0061] 1: Refrigerator, 2: Server (refrigerator management device), 3: Portable terminal, 114: RAM, 100: Camera lighting unit, 12: Left door, 117: Door sensor, 116: Refrigerating chamber, 21: CPU, 22: ROM,, 23: RAM, 25: Communication unit, 31: CPU, 31b: Housing, 31a: Lens, 32: ROM, 33: RAM, 34: Storage unit, 35: Communication unit, 36: Touch panel display, 40, 40a, 40b, 40c: Food ingredient list / upper screen, 402: Thumbnail, 403: Product name, 405: Remaining days, 406: "+" button, 41, 41b: List method selection screen, 411: "Image input from refrigerator camera" button, 412: "Input food ingredient image with camera" button, 413: "Voice input food ingredient name" button, 414: "Text input food ingredient name" button, 42, 42b, 42c, 42d: Range selection screen, 421: Storage room image, 422: Range selection frame, 423: Indoor selection button, 424: List import button, 425: Food ingredient detection button, 43, 43b, 43c, 43d: Temporary list screen, 431: Thumbnail, 432: Product name, 433: Storage location, 435: List append button, 436: "Import from other images" button, 437: Delete button, 438: List append button, 439: Input area, 46, 46b: Home screen, 461: Notification area, 462: "Check the contents inside the refrigerator" button, 463: Usage housekeeper button, 461b: Notification area, 464: "View details" link, 491: Home button, 492: Food ingredient management button, 493: Operating status button, 494: Help button, 495: Setting button, 52: Display location selection screen, 521: Left door, 522: Storage room, 523: Right door, 524: Upper vegetable room, 525: Lower freezer, 526: Cancel button, 54b: Import image information screen, 54: Import image information screen, 541: Cropped image, 542: Product name, 543: Storage room image, 544: Crop frame, 551: Push notification dialog box, 55: Sleep screen, 80: Image acquisition unit, 81: Food ingredient recognition unit, 82: Range specification unit, 83: Temporary list presentation unit, 84: Control unit, 85: Food ingredient list, 86: Official list presentation unit, 87: Food ingredient image extraction unit, 851: Stored food ingredient list, 88: Warning unit, 14: Ice making room, 15: Upper freezer, 16: Vegetable room, 17: Lower freezer, 123: Door shelf, 101: Main body, 102: Support part, 103: Lens, 122: Inner plate, 133: Door shelf, 124: Rotary partition, 125: Shooting button, 135: Shooting button. Detailed implementation

[0062] Hereinafter, embodiments for implementing the present invention will be described in detail with reference to the respective drawings.

[0063] Figure 1 It is a structural diagram of a cold storage management system.

[0064] In the cold storage management system of the present embodiment, the cold storage 1, the server 2, and the portable terminal 3 cooperate with each other to provide services to the user. In addition, the cold storage 1, the server 2, and the portable terminal 3 are connected to the Internet and are respectively assigned different IP addresses. In the present embodiment, the server 2 is a computer and operates as a cold storage management device for managing the cold storage 1.

[0065] Refer to Figure 2 The structure of the cold storage 1 will be described.

[0066] As a hardware structure, the cold storage 1 includes a cold storage camera 10, an arithmetic unit 112, a ROM (Read Only Memory) 113, a RAM (Random Access Memory) 114, a communication unit 115, and a camera lighting unit 100. The cold storage 1 also includes a left door 12, a right door 13, a door sensor 117, and a cold storage chamber 116.

[0067] The arithmetic unit 112 is, for example, a CPU (Central Processing Unit), which comprehensively controls the cold storage 1. The RAM 114 is a working area of the arithmetic unit 112. In addition, the RAM 114 is a non-temporary or temporary recording medium for storing various programs and data, and can store programs (not shown).

[0068] The cold storage camera 10 is an imaging unit that captures the cold storage chamber 116 and other storage chambers, and is, for example, a fish-eye camera or a panoramic camera. In the present embodiment, the cold storage camera 10 is installed above the cold storage chamber 116, but it can also be installed inside the cold storage chamber 116, on the left door 12, or on the right door 13, for example. Instead of or in addition to the cold storage camera 10, a camera (not shown) provided in the portable terminal 3 prepared by the user can also be used. The camera lighting unit 100 is an indicator light that emits light when the cold storage camera 10 takes a picture. The left door 12 and the right door 13 are doors provided on the cold storage chamber 116, and their opening and closing states are detected by the door sensor 117. When the user opens the left door 12 or / and the right door 13, the cold storage camera 10 operates to capture the cold storage chamber 116. The cold storage chamber 116 is a storage for storing items represented by food.

[0069] The communication unit 115 is connected to the network and exchanges data with the server 2 as an external device.

[0070] Refer to Figure 3Describe the structure of Server 2. Server 2 has a CPU 21, a ROM 22, a RAM 23, a storage unit 24, and a communication unit 25, respectively.

[0071] The CPU 21 comprehensively controls Server 2. The ROM 22 is a non-volatile memory that stores, for example, the BIOS (Basic Input / Output System). The RAM 23 is the working area of the CPU 21. In addition, the RAM 23 is a non-transitory or transitory recording medium for storing various programs and data, and can store programs (not shown).

[0072] The storage unit 24 is a non-transitory or transitory recording medium for storing various programs and data, and is composed of, for example, an HDD (Hard Disk Drive), a flash memory, etc. The storage unit 24 stores a refrigerator management program 241 for managing the refrigerator 1. In addition, the storage units 24 of the storage servers 6a and 6b are preferably large-capacity and can be accessed at high speed. The communication unit 25 is connected to the network and exchanges data with external devices.

[0073] Refer to Figure 4 Describe the structure of the portable terminal 3. The portable terminal 3 has a CPU 31, a ROM 32, a RAM 33, a storage unit 34, a communication unit 35, and a touch panel display 36, respectively.

[0074] The CPU 31 comprehensively controls the portable terminal 3. The ROM 32 is a non-volatile memory that stores, for example, the BIOS. The RAM 33 is the working area of the CPU 31. In addition, the RAM 33 is a non-transitory or transitory recording medium for storing various programs and data, and can store programs (not shown). In addition, the information processing performed by the arithmetic unit 112, the CPU 21, the CPU 31, etc. can be performed by any one of them within the range without obstacles. For example, in the present embodiment, the processing described as being performed by the CPU 21 can also be performed by the arithmetic unit 112 or the CPU 31 if there is no obstacle from the perspective of the entire system.

[0075] The storage unit 34 is a non-transitory or transitory recording medium for storing various programs and data, and is composed of, for example, a flash memory, etc. The communication unit 35 is connected to the network and exchanges data with external devices. The touch panel display 36 is formed by laminating a touch panel on a planar display. The touch panel display 36 is a display unit for displaying characters, images, or graphics, and is also an input unit for detecting user operations.

[0076] Figure 5 It is a logical module diagram of the image management system.

[0077] The image acquisition unit 80 acquires an image of the storage room of the refrigerator 1 captured by the refrigerator camera 10. This image acquisition unit 80 is specifically implemented by the CPU 21 of the server 2 executing the refrigerator management program 241.

[0078] The food ingredient recognition unit 81 recognizes the presence of each food ingredient from the image. The food ingredient recognition unit 81 may also recognize information such as the name of the food ingredient and the position of the food ingredient.

[0079] The range specification unit 82 can specify a partial range of the image. A partial range of the captured image of the storage room specified by the range specification unit 82 may also be specified in advance. The food ingredient recognition unit 81 recognizes each food ingredient from the image before the range is specified by the range specification unit 82.

[0080] The temporary list presentation unit 83 presents the images of the food ingredients recognized by the food ingredient recognition unit 81 and / or the recognized food ingredient variety names that are included in the above range specified by the range specification unit 82 in the display unit.

[0081] The control unit 84 adds the food ingredient variety name selected by the user among the food ingredients recognized by the food ingredient recognition unit 81 and presented by the temporary list presentation unit 83 and / or the recognized food ingredient variety names to the food ingredient list 85 and updates it.

[0082] The official list presentation unit 86 presents the images of the food ingredients registered in the food ingredient list 85 by the control unit 84 in the display unit.

[0083] The control unit 84 updates the food ingredient list 85 based on the information of the food ingredients included in the range obtained by the range specification unit 82.

[0084] The food ingredient image extraction unit 87 extracts the images of the respective food ingredients from the image acquired by the image acquisition unit 80. Then, the CPU 21 updates the staying food ingredient list 851.

[0085] The control unit 84 compares the images extracted by the food ingredient image extraction unit 87 with the images already registered in the staying food ingredient list 851, and if there is no image determined to be the same food ingredient in the staying food ingredient list 851, registers the food ingredient in the staying food ingredient list 851.

[0086] The control unit 84 compares the images extracted by the food ingredient image extraction unit 87 with the images already registered in the staying food ingredient list 851, and if there is an image among the images already registered in the staying food ingredient list 851 that is not consistent with any of the food ingredients extracted by the food ingredient image extraction unit 87, deletes the food ingredient from the staying food ingredient list 851.

[0087] Alternatively, for the image extracted by the food ingredient image extraction unit 87, the CPU 21 combines it with the cropping coordinate information used when extracting the image of the food ingredient and registers it in the remaining food ingredient list 851. When comparing the image extracted by the food ingredient image extraction unit 87 with the images registered in the remaining food ingredient list 851, the change in the coordinate information of the cropping position of the image is combined with the image of the food ingredient.

[0088] If there is a food ingredient in the remaining food ingredient list 851 whose remaining time exceeds the specified value, the warning unit 88 sends a warning message to the portable terminal 3 to display a warning. At this time, the "refrigerator concierge app" of the portable terminal 3 displays the warning to the user through a push notification or the home screen.

[0089] Figure 6 It is a front view of the refrigerator 1.

[0090] The refrigerator 1 is a device for cooling foods, etc., and includes a left door 12 and a right door 13 of the refrigerating chamber 116 and a refrigerator camera 10. The refrigerator 1 has a plurality of storage chambers inside its housing. As the storage chambers of the refrigerator 1, a refrigerating chamber 116, an ice-making chamber 14 arranged horizontally, an upper freezing chamber 15, a vegetable chamber 16, and a lower freezing chamber 17 are provided in order from top to bottom. In addition, the arrangement of the vegetable chamber 16 and the lower freezing chamber 17 is not limited to this. As the storage chambers of the refrigerator 1, a refrigerating chamber 116, an ice-making chamber 14 arranged horizontally, an upper freezing chamber 15, and a lower freezing chamber 17 and a vegetable chamber 16 may be provided in order from top to bottom.

[0091] The housing of the refrigerator 1 is configured such that a heat insulating material (not shown) such as polyurethane foam is filled between a steel outer box and a resin inner box (not shown). A plurality of openings corresponding to each chamber are provided on the front side (front side) of the housing of the refrigerator 1. For example, when putting foods, etc. through the opening of the refrigerating chamber 116, the left door 12 and the right door 13 are opened. In addition, when the left door 12 and the right door 13 are closed, the opening of the refrigerating chamber 116 becomes a closed state. In this way, the left door 12 and the right door 13 have functions such as closing the opening of the refrigerating chamber 116. The same applies to the other doors.

[0092] As French doors that form the refrigerating chamber 116 together with the housing, the refrigerator 1 has a left door 12 and a right door 13. The left door 12 can rotate about the axis of the hinge 121 at the left end (refer to Figure 9 ). The same applies to the right door 13. In addition, in the refrigerator 1, as pull-out doors, in addition to the ice-making chamber door and the upper freezing chamber door, there are also a vegetable chamber door and a lower freezing chamber door.

[0093] In the refrigerating chamber 116, a plurality of shelves are provided to partition the refrigerating chamber 116 as specified. Inside the left door 12 on the storage side, a plurality of door shelves 123 for storing food and the like are provided (refer to Figure 8 ). The same applies to the left door 12. In the ice-making chamber 14, an ice-making chamber container that is pulled out integrally with the ice-making chamber door (not shown) is provided. Similarly, in the upper freezing chamber 15, an upper freezing chamber container (not shown) is provided, in the vegetable chamber 16, a vegetable chamber container (not shown) is provided, and in addition, in the lower freezing chamber 17, a lower freezing chamber container (not shown) is provided.

[0094] In addition, although not shown, the refrigerator 1 has a compressor, a radiator (condenser), a capillary tube (throttling mechanism), and an evaporator. And the refrigerant circulates successively through the compressor, the radiator, the capillary tube, and the evaporator, and the air in the storage chamber is cooled by heat exchange with the refrigerant flowing in the evaporator. Figure 6 The refrigerator camera 10 shown is a device for photographing the storage chamber and the like of the refrigerator 1, and is provided on the upper surface of the housing.

[0095] Figure 7 is a side view of the refrigerator 1.

[0096] As Figure 7 shown, the refrigerator camera 10 has a main body portion 101 and a support portion 102. The main body portion 101 has the function of photographing the storage chamber and the like. The support portion 102 supports the main body portion 101 and is provided on the upper surface of the housing of the refrigerator 1.

[0097] Near the front end of the main body portion 101, a lens 103 is provided (also refer to Figure 7 ). The lens 103 is an optical element that refracts and converges light. For example, a fish-eye lens is used as such a lens 103. And the lens 103 is in a state with the surface facing downward so that when the left door 12 or / and the right door 13 (refer to Figure 6 ) is opened, the refrigerator camera 10 can photograph the refrigerating chamber 116 (refer to Figure 8 ). The installation position of the refrigerator camera 10 enables it to photograph a living space range including the living space outside the refrigerating chamber 116.

[0098] As Figure 7 shown, the lens 103 is located on the front side relative to the front end (the opening of the housing). More preferably, the lens 103 is located at a position more forward than the front surfaces of the left door 12 and the right door 13 in the closed state (refer to Figure 6 ). Thereby, for example, when the left door 12 and the right door 13 of the refrigerating chamber 116 are opened, the refrigerating chamber 116 easily enters the field of view of the lens 103. In addition, in a state where a pull-out door such as a vegetable chamber door is opened, the vegetable chamber 16 and the like can also be photographed by the refrigerator camera 10 (refer to Figure 6)。In this way, the lens 103 is set at a position where it can look down on each storage compartment from above.

[0099] Figure 8 It is a front view of the state where the left door 12 and the right door 13 of the refrigerator compartment 1 are open.

[0100] As described above, a plurality of door shelves 123 for storing foods and the like are provided on the inner plate 122 of the left door 12 (the same applies to the right door 13). And when the left door 12 and the right door 13 are opened, the foods and the like in the refrigerator compartment 116 and the door shelves 123, 133 enter the field of view of the lens 103 of the refrigerator camera 10 from above.

[0101] Figure 8 The shown rotary partition 124 is a partition for suppressing the leakage of cold air from the gap between the left door 12 and the right door 13. In Figure 8 the example, a rotary partition 124 is provided at the end of the left door 12 on the side opposite to the axis of the hinge 121 (refer to Figure 9 ), and rotates according to regulations as the left door 12 opens and closes.

[0102] The refrigerator camera 10 automatically takes pictures when the user opens at least one of the left door 12 and the right door 13 of the refrigerator 1, any one of the vegetable compartment door or the lower freezer compartment door. It is preferable to take pictures after a specified time has elapsed since any door was opened.

[0103] In addition, the refrigerator camera 10 can also be used to manually take pictures of the refrigerator compartment 116 and the like. When taking pictures of the refrigerator compartment 116, a shooting button 125 for the user to press is provided on the left door 12, and a shooting button 135 is provided on the right door 13. In Figure 8 the example, the shooting button 125 is provided at the lower part of the surface of the left door 12 on the side opposite to the hinge 121 of the left door 12 (refer to Figure 9 ) (the same applies to the right door 13). In other words, in the state where the left door 12 and the right door 13 are closed (refer to Figure 6 ), the shooting button 125 is provided on the surface where one of the left door 12 and the right door 13 faces the other ( Figure 8 near the rotary partition 124).

[0104] The shooting performed when at least one of these shooting buttons 125, 135 is pressed by the user is called "manual shooting". In addition, the shooting performed by the arithmetic unit 112 based on the time elapsed since the left door 12 and the right door 13 started to open is called "automatic shooting".

[0105] Figure 9 It is a top view of the state where the left door 12 and the right door 13 of the refrigerator 1 are open.

[0106] In Figure 9 's example, in the left - right direction, the main body 101 of the refrigerator camera 10 is set at a position slightly to the left of the center of the housing of the refrigerator 1. More specifically, the main body 101 of the refrigerator camera 10 is arranged directly above the seam between the left door 12 and the right door 13 in the closed state (also refer to Figure 6 and Figure 7 ). By arranging the main body 101 in this way, for example, when the left door 12 and the right door 13 are opened, foods on the door shelves 123, 133 (refer to Figure 8 ) etc. can easily enter the field of view of the refrigerator camera 10.

[0107] In addition, when using the refrigerator 1, in most cases, the user will be aware of the seam (boundary part) between the left door 12 and the right door 13. Therefore, by arranging the main body 101 directly above the seam between the left door 12 and the right door 13, compared with the case where the position of the main body 101 in the left - right direction deviates from the seam, it is possible to suppress the user from feeling discomfort.

[0108] Figure 10 is a perspective view when looking at the refrigerator camera 10 from the obliquely downward direction.

[0109] Figure 10 The main body 101 of the refrigerator camera 10 shown not only has a lens 103 (refer to Figure 7 ), but also has a housing 31b and a camera illumination unit 100. In Figure 7 's example, the housing of the main body 101 is generally in the shape of a rectangular parallelepiped that is elongated in the front - rear direction. A circular hole (not shown in the reference numerals) is provided on the lower surface near the front end of the housing of the main body 101, and the lens 103 is exposed through this hole. In addition, the camera illumination unit 100 is provided on the rear side (inside) of the lens 103.

[0110] The camera illumination unit 100 is a light source that irradiates light to the refrigerating chamber 116 etc., so that the refrigerator camera 10 can photograph the refrigerating chamber 116 (refer to Figure 8 ) and the door shelves 123, 133 (refer to Figure 8 ) at an appropriate brightness. In addition, the camera illumination unit 100 (illumination unit) is provided outside the housing of the refrigerator 1 (refer to Figure 7 ). In this embodiment, in a state where the in - chamber lamp of the refrigerator 1 is turned off and the camera illumination unit 100 is turned on, the refrigerator camera 10 photographs the refrigerating chamber 116 etc. (refer to Figure 8 ), and the details will be described later.

[0111] As Figure 10As shown, by disposing the lens 103 on the front side of the camera illumination unit 100, in a state where the left door 12 and the right door 13 are open, the reflected light from the refrigerator compartment 116 etc. (refer to Figure 8 ) is likely to enter the lens 31a. Thus, the refrigerator camera 10 can appropriately photograph the foods stored in the refrigerator compartment 116 etc. Further, it is preferable that the camera illumination unit 100 is positioned on the front side with respect to the front end (opening of the housing) of the housing of the refrigerator 1 (refer to Figure 7 ). Thereby, the light irradiated from the camera illumination unit 100 is likely to enter the refrigerator compartment 116 etc., and thus it is easy to obtain a clear photographing result.

[0112] The refrigerator 1 is a storage, including a storage area for storing articles, i.e., the refrigerator compartment 116, and a refrigerator camera 10 for photographing the range of articles including the refrigerator compartment 116.

[0113] Figure 11 is a diagram showing an image before processing photographed by the refrigerator camera 10 of the refrigerator 1. The refrigerator compartment 116 is photographed in the lower half of the image. The door shelf 123 inside the left door 12 is photographed on the right side of the image. The door shelf 133 inside the right door 13 is photographed on the left side of the image. And, the living environment of the user is photographed in the upper half of the image. From the processed image, this part is an unnecessary part.

[0114] Figure 12 is a diagram showing the image before processing with the unnecessary part covered.

[0115] This system performs a masking process on the unnecessary part of the fish-eye image. The masking process executed by this system is a processing for deleting at least a part of the range outside the range of articles such as the refrigerator compartment 116 and the door shelves 123, 133. Thereby, when a person views the image obtained by photographing the inside of the user's refrigerator 1, it is possible to suppress the user and the user's living environment from being seen by others unintentionally.

[0116] Further, the masking process of this unnecessary part is executed by the server 2, but it may also be executed by the refrigerator 1.

[0117] Figure 13 is a diagram showing an example of the image after processing.

[0118] The server 2 performs lens distortion correction on the image before processing photographed by the refrigerator camera 10 of the refrigerator 1, and performs trapezoidal correction to process it into an image for frontal viewing. At this time, when a human face is detected in the image, the server 2 performs mosaic processing on the area where the human face is photographed.

[0119] The mosaic processing performed by this system is a processing that reduces the amount of information in at least a part of the range outside the object range. Thus, when a user views an image obtained by photographing the inside of the refrigerator 1, it is possible to prevent the user from being inadvertently seen by others. In addition, although the face detection processing and the mosaic processing are performed by the server 2, they may also be performed on the pre-processed image by the refrigerator 1.

[0120] 《First Embodiment》

[0121] In this embodiment, it is possible to specify a part of the range in the image obtained by photographing the inside of the refrigerator 1, and use the food ingredients within the range as the object to be listed. Thus, it is possible to easily generate a food list consisting only of the food ingredients that the user wishes to manage.

[0122] Figure 14 It is a flowchart of the process for updating the food ingredient list 85. In this process, a part of the range of the image is specified with a frame or the like, and the food ingredients within the range are added to the temporary list. This is the process when the CPU 21 of the server 2 executes the refrigerator management program 241.

[0123] First, Figure 5 the food ingredient recognition unit 81 performs image recognition of the food ingredients from the image captured by the refrigerator camera 10 (step S10). As the information recognized by the food ingredient recognition unit 81, at least the area where the food ingredients are captured and the area where no food ingredients are captured in the captured image are recognized. The timing for the food ingredient recognition unit 81 to perform image recognition is preferably immediately after the refrigerator camera 10 captures the image, or before the list method selection screen 41 is displayed on the portable terminal 3. Thus, when the user wants to select the required range, the food ingredient recognition unit 81 has already completed the image recognition, so the time taken from the selection of the range to the display of the list of the food ingredients included in the range can be shortened. Figure 16

[0124]

[0124] Alternatively, the food ingredient recognition unit 81 may perform image recognition after the user selects the required range. In this case, since the food ingredient recognition unit 81 does not perform unnecessary image recognition, the computational load on the CPU 21 of the server 2 can be reduced. In addition, the information recognized by the food ingredient recognition unit 81 may also include the name of the food ingredient, the name of the storage room where the food ingredient is stored, the coordinate information in the storage room, etc.

[0125] After that, when the user taps the “+” button 406 in the Figure 15 food ingredient list / upper screen 40 (step S11), the Figure 16 list method selection screen 41 is displayed on the portable terminal 3 (step S12).

[0126] Next, in Figure 16In the list-based method selection screen 41, the user taps the "Image Input from Refrigerator Camera" button 411 (step S13). As a result, the range specifying unit 82 causes the portable terminal 3 to display Figure 17 a range selection screen 42 (step S14). In this range selection screen 42, a storage room image 421 obtained by the refrigerator camera 10 photographing the storage room of the refrigerator 1 is displayed. The storage room image 421 is preferably one of the latest images obtained by the refrigerator camera 10 photographing the storage room. However, for example, any image specified by the user from among multiple recent captured images may also be displayed. The images captured by the refrigerator camera 10 are tagged in a manner that enables differentiation for each captured storage room, and preferably, the latest image of the storage room specified by the indoor selection button 423 is displayed.

[0127] When the user selects any range of the storage room image 421 obtained by photographing the storage room of the refrigerator 1 (step S15), the temporary list presenting unit 83 generates a temporary list of the ingredients within the range selected by the range selection box 422 and causes the portable terminal 3 to display Figure 18 a temporary list screen 43 (step S16). When, for example, a thumbnail 431 of this ingredient is selected and the list addition button 435 is tapped (step S17), the ingredient list 85 is updated by the control unit 84, and the ingredient list / upper screen 40a is updated as Figure 19 shown (step S18), Figure 14 and the process ends.

[0128] Through this process, the user can freely determine the space for placing the ingredients to be added to the ingredient list 85 according to their preferences, and can easily register the ingredients in the ingredient list 85. Regarding the range for extracting the ingredients to be added to the ingredient list 85, in addition to the user setting it each time in step S14, it may also be configured to automatically restore the range selection box 422 set by the user in the past, for example, in step S14, or it may be set by default. That is, the initial setting of the range can be changed.

[0129] In addition, the system can perform image recognition on all the ingredients captured in the captured image before the user specifies the range. As a result, the ingredient list can be updated immediately after the user specifies the range. It is not necessary to list the entire image, but the operability in the case of listing multiple ingredients together can be improved.

[0130] Figure 15 is the ingredient list / upper screen 40 of the "Refrigerator Butler Application". Each of the following screens shown is displayed on the touch panel display 36 of the portable terminal 3.

[0131] The ingredient list / upper screen 40 is displayed in a list form Figure 5The screen of the ingredient list 85 in the refrigerated storage 1 shown. Each ingredient is represented by a thumbnail 402, a product name 403, and the remaining days 405. The thumbnail 402 is a small image representing the appearance of the ingredient, and is an image obtained by cropping the entire area where the ingredient is recognized or a part thereof. The product name 403 represents the product name of the food. The remaining days 405 is the number of remaining days until the target date when the ingredient is used up. The target date when the ingredient is used up can be determined according to the shelf life of each ingredient, or can be arbitrarily set by the user.

[0132] Also, a "+" button 406 is shown at the lower side of the list. This "+" button 406 is a button for updating, adding, and deleting the ingredient list 85.

[0133] At the lower part of the ingredient list / upper screen 40, a home button 491, an ingredient management button 492, an operation status button 493, a help button 494, and a settings button 495 are shown.

[0134] The home button 491 is a button for transferring to the home screen of the application, and is highlighted when the home screen is being displayed. The ingredient management button 492 is a button for transferring to the ingredient management screen that displays the ingredient list, and is highlighted when the ingredient management screen is being displayed. The operation status button 493 is a button for transferring to the screen showing the operation status of the refrigerated storage 1, and is highlighted when the screen showing the operation status is being displayed.

[0135] The help button 494 is a button for transferring to the screen that helps the users of the application, and is highlighted when the help screen is being displayed. The settings button 495 is a button for transferring to the screen for setting the application, and is highlighted when the settings screen is being displayed. These buttons are commonly displayed at the lower part of the screen of the application.

[0136] Figure 16 It is the list method selection screen 41.

[0137] The list method selection screen 41 is Figure 15 with the ingredient list / upper screen 40 shown as the background and is displayed at a layer in front of it. On the list method selection screen 41, a "Image input from the refrigerated storage camera" button 411, a "Ingredient image input with the camera" button 412, a "Ingredient name voice input" button 413, and a "Ingredient name text input" button 414 are shown.

[0138] Here, when the user taps the "Image input from the refrigerated storage camera" button 411, it transfers to Figure 17The range selection screen 42. When the user taps the "Input food material image with camera" button 412, it transfers to a camera shooting screen (not shown). When the user taps the "Voice input food material name" button 413, it transfers to a voice input screen (not shown). When the user taps the "Text input food material name" button 414, it transfers to a text input screen (not shown).

[0139] Figure 17 is the range selection screen 42 of the storage room image 421.

[0140] In the range selection screen 42, the storage room image 421, the range selection frame 422, the indoor selection button 423, and the food material detection button 425 are displayed. The storage room image 421 is an image obtained by the refrigerator camera 10 shooting any storage room of the refrigerator 1.

[0141] The range selection frame 422 is a frame for selecting the range in the storage room image 421 where the food materials to be registered in the food material list 85 are stored. The indoor selection button 423 is a button for determining which of the storage rooms of the refrigerator 1 to select. The food material detection button 425 is a button for detecting the food materials included in the range selection frame 422 and transferring to Figure 18 the temporary list screen 43. In addition, the range selection frame 422 can use a shape other than a quadrilateral to select the range, and multiple ranges can also be selected with the range selection frame 422 in one storage room image 421. Regarding the setting of the range, for example, the range can be freely set by a so-called pinch operation where the user touches the touch panel display 36 with multiple fingers at the same time, etc.

[0142] In addition to executing the partial selection mode of selecting a part of the storage room image 421 using the range selection frame 422 like this, it can also be executed separately from the all-selection mode of selecting the entire storage room image 421 without using the range selection frame 422. In addition, the range selection frame 422 can be configured to be able to select the entire image, or can be configured not to be able to select the entire image in order to clarify the difference from the all-selection mode.

[0143] Figure 18 is the temporary list screen 43 of the food materials.

[0144] In the temporary list screen 43, the temporary lists of each food material are displayed. Each food material is represented by a thumbnail 431, a product name 432, and a storage location 433. These food materials are a list of the food materials included in the range selection frame 422. In this embodiment, the product name 432 is defaultly displayed in the state of "other food materials" input in the temporary list, and the user follows the usage Figure 29Input the name of the food ingredient using the method described below. Additionally, when the food ingredient name is recognized by the food ingredient recognition unit 81, the recognized food ingredient name can be automatically input as the product name 432, and in this state, a temporary list is displayed.

[0145] The thumbnail 431 is a small image representing the appearance of the food ingredient, and is obtained by cropping the area where the food ingredient recognized by the food ingredient recognition unit 81 is photographed. The food ingredient recognition unit 81 can crop the entire food ingredient or only a part of the food ingredient. The product name 434 represents the product name of the food ingredient. The storage location 433 uses an icon to indicate the storage room where the food is detected. When the user selects a food by, for example, tapping the thumbnail 431 and then taps the list addition button 435 in the lower right, the food ingredient is added to the food ingredient list 85.

[0146] Figure 19 It is the updated food ingredient list / upper screen 40a.

[0147] The food list on the food ingredient list / upper screen 40a has been updated, and beverages, beer, cakes, and natto as other food ingredients have been added. It is indicated that it is newly added, for example, within the thumbnail of the most recently added food ingredient. For example, the "NEW" mark is displayed.

[0148] 《Other detailed operations》

[0149] Refer to the following Figures 20 to 30 , and explain other operations of specifying a range of a part of the food ingredient in the image with a frame or the like and adding the food ingredient within the range to the temporary list.

[0150] First, Figure 20 It is the food ingredient list / upper screen 40b.

[0151] This food ingredient list / upper screen 40b shows the same elements as Figure 15 the food ingredient list / upper screen 40 shown. Here, when the user taps the "+" button 406, it transfers to Figure 21 the list method selection screen 41b.

[0152] Figure 21 It is the list method selection screen 41b.

[0153] The list method selection screen 41b shows the same elements as Figure 16 the list method selection screen 41 shown. Here, when the user taps the "Image input from the refrigerator camera" button 411, it transfers to Figure 23 the range selection screen 42.

[0154] In addition, it can also be that when the user taps the "Image input from the refrigerator camera" button 411, it transfers to Figure 24The range selection screen 42c. In this case, the range selection box 422 of the range selection screen 42c can be displayed in a state where the entire storage room image 421 is selected, or can be displayed in an initial setting state of the range selection box 422 set by the user, or in a state where the range selection box 422 selected by the user in the past is restored.

[0155] Figure 22 is the home screen 46.

[0156] In the home screen 46, a notification area 461, an "Inspect the contents of the refrigerator" button 462, and a usage steward button 463 are displayed.

[0157] In the notification area 461, a dialog box of tomatoes as the current season's ingredients and "This month's recommended tomatoes" is displayed.

[0158] When the "Inspect the contents of the refrigerator" button 462 is tapped, it transfers to Figure 23 the range selection screen 42b that displays the storage room image 421 for inspecting the internal items.

[0159] When the usage steward button 463 is tapped, it transfers to a guidance screen of the usage method (not shown).

[0160] Figure 23 is the range selection screen 42b, which displays the storage room image 421 for inspecting the internal items of the refrigerator 1.

[0161] The range selection screen 42b displays the storage room image 421, an indoor selection button 423, and a list import button 424.

[0162] Here, the user can perform operations such as tapping and pinching on the storage room image 421, thereby transferring to Figure 24 the range selection screen 42c that displays the range selection box 422.

[0163] Figure 24 is the range selection screen 42c, which displays the internal item inspection image of the refrigerator 1 with a specified range.

[0164] The range selection screen 42c displays the range selection box 422 on the storage room image 421, and also displays the indoor selection button 423 and the food ingredient detection button 425. The range selection box 422 is a box that has selected a part of the storage room image 421 through the operation of the user. Whether a certain food ingredient is included and specified within the range selection box 422 is determined by whether the center of the figure (center) where the food ingredient is recognized is included within the range selection box 422. For example, in Figure 24Among them, for the plastic bottle-shaped food ingredient located at the upper right corner of the range selection box 422, a part of its right end is outside the range, but the center of the plastic bottle-shaped food ingredient is within the range. Therefore, the plastic bottle-shaped food ingredient is regarded as the selected food ingredient.

[0165] When the user taps the indoor selection button 423, it transfers to Figure 27 the display location selection screen 52. When the user taps the food ingredient detection button 425, it transfers to Figure 25 the temporary list screen 43b of the food ingredients.

[0166] Figure 25 This is the temporary list screen 43b of the food ingredients. Since the temporary list screen 43b of the food ingredients is a screen related to food ingredient management, the food ingredient management button 492 is highlighted and displayed.

[0167] In the temporary list screen 43b, the temporary lists of each food ingredient are displayed. Each food ingredient is represented by a thumbnail 431, a product name 432, and a storage location 433. These food ingredients are the lists of the food ingredients included within the range selection box 422. The temporary list can also be generated by sequentially specifying the range for multiple images as described below.

[0168] In the temporary list screen 43b, the "Import from Other Images" button 436 and the list append button 435 are also displayed. When the user taps the "Import from Other Images" button 436, it transfers to Figure 23 the range selection screen 42b. Here, when further performing a left-right swipe operation on the storage room image 421, the images taken before or after the currently displayed image of the selected storage room (here, the refrigerating chamber) are displayed.

[0169] On the other hand, when tapping the indoor selection button 423 and transferring to Figure 27 the display location selection screen 52, and then tapping any one of the lower freezer 525, upper vegetable compartment 524, right door 523, storage room 522, left door 521, which are the icons of the required storage rooms, it transfers again to, for example, Figure 23 the range selection screen 42b or Figure 28 the range selection screen 42d to display the latest images of the specified storage room. Then, when a range selection is made using the range selection box 422 and the food ingredient detection button 425 is tapped, the food ingredients included within the range selection box 422 are added to the temporary list.

[0170] After completing the addition of the food ingredients of the required multiple images to the temporary list, after selecting the food ingredient to be finally added to the food ingredient list 85 and tapping the list append button 435, multiple images of food can be added to the food ingredient list 85 at one time.

[0171] Figure 26 It is the ingredient list / upper screen 40c of the updated ingredients.

[0172] The ingredient list / upper screen 40c and Figure 15 The shown ingredient list / upper screen 40 have the same elements. This ingredient list / upper screen 40c shows the updated ingredient list 85, with drinks, beer, and cakes newly added, as well as natto as other ingredients. In this embodiment, in order to minimize the information displayed on the ingredient list / upper screen 40c and prevent the screen from being complicated, the information corresponding to the storage location 433 shown in the temporary list screen 43b is not displayed, but this information can also be displayed. In addition, it can also be set by the user whether to display the information corresponding to the storage location 433 in the ingredient list / upper screen 40c.

[0173] Figure 27 It is the display location selection screen 52.

[0174] In this display location selection screen 52, each storage room of the refrigerator 1 is selectively displayed. Specifically, the display location selection screen 52 schematically shows the left door 521, the storage room 522, the right door 523, the upper vegetable room 524, the lower freezer 525, and a cancel button 526 is shown below it. Here, when the right door 523 is tapped, it transfers to Figure 28 The shown range selection screen 42d, and the right door part is displayed as the storage room image 421. When the storage room 522 is tapped, it transfers to the range selection screen 42, and the storage room part is displayed as the storage room image 421.

[0175] Figure 28 It is the range selection screen 42d.

[0176] In the range selection screen 42d, the storage room image 421 representing the right door part is shown. A range selection box 422 is shown on the storage room image 421. In this state, when the user taps the ingredient detection button 425, it transfers to Figure 29 The temporary list screen 43c, and the recognition results of the ingredients included in the range selection box 422 are displayed as a temporary list.

[0177] Figure 29 It is a diagram showing the tapping action of the text part of the temporary list screen 43c.

[0178] The temporary list screen 43c shows drinks and beer, and a text input area 439 is shown below it. The user can arbitrarily set the names of the ingredients displayed on the temporary list screen 43c.

[0179] Figure 30 It is a diagram showing the sliding action of the ingredients on the temporary list screen 43d.

[0180] The temporary list screen 43d displays beverages, beer, cakes, and other food ingredients, and a list addition button 438 is displayed at its lower part. By tapping the list addition button 438, the content of the temporary list can be saved to the food ingredient list 85.

[0181] In addition, the row where the beer is located is shifted to the right due to a sliding operation, and a delete button 437 is displayed at the right end. By the user tapping the delete button 437, the beer as the corresponding food ingredient can be deleted from the temporary list. In addition, by the user double-tapping the thumbnail 431 of the cake, it is possible to transfer to Figure 31 the imported image information screen 54. These deletions or transfers (jumps) to the imported image information screen 54 can also be performed in the food ingredient list 85.

[0182] Figure 31 It is the imported image information screen 54.

[0183] The imported image information screen 54 displays a cropped image 541, a product name 542, a storage room image 543, and a crop frame 544 located above it, and transfers to this screen when the thumbnail 402 of the food ingredient list / upper screen 40 or the thumbnail 431 of the temporary list screen 43 is tapped. The cropped image 541 is an image of the cake that has been photographed and cropped. The product name 542 is displayed as "cake", and a string representing this food is displayed. The storage room image 543 is an image of the storage room where this food is stored. The crop frame 544 is a frame representing the range in which this food is recognized in the storage room image 543.

[0184] "Detection of Stagnation of Shelved Food Ingredients (Long-Term Storage Items)"

[0185] In the stagnation detection system of shelved food ingredients, when each food ingredient is cropped from the latest in-library image, it is compared with the cropped images in the stagnant food ingredient list, and the number of days elapsed since the food ingredient was stored in the library is calculated. Thereby, the stagnation detection process is simplified and the stagnation detection accuracy is improved.

[0186] In the present embodiment, in the case where the orientation of the food ingredient has changed due to re-storage or the like, it can be recognized as a different food. Thus, it is easy to detect food ingredients that the user has forgotten about and that have been left untouched for a long time. In other words, when the orientation has changed, it is considered that the user frequently uses this food ingredient and it is not recognized as a shelved food ingredient. The same judgment can also be made in the case where the coordinates (storage location) of the food ingredient have changed significantly due to re-storage or the like.

[0187] In addition, it is also possible to arbitrarily set a threshold for consistency determination in such a way that even if the storage location or orientation of the food ingredients changes due to the user re-storing them, etc., it is regarded as the same food ingredient. In this case, even for the food ingredients being used by the user, the total number of days stored in the refrigerator can be counted.

[0188] Figure 32 It is a flowchart of the determination process for residence.

[0189] Figure 32 The flowchart shows the steps when the CPU 21 of the server 2 executes the refrigerator management program 241.

[0190] The food ingredient recognition unit 81 recognizes all the food ingredients in the image, and the food ingredient image extraction unit 87 crops the range of the image related to the food ingredient (step S20). Then, the control unit 84 repeats the processing of steps S21 to S26 for all the cropped images.

[0191] The control unit 84 compares the image group in the list of staying food ingredients 851 with the cropped image, and searches in the list of staying food ingredients 851 to see if there is an image that is the same as the latest cropped image (step S22). Here, the list of staying food ingredients 851 is information inside the server 2.

[0192] In step S23, the control unit 84 determines whether this cropped image is the same as any one of the images in the list of staying food ingredients 851. If there is an image in the list of staying food ingredients that is the same as the cropped image (yes), the process proceeds to step S24, and the control unit 84 increments the number of days passed for this food ingredient in the list of staying food ingredients by 1.

[0193] If there is no image in the list of staying food ingredients 851 that is the same as the cropped image (no), the control unit 84 determines that it is a newly staying food ingredient and proceeds to step S25, and adds this food ingredient to the list of staying food ingredients 851.

[0194] Next, in step S26, if not all the cropped images have been processed yet, the process returns to step S21. If all the cropped images have been processed repeatedly, the process proceeds to step S27.

[0195] In step S27, the control unit 84 determines whether there is a food ingredient in the list of staying food ingredients 851 that has not been determined to be the same as any of the food ingredients in the cropped images. If there is a food ingredient in the list of staying food ingredients 851 that has not been determined to be the same as any of the food ingredients in the cropped images (yes), the control unit 84 determines that this food ingredient has been consumed and deletes it from the list of staying food ingredients 851, Figure 32 and the processing ends. If all the food ingredients in the list of staying food ingredients 851 are determined to be the same as the food ingredients in the cropped images (no), then Figure 32The processing is completed.

[0196] In addition, in this embodiment, the feature amounts of the food material images are compared with each other. However, when the food material identification unit 81 identifies the variety name together, the comparison can also be performed in combination with the variety name identified from the food material image. Thereby, the comparison accuracy can be improved. The determination of whether the images are the same or different is made based on the distances of multi-dimensional numerical feature vectors such as color and shape. By setting a relatively high threshold value for determining identity in the comparison, the food material identification unit 81 can identify different food materials when the orientation of the food material changes slightly due to re-storage or the like. Thereby, it is possible to manage only the food materials that have been left untouched for a long time because the user has forgotten their existence in the retention food material list 851.

[0197] Further, in this embodiment, when comparing the feature amounts of the food material images with each other, the information on the storage location (cropping position of the cropped image) of the food material in the warehouse is not considered. Therefore, by setting a relatively low threshold value for determining identity in the comparison, even if the storage location of the food material changes before and after re-storage and the orientation of the food material changes slightly, the food materials can be regarded as the same. Thereby, even for the food materials that the user is aware of and is using, the total number of days stored in the refrigerator can be used as the elapsed number of days for counting.

[0198] In addition, when comparing the feature amounts of the food material images with each other, the information on the storage location (cropping position of the cropped image) of the food material in the warehouse can also be considered. In this case, for example, when there are multiple food materials of the same type in the warehouse and the comparison is made for each individual food material of the same type, since the cropping positions of the cropped images are different, it can also be determined that they are different.

[0199] Figure 33 This is an example of a timing chart for detecting long-term retained items.

[0200] This timing chart shows, from top to bottom, the date, the camera shooting process, the food material image cropping process, the retention determination process, the state of the home screen, and the state of the push notification.

[0201] Each arrow in the date row represents each day. The camera shooting process is indicated by an inverted triangle indicating the execution of the process. For example, it is executed by automatic shooting when the user of the refrigerator 1 opens the door, or by manual shooting when at least one of the shooting buttons 125 and 135 is pressed by the user.

[0202] The food material image cropping process is indicated by an inverted triangle indicating the execution of the process. For example, it is executed each time the camera shooting process is performed.

[0203] The retention determination process is Figure 32The processing shown is executed at a specified time (e.g., 3:00 am every day) based on the information of the timing represented by an inverted triangle. The cropped image of the latest image captured before the specified time is used in this residence determination.

[0204] The determination process of residence is a process of comparing the cropped image of the latest image at the time of determination with the cropped image of the nearest image more than 24 hours ago. That is, in the case where there is a cropped image of the previous day, it is compared with the nearest cropped image among them. In the case where there is no cropped image of the previous day, it is compared with the nearest cropped image among the cropped images of the nearest date.

[0205] In this case, the number of days elapsed obtained by comparing with the previous captured image is added to the residence days of the food. In addition, in this case, since the image cropping process is performed every time a shot is taken, the image cropping process has been completed at the specified time. In the case where the server 2 undertakes the cropping processes of different refrigerators used by multiple users together, even if the specified times set by each user are concentrated at a specific time, it is possible to prevent the food material image cropping process executed by the server 2 from being concentrated at a specific time, and stable operation of the server 2 can be achieved.

[0206] The state of the home page screen and push notifications are used to detect long-term stored items (stored food materials) and push notifications to users. The push notification is performed when the number of residence days is equal to the specified number of days. For example, the user can set that if the residence days are 10 days or more, a reminder display is made on the home page screen 46b of the application as Figure 36 shown, and a push notification is sent to the portable terminal 3 as Figure 35 shown. In addition, the shelf life of the food material can also be used as the specified number of days.

[0207] Figure 34 is another example of the timing chart for the detection of long-term stored items.

[0208] This timing chart sequentially shows the date, camera shooting process, food material image cropping process, residence determination process, state of the home page screen, and state of push notifications from top to bottom.

[0209] Each arrow in the date row represents each day. The camera shooting process is represented by an inverted triangle indicating the implementation of the process, for example, when the user of refrigerator 1 opens the door.

[0210] The food material image cropping process is represented by an inverted triangle indicating the implementation of the process and is executed at a specified time (e.g., 3:00 am every day).

[0211] The residence determination process is Figure 32The processing shown is performed after the ingredient listing process based on the information of the timing represented by an inverted triangle. The cropped image of the latest image captured before a specified time is used in this residence determination. The residence determination process is a process of comparing the cropped image of the latest image at the time of determination with the cropped image of the nearest image more than 24 hours ago. That is, in the case where there is a cropped image from the previous day, it is compared with the nearest cropped image among them. In the case where there is no cropped image from the previous day, it is compared with the nearest cropped image among the cropped images of the nearest date. In this case, the system of the present embodiment adds the number of elapsed days obtained from the comparison with the previous captured image to the residence days of this food.

[0212] In addition, in this case, since only the captured image just before is cropped at the specified time, the number of times of the ingredient image cropping process is reduced. Therefore, the operation rate of the CPU 21 of the server 2 can be suppressed, and resources for performing any other process can be ensured.

[0213] The state of the home screen and the push notification are used to detect long-term stored items (stored ingredients) and push notifications to the user. In the case where the specified number of days is 10 days, if the residence days are 10 days or more, as Figure 36 shown, a reminder display described later is performed on the home screen 46b of the application, and as Figure 35 shown, a push notification is sent to the portable terminal 3.

[0214] In addition, the availability of the push notification can be turned on / off with the setting item "Notification of Detection of Long-Term Stored Items". The timing of the notification can also be set with the setting item "Notification at a Specific Time". The timing of the determination can also be set with the setting item "Once a Day · AM 3:00". The determination condition for "long-term stored items" can be set with "Determination Time for Detecting Long-Term Stored Items".

[0215] Figure 35 It is a diagram showing a push notification dialog box 551 for stored ingredients.

[0216] The push notification dialog box 551 for stored ingredients is displayed on the sleep screen 55. When a long-term stored item is detected, Figure 5 the warning unit 88 sends a push notification to the portable terminal 3. Then, the CPU 31 of the portable terminal 3 displays the push notification dialog box 551 on the touch panel display 36 by executing a refrigerated storage housekeeper application / program (not shown).

[0217] Figure 36 It is the home screen 46b of the refrigerated storage housekeeper application in the case where a long-term stored item (stored ingredient) is detected.

[0218] On the home screen 46b, a notification area 461b, a "Check the contents inside the refrigerator" button 462, and a usage housekeeper button 463 are displayed. A link 464 for "View Details" is displayed in the notification area 461b.

[0219] In the notification area 461b, boxed natto as a long-term storage item, a dialog box for "There are ingredients for long-term storage", and a reminder display including the "View Details" link 464 are displayed. When the "View Details" link 464 is tapped, it transfers to the temporary list screen 43e of the detained food.

[0220] Figure 37 It is the temporary list screen 43e of the detained ingredients.

[0221] In this temporary list screen 43e of the detained ingredients, a thumbnail 431 and a product name 432 are displayed. Also, a list addition button 435 is displayed at the lower part of the temporary list screen 43e. When the user taps the list addition button 435, it transfers to the ingredient list / upper screen 40. Additionally, when the user taps the thumbnail 431 of any ingredient, it transfers to Figure 38 the imported image information screen 54b. The detained ingredients may be ingredients that were forgotten to be added to the ingredient list 85, so a temporary list based on the detained ingredients is presented.

[0222] Figure 38 It is the imported image information screen 54b.

[0223] The imported image information screen 54b displays a cropped image 541, a product name 542, a storage room image 543, and a crop frame 544 located thereon. The cropped image 541 is an image of the boxed natto taken and cropped. In the product name 542, it is displayed as "Other ingredients", and a string representing the food is shown. The storage room image 543 is an image of the storage room where the food is stored. The crop frame 544 is a frame indicating the range in the storage room image 543 where the food is recognized.

[0224] The structure and effects of the present invention will be described below. [1]

[0226] A refrigerator management device (2), characterized by comprising:

[0227] An image acquisition unit (80) that acquires an image of a storage room of the refrigerator (1);

[0228] An ingredient recognition unit (81) that recognizes each ingredient from the image;

[0229] A range specification unit (82) that can specify a partial range of the image; and

[0230] The control unit (84) updates the food ingredient list based on the information of the food ingredients included in the range acquired by the range specifying unit (82) among the food ingredients identified by the food ingredient identifying unit (81).

[0231] Thereby, a list of the food ingredients stored in the refrigerator can be easily generated. [2]

[0233] The refrigerator management device (2) according to [1], characterized in that

[0234] The food ingredient identifying unit (81) identifies each food ingredient from the image before the range specifying unit (82) specifies a range.

[0235] Thereby, a temporary list of food ingredients can be immediately displayed after the range is specified. [3]

[0237] The refrigerator management device (2) according to [1], characterized in that

[0238] It further includes a temporary list presenting unit (83) that presents images of the food ingredients included in the range specified by the range specifying unit (82) among the food ingredients identified by the food ingredient identifying unit (81) on the display unit,

[0239] The control unit (84) adds the food ingredients selected by the user among the food ingredients presented by the temporary list presenting unit (83) to the food ingredient list and updates it.

[0240] Thereby, errors in food ingredient image recognition can be corrected through manual operations, and a more accurate food ingredient list can be generated. [4]

[0242] The refrigerator management device (2) according to [1], characterized in that

[0243] A partial range of the image of the storage room specified by the range specifying unit (82) is specified in advance,

[0244] The food ingredient identifying unit (81) identifies each food ingredient included in the range specified in advance among the images of the storage room.

[0245] Thereby, image recognition is only performed on the food ingredients in the pre-determined range, so that the user is aware of this range and places the food ingredients to be recognized in this range, and a more accurate food ingredient list can be generated. [5]

[0247] The refrigerator management device (2) according to [1], characterized in that

[0248] The range specified by the range specifying unit (82) can be changed by the user on the screen where the image is displayed.

[0249] Thereby, the user can make the refrigerator 1 recognize the food ingredients within the required range. [6]

[0251] The refrigerator management device (2) according to [1], characterized in that it further comprises:

[0252] A food ingredient image extraction unit (87) that extracts images of individual food ingredients from the images obtained by the image acquisition unit (80); and

[0253] A list of staying food ingredients (851) for judging the staying of food ingredients.

[0254] Thereby, it is possible to list the foods that have stayed in the refrigerator for a long time. [7]

[0256] The refrigerator management device (2) according to [6], characterized in that

[0257] The control unit (84) compares the images extracted by the food ingredient image extraction unit (87) with the images registered in the list of staying food ingredients (851), and when it is judged that they are the same food ingredient, updates the staying time of this food ingredient.

[0258] Thereby, it is possible to update the staying time of the foods registered in the list of staying foods. [8]

[0260] The refrigerator management device (2) according to [6], characterized in that

[0261] The control unit (84) compares the images extracted by the food ingredient image extraction unit (87) with the images registered in the list of staying food ingredients (851), and when there is no image judged to be the same food ingredient in the list of staying food ingredients (851), registers this food ingredient in the list of staying food ingredients (851).

[0262] Thereby, it is possible to register the newly stored foods in the refrigerator into the list of staying foods. [9]

[0264] The refrigerator management device (2) according to [6], characterized in that

[0265] The control unit (84) compares the image extracted by the food material image extraction unit (87) with the images registered in the remaining food material list (851). If there is a food material in the images registered in the remaining food material list (851) that does not match any of the food materials extracted by the food material image extraction unit (87), the control unit deletes that food material from the remaining food material list (851).

[0266] Thereby, foods that have been consumed and no longer exist in the refrigerator can be deleted from the remaining food list.

[10]

[0268] The refrigerator management device (2) according to any one of [7] to [9], characterized in that

[0269] For the image extracted by the food material image extraction unit (87), the control unit (84) also combines it with the cropping coordinate information used when extracting the image of the food material and registers it in the remaining food material list (851). When comparing the image extracted by the food material image extraction unit (87) with the images registered in the remaining food material list (851), the cropping coordinate information of the image is used in combination with the image of the food material.

[0270] Thereby, when the position of the food in the refrigerator changes, the number of days elapsed can be initialized.

[11]

[0272] The refrigerator management device (2) according to

[10] , characterized in that

[0273] It further includes a warning unit (88). If the retention time of the food materials registered in the remaining food material list (851) exceeds a specified value, the warning unit gives a warning.

[0274] Thereby, the user can be warned about foods that have been retained in the refrigerator for longer than the specified time.

[12]

[0276] A refrigerator management method, characterized by including:

[0277] A step of obtaining an image of the storage compartment of the refrigerator (1) by an image acquisition unit (80);

[0278] A step of identifying each food material from the image by a food material identification unit (81);

[0279] A step that a range specifying unit (82) can specify a partial range of the image; and

[0280] A step of updating a food ingredient list by a control unit (84) based on information of food ingredients among those identified by the food ingredient identification unit (81) and included in the range specified by the range specifying unit (82).

[0281] Thereby, a list of food ingredients stored in the refrigerator can be easily generated.

[13]

[0283] A refrigerator management program (241) for causing a computer to execute the following steps:

[0284] A step of acquiring an image of a storage compartment of the refrigerator (1);

[0285] A step of identifying each food ingredient from the image;

[0286] A step of being able to specify a partial range of the image; and

[0287] A step of updating a food ingredient list based on information of food ingredients identified from the image and included in the range.

[0288] Thereby, a list of food ingredients stored in the refrigerator can be easily generated.

[0289] (Modification example)

[0290] The present invention is not limited to the above-described embodiments and includes various modification examples. For example, the above-described embodiments are embodiments described in detail for easy understanding of the present invention and are not limited to having all the structures described. A part of the structure of a certain embodiment can be replaced with the structure of another embodiment, and the structure of another embodiment can also be added to the structure of a certain embodiment. In addition, for a part of the structure of each embodiment, other structures can also be added, deleted, or replaced.

[0291] Part or all of the above-described structures, functions, processing units, processing elements, etc. can also be implemented by hardware such as an integrated circuit. The above-described structures, functions, etc. can also be implemented by software by a processor interpreting and executing a program for realizing each function. Information such as a program, a table, a file, etc. for realizing each function can be placed in a recording device such as a memory, a hard disk, an SSD (Solid State Drive), or a recording medium such as a flash memory card, a DVD (Digital Versatile Disk).

[0292] In each embodiment, the control lines and information lines represent parts considered necessary for explanation and do not necessarily represent all the control lines and information lines in the product. In fact, it can be considered that almost all the structures are interconnected.

[0293] As a modification example of the present invention, for example, there are the following (a) to (f).

[0294] (a) When comparing between foodstuffs, not only the information of the feature amounts of the cut images of the foodstuffs can be compared, but also the cut coordinate information of the images can be combined. Thereby, even when there are multiple foodstuffs of the same type, it is possible to detect the placement of the foodstuffs.

[0295] (b) The range selection box is not limited to a rectangle, and can also be an amorphous shape, a combination of multiple rectangles, without limitation.

[0296] (c) What operates as the cold storage management device is not limited to a server. For example, a cold storage or a mobile terminal can also operate as the cold storage management device, without limitation.

[0297] (d) What photographs the storage room of the cold storage is not limited to the cold storage camera. The storage room of the cold storage can also be photographed one by one by the portable terminal held by the user.

[0298] (e) When identifying a foodstuff from the photographed image of the storage room of the cold storage, the information of the foodstuff can also be registered in the foodstuff list without generating a temporary list.

[0299] (f) The residence time for issuing a warning to the foodstuff with a long residence time can also be determined by whether it exceeds the shelf life of each foodstuff, without limitation.

Claims

1. A cold storage management device, characterized in that, Comprising: An image acquisition unit that acquires an image of a storage chamber of a refrigerator; An ingredient recognition unit that recognizes each ingredient from the image; A range specifying unit that can specify a partial range of the image; And A control unit that updates an ingredient list based on information of ingredients included in the range acquired by the range specifying unit among the ingredients recognized by the ingredient recognition unit.

2. The refrigerator management device according to claim 1, characterized in that The ingredient recognition unit recognizes each ingredient from the image before the range is specified by the range specifying unit.

3. The refrigerator management device according to claim 1, characterized in that It further includes a temporary list presentation unit that presents images of ingredients included in the range specified by the range specifying unit among the ingredients recognized by the ingredient recognition unit on a display unit, The control unit adds the ingredients selected by the user among the ingredients presented by the temporary list presentation unit to the ingredient list for updating.

4. The refrigerator management device according to claim 1, characterized in that A partial range of the image of the storage chamber specified by the range specifying unit is pre-specified, The ingredient recognition unit recognizes each ingredient included in the pre-specified range in the image of the storage chamber.

5. The refrigerator management device according to claim 1, characterized in that The range specified by the range specifying unit can be changed by the user on the screen where the image is displayed.

6. The cold storage management device according to claim 1, characterized in that, It further includes: An ingredient image extraction unit that extracts images of individual ingredients from the image acquired by the image acquisition unit; And A retained ingredient list for judging the retention of ingredients.

7. The refrigerator management device according to claim 6, characterized in that The control unit compares the images extracted by the ingredient image extraction unit with the images registered in the retained ingredient list, and updates the retention time of the ingredient when it is judged to be the same ingredient.

8. The refrigerator management device according to claim 6, characterized in that The control unit compares the images extracted by the ingredient image extraction unit with the images registered in the retained ingredient list, and registers the ingredient in the retained ingredient list when there is no image judged to be the same ingredient in the retained ingredient list.

9. The refrigerator management device according to claim 6, characterized in that The control unit compares the images extracted by the ingredient image extraction unit with the images registered in the retained ingredient list, and deletes the ingredient from the retained ingredient list if there is an ingredient in the images registered in the retained ingredient list that is inconsistent with any of the ingredients extracted by the ingredient image extraction unit.

10. The refrigerator management device according to any one of claims 7 to 9, characterized in that For the image extracted by the food material image extraction unit, the control unit also combines it with the cropping coordinate information used when extracting the image of the food material and registers it in the list of remaining food materials. When comparing the image extracted by the food material image extraction unit with the images registered in the list of remaining food materials, the cropping coordinate information of the image used together with the image of the food material is used.

11. The cold storage management device according to claim 10, characterized in that it further includes a warning unit that gives a warning if the retention time of any of the food materials registered in the list of remaining food materials exceeds a specified value.

12. A cold storage management method, characterized in that, It includes: a step of obtaining an image of a storage room of a cold storage by an image acquisition unit; a step of identifying each food material from the image by a food material identification unit; a step in which a range specifying unit can specify a partial range of the image; and a step of updating a food material list by a control unit based on information on the food materials included in the range specified by the range specifying unit among the food materials identified by the food material identification unit.

13. A cold storage management program for causing a computer to execute the following steps: a step of obtaining an image of a storage room of a cold storage; a step of identifying each food material from the image; a step of being able to specify a partial range of the image; and a step of updating a food material list based on information on the food materials included in the range among the food materials identified from the image.

Images