Refrigerator management device, refrigerator management method, and refrigerator management program

Abstract

To easily create a list of ingredients stored in a refrigerator.SOLUTION: A server, which is a refrigerator management device, comprises: an image acquisition unit 80 which acquires images of a storage room equipped in a refrigerator; an ingredient recognition unit 81 which recognizes each ingredient from the image; a range specification unit 82 which is able to specify a range of a part of the image; and a control unit 84 which updates an ingredient list 85 on the basis of the information of the ingredients included in the range acquired by the range specification unit 82 among the ingredients recognized by the ingredient recognition unit 81.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a refrigerator management device, a refrigerator management method, and a refrigerator management program.

Background Art

[0002] As the background art of this technical field, the abstract of Patent Document 1 describes that "a refrigerator management device 20 is provided with a storage room internal image storage control unit 938 that stores, as a storage room internal image associated with the storage rooms 61 to 64 designated by a storage room designation unit 936, an image acquired from an imaging unit 26 that captures an image."

[0003] Also, the abstract of Patent Document 2 describes that "an imaging control unit 111 that generates a food material image PC, which is an image of a food material, every predetermined time, an extraction unit 112 that extracts, based on the food material image PC, a food material name candidate CFN that is a candidate for the food material name FN corresponding to the food material every time the imaging unit 13 generates the food material image PC, and a display control unit 113 that displays the food material image PC on an LCD 121 every time the imaging unit 13 generates the food material image PC and displays the food material name candidate CFN as the food material name FN on the LCD 121 when a predetermined condition is satisfied, are made to function."

[0004] Patent Document 3 describes an invention of a refrigerator system that can display an image indicating the position of a change together with an image inside the refrigerator before the change when there is a change in the stored items inside the refrigerator.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0006] According to the invention described in Patent Document 1, a list of foodstuffs stored in a refrigerator can be easily created. However, since the device of the present invention does not have an image recognition function, it requires complicated operations for the user. Further, simply recognizing an image of a storage compartment has a problem in that manual work must be used in combination when the reliability of image recognition is low.

[0007] Further, according to the invention described in Patent Document 2, when registering a plurality of foodstuffs that appear in a certain range, it is necessary to touch and select them one by one, and there is room for improvement in operability. According to the invention described in Patent Document 3, when the number or area of regions where there are differences between a new image of a refrigerator and a previous image is equal to or greater than a predetermined value, the differences are highlighted or the difference information is cut out and displayed, so that neglected foodstuffs can be detected. In the invention described in Patent Document 3, since both of the new and old images to be compared are those showing the entire interior of the refrigerator, the comparison process is inefficient. Further, in the invention described in Patent Document 3, when a foodstuff taken out once is restocked in another place, there is a possibility that it may be counted as a difference, and there is room for improvement.

[0008] Therefore, an object of the present invention is to enable an easy creation of a list of foodstuffs stored in a refrigerator.

Means for Solving the Problems

[0009] In order to solve the above-described problems, a refrigerator management device of the present invention includes an image acquisition unit that acquires an image of a storage compartment provided in a refrigerator, a foodstuff recognition unit that recognizes each foodstuff from the image, a range designation unit that can designate a partial range of the image, and a control unit that updates a foodstuff list based on information on foodstuffs included in the range acquired by the range designation unit among the foodstuffs recognized by the foodstuff recognition unit.

[0010] The refrigerator management method of the present invention comprises: a step in which an image acquisition unit acquires an image of a storage compartment provided in the refrigerator; a step in which a food recognition unit recognizes each food item from the image; a step in which a range designation unit can designate a partial range of the image; and a step in which a control unit updates a food list based on information on the food items included in the range designated by the range designation unit among the food items recognized by the food recognition unit.

[0011] The refrigerator management program of the present invention is for causing a computer to execute a procedure for acquiring an image of a storage compartment provided in the refrigerator, a procedure for recognizing each food item from the image, a procedure for being able to designate a partial range of the image, and a procedure for updating a food list based on information on the food items included in the range among the food items recognized from the image.

[0012] Other means will be described in the mode for carrying out the invention.

Effects of the Invention

[0013] According to the present invention, it becomes possible to easily create a list of the food items stored in the refrigerator.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Mode for Carrying Out the Invention

[0015] Hereinafter, the mode for carrying out the present invention will be described in detail with reference to each figure. FIG. 1 is a configuration diagram of a refrigerator management system. The refrigerator management system of the present embodiment provides services to the user in cooperation with the refrigerator 1, the server 2, and the mobile terminal 3. Note that the refrigerator 1, the server 2, and the mobile terminal 3 are connected to the Internet and are assigned different IP addresses. In the present embodiment, the server 2 is a computer and functions as a refrigerator management device for managing the refrigerator 1.

[0016] Referring to FIG. 2, the configuration of the refrigerator 1 will be described. As a hardware configuration, the refrigerator 1 includes a refrigerator camera 10, an arithmetic unit 112, a ROM (Read Only Memory) 113, a RAM (Random Access Memory) 114, a communication unit 115, and a camera illumination unit 100. The refrigerator 1 further includes a left door 12, a right door 13, a door sensor 117, and a refrigerating chamber 116.

[0017] The arithmetic unit 112 is, for example, a CPU (Central Processing Unit) that comprehensively controls the refrigerator 1. The RAM 114 serves as the working area of the arithmetic unit 112. Also, the RAM 114 is a non-temporary or temporary recording medium that stores various programs and data, and a program (not shown) is stored therein.

[0018] The refrigerator camera 10 is an imaging unit that photographs the refrigerator compartment 116 and other storage compartments, and is, for example, a fish-eye camera or a panoramic camera. In the present embodiment, the refrigerator camera 10 is attached above the refrigerator compartment 116, but may be attached, for example, inside the refrigerator compartment 116, to the left door 12, or to the right door 13. Instead of or in addition to the refrigerator camera 10, a camera (not shown) provided in, for example, a mobile terminal 3 prepared by the user may be used. The camera illumination unit 100 is an indicator that lights up when the refrigerator camera 10 is taking an image. The left door 12 and the right door 13 are doors provided in the refrigerator compartment 116, and their open / closed states are detected by a door sensor 117. When the user opens the left door 12 or / and the right door 13, the refrigerator camera 10 operates to photograph the refrigerator compartment 116. The refrigerator compartment 116 is a storage compartment that stores articles typified by food.

[0019] The communication unit 115 connects to a network and transmits and receives data to and from a server 2, which is an external device.

[0020] Referring to FIG. 3, the configuration of the server 2 will be described. The server 2 includes a CPU 21, a ROM 22, a RAM 23, a storage unit 24, and a communication unit 25, respectively.

[0021] The CPU 21 comprehensively controls the server 2. The ROM 22 is a non-volatile memory, and stores, for example, a BIOS (Basic I / O System). The RAM 23 serves as the working area of the CPU 21. Also, the RAM 23 is a non-temporary or temporary recording medium that stores various programs and data, and a program (not shown) is stored therein.

[0022] The storage unit 24 is a non-temporary or temporary recording medium that stores various programs and data, and is composed of, for example, an HDD (Hard Disk Drive), a flash memory, etc. The storage unit 24 stores a refrigerator management program 241 for managing the refrigerator 1. Note that the storage units 24 of the storage servers 6a and 6b are preferably large-capacity and capable of high-speed access. The communication unit 25 connects to a network and transmits and receives data to and from external devices.

[0023] Referring to FIG. 4, the configuration of the mobile terminal 3 will be described. The mobile terminal 3 includes a CPU 31, a ROM 32, a RAM 33, a storage unit 34, a communication unit 35, and a touch panel display 36, respectively. The CPU 31 controls the mobile terminal 3 in an overall manner. The ROM 32 is a non-volatile memory, and stores, for example, a BIOS. The RAM 33 serves as a working area for the CPU 31. Also, the RAM 33 is a non-temporary or temporary recording medium that stores various programs and data, and stores a program (not shown). Note that the information processing performed by the arithmetic unit 112, the CPU 21, the CPU 31, etc. may be performed by any of them as long as there is no problem. For example, the processing described as being performed by the CPU 21 in the present embodiment may be performed by the arithmetic unit 112 or the CPU 31 as long as there is no problem in the entire system.

[0024] The storage unit 34 is a non-temporary or temporary recording medium that stores various programs and data, and is composed of, for example, a flash memory, etc. The communication unit 35 connects to a network and transmits and receives data to and from external devices. The touch panel display 36 is configured by laminating a touch panel on a flat display. The touch panel display 36 is a display means for displaying characters, images, or graphics, and is also an input means for detecting user operations.

[0025] FIG. 5 is a logical block diagram of the image management system. The image acquisition unit 80 acquires an image of the storage compartment provided in the refrigerator 1 captured by the refrigerator camera 10. This image acquisition unit 80 is realized by the CPU 21 of the server 2 executing the refrigerator management program 241.

[0026] The food ingredient recognition unit 81 recognizes the presence of each food ingredient from the image. The food ingredient recognition unit 81 may also recognize information such as the name of the food ingredient and the position of the food ingredient. The range specification unit 82 can specify a partial range of the image. A partial range of the captured image of the storage compartment specified by the range specification unit 82 may be specified in advance. The food ingredient recognition unit 81 recognizes each food ingredient from the image before the range is specified by the range specification unit 82.

[0027] The provisional list presentation unit 83 presents, to the display unit, the images of the food ingredients included in the range specified by the range specification unit 82 among the food ingredients recognized by the food ingredient recognition unit 81 and / or the recognized food ingredient item names. The control unit 84 adds and updates to the food ingredient list 85 the food ingredients selected by the user among the food ingredients recognized by the food ingredient recognition unit 81 and presented by the provisional list presentation unit 83 and / or the recognized food ingredient item names.

[0028] The main list presentation unit 86 presents, to the display unit, the images of the food ingredients registered in the food ingredient list 85 by the control unit 84. The control unit 84 updates the food ingredient list 85 based on the information of the food ingredients included in the range acquired by the range specification unit 82.

[0029] The food ingredient image extraction unit 87 extracts the images of the respective food ingredients from the image acquired by the image acquisition unit 80. Then the CPU 21 updates the staying food ingredient list 851. The control unit 84 compares the image extracted by the food ingredient image extraction unit 87 with the images already registered in the staying food ingredient list 851. If there is no image in the staying food ingredient list 851 that is determined to be the same food ingredient, the control unit 84 registers the food ingredient in the staying food ingredient list 851.

[0030] The control unit 84 compares the image extracted by the food material image extraction unit 87 with the images registered in the staying food material list 851, and if there is any image registered in the staying food material list 851 that does not match any of the food materials extracted by the food material image extraction unit 87, the control unit 84 deletes the food material from the staying food material list 851. The CPU 21 may further register in the staying food material list 851 by combining the cut-out coordinate information used for extracting the image of the food material with the image extracted by the food material image extraction unit 87, and may combine the change in the coordinate information of the cut-out position of the image together with the image of the food material in the comparison between the image extracted by the food material image extraction unit 87 and the images registered in the staying food material list 851.

[0031] If the staying period of the food materials registered in the staying food material list 851 exceeds a predetermined value, the warning unit 88 transmits warning information to the mobile terminal 3 to display a warning. At this time, the refrigerator concierge app of the mobile terminal 3 displays a push notification or a warning on the home screen for the user.

[0032] FIG. 6 is a front view of the refrigerator 1. The refrigerator 1 is a device for cooling foods and the like, and includes a left door 12 and a right door 13 of the refrigerating chamber 116, and a refrigerator camera 10. The refrigerator 1 has a plurality of storage chambers inside its housing. As the storage chambers of the refrigerator 1, in order from the top, there are provided a refrigerating chamber 116, an ice-making chamber 14 and an upper freezing chamber 15 arranged side by side left and right, a vegetable chamber 16, and a lower freezing chamber 17. Note that the arrangement of the vegetable chamber 16 and the lower freezing chamber 17 is not limited to this, and as the storage chambers of the refrigerator 1, in order from the top, there may be provided a refrigerating chamber 116, an ice-making chamber 14 and an upper freezing chamber 15 arranged side by side left and right, a lower freezing chamber 17, and a vegetable chamber 16.

[0033] The housing of the refrigerator 1 is configured such that a heat insulating material (not shown) such as foamed urethane is filled between an outer case made of a steel plate and an inner case (not shown) made of resin. A plurality of openings corresponding to each compartment are provided on the front side (front surface side) of the housing of the refrigerator 1. For example, when food or the like is put in through the opening of the refrigerating compartment 116, the left door 12 and the right door 13 are opened. Also, when the left door 12 and the right door 13 are closed, the opening of the refrigerating compartment 116 is in a closed state. Thus, the left door 12 and the right door 13 have functions such as closing the opening of the refrigerating compartment 116. The same applies to each of the other doors.

[0034] The refrigerator 1 includes a left door 12 and a right door 13 as French doors that form the refrigerating compartment 116 together with the housing. The left door 12 is rotatable about the axis of the hinge 121 (see FIG. 9) at the left end. The same applies to the right door 13. Also, the refrigerator 1 includes a pull-out door, and in addition to the ice-making compartment door and the upper freezer compartment door, it includes a vegetable compartment door and a lower freezer compartment door.

[0035] A plurality of shelf boards for partitioning the refrigerating compartment 116 into a predetermined configuration are provided in the refrigerating compartment 116. A plurality of door pockets 123 (see FIG. 8) for storing food or the like are provided on the inner side of the left door 12. The same applies to the right door 13. An ice-making compartment container (not shown) that is pulled out integrally with the ice-making compartment door is provided in the ice-making compartment 14. Similarly, an upper freezer compartment container (not shown) is provided in the upper freezer compartment 15, a vegetable compartment container (not shown) is provided in the vegetable compartment 16, and a lower freezer compartment container (not shown) is provided in the lower freezer compartment 17.

[0036] Also, although not shown, the refrigerator 1 includes a compressor, a radiator (condenser), a capillary tube (throttling mechanism), and an evaporator. Then, the refrigerant circulates sequentially through the compressor, the radiator, the capillary tube, and the evaporator, and the air in the storage compartment is cooled by heat exchange with the refrigerant flowing through the evaporator. The refrigerator camera 10 shown in FIG. 6 images the storage compartment and the like of the refrigerator 1 and is installed on the upper surface of the housing.

[0037] FIG. 7 is a side view of the refrigerator 1. As shown in FIG. 7, the refrigerator camera 10 includes a main body portion 101 and a support portion 102. The main body portion 101 has a function of imaging a storage room or the like. The support portion 102 supports the main body portion 101 and is installed on the upper surface of the housing of the refrigerator 1.

[0038] Near the front end of the main body portion 101, a lens 103 (also see FIG. 7) is provided. The lens 103 is an optical element that refracts and focuses light. For example, a fish-eye lens is used as such a lens 103. And when the left door 12 or / and the right door 13 (see FIG. 6) is opened, the lens 103 faces downward so that the refrigerator camera 10 can image the refrigerating chamber 116 (see FIG. 8). The refrigerator camera 10 is attached at a position where it may image a living space range including the living space outside the refrigerating chamber 116.

[0039] As shown in FIG. 7, the lens 103 is located in front of the front end (opening of the housing) of the housing. More preferably, the lens 103 is located further in front of the front surfaces of the closed left door 12 and right door 13 (see FIG. 6). Thereby, for example, when the left door 12 and the right door 13 of the refrigerating chamber 116 are opened, the refrigerating chamber 116 easily enters the field of view of the lens 103. It is also possible to image the vegetable chamber 16 or the like (see FIG. 6) with the refrigerator camera 10 when a drawer-type door such as a vegetable chamber door is opened. In this way, the lens 103 is provided at a position where each storage chamber can be looked down upon from above.

[0040] FIG. 8 is a front view of the refrigerator 1 with the left door 12 and the right door 13 opened. As described above, a plurality of door pockets 123 for accommodating foods and the like are provided on the inner plate 122 of the left door 12 (the same applies to the right door 13). When the left door 12 and the right door 13 are opened, the foods in the refrigerating chamber 116 and the door pockets 123, 133 are panoramically included in the field of view of the lens 103 of the refrigerator camera 10.

[0041] The rotary partition 124 shown in Fig. 8 is a partition body for suppressing cold air leakage from the gap between the left door 12 and the right door 13. In the example of Fig. 8, in the left door 12, the rotary partition 124 is provided at the end opposite to the axis of the hinge 121 (see Fig. 9), and is adapted to rotate in a predetermined manner as the left door 12 opens and closes.

[0042] The refrigerator camera 10 automatically takes a picture when at least one of the left door 12 and the right door 13 of the refrigerator 1, the vegetable compartment door, or the lower freezer compartment door is opened by the user. It is desirable to take the picture after a predetermined time has elapsed since any of the doors was opened. Alternatively, the refrigerator camera 10 may be used to manually take a picture of the refrigerator compartment 116 or the like. When imaging the refrigerator compartment 116, a shooting button 125 pushed by the user is provided on the left door 12, and a shooting button 135 is provided on the right door 13. In the example of Fig. 8, in the left door 12, the shooting button 125 is provided at the lower part of the surface on the side opposite to the hinge 121 (see Fig. 9) of the left door 12 (the same applies to the right door 13). From another perspective, in the state where the left door 12 and the right door 13 are closed (see Fig. 6), the shooting button 125 is provided on the surface where one of the left door 12 and the right door 13 faces the other (near the rotary partition 124 in Fig. 8).

[0043] The shooting performed when at least one of these shooting buttons 125 and 135 is pushed by the user is called "manual shooting". Also, the shooting performed by the arithmetic unit 112 based on the elapsed time from the start timing of the opening of the left door 12 and the right door 13 is called "automatic shooting".

[0044] Fig. 9 is a plan view of the state where the left door 12 and the right door 13 of the refrigerator 1 are opened. In the example of FIG. 9, in the left-right direction, the main body 101 of the refrigerator camera 10 is disposed slightly to the left of the center of the housing of the refrigerator 1. More specifically, the main body 101 of the refrigerator camera 10 is disposed directly above the joint of the left door 12 and the right door 13 in the closed state (see also FIGS. 6 and 7). By disposing the main body 101 in this way, for example, when the left door 12 and the right door 13 are opened, foods and the like in the door pockets 123, 133 (see FIG. 8) are more likely to enter the field of view of the refrigerator camera 10.

[0045] Also, when using the refrigerator 1, the user often pays attention to the joint (boundary portion) of the left door 12 and the right door 13. Therefore, by disposing the main body 101 directly above the joint of the left door 12 and the right door 13, it is possible to suppress the user from feeling discomfort as compared with the case where the position of the main body 101 in the left-right direction is deviated from the joint.

[0046] FIG. 10 is a perspective view when the refrigerator camera 10 is looked up from an obliquely downward direction. The main body 101 of the refrigerator camera 10 shown in FIG. 10 includes, in addition to the lens 103 described above (see also FIG. 7), a case 31b and a camera illumination unit 100. In the example of FIG. 7, the case of the main body 101 generally has an elongated rectangular parallelepiped shape in the front-rear direction. A circular hole (reference numeral not shown) is provided on the lower surface near the front end of the case of the main body 101, and the lens 103 is exposed through this hole. Further, the camera illumination unit 100 is provided on the rear side (inner side) of the lens 103.

[0047] The camera illumination unit 100 is a light source that irradiates light to the refrigerating chamber 116 or the like so that the refrigerator camera 10 can photograph the refrigerating chamber 116 or the door pockets 123, 133 (see FIG. 8) under appropriate brightness. Further, the camera illumination unit 100 (illumination unit) is provided outside the housing of the refrigerator 1 (see FIG. 7). Although details will be described later, in the present embodiment, the refrigerator camera 10 photographs the refrigerating chamber 116 or the like (see FIG. 8) with the camera illumination unit 100 turned on while turning off the interior light of the refrigerator 1.

[0048] As shown in FIG. 10, by providing the lens 103 on the front side of the camera illumination unit 100, the reflected light from the refrigerator compartment 116 etc. (see FIG. 8) is likely to enter the lens 31a when the left door 12 and the right door 13 are open. Therefore, the refrigerator camera 10 can appropriately photograph the food stored in the refrigerator compartment 116 etc. It is preferable that the camera illumination unit 100 is located on the front side of the front end (opening of the housing) of the housing of the refrigerator 1 (see FIG. 7). As a result, the light irradiated from the camera illumination unit 100 is likely to enter the refrigerator compartment 116 etc., and thus a clear photographing result is likely to be obtained.

[0049] The refrigerator 1 is a storage cabinet including a refrigerator compartment 116 which is a storage area for storing articles, and a refrigerator camera 10 for imaging an article range including the refrigerator compartment 116. FIG. 11 is a diagram showing an image before processing photographed by the refrigerator camera 10 of the refrigerator 1. The refrigerator compartment 116 is shown in the lower half of the image. The inner door pocket 123 of the left door 12 is shown on the right side of the image. The inner door pocket 133 of the right door 13 is shown on the left side of the image. And the user's living environment is reflected in the upper half of the image. This part is an unnecessary part when viewed from the processed image.

[0050] FIG. 12 is a diagram showing an image before processing with the unnecessary part filled in. This system performs a masking process on the unnecessary part of the fisheye image. The masking process executed by this system is a processing process for deleting at least a part of the range outside the article range such as the refrigerator compartment 116 and the door pockets 123, 133. Thereby, when the user views the image of the inside of the refrigerator 1, it is possible to prevent the user and the user's living environment from being inadvertently seen by others. Note that the masking process for this unnecessary part is executed by the server 2, but it may also be executed by the refrigerator 1.

[0051] FIG. 13 is a diagram showing an example of the processed image. Server 2 corrects the lens distortion of the pre - processed image taken by the refrigerator camera 10 of the refrigerator 1, performs trapezoidal correction, and processes it into a front - view image. At this time, when Server 2 detects a face in the image, it performs mosaic processing on the area where the face appears.

[0052] The mosaic processing executed by this system is a processing operation that reduces the amount of information in at least a partial range outside the article range. Thereby, when allowing the user to view the image taken inside the refrigerator 1, it is possible to prevent the user from being inadvertently seen by others. Note that this face detection process and mosaic processing are executed by Server 2, but the refrigerator 1 may also execute them on the pre - processed image.

[0053] 《First Embodiment》 In this embodiment, it is possible to specify a partial range of the image taken inside the refrigerator 1, and the food ingredients within the range are targeted for listing. Thereby, it is possible to easily create a food list consisting only of the food ingredients that the user wishes to manage.

[0054] FIG. 14 is a flowchart of the process for updating the food ingredient list 85. This process designates a partial range of the image with a frame or the like and adds the food ingredients within the range to a temporary list, which is the process when the CPU 21 of Server 2 executes the refrigerator management program 241. First, the food ingredient recognition unit 81 in FIG. 5 recognizes food ingredients from the image taken by the refrigerator camera 10 (step S10). The information recognized by the food ingredient recognition unit 81 is at least to recognize the area where the food ingredient appears and the area where it does not appear in the taken image. The timing at which the food ingredient recognition unit 81 performs image recognition is preferably immediately after the refrigerator camera 10 takes an image or before the listing method selection screen 41 in FIG. 16 is displayed on the mobile terminal 3. Thereby, at the timing when the user selects the desired range, since the food ingredient recognition unit 81 has already completed image recognition, the time from the selection of the range to the list display of the food ingredients included in the range can be shortened.

[0055] Also, after the user selects a desired range, the food ingredient recognition unit 81 may perform image recognition. In this case, since the food ingredient recognition unit 81 does not perform unnecessary image recognition, the computational load on the CPU 21 of the server 2 can be reduced. Note that the information recognized by the food ingredient recognition unit 81 may include the name of the food ingredient, the name of the storage room where the food ingredient is stored, the coordinate information in the storage room, and the like.

[0056] Thereafter, when the user taps the “+” button 406 on the food ingredient list top screen 40 in FIG. 15 (step S11), the list creation method selection screen 41 in FIG. 16 is displayed on the mobile terminal 3 (step S12). Next, on the list creation method selection screen 41 in FIG. 16, the user taps the “Input from the image of the refrigerator camera” button 411 (step S13). As a result, the range specifying unit 82 causes the range selection screen 42 in FIG. 17 to be displayed on the mobile terminal 3 (step S14). On this range selection screen 42, a storage room image 421 of the storage room of the refrigerator 1 taken by the refrigerator camera 10 is displayed. It is desirable that the storage room image 421 is one of the latest images taken by the refrigerator camera 10 of the storage room. However, for example, an arbitrary image specified by the user may be displayed from among a plurality of recent captured images. The images taken by the refrigerator camera 10 are labeled so that they can be distinguished for each captured storage room, and it is preferable to display the latest image of the storage room specified by the indoor selection button 423.

[0057] When the user selects any range of the storage room image 421 of the storage room of the refrigerator 1 (step S15), the temporary list presenting unit 83 creates a temporary list for the food ingredients within the selected range in the range selection frame 422, and causes the temporary list screen 43 in FIG. 18 to be displayed on the mobile terminal 3 (step S16). When, for example, a thumbnail 431 of this food ingredient is selected and the list addition button 435 is tapped (step S17), the food ingredient list 85 is updated by the control unit 84, and the food ingredient list top screen 40a is updated as shown in FIG. 19 by the main list presenting unit 86 (step S18), and the process in FIG. 14 ends.

[0058] Through this process, the user can decide on their own, according to their preferences, the space for placing the ingredients they want to add to the ingredient list 85, and can easily register the ingredients in the ingredient list 85. The range for extracting the ingredients to be added to the ingredient list 85 can be set by the user each time in step S14, or the system can automatically restore the range selection frame 422 set by the user in the past, for example, in step S14, or it can be set by default. That is, the initial setting of the range can be changed.

[0059] Note that before the user specifies the range, the system may finish the image recognition of all the ingredients shown in the captured image. As a result, after the user specifies the range, the ingredient list can be updated immediately. Although it is not necessary to list the entire image, the operability when wanting to list multiple ingredients at once can be improved.

[0060] Figure 15 shows the ingredient list top screen 40 of the refrigerator concierge app. Each screen shown in the following description is displayed on the touch panel display 36 of the mobile terminal 3. The ingredient list top screen 40 is a screen that displays the ingredient list 85 in the refrigerator 1 shown in FIG. 5 in a list format. Each ingredient is indicated by a thumbnail 402, a product name 403, and the remaining days 405. The thumbnail 402 is a small image showing the appearance of this ingredient, which is the whole or a part cut out from the recognized area of the ingredient. The product name 403 indicates the product name of this ingredient. The remaining days 405 is the number of remaining days until the target date for using up this ingredient. The target date for using up can be determined based on the expiration date of each ingredient, or can be arbitrarily set by the user.

[0061] And a "+" button 406 is shown at the lower side of the list. This "+" button 406 is a button for updating, adding, and deleting the ingredient list 85. At the lower part of the ingredient list top screen 40, a home button 491, an ingredient management button 492, an operation status button 493, a support button 494, and a setting button 495 are shown.

[0062] The home button 491 is a button for transitioning to the home screen of this application and is highlighted when the home screen is being displayed. The food ingredient management button 492 is a button for transitioning to the food ingredient management screen that displays the food ingredient list and is highlighted when the food ingredient management screen is being displayed. The operation status button 493 is a button for transitioning to the screen showing the operation status of this refrigerator 1 and is highlighted when the screen showing the operation status is being displayed.

[0063] The support button 494 is a button for transitioning to the screen that supports the user of this application and is highlighted when the support screen is being displayed. The settings button 495 is a button for transitioning to the screen for setting this application and is highlighted when the settings screen is being displayed. These buttons are commonly displayed at the bottom of the screens of this application.

[0064] Figure 16 is the listing method selection screen 41. The listing method selection screen 41 is displayed in the front layer with the food ingredient list - top screen 40 shown in Figure 15 as the background. On the listing method selection screen 41, a "Input from the image of the refrigerator camera" button 411, a "Input food ingredient image with the camera" button 412, a "Voice input of food ingredient name" button 413, and a "Text input of food ingredient name" button 414 are shown.

[0065] When the user taps the "Input from the image of the refrigerator camera" button 411 here, it transitions to the range selection screen 42 in Figure 17. When the user taps the "Input food ingredient image with the camera" button 412, it transitions to a camera shooting screen (not shown). When the user taps the "Voice input of food ingredient name" button 413, it transitions to a voice input screen (not shown). When the user taps the "Text input of food ingredient name" button 414, it transitions to a text input screen (not shown).

[0066] Figure 17 is the range selection screen 42 for the storage room image 421. On the range selection screen 42, a storage room image 421, a range selection frame 422, an indoor selection button 423, and a food detection button 425 are displayed. The storage room image 421 is an image taken by the refrigerator camera 10 of one of the storage rooms of the refrigerator 1.

[0067] The range selection frame 422 is a frame for selecting the range in which the food to be registered in the food list 85 is stored in the storage room image 421. The indoor selection button 423 is a button for determining which of the storage rooms of the refrigerator 1 to select. The food detection button 425 is a button for detecting the food contained in the range selection frame 422 and transitioning to the temporary list screen 43 in FIG. 18. Note that the range selection frame 422 may be able to select a range in a shape other than a rectangle, or a plurality of ranges of the range selection frame 422 may be selectable in a single storage room image 421. The range can be freely set by, for example, a so-called pinch operation in which the user touches the touch panel display 36 with a plurality of fingers at the same time.

[0068] In addition to executing a partial selection mode in which a part of the storage room image 421 is selected using the range selection frame 422 in this way, a full selection mode in which the entire storage room image 421 is selected without using the range selection frame 422 may be distinguished and executed. Also, the range selection frame 422 may be able to select the entire image, or may not be able to select the entire image in order to clarify the difference from the full selection mode.

[0069] FIG. 18 is a temporary list screen 43 for food. On the temporary list screen 43, a temporary list of each food is displayed. Each food is indicated by a thumbnail 431, a product name 432, and a storage location 433. These foods are a list of the foods contained in the range selection frame 422. In the present embodiment, the product name 432 is displayed in the temporary list with "other food" input by default, and the user inputs the food name by the method described later with reference to FIG. 29. Note that when the food recognition unit 81 recognizes the food name of the food, the temporary list may be displayed with the recognized food name automatically input as the product name 432.

[0070] The thumbnail 431 is a small image showing the appearance of this food ingredient, and is a cut-out of the area where the food ingredient recognized by the food ingredient recognition unit 81 is shown. The food ingredient recognition unit 81 may cut out all of the food ingredient or only a part of it. The product name 434 indicates the product name of this food ingredient. The storage location 433 indicates the storage compartment where this food was detected, using an icon. When the user taps this thumbnail 431, for example, to select the food, and then taps the list addition button 435 in the lower right, this food ingredient is added to the food ingredient list 85.

[0071] Figure 19 is the updated food ingredient list - top screen 40a. The food list on the food ingredient list - top screen 40a has been updated, and beverages, beer, cake, and the other food ingredient natto have been added. In the thumbnail of the most recently added food ingredient, for example, this is indicated. For example, a "NEW" mark is displayed.

[0072] 《Other detailed operations》 Regarding other operations of specifying a range of some of the food ingredients in the image with a frame or the like and adding the food ingredients within the range to a temporary list, it will be described with reference to FIGS. 20 to 30 below. First,

[0073] Figure 20 is the food ingredient list - top screen 40b. This food ingredient list - top screen 40b displays the same elements as the food ingredient list - top screen 40 shown in FIG. 15. Here, when the user taps the "+" button 406, the user transitions to the list creation method selection screen 41b in FIG. 21.

[0074] Figure 21 is the list creation method selection screen 41b. The list creation method selection screen 41b displays the same elements as the list creation method selection screen 41 shown in FIG. 16. Here, when the user taps the "Input from the image of the refrigerator camera" button 411, the user transitions to the range selection screen 42 in FIG. 23. In addition, when the user taps the "Input from Refrigerator Camera Image" button 411, the user may be transitioned to the range selection screen 42c in FIG. 24. In this case, the range selection frame 422 on the range selection screen 42c may be displayed in a state where the entire storage room image 421 is selected, or may be displayed in the initial setting state of the range selection frame 422 set by the user, or in a state where the range selection frame 422 selected by the user in the past is restored.

[0075] FIG. 22 is the home screen 46. On the home screen 46, a notification area 461, a "Check Contents of Refrigerator" button 462, and a usage concierge button 463 are displayed. In the notification area 461, tomatoes, which are seasonal ingredients, and a speech bubble saying "Tomatoes are recommended this month" are displayed.

[0076] When the "Check Contents of Refrigerator" button 462 is tapped, the user is transitioned to the range selection screen 42b that displays the storage room image 421 for checking the contents in FIG. 23. When the usage concierge button 463 is tapped, the user is transitioned to a guidance screen for the usage method (not shown).

[0077] FIG. 23 is the range selection screen 42b that displays the storage room image 421 for checking the contents of the refrigerator 1. The range selection screen 42b displays the storage room image 421, an indoor selection button 423, and a list import button 424. Here, when the user performs, for example, a tap operation or a pinch operation on the storage room image 421, the user is transitioned to the range selection screen 42c in FIG. 24, and the range selection frame 422 is displayed.

[0078] FIG. 24 is the range selection screen 42c that displays the middle check image of the refrigerator 1 with a specified range. The range selection screen 42c has a range selection frame 422 displayed on the storage room image 421, and further has an indoor selection button 423 and a food detection button 425 displayed thereon. This range selection frame 422 is a part of the storage room image 421 selected by the user's operation. Whether or not a certain food is included and specified in the range selection frame 422 is determined by whether or not the recognized centroid (center) of the food is included within the range selection frame 422. For example, in FIG. 24, the PET bottle-like food at the upper right corner of the range selection frame 422 has a part of its right end outside the range, while the centroid (center) of the PET bottle-like food is within the range. Therefore, the PET bottle-like food is treated as being selected.

[0079] When the user taps the indoor selection button 423, the display location selection screen 52 in FIG. 27 is transitioned to. When the user taps the food detection button 425, the provisional list screen 43b of the food in FIG. 25 is transitioned to.

[0080] FIG. 25 is the provisional list screen 43b of the food. Since the provisional list screen 43b of the food is a screen related to food management, the food management button 492 is highlighted. On the provisional list screen 43b, a provisional list of each food is displayed. Each food is indicated by a thumbnail 431, a product name 432, and a storage location 433. These foods are a list of the foods included in the range selection frame 422. The provisional list may be created by sequentially specifying ranges for a plurality of images as follows.

[0081] On the provisional list screen 43b, there are further displayed a "import from another image" button 436 and a list addition button 435. When the user taps the "import from another image" button 436, the range selection screen 42b in FIG. 23 is transitioned to. Here, further, when the storage room image 421 is swiped left or right, an image captured before or after the currently displayed image of the selected storage room (here, the refrigerator) is displayed.

[0082] On one hand, tap the indoor selection button 423 to transition to the display location selection screen 52 in Fig. 27. Then, if you further tap any one of the icons of the desired storage compartments, namely the lower freezer 525, the upper vegetable compartment 524, the right door 523, the storage compartment 522, and the left door 521, it will transition again to, for example, the range selection screen 42b in Fig. 23 or the range selection screen 42d in Fig. 28 so as to display the latest image of the designated storage compartment. Then, make a range selection using the range selection frame 422 and tap the food ingredient detection button 425, and the food ingredients contained in the range selection frame 422 will be added to the temporary list.

[0083] After finishing adding the food ingredients of the desired multiple images to the temporary list, finally select the food ingredients you want to add to the food ingredient list 85 and then tap the list addition button 435, and the food ingredients of the multiple images can be added to the food ingredient list 85 at once.

[0084] Fig. 26 is the food ingredient list - top screen 40c of the updated food ingredients. The food ingredient list - top screen 40c has elements similar to those of the food ingredient list - top screen 40 shown in Fig. 15. This food ingredient list - top screen 40c displays the updated food ingredient list 85, and newly added are beverages, beer, cake, and natto as other food ingredients. In this embodiment, to minimize the information displayed on the food ingredient list - top screen 40c and prevent the screen from being cluttered, the information corresponding to the storage location 433 shown on the temporary list screen 43b is not displayed, but it may also be displayed in this manner. Also, it may be possible for the user to set whether to display the information corresponding to the storage location 433 on the food ingredient list - top screen 40c.

[0085] Fig. 27 is the display location selection screen 52. On this display location selection screen 52, each storage compartment of the refrigerator 1 is displayed in a selectable manner. Specifically, on the display location selection screen 52, the left door 521, the storage compartment 522, the right door 523, the upper vegetable compartment 524, and the lower freezer compartment 525 are schematically displayed, and a cancel button 526 is displayed below them. When the right door 523 is tapped here, it transitions to the range selection screen 42d shown in FIG. 28, and the right door portion is displayed as the storage compartment image 421. When the storage compartment 522 is tapped, it transitions to the range selection screen 42, and the storage compartment portion is displayed as the storage compartment image 421.

[0086] FIG. 28 is the range selection screen 42d. On the range selection screen 42d, the storage compartment image 421 with the right door portion displayed is shown. Above the storage compartment image 421, a range selection frame 422 is displayed. In this state, when the user taps the food detection button 425, it transitions to the temporary list screen 43c in FIG. 29, and the recognition results of the foods included in the range selection frame 422 are displayed as a temporary list.

[0087] FIG. 29 is a diagram showing the tap operation on the text portion of the temporary list screen 43c. On the temporary list screen 43c, beverages and beer are displayed, and a Japanese flick input area 439 is displayed below them. The user can arbitrarily set the names of the foods displayed on the temporary list screen 43c.

[0088] FIG. 30 is a diagram showing the swipe operation on the foods on the temporary list screen 43d. On the temporary list screen 43d, beverages, beer, cake, and other foods are displayed, and a list addition button 438 is displayed below them. By tapping the list addition button 438, the contents of the temporary list can be saved to the food list 85.

[0089] Note that the row related to beer is shifted to the right by swiping, and the delete button 437 is displayed at the left end. By tapping the delete button 437 by the user, the corresponding food ingredient, beer, can be deleted from the temporary list. Also, when the user double-taps the thumbnail 431 of the cake, the user is transferred to the captured image information screen 54 in FIG. 31. These deletion and transition to the captured image information screen 54 can also be performed in the food ingredient list 85.

[0090] FIG. 31 shows the captured image information screen 54. The captured image information screen 54 displays a cut-out image 541, a product name 542, a storage room image 543, and a cut-out frame 544 positioned thereon, and is transferred when the thumbnail 402 on the food ingredient list / top screen 40 or the thumbnail 431 on the temporary list screen 43 is tapped. The cut-out image 541 is an image obtained by photographing and cutting out this cake. The product name 542 is displayed as "cake", and a character string indicating this food is displayed. The storage room image 543 is an image of the storage room in which this food is stored. The cut-out frame 544 is a frame indicating the range in which this food was recognized in the storage room image 543.

[0091] 《Detection of Stagnation of Abandoned Food Ingredients (Long-Term Storage Items)》 In the abandoned food ingredient stagnation detection system, when each food ingredient is cut out from the latest in-store image, the number of days elapsed since the food ingredient was stored in the store is calculated by comparing it with the cut-out image in the stagnant food ingredient list. As a result, the stagnation detection process is simplified and the stagnation detection accuracy is improved.

[0092] In the present embodiment, when the orientation of the food ingredient changes due to restocking or the like, it can be recognized as a different food. As a result, it becomes easier to detect food ingredients that the user has forgotten the existence of and that have been left without movement for a long time. In other words, when the orientation changes, it is considered that the user frequently uses it, and it is not recognized as an abandoned food ingredient. The same determination can be made when the coordinates (storage position) of the food ingredient change significantly due to restocking or the like.

[0093] Note that the threshold value for determining a match may be arbitrarily set so that the same food ingredients are regarded as such even when the storage location or orientation of the food ingredients has changed due to restocking or the like by the user. In this case, even for the food ingredients being used by the user, the total number of days of storage in the refrigerator can be counted.

[0094] FIG. 32 is a flowchart of the stay determination process. The flowchart of FIG. 32 shows the procedure when the CPU 21 of the server 2 executes the refrigerator management program 241. The food ingredient recognition unit 81 recognizes all the food ingredients in the image, and the food ingredient image extraction unit 87 cuts out the range of the image related to the food ingredient (step S20). Then, the control unit 84 repeats the processes from step S21 to S26 for all the cut-out images.

[0095] The control unit 84 collates the image group in the stay food ingredient list 851 with the cut-out image, and searches for whether there is an image in the stay food ingredient list 851 that matches the latest cut-out image (step S22). Note that the stay food ingredient list 851 is information inside the server 2.

[0096] In step S23, the control unit 84 determines whether this cut-out image matches any of the images in the stay food ingredient list 851. If there is an image in the stay food ingredient list that matches the cut-out image (Yes), the process proceeds to step S24, and the control unit 84 adds 1 to the number of days elapsed for the food ingredient in the stay list. If there is no image in the stay food ingredient list 851 that matches the cut-out image (No), the control unit 84 determines that the food ingredient is newly staying and proceeds to step S25, and adds this food ingredient to the stay food ingredient list 851.

[0097] Then, in step S26, if any of the cut-out images has not been processed, the process returns to step S21, and if all the cut-out images have been processed, the process proceeds to step S27. In step S27, the control unit 84 determines whether there is any food material in the remaining food material list 851 that has not been determined to match the food material in any of the cut-out images. If there is any food material in the remaining food material list 851 that has not been determined to match the food material in any of the cut-out images (Yes), the control unit 84 determines that this food material has been consumed, deletes it from the remaining food material list 851, and the process of FIG. 32 ends. If all the food materials in the remaining food material list 851 have been determined to match the food materials in the cut-out images (No), the process of FIG. 32 ends.

[0098] Note that in this embodiment, the feature amounts of the food material images are collated. However, when the food material recognition unit 81 recognizes even the item name, it may be collated in combination with the item name recognized from the food material image. Thereby, it is possible to improve the collation accuracy. The process of determining whether the images match or not is determined by the distance of the feature vectors in which the color, shape, etc. are digitized in multiple dimensions. The food material recognition unit 81 can recognize different food materials when the orientation of the food material changes slightly due to re-stocking or the like by setting a high threshold value for determining a match in the collation. Thereby, only the food materials that the user has forgotten the existence of and has been left without movement for a long time can be managed in the remaining food material list 851.

[0099] Also, in this embodiment, when comparing the feature amounts of the food material images, the information on the storage location (cut-out position of the cut-out image) of the food material in the storage is not considered. Therefore, by setting a low threshold value for determining a match in the collation, it is possible to regard the same food material even if the storage location of the food material changes before and after re-stocking or there is a small change in the orientation of the food material. Thereby, even for the food materials that the user has recognized the existence of and is using, the total number of days stored in the refrigerator can be counted as the elapsed days.

[0100] Also, when comparing the feature amounts of food images, information on the storage location (the cut-out position of the cut-out image) of the food in the storage may be considered. In this case, for example, when there are multiple pieces of the same type of food in the storage and the same type of food, but different individual foods are being compared, since the cut-out positions of the cut-out images are different, it can be determined as a mismatch.

[0101] FIG. 33 is an example of a time chart related to the detection of long-term detained items. This time chart shows, in order from the top, the date, the camera shooting process, the food image cut-out process, the detention determination process, the state of the home screen, and the state of the push notification. The date row shows each day with each arrow. The camera shooting process is indicated by an inverted triangle to show the execution of the process. For example, it is carried out by automatic shooting at the timing when the user of refrigerator 1 opens the door, or by manual shooting when at least one of the shooting buttons 125, 135 is pressed by the user.

[0102] The food image cut-out process is indicated by an inverted triangle to show the execution of the process. For example, it is carried out each time the camera shooting process is performed. The detention determination process is the process shown in FIG. 32 and is executed at a predetermined time (for example, 3:00 am every day) based on the timing information indicated by the inverted triangle. The cut-out image of the latest image taken before the predetermined time is used for this detention determination.

[0103] The detention determination process is a process of comparing the cut-out image of the latest image at the determination timing with the cut-out image of the nearest image more than 24 hours ago. That is, if there is a cut-out image from the previous day, it is compared with the nearest cut-out image among them. If there is no cut-out image from the previous day, it is compared with the nearest cut-out image among the cut-out images of the nearest date.

[0104] In this case, the number of days elapsed obtained from the comparison with the previous captured image is added to the residence days of this food. Note that in this case, since the image cutting process is performed each time a photo is taken, the image cutting process has been completed at the timing of a predetermined time. When the server 2 collectively undertakes the cutting processes of different refrigerators used by multiple users, even when the predetermined times set by each user concentrate on a specific time, it is possible to prevent the food ingredient image cutting process executed by the server 2 from concentrating at a specific time, enabling stable operation of the server 2.

[0105] The state of the home screen and the push notification are for detecting long-term stored items (stored food ingredients) and sending a push notification to the user. The push notification is sent when the number of residence days becomes equal to a predetermined number of days. For example, the user can set to display a reminder on the home screen 46b of the app as shown in FIG. 36 and send a push notification to the mobile terminal 3 as shown in FIG. 35 when the residence days are 10 days or more. Also, the expiration date of the food ingredient may be set as a predetermined number of days.

[0106] FIG. 34 is another example of a time chart related to the detection of long-term stored items. This time chart shows the date, camera shooting process, food ingredient image cutting process, residence determination process, state of the home screen, and state of the push notification in order from the top. The date row indicates each day with each arrow. The camera shooting process is indicated by an inverted triangle to show the execution of the process, and is executed, for example, at the timing when the user of refrigerator 1 opens the door.

[0107] The food ingredient image cutting process is indicated by an inverted triangle to show the execution of the process, and is executed at a predetermined time (for example, 3:00 am every day). The residence determination process is the process shown in FIG. 32 and is executed after the food ingredient listing process based on the timing information shown by an inverted triangle. The cutout image of the latest image taken before a predetermined time is used for this residence determination. The residence determination process is a process of comparing the cutout image of the latest image at the determination timing with the cutout image of the nearest image more than 24 hours ago. That is, if there is a cutout image from the previous day, it is compared with the nearest cutout image among them. If there is no cutout image from the previous day, it is compared with the nearest cutout image among the cutout images of the nearest date. In this case, the system of this embodiment adds the number of elapsed days obtained from the comparison with the previous captured image on the residence day of this food.

[0108] In this case, since only the captured image taken immediately before is cut out at the timing of a predetermined time, the number of times of the food ingredient image cutout process is reduced. Therefore, the operating rate of the CPU 21 of the server 2 can be suppressed, and resources for performing any other process can be secured.

[0109] The state of the home screen and the push notification are for detecting long-term storage items (stored food ingredients) and pushing notifications to the user. When the predetermined number of days is 10 days, if the residence day is 10 days or more, a reminder display described later is performed on the home screen 46b of the app as shown in FIG. 36, and a push notification is sent to the mobile terminal 3 as shown in FIG. 35.

[0110] Note that the availability of the push notification may be able to be turned on / off in the setting item "Notification of detection of long-term storage items". The timing of the notification may be able to be set in the setting item "Notification at a fixed time". The determination timing may be able to be set in the setting item "Once a day · 3:00 AM". The determination condition for "long-term storage items" may be able to be set in the "Determination period for detection of long-term storage items".

[0111] FIG. 35 is a diagram showing a push notification dialog 551 related to stored food ingredients. On the sleep screen 55, a push notification dialog 551 related to the remaining food is displayed. When a long-term storage item is detected, the warning section 88 in FIG. 5 sends a push notification to the mobile terminal 3. Then, the CPU 31 of the mobile terminal 3 executes a refrigerator concierge application program (not shown) to display the push notification dialog 551 on the touch panel display 36.

[0112] FIG. 36 is the home screen 46b of the refrigerator concierge application when a long-term storage item (remaining food) is detected. On the home screen 46b, a notification area 461b, a "Check the Contents of the Refrigerator" button 462, and a usage concierge button 463 are displayed. In the notification area 461b, a "View in More Detail" link 464 is displayed.

[0113] In the notification area 461b, a reminder display including a natto pack that is a long-term storage item, a speech bubble saying "There is food that has been stored for a long time", and the "View in More Detail" link 464 is displayed. When the "View in More Detail" link 464 is tapped, the screen transitions to the temporary list screen 43e of the remaining food.

[0114] FIG. 37 is the temporary list screen 43e of the remaining food. On this temporary list screen 43e of the remaining food, a thumbnail 431 and a product name 432 are displayed. Further, a list addition button 435 is displayed at the bottom of the temporary list screen 43e. When the user taps the list addition button 435, the screen transitions to the food list top screen 40. Also, when the user taps the thumbnail 431 of any food, the screen transitions to the captured image information screen 54b in FIG. 38. Since the remaining food may be food that has been forgotten to be added to the food list 85, a temporary list based on the remaining food is presented.

[0115] FIG. 38 is the captured image information screen 54b. The captured image information screen 54b displays a cut-out image 541, a product name 542, a storage room image 543, and a cut-out frame 544 located thereon. The cut-out image 541 is an image obtained by photographing and cutting out this natto pack. The product name 542 is displayed as "Other Ingredients", and a character string indicating this food is displayed. The storage room image 543 is an image of the storage room where this food is stored. The cut-out frame 544 is a frame indicating the range within the storage room image 543 where this food was recognized.

[0116] Hereinafter, the configuration and effects of the present invention will be described.

[0117] [1] An image acquisition unit (80) that acquires an image of a storage room provided in the refrigerator (1), A food recognition unit (81) that recognizes each food ingredient from the image, A range specification unit (82) that can specify a partial range of the image, A control unit (84) that updates a food list based on information on the food ingredients included in the range acquired by the range specification unit (82) among the food ingredients recognized by the food recognition unit (81); A refrigerator management device (2), characterized by comprising:

[0118] Thereby, it is possible to easily create a list of the food ingredients stored in the refrigerator.

[0119] [2] The food recognition unit (81) recognizes each food ingredient from the image before the range is specified by the range specification unit (82). The refrigerator management device (2) according to claim 1, characterized in that:

[0120] Thereby, it is possible to display a provisional list of food ingredients immediately after the range is specified.

[0121] [3] The refrigerator management device (2) further includes a temporary list presenting unit (83) that presents, on a display unit, images of the foodstuffs included in the range specified by the range specifying unit (82) among the foodstuffs recognized by the foodstuff recognition unit (81). The control unit (84) adds and updates, to the foodstuff list, the foodstuffs selected by the user from among the foodstuffs presented by the temporary list presenting unit (83). The refrigerator management device (2) according to claim 1, characterized in that.

[0122] This makes it possible to manually correct errors in the image recognition of foodstuffs and create a more accurate foodstuff list.

[0123] [4] A partial range of the image of the storage compartment specified by the range specifying unit (82) is specified in advance. The foodstuff recognition unit (81) recognizes each foodstuff included in the range specified in advance in the image of the storage compartment. The refrigerator management device (2) according to claim 1, characterized in that.

[0124] This enables the creation of a more accurate foodstuff list because only the foodstuffs within a predetermined range are recognized by image recognition, and the user can place the foodstuffs they want to be recognized within that range.

[0125] [5] The range specified by the range specifying unit (82) can be freely changed by the user on the screen where the image is displayed. The refrigerator management device (2) according to claim 1, characterized in that.

[0126] This enables the user to cause the refrigerator 1 to recognize the foodstuffs within the desired range.

[0127] [6] A foodstuff image extraction unit (87) that extracts images of respective foodstuffs from the image acquired by the image acquisition unit (80), A retained foodstuff list (851) used for determining the retention of foodstuffs, The refrigerator management device (2) according to claim 1, further comprising

[0128] Thereby, foods staying in the refrigerator for a long time can be listed.

[0129] [7] The control unit (84) compares the image extracted by the food ingredient image extraction unit (87) with the images registered in the staying food ingredient list (851), and when it determines that they are the same food ingredient, updates the staying period of the food ingredient. The refrigerator management device (2) according to claim 6, characterized in that

[0130] Thereby, the staying period of the foods registered in the staying food list can be updated.

[0131] [8] The control unit (84) compares the image extracted by the food ingredient image extraction unit (87) with the images registered in the staying food ingredient list (851), and when an image determined to be the same food ingredient does not exist in the staying food ingredient list (851), registers the food ingredient in the staying food ingredient list (851). The refrigerator management device (2) according to claim 6, characterized in that

[0132] Thereby, newly stored foods in the refrigerator can be registered in the staying food list.

[0133] [9] The control unit (84) compares the image extracted by the food ingredient image extraction unit (87) with the images registered in the staying food ingredient list (851), and if there is any image registered in the staying food ingredient list (851) that does not match any of the food ingredients extracted by the food ingredient image extraction unit (87), deletes the food ingredient from the staying food ingredient list (851). The refrigerator management device (2) according to claim 6, characterized in that

[0134] Thereby, foods that have been consumed and no longer exist in the refrigerator can be deleted from the staying food list.

[0135]

[10] The control unit (84) further combines the cut-out coordinate information used for extracting the image of the food material with the image extracted by the food material image extraction unit (87) and registers it in the staying food material list (851), and combines the cut-out coordinate information of the image with the image of the food material by comparing the image extracted by the food material image extraction unit (87) with the image already registered in the staying food material list (851). The refrigerator management device (2) according to any one of claims 7 to 9, characterized in that.

[0136] Thereby, when the position of the food in the refrigerator changes, the number of elapsed days can be initialized.

[0137]

[11] A warning unit (88) for warning if the staying period exceeds a predetermined value among the food materials registered in the staying food material list (851). The refrigerator management device (2) according to claim 10, further comprising.

[0138] Thereby, it is possible to warn the user about the food that has stayed in the refrigerator for a predetermined period or more.

[0139]

[12] A step of an image acquisition unit (80) acquiring an image of a storage compartment provided in the refrigerator (1); A step of a food material recognition unit (81) recognizing each food material from the image; A step of a range specifying unit (82) being able to specify a partial range of the image; A step of a control unit (84) updating a food material list based on information on the food materials included in the range specified by the range specifying unit (82) among the food materials recognized by the food material recognition unit (81); A refrigerator management method, characterized by comprising.

[0140] Thereby, it is possible to easily create a list of the food materials stored in the refrigerator.

[0141]

[13] Cause a computer to perform a procedure for acquiring an image of a storage compartment provided in a refrigerator (1); perform a procedure for recognizing each food item from the image; perform a procedure for being able to specify a partial range of the image; perform a procedure for updating a food list based on information of food items included in the range among the food items recognized from the image; and a refrigerator management program (241) for causing the above to be executed.

[0142] Thereby, it is possible to easily create a list of food items stored in the refrigerator.

[0143] (Modification example) The present invention is not limited to the above-described embodiments, and includes various modification examples. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. It is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0144] Each of the above configurations, functions, processing units, processing means, etc. may be realized by hardware such as an integrated circuit, for example, for a part or all of them. Each of the above configurations, functions, etc. may also be realized by software by a processor interpreting and executing a program for realizing each function. Information such as a program, table, file, etc. for realizing each function can be placed in a recording device such as a memory, hard disk, SSD (Solid State Drive), or a recording medium such as a flash memory card, DVD (Digital Versatile Disk).

[0145] In each embodiment, control lines and information lines show those considered necessary for explanation, and do not necessarily show all control lines and information lines on the product. In reality, it may be considered that almost all configurations are interconnected. As a modification example of the present invention, for example, there are the following (a) to (f).

[0146] (a) When comparing food ingredients, not only the information on the feature amounts of the cut-out images of the food ingredients but also the cut-out coordinate information of the images may be combined. Thereby, even when there are a plurality of the same kind of food ingredients, neglect can be detected. (b) The range selection frame is not limited to a rectangle, and may be an irregular shape or a combination of a plurality of rectangles, and is not limited.

[0147] (c) What operates as the refrigerator management device is not limited to the server, and for example, a refrigerator or a mobile terminal may operate as the refrigerator management device, and is not limited.

[0148] (d) Photographing the storage room of the refrigerator is not limited to the refrigerator camera. The storage room of the refrigerator may be sequentially photographed by a mobile terminal held by the user. (e) When recognizing food ingredients from the photographed image of the storage room of the refrigerator, the information on the food ingredients may be registered in the food ingredient list without creating a temporary list. (f) The retention period for warning about food ingredients that have stayed for a long time may be determined by whether or not the expiration date of each food ingredient has been exceeded, and is not limited.

Explanation of Reference Numerals

[0149] 1 Refrigerator 2 Server (Refrigerator Management Device) 3 Mobile Terminal 114 RAM 100 Camera Illumination Unit 12 Left Door 117 Door Sensor 116 Refrigerating Chamber 21 CPU 22 ROM 23 RAM 25 Communication Unit 31 CPU 31b Case 31a Lens 32 ROM 33 RAM 34 Memory unit 35 Communication unit 36 Touch panel display 40, 40a, 40b, 40c Food list - Top screen 402 Thumbnail 403 Product name 405 Remaining days 406 "+" button 41, 41b List creation method selection screen 411 "Input from refrigerator camera image" button 412 "Input food image with camera" button 413 "Input food name by voice" button 414 "Input food name in text" button 42, 42b, 42c, 42d Range selection screen 421 Storage room image 422 Range selection frame 423 Indoor selection button 424 List import button 425 Food detection button 43, 43b, 43c, 43d Temporary list screen 431 Thumbnail 432 Product name 433 Storage location 435 List addition button 436 "Import from another image" button 437 Delete button 438 List addition button 439 Japanese flick input area 46, 46b Home screen 461 Notification area 462 "Check the contents of the refrigerator" button 463 Usage concierge button 461b Notification area 464 "View in more detail" link 491 Home button 492 Food management button 493 Operation status button 494 Support button 495 Settings button 52 represents the location selection screen 521 Left door 522 Storage room 523 Right door 524 Upper vegetable compartment 525 Lower freezer 526 Cancel button 54b Imported image information screen 54 Imported image information screen 541 Cut-out image 542 Product name 543 Storage room image 544 Cut-out frame 551 Push notification dialog 55 Sleep screen 80 Image acquisition unit 81 Food recognition unit 82 Range specification unit 83 Temporary list presentation unit 84 Control unit 85 Food list 86 Main list presentation unit 87 Food image extraction unit 851 Stored food list 88 Warning unit 14 Ice making compartment 15 Upper freezer 16 Vegetable compartment 17 Lower freezer 123 Door pocket 101 Main body part 102 Support part 103 Lens 122 Inner panel 133 Door pocket 124 Rotary partition 125 Shooting button 135 Shooting button

Claims

1. An image acquisition unit that acquires an image of a storage room provided in a refrigerator; A food recognition unit that recognizes each food from the image; A range designation unit that can designate a partial range of the image; A control unit that updates a food list based on information on foods included in the range acquired by the range designation unit among the foods recognized by the food recognition unit; A refrigerator management device characterized by comprising the above.

2. The food recognition unit recognizes each food from the image before the range is designated by the range designation unit. The refrigerator management device according to claim 1, characterized by the above.

3. The refrigerator management device further comprises a temporary list presentation unit that presents, to a display unit, images of foods included in the range designated by the range designation unit among the foods recognized by the food recognition unit. The control unit adds and updates, to the food list, those selected by the user among the foods presented by the temporary list presentation unit. The refrigerator management device according to claim 1, characterized by the above.

4. A partial range of the image of the storage room designated by the range designation unit is designated in advance. The food recognition unit recognizes each food included in the range designated in advance in the image of the storage room. The refrigerator management device according to claim 1, characterized by the above.

5. The range designated by the range designation unit can be freely changed by the user on the screen where the image is displayed. The refrigerator management device according to claim 1, characterized by the above.

6. A food image extraction unit that extracts images of respective foods from the image acquired by the image acquisition unit; and a retention food list used for determining the retention of foods. The refrigerator management device according to claim 1, characterized by further comprising the above.

7. The control unit compares the image extracted by the food image extraction unit with the images registered in the retention food list. When it determines that they are the same food, it updates the retention period of the food. The refrigerator management device according to claim 6, characterized by the above.

8. The control unit compares the image extracted by the food image extraction unit with the images registered in the retention food list. When an image determined to be the same food does not exist in the retention food list, it registers the food in the retention food list. The refrigerator management device according to claim 6, characterized by the above.

9. ​ ​ ​ The control unit compares the image extracted by the food material image extraction unit with the images registered in the remaining food material list. If there is an image registered in the remaining food material list that does not match any of the food materials extracted by the food material image extraction unit, the control unit deletes the food material from the remaining food material list. The refrigerator management device according to claim 6, characterized in that.

10. The control unit further combines the cut-out coordinate information used for extracting the image of the food material with the image extracted by the food material image extraction unit and registers it in the remaining food material list. When comparing the image extracted by the food material image extraction unit with the images registered in the remaining food material list, the control unit combines the cut-out coordinate information of the image together with the image of the food material. The refrigerator management device according to any one of claims 7 to 9, characterized in that.

11. A warning unit that gives a warning if the residence period of the food materials registered in the remaining food material list exceeds a predetermined value. The refrigerator management device according to claim 10, further comprising the warning unit.

12. A step of an image acquisition unit acquiring an image of a storage chamber provided in the refrigerator; A step of a food material recognition unit recognizing each food material from the image; A step of a range specification unit being able to specify a partial range of the image; A step of a control unit updating a food material list based on information on the food materials included in the range specified by the range specification unit among the food materials recognized by the food material recognition unit; A refrigerator management method, characterized by comprising the steps.

13. On a computer, A procedure for acquiring an image of a storage chamber provided in the refrigerator; A procedure for recognizing each food material from the image; A procedure for being able to specify a partial range of the image; A procedure for updating a food material list based on information on the food materials included in the range among the food materials recognized from the image; A refrigerator management program for causing the steps to be executed.

Images