Program, terminal device and terminal device control method

Abstract

To provide a program, a terminal device and a terminal device control method which enable a user to intuitively assess what kind of food is stored in a refrigerator.SOLUTION: A program causes a computer of a terminal device mounted with a display section to function as a display control section which displays: a refrigerator interior image representing an interior of a refrigerator viewed from a front side thereof on the display section; and a food image representing food stored in the refrigerator in an area on the refrigerator interior image corresponding to a storage area of the food inside the refrigerator.SELECTED DRAWING: Figure 16

Description

Technical Field

[0001] The present disclosure relates to a program, a terminal device, and a control method for the terminal device.

Background Art

[0002] Patent Document 1 discloses a technique in which when an object passing through an opening of a refrigerator is detected, a camera installed above the refrigerator executes shooting, and information on food stored in the refrigerator is displayed on the refrigerator or a terminal device based on the shooting result.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides a program, a terminal device, and a control method for the terminal device that enable a user to intuitively grasp what kind of food is stored in a refrigerator.

Means for Solving the Problems

[0005] The program in the present disclosure causes a computer of a terminal device including a display unit to display, by the display unit, an in-compartment image showing the inside of the refrigerator as viewed from the front of the refrigerator, and causes the computer to function as a display control unit that displays a food image indicating the food in an area corresponding to the storage area of the food stored in the in-compartment image.

[0006] In addition, the terminal device in the present disclosure includes a display unit, and a display control unit that displays, on the display unit, an in-compartment image showing the inside of the refrigerator as viewed from the front of the refrigerator, and displays a food image indicating the food in an area corresponding to the storage area of the food stored in the refrigerator in the displayed in-compartment image.

[0007] In addition, the control method of the terminal device in the present disclosure displays an in-compartment image showing the inside of the refrigerator as viewed from the front of the refrigerator, and displays a food image indicating the food in an area corresponding to the storage area of the food stored in the refrigerator in the displayed in-compartment image.

Advantages of the Invention

[0008] In the program, terminal device, and control method of the terminal device in the present disclosure, an image of the food is displayed in an area corresponding to the storage area of the food stored in the refrigerator within the image showing the inside of the refrigerator. Therefore, the user can intuitively grasp what kind of food is stored in the refrigerator.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

[0010] (Findings etc. Underlying the Present Disclosure) When the inventors arrived at the idea of the present disclosure, there was a technology in which a terminal device displays a list of foods stored in a refrigerator. However, as in the conventional technology, the inventors discovered the problem that users cannot intuitively grasp which foods are stored in the refrigerator just by simply displaying a list of foods, and in order to solve this problem, they came to constitute the subject matter of the present disclosure. Therefore, the present disclosure provides a program, a terminal device, and a control method for the terminal device that enable a user to intuitively grasp what kind of foods are stored in a refrigerator.

[0011] Hereinafter, embodiments will be described in detail with reference to the drawings. However, there may be cases where more detailed explanations than necessary are omitted. For example, there may be cases where detailed explanations of well-known matters are omitted, or duplicate explanations for substantially the same configurations are omitted. It should be noted that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims thereby.

[0012] (Embodiment 1) First, Embodiment 1 will be described. [1-1. Configuration] [1-1-1. Configuration of Food Management System] FIG. 1 is a diagram showing the configuration of a food management system 1000.

[0013] The food management system 1000 is a system that manages the storage date and time, etc. of foods stored in the refrigerator 1 for the foods stored in the refrigerator 1. The refrigerator 1 corresponds to an example of the storage in the present disclosure. The food corresponds to an example of the article in the present disclosure.

[0014] The food management system 1000 includes a refrigerator 1. The refrigerator 1 includes a main cabinet 10 with an open front. In the main cabinet 10, a refrigerating compartment 11, an ice-making compartment 12, a fresh freezing compartment 13, a freezing compartment 14, and a vegetable compartment 15 are formed. A rotary left door 11A and a right door 11B are provided at the opening in the front of the refrigerating compartment 11. Drawers 12A, 13A, 14A, 15A for storing food are provided in each of the ice-making compartment 12, the fresh freezing compartment 13, the freezing compartment 14, and the vegetable compartment 15. The drawers 14A, 15A of the present disclosure include two cases CA, an upper case JCA and a lower case GCA. The upper case JCA corresponds to an example of the first case of the present disclosure. The lower case GCA corresponds to an example of the second case of the present disclosure.

[0015] In the following description, when the left door 11A and the right door 11B are not distinguished, they are referred to as "doors" and given the reference numeral "11C". Also, in the following description, when the drawers 12A, 13A, 14A, 15A are not distinguished, they are referred to as "drawers" and given the reference numeral "16A".

[0016] The food management system 1000 includes a camera unit 2. The camera unit 2 is a device that performs photographing and detects food entering and leaving the refrigerator 1 from the photographed images. The camera unit 2 is provided on the upper surface 10B of the main cabinet 10 of the refrigerator 1. The camera unit 2 corresponds to an example of the photographing device of the present disclosure.

[0017] The food management system 1000 includes a terminal device 3. The terminal device 3 is, for example, a smartphone or a tablet terminal. An application program for managing the food in the refrigerator 1 is installed in the terminal device 3, and the terminal device 3 communicates with a food management server 4 through the functions of this application program. In the following description, this application program is referred to as the "food management app" and given the reference numeral "311". The food management app 311 corresponds to an example of the program and application program of the present disclosure.

[0018] In FIG. 1, the user P at home is shown by a solid line, and the user P who has gone out from the home H is shown by a dotted line. When the terminal device 3 is used by the user P at home, it communicates with the food management server 4 through the communication device 5 or without passing through the communication device 5 by the function of the food management application 311. Further, when the terminal device 3 is used by the user P who has gone out from the home H and a communication connection cannot be established with the communication device 5, it communicates with the food management server 4 without passing through the communication device 5 by the function of the food management application 311.

[0019] The communication device 5 is connected to the global network GN including the Internet, a telephone network, and other communication networks, and communicates with the food management server 4 connected to the global network GN. The communication device 5 is an interface device for connecting the terminal device 3 to the global network GN.

[0020] The food management system 1000 includes a food management server 4. The food management server 4 is a server device that manages the food in the refrigerator 1 and is connected to the global network GN. In each figure, the food management server 4 is represented by one block, but this does not necessarily mean that the food management server 4 is constituted by a single server device. For example, the food management server 4 may be constituted by including a plurality of server devices with different processing contents.

[0021] Next, the configurations of the camera unit 2, the terminal device 3, and the food management server 4 will be described.

[0022] [1-1-2. Configuration of Camera Unit] The camera unit 2 will be described with reference to FIGS. 2-4. In FIGS. 2 to 4, the X-axis, Y-axis, and Z-axis are illustrated. The X-axis, Y-axis, and Z-axis are orthogonal to each other. The Z-axis indicates the vertical direction. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the left-right direction and also indicates the width direction of the refrigerator 1 and the camera unit 2. The Y-axis indicates the front-rear direction and also indicates the depth direction of the refrigerator 1 and the camera unit 2. The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the front direction. The positive direction of the Z-axis indicates the upward direction.

[0023] FIG. 2 is a side view of the camera unit 2 installed in the refrigerator 1 as viewed from the right side. FIG. 3 is a perspective view of the position regulating member 205 in contact with the front edge 10A of the main body 10 as viewed from the left front side.

[0024] As shown in FIGS. 2 and 3, the camera unit 2 includes a main body 201 having an outer box shape installed on the upper surface 10B of the main body 10. Further, the camera unit 2 includes a photographing member 202 extending forward from the main body 201 above the front part of the main body 201. The photographing member 202 includes a refrigerator chamber camera 203 and a drawer camera 204. Each of the refrigerator chamber camera 203 and the drawer camera 204 is provided in front of the main body 201 at the lower part of the photographing member 202.

[0025] The main body 201 includes a position regulating member 205 that regulates the position of the photographing member 202 in the front-rear direction of the refrigerator 1 by coming into contact with the front edge 10A of the main body 10. The position regulating member 205 is a position regulating piece extending downward from the bottom surface 206 of the main body 201 in the installed state. The position regulating member 205 extends in the left-right direction of the main body 201 on the front surface of the main body 201 and has a mark 207 at the center in the left-right direction of the main body 201. This mark 207 indicates the central position of the camera unit 2 in the left-right direction.

[0026] The length of the position regulating member 205 is formed to be shorter than the length from the upper surface 10B of the main cabinet 10 to the packing provided between the refrigerator 1 and the door 11C in the installed state. More specifically, the length of the position regulating member 205 is formed to be shorter than the length from the upper surface 10B of the main cabinet 10 to the door packing provided substantially parallel to the upper surface 10B at the upper part of the inner surface of the door 11C in the installed state.

[0027] The main body 201 is fixed at the position regulated by the position regulating member 205 by the built-in magnet 208. The means for fixing the main body 201 is not limited to the magnet 208, and other means such as double-sided tape may be used.

[0028] FIG. 4 is a front view of the refrigerator 1 and the camera unit 2 as viewed from the front side. In the refrigerator 1 shown in FIG. 4, the left door 11A and the right door 11B are in the open state. FIG. 5 is a plan view of the refrigerator 1 and the camera unit 2 as viewed from the right side. In the refrigerator 1 shown in FIG. 5, the drawer 15A is in the open state, that is, the pulled-out state. In the camera unit 2 shown in FIGS. 4 and 5, the mark 207 is located substantially at the center in the left-right direction of the refrigerator 1, and the position regulating member 205 is in contact with the front edge 10A.

[0029] The camera unit 2 has a refrigerator camera 203 and a drawer camera 204. The refrigerator camera 203 photographs the food entering and leaving the refrigerator compartment 11 by photographing downward from the front upper part of the refrigerator 1.

[0030] For example, the photographing range of the refrigerator camera 203 includes the range A1 shown in FIG. 4 in the front view. The range A1 is a range including the door pocket 111 provided on the door 11C and the opening of the front surface of the refrigerator compartment 11 when the door 11C is in the open state. Also, the photographing range of the refrigerator camera 203 includes the range A3 shown in FIG. 5 in the side view. The range A3 is a range including from the open door 11C to the front end of the shelf 112 in the front-rear direction and including the opening of the refrigerator compartment 11 in the vertical direction.

[0031] The extraction camera 204 captures images of the food items being put in and taken out of the drawer-type storage compartment by shooting from the upper front to the lower front of the refrigerator 1.

[0032] For example, the shooting range of the extraction camera 204 includes the range A2 shown in FIG. 4 in the front view. The range A2 is a range including the drawer 16A in the front view of the refrigerator 1. Also, the shooting range of the extraction camera 204 includes the range A4 shown in FIG. 5 in the side view. The range A4 is a range including the drawer 16A in the most extended state in the front-rear direction.

[0033] The camera unit 2 includes a distance measuring sensor 209 for the left door, a distance measuring sensor 210 for the right door, and a distance measuring sensor 211 for extraction.

[0034] The distance measuring sensor 209 for the left door is a distance measuring sensor for detecting the opening and closing state of the left door 11A. The distance measuring sensor 209 for the left door detects the separation distance from the left door 11A as the opening and closing state of the left door 11A.

[0035] The distance measuring sensor 210 for the right door is a distance measuring sensor for detecting the opening and closing state of the right door 11B. The distance measuring sensor 210 for the right door detects the separation distance from the right door 11B as the opening and closing state of the right door 11B.

[0036] The distance measuring sensor 211 for extraction is a distance measuring sensor for detecting the opening and closing state of the drawer 16A. Note that when the drawer 16A is in the open state, it means it is in the state of being pulled out from the corresponding drawer-type storage compartment. Also, when the drawer 16A is in the closed state, it means it is in the state of being stored in the corresponding drawer-type storage compartment. The distance measuring sensor 211 for extraction detects the separation distance from the drawer 16A as the opening and closing state of the drawer 16A. In the present embodiment, the distance measuring sensor 211 for extraction detects the opening and closing states of each of the drawers 14A and 15A.

[0037] The camera unit 2 has a human presence sensor 212. The human presence sensor 212 outputs a detection value to the camera unit control section 20. The human presence sensor 212 is provided at a predetermined position of the imaging member 202. The human presence sensor 212 corresponds to an example of the "sensor" of the present disclosure.

[0038] Note that the arrangement order and installation positions of the respective sensors and cameras in FIGS. 4 and 5 are illustrated for convenience of explanation of the camera unit 2, and these arrangement order and installation positions are not limited to the arrangement order and installation positions shown in FIGS. 4 and 5.

[0039] FIG. 6 is a block diagram showing the configuration of the camera unit 2. The camera unit 2 includes a camera unit control section 20, a camera unit communication section 21, a sensor section 22, an imaging section 23, and a power supply section 24.

[0040] The camera unit control section 20 includes a camera unit processor 220 which is a processor such as a CPU (Central Processing Unit) or an MPU (Micro-processing unit), and a camera unit storage section 230. The camera unit control section 20 controls each part of the camera unit 2 by the camera unit processor 220 reading out and executing a control program stored in the camera unit storage section 230. The camera unit processor 220 functions as a camera unit communication control section 221, a detection value processing section 222, an imaging control section 223, a recording control section 224, and a food detection section 225 by executing the control program stored in the camera unit storage section 230.

[0041] The camera unit storage unit 230 includes a memory that stores programs executed by the camera unit processor 220 and data processed by the camera unit processor 220. The camera unit storage unit 230 stores a control program executed by the camera unit processor 220, the camera unit ID 4112 that is the identification information of the camera unit 2, and various other data. The camera unit storage unit 230 has a non-volatile storage area. Also, the camera unit storage unit 230 may include a volatile storage area and constitute the work area of the camera unit processor 220.

[0042] The camera unit communication unit 21 is a communication interface including a communication circuit and a connector etc. that conform to a predetermined communication standard, and communicates with the food management server 4 and the terminal device 3 under the control of the camera unit control unit 20. In the present embodiment, the communication standard used by the camera unit communication unit 21 is exemplified by a wireless communication standard, but a wired communication standard may also be used.

[0043] The sensor unit 22 includes a distance measurement sensor 209 for the left door, a distance measurement sensor 210 for the right door, a distance measurement sensor 211 for pulling out, and a human sensor 212, and outputs a detection value for each sensor to the camera unit control unit 20.

[0044] The photographing unit 23 includes a refrigerator camera 203 and a pulling-out camera 204. The photographing unit 23 outputs the photographing results of the refrigerator camera 203 and the pulling-out camera 204 to the camera unit control unit 20. The refrigerator camera 203 and the pulling-out camera 204 photograph moving images. Hereinafter, when the refrigerator camera 203 and the pulling-out camera 204 are not distinguished, they are referred to as "camera" and given the symbol "213".

[0045] The power supply unit 24 includes hardware such as a power supply circuit, and supplies power to each part of the camera unit 2. In the present embodiment, the power cable 7 of the camera unit 2 is connected to the commercial AC power supply 8, and the power supply unit 24 supplies power to each of the camera units 2 based on the power supplied from the external commercial AC power supply 8.

[0046] Note that the power supply source for the power supply unit 24 is not limited to the external commercial AC power supply 8, and it may also be the refrigerator 1 or a battery. When the power supply source is the former, the power supply unit 24 and the refrigerator 1 are equipped with hardware such as a port that complies with the USB power supply capable standard. And when the power supply source is the former, the USB cable of the camera unit 2 is connected to the USB port of the refrigerator 1, and the refrigerator 1 supplies power to the power supply unit 24 via the USB cable. When the power supply source is the latter, the camera unit 2 is equipped with a battery.

[0047] As described above, the camera unit control unit 20 functions as a camera unit communication control unit 221, a detection value processing unit 222, a shooting control unit 223, a recording control unit 224, and a food detection unit 225. The food detection unit 225 corresponds to an example of the article detection unit.

[0048] The camera unit communication control unit 221 communicates with the food management server 4 via the camera unit communication unit 21.

[0049] Based on the detection values of each sensor output from the sensor unit 22, the detection value processing unit 222 determines whether each of the left door 11A, right door 11B, drawers 14A and 15A is in an open state or a closed state. When the detection value processing unit 222 determines that either the left door 11A or the right door 11B is in an open state, it outputs door open state information indicating that the door 11C is in an open state to the recording control unit 224 and the food detection unit 225. Also, when the detection value processing unit 222 determines that both the left door 11A and the right door 11B are in a closed state, it outputs door closed state information indicating that the door 11C is in a closed state to the recording control unit 224 and the food detection unit 225. When the detection value processing unit 222 determines that either the drawer 14A or 15A is in an open state, it outputs drawer open state information indicating that either the drawer 14A or 15A is in an open state to the recording control unit 224 and the food detection unit 225. Also, when the detection value processing unit 222 determines that both the drawers 14A and 15A are in a closed state, it outputs drawer closed state information indicating that the drawers 14A and 15A are in a closed state to the recording control unit 224 and the food detection unit 225.

[0050] Further, based on the detection value of the human presence sensor 212 output from the sensor unit 22, the detection value processing unit 222 determines whether a person is present around the refrigerator 1. When the detection value processing unit 222 determines that a person is present around the refrigerator 1, it outputs presence / absence information indicating the presence of a person to the shooting control unit 223. Also, when the detection value processing unit 222 determines that no person is present around the refrigerator 1, it outputs presence / absence information indicating the absence of a person to the shooting control unit 223.

[0051] When the shooting control unit 223 receives presence / absence information indicating the presence of a person output from the detection value processing unit 222, it starts shooting with the refrigerator compartment camera 203 and the drawer camera 204. When the shooting control unit 223 receives presence / absence information indicating the absence of a person output from the detection value processing unit 222, it ends shooting with the refrigerator compartment camera 203 and the drawer camera 204.

[0052] When the detection value processing unit 222 outputs door open state information, the recording control unit 224 starts recording the shooting result of the refrigerator compartment camera 203. When the detection value processing unit 222 outputs door closed state information, the recording control unit 224 ends the recording of the shooting result of the refrigerator compartment camera 203 that has been started. Also, when the detection value processing unit 222 outputs drawer open state information, the recording control unit 224 starts recording the shooting result of the drawer camera 204. When the detection value processing unit 222 outputs drawer closed state information, the recording control unit 224 ends the recording of the shooting result of the drawer camera 204 that has been started. When the recording control unit 224 ends the recording, it generates the moving image from the start to the end of the recording as one recording file 231 and stores it in the camera unit storage unit 230.

[0053] The food detection unit 225 detects the food that has been put in and taken out of the refrigerator 1 based on the recording file 231 stored in the camera unit storage unit 230.

[0054] FIG. 7 is a block diagram showing the configurations of the terminal device 3 and the food management server 4.

[0055] [1-1-3. Configuration of Food Management Server] First, the configuration of the food management server 4 will be described. The food management server 4 includes a server control unit 40 and a server communication unit 41.

[0056] The server control unit 40 includes a server processor 400 which is a processor such as a CPU or an MPU, and a server storage unit 410. The server control unit 40 controls each part of the food management server 4 by the server processor 400 reading and executing the control program stored in the server storage unit 410. The server processor 400 functions as a server communication control unit 401 and an information processing unit 402 by executing the control program stored in the server storage unit 410.

[0057] The server storage unit 410 has a memory that stores programs executed by the server processor 400 and data processed by the server processor 400. The server storage unit 410 stores a control program executed by the server processor 400, a refrigerator food management database 411, a taken-out food management database 412, and various other data. The server storage unit 410 has a non-volatile storage area. The server storage unit 410 may include a volatile storage area and constitute a work area of the server processor 400.

[0058] FIG. 8 is a diagram showing an example of one record stored in the refrigerator food management database 411. The refrigerator food management database 411 is a database that stores various information related to foods stored in the refrigerator 11. In the following description, one record stored in the refrigerator food management database 411 is referred to as a "refrigerator food management record" and is denoted by the symbol "RR".

[0059] The refrigerator food management record RR has an account ID 4111, a camera unit ID 4112, refrigerator model information 4113, in-refrigerator image data 4114, in-out information 4115, food image data 4116, storage date and time information 4117, divided area number 4118, memo input information 4119, memo information 4130, deletion information 4131, and deletion date and time information 4132.

[0060] The account ID 4111 is identification information that identifies the account assigned to the user P who uses the food management application 311.

[0061] The refrigerator model information 4113 is information indicating the model of the refrigerator 1 used by the user P indicated by the account ID 4111 associated in the same record.

[0062] The refrigerated compartment interior image data 4114 is the image data of an image showing the interior of the refrigerated compartment 11 of the refrigerator 1. Hereinafter, the image showing the interior of the refrigerated compartment 11 is referred to as the "refrigerated compartment interior image" and is denoted by the symbol "RG". The refrigerated compartment interior image data 4114 shows the refrigerated compartment interior image RG corresponding to the model indicated by the refrigerator model information 4113 associated in the same record. The refrigerated compartment interior image RG corresponds to an example of the interior image.

[0063] Figure 9 is a diagram showing an example of the refrigerated compartment interior image RG. The refrigerated compartment interior image RG shown in Figure 9 includes a left door pocket image RG1, a refrigerated compartment front image RG2, and a right door pocket image RG3. In the refrigerated compartment interior image RG shown in Figure 9, the left door pocket image RG1, the refrigerated compartment front image RG2, and the right door pocket image RG3 are arranged in this order from the left in the figure. The left door pocket image RG1 is an image showing the door pocket 111 formed in the left door 11A. The refrigerated compartment front image RG2 is an image of the refrigerated compartment 11 seen from the front when the left door 11A and the right door 11B are in the open state. The right door pocket image RG3 is an image showing the door pocket 111 formed in the right door 11B.

[0064] The refrigerated compartment interior image RG has its image area divided into a plurality of areas. Each of these areas is referred to as a "divided area" and is denoted by the symbol "BA". The refrigerated compartment interior image RG shown in Figure 9 is divided into 15 divided areas BA. Different numbers are assigned to each of the divided areas BA. In the following description, this number is referred to as the divided area number and is denoted by the symbol "4118". Note that in Figure 9, which divided area BA each divided area number is assigned to is illustrated for convenience, and the divided area number 4118 is not displayed in the refrigerated compartment interior image RG displayed on the refrigerated compartment home screen GM2 described later.

[0065] Returning to the description of Figure 8, the in / out storage presence / absence information 4115 is information indicating whether there has been a new in / out storage.

[0066] The food image data 4116 is the image data of food. In the present embodiment, the image indicated by this food image data 4116 is the captured image of the refrigerator camera 203.

[0067] The storage date and time information 4117 is information indicating the storage date and time when the food is stored in the refrigerator 1.

[0068] The refrigerator food management record RR has a divided area number 4118 for each food image data 4116.

[0069] The memo input presence / absence information 4119 is information indicating whether a memo is input or not.

[0070] The memo information 4130 is information indicating the input memo.

[0071] The deletion information 4131 is information indicating whether the associated sub-record is a sub-record to be deleted from the refrigerator food management record RR. A sub-record is a record stored in the refrigerator food management record RR and has the food image data 4116, the storage date and time information 4117, the divided area number 4118, the memo input presence / absence information 4119, and the memo information 4130.

[0072] The deletion date and time information 4132 is information indicating the date and time when the associated sub-record is to be deleted. When the deletion information 4131 indicates a deletion target, the information processing unit 402 described later deletes the sub-record, the deletion information 4131 associated with this sub-record, and the deletion date and time information 4132 from the refrigerator food management record RR at a predetermined timing.

[0073] FIG. 10 is a diagram showing one record stored in the taken-out food management database 412. The taken-out food management database 412 is a database that stores various information related to the food in the drawer-type storage compartment. Note that the taken-out food management database 412 is a database related to the vegetable compartment 15 in the description with reference to FIGS. 23-25. In the following description, one record stored in the withdrawal food management database 412 is referred to as a "withdrawal food management record" and is denoted by the symbol "HR".

[0074] The withdrawal food management record HR has an account ID 4111, a camera unit ID 4112, update date and time information 4121, upper case information 4122, and lower case information 4124.

[0075] The update date and time information 4121 is information indicating the date and time when the content of the withdrawal food management record HR was updated.

[0076] The upper case information 4122 has upper case image data 4123, food image data 4116, memo input presence / absence information 4119, and memo information 4130.

[0077] The upper case image data 4123 is image data of an upper case image JCG that is an image of an upper case JCA. The upper case image JCG is a captured image by the withdrawal camera 204. The upper case image JCG corresponds to an example of the first case image of the present disclosure.

[0078] In the present embodiment, the image indicated by the food image data 4116 included in the upper case information 4122 is an image cut out from the upper case image JCG.

[0079] The lower case information 4124 has lower case image data 4125, food image data 4116, memo input presence / absence information 4119, and memo information 4130.

[0080] The lower case image data 4125 is image data of a lower case image GCG that is an image of a lower case GCA. The lower case image GCG is a captured image by the withdrawal camera 204. The lower case image GCG corresponds to an example of the second case image of the present disclosure. Hereinafter, when the upper case image JCG and the lower case image GCG are not distinguished, they are referred to as "case images" and denoted by the symbol "CG".

[0081] In this embodiment, the image shown by the food image data 4116 included in the lower case information 4124 is an image cut out from the lower case image GCG.

[0082] Returning to the description of FIG. 7, the server communication unit 41 is a communication interface including components related to communication such as a wireless circuit and an antenna according to a predetermined communication standard, and communicates with a device connected to the global network GN according to the predetermined communication standard. In this embodiment, the server communication unit 41 communicates with the camera unit 2 and the terminal device 3.

[0083] As described above, the server control unit 40 functions as a server communication control unit 401 and an information processing unit 402.

[0084] The server communication control unit 401 communicates with the camera unit 2 and the terminal device 3 via the server communication unit 41.

[0085] The information processing unit 402 performs various information processes based on the information received by the server communication control unit 401 from the camera unit 2 and the terminal device 3.

[0086] [1-1-4. Configuration of Terminal Device] Next, the functional configuration of the terminal device 3 will be described. The terminal device 3 includes a terminal control unit 30, a terminal communication unit 31, and a touch panel 32. The touch panel 32 corresponds to an example of a display unit.

[0087] The terminal control unit 30 includes a terminal processor 300 which is a processor such as a CPU or an MPU, and a terminal storage unit 310. The terminal control unit 30 controls each part of the terminal device 3 by the terminal processor 300 reading out and executing a control program stored in the terminal storage unit 310. A food management application 311 is installed in the terminal device 3. The terminal processor 300 functions as an application execution unit 301 by reading out and executing the food management application 311. The terminal processor 300 corresponds to an example of the computer of the present disclosure. Further, the application execution unit 301 corresponds to an example of the display control unit of the present disclosure.

[0088] The terminal storage unit 310 has a memory that stores programs executed by the terminal processor 300 and data processed by the terminal processor 300. The terminal storage unit 310 stores a control program executed by the terminal processor 300, the food management application 311, the account ID 4111, and various other data. The terminal storage unit 310 has a non-volatile storage area. Further, the terminal storage unit 310 may include a volatile storage area and constitute a work area of the terminal processor 300.

[0089] The terminal communication unit 31 is a communication interface including components such as a wireless circuit and an antenna according to a predetermined communication standard, and communicates with a device connected to the global network GN according to the predetermined communication standard. In the present disclosure, a wireless communication standard is exemplified as the communication standard used by the terminal communication unit 31.

[0090] The touch panel 32 includes a display panel such as a liquid crystal display panel and a touch sensor provided over or integrally with the display panel.

[0091] As described above, the terminal control unit 30 functions as the application execution unit 301. The application execution unit 301 transmits and receives various information to and from the food management server 4 via the terminal communication unit 31.

[0092] Further, the application execution unit 301 causes the touch panel 32 to display a user interface related to the management of the food in the refrigerator 1. In the following description, this user interface is referred to as the "application UI" and is denoted by the reference numeral "320". By causing the touch panel 32 to display the application UI 320, the application execution unit 301 provides various information related to the food in the refrigerator 1 to the user P and accepts various inputs related to the food in the refrigerator 1 from the user P.

[0093] [1-2. Operations] Next, the operations of each part related to the food management system 1000 will be described.

[0094] [1-2-1. Installation of Camera Unit] The camera unit 2 is separate from the refrigerator 1. Therefore, the camera unit 2 is installed on the upper surface 10B of the refrigerator 1 by the user P. Since the position regulating member 205 regulates the position of the camera unit 2 in the front-rear direction of the refrigerator 1 as described above, the user P adjusts the position of the camera unit 2 in the left-right direction of the refrigerator 1 when installing the camera unit 2. Hereinafter, a plurality of examples of the method for adjusting the position of the camera unit 2 in the left-right direction of the refrigerator 1 will be illustrated.

[0095] [1-2-1-1. First Position Adjustment]

[0096] In the first position adjustment, the user P uses the app UI 320. In the first position adjustment, the app execution unit 301 causes the app UI 320 of the adjustment screen GM1 to be displayed on the touch panel 32.

[0097] FIG. 11 is a diagram showing an example of the adjustment screen GM1. The adjustment screen GM1 is provided with a shooting result display area HA1. The shooting result display area HA1 is a rectangular area and is an area for displaying the shooting result of either the refrigerator compartment camera 203 or the drawer camera 204. In the shooting result display area HA1, a line L1 extending in the vertical direction is provided approximately at the center in the left-right direction in the figure.

[0098] The adjustment screen GM1 includes adjustment method information J1. The adjustment method information J1 prompts the user to adjust the position so that the center of the refrigerator 1 in the left-right direction that appears in the shooting result display area HA1 becomes the line L1 provided in the shooting result display area HA1.

[0099] User P installs the camera unit 2 in the refrigerator 1 according to the adjustment method indicated by the adjustment method information J1. As a result, the camera unit 2 enables each of the refrigerator compartment camera 203 and the drawer camera 204 to include the food entering and leaving the refrigerator 1 within the shooting range. Also, the camera unit 2 enables each distance measuring sensor included therein to detect the opening and closing of the corresponding door 11C or drawer 16A.

[0100] In the first position adjustment, the camera unit 2 transmits the shooting result of either the refrigerator compartment camera 203 or the drawer camera 204 to the terminal device 3. In the first position adjustment, the camera unit 2 and the terminal device 3 communicate directly via the food management server 4 or by means of short-range wireless communication or the like.

[0101] The camera unit communication control unit 221 of the camera unit 2 transmits the shooting result of either the refrigerator compartment camera 203 or the drawer camera 204 to the terminal device 3. Then, the app execution unit 301 displays the received shooting result in the shooting result display area HA1.

[0102] [1-2-1-2. Second Position Adjustment] In the second position adjustment, user P uses the center position recording paper CY.

[0103] FIG. 12 is a diagram showing an example of the center position recording paper CY. The center position recording paper CY is a paper on which a center mark MR2 indicating the center position in the left-right direction of the refrigerator 1 is recorded. In addition to the center mark MR2, the left end mark MR1 indicating the left end position in the left-right direction of the refrigerator 1 and the right end mark MR3 indicating the right end position in the left-right direction of the refrigerator 1 are recorded on the center position recording paper CY.

[0104] The central position recording paper CY is packed together with the refrigerator 1 and reaches the user P. Note that the camera unit 2 may or may not be packed together with the central position recording paper CY and the refrigerator 1. The length of the central position recording paper CY packed together with the refrigerator 1 from the left end mark MR1 to the right end mark MR3 corresponds to the width of this refrigerator 1.

[0105] In the second position adjustment, the user P uses the central position recording paper CY to grasp the central position of the refrigerator 1 in the left - right direction. For example, the user P attaches the central position recording paper CY to the refrigerator 1 so that the left end mark MR1 is at the left end of the refrigerator 1 and the right end mark MR3 is at the right end of the refrigerator 1, and grasps the central position of the refrigerator 1 in the left - right direction with the central mark MR2 as a reference. Then, the user P adjusts the position of the camera unit 2 so that the mark 207 on the position regulating member 205 of the camera unit 2 is at the grasped central position.

[0106] By the second position adjustment, the camera unit 2 can perform various detections as in the first position adjustment.

[0107] [1 - 2 - 1 - 3. Third position adjustment] In the third position adjustment, a mark indicating the central position of the refrigerator 1 in the left - right direction is attached to a predetermined position of the refrigerator 1. In the third position adjustment, the mark 207 on the position regulating member 205 and the mark attached to the refrigerator 1 are used.

[0108] In the third position adjustment, the user P adjusts the position of the camera unit 2 so that the position of the mark attached to the refrigerator 1 and the position of the mark 207 on the position regulating member 205 coincide in the left - right direction of the refrigerator 1.

[0109] By the third position adjustment, the camera unit 2 can perform various detections as in the first position adjustment.

[0110] Next, the operation of the food management system 1000 after the camera unit 2 is installed in the refrigerator 1 will be described. [1-2-2. Operations related to recording] First, the operations related to recording will be described. FIG. 13 is a flowchart FA showing the operation of the camera unit 2.

[0111] In the operation shown in FIG. 13, the sensor unit 22 outputs the detection values of the respective sensors to the camera unit control unit 20. And in the operation shown in FIG. 13, the detection value processing unit 222 executes processing based on the various detection values received from the sensor unit 22.

[0112] Based on the detection value of the human sensor 212, the detection value processing unit 222 determines whether a person is present around the refrigerator 1 (step SA1).

[0113] When the detection value processing unit 222 determines that a person is present around the refrigerator 1 (step SA1: YES), it outputs presence / absence information indicating the presence of a person to the shooting control unit 223 (step SA2).

[0114] When the shooting control unit 223 receives the presence / absence information indicating the presence of a person from the detection value processing unit 222, it starts shooting with the refrigerator compartment camera 203 and the drawer camera 204 (step SA3).

[0115] Next, based on the detection values of the various sensors output from the sensor unit 22, the detection value processing unit 222 determines whether any of the left door 11A, right door 11B, drawers 14A, 15A is in an open state (step SA4).

[0116] When the detection value processing unit 222 determines that none of the left door 11A, right door 11B, drawers 14A, 15A is in an open state (step SA4: NO), it performs the processing of step SA16.

[0117] When the detection value processing unit 222 determines that any one of the left door 11A, the right door 11B, the drawers 14A, 15A is in the open state (step SA4: YES), it determines whether the door 11C is in the open state or any one of the drawers 14A, 15A is in the open state (step SA5).

[0118] When the detection value processing unit 222 determines that the door 11C is in the open state (step SA5: door), it outputs door open state information to the recording control unit 224 (step SA6).

[0119] Next, when the recording control unit 224 receives the door open state information from the detection value processing unit 222, it starts recording the shooting result of the refrigerator compartment camera 203 (step SA7).

[0120] Next, the detection value processing unit 222 determines whether the door 11C is in the closed state (step SA8).

[0121] When the detection value processing unit 222 determines that the door 11C is in the closed state (step SA8: YES), it outputs door closed state information to the recording control unit 224 (step SA9).

[0122] Next, when the recording control unit 224 receives the door closed state information from the detection value processing unit 222, it ends the recording of the shooting result of the refrigerator compartment camera 203 (step SA10). As a result, one recording file 231 of the refrigerator compartment camera 203 is generated in the camera unit storage unit 230.

[0123] Returning to the description of step SA5, when the detection value processing unit 222 determines that any one of the drawers 14A, 15A is in the open state (step SA5: drawer), it outputs drawer open state information to the recording control unit 224 (step SA11).

[0124] Next, when the recording control unit 224 receives the drawer open state information from the detection value processing unit 222, it starts recording the shooting result of the drawer camera 204 (step SA12).

[0125] Next, the detection value processing unit 222 determines whether either of the open drawers 14A and 15A has become closed (step SA13).

[0126] When the detection value processing unit 222 determines that either of the open drawers 14A and 15A has become closed (step SA13: YES), it outputs drawer closed state information to the recording control unit 224 (step SA14).

[0127] Next, when the recording control unit 224 receives the drawer closed state information from the detection value processing unit 222, it ends the recording of the shooting result of the drawer camera 204 (step SA15). As a result, one recording file 231 of the drawer camera 204 is generated in the camera unit storage unit 230.

[0128] The detection value processing unit 222 determines whether there is a person around the refrigerator 1 (step SA16).

[0129] When the detection value processing unit 222 determines that there is a person around the refrigerator 1 (step SA16: YES), it performs the processing from step SA4 and later again.

[0130] On the other hand, when the detection value processing unit 222 determines that there is no person around the refrigerator 1 (step SA16: NO), it outputs presence / absence information indicating the absence of a person to the shooting control unit 223 (step SA17).

[0131] When the shooting control unit 223 receives the presence / absence information indicating the absence of a person from the detection value processing unit 222, it ends the shooting of the refrigerator compartment camera 203 and the drawer camera 204 (step SA18).

[0132] [1-2-3. Operations related to food detection] Next, the operations related to food detection will be described. FIG. 14 is a flowchart showing the operations of the food management system 1000. In FIG. 14, flowchart FB shows the operation of camera unit 2, and flowchart FC shows the operation of food management server 4.

[0133] The food detection unit 225 of the camera unit 2 determines whether a recording file 231 is stored in the camera unit storage unit 230 (step SB1).

[0134] When the food detection unit 225 determines that the recording file 231 is stored in the camera unit storage unit 230 (step SB1: YES), it determines whether the stored recording file 231 is the recording file 231 of the refrigerator camera 203 or the recording file 231 of the extraction camera 204 (step SB2). The determination in step SB2 is made based on the information output by the detection value processing unit 222.

[0135] When the food detection unit 225 determines that it is the recording file 231 of the refrigerator camera 203 (step SB2: refrigerator camera), it detects food based on the stored recording file 231 (step SB3).

[0136] In step SB3, for example, the food detection unit 225 detects food from the recording file 231 of the refrigerator camera 203 based on feature amounts such as shape and color. The data necessary for food detection is stored in a storage area accessible by the food detection unit 225.

[0137] The food detection unit 225 determines whether it can detect food from the recording file 231 of the refrigerator camera 203 (step SB4).

[0138] When the food detection unit 225 determines that it cannot detect food from the recording file 231 of the refrigerator camera 203 (step SB4: NO), it transfers the process to step SB10.

[0139] On the other hand, when the food detection unit 225 can detect food from the recording file 231 of the refrigerator camera 203 (step SB4: YES), it performs the incoming / outgoing determination process (step SB5).

[0140] The incoming / outgoing determination process is a process for determining whether the food detected in step SB3 is food stored in the refrigerator 1 or food taken out of the refrigerator 1. In the incoming / outgoing determination process, the incoming / outgoing determination image HG is superimposed on the moving image shown in the recorded file 231 of the refrigerator camera 203.

[0141] FIG. 15 is a diagram for explaining the incoming / outgoing determination image HG. The incoming / outgoing determination image HG shown in FIG. 15 shows a case where it is superimposed on a moving image in which the user P stores an egg pack in the refrigerator 1.

[0142] The incoming / outgoing determination image HG is provided with a boundary line L2. The boundary line L2 has boundary lines L21, L22, L23, L24, and L25. The boundary line L21 is a line extending from the left end to the right in the figure of the incoming / outgoing determination image HG at the upper middle part in the figure of the incoming / outgoing determination image HG. The boundary line L22 is a line connecting the right end of the boundary line L21 and the left end of the boundary line L23. The boundary line L23 is a line extending from the position in the horizontal direction of the right end of the boundary line L21 to the right at approximately the center in the vertical direction of the incoming / outgoing determination image HG. The boundary line L24 is a line connecting the right end of the boundary line L23 and the left end of the boundary line L24. The boundary line L25 is a line extending from the position in the horizontal direction of the right end of the boundary line L23 to the right end of the incoming / outgoing determination image HG at the upper part of the incoming / outgoing determination image HG.

[0143] The image area of the incoming / outgoing determination image HG is divided by the boundary line L2 into a concave area HG1 that is a concave area and a convex area HG2 that is a convex area.

[0144] The food detection unit 225 discriminates whether the food detected in step SB3 has advanced from the convex region HG2 to the concave region HG1 or from the concave region HG1 to the convex region HG1 by performing image analysis on the moving image overlaid with the incoming / outgoing determination image HG. When the food detection unit 225 determines that the food detected in step SB3 has advanced from the convex region HG2 to the concave region HG1, it determines that the food detected in step SB3 has been stored in the refrigerator 1. On the other hand, when the food detection unit 225 determines that the food detected in step SB3 has advanced from the concave region HG1 to the convex region HG2, it determines that the food detected in step SB3 has been shipped out from the refrigerator 1.

[0145] Returning to the explanation of the flowchart FB in FIG. 14, the food detection unit 225 performs divided area number identification processing (step SB6).

[0146] The divided area number identification processing is a process of identifying the divided area BA corresponding to the storage area where the food detected in step SB3 has been received or shipped out, and identifying the divided area number 4118 assigned to the identified divided area BA.

[0147] For example, the food detection unit 225 identifies the storage area of the refrigerator compartment 11 where the food detected in step SB3 has been received or shipped out by image analysis. Also, for example, marks for identifying the storage area are attached to the tips of each of the shelves 112 and inside each of the door pockets 111, and the food detection unit 225 identifies the storage area based on the positional relationship between this mark in the moving image and the detected food. Next, the food detection unit 225 determines which divided area BA of the refrigerator compartment interior image RG the identified storage area corresponds to, and identifies the divided area number 4118 assigned to the determined divided area BA. Note that the correspondence relationship between the storage area, the divided area BA, and the divided area number 4118 is stored as data in the camera unit storage unit 230.

[0148] When the food detection unit 225 performs the divided area number identification process, it extracts the image of each food detected in step SB3 from the recording file 231 of the refrigerator camera 203 (step SB7). Hereinafter, the image of the food extracted from the shooting result of the camera 213 is referred to as a "shot food image" and is given the symbol "FG". The shot food image FG corresponds to an example of the food image of the present disclosure.

[0149] Next, the food detection unit 225 generates first update information (step SB8).

[0150] The first update information generated in step SB8 includes the camera unit ID 4112 of the camera unit storage unit 230. Also, the first update information generated in step SB8 includes date and time information indicating the date and time when the recording file 231 of the refrigerator camera 203 was generated. Also, the first update information generated in step SB8 includes, for each food detected in step SB3, food image data 4116 indicating the shot food image FG cut out in step SB7, incoming / outgoing determination result information indicating the determination result of the incoming / outgoing determination process, and the divided area number 4118 specified in the divided area number identification process.

[0151] Next, the camera unit communication control unit 221 transmits the first update information generated by the food detection unit 225 to the food management server 4 (step SB9).

[0152] The food detection unit 225 deletes the recording file 231 from the camera unit storage unit 230 (step SB10).

[0153] Returning to the description of step SB2, when the food detection unit 225 determines that it is the recording file 231 of the drawer camera 204 (step SB2: drawer camera), it determines whether at least one of the upper case JCA and the lower case GCA shown in the moving image of the recording file 231 of the drawer camera 204 is pulled out by a predetermined threshold or more (step SB11).

[0154] For example, the food detection unit 225 distinguishes the case CA in the moving image based on feature amounts such as shape and color. Then, the food detection unit 225 determines whether the upper case JCA and the lower case GCA are pulled out from the refrigerator 1 by a predetermined threshold value or more by analyzing the images of the recording file 231. Also, for example, the upper case JCA and the lower case GCA are each marked with a mark for identifying the case CA at a predetermined position, and the food detection unit 225 reads this mark from the moving image to distinguish the case CA in the moving image. Then, the food detection unit 225 calculates the moving distance of this mark in the moving image to determine whether it is pulled out from the refrigerator 1 by a predetermined threshold value or more.

[0155] When the food detection unit 225 determines that neither the upper case JCA nor the lower case GCA has been pulled out by a predetermined threshold value or more (step SB11: NO), the process of step SB18 is performed.

[0156] On the other hand, when the food detection unit 225 determines that at least one of the upper case JCA and the lower case GCA has been pulled out by a predetermined threshold value or more (step SB11: YES), it cuts out the case image CG from the recording file 231 (step SB12).

[0157] In step SB12, when only the upper case JCA has been pulled out by a predetermined threshold value or more, the food detection unit 225 cuts out the upper case image JCG from the recording file 231. In step SB12, when only the lower case GCA has been pulled out by a predetermined threshold value or more, the food detection unit 225 cuts out the lower case image GCG from the recording file 231. In step SB12, when both the upper case JCA and the lower case GCA have been pulled out by a predetermined threshold value or more, the food detection unit 225 cuts out the upper case image JCG and the lower case image GCG from the recording file 231.

[0158] Next, the food detection unit 225 detects food from the cut-out case image CG (step SB13).

[0159] Next, the food detection unit 225 determines whether it has detected food from the case image CG (step SB14).

[0160] When the food detection unit 225 determines that it cannot detect food (step SB14: NO), it performs the process of step SB16.

[0161] On the other hand, when the food detection unit 225 determines that it has detected food (step SB14: YES), it cuts out a photographed food image FG for each detected food from the case image CG (step SB15).

[0162] Next, the food detection unit 225 generates second update information (step SB16).

[0163] When only the upper case image JCG is cut out, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage unit 230, the upper case image data 4123 indicating the cut-out upper case image JCG, and the food image data 4116 indicating the photographed food image FG cut out from the upper case image JCG. Note that when a negative determination is made in step SB14, the food image data 4116 is not included in this second update information.

[0164] Also, when only the lower case image GCG is cut out, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage unit 230, the lower case image data 4125 indicating the cut-out lower case image GCG, and the food image data 4116 indicating the photographed food image FG cut out from the lower case image GCG. Note that when a negative determination is made in step SB14, the food image data 4116 is not included in this second update information.

[0165] Also, when the upper case image JCG and the lower case image GCG are cut out, the second update information generated in step SB16 includes the camera unit ID 4112 in the camera unit storage unit 230, the upper case image data group, and the lower case image data group. The upper case image data group includes the upper case image data 4123 indicating the cut-out upper case image JCG and the food image data 4116 indicating the photographed food image FG cut out from the upper case image JCG. The lower case image data group includes the lower case image data 4125 indicating the cut-out lower case image GCG and the food image data 4116 indicating the photographed food image FG cut out from the lower case image GCG. Note that when a negative determination is made in step SB14, the food image data 4116 is not included in this second update information.

[0166] Next, the camera unit communication control unit 221 transmits the second update information generated by the food detection unit 225 to the food management server 4 (step SB17).

[0167] Next, the food detection unit 225 deletes the recording file 231 from the camera unit storage unit 230 (step SB18).

[0168] As shown in the flowchart FC, the server communication control unit 401 of the food management server 4 determines whether it has received either the first update information or the second update information (step SC1).

[0169] When it is determined that the server communication control unit 401 has received the update information (step SC1: YES), the information processing unit 402 updates the database based on the update information received by the server communication control unit 401 (step SC2).

[0170] Step SC2 will be described in detail. When the update information received by the server communication control unit 401 is the first update information, the information processing unit 402 identifies the refrigerator compartment food management record RR having the camera unit ID 4112 included in the first update information. Then, the information processing unit 402 updates the presence / absence of incoming / outgoing information 4115 of the identified refrigerator compartment food management record RR to information indicating "present".

[0171] Next, when the incoming / outgoing determination result information included in the first update information indicates incoming, the information processing unit 402 generates a sub-record, associates the blank deletion information 4131 and the deletion date / time information 4132 with the generated sub-record, and stores them in the identified refrigerator compartment food management record RR. The sub-record generated here includes the food image data 4116 included in the first update information, the storage date / time information 4117 indicating the same date / time as the date / time information included in the first update information, the divided area number 4118 included in the first update information, the presence / absence information 4119 of blank memo input, and the blank memo information 4130.

[0172] Also, when the incoming / outgoing determination result information included in the first update information indicates outgoing, the information processing unit 402 identifies, from the refrigerator compartment food management record RR, a sub-record including the photographed food image FG indicated by the food image data 4116 included in the first update information and the food image data 4116 indicating a photographed food image FG with a high degree of coincidence. This identification is performed for each food image data 4116 included in the first update information. Then, the information processing unit 402 updates the deletion information 4131 associated with the identified sub-record to information indicating that it is a deletion target. Also, the information processing unit 402 updates the deletion date / time information 4132 associated with the identified sub-record to information indicating the current date / time as the deletion date / time.

[0173] Furthermore, step SC2 will be described in detail. When the update information received by the server communication control unit 401 is the second update information, the information processing unit 402 identifies the taken-out food management record HR having the camera unit ID 4112 included in the second update information.

[0174] When it is the second update information when only the upper case image JCG is cut out, the information processing unit 402 updates the upper case image data 4123 and the food image data 4116 included in the upper case information 4122 of the identified withdrawal food management record HR to the data included in the second update information. When it is the second update information when only the lower case image GCG is cut out, the information processing unit 402 updates the lower case image data 4125 and the food image data 4116 included in the lower case information 4124 of the identified withdrawal food management record HR to the data included in the second update information. When it is the second update information when the upper case image JCG and the lower case image GCG are cut out, the information processing unit 402 appropriately updates the upper case image data 4123 and the food image data 4116 included in the upper case information 4122 of the identified withdrawal food management record HR to the data included in the upper case image data group. Further, the information processing unit 402 updates the lower case image data 4125 and the food image data 4116 included in the lower case information 4124 of the identified withdrawal food management record HR to the data included in the lower case image data group.

[0175] In addition, the information processing unit 402 updates the update date and time information 4121 of the identified withdrawal food management record HR to the information indicating the date and time when the withdrawal food management record HR was updated.

[0176] [1-2-4. Operations related to the app UI] Next, the operations related to the app UI 320 will be described. In the present embodiment, on the app UI 320, a screen related to the refrigerator 11 and a screen related to the drawer-type storage compartment are displayed.

[0177] [1-2-4-1. Screen related to the refrigerator]

[0178] The screen related to the refrigerator 11 includes a refrigerator home screen GM2, a refrigerator list screen GM3, and a food information addition list screen GM4.

[0179] FIG. 16 is a diagram showing an example of the refrigerator home screen GM2.

[0180] The refrigerator home screen GM2 has a food image display area HA2. The food image display area HA2 is an area for displaying photographed food images FG. The food image display area HA2 displays the refrigerator interior image RG and superimposes and displays the photographed food image FG on the refrigerator interior image RG. Each of the photographed food images FG is displayed superimposed on a divided area BA indicated by a divided area number 4118 associated with food image data 4116 in the refrigerator interior image RG.

[0181] The food image display area HA2 in FIG. 16 displays the refrigerator interior image RG shown in FIG. 9. For example, in the refrigerator food management record RR, it is assumed that a divided area number 4118 of "4" is associated with the food image data 4116 indicating the photographed food image FG1 in FIG. 16. In this case, the photographed food image FG1 is displayed superimposed on the divided area BA to which the divided area number 4118 of "4" is assigned.

[0182] The default food image display area HA2 displays the refrigerator interior image RG with the left door pocket image RG1 and the right door pocket image RG3 cut off. In the food image display area HA2, a scroll operation in the left-right direction of the figure is accepted from the user P. The food image display area HA2 changes the display range of the refrigerator interior image RG according to the scroll amount and scroll direction of the accepted scroll operation.

[0183] For example, assume that the scroll direction is to the right and the scroll amount is such that the left end of the left door pocket image RG1 is displayed. In this case, the food image display area HA2 displays the refrigerator interior image RG with the right door pocket image RG3 completely cut off. Note that the photographed food image FG and the memo input icon MIC move following the change in the display range of the refrigerator interior image RG, in other words, accompanying the movement of the superimposed divided area BA. The memo input icon MIC will be described later.

[0184] When the food image display area HA2 displays a plurality of captured food images FG in one divided area BA, they are arranged and displayed in the left - right direction of the figure. The food image display area HA2 can simultaneously arrange and display a maximum of 5 captured food images FG in one divided area BA. When there are 6 or more captured food images FG to be displayed in one divided area BA, the food image display area HA2 accepts a scroll operation from the user P. When accepting a scroll operation, the one divided area BA displays the captured food images FG from the 6th onwards. When there are 6 or more captured food images FG to be displayed in one divided area BA and a scroll operation in the left - right direction is accepted, while maintaining the maximum number of images that can be arranged and displayed, the food image display area HA2 displays the captured food images FG from the 6th onwards according to the scroll amount. At this time, the food image display area HA2 makes the captured food images FG from the 1st to the 5th that were being displayed non - visible, or displays the captured food images FG from the 1st onwards after the captured food images FG from the 6th onwards, according to the scroll amount and the number of captured food images FG to be displayed. Note that the memo input icon MIC moves following the movement of the associated captured food image FG. That is, when the associated captured food image FG becomes non - visible due to a scroll operation, the memo input icon MIC also becomes non - visible.

[0185] Each of the captured food images FG displayed in the food image display area HA2 accepts the input of a memo. The memo is information that the user P wants to leave for the food in the refrigerator 1.

[0186] Figure 17 is a diagram showing the memo input procedure.

[0187] The user P performs a predetermined touch operation such as a long - press operation on the captured food image FG for which the user wants to input a memo on the refrigerator home screen GM2. When the application execution unit 301 accepts a predetermined touch operation on one captured food image FG, it displays a memo editing image MHG in association with the captured food image FG on which the predetermined touch operation was performed.

[0188] The memo editing image MHG has a memo input area MNA, a memo save button B1, a memo deletion button B2, and a food deletion button B3.

[0189] The memo input area MNA is an area for inputting a memo. When the memo input area MNA is touched, the application execution unit 301 displays a software keyboard on the touch panel 32. Then, the application execution unit 301 inputs a memo corresponding to an operation on the software keyboard into the memo input area MNA.

[0190] The memo save button B1 is a software button for saving the memo input in the memo input area MNA. The memo save button B1 cannot be touched when no memo is input in the memo input area MNA. When the memo save button B1 is touched, the application execution unit 301 displays, on the screen related to the refrigerator 11, the photographed food image FG with a memo input in a display mode different from the display mode of the photographed food image FG without a memo input. In the present embodiment, the application execution unit 301 displays the frame of the photographed food image FG with a memo input in a different color, and associates and displays a memo input icon MIC, which is an icon indicating that a memo has been input, with the photographed food image FG with a memo input. The memo input icon MIC is touch-operable. When the memo input icon MIC is touched, the input memo is displayed in the application UI 320. The memo input icon MIC corresponds to an example of the memo input image of the present disclosure.

[0191] The memo deletion button B2 is a button for deleting the input memo. When the memo deletion button B2 is touched, the memo input in the photographed food image FG is deleted.

[0192] The food deletion button B3 is a button for deleting the photographed food image FG from the food image display area HA2.

[0193] Returning to the description of the refrigerator home screen GM2 in FIG. 16, the food image display area HA2 has a list display button B4. The list display button B4 is a touch-operable software button.

[0194] When the list display button B4 is touch-operated, in the app UI320, the screen transitions to the refrigerator list screen GM3.

[0195] The refrigerator list screen GM3 includes a storage list screen GM31, a new addition list screen GM32, and a deletion target list screen GM33.

[0196] The storage list screen GM31 will be described. FIG. 18 is a diagram showing the storage list screen GM31.

[0197] The storage list screen GM31 has a stored food information display area HA3. The stored food information display area HA3 displays a list of first food information J21 for each food stored in the refrigerator 11. The first food information J21 includes a photographed food image FG, a storage date and time information 4117, and divided area information J3. The divided area information J3 is a refrigerator interior image RG in which the divided area BA is shown in color.

[0198] In the photographed food image FG of the first food information J21, when a memo is input, the frame is displayed in a different color and a memo input icon MIC is displayed in association. In the stored food information display area HA3, a scroll operation is possible in the vertical direction of the figure. The user P can display the first food information J21 that cannot be fully displayed in the stored food information display area HA3 by performing a scroll operation in the stored food information display area HA3.

[0199] The storage list screen GM31 has a first switching button B5 and a second switching button B6.

[0200] The first switching button B5 is a software button for switching the order of the first food information J21 displayed in the stored food information display area HA3 to the next order. That is, when the first switching button B5 is touch-operated, the stored food information display area HA3 displays the first food information J21 having the photographed food image FG with a memo input above the first food information J21 having the photographed food image FG without a memo input. Note that displaying above means displaying such that the amount of scrolling during display is reduced. Also, when there are a plurality of first food information J21 having the photographed food image FG with a memo input, the stored food information display area HA3 displays the food information J2 with an older storage date and time indicated by the storage date and time information 4117 above. Further, when there are a plurality of first food information J21 having the photographed food image FG without a memo input, the stored food information display area HA3 displays the first food information J21 with an older storage date and time indicated by the storage date and time information 4117 above. Note that the left storage list screen GM31 in the figure of FIG. 18 shows the storage list screen GM31 when the first switching button B5 is touch-operated.

[0201] The second switching button B6 is a software button for switching the order of the first food information J21 displayed in the stored food information display area HA3 to the following order. That is, when the second switching button B6 is touch-operated, the stored food information display area HA3 displays the first food information J21 arranged for each divided area BA. The order for each divided area BA may be an order predetermined by the user P. In one divided area BA, the stored food information display area HA3 displays the first food information J21 in the same order as when the first switching button B5 is touch-operated. Note that the right storage list screen GM31 in the figure of FIG. 18 shows the storage list screen GM31 when the second switching button B6 is touch-operated.

[0202] The storage list screen GM31 has a storage list display button B7, a new addition list display button B8, and a deletion target list display button B9.

[0203] The accommodation list display button B7 is a software button for switching the screen displayed by the app UI 320 to the accommodation list screen GM31. The new addition list display button B8 is a software button for switching the screen displayed by the app UI 320 to the new addition list screen GM32. The deletion target list display button B9 is a software button for switching the screen displayed by the app UI 320 to the deletion target list screen GM33.

[0204] Figure 19 is a diagram showing an example of the new addition list screen GM32. The new addition list screen GM32 has a new addition food information display area HA4. The new addition food information display area HA4 displays the first food information J21 regarding the food newly stored in the refrigerator 11 in a list. Note that the food newly stored in the refrigerator 11 means the food stored within a predetermined period (e.g., 3 days) from the current date and time. The new addition food information display area HA4 displays the first food information J21 of the food corresponding to the newly stored food, with the storage date and time indicated by the storage date and time information 4117 in descending order of the newness of the storage date and time at the top.

[0205] Each of the first food information J21 displayed in the new addition food information display area HA4 accepts a predetermined touch operation on the photographed food image FG. On the new addition list screen GM32, the first edited image HNG1 is displayed in association with the photographed food image FG that has been subjected to the predetermined touch operation.

[0206] The first edited image HNG1 has a food deletion button B10, an area correction button B11, and a return button B12. The food deletion button B10 is a software button for designating the food indicated by the photographed food image FG associated with the first edited image HNG1 as a deletion target from the refrigerator home screen GM2 when touched. The area correction button B11 is a software button for correcting the divided area BA indicated by the divided area information J3 when touched. The return button B12 is a software button for stopping the display of the first edited image HNG1.

[0207] The new additional food information display area HA4 has a food information addition button B13 at the end of the list of the first food information J21. The food information addition button B13 is a software button for manually adding new first food information J21. Referring to FIG. 20, the addition procedure of the first food information J21 will be described.

[0208] FIG. 20 is a diagram for explaining the addition procedure of the first food information J21.

[0209] When the food information addition button B13 is touch-operated, in the app UI320, a food information addition list screen GM4 is displayed. The food information addition list screen GM4 has a plurality of addition buttons B14 which are software buttons. Also, the food information addition list screen GM4 has a blank image BG and a food name input area SNA for inputting a food name. When the blank image BG is touch-operated, within the blank image BG, it becomes an image showing the food name input in the food name input area SNA.

[0210] When the addition button B14 on the food information addition list screen GM4 is touch-operated, in the app UI320, an image of the inside of the refrigerator RG is displayed. Here, the displayed image of the inside of the refrigerator RG is an image in which each of the divided areas BA is assigned a divided area number 4118, and each of the divided areas BA can be selected by a touch operation. When a divided area BA is selected in this image of the inside of the refrigerator RG, in the app UI320, the next first food information J21 is added.

[0211] That is, the first food information J21 to be added is an image of the food corresponding to the touch-operated addition button B14, storage date and time information 4117 indicating the date and time when the first food information J21 was added, and division area information J3 indicating the selected division area BA. Note that the image of the food indicated by the first food information J21 to be added may be an icon. In this embodiment, it is assumed that the image superimposed on the refrigerator interior image RG of the refrigerator home screen GM2 is the shadow food image FG. However, when the first food information J21 is added, an icon is displayed on the refrigerator interior image RG together with the photographed food image FG.

[0212] Note that in the procedure for adding the first food information J21, there may be a procedure in which the user P inputs the storage date and time. In this case, the first food information J21 to be added includes the storage date and time information 4117 indicating the storage date and time input by the user P.

[0213] Returning to the description of FIG. 19, the new addition list screen GM32 has a storage list display button B7, a new addition list display button B8, and a deletion target list display button B9.

[0214] FIG. 21 is a diagram showing an example of the deletion target list screen GM33.

[0215] The deletion target list screen GM33 has a deletion target food information display area HA5. The deletion target food information display area HA5 displays a list of the second food information J22 for each food to be deleted. Note that the food to be deleted indicates the food to be deleted from the refrigerator home screen GM2 and the food determined to have been shipped out from the refrigerator 11. The second food information J22 includes a photographed food image FG, deletion date and time information 4132, and division area information J3. The deletion target list screen GM33 displays the second food information J22 with the latest deletion date and time at the top.

[0216] Each of the second food information J22 displayed in the food information display area HA5 to be deleted accepts a predetermined touch operation on the photographed food image FG. On the deletion target list screen GM33, a second edited image HNG2 is displayed in association with the photographed food image FG that has received the predetermined touch operation.

[0217] The second edited image HNG2 has a delete cancellation button B15 and a return button B16. The delete cancellation button B15 is a software button for setting the photographed food image FG associated with the second edited image HNG2 as the image displayed on the refrigerator home screen GM2. The return button B16 is a software button for stopping the display of the second edited image HNG2.

[0218] [1-2-4-2. Operation of the food management system] Next, the operation of the food management system 1000 when displaying a screen related to the refrigerator 11 will be described.

[0219] FIG. 22 is a flowchart showing the operation of the food management system 1000. In FIG. 22, the flowchart FD shows the operation of the terminal device 3, and the flowchart FE shows the operation of the food management server 4.

[0220] The app execution unit 301 of the terminal device 3 determines whether to request the refrigerator food management record RR from the food management server 4 (step SD1).

[0221] For example, when the app execution unit 301 receives a touch operation instructing the display of the refrigerator home screen GM2 in the app UI320, it makes an affirmative determination in step SD1. Also, for example, when a touch operation for instructing an update of the display is performed on the refrigerator home screen GM2, the app execution unit 301 makes an affirmative determination in step SD1.

[0222] When the application execution unit 301 determines that it needs to request the refrigerator management record RR (step SD1: YES), it sends the first record request information for requesting the refrigerator management record RR to the food management server 4 (step SD2). The first record request information sent in step SD2 includes the account ID 4111 stored in the terminal storage unit 310.

[0223] As shown in the flowchart FF, the server communication control unit 401 of the food management server 4 receives the first record request information through the server communication unit 41 (step SE1).

[0224] Next, the information processing unit 402 identifies the refrigerator food management record RR including the account ID 4111 included in the first record request information received by the server communication control unit 401 from the refrigerator food management database 411 (step SE2).

[0225] Then, the server communication control unit 401 sends the refrigerator food management record RR identified by the information processing unit 402 to the terminal device 3 (step SE3). Note that the information processing unit 402 updates the in / out stock presence / absence information 4115 of the sent refrigerator food management record RR to information indicating "none".

[0226] Referring to the flowchart FD, when the application execution unit 301 of the terminal device 3 receives the refrigerator food management record RR from the food management server 4 (step SD3), it generates a screen related to the refrigerator 11 based on this refrigerator food management record RR (step SD4).

[0227] Step SD4 will be described in detail. The application execution unit 301 generates a refrigerator home screen GM2 for displaying the refrigerator interior image RG shown by the refrigerator interior image data 4114 included in the refrigerator food management record RR in the food image display area HA2. In addition, for each sub-record, the app execution unit 301 generates a refrigerator home screen GM2 in which the photographed food image FG indicated by the food management image data 4116 is superimposed on the divided area BA indicated by the divided area number 4118. Also, when the memo input presence / absence information 4119 indicates that there is a memo, for each sub-record, the app execution unit 301 generates a refrigerator home screen GM2 in which the memo input icon MIC is displayed in association with the photographed food image FG.

[0228] Furthermore, step SD4 will be described in detail. The app execution unit 301 generates first food information J21 for each food image data 4116. This first food information J21 includes the photographed food image FG indicated by the food image data 4116, the storage date and time information 4117 associated with this food image data 4116, and the divided area information J3 in which the divided area BA indicated by the divided area number 4118 is shown in color. Then, the app execution unit 301 generates a storage list screen GM31 that displays a list of the first food information J21 related to the sub-records associated with the deletion information 4131 that does not indicate a deletion target. Also, the app execution unit 301 generates a new addition list screen GM32 that displays a list of the first food information J21 among the first food information J21 displayed on the storage list screen GM31, where the storage date and time indicated by the storage date and time information 4117 is within a predetermined period from the current time.

[0229] Furthermore, step SD4 will be described in detail. The app execution unit 301 generates second food information J22 for each food image data 4116. The second food information J22 includes the photographed food image FG indicated by the food image data 4116, the deletion date and time information 4132 associated with this food image data 4116, and the divided area information J3 in which the divided area BA indicated by the divided area number 4118 is shown in color. Then, the app execution unit 301 generates a deletion target list screen GM33 that displays a list of the generated second food information J22.

[0230] Returning to the description of flowchart FD, the app execution unit 301 displays a screen related to the refrigerator compartment 11 in response to a touch operation (step SD5). When the incoming / outgoing presence / absence information 4115 of the refrigerator compartment food management record RR received in step SD3 indicates "present", when the app execution unit 301 first displays the refrigerator compartment home screen GM2 in step SD5, the app execution unit 301 pops up a message on the refrigerator compartment home screen GM2 indicating that there has been a new incoming / outgoing.

[0231] Next, the operation of the food management system 1000 when an operation is performed on the screen related to the refrigerator compartment 11 will be described.

[0232] When the memo save button B1 is touched, the app execution unit 301 transmits information for updating a sub-record related to the photographed food image FG of the operation target to the food management server 4. This information includes memo input presence / absence information 4119 indicating that there is a memo, and memo information 4130 indicating the input memo. The food management server 4 appropriately updates the sub-record based on the received information.

[0233] When the memo deletion button B2 is touched, the app execution unit 301 transmits information for updating a sub-record related to the photographed food image FG of the operation target to the food management server 4. This information includes memo input presence / absence information 4119 indicating that there is no memo input. The food management server 4 appropriately updates the sub-record based on the received information.

[0234] When the food deletion buttons B3 and B10 are touched, the app execution unit 301 transmits information for updating a sub-record related to the photographed food image FG of the operation target to the food management server 4. This information includes deletion information 4131 indicating the deletion target. The food management server 4 appropriately updates the sub-record based on the received information.

[0235] When the area correction button B11 is touched and the divided area BA is corrected, the application execution unit 301 transmits information for updating the sub-record related to the photographed food image FG to be operated on to the food management server 4. This information includes the divided area number 4118 of the corrected divided area BA. The food management server 4 appropriately updates the sub-record based on the received information.

[0236] When the first food information J21 is added in the additional procedure shown in FIG. 20, the application execution unit 301 transmits information for adding the sub-record related to the added first food information J21 to the food management server 4. The food management server 4 appropriately updates the sub-record based on the received information.

[0237] When the delete cancellation button B15 is touched, the application execution unit 301 transmits information for updating the sub-record related to the photographed food image FG to be operated on to the food management server 4. This information includes the deletion information 4131 indicating a blank. The food management server 4 appropriately updates the sub-record based on the received information.

[0238] [1-2-4-3. Screen related to the drawer-type storage room] Next, the screen related to the drawer-type storage room will be described. In the description using FIGS. 23 to 25, the vegetable room 15 is exemplified as the drawer-type storage room. In this description, the database shown in FIG. 10 is a database that stores various information related to the foods in the vegetable room 15.

[0239] The screen related to the vegetable room 15 includes a vegetable room home screen GM5 and a food image list screen GM6.

[0240] FIG. 23 is a diagram showing an example of the vegetable room home screen GM5.

[0241] The vegetable room home screen GM5 has a case image display area HA6. The case image display area HA6 is an area for displaying the case image CG. The case image display area HA6 displays the upper case image JCG superimposed on the lower case image GCG. In the default case image display area HA6, the upper case image JCG is superimposed on the entire image area of the lower case image GCG, and only the upper case image JCG is displayed.

[0242] The case image display area HA6 accepts a scroll operation in the vertical direction of the figure. The case image display area HA6 changes the display range of the upper case image JCG according to the scroll amount and scroll direction of the accepted scroll operation. For example, when the scroll direction of the accepted scroll operation is upward, the case image display area HA6 displays the upper case image JCG with a part cut off. In the case image display area HA6, when the upper case image JCG moves upward, the lower case image GCG that was superimposed on the upper case image JCG will be displayed.

[0243] Each of the case images CG displayed by the case image display area HA6 accepts the input of a memo. The procedure for memo input is the same as that in the case of the refrigerator compartment 11, except that the image operated by the user P is the photographed food image FG or the case image CG. As shown in Fig. 23, the memo input icon MIC associated with the upper case image JCG moves following the movement of the upper case image JCG.

[0244] The vegetable compartment home screen GM5 has a list display button B17. The list display button B17 is a touch-operable software button.

[0245] When the list display button B17 is touch-operated, in the app UI320, the food image list screen GM6 is displayed.

[0246] Incidentally, the vegetable compartment home screen GM5 may have a switching button for switching the case image CG displayed in the case image display area HA6 to the upper case image JCG or the lower case image GCG. In the case of this configuration, the app execution unit 301 switches the case image CG to be displayed to the upper case image JCG or the lower case image GCG according to the operation on this switching button.

[0247] FIG. 24 is a diagram showing an example of the food image list screen GM6.

[0248] The food image list screen GM6 has update date and time information 4121. The food image list screen GM6 in FIG. 24 has update date and time information 4121 indicating an update date and time of "12 / 1 10:05".

[0249] The food image list screen GM6 displays a list of photographed food images FG for each case. The photographed food image FG displayed in association with the item of the upper case JCA is an image cut out from the upper case image JCG. The photographed food image FG displayed in association with the item of the lower case GCA is an image cut out from the lower case image GCG.

[0250] [1-2-4-5. Operation of the Food Management System] Next, the operation of the food management system 1000 when displaying a screen related to the vegetable compartment 15 will be described.

[0251] FIG. 25 is a flowchart showing the operation of the food management system 1000. In FIG. 11, the flowchart FF shows the operation of the terminal device 3, and the flowchart FJ shows the operation of the food management server 4.

[0252] The app execution unit 301 of the terminal device 3 determines whether or not to request the withdrawal management record HR from the food management server 4 (step SF1).

[0253] For example, when the app execution unit 301 receives a touch operation that instructs the display of the vegetable room home screen GM5 in the app UI 320, it makes an affirmative determination in step SF1. Also, for example, when a touch operation for instructing an update of the display is performed on the vegetable room home screen GM5, the app execution unit 301 makes an affirmative determination in step SF1.

[0254] When the app execution unit 301 determines that it is necessary to request the withdrawal management record RR (step SF1: YES), it transmits the second record request information for requesting the withdrawal management record HR to the food management server 4 (step SF2). The second record request information transmitted in step SF2 includes the account ID 4111 stored in the terminal storage unit 310.

[0255] Referring to the flowchart FJ, the server communication control unit 401 of the food management server 4 receives the second record request information (step SJ1).

[0256] Next, the information processing unit 402 identifies the withdrawal management record HR including the account ID 4111 included in the second record request information received by the server communication control unit 401 from the withdrawal food management database 412 (step SJ2).

[0257] Then, the server communication control unit 401 transmits the withdrawal management record HR identified by the information processing unit 402 to the terminal device 3 (step SJ3).

[0258] Referring to the flowchart FJ, when the app execution unit 301 of the terminal device 3 receives the withdrawal food management record HR from the food management server 4 (step SF3), it generates a screen related to the vegetable room 15 (step SF4).

[0259] Step SF4 will be described in detail. The app execution unit 301 generates a vegetable room home screen GM5 that has a case image display area HA6 in which an upper case image JCG indicated by upper case image data 4123 included in the same record is superimposed on a lower case image GCG indicated by lower case image data 4125 included in the withdrawn food management record HR. Also, when the memo input presence / absence information 4119 indicates that there is a memo, the app execution unit 301 generates a vegetable room home screen GM5 in which a memo input icon MIC is displayed in association with the corresponding case image CG.

[0260] Furthermore, step SF4 will be described in detail. The app execution unit 301 generates a food image list screen GM6 in which captured food images FG indicated by food image data 4116 are arranged for each case CA.

[0261] Returning to the description of the flowchart FF, the app execution unit 301 displays a screen related to the vegetable room 15 in response to a touch operation (step SF5).

[0262] Next, the operation of the food management system 1000 when an operation is performed on the screen related to the vegetable room 15 will be described.

[0263] When the memo save button B1 is touched, the app execution unit 301 transmits information for updating the memo input presence / absence information 4119 and the memo information 4130 to the food management server 4. This information includes information indicating whether the operation target is the upper case image JCG or the lower case image GCG, the memo input presence / absence information 4119 indicating that there is a memo, and the memo information 4130 indicating the input memo. The food management server 4 appropriately updates the withdrawn food management record HR based on the received information.

[0264] When the memo deletion button B2 is touched, the app execution unit 301 transmits information for updating the memo input presence / absence information 4119 and the memo information 4130 to the food management server 4. This information includes information indicating whether the memo deletion target is the upper case image JCG or the lower case image GCG, and the memo input presence / absence information 4119 indicating that there is no memo. The food management server 4 appropriately updates the drawer food management record HR based on the received information.

[0265] Note that the vegetable compartment 15 was exemplified as the drawer-type storage compartment described with reference to FIGS. 23 to 25. However, the screen display shown in FIGS. 23 to 25 and the operation of the food management system 1000 may be similarly performed for the freezer compartment 14. In this case, the database shown in FIG. 10 indicates a database that stores various information related to the foods in the freezer compartment 14.

[0266] [1-3. Effects] As described above, the camera unit 2 includes a main body 201 installed on the upper surface 10B of the refrigerator 1, and a photographing member 202 extending forward from the main body 201 of the refrigerator 1 and having a camera 213 that photographs downward from the upper front of the refrigerator 1. The main body 201 is in contact with the front edge 10A of the refrigerator 1 and includes a position regulating member 205 that regulates the position of the photographing member 202 in the front-rear direction of the refrigerator 1.

[0267] According to this, since the position of the photographing member 202 in the front-rear direction of the refrigerator 1 is regulated by the position regulating member 205, the camera 213 is defined at a position where it can photograph downward from the upper front of the refrigerator 1. Therefore, the foods entering and leaving the refrigerator 1 can be accurately included in the photographing result of the camera 213, and the foods entering and leaving the refrigerator 1 can be accurately photographed.

[0268] The position regulating member 205 is a position regulating piece that extends downward from the bottom surface 206 of the main body 201 in the installed state.

[0269] According to this, the position of the camera unit 2 can be regulated by utilizing the gap formed between the main cabinet 10 and the door 11C. Therefore, since the camera unit 2 can be installed with respect to the existing refrigerator 1, even in the existing refrigerator 1, the food items that are put into and taken out of the refrigerator 1 can be accurately photographed.

[0270] The position regulating member 205 is formed shorter than the length up to the packing provided between the refrigerator 1 and the door 11C of the refrigerator 1.

[0271] According to this, the position of the camera unit 2 can be regulated by utilizing the gap formed between the main cabinet 10 and the door 11C without inhibiting the cooling effect of the refrigerating chamber 11. Therefore, without inhibiting the cooling effect of the refrigerating chamber 11, the food items that are put into and taken out of the refrigerator 1 can be accurately photographed.

[0272] The position regulating member 205 extends in the width direction of the main body 201.

[0273] According to this, since a wide contact area with the main cabinet 10 can be ensured, the installation posture of the camera unit 2 can be stabilized. Therefore, the food items that are put into and taken out of the refrigerator 1 can be photographed more accurately.

[0274] The position regulating member 205 has a mark 207 for adjusting the position of the camera unit 2 in the width direction of the refrigerator 1.

[0275] According to this, by using the mark 207 as a guide, the user P can easily install the camera unit 2 in the left - right direction of the refrigerator 1. Therefore, the user P can easily install the camera unit 2 so that the food items that are put into and taken out of the refrigerator 1 can be accurately photographed.

[0276] The position regulating member 205 has the mark 207 at the center in the width direction of the main body 201.

[0277] According to this, by using the mark 207 as a reference, the user P can easily install the camera unit 2 at the center in the left - right direction of the refrigerator 1. Thus, the user P can easily install the camera unit 2 so that the food entering and leaving the refrigerator 1 can be photographed more accurately.

[0278] Also, as described above, the camera unit 2 includes a camera 213 that captures the refrigerator 1 as a moving image from the front - upper side of the refrigerator 1, a human - presence sensor 212 that detects whether a person exists around the refrigerator 1, a shooting control unit 223 that causes the camera 213 to start shooting when the human - presence sensor 212 detects that a person exists around the refrigerator 1, and a recording control unit 224 that starts recording the shooting result of the camera 213 at a predetermined timing after the shooting control unit 223 causes the camera 213 to start shooting. Also, the control method of the camera unit 2 is such that when the human - presence sensor 212 that detects whether a person exists around the refrigerator 1 detects that a person exists around the refrigerator 1, it causes the camera 213 to start shooting, and at a predetermined timing after the camera 213 starts shooting, it starts recording the shooting result of the camera 213.

[0279] According to the camera unit 2 and the control method of the camera unit 2, when the human - presence sensor 212 detects that a person exists around the refrigerator 1, by causing the camera 213 to start shooting, the possibility that the food entering and leaving the refrigerator 1 is included in the shooting result of the camera 213 before the door 11C or the drawer 16A of the refrigerator 1 opens can be increased. Therefore, the food entering and leaving the refrigerator 1 can be photographed accurately.

[0280] The recording control unit 224 starts recording the shooting result of the camera 213 when the door 11C or the drawer 16A of the refrigerator 1 opens after the shooting control unit 223 causes the camera 213 to start shooting.

[0281] According to this, since recording is started at a timing when it is highly likely that the food inventory of the refrigerator 1 starts, it is possible to suppress unnecessary recording in a situation where no food is taken in or out of the refrigerator 1. Further, since the recording control unit 224 can suppress an increase in the data capacity of the recording file 231, it is possible to prevent an increase in the processing target of the food detection unit 225 and an increase in the processing time and processing load of the food detection unit 225.

[0282] After starting the recording, when the door 11C or the drawer 16A of the opened refrigerator 1 closes, the recording control unit 224 stops the recording.

[0283] According to this, since recording is ended at a timing when it is highly likely that the food inventory of the refrigerator 1 ends, it is possible to suppress unnecessary recording in a situation where no food is taken in or out of the refrigerator 1. Further, since the recording control unit 224 can suppress an increase in the data capacity of the recording file 231, it is possible to prevent an increase in the processing target of the food detection unit 225 and an increase in the processing time and processing load of the food detection unit 225.

[0284] When the recording control unit 224 stops the recording, it generates the shooting result from when the door 11C or the drawer 16A of the refrigerator 1 is opened until it closes as one recording file 231.

[0285] According to this, since the shooting result from when the door 11C or the drawer 16A of the refrigerator 1 is opened until it closes is generated as one recording file 231, it is possible to quickly generate one recording file 231 while suppressing an increase in the data capacity of one recording file 231. Further, since it is possible to quickly generate one recording file 231 while suppressing an increase in the data capacity of one recording file 231, after food is taken in or out of the refrigerator 1, the food detection unit 225 can quickly start processing related to the recording file 231.

[0286] After the recording control unit 224 stops the recording, when the human presence sensor 212 detects that there is no person around the refrigerator 1, the shooting control unit 223 causes the camera 213 to end the shooting.

[0287] According to this, when the human presence sensor 212 detects that there is no person around the refrigerator 1, in order to cause the camera 213 to end shooting, it is possible to prevent the camera 213 from shooting in a situation where food is not put into or taken out of the refrigerator 1. Therefore, while preventing an increase in power consumption, it is possible to accurately shoot the food put into and taken out of the refrigerator 1.

[0288] The camera unit 2 includes a food detection unit 225 that detects food put into and taken out of the refrigerator 1 based on the recording result of the recording control unit 224.

[0289] According to this, since food can be detected based on the shooting result of accurately shooting the food put into and taken out of the refrigerator 1, it is possible to accurately detect the food put into and taken out of the refrigerator 1.

[0290] Also, as described above, the food management application 311 causes the terminal processor 300 of the terminal device 3 including the touch panel 32 to display the refrigerated compartment interior image RG on the touch panel 32, and in the displayed refrigerated compartment interior image RG, functions as an application execution unit 301 that displays the photographed food image FG in the divided area BA corresponding to the storage area of the food stored in the refrigerated compartment 11. The terminal device 3 also includes a touch panel 32 and an application execution unit 301 that displays the refrigerated compartment interior image RG on the touch panel 32 and displays the photographed food image FG in the divided area BA corresponding to the storage area of the food stored in the refrigerated compartment 11 in the displayed refrigerated compartment interior image RG. The control method of the terminal device 3 is to display the refrigerated compartment interior image RG and display the photographed food image FG in the divided area BA corresponding to the storage area of the food stored in the refrigerated compartment 11 in the displayed refrigerated compartment interior image RG.

[0291] According to this food management application 311, terminal device 3, and control method of terminal device 3, in an image showing the interior of the refrigerator compartment, this captured food image FG is displayed in a divided area BA corresponding to the storage area of the food stored in the refrigerator. Therefore, user P can intuitively understand what kind of food is stored in refrigerator compartment 11.

[0292] The captured food image FG is an image captured by the refrigerator compartment camera 203.

[0293] The captured food image FG is an image captured by the refrigerator compartment camera 203. According to this, since an image of the food actually stored in refrigerator 1 is provided to user P, user P can more intuitively understand what kind of food is stored in refrigerator compartment 11.

[0294] The refrigerator compartment interior image RG is an image corresponding to the model of refrigerator 1 used by user P of terminal device 3.

[0295] According to this, since the refrigerator compartment interior image RG corresponding to refrigerator compartment 11 of the actually used refrigerator 1 is provided to user P, user P can more intuitively understand what kind of food is stored in refrigerator compartment 11.

[0296] The refrigerator compartment interior image RG is divided into a plurality of divided areas BA. The number of divided areas BA in the refrigerator compartment interior image RG is a number corresponding to the model of refrigerator 1 used by user P of terminal device 3.

[0297] According to this, user P can grasp what kind of food is stored in the storage area of refrigerator compartment 11 of the actually used refrigerator 1. Therefore, user P can more intuitively understand what kind of food is stored in refrigerator compartment 11.

[0298] When the application execution unit 301 displays a plurality of captured food images FG in one divided area BA, the plurality of captured food images FG are arranged and displayed.

[0299] According to this, for each divided area BA, a plurality of photographed food images FG can be displayed in an easy-to-view manner, so that the user P can intuitively and easily grasp what kind of food is stored in the refrigerator 11.

[0300] In the divided area BA of 1, an upper limit number of photographed food images FG that can be displayed simultaneously is set.

[0301] According to this, it is possible to prevent the visibility from decreasing when there are many photographed food images FG displayed in the divided area BA of 1. Therefore, the user P can intuitively and easily grasp what kind of food is stored in the refrigerator 11.

[0302] When the number of photographed food images FG to be displayed in the divided area BA of 1 exceeds the upper limit number, the application execution unit 301 displays the photographed food images FG in a switchable manner within the range of the upper limit number.

[0303] According to this, while preventing the visibility from decreasing when there are many photographed food images FG displayed in the divided area BA of 1, the user P can view the photographed food images FG that cannot be fully displayed.

[0304] The application execution unit 301 accepts input of a memo from the user P of the terminal device 3 for each photographed food image FG, and displays a memo input icon MIC indicating that the memo has been accepted in association with the photographed food image FG.

[0305] According to this, the user P can grasp for which food the memo has been input. Therefore, the user P can intuitively grasp what kind of food is stored in the refrigerator 11 and also grasp for which food the memo has been input.

[0306] The food management application 311 is an application program that can be installed in the terminal device 3.

[0307] According to this, a terminal device 3 that does not have a function of displaying the image RG inside the refrigerator compartment and the photographed food image FG can be made into a terminal device 3 that can display these images by installing the food management application 311.

[0308] Also, as described above, the food management application 311 causes the terminal processor 300 of the terminal device 3 equipped with the touch panel 32 to function as an application execution unit 301 that displays a photographed image of the extraction camera 204 that photographs the extraction 16A of the refrigerator 1 having the upper case JCA and the lower case GCA. The application execution unit 301 can display the upper case image JCG and the lower case image GCG, and accepts input of a memo from the user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG. When the application execution unit 301 accepts input of a memo from the user P for the upper case image JCG, it displays a memo input icon MIC indicating that the input of the memo has been accepted in association with the upper case image JCG, and when it accepts input of a memo from the user P for the lower case image GCG, it displays the memo input icon MIC in association with the lower case image GCG.

[0309] Also, the terminal device 3 includes a touch panel 32 and an application execution unit 301 that displays a photographed image of the extraction camera 204. The application execution unit 301 can display the upper case image JCG and the lower case image GCG, and accepts input of a memo from the user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG. When the application execution unit 301 accepts input of a memo from the user P for the upper case image JCG, it displays a memo input icon MIC indicating that the input of the memo has been accepted in association with the upper case image JCG, and when it accepts input of a memo from the user P for the lower case image GCG, it displays the memo input icon MIC in association with the lower case image GCG.

[0310] In addition, the control method of the terminal device 3 can display the upper case image JCG and the lower case image GCG, and for each of the upper case image JCG and the lower case image GCG, it accepts a memo input from the user P of the terminal device 3. When accepting a memo input from the user P for the upper case image JCG, it displays a memo input icon MIC indicating that the memo input has been accepted in association with the upper case image JCG. When accepting a memo input from the user P for the lower case image GCG, it displays the memo input icon MIC in association with the lower case image GCG.

[0311] According to this, in the food management application 311, the terminal device 3, and the control method of the terminal device 3, a photographed image of the case CA included in the drawer 16A is displayed, and the user P can input a memo to the displayed photographed image. Therefore, the user P can intuitively grasp the food stored in the drawer 16A of the refrigerator 1, and the convenience of the user P can be improved.

[0312] The application execution unit 301 displays the upper case image JCG superimposed on the lower case image GCG, and when accepting an operation to move the upper case image JCG from the user P, it moves the first case image to display the lower case image GCG.

[0313] According to this, the lower case image GCG can be displayed in a manner of actually moving the case CA. Therefore, the user P can more intuitively grasp the food stored in the drawer 16A of the refrigerator 1, and the convenience of the user P can be improved.

[0314] When the application execution unit 301 is displaying the memo input icon MIC in association with the upper case image JCG, it moves the memo input icon MIC together with the upper case image JCG.

[0315] According to this, even when the user P moves the upper case image JCG, the association between the upper case image JCG and the memo input icon MIC can be maintained. Therefore, even when the user P moves the upper case image JCG, the user P can appropriately grasp which image the memo input icon MIC relates to, and the convenience of the user P can be improved.

[0316] The application execution unit 301 displays a switching button for switching the captured image to be displayed to the upper case image JCG or the lower case image GCG, and switches the captured image to be displayed to the upper case image JCG or the lower case image GCG according to an operation on the switching button.

[0317] According to this, since the case image CG to be displayed can be easily switched by operating the switching button, the convenience of the user P can be improved.

[0318] The upper case image JCG is a captured image when the upper case JCA is pulled out from the refrigerator 1 by a predetermined threshold value or more. The lower case image GCG is a captured image when the lower case GCA is pulled out from the refrigerator 1 by a predetermined threshold value or more.

[0319] According to this, since the captured image in which as many foods as possible are captured can be used as the case image CG, the user P can intuitively grasp the foods stored in the drawer 16A of the refrigerator 1, and the convenience of the user P can be improved.

[0320] The food management application 311 is an application program that can be installed on the terminal device 3.

[0321] According to this, a terminal device 3 that does not have a function of displaying the case image CG and a function of accepting memo input for the displayed case image CG can be made into a terminal device 3 having these functions by installing the food management application 311.

[0322] (Embodiment 2) Next, Embodiment 2 will be described. In the description of Embodiment 2, for the same components as those of each part of the food management system 1000 in Embodiment 1, the same reference numerals will be given and the detailed description will be omitted as appropriate.

[0323] [2-1. Configuration] FIG. 26 is a block diagram showing the configurations of the terminal device 3 and the food management server 4 in Embodiment 2. As is clear from comparing FIG. 26 with FIG. 7, the server processor 400 in Embodiment 2 functions as a food recognition unit 403 in addition to the server communication control unit 401 and the information processing unit 402 by executing a control program stored in the server storage unit 410.

[0324] The food recognition unit 403 performs food recognition processing based on the photographed food image FG shown by the food image data 4116 included in each of the first update information and the second update information.

[0325] The food recognition processing is a process of recognizing a food from the photographed food image FG and acquiring an icon indicating the same type of food as the recognized food from an icon database. This icon database stores an icon indicating a food for each food. Hereinafter, this icon will be referred to as a food icon. The icon database may be stored in the food management server 4 or may be stored in an external device that can communicate with the food management server 4. In the present embodiment, the food icon corresponds to an example of the food image of the present disclosure.

[0326] For example, the food recognition unit 403 recognizes a food as follows. The food recognition unit 403 identifies the image in the food image database that most closely matches the photographed food image FG indicated by the food image data 4116 included in the update information. The food image database is a database in which image data is stored for each food item. The degree of match is determined based on feature quantities such as shape and color, for example. Then, the food recognition unit 403 recognizes that the food corresponding to the identified image is the food indicated by the food image data 4116 included in the update information.

[0327] Note that the food recognition unit 403 may recognize food by means of AI (Artificial Intelligence). For example, perform machine learning processing to learn feature quantities such as the color and shape of food from the photographed food image FG for teaching, and construct a learned model. This learned model is stored in the server storage unit 410. The food recognition unit 403 uses the food image data 4116 included in the update information as input data, refers to the learned model, and recognizes the food indicated by the photographed food image FG. Here, this learned model may be stored in a device other than the food management server 4 connected to the global network GN, and the food may be recognized in the food management server 4. In the case of this configuration, the food recognition unit 403 transmits the food image data 4116 included in the update information to the device storing the learned model, and acquires the food identification result from this device. Then, the food recognition unit 403 recognizes that the food indicated by the acquired identification result is the food indicated by the photographed food image FG indicated by the food image data 4116 included in the update information.

[0328] When the food recognition unit 403 recognizes food, it acquires the food icon of the recognized food from the icon database. Then, in Embodiment 2, instead of or together with the food image data 4116 of the refrigerator food management record RR, the icon data of the food icon acquired by the food recognition unit 403 is appropriately stored. Also, in Embodiment 2, instead of or together with the food image data 4116 of the drawer food management record HR, the icon data of the food icon acquired by the food recognition unit 403 is appropriately stored.

[0329] As a result, in the second embodiment, a food icon can be displayed in the app UI 320 instead of the photographed food image FG.

[0330] [2-2. Effects, etc.] According to the second embodiment, the same effects as those of the first embodiment are achieved.

[0331] (Other embodiments) As described above, the above embodiments have been described as examples disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments with changes, replacements, additions, omissions, etc. Further, it is also possible to combine the respective components described in the above embodiments to form a new embodiment. Therefore, other embodiments will be exemplified below.

[0332] In the above-described embodiment, the refrigerator 1 is exemplified as the storage. However, the storage of the present disclosure is not limited to the refrigerator 1, and may be a warehouse installed in an office or the like, for example. In the above-described embodiment, food is exemplified as the article of the present disclosure, but the article of the present disclosure is not limited to food, and may be other articles such as books according to the storage.

[0333] In the above-described embodiment, a configuration in which the position regulating member 205 has the mark 207 at the center in the left-right direction of the main body 201 is exemplified. However, the position of the mark 207 is not limited to the center in the left-right direction of the main body 201. The position of the mark 207 may be other than the center in the left-right direction of the main body 201. In this case, in the center position recording paper CY, each mark is appropriately recorded so that the camera unit 2 adjusted in position based on the mark 207 can appropriately perform various detections.

[0334] In the above-described embodiments, as the in-warehouse image of the present disclosure, the in-warehouse image RG of the refrigerator compartment 11 when viewed from the front with the left door 11A and the right door 11B in the open state is exemplified. However, the in-warehouse image of the present disclosure is not limited to this in-warehouse image RG of the refrigerator compartment, and may be an image of the refrigerator compartment when viewed from the front with a single-opening door in the open state, or in a refrigerator 1 where one in-warehouse includes a refrigerator compartment and a freezer compartment, an image of this one in-warehouse when viewed from the front with the door in the open state.

[0335] The shooting range of the extraction camera 204 may be a range combining one or a plurality of arbitrary extraction compartments according to the type of the extraction compartment of the screen displayed by the app UI 320.

[0336] The food management system 1000 may manage other elements such as the remaining quantity and the expiration date in addition to the storage date and time.

[0337] The main body 201 and the shooting member 202 included in the camera unit 2 may be configured separately.

[0338] The camera unit 2 may be configured to transmit the case image CG to the food management server 4 even when it is not pulled out by a predetermined threshold or more. In the case of this configuration, the food management server 4 stores the case image CG when it is not pulled out by a predetermined threshold or more and provides it to the terminal device 3. Then, the app execution unit 301 displays this case image CG on the app UI 320 so as to be interchangeable with each of the upper case image JCG and the lower case image GCG.

[0339] In the above-described embodiments, the case where the user P installs the camera unit 2 in the refrigerator 1 is exemplified, but the camera unit 2 may be shipped in a state where it is installed in the refrigerator 1.

[0340] The server processor 400 may function as the food detection unit 225, and the camera unit processor 220 may not function as the food detection unit 225. In the case of this configuration, the camera unit 2 transmits the recording file 231 to the food management server 4.

[0341] In the above-described embodiments, the human presence sensor 212 is exemplified as the sensor of the present disclosure. However, the sensor of the present disclosure is not limited to the human presence sensor 212, and may be an illuminance sensor, an electrostatic sensor provided in the gripping portion of the refrigerator 1, or the like. In the case of this electrostatic sensor, the refrigerator 1 and the camera unit 2 can communicate with each other, and the refrigerator 1 transmits the detection value of the electrostatic sensor to the camera unit 2.

[0342] The terminal device 3 may store the refrigerator compartment food management record RR and the taken-out food management record HR, and the food management application 311 may manage various information related to the food detected by the camera unit 2. In the case of this configuration, the application execution unit 301 appropriately updates the contents of each record stored in the terminal device 3 in the same manner as the above-described food management server 4. As a result, the food management system 1000 does not need to include the food management server 4, and the system configuration becomes simple. Also, in the case of this configuration, the application execution unit 301 may function as the food recognition unit 403.

[0343] For example, the types of rooms formed in the main body 10 of the refrigerator 1 are not limited to the refrigerator compartment 11, the ice-making compartment 12, the fresh freezing compartment 13, the freezing compartment 14, and the vegetable compartment 15, and may be at least one, or even another type of room may be formed. Also, the number of doors provided at the opening in the front of the refrigerator compartment 11 may be one.

[0344] For example, the cases provided with the drawers 14A and 15A are not limited to two stages, and may be more. In this case, a certain case corresponds to the first case of the present disclosure, and the case below this certain case corresponds to the second case of the present disclosure.

[0345] For example, the camera unit processor 220, the terminal processor 300, and the server processor 400 may be configured by a single processor, or may be configured by a plurality of processors.

[0346] The components shown in FIGS. 6, 7, and 26 are merely examples, and the specific implementation forms are not particularly limited. That is, it is not necessarily required to implement hardware corresponding to each component individually, and of course, it is also possible to adopt a configuration in which one processor executes a program to realize the functions of each component. In addition, in the above-described embodiments, part of the functions realized by software may be implemented as hardware, or part of the functions realized by hardware may be realized by software. Furthermore, regarding the specific detailed configurations of the other components of the camera unit 2, the terminal device 3, and the food management server 4, they can be arbitrarily changed without departing from the spirit of the present disclosure.

[0347] Also, for example, the step units of the operations shown in FIGS. 13, 14, 22, and 25 are divided according to the main processing contents in order to facilitate the understanding of the operations of the respective components of the food management system 1000, and the present disclosure is not limited by the way of dividing the processing units and their names.

[0348] Note that the above-described embodiments are for exemplifying the technologies in the present disclosure, and various changes, replacements, additions, omissions, etc. can be made within the scope of the claims or their equivalents.

Industrial Applicability

[0349] As described above, the program, the terminal device, and the control method of the terminal device according to the present invention can be used for the purpose of displaying images of foods entering and leaving the refrigerator.

Explanation of Reference Numerals

[0350] 1 Refrigerator (storage) 2 Camera unit (imaging device) 3 Terminal device 4 Food management server 10 Main cabinet 10A Front edge 10B Upper surface 11 Refrigerating chamber 11A Left door 11B Right door 11C Door 12 Ice Making Chamber (Drawer - type Storage Chamber) 12A, 13A, 14A, 15A, 16A Drawer 13 Fresh Freezing Chamber (Drawer - type Storage Chamber) 14 Freezing Chamber (Drawer - type Storage Chamber) 15 Vegetable Chamber (Drawer - type Storage Chamber) 32 Touch Panel (Display Section) 201 Main Body 202 Photographing Member 203 Refrigerator Camera 204 Drawer Camera 205 Position Regulation Member 206 Bottom Surface 207 Mark 212 Human Sensor (Sensor) 213 Camera 223 Photographing Control Section 224 Recording Control Section 225 Food Detection Section (Article Detection Section) 231 Recording File 300 Terminal Processor (Computer) 301 App Execution Section (Display Control Section) 311 Food Management App (Program, Application Program) 1000 Food Management System BA Partition Area (Area) FG, FG1 Photographed Food Image (Food Image) GCA Lower Case (Second Case) GCG Lower Case Image (Second Case Image) JCA Upper Case (First Case) JCG Upper Case Image (First Case Image) MIC Memo Input Icon (Memo Input Image) P User RG Refrigerator Interior Image (Interior Image)

Claims

1. A computer of a terminal device including a display unit is configured to cause the display unit to display an in - refrigerator image showing the inside of the refrigerator as seen from the front of the refrigerator, and in the displayed in - refrigerator image, cause a food image indicating the food to be displayed in an area corresponding to the storage area of the food stored in the refrigerator, thereby functioning as a display control unit. A program.

2. The food image is a captured image of a refrigerator compartment camera that captures food entering and leaving the refrigerator. The program according to claim 1.

3. The food image is an icon corresponding to the type of the food. The program according to claim 1.

4. The in - refrigerator image is an image corresponding to the model of the refrigerator used by the user of the terminal device. The program according to any one of claims 1 to 3.

5. The in - refrigerator image is divided into a plurality of areas, and the number of the areas of the in - refrigerator image is a number corresponding to the model of the refrigerator used by the user of the terminal device. The program according to claim 4.

6. The display control unit when displaying a plurality of the food images in one of the areas, arranges and displays the plurality of the food images. The program according to claim 5.

7. An upper limit number of the food images that can be simultaneously displayed is set for one of the areas. The program according to claim 6.

8. The display control unit when the number of the food images to be displayed in one of the areas exceeds the upper limit number, displays the food images so that they can be switched within the range of the upper limit number. The program according to claim 7.

9. The display control unit receives an input of a memo for each of the food images from the user of the terminal device, and displays a memo input image indicating that the memo has been received in association with the food image. The program according to any one of claims 1 to 8.

10. An application program installable on the terminal device. The program according to any one of claims 1 to 9.

11. A display unit, and a display control unit that causes the display unit to display an in - refrigerator image showing the inside of the refrigerator as seen from the front of the refrigerator, and in the displayed in - refrigerator image, causes a food image indicating the food to be displayed in an area corresponding to the storage area of the food stored in the refrigerator. A terminal device.

12. Displays an in - refrigerator image showing the inside of the refrigerator as seen from the front of the refrigerator. In the displayed image of the interior of the storage, a food image indicating the food is displayed in an area corresponding to the storage area of the food stored in the storage. A control method for a terminal device.

Images