Program, terminal device, and method for controlling the terminal device

Abstract

To provide a program, a terminal device and a terminal device control method which enable a user to intuitively assess food stored in a drawer inside a refrigerator and can improve user convenience.SOLUTION: A program causes a computer of a terminal device to function as a display control section which displays images taken with cameras imaging interiors of a first case and a second case which is a case positioned below the first case. The display control section: can display a first case image and a second case image; receives input of a user's memo with respect to the first case image and the second case image; displays a memo input image in association with the first case image when receiving the memo with respect to the first case image; and displays the memo input image in association with the second case image when receiving the memo with respect to the second case image.SELECTED DRAWING: Figure 23

Description

Technical Field

[0001] The present disclosure relates to a program, a terminal device, and a control method for a terminal device.

Background Art

[0002] Patent Document 1 discloses a technique in which when an object passing through an opening of a refrigerator is detected, a camera installed above the refrigerator performs shooting, and the refrigerator or a terminal device displays information on food stored in the refrigerator based on the shooting result.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides a program, a terminal device, and a control method for a terminal device that enable a user to intuitively grasp food stored in a drawer of a refrigerator and improve the convenience for the user.

Means for Solving the Problems

[0005] The program in the present disclosure causes a computer of a terminal device including a display unit to capture an image of a drawer of a refrigerator having a first case and a second case which is a lower case of the first case mosquito function as a display control unit that displays a captured image of the drawer, and the display control unit can display a first case image which is the captured image of the first case and a second case image which is the captured image of the second case, The first case image is displayed superimposed on the image area of ​​the second case image. When the terminal device receives an operation from a user to move the first case image in the direction of movement of the first case, the display range of the first case image is changed in response to the operation, and the display range of the second case image is changed in response to the change in the display range of the first case image. 。

[0006] In addition, the terminal device in the present disclosure includes a display unit and captures an image of a drawer of a refrigerator having a first case and a second case which is a lower case of the first case mosquitoThe system comprises a display control unit that displays the image captured by the camera, the display control unit being capable of displaying a first case image, which is the image captured by the first case, and a second case image, which is the image captured by the second case. The first case image is displayed superimposed on the image area of ​​the second case image. When the user requests an operation to move the first case image in the direction of movement of the first case, the display range of the first case image is changed in response to the operation, and the display range of the second case image is changed in response to the change in the display range of the first case image. .

[0007] Furthermore, the terminal device control method in this disclosure involves photographing the drawer of a refrigerator having a first case and a second case which is the lower case of the first case. mosquito A control method for a terminal device that displays images captured by a camera, wherein the device is capable of displaying a first case image, which is the captured image of the first case, and a second case image, which is the captured image of the second case. The first case image is displayed superimposed on the image area of ​​the second case image. When the terminal device receives an operation from a user to move the first case image in the direction of movement of the first case, the display range of the first case image is changed in response to the operation, and the display range of the second case image is changed in response to the change in the display range of the first case image. . [Effects of the Invention]

[0008] The program, terminal device, and control method for the terminal device in this disclosure display a photograph of the case contained in the drawer, and the user can input notes into the displayed photograph. Therefore, the user can intuitively understand the food stored in the refrigerator drawer and improve user convenience. [Brief explanation of the drawing]

[0009] [Figure 1] Diagram showing the configuration of the food management system in Embodiment 1 [Figure 2] Side view of the camera unit installed in the refrigerator in Embodiment 1, viewed from the right side. [Figure 3] Perspective view of the position regulating member in contact with the front edge of the main box body in Embodiment 1, as seen from the left front side. [Figure 4] Front view of the refrigerator and camera unit in Embodiment 1, as seen from the front. [Figure 5] Plan view of the refrigerator and camera unit in Embodiment 1, viewed from the right side. [Figure 6] Block diagram showing the configuration of the camera unit in Embodiment 1 [Figure 7]Block diagram showing the configuration of the terminal device and the food management server in Embodiment 1 [Figure 8] Diagram showing an example of the refrigerator management database in Embodiment 1 [Figure 9] Diagram showing an example of the in-refrigerator image in Embodiment 1 [Figure 10] Diagram showing an example of the drawer management database in Embodiment 1 [Figure 11] Diagram showing an example of the adjustment screen in Embodiment 1 [Figure 12] Diagram showing an example of the central position recording paper in Embodiment 1 [Figure 13] Flowchart showing the operation of the camera unit in Embodiment 1 [Figure 14] Flowchart showing the operation of the food management system in Embodiment 1 [Figure 15] Diagram for explaining the incoming / outgoing determination image in Embodiment 1 [Figure 16] Diagram showing an example of the refrigerator home screen in Embodiment 1 [Figure 17] Diagram showing the input procedure of the memo in Embodiment 1 [Figure 18] Diagram showing an example of the storage list screen in Embodiment 1 [Figure 19] Diagram showing an example of the new addition list screen in Embodiment 1 [Figure 20] Diagram for explaining the addition procedure of food information in Embodiment 1 [Figure 21] Diagram showing an example of the deletion target list screen in Embodiment 1 [Figure 22] Flowchart showing the operation of the food management system in Embodiment 1 [Figure 23] Diagram showing an example of the vegetable room home screen in Embodiment 1 [Figure 24] Diagram showing an example of the food image list screen in Embodiment 1 [Figure 25] Flowchart showing the operation of the food management system in Embodiment 1 [Figure 26] Block diagram showing the configuration of the terminal device and food management server in Embodiment 2 [Modes for carrying out the invention]

[0010] (Knowledge and other information that formed the basis of this disclosure) At the time the inventors conceived of this disclosure, there was a technology that allowed a terminal device to display a list of foods stored in a refrigerator. However, the inventors discovered that with the conventional technology, simply displaying a list of foods did not allow the user to intuitively understand what foods were stored in the drawer-type storage compartments such as the vegetable compartment or freezer compartment. Furthermore, the inventors discovered that with the conventional technology, simply displaying a list of foods only allowed the user to view the list, leaving room for improvement in terms of user convenience. The inventors then arrived at the subject matter of this disclosure in order to solve these problems. Therefore, this disclosure provides a program, a terminal device, and a control method for the terminal device that enable users to intuitively understand the food stored in the drawers of a refrigerator and improve user convenience.

[0011] The embodiments will be described in detail below with reference to the drawings. However, some unnecessarily detailed explanations may be omitted. For example, detailed explanations of already well-known matters or redundant explanations of substantially identical configurations may be omitted. The attached drawings and the following description are provided to enable those skilled in the art to fully understand this disclosure and are not intended to limit the subject matter described in the claims.

[0012] (Embodiment 1) First, let me describe Embodiment 1. [1-1. Structure] [1-1-1. Food Management System Configuration] Figure 1 shows the configuration of the food management system 1000.

[0013] The food management system 1000 is a system that manages the date and time of storage of food stored in refrigerator 1. Refrigerator 1 corresponds to an example of a storage facility in this disclosure. Food corresponds to an example of an article in this disclosure.

[0014] The food management system 1000 includes a refrigerator 1. The refrigerator 1 includes a main box 10 with an open front. The main box 10 has a refrigerator compartment 11, an ice-making compartment 12, a fresh freezing compartment 13, a freezer compartment 14, and a vegetable compartment 15. The front opening of the refrigerator compartment 11 is provided with a rotating left door 11A and a right door 11B. The ice-making compartment 12, the fresh freezing compartment 13, the freezer compartment 14, and the vegetable compartment 15 are each provided with drawers 12A, 13A, 14A, and 15A for storing food. The drawers 14A and 15A of this disclosure include two cases CA, an upper case JCA and a lower case GCA. The upper case JCA corresponds to an example of the first case of this disclosure. The lower case GCA corresponds to an example of the second case of this disclosure.

[0015] In the following description, when the left door 11A and the right door 11B are not distinguished, they will be referred to as "doors" and designated as "11C". Also, in the following description, when the drawers 12A, 13A, 14A, and 15A are not distinguished, they will be referred to as "drawers" and designated as "16A".

[0016] The food management system 1000 includes a camera unit 2. The camera unit 2 is a device that takes pictures and detects food being put into and taken out of the refrigerator 1 from the captured images. The camera unit 2 is installed on the upper surface 10B of the main casing 10 of the refrigerator 1. Camera unit 2 corresponds to an example of the imaging device of this disclosure.

[0017] The food management system 1000 includes a terminal device 3. The terminal device 3 is, for example, a smartphone or a tablet. The terminal device 3 has an application program installed for managing the food in the refrigerator 1, and communicates with the food management server 4 through the functions of this application program. In the following explanation, this application program will be referred to as the "Food Management App" and assigned the code "311". Food management app 311 corresponds to an example of a program and application program of this disclosure.

[0018] In Figure 1, users P who are at home are shown with a solid line, and users P who have left home H are shown with a dotted line. When terminal device 3 is used by user P who is at home, it communicates with the food management server 4 via the communication device 5, or without going through the communication device 5, according to the functions of the food management application 311. Also, when terminal device 3 is used by user P who has left home H and cannot establish a communication connection with the communication device 5, it communicates with the food management server 4 without going through the communication device 5, according to the functions of the food management application 311.

[0019] Communication device 5 connects to a global network GN, which includes the Internet, telephone networks, and other communication networks, and communicates with a food management server 4 that is connected to the global network GN. Communication device 5 is an interface device for connecting terminal device 3 to the global network GN.

[0020] The food management system 1000 includes a food management server 4. The food management server 4 is a server device that manages the food in refrigerator 1 and is connected to the global network GN. In each diagram, the food management server 4 is represented by a single block, but this does not necessarily mean that the food management server 4 is composed of a single server device. For example, the food management server 4 may be composed of multiple server devices with different processing functions.

[0021] Next, we will describe the configuration of the camera unit 2, the terminal device 3, and the food management server 4.

[0022] [1-1-2. Camera Unit Configuration] The camera unit 2 will be described with reference to Figures 2-4. Figures 2-4 illustrate the X, Y, and Z axes. The X, Y, and Z axes are orthogonal to each other. The Z axis indicates the vertical direction. The X and Y axes are parallel to the horizontal direction. The X axis indicates the left-right direction and also represents the width direction of refrigerator 1 and camera unit 2. The Y axis indicates the front-back direction and also represents the depth direction of refrigerator 1 and camera unit 2. The positive direction of the X axis is to the right. The positive direction of the Y axis is forward. The positive direction of the Z axis is upward.

[0023] Figure 2 is a side view of the camera unit 2 installed in the refrigerator 1, viewed from the right side. Figure 3 is a perspective view of the position regulating member 205, which is in contact with the front edge 10A of the main casing 10, viewed from the left front side.

[0024] As shown in Figures 2 and 3, the camera unit 2 comprises a box-shaped body 201 installed on the upper surface 10B of the main casing 10. Furthermore, the camera unit 2 includes a shooting member 202 extending forward from the body 201, located above the front of the body 201. The shooting member 202 comprises a refrigerator compartment camera 203 and a drawer camera 204. The refrigerator compartment camera 203 and the drawer camera 204 are each provided in front of the body 201 at the lower part of the shooting member 202.

[0025] The main body 201 includes a position regulating member 205 that contacts the front edge 10A of the main box 10 to regulate the position of the imaging member 202 in the front-rear direction of the refrigerator 1. The position regulating member 205 is a position regulating piece that extends downward from the bottom surface 206 of the main body 201 when installed. The position regulating member 205 extends in the left-right direction of the main body 201 on the front surface of the main body 201 and has a mark 207 in the left-right center of the main body 201. This mark 207 indicates the left-right center position of the camera unit 2.

[0026] The length of the position regulating member 205 is shorter than the length from the top surface 10B of the main box 10 to the gasket provided between the refrigerator 1 and the door 11C when installed. More specifically, the length of the position regulating member 205 is shorter than the length from the top surface 10B of the main box 10 to the door gasket provided on the upper inner surface of the door 11C, substantially parallel to the top surface 10B when installed.

[0027] The main body 201 is fixed in the position restricted by the position regulating member 205 by the built-in magnet 208. The means for fixing the main body 201 is not limited to the magnet 208, but may also be other means such as double-sided tape.

[0028] Figure 4 is a front view of the refrigerator 1 and camera unit 2 as seen from the front. In Figure 4, the left door 11A and the right door 11B of the refrigerator 1 are open. Figure 5 is a top view of the refrigerator 1 and camera unit 2 as seen from the right side. In Figure 5, the drawer 15A of the refrigerator 1 is open, i.e., pulled out. In Figures 4 and 5, the camera unit 2 has a mark 207 located approximately in the center in the left-right direction of the refrigerator 1, and a position regulating member 205 is in contact with the front edge 10A.

[0029] Camera unit 2 includes a camera 203 for the refrigerator compartment and a camera 204 for the drawer. The refrigerator compartment camera 203 photographs food being put in and taken out of the refrigerator compartment 11 by taking pictures from the front upper part of the refrigerator 1 downwards.

[0030] For example, the shooting range of the refrigerator compartment camera 203 includes the range A1 shown in Figure 4 when viewed from the front. Range A1 includes the door pocket 111 provided in the door 11C and the front opening of the refrigerator compartment 11 when the door 11C is open. In addition, the shooting range of the refrigerator compartment camera 203 includes the range A3 shown in Figure 5 when viewed from the side. Range A3 includes the range from the open door 11C to the front end of the shelf 112 in the front-to-back direction, and the range including the opening of the refrigerator compartment 11 in the up-to-down direction.

[0031] The drawer camera 204 photographs food items being put in and taken out of the drawer-type storage compartment by shooting from the front upper part of the refrigerator 1 downwards.

[0032] For example, the shooting range of the drawer camera 204 includes range A2 shown in Figure 4 when viewed from the front. Range A2 is the range that includes drawer 16A when viewed from the front of refrigerator 1. Also, the shooting range of the drawer camera 204 includes range A4 shown in Figure 5 when viewed from the side. Range A4 is the range that includes drawer 16A in its fully extended state in the front-to-back direction.

[0033] The camera unit 2 includes a distance measuring sensor 209 for the left door, a distance measuring sensor 210 for the right door, and a distance measuring sensor 211 for the pull-out mechanism.

[0034] The left door distance sensor 209 is a distance sensor for detecting the open / closed state of the left door 11A. The left door distance sensor 209 detects the distance from the left door 11A as the open / closed state of the left door 11A.

[0035] The right door distance sensor 210 is a distance sensor for detecting the open / closed state of the right door 11B. The right door distance sensor 210 detects the distance from the right door 11B as the open / closed state of the right door 11B.

[0036] The drawer distance sensor 211 is a distance sensor for detecting the open / closed state of drawer 16A. When drawer 16A is in the open state, it is pulled out from the corresponding drawer-type storage compartment. When drawer 16A is in the closed state, it is stored in the corresponding drawer-type storage compartment. The drawer distance sensor 211 detects the distance from drawer 16A to determine its open / closed state. In this embodiment, the drawer distance sensor 211 detects the open / closed state of drawers 14A and 15A, respectively.

[0037] The camera unit 2 has a human presence sensor 212. The human presence sensor 212 outputs the detected value to the camera unit control unit 20. The human presence sensor 212 is installed at a predetermined position on the imaging member 202. The motion sensor 212 corresponds to an example of a “sensor” in this disclosure.

[0038] Note that the arrangement and installation positions of each sensor and camera in Figures 4 and 5 are illustrated for the convenience of explaining the camera unit 2, and their arrangement and installation positions are not limited to those shown in Figures 4 and 5.

[0039] Figure 6 is a block diagram showing the configuration of camera unit 2. The camera unit 2 comprises a camera unit control unit 20, a camera unit communication unit 21, a sensor unit 22, an imaging unit 23, and a power supply unit 24.

[0040] The camera unit control unit 20 includes a camera unit processor 220, which is a processor such as a CPU (Central Processing Unit) or MPU (Micro-processing unit), and a camera unit storage unit 230. The camera unit control unit 20 controls each part of the camera unit 2 by having the camera unit processor 220 read and execute control programs stored in the camera unit storage unit 230. By executing the control programs stored in the camera unit storage unit 230, the camera unit processor 220 functions as the camera unit communication control unit 221, the detection value processing unit 222, the shooting control unit 223, the recording control unit 224, and the food detection unit 225.

[0041] The camera unit storage unit 230 includes memory for storing programs executed by the camera unit processor 220 and data processed by the camera unit processor 220. The camera unit storage unit 230 stores control programs executed by the camera unit processor 220, camera unit ID 4112 which is identification information for camera unit 2, and various other data. The camera unit storage unit 230 has a non-volatile storage area. Alternatively, the camera unit storage unit 230 may also have a volatile storage area which constitutes the work area of ​​the camera unit processor 220.

[0042] The camera unit communication unit 21 is a communication interface equipped with communication circuits and connectors that conform to a predetermined communication standard, and communicates with the food management server 4 and terminal device 3 under the control of the camera unit control unit 20. In this embodiment, the communication standard used by the camera unit communication unit 21 is exemplified as a wireless communication standard, but a wired communication standard may also be used.

[0043] The sensor unit 22 includes a distance measuring sensor 209 for the left door, a distance measuring sensor 210 for the right door, a distance measuring sensor 211 for the pull-out mechanism, and a human presence sensor 212, and outputs the detected value for each sensor to the camera unit control unit 20.

[0044] The imaging unit 23 includes a refrigerator compartment camera 203 and a drawer camera 204. The imaging unit 23 outputs the imaging results of the refrigerator compartment camera 203 and the drawer camera 204 to the camera unit control unit 20. The refrigerator compartment camera 203 and the drawer camera 204 capture moving images. Hereafter, when not distinguishing between the refrigerator compartment camera 203 and the drawer camera 204, they will be referred to as "camera" and denoted with the code "213".

[0045] The power supply unit 24 is equipped with hardware such as a power supply circuit and supplies power to each part of the camera unit 2. In this embodiment, the power cable 7 of the camera unit 2 is connected to a commercial AC power supply 8, and the power supply unit 24 supplies power to each part of the camera unit 2 based on the power supplied from the external commercial AC power supply 8.

[0046] The power source for the power supply unit 24 is not limited to an external commercial AC power supply 8, but may also be the refrigerator 1 or a battery. If the power source is the former, the power supply unit 24 and the refrigerator 1 are equipped with hardware such as a port that conforms to the USB power supply standard. In the former case, the USB cable of the camera unit 2 is connected to the USB port of the refrigerator 1, and the refrigerator 1 supplies power to the power supply unit 24 via the USB cable. If the power source is the latter, the camera unit 2 is equipped with a battery.

[0047] As described above, the camera unit control unit 20 functions as a camera unit communication control unit 221, a detected value processing unit 222, a shooting control unit 223, a recording control unit 224, and a food detection unit 225. The food detection unit 225 corresponds to an example of an item detection unit.

[0048] The camera unit communication control unit 221 communicates with the food management server 4 via the camera unit communication unit 21.

[0049] The detection value processing unit 222 determines whether the left door 11A, the right door 11B, and the drawers 14A and 15A are open or closed based on the detection values ​​of each sensor output from the sensor unit 22. If the detection value processing unit 222 determines that either the left door 11A or the right door 11B is open, it outputs door open state information indicating that door 11C is open to the recording control unit 224 and the food detection unit 225. If the detection value processing unit 222 determines that either the left door 11A or the right door 11B is closed, it outputs door closed state information indicating that door 11C is closed to the recording control unit 224 and the food detection unit 225. If the detection value processing unit 222 determines that either the drawers 14A or 15A is open, it outputs drawer open state information indicating that either the drawers 14A or 15A is open to the recording control unit 224 and the food detection unit 225. Furthermore, if the detection value processing unit 222 determines that both drawers 14A and 15A are in the closed state, it outputs drawer closed state information indicating that drawers 14A and 15A are in the closed state to the recording control unit 224 and the food detection unit 225.

[0050] Furthermore, the detection value processing unit 222 determines whether or not a person is present around the refrigerator 1 based on the detection value of the human presence sensor 212 output from the sensor unit 22. If the detection value processing unit 222 determines that a person is present around the refrigerator 1, it outputs presence / absence information indicating the presence of a person to the shooting control unit 223. If the detection value processing unit 222 determines that there is no person present around the refrigerator 1, it outputs presence / absence information indicating the absence of a person to the shooting control unit 223.

[0051] If the detection value processing unit 222 outputs presence / absence information indicating the presence of a person, the shooting control unit 223 instructs the refrigerator camera 203 and the drawer camera 204 to start shooting. If the detection value processing unit 222 outputs presence / absence information indicating the absence of a person, the shooting control unit 223 instructs the refrigerator camera 203 and the drawer camera 204 to stop shooting.

[0052] When the detection value processing unit 222 outputs door open status information, the recording control unit 224 starts recording the results of the refrigerator compartment camera 203. When the detection value processing unit 222 outputs door closed status information, the recording control unit 224 stops recording the results of the refrigerator compartment camera 203 that was started. Also, when the detection value processing unit 222 outputs drawer open status information, the recording control unit 224 starts recording the results of the drawer camera 204 that was started. When the detection value processing unit 222 outputs drawer closed status information, the recording control unit 224 stops recording the results of the drawer camera 204 that was started. When the recording control unit 224 finishes recording, it generates the video footage from the start to the end of recording as a single recording file 231 and stores it in the camera unit storage unit 230.

[0053] The food detection unit 225 detects food items being put into or taken out of the refrigerator 1 based on the recorded video file 231 stored in the camera unit storage unit 230.

[0054] Figure 7 is a block diagram showing the configuration of the terminal device 3 and the food management server 4.

[0055] [1-1-3. Food Management Server Configuration] First, let's explain the configuration of food management server 4. The food management server 4 comprises a server control unit 40 and a server communication unit 41.

[0056] The server control unit 40 includes a server processor 400, which is a processor such as a CPU or MPU, and a server storage unit 410. The server control unit 40 controls each part of the food management server 4 by having the server processor 400 read and execute control programs stored in the server storage unit 410. The server processor 400 functions as a server communication control unit 401 and an information processing unit 402 by executing the control programs stored in the server storage unit 410.

[0057] The server storage unit 410 has memory for storing programs executed by the server processor 400 and data processed by the server processor 400. The server storage unit 410 stores control programs executed by the server processor 400, the refrigerator food management database 411, the drawer food management database 412, and various other data. The server storage unit 410 has a non-volatile storage area. The server storage unit 410 may also have a volatile storage area and constitute the work area of ​​the server processor 400.

[0058] Figure 8 shows an example of a record stored in the refrigerator food management database 411. The refrigerated compartment food management database 411 is a database that stores various information related to the food stored in the refrigerated compartment 11. In the following explanation, a single record stored in the refrigerator food management database 411 will be referred to as a "refrigerated food management record" and will be denoted with the symbol "RR".

[0059] The refrigerator food management record RR includes account ID 4111, camera unit ID 4112, refrigerator model information 4113, refrigerator interior image data 4114, in / out information 4115, food image data 4116, storage date and time information 4117, divided area number 4118, memo input information 4119, memo information 4130, deletion information 4131, and deletion date and time information 4132.

[0060] Account ID 4111 is identification information that identifies the account assigned to user P who uses the food management app 311.

[0061] Refrigerator model information 4113 indicates the model of refrigerator 1 used by user P, whose account ID 4111 corresponds to the same record.

[0062] The refrigerator compartment interior image data 4114 is image data of the interior of refrigerator compartment 11 of refrigerator 1. Hereinafter, the image showing the interior of refrigerator compartment 11 will be referred to as the "refrigerator compartment interior image" and will be denoted with the code "RG". The refrigerator compartment interior image data 4114 represents the refrigerator compartment interior image RG corresponding to the refrigerator model indicated by the refrigerator model information 4113 associated with the same record. Image RG of the refrigerator compartment interior corresponds to an example of an interior image.

[0063] Figure 9 shows an example of an image of the inside of a refrigerator compartment (RG). The refrigerator interior image RG shown in Figure 9 includes the left door pocket image RG1, the front of the refrigerator compartment image RG2, and the right door pocket image RG3. The refrigerator interior image RG shown in Figure 9 is arranged in this order from left to right in the figure. The left door pocket image RG1 is an image showing the door pocket 111 formed in the left door 11A. The front of the refrigerator compartment image RG2 is an image of the refrigerator compartment 11 viewed from the front when the left door 11A and the right door 11B are open. The right door pocket image RG3 is an image showing the door pocket 111 formed in the right door 11B.

[0064] The refrigerator interior image RG is divided into multiple areas. Each of these areas is called a "divided area" and is denoted with the code "BA". The refrigerator interior image RG shown in Figure 9 is divided into 15 divided areas BA. Each divided area BA is assigned a different number. In the following explanation, this number will be called the divided area number and will be denoted with the code "4118". Note that Figure 9 shows which divided area number is assigned to which divided area BA for convenience, and the divided area number 4118 will not be displayed in the refrigerator interior image RG shown on the refrigerator home screen GM2, which will be described later.

[0065] Returning to the explanation of Figure 8, the inbound / outbound information 4115 indicates whether or not there have been any new inbound or outbound transactions.

[0066] Food image data 4116 is image data of food. In this embodiment, the image shown by this food image data 4116 is an image captured by the refrigerator camera 203.

[0067] Storage date and time information 4117 indicates the date and time when the food was stored in refrigerator 1.

[0068] The refrigerated food management record RR has a divided area number 4118 for each food image data 4116.

[0069] The memo input status information 4119 indicates whether or not a memo has been entered.

[0070] Memo information 4130 is information indicating the entered memo.

[0071] Deletion information 4131 indicates whether the corresponding subrecord is a subrecord to be deleted from the refrigerator food management record RR. A subrecord is a record stored in the refrigerator food management record RR and has food image data 4116, storage date and time information 4117, division area number 4118, memo input status information 4119, and memo information 4130.

[0072] The deletion date and time information 4132 indicates the date and time when the corresponding subrecord was targeted for deletion. The information processing unit 402, described later, deletes the subrecord, the corresponding deletion information 4131, and the deletion date and time information 4132 from the refrigerator food management record RR at a predetermined timing if the deletion information 4131 indicates that the item is to be deleted.

[0073] Figure 10 shows a single record stored in the food drawer management database 412. The drawer food management database 412 is a database that stores various information related to food in the drawer-type storage compartment. In the explanation referring to Figures 23-25, the drawer food management database 412 is the database related to the vegetable compartment 15. In the following explanation, a single record stored in the retrieved food management database 412 is referred to as a "retrieved food management record" and is denoted by the code "HR".

[0074] The food drawer management record HR includes account ID 4111, camera unit ID 4112, update date and time information 4121, upper case information 4122, and lower case information 4124.

[0075] Update date and time information 4121 indicates the date and time when the contents of the food drawer management record HR were updated.

[0076] The upper case information 4122 includes upper case image data 4123, food image data 4116, memo input status information 4119, and memo information 4130.

[0077] Upper case image data 4123 is the image data of upper case image JCG, which is an image of upper case JCA. Upper case image JCG is an image captured by the drawer camera 204. The upper case image JCG corresponds to an example of the first case image of this disclosure.

[0078] In this embodiment, the image shown by the food image data 4116 contained in the upper case information 4122 is an image extracted from the upper case image JCG.

[0079] The lower case information 4124 includes lower case image data 4125, food image data 4116, memo input status information 4119, and memo information 4130.

[0080] Lower case image data 4125 is the image data of lower case image GCG, which is the image of lower case GCA. Lower case image GCG is an image captured by the drawer camera 204. The lower case image GCG corresponds to an example of the second case image of this disclosure. Hereafter, when the upper case image JCG and the lower case image GCG are not distinguished, they will be referred to as "case images" and denoted with the symbol "CG".

[0081] In this embodiment, the image shown by the food image data 4116 contained in the lower case information 4124 is an image extracted from the lower case image GCG.

[0082] Returning to the explanation of Figure 7, the server communication unit 41 is a communication interface equipped with a wireless circuit, antenna, and other communication-related configurations in accordance with a predetermined communication standard, and communicates with devices connected to the global network GN in accordance with the predetermined communication standard. In this embodiment, the server communication unit 41 communicates with the camera unit 2 and the terminal device 3.

[0083] As described above, the server control unit 40 functions as a server communication control unit 401 and an information processing unit 402.

[0084] The server communication control unit 401 communicates with the camera unit 2 and the terminal device 3 via the server communication unit 41.

[0085] The information processing unit 402 performs various information processing based on the information received by the server communication control unit 401 from the camera unit 2 and the terminal device 3.

[0086] [1-1-4. Terminal Device Configuration] Next, we will describe the functional configuration of terminal device 3. The terminal device 3 comprises a terminal control unit 30, a terminal communication unit 31, and a touch panel 32. The touch panel 32 corresponds to an example of a display unit.

[0087] The terminal control unit 30 includes a terminal processor 300, which is a processor such as a CPU or MPU, and a terminal storage unit 310. The terminal control unit 30 controls each part of the terminal device 3 by having the terminal processor 300 read and execute control programs stored in the terminal storage unit 310. A food management application 311 is installed on the terminal device 3. The terminal processor 300 functions as an application execution unit 301 by reading and executing the food management application 311. The terminal processor 300 corresponds to an example of the computer described herein. The application execution unit 301 corresponds to an example of the display control unit described herein.

[0088] The terminal storage unit 310 has memory for storing programs executed by the terminal processor 300 and data processed by the terminal processor 300. The terminal storage unit 310 stores control programs executed by the terminal processor 300, food management applications 311, account IDs 4111, and various other data. The terminal storage unit 310 has a non-volatile storage area. Alternatively, the terminal storage unit 310 may also have a volatile storage area that constitutes the work area of ​​the terminal processor 300.

[0089] The terminal communication unit 31 is a communication interface equipped with a wireless circuit, antenna, and other components that conform to a predetermined communication standard, and communicates with devices connected to the global network GN in accordance with the predetermined communication standard. In this disclosure, wireless communication standards are given as examples of communication standards used by the terminal communication unit 31.

[0090] The touch panel 32 comprises a display panel such as a liquid crystal display panel, and a touch sensor that is superimposed on or integrated with the display panel.

[0091] As described above, the terminal control unit 30 functions as the application execution unit 301. The application execution unit 301 sends and receives various information to and from the food management server 4 via the terminal communication unit 31.

[0092] Furthermore, the application execution unit 301 displays a user interface related to the management of food in the refrigerator 1 on the touch panel 32. In the following description, this user interface will be referred to as the "application UI" and denoted as "320". By displaying the application UI 320 on the touch panel 32, the application execution unit 301 provides various information related to the food in the refrigerator 1 to the user P, and also receives various inputs related to the food in the refrigerator 1 from the user P.

[0093] [1-2. Operation] Next, we will explain the operation of each part of the food management system 1000.

[0094] [1-2-1. Installation of the camera unit] The camera unit 2 is separate from the refrigerator 1. Therefore, the camera unit 2 is installed on the top surface 10B of the refrigerator 1 by the user P. As described above, the position regulating member 205 regulates the position of the camera unit 2 in the front-to-back direction of the refrigerator 1, so when installing the camera unit 2, the user P adjusts its position in the left-to-right direction of the refrigerator 1. The following are several examples of methods for adjusting the position of the camera unit 2 in the left-right direction of the refrigerator 1.

[0095] [1-2-1-1. First position adjustment]

[0096] In the first position adjustment, user P uses the app UI320. In the first position adjustment, the application execution unit 301 displays the application UI 320 of the adjustment screen GM1 on the touch panel 32.

[0097] Figure 11 shows an example of the adjustment screen GM1. The adjustment screen GM1 is provided with a shooting result display area HA1. The shooting result display area HA1 is a rectangular area that displays the shooting results of either the refrigerator compartment camera 203 or the drawer camera 204. The shooting result display area HA1 has a line L1 extending vertically, located approximately in the center in the horizontal direction in the figure.

[0098] The adjustment screen GM1 includes adjustment method information J1. The adjustment method information J1 prompts the user to adjust the position so that the horizontal center of the refrigerator 1 as seen in the image result display area HA1 aligns with the line L1 provided in the image result display area HA1.

[0099] User P installs the camera unit 2 in the refrigerator 1 according to the adjustment method indicated in the adjustment method information J1. This allows the camera unit 2 to ensure that both the refrigerator compartment camera 203 and the drawer camera 204 include food items being put into and taken out of the refrigerator 1 in their shooting range. In addition, the camera unit 2's distance measuring sensors can detect the opening and closing of the corresponding door 11C or drawer 16A.

[0100] In the first position adjustment, the camera unit 2 transmits the image captured by either the refrigerator camera 203 or the drawer camera 204 to the terminal device 3. During the first position adjustment, the camera unit 2 and the terminal device 3 communicate via the food management server 4 or directly via short-range wireless communication.

[0101] The camera unit communication control unit 221 of camera unit 2 transmits the shooting result from either the refrigerator compartment camera 203 or the drawer camera 204 to the terminal device 3. The application execution unit 301 then displays the received shooting result in the shooting result display area HA1.

[0102] [1-2-1-2. Second position adjustment] In the second position adjustment, user P uses the central position recording sheet CY.

[0103] Figure 12 shows an example of a central position recording sheet CY. The center position recording sheet CY is a sheet of paper on which the center mark MR2, which indicates the center position of the refrigerator 1 in the left-right direction, is recorded. In addition to the center mark MR2, the center position recording sheet CY also has the left edge mark MR1, which indicates the left edge position of the refrigerator 1 in the left-right direction, and the right edge mark MR3, which indicates the right edge position of the refrigerator 1 in the left-right direction.

[0104] The central position recording sheet CY is included with refrigerator 1 and delivered to user P. The camera unit 2 may or may not be included with the central position recording sheet CY and refrigerator 1. When included with refrigerator 1, the length from the leftmost mark MR1 to the rightmost mark MR3 of the central position recording sheet CY corresponds to the width of refrigerator 1.

[0105] In the second position adjustment, user P uses the center position recording sheet CY to determine the location of the center of refrigerator 1 in the left-right direction. For example, user P attaches the center position recording sheet CY to refrigerator 1 so that the left edge mark MR1 is at the left edge of refrigerator 1 and the right edge mark MR3 is at the right edge of refrigerator 1, and uses the center mark MR2 as a guide to determine the location of the center of refrigerator 1 in the left-right direction. Then, user P adjusts the position of camera unit 2 so that the mark 207 on the position regulating member 205 of camera unit 2 corresponds to the determined center position.

[0106] The second position adjustment enables the camera unit 2 to perform various detections, similar to the first position adjustment.

[0107] [1-2-1-3. Third position adjustment] In the third position adjustment, a mark indicating the center position of the refrigerator 1 in the left-right direction is placed at a predetermined position on the refrigerator 1. In the third position adjustment, the mark 207 on the position regulating member 205 and the mark placed on the refrigerator 1 are used.

[0108] In the third position adjustment, user P adjusts the position of the camera unit 2 so that the position of the mark attached to the refrigerator 1 coincides with the position of the mark 207 on the position regulating member 205 in the left-right direction of the refrigerator 1.

[0109] The third position adjustment enables the camera unit 2 to perform various detections, similar to the first position adjustment.

[0110] Next, we will describe the operation of the food management system 1000 after the camera unit 2 has been installed in the refrigerator 1. [1-2-2. Actions related to recording] First, let's explain the operations related to recording. Figure 13 is a flowchart FA showing the operation of camera unit 2.

[0111] In the operation shown in Figure 13, the sensor unit 22 outputs the detected values ​​of each sensor to the camera unit control unit 20. Then, in the operation shown in Figure 13, the detection value processing unit 222 performs processing based on the various detected values ​​received from the sensor unit 22.

[0112] The detection value processing unit 222 determines whether or not there is a person around the refrigerator 1 based on the detection value of the human presence sensor 212 (step SA1).

[0113] If the detection value processing unit 222 determines that a person is present around the refrigerator 1 (step SA1: YES), it outputs presence information indicating the presence of a person to the shooting control unit 223 (step SA2).

[0114] When the imaging control unit 223 receives presence information from the detection value processing unit 222 indicating the presence of a person, it causes the refrigerator camera 203 and the drawer camera 204 to start taking pictures (step SA3).

[0115] Next, the detection value processing unit 222 determines whether the left door 11A, the right door 11B, or the drawers 14A and 15A are in the open state based on the detection values ​​of the various sensors output from the sensor unit 22 (step SA4).

[0116] If the detection value processing unit 222 determines that none of the left door 11A, right door 11B, drawers 14A, and 15A are open (step SA4: NO), it performs the process in step SA16.

[0117] If the detection value processing unit 222 determines that either the left door 11A, the right door 11B, or the drawers 14A or 15A are open (step SA4: YES), it then determines whether the door 11C is open or whether either the drawers 14A or 15A are open (step SA5).

[0118] If the detection value processing unit 222 determines that the door 11C is in an open state (step SA5: door), it outputs door open state information to the recording control unit 224 (step SA6).

[0119] Next, when the recording control unit 224 receives door open status information from the detection value processing unit 222, it starts recording the results of the camera 203 for the refrigerator compartment (step SA7).

[0120] Next, the detection value processing unit 222 determines whether or not the door 11C is in the closed state (step SA8).

[0121] If the detection value processing unit 222 determines that the door 11C is in a closed state (step SA8: YES), it outputs door closed state information to the recording control unit 224 (step SA9).

[0122] Next, when the recording control unit 224 receives door closed status information from the detection value processing unit 222, it terminates recording of the results captured by the refrigerator camera 203 (step SA10). As a result, one recording file 231 of the refrigerator camera 203 is generated in the camera unit storage unit 230.

[0123] Returning to the explanation of step SA5, if the detection value processing unit 222 determines that either drawer 14A or 15A is in the open state (step SA5: drawer), it outputs drawer open state information to the recording control unit 224 (step SA11).

[0124] Next, when the recording control unit 224 receives the pull-out closed state information from the detection value processing unit 222, it starts recording the shooting results of the pull-out camera 204 (step SA12).

[0125] Next, the detection value processing unit 222 determines whether either of the open drawers 14A or 15A has been closed (step SA13).

[0126] If the detection value processing unit 222 determines that either of the open drawers 14A or 15A has closed (step SA13: YES), it outputs drawer closed status information to the recording control unit 224 (step SA14).

[0127] Next, when the recording control unit 224 receives the pull-out closed state information from the detection value processing unit 222, it terminates recording of the shooting results of the pull-out camera 204 (step SA15). As a result, one recording file 231 of the pull-out camera 204 is generated in the camera unit storage unit 230.

[0128] The detection value processing unit 222 determines whether or not there is a person around the refrigerator 1 (step SA16).

[0129] If the detection processing unit 222 determines that a person is present around the refrigerator 1 (step SA16: YES), it will perform the processing from step SA4 onwards again.

[0130] On the other hand, if the detection value processing unit 222 determines that there are no people around the refrigerator 1 (step SA16: NO), it outputs presence / absence information indicating the absence of people to the shooting control unit 223 (step SA17).

[0131] When the imaging control unit 223 receives presence / absence information from the detection value processing unit 222 indicating that no person is present, it causes the refrigerator camera 203 and the drawer camera 204 to stop imaging (step SA18).

[0132] [1-2-3. Actions related to food detection] Next, we will explain the process involved in food detection. Figure 14 is a flowchart showing the operation of the food management system 1000. In Figure 14, flowchart FB shows the operation of camera unit 2, and flowchart FC shows the operation of food management server 4.

[0133] The food detection unit 225 of the camera unit 2 determines whether or not the video recording file 231 has been stored in the camera unit storage unit 230 (step SB1).

[0134] If the food detection unit 225 determines that a recording file 231 has been stored in the camera unit storage unit 230 (step SB1: YES), it determines whether the stored recording file 231 is from the refrigerator camera 203 or the drawer camera 204 (step SB2). The determination in step SB2 is made based on the information output by the detection value processing unit 222.

[0135] If the food detection unit 225 determines that the recording file 231 is from the refrigerator camera 203 (step SB2: refrigerator camera), it detects food based on the stored recording file 231 (step SB3).

[0136] In step SB3, for example, the food detection unit 225 detects food from the recording file 231 of the refrigerator camera 203 based on characteristic quantities such as shape and color. The data necessary for food detection is stored in a memory area accessible to the food detection unit 225.

[0137] The food detection unit 225 determines whether or not food was detected from the recorded file 231 of the refrigerator camera 203 (step SB4).

[0138] If the food detection unit 225 determines that it could not detect food from the recorded file 231 of the refrigerator camera 203 (step SB4: NO), it proceeds to step SB10.

[0139] Meanwhile, if the food detection unit 225 can detect food from the recorded file 231 of the refrigerator camera 203 (step SB4: YES), it performs an in / out determination process (step SB5).

[0140] The in / out determination process determines whether the food detected in step SB3 was placed into refrigerator 1 or removed from refrigerator 1. In the in / out determination process, the in / out determination image HG is superimposed on the video image shown by the recording file 231 from the refrigerator camera 203.

[0141] Figure 15 is a diagram illustrating the inbound / outbound determination image HG. The in / out detection image HG shown in Figure 15 illustrates the case where it is superimposed on a video of user P placing an egg carton into refrigerator 1.

[0142] The inbound / outbound detection image HG has a boundary line L2. Boundary line L2 has boundary lines L21, L22, L23, L24, and L25. Boundary line L21 is a line extending to the right from the left edge of the entry / exit determination image HG in the upper part of the figure. Boundary line L22 is a line connecting the right edge of boundary line L21 and the left edge of boundary line L23. Boundary line L23 is a line extending to the right from the position of the right edge of boundary line L21 in the left-right direction, approximately in the vertical center of the entry / exit determination image HG. Boundary line L24 is a line connecting the right edge of boundary line L23 and the left edge of boundary line L24. Boundary line L25 is a line extending to the right from the position of the right edge of boundary line L23 in the left-right direction to the right edge of the entry / exit determination image HG in the upper part of the entry / exit determination image HG.

[0143] The image area of ​​the inbound / outbound judgment image HG is divided by the boundary line L2 into a concave region HG1, which is a concave region, and a convex region HG2, which is a convex region.

[0144] The food detection unit 225 performs image analysis on the moving image overlaid with the in / out determination image HG to determine whether the food detected in step SB3 has moved from the convex region HG2 to the concave region HG1, or from the concave region HG1 to the convex region HG2. If the food detection unit 225 determines that the food detected in step SB3 has moved from the convex region HG2 to the concave region HG1, it determines that the food detected in step SB3 has been placed in the refrigerator 1. On the other hand, if the food detection unit 225 determines that the food detected in step SB3 has moved from the concave region HG1 to the convex region HG2, it determines that the food detected in step SB3 has been placed out of the refrigerator 1.

[0145] Returning to the explanation of flowchart FB in Figure 14, the food detection unit 225 performs the division area number identification process (step SB6).

[0146] The division area number identification process involves identifying the division area BA corresponding to the storage area where the food detected in step SB3 was stored, and then identifying the division area number 4118 assigned to the identified division area BA.

[0147] For example, the food detection unit 225 identifies the storage area of ​​the refrigerator compartment 11 where the food detected in step SB3 was placed, through image analysis. Alternatively, for example, marks identifying the storage area are placed at the ends of each shelf 112 and inside each door pocket 111, and the food detection unit 225 identifies the storage area based on the positional relationship between these marks and the detected food in the moving image. Next, the food detection unit 225 determines which divided area BA of the refrigerator compartment image RG corresponds to the identified storage area and identifies the divided area number 4118 assigned to the determined divided area BA. The correspondence between the storage area, divided area BA, and divided area number 4118 is stored as data in the camera unit storage unit 230.

[0148] When the food detection unit 225 performs the division area number identification process, it extracts images of each food item detected in step SB3 from the recording file 231 of the refrigerator camera 203 (step SB7). Hereinafter, the images of food extracted from the camera 213's shooting results are referred to as "captured food images" and are denoted with the code "FG". The captured food image FG corresponds to an example of a food image in this disclosure.

[0149] Next, the food detection unit 225 generates the first update information (step SB8).

[0150] The first update information generated in step SB8 includes the camera unit ID 4112 of the camera unit storage unit 230. Furthermore, the first update information generated in step SB8 includes date and time information indicating the date and time when the recording file 231 from the refrigerator camera 203 was created. Furthermore, the first update information generated in step SB8 includes, for each food item detected in step SB3, food image data 4116 showing the photographed food image FG extracted in step SB7, inbound / outbound determination result information showing the determination result of the inbound / outbound determination process, and the divided area number 4118 identified in the divided area number identification process.

[0151] Next, the camera unit communication control unit 221 transmits the first update information generated by the food detection unit 225 to the food management server 4 (step SB9).

[0152] The food detection unit 225 erases the recorded file 231 from the camera unit storage unit 230 (step SB10).

[0153] Returning to the explanation of step SB2, if the food detection unit 225 determines that it is the recording file 231 of the drawer camera 204 (step SB2: drawer camera), it determines whether at least one of the upper case JCA and the lower case GCA shown in the video of the recording file 231 of the drawer camera 204 has been drawn out to a predetermined threshold or higher (step SB11).

[0154] For example, the food detection unit 225 distinguishes case CA in the video based on feature quantities such as shape and color. The food detection unit 225 then determines, by analyzing the image of the recorded file 231, whether the upper case JCA and the lower case GCA have been pulled out of the refrigerator 1 above a predetermined threshold. Furthermore, for example, the upper case JCA and the lower case GCA each have marks at predetermined positions to identify case CA, and the food detection unit 225 distinguishes case CA in the video by reading these marks from the video. The food detection unit 225 then calculates the distance traveled by these marks in the video to determine whether or not the food has been pulled out of the refrigerator 1 by a predetermined threshold.

[0155] If the food detection unit 225 determines that neither the upper case JCA nor the lower case GCA has been drawn out above a predetermined threshold (step SB11: NO), it performs the process in step SB18.

[0156] On the other hand, if the food detection unit 225 determines that at least one of the upper case JCA and the lower case GCA has been extracted to a predetermined threshold or higher (step SB11: YES), it extracts the case image CG from the recording file 231 (step SB12).

[0157] In step SB12, if the food detection unit 225 extracts the upper case image JCG from the recording file 231 if only the upper case JCA is extracted above a predetermined threshold. In step SB12, if the food detection unit 225 extracts the lower case image GCG from the recording file 231 if only the lower case GCA is extracted above a predetermined threshold. In step SB12, if the upper case JCA and lower case GCA are extracted to a predetermined threshold or higher, the food detection unit 225 extracts the upper case image JCG and the lower case image GCG from the recording file 231.

[0158] Next, the food detection unit 225 detects food from the cut-out case image CG (step SB13).

[0159] Next, the food detection unit 225 determines whether or not food was detected from the case image CG (step SB14).

[0160] If the food detection unit 225 determines that it cannot detect food (step SB14: NO), it performs the process in step SB16.

[0161] On the other hand, if the food detection unit 225 determines that it has detected food (step SB14: YES), it extracts a photographed food image FG from the case image CG for each detected food item (step SB15).

[0162] Next, the food detection unit 225 generates second update information (step SB16).

[0163] If only the upper case image JCG is extracted, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage unit 230, the upper case image data 4123 showing the extracted upper case image JCG, and the food image data 4116 showing the photographed food image FG extracted from the upper case image JCG. However, if a negative determination is made in step SB14, the food image data 4116 is not included in this second update information.

[0164] Furthermore, if only the lower case image GCG is extracted, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage unit 230, the lower case image data 4125 showing the extracted lower case image GCG, and the food image data 4116 showing the photographed food image FG extracted from the lower case image GCG. However, if a negative determination is made in step SB14, the food image data 4116 is not included in this second update information.

[0165] Furthermore, if the upper case image JCG and the lower case image GCG are extracted, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage unit 230, the upper case image data group, and the lower case image data group. The upper case image data group includes upper case image data 4123 showing the extracted upper case image JCG, and food image data 4116 showing the photographed food image FG extracted from the upper case image JCG. The lower case image data group includes lower case image data 4125 showing the extracted lower case image GCG, and food image data 4116 showing the photographed food image FG extracted from the lower case image GCG. Note that if a negative determination is made in step SB14, this second update information does not include food image data 4116.

[0166] Next, the camera unit communication control unit 221 transmits the second update information generated by the food detection unit 225 to the food management server 4 (step SB17).

[0167] Next, the food detection unit 225 deletes the recorded file 231 from the camera unit storage unit 230 (step SB18).

[0168] As shown in flowchart FC, the server communication control unit 401 of the food management server 4 determines whether or not it has received either the first update information or the second update information (step SC1).

[0169] If the server communication control unit 401 determines that it has received update information (step SC1: YES), the information processing unit 402 updates the database based on the update information received by the server communication control unit 401 (step SC2).

[0170] Step SC2 will be described in detail. If the update information received by the server communication control unit 401 is the first update information, the information processing unit 402 identifies the refrigerator food management record RR having camera unit ID 4112 included in the first update information. Then, the information processing unit 402 updates the in / out information 4115 of the identified refrigerator food management record RR to information indicating "present".

[0171] Next, if the inbound / outbound determination result information included in the first update information indicates inbound, the information processing unit 402 generates a subrecord, associates the generated subrecord with blank deletion information 4131 and deletion date and time information 4132, and stores it in the identified refrigerated food management record RR. The subrecord generated here includes food image data 4116 included in the first update information, storage date and time information 4117 indicating the same date and time as the date and time information included in the first update information, a divided area number 4118 included in the first update information, blank memo input presence / absence information 4119, and blank memo information 4130.

[0172] Furthermore, if the inbound / outbound determination result information included in the first update information indicates outbound, the information processing unit 402 identifies a subrecord from the refrigerator food management record RR that contains food image data 4116 indicating a photographed food image FG with a high degree of match to the food image data 4116 included in the first update information. This identification is performed for each food image data 4116 included in the first update information. The information processing unit 402 then updates the deletion information 4131 associated with the identified subrecord to information indicating that it is subject to deletion. The information processing unit 402 also updates the deletion date and time information 4132 associated with the identified subrecord to information indicating the current date and time as the deletion date and time.

[0173] Furthermore, step SC2 will be described in detail. If the update information received by the server communication control unit 401 is the second update information, the information processing unit 402 identifies the retrieved food management record HR having camera unit ID 4112 included in the second update information.

[0174] If the second update information is obtained when only the upper case image JCG is extracted, the information processing unit 402 updates the upper case image data 4123 and food image data 4116 contained in the upper case information 4122 of the identified retrieved food management record HR with the data contained in the second update information. If the second update information is obtained when only the lower case image GCG is extracted, the information processing unit 402 updates the lower case image data 4125 and food image data 4116 contained in the lower case information 4124 of the identified retrieved food management record HR with the data contained in the second update information. If the upper case image JCG and lower case image GCG are extracted, and this is the second update information, the information processing unit 402 appropriately updates the upper case image data 4123 and food image data 4116 contained in the upper case information 4122 of the identified retrieved food management record HR with the data contained in the upper case image data group. The information processing unit 402 also updates the lower case image data 4125 and food image data 4116 contained in the lower case information 4124 of the identified retrieved food management record HR with the data contained in the lower case image data group.

[0175] Furthermore, the information processing unit 402 updates the update date and time information 4121 of the identified retrieved food management record HR to information indicating the date and time when the retrieved food management record HR was updated.

[0176] [1-2-4. Actions related to the app UI] Next, we will explain the operations related to the application UI320. In this embodiment, the application UI320 displays a screen related to the refrigerator compartment 11 and a screen related to the drawer-type storage compartment.

[0177] [1-2-4-1. Screens related to the refrigerator compartment]

[0178] The screens related to the refrigerator compartment 11 include the refrigerator compartment home screen GM2, the refrigerator compartment list screen GM3, and the food information addition list screen GM4.

[0179] Figure 16 shows an example of the GM2 refrigerator compartment home screen.

[0180] The refrigerator compartment home screen GM2 has a food image display area HA2. The food image display area HA2 is the area for displaying the captured food images FG. The food image display area HA2 displays the refrigerator interior image RG, and displays the captured food images FG superimposed on this refrigerator interior image RG. Each of the captured food images FG is displayed superimposed on the refrigerator interior image RG in the segmented area BA indicated by segmented area number 4118 corresponding to the food image data 4116.

[0181] The food image display area HA2 in Figure 16 displays the refrigerator interior image RG shown in Figure 9. For example, in the refrigerator food management record RR, suppose that the food image data 4116 representing the photographed food image FG1 in Figure 16 is associated with the division area number 4118, which is "4". In this case, the photographed food image FG1 is displayed superimposed on the division area BA to which the division area number 4118, "4", is assigned.

[0182] The default food image display area HA2 displays the refrigerator compartment interior image RG with the left door pocket image RG1 and the right door pocket image RG3 partially cut off. The food image display area HA2 accepts scrolling operations from the user P in the left-right direction of the image. The food image display area HA2 changes the display range of the refrigerator compartment interior image RG according to the scroll amount and direction of the received scrolling operation.

[0183] For example, suppose the scrolling direction is to the right, and the scroll amount is such that the left door pocket image RG1 is displayed to its left edge. In this case, the food image display area HA2 will display the refrigerator interior image RG, with the right door pocket image RG3 completely cut off. The captured food image FG and the memo input icon MIC will move in accordance with the change in the display range of the refrigerator interior image RG, in other words, they will move along with the movement of the superimposed divided area BA. The memo input icon MIC will be explained later.

[0184] The food image display area HA2 displays multiple photographed food images FG in a single divided area BA, arranging them side-by-side horizontally. The food image display area HA2 can simultaneously display up to 5 photographed food images FG in a single divided area BA. If there are 6 or more photographed food images FG to be displayed in a single divided area BA, the food image display area HA2 accepts a scroll operation from the user P. When a scroll operation is accepted, the divided area BA displays the 6th and subsequent photographed food images FG. If there are 6 or more photographed food images FG to be displayed in a single divided area BA, and a horizontal scroll operation is accepted, the food image display area HA2 displays the 6th and subsequent photographed food images FG according to the scroll amount, while maintaining the maximum number of images that can be displayed side-by-side. In this case, the food image display area HA2 will hide the first five food images FG that were being displayed, or display the first five food images FG after the sixth and subsequent food images FG. The memo input icon MIC will move in accordance with the movement of the associated food images FG. That is, when the associated food images FG are hidden by scrolling, the memo input icon MIC will also be hidden.

[0185] Each of the photographed food images FG displayed by the food image display area HA2 accepts memo input. The memo is information that user P wants to leave for the food in refrigerator 1.

[0186] Figure 17 shows the procedure for entering notes.

[0187] User P performs a predetermined touch operation, such as a long press, on the food image FG on the refrigerator home screen GM2 in which they wish to enter a memo. When the application execution unit 301 receives the predetermined touch operation on the food image FG, it displays a memo editing image MHG associated with the food image FG on which the predetermined touch operation was performed.

[0188] The memo editing image MHG has a memo input area MNA, a memo save button B1, a memo delete button B2, and a food delete button B3.

[0189] The memo input area MNA is an area for entering memos. When the memo input area MNA is touched, the application execution unit 301 displays a software keyboard on the touch panel 32. The application execution unit 301 then inputs memos corresponding to the operation on the software keyboard into the memo input area MNA.

[0190] The memo save button B1 is a software button for saving memos entered in the memo input area MNA. The memo save button B1 cannot be touched if no memo has been entered in the memo input area MNA. When the memo save button B1 is touched, the application execution unit 301 displays the photographed food image FG with the memo entered in a different display mode than the photographed food image FG without a memo entered on the screen related to the refrigerator compartment 11. In this embodiment, the application execution unit 301 displays the frame of the photographed food image FG with the memo entered in a different color, and also displays the memo input icon MIC, which is an icon indicating that a memo has been entered, associated with the photographed food image FG with the memo entered. The memo input icon MIC is touchable. When the memo input icon MIC is touched, the entered memo is displayed in the application UI 320. The memo input icon MIC corresponds to an example of a memo input image in this disclosure.

[0191] The memo erase button B2 is used to erase the entered memo. When the memo erase button B2 is touched, the memo entered in the photographed food image FG is erased.

[0192] The food deletion button B3 is used to delete the captured food image FG from the food image display area HA2.

[0193] Returning to the description of the refrigerator compartment home screen GM2 in Figure 16, the food image display area HA2 has a list display button B4. The list display button B4 is a touch-operable software button.

[0194] When the list display button B4 is touched, the app UI320 transitions to the refrigerator compartment list screen GM3.

[0195] The refrigerator compartment list screen GM3 includes the storage list screen GM31, the new item addition list screen GM32, and the deletion target list screen GM33.

[0196] This explains the containment list screen GM31. Figure 18 shows the GM31 storage list screen.

[0197] The storage list screen GM31 has a storage food information display area HA3. The storage food information display area HA3 displays a list of first food information J21 for each food item stored in the refrigerator compartment 11. The first food information J21 includes a photographed food image FG, storage date and time information 4117, and divided area information J3. The divided area information J3 is a refrigerator compartment interior image RG in which divided areas BA are indicated by color.

[0198] If a memo has been entered, the food image FG of the first food information J21 will be displayed with a different colored border and the memo input icon MIC will be displayed in association with it. Furthermore, the food information display area HA3 allows for vertical scrolling of the diagram. By scrolling in the food information display area HA3, user P can display the first food information J21 that cannot be fully displayed in the food information display area HA3.

[0199] The storage list screen GM31 has a first switch button B5 and a second switch button B6.

[0200] The first switching button B5 is a software button used to switch the sorting order of the first food information J21 displayed in the food information display area HA3 to the following sorting order. In other words, when the first toggle button B5 is touched, the stored food information display area HA3 displays the first food information J21 that has a photographed food image FG with a memo entered, prioritizing it over the first food information J21 that has a photographed food image FG without a memo entered. Displaying it higher means displaying it in a way that reduces the amount of scrolling required when viewing it. Furthermore, if there are multiple first food information J21 with photographed food images FG with memos entered, the stored food information J2 with the oldest storage date and time indicated by the storage date and time information 4117 will be displayed higher. Furthermore, if there are multiple first food information J21 with photographed food images FG without memos entered, the stored food information J21 with the oldest storage date and time indicated by the storage date and time information 4117 will be displayed higher. Note that the storage list screen GM31 on the left in Figure 18 shows the storage list screen GM31 when the first switch button B5 is touched.

[0201] The second switching button B6 is a software button used to switch the sorting order of the first food information J21 displayed in the food information display area HA3 to the following sorting order. In other words, when the second toggle button B6 is touched, the food information display area HA3 displays the first food information J21 in order for each divided area BA. The order of the items in each divided area BA may be an order predetermined by the user P. In one divided area BA, the food information display area HA3 displays the first food information J21 in the same order as when the first toggle button B5 is touched. Note that the storage list screen GM31 on the right in Figure 18 shows the storage list screen GM31 when the second switch button B6 is touched.

[0202] The storage list screen GM31 has a storage list display button B7, a new addition list display button B8, and a deletion target list display button B9.

[0203] The storage list display button B7 is a software button used to switch the screen displayed by the application UI320 to the storage list screen GM31. The "Add New" list display button B8 is a software button used to switch the screen displayed by the app UI320 to the "Add New" list screen GM32. The delete target list display button B9 is a software button used to switch the screen displayed by the application UI320 to the delete target list screen GM33.

[0204] Figure 19 shows an example of the new addition list screen GM32. The newly added list screen GM32 has a newly added food information display area HA4. The newly added food information display area HA4 displays a list of first food information J21 for foods newly placed in the refrigerator compartment 11. Food newly placed in the refrigerator compartment 11 means food placed within a predetermined period (for example, 3 days) from the current date and time. The newly added food information display area HA4 displays the first food information J21 for foods that are newly placed, prioritizing those with the most recent placement date and time indicated by the placement date and time information 4117.

[0205] Each of the first food information J21 displayed in the newly added food information display area HA4 accepts a predetermined touch operation on the photographed food image FG. On the newly added list screen GM32, the first edited image HNG1 is displayed in association with the photographed food image FG that has been touched.

[0206] The first edited image HNG1 has a food delete button B10, an area modification button B11, and a back button B12. The food deletion button B10 is a software button that, when touched, targets the food item indicated by the photographed food image FG associated with the first edited image HNG1 for deletion from the refrigerator home screen GM2. The area correction button B11 is a software button that, when touched, corrects the divided area BA indicated by the divided area information J3. The back button B12 is a software button used to stop the display of the first edited image HNG1.

[0207] The newly added food information display area HA4 has a food information add button B13 at the end of the list of first food information J21. The food information add button B13 is a software button for manually adding new first food information J21. Refer to Figure 20 to explain the additional steps for the first food information J21.

[0208] Figure 20 is a diagram illustrating the additional steps for the first food information J21.

[0209] When the food information addition button B13 is touched, the food information addition list screen GM4 is displayed in the application UI320. The food information addition list screen GM4 has multiple software buttons called add buttons B14. The food information addition list screen GM4 also has a blank image BG and a food name input area SNA where the food name is entered. When the blank image BG is touched, the blank image BG becomes an image showing the food name entered in the food name input area SNA.

[0210] When the add button B14 is touched on the food information addition list screen GM4, the refrigerator interior image RG is displayed in the app UI320. The displayed refrigerator interior image RG is an image in which each of the divided areas BA is assigned a divided area number 4118, and each of the divided areas BA can be selected by touch operation. When a divided area BA is selected in this refrigerator interior image RG, the next first food information J21 is added in the app UI320.

[0211] In other words, the added first food information J21 consists of an image of the food corresponding to the touched add button B14, storage date and time information 4117 indicating the date and time the first food information J21 was added, and divided area information J3 indicating the selected divided area BA. Note that the image of the food shown in the added first food information J21 may be an icon. In this embodiment, it is assumed that the image superimposed on the refrigerator interior image RG of the refrigerator home screen GM2 is a shadow food image FG, but when the first food information J21 is added, an icon is displayed on the refrigerator interior image RG along with the captured food image FG.

[0212] Furthermore, the procedure for adding the first food information J21 may include a step in which user P inputs the storage date and time. In this case, the added first food information J21 will include storage date and time information 4117 indicating the storage date and time entered by user P.

[0213] Returning to the explanation of Figure 19, the new addition list screen GM32 has a storage list display button B7, a new addition list display button B8, and a deletion target list display button B9.

[0214] Figure 21 shows an example of the GM33 deletion target list screen.

[0215] The deletion target list screen GM33 has a deletion target food information display area HA5. The deletion target food information display area HA5 displays a list of secondary food information J22 for each food item to be deleted. The foods to be deleted refer to foods to be deleted from the refrigerator home screen GM2, and foods that have been determined to have been taken out of the refrigerator 11. The secondary food information J22 includes a photographed food image FG, deletion date and time information 4132, and divided area information J3. The deletion target list screen GM33 displays the secondary food information J22 in descending order of deletion date and time.

[0216] Each of the second food information J22 displayed in the deletion target food information display area HA5 is determined by the captured food image FG receiving a predetermined touch operation. On the deletion target list screen GM33, the second edited image HNG2 is displayed in association with the captured food image FG that has received the predetermined touch operation.

[0217] The second edited image HNG2 has a delete undo button B15 and a back button B16. The delete / undo button B15 is a software button that sets the photographed food image FG, to which the second edited image HNG2 is associated, as the image displayed on the refrigerator home screen GM2. The back button B16 is a software button used to stop the display of the second edited image, HNG2.

[0218] [1-2-4-2. Operation of the Food Management System] Next, we will explain the operation of the food management system 1000 when displaying a screen related to the refrigerator compartment 11.

[0219] Figure 22 is a flowchart showing the operation of the food management system 1000. In Figure 22, flowchart FD shows the operation of terminal device 3, and flowchart FE shows the operation of food management server 4.

[0220] The application execution unit 301 of the terminal device 3 determines whether or not to request the refrigerator food management record RR from the food management server 4 (step SD1).

[0221] For example, in step SD1, the application execution unit 301 makes a positive determination when it receives a touch operation in the application UI 320 that instructs the display of the refrigerator compartment home screen GM2. Furthermore, for example, if a touch operation is performed on the refrigerator home screen GM2 to instruct the display to be updated, the application execution unit 301 makes a positive determination in step SD1.

[0222] If the application execution unit 301 determines that it is requesting a refrigerator management record RR (step SD1: YES), it sends first record request information requesting the refrigerator management record RR to the food management server 4 (step SD2). The first record request information sent in step SD2 includes account ID 4111 stored in the terminal storage unit 310.

[0223] As shown in flowchart FF, the server communication control unit 401 of the food management server 4 receives the first record request information from the server communication unit 41 (step SE1).

[0224] Next, the information processing unit 402 identifies the refrigerator food management record RR containing the account ID 4111 included in the first record request information received by the server communication control unit 401 from the refrigerator food management database 411 (step SE2).

[0225] Then, the server communication control unit 401 transmits the refrigerator food management record RR identified by the information processing unit 402 to the terminal device 3 (step SE3). The information processing unit 402 also updates the in / out information 4115 of the transmitted refrigerator food management record RR to indicate "none".

[0226] Referring to the flowchart FD, when the application execution unit 301 of the terminal device 3 receives the refrigerator food management record RR from the food management server 4 (step SD3), it generates a screen related to the refrigerator 11 based on this refrigerator food management record RR (step SD4).

[0227] Step SD4 will be described in detail. The application execution unit 301 generates a refrigerator home screen GM2 that displays the refrigerator interior image RG, indicated by the refrigerator interior image data 4114 included in the refrigerator food management record RR, in the food image display area HA2. Furthermore, the application execution unit 301 generates a refrigerator home screen GM2 for each subrecord by superimposing the photographed food image FG indicated by the food management image data 4116 onto the divided area BA indicated by the divided area number 4118. Furthermore, if the memo input status information 4119 indicates that a memo exists, the application execution unit 301 generates a refrigerator home screen GM2 for each sub-record, displaying the memo input icon MIC associated with the photographed food image FG.

[0228] Furthermore, Step SD4 will be described in detail. The application execution unit 301 generates a first food information J21 for each food image data 4116. This first food information J21 includes the photographed food image FG indicated by the food image data 4116, the storage date and time information 4117 associated with this food image data 4116, and the divided area information J3 indicated by the divided area number 4118, which is divided area BA, shown in color. The application execution unit 301 then generates a storage list screen GM31 that displays a list of first food information J21 related to subrecords associated with deletion information 4131 that does not indicate the item to be deleted. The application execution unit 301 also generates a new addition list screen GM32 that displays a list of first food information J21 from the first food information J21 displayed on the storage list screen GM31, where the storage date and time indicated by the storage date and time information 4117 indicates a date and time within a predetermined period from the current time.

[0229] Furthermore, Step SD4 will be described in detail. The application execution unit 301 generates a second food information J22 for each food image data 4116. The second food information J22 includes the photographed food image FG indicated by the food image data 4116, the deletion date and time information 4132 associated with this food image data 4116, and the division area information J3 indicated by the division area number 4118, which is shown in color as division area BA. The application execution unit 301 then generates a deletion target list screen GM33 that displays a list of the generated second food information J22.

[0230] Returning to the explanation of flowchart FD, the application execution unit 301 displays a screen related to the refrigerator compartment 11 in response to a touch operation (step SD5). If the in / out information 4115 of the refrigerator compartment food management record RR received in step SD3 indicates "Yes", then when the refrigerator compartment home screen GM2 is displayed for the first time in step SD5, the application execution unit 301 displays a pop-up on the refrigerator compartment home screen GM2 indicating that there has been a new in / out.

[0231] Next, we will explain the operation of the food management system 1000 when it is operated on the screen related to the refrigerator compartment 11.

[0232] When the memo save button B1 is touched, the application execution unit 301 sends information to the food management server 4 to update the subrecord related to the photographed food image FG of the target. This information includes memo input status information 4119 indicating whether a memo exists, and memo information 4130 indicating the entered memo. The food management server 4 updates the subrecord appropriately based on the received information.

[0233] When the memo erase button B2 is touched, the application execution unit 301 sends information to the food management server 4 to update the subrecord related to the photographed food image FG being operated on. This information includes memo input status information 4119 indicating whether or not a memo has been entered. The food management server 4 updates the subrecord appropriately based on the received information.

[0234] When the food deletion buttons B3 and B10 are touched, the application execution unit 301 sends information to the food management server 4 to update the subrecord related to the captured food image FG of the target food. This information includes deletion information 4131 indicating the item to be deleted. The food management server 4 updates the subrecord appropriately based on the received information.

[0235] When the area correction button B11 is touched and the divided area BA is corrected, the application execution unit 301 sends information to the food management server 4 to update the subrecord related to the photographed food image FG being operated on. This information includes the divided area number 4118 of the corrected divided area BA. The food management server 4 updates the subrecord appropriately based on the received information.

[0236] When the application execution unit 301 adds the first food information J21 using the additional procedure shown in Figure 20, it sends information to the food management server 4 to add a subrecord related to the added first food information J21. The food management server 4 updates the subrecord appropriately based on the received information.

[0237] When the delete cancellation button B15 is touched, the application execution unit 301 sends information to the food management server 4 to update the subrecord related to the captured food image FG. This information includes a delete information 4131 indicating a blank space. The food management server 4 updates the subrecord appropriately based on the received information.

[0238] [1-2-4-3. Screen related to the drawer-type storage compartment] Next, we will explain the screen related to the pull-out storage compartment. In the explanation using Figures 23-25, the vegetable compartment 15 will be used as an example of a pull-out storage compartment. In this explanation, the database shown in Figure 10 is a database that stores various information related to the food in the vegetable compartment 15.

[0239] The screens related to the vegetable compartment 15 include the vegetable compartment home screen GM5 and the food image list screen GM6.

[0240] Figure 23 shows an example of the GM5 vegetable compartment home screen.

[0241] The vegetable compartment home screen GM5 has a case image display area HA6. The case image display area HA6 is an area for displaying the case image CG. The case image display area HA6 displays the upper case image JCG superimposed on the lower case image GCG. In the default case image display area HA6, the upper case image JCG is superimposed on the entire image area of the lower case image GCG, and only the upper case image JCG is displayed.

[0242] The case image display area HA6 accepts a scroll operation in the vertical direction of the figure. The case image display area HA6 changes the display range of the upper case image JCG according to the scroll amount and scroll direction of the accepted scroll operation. For example, when the scroll direction of the accepted scroll operation is upward, the case image display area HA6 displays the upper case image JCG with a part cut off. In the case image display area HA6, when the upper case image JCG moves upward, the lower case image GCG that was superimposed on the upper case image JCG will be displayed.

[0243] Each of the case images CG displayed by the case image display area HA6 accepts the input of a memo. The procedure for memo input is the same as that in the case of the refrigerator compartment 11, except that the image operated by the user P is the photographed food image FG or the case image CG. As shown in FIG. 23, the memo input icon MIC associated with the upper case image JCG moves following the movement of the upper case image JCG.

[0244] The vegetable compartment home screen GM5 has a list display button B17. The list display button B17 is a touch-operable software button.

[0245] When the list display button B17 is touch-operated, in the app UI320, a food image list screen GM6 is displayed.

[0246] Incidentally, the vegetable room home screen GM5 may have a switching button for switching the case image CG displayed in the case image display area HA6 to the upper case image JCG or the lower case image GCG. In the case of this configuration, the app execution unit 301 switches the case image CG to be displayed to the upper case image JCG or the lower case image GCG according to the operation on this switching button.

[0247] FIG. 24 is a diagram showing an example of the food image list screen GM6.

[0248] The food image list screen GM6 has update date and time information 4121. The food image list screen GM6 in FIG. 24 has update date and time information 4121 indicating an update date and time of "12 / 1 10:05".

[0249] The food image list screen GM6 displays a list of photographed food images FG for each case. The photographed food image FG displayed in association with the item of the upper case JCA is an image cut out from the upper case image JCG. The photographed food image FG displayed in association with the item of the lower case GCA is an image cut out from the lower case image GCG.

[0250] [1-2-4-5. Operation of the food management system] Next, the operation of the food management system 1000 when displaying the screen related to the vegetable room 15 will be described.

[0251] FIG. 25 is a flowchart showing the operation of the food management system 1000. In FIG. 11, the flowchart FF shows the operation of the terminal device 3, and the flowchart FJ shows the operation of the food management server 4.

[0252] The app execution unit 301 of the terminal device 3 determines whether to request the withdrawal management record HR from the food management server 4 (step SF1).

[0253] For example, in step SF1, the application execution unit 301 makes a positive determination when it receives a touch operation in the application UI 320 that instructs the display of the vegetable compartment home screen GM5. Furthermore, for example, if a touch operation is performed on the vegetable compartment home screen GM5 to instruct the display to be updated, the application execution unit 301 makes a positive determination in step SF1.

[0254] If the application execution unit 301 determines that it is requesting withdrawal management record RR (step SF1: YES), it sends second record request information requesting withdrawal management record HR to the food management server 4 (step SF2). The second record request information sent in step SF2 includes account ID 4111 stored in the terminal storage unit 310.

[0255] Referring to flowchart FJ, the server communication control unit 401 of the food management server 4 receives the second record request information (step SJ1).

[0256] Next, the information processing unit 402 identifies the withdrawal management record HR containing account ID 4111, which is included in the second record request information received by the server communication control unit 401, from the withdrawal food management database 412 (step SJ2).

[0257] Then, the server communication control unit 401 transmits the retrieval management record HR identified by the information processing unit 402 to the terminal device 3 (step SJ3).

[0258] Referring to flowchart FJ, when the application execution unit 301 of terminal device 3 receives the food drawer management record HR from the food management server 4 (step SF3), it generates a screen related to the vegetable compartment 15 (step SF4).

[0259] Step SF4 will be described in detail. The application execution unit 301 generates a vegetable compartment home screen GM5 having a case image display area HA6 in which the upper case image JCG, indicated by the upper case image data 4123 included in the same record, is superimposed on the lower case image GCG indicated by the lower case image data 4125 included in the drawer food management record HR. Furthermore, if the memo input status information 4119 indicates that a memo exists, the application execution unit 301 generates a vegetable compartment home screen GM5 in which the memo input icon MIC is associated with the corresponding case image CG.

[0260] Furthermore, we will elaborate on step SF4. The application execution unit 301 generates a food image list screen GM6 for each case CA, which displays the captured food images FG indicated by the food image data 4116.

[0261] Returning to the explanation of flowchart FF, the application execution unit 301 displays a screen related to the vegetable compartment 15 in response to a touch operation (step SF5).

[0262] Next, we will explain the operation of the food management system 1000 when it is operated on the screen related to the vegetable compartment 15.

[0263] When the memo save button B1 is touched, the application execution unit 301 sends information to the food management server 4 to update the memo input status information 4119 and the memo information 4130. This information includes information indicating whether the operation target is the upper case image JCG or the lower case image GCG, memo input status information 4119 indicating whether a memo exists, and memo information 4130 indicating the entered memo. The food management server 4 appropriately updates the retrieved food management record HR based on the received information.

[0264] When the memo deletion button B2 is touched, the app execution unit 301 transmits information for updating the memo input presence / absence information 4119 and the memo information 4130 to the food management server 4. This information includes information indicating whether the memo deletion target is the upper case image JCG or the lower case image GCG, and the memo input presence / absence information 4119 indicating that there is no memo. The food management server 4 appropriately updates the withdrawal food management record HR based on the received information.

[0265] Note that the vegetable compartment 15 was exemplified as the drawer-type storage compartment described using FIGS. 23 to 25. However, the screen display shown in FIGS. 23 to 25 and the operation of the food management system 1000 may be similarly performed for the freezer compartment 14. In this case, the database shown in FIG. 10 indicates a database that stores various information related to the foods in the freezer compartment 14.

[0266] [1-3. Effects] As described above, the camera unit 2 includes a main body 201 installed on the upper surface 10B of the refrigerator 1, and a photographing member 202 extending from the main body 201 forward of the refrigerator 1 and having a camera 213 that photographs downward from the upper front of the refrigerator 1. The main body 201 is provided with a position regulating member 205 that abuts against the front edge 10A of the refrigerator 1 and regulates the position of the photographing member 202 in the front-rear direction of the refrigerator 1.

[0267] According to this, since the position of the photographing member 202 in the front-rear direction of the refrigerator 1 is regulated by the position regulating member 205, the camera 213 is defined at a position where it can photograph downward from the upper front of the refrigerator 1. Therefore, the foods being loaded into and unloaded from the refrigerator 1 can be accurately included in the photographing result of the camera 213, and the foods being loaded into and unloaded from the refrigerator 1 can be accurately photographed.

[0268] The position regulating member 205 is a position regulating piece that extends downward from the bottom surface 206 of the main body 201 in the installed state.

[0269] According to this, the position of the camera unit 2 can be restricted by utilizing the gap formed between the main body 10 and the door 11C. Therefore, since the camera unit 2 can be installed on an existing refrigerator 1, even an existing refrigerator 1 can accurately photograph food being put in and taken out of the refrigerator 1.

[0270] The position regulating member 205 is formed to be shorter than the length to the gasket provided between the refrigerator 1 and the door 11C of the refrigerator 1.

[0271] According to this, the position of the camera unit 2 can be restricted by utilizing the gap formed between the main box 10 and the door 11C without hindering the cooling effect of the refrigerator compartment 11. Therefore, food being put into and taken out of the refrigerator 1 can be photographed with high accuracy without hindering the cooling effect of the refrigerator compartment 11.

[0272] The position regulating member 205 extends in the width direction of the main body 201.

[0273] This allows for a larger contact area with the main box 10, thus stabilizing the installation position of the camera unit 2. Therefore, food being put into and taken out of the refrigerator 1 can be photographed with greater accuracy.

[0274] The position regulating member 205 has marks 207 for adjusting the position of the camera unit 2 in the width direction of the refrigerator 1.

[0275] According to this, by using Mark 207 as a guide, user P can easily install camera unit 2 on the left and right sides of refrigerator 1. Therefore, user P can easily install camera unit 2 so that food being put in and taken out of refrigerator 1 can be photographed with high accuracy.

[0276] The position regulating member 205 has a mark 207 in the center of the width direction of the main body 201.

[0277] According to this, by using Mark 207 as a guide, user P can easily install camera unit 2 in the center of refrigerator 1 in the left-right direction. Therefore, user P can easily install camera unit 2 to capture more accurate images of food being put into and taken out of refrigerator 1.

[0278] As explained, the camera unit 2 includes a camera 213 that captures moving images of the refrigerator 1 from the front and above, a shooting control unit 223 that causes the camera 213 to start shooting when a human presence sensor 212, which detects whether or not there are people around the refrigerator 1, detects that there are people around the refrigerator 1, and a recording control unit 224 that starts recording the results of the camera 213's shooting at a predetermined timing after the shooting control unit 223 has caused the camera 213 to start shooting. Furthermore, the control method for the camera unit 2 is as follows: when the motion sensor 212, which detects whether or not there are people around the refrigerator 1, detects that there are people around the refrigerator 1, the camera 213 is instructed to start shooting, and at a predetermined timing after the camera 213 has started shooting, the recording of the results captured by the camera 213 is started.

[0279] According to the camera unit 2 and the control method for the camera unit 2, when the motion sensor 212 detects the presence of a person around the refrigerator 1, the camera 213 is instructed to start taking pictures. This increases the likelihood that food items being placed in or taken out of the refrigerator 1 before the door 11C or drawer 16A is opened can be included in the camera 213's images. Therefore, food items being placed in or taken out of the refrigerator 1 can be photographed with high accuracy.

[0280] The recording control unit 224 starts recording the results of the camera 213's shooting when the door 11C or drawer 16A of the refrigerator 1 is opened after the shooting control unit 223 has started shooting with the camera 213.

[0281] According to this, recording is started at a time when food is likely to be added to or removed from the refrigerator 1, thus preventing unnecessary recording when no food is being added or removed. In addition, the recording control unit 224 can suppress an increase in the data capacity of the recording file 231, thereby preventing an increase in the processing target of the food detection unit 225 and thus preventing an increase in the processing time and processing load of the food detection unit 225.

[0282] The recording control unit 224 stops recording if the door 11C or drawer 16A of the refrigerator 1, which was open after recording has started, is closed.

[0283] According to this, recording is terminated at a time when it is highly likely that the loading and unloading of food has finished, thus preventing unnecessary recording when no food is being loaded or unloaded from refrigerator 1. In addition, the recording control unit 224 can suppress an increase in the data capacity of the recording file 231, thereby preventing an increase in the processing target of the food detection unit 225 and thus preventing an increase in the processing time and processing load of the food detection unit 225.

[0284] When recording is stopped, the recording control unit 224 generates a recording file 231 containing the results of the recording from the time the door 11C or drawer 16A of the refrigerator 1 is opened until it is closed.

[0285] According to this, the recording results from when the door 11C or drawer 16A of the refrigerator 1 is opened until it is closed are generated as a recording file 231, so that the recording file 231 can be generated quickly while suppressing an increase in the data capacity of the recording file 231. Furthermore, because the recording file 231 can be generated quickly while suppressing an increase in the data capacity of the recording file 231, the food detection unit 225 can quickly start processing related to the recording file 231 after food is placed in or taken out of the refrigerator 1.

[0286] If the recording control unit 224 stops recording and the motion sensor 212 detects that there are no people around the refrigerator 1, the shooting control unit 223 instructs the camera 213 to stop shooting.

[0287] According to this, if the motion sensor 212 detects that there are no people around the refrigerator 1, the camera 213 will stop taking pictures, thus preventing the camera 213 from taking pictures when no food is being put in or taken out of the refrigerator 1. Therefore, it is possible to accurately photograph food being put in or taken out of the refrigerator 1 while preventing an increase in power consumption.

[0288] The camera unit 2 includes a food detection unit 225 that detects food items being put into or taken out of the refrigerator 1 based on the recording results from the recording control unit 224.

[0289] According to this, food can be detected based on the results of accurately photographing the food being put into and taken out of refrigerator 1, thus enabling accurate detection of food being put into and taken out of refrigerator 1.

[0290] As explained, the food management application 311 causes the terminal processor 300 of the terminal device 3 equipped with a touch panel 32 to function as an application execution unit 301 that displays a refrigerator interior image RG via the touch panel 32, and displays a photographed food image FG in the divided area BA corresponding to the food storage area of ​​the refrigerator compartment 11 within the displayed refrigerator interior image RG. Furthermore, the terminal device 3 includes a touch panel 32 and an application execution unit 301 that displays a refrigerator interior image RG via the touch panel 32, and displays a photographed food image FG in the divided area BA corresponding to the food storage area of ​​the refrigerator compartment 11 within the displayed refrigerator interior image RG. Furthermore, the control method for terminal device 3 involves displaying an image RG of the inside of the refrigerator compartment, and displaying a photographed food image FG in the divided area BA corresponding to the storage area of ​​the food stored in the refrigerator compartment 11 within the displayed image RG of the inside of the refrigerator compartment.

[0291] According to this food management app 311, terminal device 3, and control method for terminal device 3, the captured food image FG is displayed in the image showing the inside of the refrigerator compartment in the corresponding divided area BA of the storage area for the food stored in the refrigerator. Therefore, the user P can intuitively understand what kind of food is stored in the refrigerator compartment 11.

[0292] Image FG of the captured food is an image taken by camera 203 for the refrigerator compartment.

[0293] According to this, since the user P is provided with images of the food actually stored in refrigerator 1, user P can more intuitively understand what kind of food is stored in refrigerator compartment 11.

[0294] The refrigerator interior image RG is an image corresponding to the model of refrigerator 1 used by user P of terminal device 3.

[0295] According to this, a refrigerator compartment image RG corresponding to the refrigerator compartment 11 of the refrigerator 1 that is actually being used is provided to the user P, so that the user P can more intuitively understand what kind of food is stored in the refrigerator compartment 11.

[0296] The refrigerator interior image RG is divided into multiple divided areas BA. The number of divided areas BA in the refrigerator interior image RG corresponds to the model of refrigerator 1 used by user P of terminal device 3.

[0297] According to this, user P can understand what kind of food is stored in the storage area of ​​the refrigerator compartment 11 of the refrigerator 1 that is actually being used. Therefore, user P can more intuitively understand what kind of food is stored in the refrigerator compartment 11.

[0298] When the application execution unit 301 displays multiple photographed food images FG in a single divided area BA, it displays the multiple photographed food images FG side by side.

[0299] According to this, multiple photographed food images FG can be displayed clearly for each divided area BA, allowing the user P to intuitively and easily understand what kind of food is stored in the refrigerator compartment 11.

[0300] In the divided area BA, there is a limit set on the number of photographed food images FG that can be displayed simultaneously.

[0301] This prevents a decrease in visibility when there are many photographed food images FG displayed in the divided area BA. Therefore, the user P can intuitively and easily understand what kind of food is stored in the refrigerator compartment 11.

[0302] If the number of photographed food images FG to be displayed in one divided area BA exceeds the upper limit, the application execution unit 301 displays the photographed food images FG in a switchable manner within the range of the upper limit.

[0303] According to this, while preventing a decrease in visibility when there are many photographed food images FG displayed in the divided area BA, the user P can still see the photographed food images FG that cannot be displayed.

[0304] The application execution unit 301 receives memo input from user P of terminal device 3 for each photographed food image FG, and displays a memo input icon MIC associated with the photographed food image FG to indicate that a memo has been received.

[0305] According to this, user P can understand which food items have had notes entered into them. Therefore, user P can intuitively understand what kind of food items are stored in the refrigerator compartment 11, and also understand which food items have had notes entered into them.

[0306] The food management app 311 is an application program that can be installed on terminal device 3.

[0307] According to this, a terminal device 3 that does not have the function to display the refrigerator interior image RG and the photographed food image FG can be made into a terminal device 3 that can display these images by installing the food management application 311.

[0308] As explained, the food management application 311 causes the terminal processor 300 of the terminal device 3 equipped with a touch panel 32 to function as an application execution unit 301 that displays images captured by a drawer camera 204 that photographs the drawer 16A of the refrigerator 1 having an upper case JCA and a lower case GCA. The application execution unit 301 is capable of displaying the upper case image JCG and the lower case image GCG, and accepts memo input from the user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG. When the application execution unit 301 receives memo input from the user P for the upper case image JCG, it displays a memo input icon MIC associated with the upper case image JCG to indicate that memo input has been received, and when it receives memo input from the user P for the lower case image GCG, it displays a memo input icon MIC associated with the lower case image GCG.

[0309] The terminal device 3 also includes a touch panel 32 and an application execution unit 301 that displays images captured by the pull-out camera 204. The application execution unit 301 is capable of displaying an upper case image JCG and a lower case image GCG, and accepts memo input from the user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG. When the application execution unit 301 receives memo input from the user P for the upper case image JCG, it displays a memo input icon MIC associated with the upper case image JCG to indicate that memo input has been received, and when it receives memo input from the user P for the lower case image GCG, it displays a memo input icon MIC associated with the lower case image GCG.

[0310] Furthermore, the control method for terminal device 3 is capable of displaying the upper case image JCG and the lower case image GCG. For each of the upper case image JCG and the lower case image GCG, the terminal device 3 accepts memo input from user P. When memo input is received from user P for the upper case image JCG, a memo input icon MIC indicating that memo input has been received is displayed in association with the upper case image JCG. When memo input is received from user P for the lower case image GCG, the memo input icon MIC is displayed in association with the lower case image GCG.

[0311] According to this, the food management app 311, the terminal device 3, and the control method for the terminal device 3 display a photograph of the case CA in drawer 16A, and the user P can input notes on this displayed photograph. As a result, the user P can intuitively understand the food stored in drawer 16A of the refrigerator 1, and the user P's convenience can be improved.

[0312] The application execution unit 301 displays the upper case image JCG superimposed on the lower case image GCG, and when it receives an operation from user P to move the upper case image JCG, it moves the first case image to display the lower case image GCG.

[0313] According to this, the lower case image GCG can be displayed in a manner that moves the actual case CA. Therefore, user P can more intuitively grasp the food stored in drawer 16A of refrigerator 1, and user P's convenience can be improved.

[0314] If the application execution unit 301 is displaying the memo input icon MIC in association with the upper case image JCG, it moves the memo input icon MIC together with the upper case image JCG.

[0315] According to this, even if user P moves the upper case image JCG, the association between the upper case image JCG and the memo input icon MIC can be maintained. Therefore, even if user P moves the upper case image JCG, user P can properly understand which image the memo input icon MIC relates to, thereby improving user convenience.

[0316] The application execution unit 301 displays a toggle button to switch between the upper case image JCG and the lower case image GCG, and switches the displayed captured image to the upper case image JCG or the lower case image GCG in response to the operation of the toggle button.

[0317] According to this, the user P can easily switch between displayed case images CG by operating a switch button, thereby improving user convenience.

[0318] The upper case image JCG is an image taken when the upper case JCA is pulled out of refrigerator 1 by a predetermined threshold or more. The lower case image GCG is an image taken when the lower case GCA is pulled out of refrigerator 1 by a predetermined threshold or more.

[0319] According to this, a photograph that shows as many food items as possible can be used as a case image CG, allowing user P to intuitively understand the food items stored in drawer 16A of refrigerator 1, and improving user P's convenience.

[0320] The food management app 311 is an application program that can be installed on terminal device 3.

[0321] According to this, a terminal device 3 that does not have the function of displaying a case image CG or the function of accepting memo input for the displayed case image CG can be made into a terminal device 3 that has these functions by installing the food management application 311.

[0322] (Embodiment 2) Next, Embodiment 2 will be described. In the description of Embodiment 2, the same reference numerals are used for components that are the same as those of the food management system 1000 in Embodiment 1, and detailed descriptions will be omitted as appropriate.

[0323] [2-1. Structure] Figure 26 is a block diagram showing the configuration of the terminal device 3 and the food management server 4 in Embodiment 2. As is clear from comparing Figure 26 and Figure 7, in Embodiment 2, the server processor 400 functions as a food recognition unit 403 in addition to the server communication control unit 401 and information processing unit 402 by executing a control program stored in the server storage unit 410.

[0324] The food recognition unit 403 performs food recognition processing based on the captured food image FG indicated by the food image data 4116 contained in the first update information and the second update information, respectively.

[0325] The food recognition process involves recognizing food from a captured food image FG and retrieving an icon representing the same type of food as the recognized food from an icon database. This icon database stores an icon representing each food item. Hereinafter, this icon will be referred to as a food icon. The icon database may be stored on the food management server 4, or it may be stored on an external device that can communicate with the food management server 4. In this embodiment, the food icon corresponds to an example of a food image in this disclosure.

[0326] For example, the food recognition unit 403 recognizes food as follows: The food recognition unit 403 identifies the image with the highest degree of match to the photographed food image FG indicated by the food image data 4116 included in the update information, from the images stored in the food image database. The food image database is a database in which image data is stored for each food item. The degree of match is determined based on feature quantities such as shape and color. The food recognition unit 403 then recognizes that the food item corresponding to the identified image is the food item indicated by the food image data 4116 included in the update information.

[0327] The food recognition unit 403 may also perform food recognition using AI (Artificial Intelligence). For example, a machine learning process is performed to learn feature quantities such as the color and shape of food from training images of photographed food FG, and a trained model is constructed. This trained model is stored in the server storage unit 410. The food recognition unit 403 uses the food image data 4116 included in the update information as input data, refers to the trained model, and recognizes the food indicated by the photographed food image FG. Here, the trained model may be stored in a device other than the food management server 4 connected to the global network GN, and the food may be recognized by the food management server 4. In this configuration, the food recognition unit 403 transmits the food image data 4116 included in the update information to the device that stores the trained model, and obtains the food identification result from this device. The food recognition unit 403 then recognizes the food indicated by the obtained identification result as the food indicated by the photographed food image FG indicated by the food image data 4116 included in the update information.

[0328] When the food recognition unit 403 recognizes a food item, it retrieves the food icon for the recognized food item from the icon database. In Embodiment 2, the icon data of the food icon retrieved by the food recognition unit 403 is appropriately stored in place of, or together with, the food image data 4116 held in the refrigerator compartment food management record RR. Also in Embodiment 2, the icon data of the food icon retrieved by the food recognition unit 403 is appropriately stored in place of, or together with, the food image data 4116 held in the drawer food management record HR.

[0329] As a result, Embodiment 2 can display a food icon in the app UI320 instead of the captured food image FG.

[0330] [2-2. Effects, etc.] According to Embodiment 2, the same effects as in Embodiment 1 are achieved.

[0331] (Other embodiments) As described above, the above embodiments have been explained as examples disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments. Therefore, other embodiments are illustrated below.

[0332] In the embodiments described above, a refrigerator 1 was given as an example of a storage facility. However, the storage facility of this disclosure is not limited to a refrigerator 1, and may be a warehouse installed in an office, for example. In the embodiments described above, food was given as an example of the articles of this disclosure, but the articles of this disclosure are not limited to food, and may be other articles such as books, depending on the storage facility.

[0333] In the embodiment described above, a configuration was illustrated in which the position regulating member 205 has a mark 207 at the center of the main body 201 in the left-right direction. However, the position of the mark 207 is not limited to the center of the main body 201 in the left-right direction. The position of the mark 207 may be other than the center of the main body 201 in the left-right direction. In this case, the marks are appropriately recorded on the central position recording sheet CY so that the camera unit 2, whose position has been adjusted based on the mark 207, can appropriately perform various detections.

[0334] In the embodiments described above, the refrigerator interior image RG, which is a front view of the refrigerator compartment 11 when the left door 11A and the right door 11B are open, was given as an example of the interior image of the disclosure. However, the interior image of the disclosure is not limited to this refrigerator interior image RG, and may also be a front view of the refrigerator compartment when one door is open, or in the case of a refrigerator 1 in which the interior includes a refrigerator compartment and a freezer compartment, it may be a front view of the interior of the refrigerator 1 when the door is open.

[0335] The shooting range of the drawer camera 204 may be a combination of one or more arbitrary drawer storage compartments, depending on the type of drawer storage compartment displayed on the screen shown by the application UI 320.

[0336] The food management system 1000 may also manage other factors besides the date and time of storage, such as remaining quantity and expiration date.

[0337] The main body 201 and the shooting element 202 of the camera unit 2 may be constructed as separate components.

[0338] The camera unit 2 may be configured to send the case image CG to the food management server 4 even if it does not exceed a predetermined threshold. In this configuration, the food management server 4 stores the case image CG when it does not exceed a predetermined threshold and provides it to the terminal device 3. The application execution unit 301 then displays this case image CG in the application UI 320, interchangeably with the upper case image JCG and the lower case image GCG, respectively.

[0339] In the embodiment described above, the case in which user P installs the camera unit 2 into the refrigerator 1 was illustrated, but the camera unit 2 may be shipped already installed in the refrigerator 1.

[0340] The server processor 400 may function as the food detection unit 225, while the camera unit processor 220 may not function as the food detection unit 225. In this configuration, the camera unit 2 sends the recorded file 231 to the food management server 4.

[0341] In the embodiment described above, a human presence sensor 212 was given as an example of the sensor of this disclosure. However, the sensor of this disclosure is not limited to the human presence sensor 212, and may be an illuminance sensor, an electrostatic sensor provided in the gripping part of the refrigerator 1, etc. In the case of this electrostatic sensor, the refrigerator 1 and the camera unit 2 can communicate with each other, and the refrigerator 1 transmits the detected value of the electrostatic sensor to the camera unit 2.

[0342] Alternatively, the terminal device 3 may store the refrigerator compartment food management record RR and the drawer food management record HR, and the food management application 311 may manage various information related to the food detected by the camera unit 2. In this configuration, the application execution unit 301 updates the contents of each record stored by the terminal device 3 as appropriate, similar to the food management server 4 described above. This simplifies the system configuration as the food management system 1000 does not need to have a food management server 4. In this configuration, the application execution unit 301 may also function as a food recognition unit 403.

[0343] For example, the types of compartments formed in the main body 10 of the refrigerator 1 are not limited to the refrigerator compartment 11, ice-making compartment 12, fresh freezing compartment 13, freezer compartment 14, and vegetable compartment 15; there may be fewer than 1, or other types of compartments may be formed. Also, the number of doors provided in the front opening of the refrigerator compartment 11 may be just one.

[0344] For example, the cases in drawers 14A and 15A are not limited to two levels, but may be more. In this case, one case corresponds to the first case of this disclosure, and the lower case of this case corresponds to the second case of this disclosure.

[0345] For example, the camera unit processor 220, the terminal processor 300, and the server processor 400 may be composed of a single processor or multiple processors.

[0346] The parts shown in Figures 6, 7, and 26 are examples, and the specific implementation is not particularly limited. In other words, it is not necessarily required that hardware corresponding to each part be implemented individually, and it is certainly possible to configure the system so that a single processor executes a program to realize the functions of each part. Furthermore, some of the functions realized by software in the above-described embodiment may be implemented by hardware, or some of the functions realized by hardware may be realized by software. In addition, the specific detailed configurations of other parts of the camera unit 2, terminal device 3, and food management server 4 can also be arbitrarily changed without departing from the spirit of this disclosure.

[0347] Furthermore, for example, the operational step units shown in Figures 13, 14, 22, and 25 are divided according to the main processing content in order to facilitate understanding of the operation of each part of the food management system 1000, and this disclosure is not limited by the way the processing units are divided or the names of those units.

[0348] Since the embodiments described above are for illustrative purposes of the technology described herein, various modifications, substitutions, additions, omissions, etc., can be made within the claims or their equivalents. [Industrial applicability]

[0349] As described above, the program, terminal device, and control method for the terminal device according to the present invention can be used for the purpose of displaying images of cases in a pull-out storage compartment. [Explanation of Symbols]

[0350] 1. Refrigerator (storage room) 2. Camera unit (imaging device) 3 Terminal devices 4. Food Management Server 10 Main box 10A Front edge 10B Top surface 11 Refrigerator 11A Left Door 11B Right Door 11C Door 12. Ice-making compartment (drawer-type storage compartment) 12A, 13A, 14A, 15A, 16A drawer 13. Fresh Freezing Chamber (Drawer-type storage chamber) 14. Freezer compartment (drawer-type storage compartment) 15. Vegetable compartment (drawer-type storage compartment) 32 Touch panel (display unit) 201 Main Unit 202 Shooting Equipment 203 Camera for Refrigerator 204 Drawer Camera 205 Position regulating member 206 Bottom 207 Mark 212 Motion Sensor (Sensor) 213 Camera 223 Imaging Control Unit 224 Recording Control Unit 225 Food detection unit (object detection unit) 231 Recording files 300 Terminal Processors (Computers) 301 Application Execution Unit (Display Control Unit) 311 Food Management App (Program, Application Program) 1000 Food Management Systems BA divided area (area) FG, FG1 Food images (Food images) GCA Lower Case (Second Case) GCG lower case image (second case image) JCA Upper Case (Case 1) JCG upper case image (first case image) MIC memo input icon (memo input image) P User RG Refrigerator Interior Image (Interior Image)

Claims

1. A computer in a terminal device equipped with a display unit, It functions as a display control unit that displays images captured by a camera that photographs the drawer of a refrigerator having a first case and a second case which is the lower case of the first case. The display control unit, The system can display the first case image, which is the image taken of the first case, and the second case image, which is the image taken of the second case. The first case image is displayed superimposed on the image area of ​​the second case image. When the terminal device receives an operation from the user to move the first case image in the direction of movement of the first case, In accordance with the above operation, the display range of the first case image is changed, The display range of the second case image is changed in accordance with the change in the display range of the first case image. program.

2. The display control unit is The first case image is displayed superimposed on the second case image. When the user requests that the first case image be moved, the first case image is moved and the second case image is displayed. The program according to claim 1.

3. The display control unit is The system further includes a function to display item images extracted from the first case image and the second case image as a list screen. A list display button is displayed to switch to the aforementioned list screen. In response to an operation on the list display button, the displayed screen is switched to the list screen. The program according to claim 1 or 2.

4. The first case image is the image taken when the first case is pulled out of the refrigerator by a predetermined threshold or more. The second case image is the image taken when the second case is pulled out of the refrigerator by a predetermined threshold or more. The program according to any one of claims 1 to 3.

5. The aforementioned terminal device is an application program that can be installed. The program according to any one of claims 1 to 4.

6. Display unit and The system includes a display control unit that displays images captured by a camera that photographs the drawer of a refrigerator having a first case and a second case which is a lower case of the first case, The display control unit, The system can display the first case image, which is the image taken of the first case, and the second case image, which is the image taken of the second case. The first case image is displayed superimposed on the image area of ​​the second case image. When the user requests an operation to move the first case image in the direction of movement of the first case, In accordance with the above operation, the display range of the first case image is changed, The display range of the second case image is changed in accordance with the change in the display range of the first case image. Terminal device.

7. A control method for a terminal device that displays images captured by a camera that photographs the drawer of a refrigerator having a first case and a second case which is a lower case of the first case, The system can display the first case image, which is the image taken of the first case, and the second case image, which is the image taken of the second case. The first case image is displayed superimposed on the image area of ​​the second case image. When the terminal device receives an operation from the user to move the first case image in the direction of movement of the first case, In accordance with the above operation, the display range of the first case image is changed, The display range of the second case image is changed in accordance with the change in the display range of the first case image. A method for controlling terminal devices.

Images