Program, terminal device and terminal device control method

Abstract

To provide a program, a terminal device and a terminal device control method which enable a user to intuitively assess food stored in a drawer inside a refrigerator and can improve user convenience.SOLUTION: A program causes a computer of a terminal device to function as a display control section which displays images taken with cameras imaging interiors of a first case and a second case which is a case positioned below the first case. The display control section: can display a first case image and a second case image; receives input of a user's memo with respect to the first case image and the second case image; displays a memo input image in association with the first case image when receiving the memo with respect to the first case image; and displays the memo input image in association with the second case image when receiving the memo with respect to the second case image.SELECTED DRAWING: Figure 23

Description

[Technical field]

[0001] The present disclosure relates to a program, a terminal device, and a control method for a terminal device. [Background technology]

[0002] Patent Document 1 discloses a technology in which, when an object passing through the opening of the refrigerator is detected, a camera installed on the top of the refrigerator takes a photograph, and the refrigerator or a terminal device displays information about the food stored in the refrigerator based on the photographed result. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-070476 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a program, a terminal device, and a method for controlling a terminal device that enable a user to intuitively grasp the foods stored in a refrigerator drawer and improve user convenience. [Means for solving the problem]

[0005] The program of the present disclosure causes a computer of a terminal device having a display unit to function as a display control unit that displays an image captured by a drawer camera that captures an image of a drawer of a refrigerator having a first case and a second case that is a lower case of the first case, and the display control unit is capable of displaying a first case image which is the captured image of the first case and a second case image which is the captured image of the second case, and accepts note input from a user of the terminal device for each of the first case image and the second case image, and when note input from the user is accepted for the first case image, displays a note input image indicating that note input has been accepted in association with the first case image, and when note input from the user is accepted for the second case image, displays the note input image in association with the second case image.

[0006] In addition, the terminal device of the present disclosure includes a display unit and a display control unit that displays an image captured by a drawer camera that captures an image of a drawer of a refrigerator having a first case and a second case that is a lower case of the first case, and the display control unit is capable of displaying a first case image that is the captured image of the first case and a second case image that is the captured image of the second case, and accepts note input from a user for each of the first case image and the second case image, and when note input from the user is accepted for the first case image, displays a note input image indicating that note input has been accepted in association with the first case image, and when note input from the user is accepted for the second case image, displays the note input image in association with the second case image.

[0007] Furthermore, a control method of a terminal device in the present disclosure is a control method of a terminal device that displays an image captured by a drawer camera that captures an image of a drawer of a refrigerator having a first case and a second case that is a lower case of the first case, and is capable of displaying a first case image that is the captured image of the first case, and a second case image that is the captured image of the second case, and accepts input of a note from a user of the terminal device for each of the first case image and the second case image, and when note input from the user is accepted for the first case image, a note input image indicating that note input has been accepted is displayed in association with the first case image, and when note input from the user is accepted for the second case image, the note input image is displayed in association with the second case image. Effect of the Invention

[0008] The program, terminal device, and control method for the terminal device disclosed herein display a photographed image of a case in a drawer, and allow the user to input a memo to the displayed photographed image, thereby enabling the user to intuitively grasp the foods stored in the refrigerator drawer and improving user convenience. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 shows a configuration of a food management system according to a first embodiment. [Diagram 2] FIG. 1 is a side view of a camera unit installed in a refrigerator according to a first embodiment, seen from the right side. [Diagram 3] FIG. 11 is a perspective view of a position restriction member abutting against a front edge of a main box body according to the first embodiment, as viewed from the left front side; [Figure 4] FIG. 1 is a front view of a refrigerator and a camera unit according to a first embodiment; [Diagram 5] FIG. 1 is a plan view of a refrigerator and a camera unit according to a first embodiment, viewed from the right side. [Figure 6] FIG. 1 is a block diagram showing a configuration of a camera unit according to a first embodiment. [Figure 7]A block diagram showing the configuration of a terminal device and a food management server according to the first embodiment. [Figure 8] FIG. 1 is a diagram showing an example of a refrigerator management database according to the first embodiment. [Figure 9] FIG. 1 is a diagram showing an example of an image of the inside of a refrigerator compartment in the first embodiment. [Figure 10] FIG. 1 shows an example of a withdrawal management database according to the first embodiment. [Figure 11] FIG. 13 shows an example of an adjustment screen in the first embodiment. [Figure 12] FIG. 1 shows an example of a central position recording sheet according to the first embodiment. [Figure 13] A flowchart showing the operation of the camera unit in the first embodiment. [Figure 14] A flowchart showing the operation of a food management system in the first embodiment. [Figure 15] FIG. 1 is a diagram for explaining an entry / exit determination image in the first embodiment; [Figure 16] FIG. 13 shows an example of a refrigerator compartment home screen in the first embodiment. [Figure 17] FIG. 1 shows a procedure for inputting a memo in the first embodiment. [Figure 18] FIG. 13 shows an example of a storage list screen in the first embodiment. [Figure 19] FIG. 13 is a diagram showing an example of a new addition list screen in the first embodiment. [Figure 20] FIG. 1 is a diagram for explaining a procedure for adding food information in the first embodiment. [Figure 21] FIG. 13 shows an example of a deletion target list screen in the first embodiment. [Figure 22] A flowchart showing the operation of a food management system in the first embodiment. [Diagram 23] FIG. 13 is a diagram showing an example of a vegetable compartment home screen in the first embodiment. [Figure 24] FIG. 13 shows an example of a food image list screen in the first embodiment. [Diagram 25] A flowchart showing the operation of a food management system in the first embodiment. [Figure 26] A block diagram showing the configuration of a terminal device and a food management server according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (The knowledge and other information that formed the basis of this disclosure) At the time when the inventors came up with the present disclosure, there was a technology in which a terminal device displays a list of foods stored in a refrigerator. However, the inventors discovered a problem with the conventional technology in that the list of foods is simply displayed, but the user cannot intuitively grasp what foods are stored in drawer-type storage compartments such as the vegetable compartment and the freezer compartment. The inventors also discovered a problem with the conventional technology in that the list of foods is simply displayed, but the user can only view the list, so there is room for improvement in improving user convenience. The inventors have come up with the subject matter of the present disclosure in order to solve these problems. Therefore, the present disclosure provides a program, a terminal device, and a control method for a terminal device that enable a user to intuitively grasp the foods stored in a refrigerator drawer and improve user convenience.

[0011] Hereinafter, the embodiments will be described in detail with reference to the drawings. However, in some cases, more detailed explanations than necessary may be omitted. For example, detailed explanations of already well-known matters or duplicate explanations of substantially the same configurations may be omitted. It should be noted that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0012] (Embodiment 1) First, the first embodiment will be described. [1-1. Configuration] [1-1-1. Food control system configuration] FIG. 1 is a diagram showing the configuration of a food management system 1000.

[0013] The food management system 1000 is a system that manages the storage date and time when the food was stored in the refrigerator 1, etc., for the food stored in the refrigerator 1. The refrigerator 1 corresponds to an example of a storage unit of the present disclosure. The food corresponds to an example of an article of the present disclosure.

[0014] The food management system 1000 includes a refrigerator 1. The refrigerator 1 includes a main box 10 with an opening at the front. The main box 10 is formed with a refrigerator compartment 11, an ice making compartment 12, a fresh freezing compartment 13, a freezer compartment 14, and a vegetable compartment 15. The opening at the front of the refrigerator compartment 11 is provided with a rotating left door 11A and a right door 11B. The ice making compartment 12, the fresh freezing compartment 13, the freezer compartment 14, and the vegetable compartment 15 are each provided with drawers 12A, 13A, 14A, and 15A for storing food. The drawers 14A and 15A of the present disclosure include two cases CA, an upper case JCA and a lower case GCA. The upper case JCA corresponds to an example of a first case in the present disclosure, and the lower case GCA corresponds to an example of a second case in the present disclosure.

[0015] In the following description, when there is no need to distinguish between the left door 11A and the right door 11B, they will be referred to as "doors" and given the symbol "11C." Also, in the following description, when there is no need to distinguish between the drawers 12A, 13A, 14A, and 15A, they will be referred to as "drawers" and given the symbol "16A."

[0016] Food management system 1000 includes a camera unit 2. Camera unit 2 is a device that takes photographs and detects food entering or leaving refrigerator 1 from the photographed images. Camera unit 2 is provided on top surface 10B of main box 10 of refrigerator 1. The camera unit 2 corresponds to an example of the imaging device of the present disclosure.

[0017] The food management system 1000 includes a terminal device 3. The terminal device 3 is, for example, a smartphone or a tablet terminal. An application program for managing food in the refrigerator 1 is installed on the terminal device 3, and the terminal device 3 communicates with the food management server 4 by using the function of this application program. In the following description, this application program is referred to as the "food management application" and is given the reference number "311." The food management application 311 corresponds to an example of a program and an application program of the present disclosure.

[0018] In Fig. 1, a user P who is at home is indicated by a solid line, and a user P who has left home H is indicated by a dotted line. When the terminal device 3 is used by a user P who is at home, it communicates with the food management server 4 via the communication device 5 or without the communication device 5, using the function of the food management app 311. When the terminal device 3 is used by a user P who has left home H and a communication connection with the communication device 5 cannot be established, it communicates with the food management server 4 via the function of the food management app 311, without the communication device 5.

[0019] The communication device 5 is connected to a global network GN including the Internet, a telephone network, and other communication networks, and communicates with the food management server 4 connected to the global network GN. The communication device 5 is an interface device for connecting the terminal device 3 to the global network GN.

[0020] Food management system 1000 includes a food management server 4. Food management server 4 is a server device that manages food in refrigerator 1 and is connected to global network GN. Note that in each figure, food management server 4 is represented by a single block, but this does not necessarily mean that food management server 4 is composed of a single server device. For example, food management server 4 may be composed of multiple server devices with different processing contents.

[0021] Next, the configurations of the camera unit 2, the terminal device 3, and the food management server 4 will be described.

[0022] [1-1-2. Camera unit configuration] The camera unit 2 will be described with reference to FIGS. 2 to 4 show the X-axis, Y-axis, and Z-axis. The X-axis, Y-axis, and Z-axis are perpendicular to each other. The Z-axis indicates the up-down direction. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the left-right direction and also indicates the width direction of refrigerator 1 and camera unit 2. The Y-axis indicates the front-rear direction and also indicates the depth direction of refrigerator 1 and camera unit 2. The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the front direction. The positive direction of the Z-axis indicates the upward direction.

[0023] Fig. 2 is a side view of the camera unit 2 installed in the refrigerator 1, seen from the right side. Fig. 3 is a perspective view of the position restricting member 205 abutting against the front edge 10A of the main box body 10, seen from the left front side.

[0024] 2 and 3, camera unit 2 includes main body 201 having a box-like appearance and installed on top surface 10B of main box 10. Furthermore, camera unit 2 includes imaging member 202 extending forward from main body 201 above the front of main body 201. Imaging member 202 includes refrigerator compartment camera 203 and drawer camera 204. Refrigerator compartment camera 203 and drawer camera 204 are each provided below imaging member 202 and forward of main body 201.

[0025] The main body 201 is provided with a position restricting member 205 that restricts the position of the imaging member 202 in the front-rear direction of the refrigerator 1 by abutting against the front end edge 10A of the main box body 10. The position restricting member 205 is a position restricting piece that extends downward from a bottom surface 206 of the main body 201 in the installed state. The position restricting member 205 extends in the left-right direction of the main body 201 on the front surface of the main body 201, and has a mark 207 in the center of the main body 201 in the left-right direction. This mark 207 indicates the center position of the camera unit 2 in the left-right direction.

[0026] The length of the position regulating member 205 is shorter than the length from the upper surface 10B of the main box body 10 to a packing provided between the refrigerator 1 and the door 11C in the installed state. More specifically, the length of the position regulating member 205 is shorter than the length from the upper surface 10B of the main box body 10 to a door packing provided approximately parallel to the upper surface 10B on the upper inner surface of the door 11C in the installed state.

[0027] The main body 201 is fixed at a position regulated by the position regulating member 205 by the built-in magnet 208. The means for fixing the main body 201 is not limited to the magnet 208, and other means such as double-sided tape may be used.

[0028] Fig. 4 is a front view of refrigerator 1 and camera unit 2 as viewed from the front. In Fig. 4, refrigerator 1 has left door 11A and right door 11B open. Fig. 5 is a plan view of refrigerator 1 and camera unit 2 as viewed from the right side. In Fig. 5, refrigerator 1 has drawer 15A open, i.e., pulled out. In Figs. 4 and 5, camera unit 2 has mark 207 located approximately in the center of refrigerator 1 in the left-right direction, and position restriction member 205 abuts front edge 10A.

[0029] The camera unit 2 has a refrigerator compartment camera 203 and a drawer camera 204 . The refrigerator compartment camera 203 photographs the food entering and leaving the refrigerator compartment 11 by photographing the front of the refrigerator 1 from above to below.

[0030] For example, the shooting range of the refrigerator compartment camera 203 includes range A1 shown in Fig. 4 when viewed from the front. Range A1 is a range that includes door pocket 111 provided in door 11C and the opening on the front side of refrigerator compartment 11 when door 11C is in the open state. Moreover, the shooting range of the refrigerator compartment camera 203 includes range A3 shown in Fig. 5 when viewed from the side. Range A3 is a range that includes from door 11C in the open state to the front end of shelf 112 in the front-to-rear direction, and also includes the opening of refrigerator compartment 11 in the up-down direction.

[0031] Drawer camera 204 photographs the food entering or leaving the drawer-type storage compartment by photographing the area from above and below the front of refrigerator 1.

[0032] For example, the shooting range of drawer camera 204 includes range A2 shown in Fig. 4 when viewed from the front. Range A2 is a range that includes drawer 16A when viewed from the front of refrigerator 1. Moreover, the shooting range of drawer camera 204 includes range A4 shown in Fig. 5 when viewed from the side. Range A4 is a range that includes drawer 16A when it is pulled out to the maximum in the front-to-rear direction.

[0033] The camera unit 2 has a left door distance measuring sensor 209 , a right door distance measuring sensor 210 , and a drawer distance measuring sensor 211 .

[0034] The left door distance measuring sensor 209 is a distance measuring sensor for detecting the open / closed state of the left door 11A. The left door distance measuring sensor 209 detects the distance from the left door 11A as the open / closed state of the left door 11A.

[0035] The right door distance measuring sensor 210 is a distance measuring sensor for detecting the open / closed state of the right door 11B. The right door distance measuring sensor 210 detects the distance from the right door 11B as the open / closed state of the right door 11B.

[0036] The drawer distance measuring sensor 211 is a distance measuring sensor for detecting the open / closed state of the drawer 16A. The open state of the drawer 16A refers to a state in which the drawer 16A is drawn out from the corresponding drawer-type storage chamber. The closed state of the drawer 16A refers to a state in which the drawer 16A is stored in the corresponding drawer-type storage chamber. The drawer distance measuring sensor 211 detects the distance from the drawer 16A as the open / closed state of the drawer 16A. In this embodiment, drawer distance measuring sensor 211 detects the open / closed state of each of drawers 14A, 15A.

[0037] The camera unit 2 has a human presence sensor 212. The human presence sensor 212 outputs a detection value to the camera unit control unit 20. The human presence sensor 212 is provided at a predetermined position of the imaging member 202. The human presence sensor 212 corresponds to an example of a "sensor" in the present disclosure.

[0038] Note that the arrangement and installation positions of each sensor and camera in Figures 4 and 5 are illustrated for the convenience of explaining camera unit 2, and these arrangement and installation positions are not limited to the arrangement and installation positions shown in Figures 4 and 5.

[0039] FIG. 6 is a block diagram showing the configuration of the camera unit 2. As shown in FIG. The camera unit 2 includes a camera unit control unit 20, a camera unit communication unit 21, a sensor unit 22, an imaging unit 23, and a power supply unit 24.

[0040] The camera unit control unit 20 includes a camera unit processor 220, which is a processor such as a CPU (Central Processing Unit) or an MPU (Micro-processing unit), and a camera unit storage unit 230. The camera unit control unit 20 controls each part of the camera unit 2 by the camera unit processor 220 reading and executing a control program stored in the camera unit storage unit 230. The camera unit processor 220 executes the control program stored in the camera unit storage unit 230, thereby functioning as a camera unit communication control unit 221, a detection value processing unit 222, an imaging control unit 223, a recording control unit 224, and a food detection unit 225.

[0041] The camera unit storage unit 230 includes a memory for storing programs executed by the camera unit processor 220 and data processed by the camera unit processor 220. The camera unit storage unit 230 stores the control programs executed by the camera unit processor 220, a camera unit ID 4112 that is identification information for the camera unit 2, and various other data. The camera unit storage unit 230 includes a non-volatile storage area. The camera unit storage unit 230 may also include a volatile storage area and constitute a work area for the camera unit processor 220.

[0042] The camera unit communication unit 21 is a communication interface equipped with a communication circuit, connector, etc. that conforms to a predetermined communication standard, and communicates with the food management server 4 and the terminal device 3 under the control of the camera unit control unit 20. In this embodiment, the communication standard used by the camera unit communication unit 21 is exemplified as a wireless communication standard, but it may also be a wired communication standard.

[0043] The sensor section 22 includes a left door distance measuring sensor 209 , a right door distance measuring sensor 210 , a drawer distance measuring sensor 211 , and a human presence sensor 212 , and outputs detection values ​​of each sensor to the camera unit control section 20 .

[0044] The photographing section 23 includes a refrigerator compartment camera 203 and a drawer camera 204. The photographing section 23 outputs the photographing results of each of the refrigerator compartment camera 203 and the drawer camera 204 to the camera unit control section 20. The refrigerator compartment camera 203 and the drawer camera 204 capture moving images. Hereinafter, when there is no need to distinguish between the refrigerator compartment camera 203 and the drawer camera 204, they will be referred to as "camera" and given the reference number "213".

[0045] Power supply unit 24 includes hardware such as a power supply circuit, and supplies power to each component of camera unit 2. In the present embodiment, power cable 7 of camera unit 2 is connected to commercial AC power source 8, and power supply unit 24 supplies power to each component of camera unit 2 based on the power supplied from external commercial AC power source 8.

[0046] The source of power supply to power supply unit 24 is not limited to the external commercial AC power supply 8, but may be the refrigerator 1 or a battery. When the source is the former, power supply unit 24 and refrigerator 1 are equipped with hardware such as a port that complies with a standard that allows USB power supply. When the source is the former, a USB cable of camera unit 2 is connected to a USB port of refrigerator 1, and refrigerator 1 supplies power to power supply unit 24 via the USB cable. When the source is the latter, camera unit 2 is equipped with a battery.

[0047] As described above, the camera unit control unit 20 functions as the camera unit communication control unit 221, the detection value processing unit 222, the photography control unit 223, the recording control unit 224, and the food detection unit 225. The food detection unit 225 corresponds to an example of an item detection unit.

[0048] The camera unit communication control unit 221 communicates with the food management server 4 via the camera unit communication unit 21.

[0049] The detection value processing unit 222 judges whether the left door 11A, the right door 11B, and the drawers 14A and 15A are in an open or closed state based on the detection values ​​of the sensors output from the sensor unit 22. When the detection value processing unit 222 judges that either the left door 11A or the right door 11B is in an open state, it outputs door open state information indicating that the door 11C is in an open state to the recording control unit 224 and the food detection unit 225. When the detection value processing unit 222 judges that both the left door 11A and the right door 11B are in a closed state, it outputs door closed state information indicating that the door 11C is in a closed state to the recording control unit 224 and the food detection unit 225. When the detection value processing unit 222 judges that either the drawers 14A or 15A is in an open state, it outputs drawer open state information indicating that either the drawers 14A or 15A is in an open state to the recording control unit 224 and the food detection unit 225. Furthermore, when the detection value processing unit 222 determines that both of the drawers 14A, 15A are closed, it outputs to the recording control unit 224 and the food detection unit 225 drawer closed state information indicating that the drawers 14A, 15A are closed.

[0050] Furthermore, the detection value processing unit 222 determines whether or not there is a person around the refrigerator 1 based on the detection value of the human presence sensor 212 output from the sensor unit 22. If the detection value processing unit 222 determines that there is a person around the refrigerator 1, it outputs presence information indicating the presence of a person to the photography control unit 223. If the detection value processing unit 222 determines that there is no person around the refrigerator 1, it outputs presence information indicating the presence of a person to the photography control unit 223.

[0051] When the detection value processing unit 222 outputs presence / absence information indicating that a person is present, the photography control unit 223 causes the refrigerator compartment camera 203 and the drawer camera 204 to start photographing. When the detection value processing unit 222 outputs presence / absence information indicating that a person is not present, the photography control unit 223 causes the refrigerator compartment camera 203 and the drawer camera 204 to end photographing.

[0052] When detection value processing unit 222 outputs door open state information, recording control unit 224 starts recording the image captured by refrigerator compartment camera 203. When detection value processing unit 222 outputs door closed state information, recording control unit 224 ends the started recording of the image captured by refrigerator compartment camera 203. When detection value processing unit 222 outputs drawer open state information, recording control unit 224 starts recording the image captured by drawer camera 204. When detection value processing unit 222 outputs drawer closed state information, recording control unit 224 ends the started recording of the image captured by drawer camera 204. When recording control unit 224 ends recording, it generates a video from the start to the end of recording as one recording file 231 and stores it in camera unit storage unit 230.

[0053] The food detection unit 225 detects food that has been put into or taken out of the refrigerator 1 based on the video recording file 231 stored in the camera unit storage unit 230.

[0054] FIG. 7 is a block diagram showing the configuration of the terminal device 3 and the food management server 4.

[0055] [1-1-3. Food management server configuration] First, the configuration of the food management server 4 will be described. The food management server 4 includes a server control unit 40 and a server communication unit 41.

[0056] The server control unit 40 includes a server processor 400, which is a processor such as a CPU or MPU, and a server memory unit 410. The server control unit 40 controls each unit of the food management server 4 by the server processor 400 reading and executing a control program stored in the server memory unit 410. The server processor 400 executes the control program stored in the server memory unit 410, thereby functioning as a server communication control unit 401 and an information processing unit 402.

[0057] The server storage unit 410 has a memory for storing programs executed by the server processor 400 and data processed by the server processor 400. The server storage unit 410 stores the control programs executed by the server processor 400, a refrigerator compartment food management database 411, a drawer food management database 412, and various other data. The server storage unit 410 has a non-volatile storage area. The server storage unit 410 may also have a volatile storage area and constitute a work area for the server processor 400.

[0058] FIG. 8 is a diagram showing an example of one record stored in the refrigerator compartment food management database 411. The refrigerator compartment food management database 411 is a database that stores various information related to the food stored in the refrigerator compartment 11. In the following description, one record stored in the refrigerator food management database 411 is referred to as a "refrigerator food management record" and is marked with the symbol "RR."

[0059] The refrigerator food management record RR has an account ID 4111, a camera unit ID 4112, refrigerator model information 4113, refrigerator interior image data 4114, entry / exit information 4115, food image data 4116, storage date and time information 4117, divided area number 4118, memo entry information 4119, memo information 4130, deletion information 4131, and deletion date and time information 4132.

[0060] The account ID 4111 is identification information that identifies an account assigned to a user P who uses the food management application 311.

[0061] The refrigerator model information 4113 is information indicating the model of the refrigerator 1 used by the user P indicated by the associated account ID 4111 in the same record.

[0062] The refrigerator compartment interior image data 4114 is image data of an image showing the interior of the refrigerator compartment 11 of the refrigerator 1. Hereinafter, the image showing the interior of the refrigerator compartment 11 will be referred to as a "refrigerator compartment interior image" and given the symbol "RG." The refrigerator compartment interior image data 4114 indicates a refrigerator compartment interior image RG corresponding to the model indicated by the corresponding refrigerator model information 4113 in the same record. The refrigerator compartment interior image RG corresponds to an example of an interior image.

[0063] FIG. 9 is a diagram showing an example of the refrigerator interior image RG. The refrigerator compartment interior image RG shown in Fig. 9 includes a left door pocket image RG1, a refrigerator compartment front image RG2, and a right door pocket image RG3. The refrigerator compartment interior image RG shown in Fig. 9 includes a left door pocket image RG1, a refrigerator compartment front image RG2, and a right door pocket image RG3 arranged in this order from the left in the figure. The left door pocket image RG1 is an image showing the door pocket 111 formed in the left door 11A. The refrigerator compartment front image RG2 is an image of the refrigerator compartment 11 viewed from the front when the left door 11A and the right door 11B are in an open state. The right door pocket image RG3 is an image showing the door pocket 111 formed in the right door 11B.

[0064] The image region of the refrigerator interior image RG is divided into a plurality of areas. Each of these areas is called a "divided area" and is given the symbol "BA". The refrigerator interior image RG shown in FIG. 9 is divided into 15 divided areas BA. A different number is assigned to each divided area BA. In the following description, these numbers are called divided area numbers and are given the symbol "4118". Note that FIG. 9 illustrates for convenience which divided area number is assigned to which divided area BA, and the divided area number 4118 is not displayed in the refrigerator interior image RG displayed on the refrigerator home screen GM2 described later.

[0065] Returning to the explanation of FIG. 8, the in-stock / out-stock presence information 4115 is information indicating whether or not there has been new in-stock / outstock.

[0066] The food image data 4116 is image data of food. In this embodiment, the image represented by the food image data 4116 is an image captured by the refrigerator compartment camera 203.

[0067] The storage date and time information 4117 is information indicating the date and time when the food was stored in the refrigerator 1.

[0068] The refrigerator compartment food management record RR has a division area number 4118 for each food image data 4116 .

[0069] The memo input presence / absence information 4119 is information indicating whether or not a memo has been input.

[0070] The memo information 4130 is information indicating the input memo.

[0071] The deletion information 4131 is information indicating whether the associated sub-record is a sub-record to be deleted from the refrigerator food management record RR. A sub-record is a record stored in the refrigerator food management record RR, and includes food image data 4116, storage date and time information 4117, divided area number 4118, memo entry presence / absence information 4119, and memo information 4130.

[0072] The deletion date and time information 4132 is information indicating the date and time when the corresponding sub-record was targeted for deletion. When the deletion information 4131 indicates a deletion target, the information processing unit 402 described later deletes the sub-record and the deletion information 4131 and deletion date and time information 4132 corresponding to this sub-record from the refrigerator food management record RR at a predetermined timing.

[0073] FIG. 10 is a diagram showing one record stored in the withdrawn food management database 412. As shown in FIG. Drawer food management database 412 is a database that stores various information related to food in the drawer-type storage compartment. In the explanation with reference to Figs. 23-25, drawer food management database 412 is a database related to vegetable compartment 15. In the following description, one record stored in the withdrawn food management database 412 is referred to as a "drawn food management record" and is given the symbol "HR."

[0074] The drawn food management record HR has an account ID 4111 , a camera unit ID 4112 , update date and time information 4121 , upper case information 4122 , and lower case information 4124 .

[0075] The update date and time information 4121 is information indicating the date and time when the contents of the withdrawn food management record HR were updated.

[0076] The upper case information 4122 includes upper case image data 4123 , food image data 4116 , memo entry presence / absence information 4119 , and memo information 4130 .

[0077] Upper case image data 4123 is image data of an upper case image JCG, which is an image of upper case JCA. The upper case image JCG is an image captured by drawing camera 204. The upper case image JCG corresponds to an example of a first case image of the present disclosure.

[0078] In this embodiment, the image indicated by the food image data 4116 contained in the upper case information 4122 is an image cut out from the upper case image JCG.

[0079] The lower case information 4124 includes lower case image data 4125 , food image data 4116 , memo entry presence / absence information 4119 , and memo information 4130 .

[0080] The lower case image data 4125 is image data of a lower case image GCG, which is an image of the lower case GCA. The lower case image GCG is an image captured by the drawer camera 204. The lower case image GCG corresponds to an example of a second case image of the present disclosure. Hereinafter, when there is no need to distinguish between the upper case image JCG and the lower case image GCG, they will be referred to as "case images" and marked with "CG."

[0081] In this embodiment, the image indicated by the food image data 4116 included in the lower case information 4124 is an image cut out from the lower case image GCG.

[0082] 7, the server communication unit 41 is a communication interface having a configuration related to communication such as a wireless circuit and an antenna according to a predetermined communication standard, and communicates with devices connected to the global network GN according to the predetermined communication standard. In this embodiment, the server communication unit 41 communicates with the camera unit 2 and the terminal device 3.

[0083] As described above, the server control unit 40 functions as the server communication control unit 401 and the information processing unit 402 .

[0084] The server communication control unit 401 communicates with the camera unit 2 and the terminal device 3 via the server communication unit 41 .

[0085] The information processing unit 402 performs various types of information processing based on the information received by the server communication control unit 401 from the camera unit 2 and the terminal device 3 .

[0086] [1-1-4. Terminal device configuration] Next, the functional configuration of the terminal device 3 will be described. The terminal device 3 includes a terminal control unit 30, a terminal communication unit 31, and a touch panel 32. The touch panel 32 corresponds to an example of a display unit.

[0087] The terminal control unit 30 includes a terminal processor 300, which is a processor such as a CPU or an MPU, and a terminal storage unit 310. The terminal control unit 30 controls each unit of the terminal device 3 by the terminal processor 300 reading and executing a control program stored in the terminal storage unit 310. A food management application 311 is installed in the terminal device 3. The terminal processor 300 functions as an application execution unit 301 by reading and executing the food management application 311. The terminal processor 300 corresponds to an example of a computer of the present disclosure. The application execution unit 301 corresponds to an example of a display control unit of the present disclosure.

[0088] The terminal storage unit 310 has a memory for storing programs executed by the terminal processor 300 and data processed by the terminal processor 300. The terminal storage unit 310 stores a control program executed by the terminal processor 300, a food management application 311, an account ID 4111, and various other data. The terminal storage unit 310 has a non-volatile storage area. The terminal storage unit 310 may also have a volatile storage area and constitute a work area for the terminal processor 300.

[0089] The terminal communication unit 31 is a communication interface having a configuration of a wireless circuit, an antenna, etc. that conforms to a predetermined communication standard, and communicates with devices connected to the global network GN in accordance with the predetermined communication standard. In the present disclosure, a wireless communication standard is exemplified as the communication standard used by the terminal communication unit 31.

[0090] The touch panel 32 includes a display panel such as a liquid crystal display panel, and a touch sensor that is provided over the display panel or is integral with the display panel.

[0091] As described above, the terminal control unit 30 functions as the application execution unit 301. The application execution unit 301 transmits and receives various information to and from the food management server 4 via the terminal communication unit 31.

[0092] The application execution unit 301 also displays a user interface related to management of food in the refrigerator 1 on the touch panel 32. In the following description, this user interface will be referred to as the "application UI" and given the reference number "320." By displaying the application UI 320 on the touch panel 32, the application execution unit 301 provides the user P with various information related to the food in the refrigerator 1, and also receives various inputs related to the food in the refrigerator 1 from the user P.

[0093] [1-2. Operation] Next, the operation of each part related to the food control system 1000 will be described.

[0094] [1-2-1. Installation of the camera unit] Camera unit 2 is separate from refrigerator 1. Therefore, camera unit 2 is installed by user P on top surface 10B of refrigerator 1. As described above, position restricting member 205 restricts the position of camera unit 2 in the front-rear direction of refrigerator 1, so user P adjusts the position of camera unit 2 in the left-right direction of refrigerator 1 when installing camera unit 2. Below, several methods for adjusting the position of the camera unit 2 in the left-right direction of the refrigerator 1 will be illustrated.

[0095] [1-2-1-1. First position adjustment]

[0096] In the first position adjustment, the user P uses the application UI 320. In the first position adjustment, the application executing unit 301 causes the touch panel 32 to display the application UI 320 of the adjustment screen GM1.

[0097] FIG. 11 is a diagram showing an example of the adjustment screen GM1. The adjustment screen GM1 is provided with a photography result display area HA1. The photography result display area HA1 is a rectangular area that displays the photography results of either the refrigerator compartment camera 203 or the drawer camera 204. The photography result display area HA1 is provided with a line L1 extending in the vertical direction at approximately the center in the horizontal direction in the figure.

[0098] The adjustment screen GM1 includes adjustment method information J1. The adjustment method information J1 prompts the user to adjust the position so that the center in the left-right direction of the refrigerator 1 shown in the photography result display area HA1 coincides with a line L1 provided in the photography result display area HA1.

[0099] User P installs camera unit 2 in refrigerator 1 according to the adjustment method indicated in adjustment method information J1. This enables camera unit 2 to capture food items being put into or taken out of refrigerator 1 with each of refrigerator compartment camera 203 and drawer camera 204. Camera unit 2 also enables each distance measuring sensor to detect whether the corresponding door 11C or drawer 16A is open or closed.

[0100] In the first position adjustment, camera unit 2 transmits the image capturing results of either refrigerator compartment camera 203 or drawer camera 204 to terminal device 3. In the first position adjustment, camera unit 2 and terminal device 3 communicate with each other via food management server 4 or directly by short-range wireless communication or the like.

[0101] The camera unit communication control unit 221 of the camera unit 2 transmits the image capturing results of either the refrigerator compartment camera 203 or the drawer camera 204 to the terminal device 3. Then, the application execution unit 301 displays the received image capturing results in the image capturing result display area HA1.

[0102] [1-2-1-2. Second position adjustment] In the second position adjustment, user P uses center position recording paper CY.

[0103] FIG. 12 is a diagram showing an example of the central position recording paper CY. The central position recording paper CY is a paper on which a central mark MR2 is recorded, which indicates the central position in the left-right direction of the refrigerator 1. In addition to the central mark MR2, the central position recording paper CY also has recorded thereon a left edge mark MR1, which indicates the left edge position of the refrigerator 1 in the left-right direction, and a right edge mark MR3, which indicates the right edge position of the refrigerator 1 in the left-right direction.

[0104] The central position recording paper CY is packed together with the refrigerator 1 and delivered to the user P. The camera unit 2 may or may not be packed together with the central position recording paper CY and the refrigerator 1. The length of the central position recording paper CY packed together with the refrigerator 1 from the left end mark MR1 to the right end mark MR3 corresponds to the width of the refrigerator 1.

[0105] In the second position adjustment, user P uses the center position recording paper CY to determine where the center position in the left-right direction of refrigerator 1 is. For example, user P attaches the center position recording paper CY to refrigerator 1 so that the left end mark MR1 is the left end of refrigerator 1 and the right end mark MR3 is the right end of refrigerator 1, and uses center mark MR2 as a landmark to determine the center position in the left-right direction of refrigerator 1. User P then adjusts the position of camera unit 2 so that mark 207 on position restriction member 205 of camera unit 2 is at the determined center position.

[0106] The second position adjustment enables the camera unit 2 to perform various detections, similar to the first position adjustment.

[0107] [1-2-1-3. Third position adjustment] In the third position adjustment, a mark indicating the center position in the left-right direction of refrigerator 1 is placed at a predetermined position on refrigerator 1. In the third position adjustment, mark 207 on position restriction member 205 and a mark placed on refrigerator 1 are used.

[0108] In the third position adjustment, user P adjusts the position of camera unit 2 in the left-right direction of refrigerator 1 so that the position of the mark on refrigerator 1 matches the position of mark 207 on position restriction member 205.

[0109] By the third position adjustment, the camera unit 2 becomes able to perform various detections, similar to the first position adjustment.

[0110] Next, the operation of food management system 1000 after camera unit 2 is installed in refrigerator 1 will be described. [1-2-2. Recording-related operations] First, the operations relating to recording will be described. FIG. 13 is a flowchart FA showing the operation of the camera unit 2.

[0111] 13, the sensor section 22 outputs detection values ​​of each sensor to the camera unit control section 20. Then, in the operation shown in FIG 13, the detection value processing section 222 executes processing based on the various detection values ​​received from the sensor section 22.

[0112] The detection value processing unit 222 determines whether or not a person is present around the refrigerator 1 based on the detection value of the human presence sensor 212 (step SA1).

[0113] When the detection value processing unit 222 determines that a person is present around the refrigerator 1 (step SA1: YES), it outputs presence information indicating the presence of a person to the photography control unit 223 (step SA2).

[0114] When the presence / absence information indicating the presence of a person is input from detection value processing section 222, photography control section 223 causes refrigerator compartment camera 203 and drawer camera 204 to start photographing (step SA3).

[0115] Next, the detection value processing unit 222 determines whether any of the left door 11A, the right door 11B, and the drawers 14A, 15A is open based on the detection values ​​of the various sensors output from the sensor unit 22 (step SA4).

[0116] When it is determined that none of the left door 11A, the right door 11B, and the drawers 14A and 15A are open (step SA4: NO), the detection value processing unit 222 performs the process of step SA16.

[0117] If the detection value processing unit 222 determines that any of the left door 11A, right door 11B, or drawers 14A, 15A has been opened (step SA4: YES), it determines whether the door 11C has been opened or whether any of the drawers 14A, 15A has been opened (step SA5).

[0118] When it is determined that the door 11C is in an open state (step SA5: door), the detection value processing unit 222 outputs door open state information to the recording control unit 224 (step SA6).

[0119] Next, when the door open state information is input from the detection value processing unit 222, the recording control unit 224 starts recording the photographing results of the refrigerator compartment camera 203 (step SA7).

[0120] Next, the detection value processing unit 222 determines whether or not the door 11C is in the closed state (step SA8).

[0121] When it is determined that the door 11C is closed (step SA8: YES), the detection value processing unit 222 outputs door closed state information to the recording control unit 224 (step SA9).

[0122] Next, when the door closed state information is input from the detection value processing unit 222, the recording control unit 224 ends recording of the photographing result of the refrigerator compartment camera 203 (step SA10). As a result, one recording file 231 of the refrigerator compartment camera 203 is generated in the camera unit storage unit 230.

[0123] Returning to the explanation of step SA5, when the detection value processing unit 222 determines that either of the drawers 14A, 15A has been opened (step SA5: drawer), it outputs drawer open state information to the recording control unit 224 (step SA11).

[0124] Next, when the drawer closed state information is input from the detection value processing unit 222, the recording control unit 224 starts recording the image captured by the drawer camera 204 (step SA12).

[0125] Next, the detection value processing unit 222 determines whether or not either of the open drawers 14A, 15A has been closed (step SA13).

[0126] When the detection value processing unit 222 determines that either of the open drawers 14A, 15A has been closed (step SA13: YES), it outputs drawer closed state information to the recording control unit 224 (step SA14).

[0127] Next, when the drawer closed state information is input from the detection value processing unit 222, the recording control unit 224 ends recording of the image captured by the drawer camera 204 (step SA15). As a result, one recording file 231 of the drawer camera 204 is generated in the camera unit storage unit 230.

[0128] The detection value processing unit 222 determines whether or not there is a person around the refrigerator 1 (step SA16).

[0129] If the detection value processing unit 222 determines that a person is present around the refrigerator 1 (step SA16: YES), it repeats the processes from step SA4 onwards.

[0130] On the other hand, if the detection value processing unit 222 determines that no person is present around the refrigerator 1 (step SA16: NO), it outputs presence information indicating that no person is present to the photography control unit 223 (step SA17).

[0131] When the presence / absence information indicating that no person is present is input from detection value processing section 222, photography control section 223 causes refrigerator compartment camera 203 and drawer camera 204 to end photography (step SA18).

[0132] [1-2-3. Food detection operations] Next, the operation relating to the detection of food will be described. FIG. 14 is a flowchart showing the operation of the food management system 1000. In FIG. 14, a flowchart FB shows the operation of the camera unit 2, and a flowchart FC shows the operation of the food management server 4.

[0133] The food detection section 225 of the camera unit 2 determines whether or not the video recording file 231 has been stored in the camera unit storage section 230 (step SB1).

[0134] When food detection unit 225 determines that recording file 231 is stored in camera unit storage unit 230 (step SB1: YES), it determines whether stored recording file 231 is recording file 231 of refrigerator compartment camera 203 or recording file 231 of drawer camera 204 (step SB2). The determination in step SB2 is made based on information output by detection value processing unit 222.

[0135] When it is determined that the recorded file is 231 of the refrigerator compartment camera 203 (step SB2: refrigerator compartment camera), the food detection unit 225 detects food based on the stored recorded file 231 (step SB3).

[0136] In step SB3, for example, food detection unit 225 detects food from video recording file 231 of refrigerator compartment camera 203 based on feature quantities such as shape and color. Data necessary for food detection is stored in a storage area accessible by food detection unit 225.

[0137] The food detection unit 225 determines whether or not food has been detected from the video recording file 231 of the refrigerator compartment camera 203 (step SB4).

[0138] If food detection unit 225 determines that no food has been detected from video recording file 231 of refrigerator compartment camera 203 (step SB4: NO), it shifts the process to step SB10.

[0139] On the other hand, if food detection unit 225 detects food from video recording file 231 of refrigerator compartment camera 203 (step SB4: YES), it performs an entry / exit determination process (step SB5).

[0140] The in / out determination process is a process for determining whether the food detected in step SB3 is food that has been put into the refrigerator 1 or food that has been taken out of the refrigerator 1. In the in / out determination process, an in / out determination image HG is superimposed on the video image shown in the video recording file 231 of the refrigerator compartment camera 203.

[0141] FIG. 15 is a diagram for explaining the warehouse entry / exit determination image HG. The storing / retrieving determination image HG shown in FIG. 15 is superimposed on a moving image of a user P storing an egg pack into refrigerator 1.

[0142] The entry / exit determination image HG has a boundary line L2. The boundary line L2 has boundaries L21, L22, L23, L24, and L25. The boundary line L21 is a line that extends from the left end of the entry / exit judgment image HG to the right in the upper part of the image. The boundary line L22 is a line that connects the right end of the boundary line L21 to the left end of the boundary line L23. The boundary line L23 is a line that extends to the right from the left-right position of the right end of the boundary line L21 at approximately the center in the up-down direction of the entry / exit judgment image HG. The boundary line L24 is a line that connects the right end of the boundary line L23 to the left end of the boundary line L24. The boundary line L25 is a line that extends to the right from the left-right position of the right end of the boundary line L23 to the right end of the entry / exit judgment image HG in the upper part of the image.

[0143] The image area of ​​the entry / exit determination image HG is divided by a boundary line L2 into a concave area HG1 that is a concave area and a convex area HG2 that is a convex area.

[0144] Food detection unit 225 performs image analysis on the moving image on which the entry / exit determination image HG is superimposed to determine whether the food detected in step SB3 has moved from convex region HG2 to concave region HG1 or from concave region HG1 to convex region HG2. If food detection unit 225 determines that the food detected in step SB3 has moved from convex region HG2 to concave region HG1, it determines that the food detected in step SB3 has been entered into refrigerator 1. On the other hand, if food detection unit 225 determines that the food detected in step SB3 has moved from concave region HG1 to convex region HG2, it determines that the food detected in step SB3 has been removed from refrigerator 1.

[0145] Returning to the explanation of the flowchart FB in FIG. 14, the food detection unit 225 performs a divided area number identification process (step SB6).

[0146] The divided area number identification process is a process for identifying the divided area BA corresponding to the storage area into which the food detected in step SB3 was stored or removed, and identifying the divided area number 4118 assigned to the identified divided area BA.

[0147] For example, food detection unit 225 uses image analysis to identify the storage area of ​​refrigerator compartment 11 into which the food detected in step SB3 has been stored or removed. Also, for example, a mark identifying the storage area is attached to the end of each shelf 112 and inside each door pocket 111, and food detection unit 225 identifies the storage area based on the positional relationship between the mark and the detected food in the video. Next, food detection unit 225 determines which divided area BA in refrigerator interior image RG the identified storage area corresponds to, and identifies divided area number 4118 assigned to the identified divided area BA. The correspondence relationship between the storage area, divided area BA, and divided area number 4118 is stored as data in camera unit storage unit 230.

[0148] After completing the divided area number identification process, food detection unit 225 cuts out food images for each food detected in step SB3 from video recording file 231 of refrigerator compartment camera 203 (step SB7). Hereinafter, the food images cut out from the photographing results of camera 213 are referred to as "photographed food images" and are assigned the symbol "FG". The photographed food image FG corresponds to an example of a food image of the present disclosure.

[0149] Next, the food detection unit 225 generates first update information (step SB8).

[0150] The first update information generated in step SB8 includes the camera unit ID 4112 of the camera unit storage section 230. Moreover, the first update information generated in step SB8 includes date and time information indicating the date and time when the video recording file 231 of the refrigerator compartment camera 203 was generated. In addition, the first update information generated in step SB8 includes, for each food detected in step SB3, food image data 4116 indicating the photographed food image FG cut out in step SB7, in-stock / out-stock determination result information indicating the determination result of the in-stock / out-stock determination process, and the divided area number 4118 identified in the divided area number identification process.

[0151] Next, the camera unit communication control unit 221 transmits the first update information generated by the food detection unit 225 to the food management server 4 (step SB9).

[0152] The food detection section 225 deletes the recording file 231 from the camera unit storage section 230 (step SB10).

[0153] Returning to the explanation of step SB2, when food detection unit 225 determines that the recording file is 231 of drawer camera 204 (step SB2: drawer camera), it determines whether at least one of upper case JCA and lower case GCA shown in the video image of recording file 231 of drawer camera 204 has been pulled out by more than a predetermined threshold (step SB11).

[0154] For example, the food detection unit 225 distinguishes the cases CA in the moving image based on feature amounts such as shape and color. Then, the food detection unit 225 determines whether the upper case JCA and the lower case GCA have been pulled out from the refrigerator 1 by a predetermined threshold or more through image analysis of the video file 231. Also, for example, upper case JCA and lower case GCA are each provided with a mark for identifying case CA at a predetermined position, and food detection unit 225 distinguishes case CA in the moving image by reading this mark from the moving image. Then, food detection unit 225 calculates the movement distance of this mark in the moving image to determine whether or not the food has been pulled out of refrigerator 1 by a predetermined threshold or more.

[0155] If the food detection unit 225 determines that neither the upper case JCA nor the lower case GCA has been pulled out by the predetermined threshold value or more (step SB11: NO), it performs the process of step SB18.

[0156] On the other hand, if food detection unit 225 determines that at least one of upper case JCA and lower case GCA has been pulled out by the predetermined threshold or more (step SB11: YES), it cuts out a case image CG from video recording file 231 (step SB12).

[0157] In step SB12, food detection unit 225 cuts out upper case image JCG from video file 231 if only upper case JCA is pulled out by a predetermined threshold or more. In step SB12, food detection unit 225 cuts out lower case image GCG from video recording file 231 if only lower case GCA is pulled out by a predetermined threshold value or more. In step SB12, food detection unit 225 cuts out upper case image JCG and lower case image GCG from video recording file 231 if upper case JCA and lower case GCA are pulled out by a predetermined threshold value or more.

[0158] Next, the food detection unit 225 detects food from the cut-out case image CG (step SB13).

[0159] Next, the food detection unit 225 determines whether or not food has been detected from the case image CG (step SB14).

[0160] If the food detection unit 225 determines that no food can be detected (step SB14: NO), it performs the process of step SB16.

[0161] On the other hand, if the food detection unit 225 determines that the food has been detected (step SB14: YES), it cuts out a photographed food image FG for each detected food from the case image CG (step SB15).

[0162] Next, the food detection unit 225 generates second update information (step SB16).

[0163] When only the upper case image JCG is cut out, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage section 230, upper case image data 4123 indicating the cut-out upper case image JCG, and food image data 4116 indicating the photographed food image FG cut out from the upper case image JCG. Note that when a negative determination is made in step SB14, the second update information does not include the food image data 4116.

[0164] Furthermore, when only the lower case image GCG is cut out, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage section 230, lower case image data 4125 indicating the cut-out lower case image GCG, and food image data 4116 indicating the photographed food image FG cut out from the lower case image GCG. Note that, when a negative determination is made in step SB14, this second update information does not include the food image data 4116.

[0165] Furthermore, when the upper case image JCG and the lower case image GCG are cut out, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage unit 230, the upper case image data group, and the lower case image data group. The upper case image data group includes upper case image data 4123 indicating the cut out upper case image JCG, and food image data 4116 indicating the photographed food image FG cut out from the upper case image JCG. The lower case image data group includes lower case image data 4125 indicating the cut out lower case image GCG, and food image data 4116 indicating the photographed food image FG cut out from the lower case image GCG. Note that, when a negative determination is made in step SB14, the second update information does not include the food image data 4116.

[0166] Next, the camera unit communication control unit 221 transmits the second update information generated by the food detection unit 225 to the food management server 4 (step SB17).

[0167] Next, the food detection section 225 deletes the recording file 231 from the camera unit storage section 230 (step SB18).

[0168] As shown in the flowchart FC, the server communication control unit 401 of the food management server 4 determines whether or not either the first update information or the second update information has been received (step SC1).

[0169] If server communication control unit 401 determines that update information has been received (step SC1: YES), information processing unit 402 updates the database based on the update information received by server communication control unit 401 (step SC2).

[0170] Step SC2 will now be described in detail. When the update information received by the server communication control unit 401 is the first update information, the information processing unit 402 identifies the refrigerator food management record RR having the camera unit ID 4112 included in the first update information. Then, the information processing unit 402 updates the incoming / outgoing presence / absence information 4115 of the identified refrigerator food management record RR to information indicating "present."

[0171] Next, when the incoming / outgoing judgment result information included in the first update information indicates incoming, the information processing unit 402 generates a sub-record, associates blank deletion information 4131 and deletion date / time information 4132 with the generated sub-record, and stores the sub-record in the identified refrigerator food management record RR. The sub-record generated here includes food image data 4116 included in the first update information, storage date / time information 4117 indicating the same date and time as the date / time information included in the first update information, divided area number 4118 included in the first update information, blank memo entry presence / absence information 4119, and blank memo information 4130.

[0172] Furthermore, when the incoming / outgoing judgment result information included in the first update information indicates outgoing, the information processing unit 402 identifies a sub-record including food image data 4116 indicating a photographed food image FG that has a high degree of match with the photographed food image FG indicated by the food image data 4116 included in the first update information from the refrigerator food management record RR. This identification is performed for each food image data 4116 included in the first update information. Then, the information processing unit 402 updates the deletion information 4131 corresponding to the identified sub-record to information indicating that the sub-record is to be deleted. Furthermore, the information processing unit 402 updates the deletion date and time information 4132 corresponding to the identified sub-record to information indicating the current date and time as the deletion date and time.

[0173] Step SC2 will now be described in detail. When the update information received by the server communication control unit 401 is the second update information, the information processing unit 402 identifies the withdrawn food management record HR having the camera unit ID 4112 included in the second update information.

[0174] If the second update information is for the case where only the upper case image JCG has been cut out, the information processing unit 402 updates the upper case image data 4123 and food image data 4116 contained in the upper case information 4122 of the identified drawer food management record HR to the data contained in the second update information. If the second update information is for the case where only the lower case image GCG has been cut out, the information processing unit 402 updates the lower case image data 4125 and food image data 4116 contained in the lower case information 4124 of the identified drawer food management record HR to the data contained in the second update information. In the second update information when the upper case image JCG and the lower case image GCG are extracted, the information processing unit 402 appropriately updates the upper case image data 4123 and food image data 4116 held in the upper case information 4122 of the identified drawer food management record HR to data included in the upper case image data group. The information processing unit 402 also updates the lower case image data 4125 and food image data 4116 held in the lower case information 4124 of the identified drawer food management record HR to data included in the lower case image data group.

[0175] Furthermore, the information processing unit 402 updates the update date and time information 4121 of the identified withdrawn food management record HR to information indicating the date and time when the withdrawn food management record HR was updated.

[0176] [1-2-4. Actions related to app UI] Next, the operation related to the application UI 320 will be described. In this embodiment, the application UI 320 displays a screen related to the refrigerator compartment 11 and a screen related to the drawer-type storage compartment.

[0177] [1-2-4-1. Refrigerator related screens]

[0178] The screens related to the refrigerator compartment 11 include a refrigerator compartment home screen GM2, a refrigerator compartment list screen GM3, and a food information addition list screen GM4.

[0179] FIG. 16 is a diagram showing an example of the refrigerator compartment home screen GM2.

[0180] The refrigerator compartment home screen GM2 has a food image display area HA2. The food image display area HA2 is an area for displaying the photographed food image FG. The food image display area HA2 displays the refrigerator compartment interior image RG, and displays the photographed food image FG superimposed on the refrigerator compartment interior image RG. Each photographed food image FG is displayed superimposed on a divided area BA indicated by a divided area number 4118 corresponding to the food image data 4116 in the refrigerator compartment interior image RG.

[0181] Food image display area HA2 in Fig. 16 displays refrigerator interior image RG shown in Fig. 9. For example, in refrigerator food management record RR, food image data 4116 showing photographed food image FG1 in Fig. 16 is associated with divided area number 4118 of "4." In this case, photographed food image FG1 is displayed superimposed on divided area BA to which divided area number 4118 of "4" is assigned.

[0182] The default food image display area HA2 displays the refrigerator interior image RG with the left door pocket image RG1 and the right door pocket image RG3 cut off. The food image display area HA2 accepts a scroll operation in the left-right direction in the figure from the user P. The food image display area HA2 changes the display range of the refrigerator interior image RG according to the scroll amount and scroll direction of the accepted scroll operation.

[0183] For example, suppose the scroll direction is rightward and the scroll amount is such that the left end of the left door pocket image RG1 is displayed. In this case, the food image display area HA2 displays the refrigerator compartment interior image RG with the right door pocket image RG3 completely cut off. The photographed food image FG and the memo input icon MIC move in accordance with the change in the display range of the refrigerator compartment interior image RG, in other words, with the movement of the superimposed divided area BA. The memo input icon MIC will be described later.

[0184] When multiple photographed food images FG are displayed in one divided area BA, the food image display area HA2 displays them side by side in the left-right direction of the figure. The food image display area HA2 can simultaneously display up to five photographed food images FG side by side in one divided area BA. The food image display area HA2 accepts a scroll operation from the user P when there are six or more photographed food images FG to be displayed in one divided area BA. When the one divided area BA accepts a scroll operation, the one divided area BA displays the sixth and subsequent photographed food images FG. When there are six or more photographed food images FG to be displayed in one divided area BA and a scroll operation in the left-right direction is accepted, the food image display area HA2 displays the sixth and subsequent photographed food images FG according to the amount of scroll while maintaining the maximum number that can be displayed side by side. At this time, the food image display area HA2 hides the first to fifth photographed food images FG that were displayed, or displays the first and subsequent photographed food images FG after the sixth and subsequent photographed food images FG, depending on the amount of scrolling and the number of photographed food images FG to be displayed. Note that the memo input icon MIC moves following the movement of the associated photographed food image FG. In other words, when the associated photographed food image FG is hidden by the scrolling operation, the memo input icon MIC is also hidden.

[0185] Each photographed food image FG displayed in the food image display area HA2 accepts input of a memo. The memo is information that the user P wants to leave for the food in the refrigerator 1.

[0186] FIG. 17 is a diagram showing a procedure for inputting a memo.

[0187] On the refrigerator compartment home screen GM2, the user P performs a predetermined touch operation, such as a long press operation, on the photographed food image FG for which the user P wants to input a memo. When the application execution unit 301 receives the predetermined touch operation on one photographed food image FG, it displays a memo edit image MHG in association with the photographed food image FG on which the predetermined touch operation was performed.

[0188] The memo edit image MHG has a memo input area MNA, a memo save button B1, a memo delete button B2, and a food delete button B3.

[0189] The memo input area MNA is an area for inputting a memo. When the memo input area MNA is touched, the application execution unit 301 displays a software keyboard on the touch panel 32. Then, the application execution unit 301 inputs a memo corresponding to the operation on the software keyboard into the memo input area MNA.

[0190] The memo save button B1 is a software button for saving a memo input in the memo input area MNA. The memo save button B1 cannot be touched if no memo is input in the memo input area MNA. When the memo save button B1 is touched, the application execution unit 301 displays the photographed food image FG to which the memo is input in a display mode different from the display mode of the photographed food image FG to which the memo is not input on the screen related to the refrigerator compartment 11. In this embodiment, the application execution unit 301 displays the frame of the photographed food image FG to which the memo is input in a different color, and displays the photographed food image FG to which the memo is input in association with a memo input icon MIC, which is an icon indicating that a memo has been input. The memo input icon MIC can be touched. When the memo input icon MIC is touched, the input memo is displayed in the application UI 320. The memo input icon MIC corresponds to an example of a memo input image of the present disclosure.

[0191] The memo erase button B2 is a button for erasing the input memo. When the memo erase button B2 is touched, the memo input in the photographed food image FG is erased.

[0192] The food delete button B3 is a button for deleting the photographed food image FG from the food image display area HA2.

[0193] Returning to the explanation of the refrigerator compartment home screen GM2 in Fig. 16, the food image display area HA2 has a list display button B4. The list display button B4 is a touch-operable software button.

[0194] When the list display button B4 is touched, the application UI 320 transitions to a refrigerator compartment list screen GM3.

[0195] The refrigerator compartment list screen GM3 includes a storage list screen GM31, a new addition list screen GM32, and a deletion target list screen GM33.

[0196] The accommodation list screen GM31 will be described. FIG. 18 is a diagram showing the accommodation list screen GM31.

[0197] The storage list screen GM31 has a stored food information display area HA3. The stored food information display area HA3 displays a list of first food information J21 for each food stored in the refrigerator compartment 11. The first food information J21 includes a photographed food image FG, storage date and time information 4117, and divided area information J3. The divided area information J3 is a refrigerator compartment interior image RG in which the divided areas BA are displayed in color.

[0198] When a memo has been entered into the photographed food image FG of the first food information J21, the frame is displayed in a different color and a memo entry icon MIC is associated with it. Note that the stored food information display area HA3 allows scrolling in the vertical direction of the figure. By scrolling in the stored food information display area HA3, the user P can display the first food information J21 that cannot be displayed in the stored food information display area HA3.

[0199] The accommodation list screen GM31 has a first switching button B5 and a second switching button B6.

[0200] The first switching button B5 is a software button for switching the order of the first food information J21 displayed in the stored food information display area HA3 to the next order. That is, when the first switching button B5 is touched, the stored food information display area HA3 displays the first food information J21 having the photographed food image FG to which a memo has been inputted, higher than the first food information J21 having the photographed food image FG to which no memo has been inputted. Displaying higher means displaying in a manner that reduces the amount of scrolling required when displaying. Also, when there are multiple pieces of first food information J21 having photographed food images FG to which a memo has been inputted, the stored food information display area HA3 displays the food information J2 having the oldest storage date and time indicated by the storage date and time information 4117 higher. Also, when there are multiple pieces of first food information J21 having photographed food images FG to which no memo has been inputted, the stored food information display area HA3 displays the first food information J21 having the oldest storage date and time indicated by the storage date and time information 4117 higher. The accommodation list screen GM31 on the left side of FIG. 18 shows the accommodation list screen GM31 when the first switching button B5 is touched.

[0201] The second switching button B6 is a software button for switching the order of the first food information J21 displayed in the stored food information display area HA3 to the next order. That is, when the second switching button B6 is touched, the contained food information display area HA3 displays the first food information J21 in each divided area BA. The order of the divided areas BA may be an order predetermined by the user P. The contained food information display area HA3 displays the first food information J21 in one divided area BA in the same order as when the first switching button B5 is touched. The accommodation list screen GM31 on the right side of FIG. 18 shows the accommodation list screen GM31 when the second switching button B6 is touched.

[0202] The accommodation list screen GM31 has an accommodation list display button B7, a new addition list display button B8, and a deletion target list display button B9.

[0203] The accommodation list display button B7 is a software button for switching the screen displayed by the application UI320 to the accommodation list screen GM31. The new addition list display button B8 is a software button for switching the screen displayed by the application UI320 to a new addition list screen GM32. The deletion target list display button B9 is a software button for switching the screen displayed by the application UI320 to a deletion target list screen GM33.

[0204] FIG. 19 is a diagram showing an example of the new addition list screen GM32. The newly added food list screen GM32 has a newly added food information display area HA4. The newly added food information display area HA4 displays a list of first food information J21 related to foods newly stored in the refrigerator compartment 11. Foods newly stored in the refrigerator compartment 11 refer to foods that have been stored within a predetermined period of time (e.g., three days) from the current date and time. The newly added food information display area HA4 displays the first food information J21 of foods corresponding to newly stored foods in order of the most recent storage date and time indicated by the storage date and time information 4117.

[0205] Each of the first food information J21 displayed in the newly added food information display area HA4 receives a predetermined touch operation on the photographed food image FG. On the newly added food list screen GM32, the first edited image HNG1 is displayed in association with the photographed food image FG that has been touched.

[0206] The first edited image HNG1 has a food delete button B10, an area correction button B11, and a back button B12. The food delete button B10 is a software button for, when touched, marking the food indicated by the photographed food image FG associated with the first edited image HNG1 as a target for deletion from the refrigerator compartment home screen GM2. The area correction button B11 is a software button for correcting the divided area BA indicated by the divided area information J3 when touched. The back button B12 is a software button for stopping the display of the first edited image HNG1.

[0207] The newly added food information display area HA4 has an Add Food Information button B13 at the end of the list of first food information J21. The Add Food Information button B13 is a software button for manually adding new first food information J21. The procedure for adding the first food information J21 will be described with reference to FIG.

[0208] FIG. 20 is a diagram illustrating the procedure for adding the first food information J21.

[0209] When the Add Food Information button B13 is touched, a food information addition list screen GM4 is displayed in the application UI320. The food information addition list screen GM4 has a plurality of add buttons B14, which are software buttons. The food information addition list screen GM4 also has a blank image BG and a food name input area SNA for inputting a food name. When the blank image BG is touched, an image showing the food name input in the food name input area SNA appears in the blank image BG.

[0210] When the Add button B14 is touched on the food information addition list screen GM4, a refrigerator interior image RG is displayed on the app UI320. Here, the displayed refrigerator interior image RG is an image in which each divided area BA is assigned a divided area number 4118, and each divided area BA can be selected by a touch operation. When a divided area BA is selected on the refrigerator interior image RG, the next first food information J21 is added on the app UI320.

[0211] That is, the first food information J21 to be added includes an image of the food corresponding to the touched Add button B14, storage date and time information 4117 indicating the date and time when the first food information J21 was added as the storage date and time, and divided area information J3 indicating the selected divided area BA. The image of the food indicated by the added first food information J21 may be an icon. In this embodiment, it is assumed that the image superimposed on the refrigerator interior image RG on the refrigerator home screen GM2 is a shadow food image FG, but when the first food information J21 is added, an icon is displayed on the refrigerator interior image RG together with the photographed food image FG.

[0212] The procedure for adding the first food information J21 may include a procedure for the user P to input the storage date and time. In this case, the added first food information J21 includes storage date and time information 4117 that indicates the storage date and time input by the user P.

[0213] Returning to the explanation of FIG. 19, the new addition list screen GM32 has an accommodation list display button B7, a new addition list display button B8, and a deletion target list display button B9.

[0214] FIG. 21 is a diagram showing an example of the deletion target list screen GM33.

[0215] The deletion target list screen GM33 has a deletion target food information display area HA5. The deletion target food information display area HA5 displays a list of second food information J22 for each food to be deleted. Note that the foods to be deleted refer to foods to be deleted from the refrigerator compartment home screen GM2 and foods determined to have been removed from the refrigerator compartment 11. The second food information J22 includes photographed food images FG, deletion date and time information 4132, and divided area information J3. The deletion target list screen GM33 displays the second food information J22 in order of most recent deletion date and time.

[0216] Each of the second food information J22 displayed in the deletion target food information display area HA5 receives a predetermined touch operation on the photographed food image FG. On the deletion target list screen GM33, the second edited image HNG2 is displayed in association with the photographed food image FG that has been touched.

[0217] The second edited image HNG2 has a delete cancel button B15 and a back button B16. The cancel deletion button B15 is a software button for setting the photographed food image FG associated with the second edited image HNG2 as the image to be displayed on the refrigerator compartment home screen GM2. The back button B16 is a software button for stopping the display of the second edited image HNG2.

[0218] [1-2-4-2. Operation of food control system] Next, the operation of food management system 1000 when displaying a screen related to refrigerator compartment 11 will be described.

[0219] Fig. 22 is a flowchart showing the operation of the food management system 1000. In Fig. 22, a flowchart FD shows the operation of the terminal device 3, and a flowchart FE shows the operation of the food management server 4.

[0220] The application execution unit 301 of the terminal device 3 determines whether or not to request the refrigerator compartment food management record RR from the food management server 4 (step SD1).

[0221] For example, when the application executing unit 301 receives a touch operation on the application UI 320 instructing to display the refrigerator compartment home screen GM2, the application executing unit 301 makes an affirmative determination in step SD1. Also, for example, when a touch operation instructing to update the display is performed on the refrigerator compartment home screen GM2, the application executing unit 301 makes an affirmative determination in step SD1.

[0222] When the application execution unit 301 determines that the refrigerator management record RR is requested (step SD1: YES), the application execution unit 301 transmits first record request information requesting the refrigerator management record RR to the food management server 4 (step SD2). The first record request information transmitted in step SD2 includes the account ID 4111 stored in the device storage unit 310.

[0223] As shown in a flowchart FF, the server communication control unit 401 of the food management server 4 receives the first record request information via the server communication unit 41 (step SE1).

[0224] Next, the information processing unit 402 identifies the refrigerator food management record RR including the account ID 4111 included in the first record request information received by the server communication control unit 401 from the refrigerator compartment food management database 411 (step SE2).

[0225] Then, the server communication control unit 401 transmits the refrigerator food management record RR identified by the information processing unit 402 to the terminal device 3 (step SE3). The information processing unit 402 updates the in-stock / out-stock presence / absence information 4115 of the transmitted refrigerator food management record RR to information indicating "absent."

[0226] Referring to the flowchart FD, when the application execution unit 301 of the terminal device 3 receives the refrigerator food management record RR from the food management server 4 (step SD3), it generates a screen related to the refrigerator compartment 11 based on this refrigerator food management record RR (step SD4).

[0227] Step SD4 will now be described in detail. The application execution unit 301 generates a refrigerator home screen GM2 that displays, in the food image display area HA2, the refrigerator interior image RG indicated by the refrigerator interior image data 4114 included in the refrigerator food management record RR. In addition, the application execution unit 301 generates a refrigerator home screen GM2 in which the photographed food image FG indicated by the food management image data 4116 is superimposed on the divided area BA indicated by the divided area number 4118 for each sub-record. Furthermore, if the memo input presence / absence information 4119 indicates that a memo is present, the application execution unit 301 generates, for each sub-record, a refrigerator home screen GM2 that displays a memo input icon MIC in association with the photographed food image FG.

[0228] Step SD4 will now be described in detail. The application execution unit 301 generates first food information J21 for each food image data 4116. This first food information J21 includes the photographed food image FG indicated by the food image data 4116, storage date and time information 4117 corresponding to this food image data 4116, and divided area information J3 indicated in color by the divided area BA indicated by the divided area number 4118. Then, the application execution unit 301 generates a storage list screen GM31 that displays a list of first food information J21 related to sub-records that are associated with deletion information 4131 that does not indicate a deletion target. In addition, the application execution unit 301 generates a new addition list screen GM32 that displays a list of first food information J21 for which the storage date and time information 4117 indicates a date and time within a predetermined period from the current time, among the first food information J21 displayed on the storage list screen GM31.

[0229] Step SD4 will now be described in detail. The application executing unit 301 generates second food information J22 for each piece of food image data 4116. The second food information J22 includes the photographed food image FG indicated by the food image data 4116, deletion date and time information 4132 corresponding to the food image data 4116, and divided area information J3 in which the divided area BA indicated by the divided area number 4118 is indicated in color. The application executing unit 301 then generates a deletion target list screen GM33 that displays a list of the generated second food information J22.

[0230] Returning to the explanation of the flowchart FD, the application execution unit 301 displays a screen related to the refrigerator compartment 11 in response to the touch operation (step SD5). If the incoming / outgoing presence / absence information 4115 of the refrigerator compartment food management record RR received in step SD3 indicates "present," the application execution unit 301 displays a pop-up on the refrigerator compartment home screen GM2 to inform the user of new incoming / outgoing items when displaying the refrigerator compartment home screen GM2 for the first time in step SD5.

[0231] Next, the operation of the food management system 1000 when an operation is performed on the screen related to the refrigerator compartment 11 will be described.

[0232] When the memo save button B1 is touched, the application execution unit 301 transmits information for updating the sub-record related to the photographed food image FG to the food management server 4. This information includes memo input presence / absence information 4119 indicating the presence of a memo, and memo information 4130 indicating the input memo. The food management server 4 updates the sub-record appropriately based on the received information.

[0233] When the memo delete button B2 is touched, the application execution unit 301 transmits information for updating the sub-record related to the photographed food image FG to the food management server 4. This information includes memo entry presence / absence information 4119 indicating that no memo has been entered. The food management server 4 updates the sub-record appropriately based on the received information.

[0234] When the food deletion buttons B3, B10 are touched, the application execution unit 301 transmits information for updating the sub-record related to the photographed food image FG to be operated, to the food management server 4. This information includes deletion information 4131 indicating the food to be deleted. The food management server 4 appropriately updates the sub-record based on the received information.

[0235] When the area correction button B11 is touched to correct the divided area BA, the application execution unit 301 transmits information for updating the sub-record for the photographed food image FG to be operated, to the food management server 4. This information includes the divided area number 4118 of the corrected divided area BA. The food management server 4 updates the sub-record appropriately based on the received information.

[0236] 20, the application execution unit 301 transmits information for adding a sub-record related to the added first food information J21 to the food management server 4. The food management server 4 updates the sub-record appropriately based on the received information.

[0237] When the cancel deletion button B15 is touched, the application execution unit 301 transmits information for updating the sub-record for the photographed food image FG to be operated, to the food management server 4. This information includes deletion information 4131 indicating blank. The food management server 4 appropriately updates the sub-record based on the received information.

[0238] [1-2-4-3. Screen related to the drawer-type containment room] Next, a screen related to the drawer-type storage compartment will be described. In the description using Fig. 23-Fig. 25, the vegetable compartment 15 is exemplified as the drawer-type storage compartment. In this description, the database shown in Fig. 10 is a database that stores various information related to the food in the vegetable compartment 15.

[0239] The screens related to the vegetable compartment 15 include a vegetable compartment home screen GM5 and a food image list screen GM6.

[0240] FIG. 23 is a diagram showing an example of the vegetable compartment home screen GM5.

[0241] The vegetable compartment home screen GM5 has a case image display area HA6. The case image display area HA6 is an area for displaying the case image CG. The case image display area HA6 displays the upper case image JCG superimposed on the lower case image GCG. In the default case image display area HA6, the upper case image JCG is superimposed on the entire image area of ​​the lower case image GCG, and only the upper case image JCG is displayed.

[0242] The case image display area HA6 accepts a scroll operation in the up and down directions in the figure. The case image display area HA6 changes the display range of the upper case image JCG according to the scroll amount and scroll direction of the accepted scroll operation. For example, if the scroll direction of the accepted scroll operation is upward, the case image display area HA6 displays a partially cut-off upper case image JCG. When the upper case image JCG moves upward in the case image display area HA6, the lower case image GCG on which the upper case image JCG was superimposed is displayed.

[0243] Each case image CG displayed in the case image display area HA6 accepts the input of a memo. The procedure for inputting a memo is the same as that for the refrigerator compartment 11, except that the image operated by the user P is the photographed food image FG or the case image CG. As shown in Fig. 23, the memo input icon MIC associated with the upper case image JCG moves following the movement of the upper case image JCG.

[0244] The vegetable compartment home screen GM5 has a list display button B17. The list display button B17 is a touch-operable software button.

[0245] When the list display button B17 is touched, a food image list screen GM6 is displayed on the application UI320.

[0246] The vegetable compartment home screen GM5 may have a switch button for switching the case image CG displayed in the case image display area HA6 to the upper case image JCG or the lower case image GCG. In this configuration, the application execution unit 301 switches the case image CG to be displayed to the upper case image JCG or the lower case image GCG in response to an operation on the switch button.

[0247] FIG. 24 is a diagram showing an example of the food image list screen GM6.

[0248] The food image list screen GM6 has update date and time information 4121. The food image list screen GM6 of Fig. 24 has update date and time information 4121 indicating the update date and time of "12 / 1 10:05".

[0249] The food image list screen GM6 displays a list of photographed food images FG for each case. The photographed food image FG displayed in association with the upper case JCA item is an image cropped from the upper case image JCG. The photographed food image FG displayed in association with the lower case GCA item is an image cropped from the lower case image GCG.

[0250] [1-2-4-5. Operation of food control system] Next, the operation of food management system 1000 when displaying a screen related to vegetable compartment 15 will be described.

[0251] 25 is a flowchart showing the operation of the food management system 1000. In Fig. 11, a flowchart FF shows the operation of the terminal device 3, and a flowchart FJ shows the operation of the food management server 4.

[0252] The application execution unit 301 of the terminal device 3 determines whether or not to request the withdrawal management record HR from the food management server 4 (step SF1).

[0253] For example, when a touch operation instructing to display the vegetable compartment home screen GM5 is received on the application UI 320, the application executing unit 301 makes an affirmative determination in step SF1. Also, for example, when a touch operation instructing to update the display is performed on the vegetable compartment home screen GM5, the application executing unit 301 makes an affirmative determination in step SF1.

[0254] When the application execution unit 301 determines that the withdrawal management record RR is requested (step SF1: YES), it transmits second record request information requesting the withdrawal management record HR to the food management server 4 (step SF2). The second record request information transmitted in step SF2 includes the account ID 4111 stored in the device storage unit 310.

[0255] Referring to flowchart FJ, the server communication control unit 401 of the food management server 4 receives the second record request information (step SJ1).

[0256] Next, the information processing unit 402 identifies the withdrawal management record HR that includes the account ID 4111 included in the second record request information received by the server communication control unit 401 from the withdrawn food management database 412 (step SJ2).

[0257] Then, the server communication control unit 401 transmits the withdrawal management record HR identified by the information processing unit 402 to the terminal device 3 (step SJ3).

[0258] Referring to flowchart FJ, when the application execution unit 301 of the terminal device 3 receives the pulled food management record HR from the food management server 4 (step SF3), it generates a screen related to the vegetable compartment 15 (step SF4).

[0259] Step SF4 will now be described in detail. The application execution unit 301 generates a vegetable compartment home screen GM5 having a case image display area HA6 in which a lower case image GCG indicated by the lower case image data 4125 included in the drawer food management record HR is superimposed with an upper case image JCG indicated by the upper case image data 4123 included in the same record. Furthermore, when the memo input presence / absence information 4119 indicates that there is a memo, the application executing unit 301 generates a vegetable compartment home screen GM5 that displays a memo input icon MIC in association with the corresponding case image CG.

[0260] Step SF4 will now be described in detail. The application execution unit 301 generates a food image list screen GM6 on which the photographed food images FG indicated by the food image data 4116 are arranged for each case CA.

[0261] Returning to the explanation of the flowchart FF, the application executing unit 301 displays a screen related to the vegetable compartment 15 in response to a touch operation (step SF5).

[0262] Next, the operation of food management system 1000 when an operation is performed on a screen related to vegetable compartment 15 will be described.

[0263] When the memo save button B1 is touched, the application execution unit 301 transmits information for updating the memo input presence / absence information 4119 and the memo information 4130 to the food management server 4. This information includes information indicating whether the operation target is the upper case image JCG or the lower case image GCG, the memo input presence / absence information 4119 indicating that a memo exists, and the memo information 4130 indicating the entered memo. The food management server 4 appropriately updates the withdrawn food management record HR based on the received information.

[0264] When the memo delete button B2 is touched, the application execution unit 301 transmits information for updating the memo input presence / absence information 4119 and the memo information 4130 to the food management server 4. This information includes information indicating whether the memo is to be deleted from the upper case image JCG or the lower case image GCG, and the memo input presence / absence information 4119 indicating that there is no memo. The food management server 4 appropriately updates the withdrawn food management record HR based on the received information.

[0265] 23-25 ​​is an example of a drawer-type storage compartment, such as the vegetable compartment 15. However, the screen displays and operations of the food management system 1000 shown in Fig. 23-25 ​​may also be performed in the same manner for the freezer compartment 14. In this case, the database shown in Fig. 10 is a database that stores various information related to the food in the freezer compartment 14.

[0266] [1-3. Effects] As described above, camera unit 2 comprises main body 201 installed on top surface 10B of refrigerator 1, and imaging member 202 extending from main body 201 to the front of the refrigerator 1 and equipped with camera 213 that captures images from the upper front to the lower front of the refrigerator 1. Main body 201 comprises position restricting member 205 that abuts front edge 10A of the refrigerator 1 and restricts the position of imaging member 202 in the front-to-rear direction of the refrigerator 1.

[0267] According to this, the position of the imaging member 202 in the front-to-rear direction of the refrigerator 1 is restricted by the position restricting member 205, so that the camera 213 is restricted to a position where it can capture images from the upper to lower front of the refrigerator 1. Therefore, food items being taken in and out of the refrigerator 1 can be accurately included in the images captured by the camera 213, and the food items being taken in and out of the refrigerator 1 can be accurately photographed.

[0268] The position restricting member 205 is a position restricting piece that extends downward from the bottom surface 206 of the main body 201 in the installed state.

[0269] According to this, the position of camera unit 2 can be regulated by utilizing the gap formed between main box body 10 and door 11C. Therefore, camera unit 2 can be installed in an existing refrigerator 1, so that food items entering and leaving refrigerator 1 can be photographed with high accuracy even in an existing refrigerator 1.

[0270] The position restricting member 205 is formed to be shorter than the length to the packing provided between the refrigerator 1 and the door 11C of the refrigerator 1.

[0271] According to this, the position of the camera unit 2 can be regulated by utilizing the gap formed between the main box body 10 and the door 11C without impeding the cooling effect of the refrigeration compartment 11. Therefore, food entering and leaving the refrigerator 1 can be photographed with high accuracy without impeding the cooling effect of the refrigeration compartment 11.

[0272] The position restriction member 205 extends in the width direction of the main body 201 .

[0273] This ensures a large contact area with the main box 10, stabilizing the installation position of the camera unit 2. As a result, food items being taken in and out of the refrigerator 1 can be photographed with higher accuracy.

[0274] The position restricting member 205 has a mark 207 for adjusting the position of the camera unit 2 in the width direction of the refrigerator 1.

[0275] According to this, by using mark 207 as a guide, user P can easily set up camera unit 2 to the left or right of refrigerator 1. Therefore, user P can easily set up camera unit 2 so that food items being taken in and out of refrigerator 1 can be photographed with high accuracy.

[0276] The position restriction member 205 has a mark 207 at the center in the width direction of the main body 201 .

[0277] According to this, by using mark 207 as a guide, user P can easily install camera unit 2 in the left-right center of refrigerator 1. Therefore, user P can easily install camera unit 2 so that food items being taken in and out of refrigerator 1 can be photographed with greater accuracy.

[0278] As described above, the camera unit 2 includes a camera 213 that captures video of the refrigerator 1 from above the front of the refrigerator 1, a capture control unit 223 that causes the camera 213 to start capturing images when the human presence sensor 212 that detects whether or not there is a person around the refrigerator 1 detects that there is a person around the refrigerator 1, and a recording control unit 224 that starts recording the images captured by the camera 213 at a predetermined timing after the capture control unit 223 has caused the camera 213 to start capturing images. Furthermore, the control method of camera unit 2 is such that, when human presence sensor 212, which detects whether or not there is a person around refrigerator 1, detects that there is a person around refrigerator 1, camera 213 is made to start capturing images, and at a predetermined timing after camera 213 has started capturing images, recording of the captured images by camera 213 is started.

[0279] According to the camera unit 2 and the control method for the camera unit 2, when the motion sensor 212 detects the presence of a person near the refrigerator 1, the camera 213 starts capturing images, which increases the likelihood that food items being taken in or out of the refrigerator 1 before the door 11C or drawer 16A of the refrigerator 1 is opened will be included in the images captured by the camera 213. This makes it possible to capture images of food items being taken in or out of the refrigerator 1 with high accuracy.

[0280] After the photography control unit 223 has caused the camera 213 to start photographing, the recording control unit 224 starts recording the photographed result of the camera 213 when the door 11C or the drawer 16A of the refrigerator 1 is opened.

[0281] According to this, since recording starts at a timing when there is a high possibility that food will start to be taken in or out of refrigerator 1, it is possible to prevent unnecessary recording in a situation where food is not being taken in or out of refrigerator 1. Furthermore, since recording control unit 224 can prevent an increase in the data capacity of recording file 231, it is possible to prevent an increase in the processing target of food detection unit 225, which in turn prevents an increase in the processing time and processing load of food detection unit 225.

[0282] After starting recording, if the door 11C or the drawer 16A of the refrigerator 1, which was open, is closed, the recording control unit 224 stops recording.

[0283] According to this, recording ends when there is a high possibility that food has been taken in or taken out, which prevents unnecessary recording when no food is being taken in or out of refrigerator 1. Furthermore, recording control unit 224 can prevent an increase in the data capacity of recording file 231, which prevents an increase in the processing target of food detection unit 225 and an increase in the processing time and processing load of food detection unit 225.

[0284] When the recording control unit 224 stops recording, it generates one recording file 231 containing the captured images from when the door 11C or the drawer 16A of the refrigerator 1 is opened to when it is closed.

[0285] According to this, the shooting results from when the door 11C or the drawer 16A of the refrigerator 1 is opened to when it is closed are generated as one recording file 231, so that the first recording file 231 can be generated quickly while preventing an increase in the data capacity of the first recording file 231. Also, since the first recording file 231 can be generated quickly while preventing an increase in the data capacity of the first recording file 231, the food detection unit 225 can start processing related to the recording file 231 quickly after food is stored in or taken out of the refrigerator 1.

[0286] If the human presence sensor 212 detects that no human is present around the refrigerator 1 after the recording control unit 224 has stopped recording, the imaging control unit 223 causes the camera 213 to end imaging.

[0287] According to this, when the human presence sensor 212 detects that no human is present around the refrigerator 1, the camera 213 is made to stop capturing images, preventing the camera 213 from capturing images when no food is being taken in or out of the refrigerator 1. This makes it possible to accurately capture images of food being taken in and out of the refrigerator 1 while preventing an increase in power consumption.

[0288] The camera unit 2 includes a food detection unit 225 that detects food entering or leaving the refrigerator 1 based on the recording results of the recording control unit 224.

[0289] According to this, since the food items taken in and out of refrigerator 1 can be detected based on the photographing results obtained by photographing the food items with high accuracy, the food items taken in and out of refrigerator 1 can be detected with high accuracy.

[0290] As described above, the food management application 311 causes the terminal processor 300 of the terminal device 3 equipped with the touch panel 32 to function as an application execution unit 301 that displays an image of the interior of the refrigerator compartment RG using the touch panel 32 and displays a photographed food image FG in a divided area BA in the displayed image of the interior of the refrigerator compartment RG that corresponds to the storage area of ​​the food stored in the refrigerator compartment 11. The terminal device 3 also includes a touch panel 32 and an application execution unit 301 that displays an image of the interior of the refrigerator compartment RG using the touch panel 32 and displays a photographed food image FG in a divided area BA in the displayed image of the interior of the refrigerator compartment RG that corresponds to the storage area of ​​the food stored in the refrigerator compartment 11. Further, the control method of the terminal device 3 displays a refrigerator compartment interior image RG, and displays a photographed food image FG in a divided area BA corresponding to the storage area of ​​the food stored in the refrigerator compartment 11 in the displayed refrigerator compartment interior image RG.

[0291] According to the food management application 311, the terminal device 3, and the control method for the terminal device 3, the photographed food image FG is displayed in a divided area BA corresponding to the storage area of ​​the food stored in the refrigerator in an image showing the inside of the refrigerator compartment. Therefore, the user P can intuitively grasp what kind of food is stored in the refrigerator compartment 11.

[0292] The photographed food image FG is an image captured by the refrigerator compartment camera 203.

[0293] This allows user P to be provided with photographed images of the foods actually stored in refrigerator 1, allowing user P to more intuitively grasp what kind of foods are stored in refrigeration compartment 11.

[0294] The refrigerator compartment interior image RG is an image corresponding to the model of the refrigerator 1 used by the user P of the terminal device 3.

[0295] According to this, the user P is provided with a refrigerator compartment interior image RG that corresponds to the refrigerator compartment 11 of the refrigerator 1 that is actually being used, so that the user P can more intuitively grasp what kind of food is stored in the refrigerator compartment 11.

[0296] The refrigerator interior image RG is divided into a plurality of divided areas BA. The number of divided areas BA included in the refrigerator interior image RG corresponds to the model of the refrigerator 1 used by the user P of the terminal device 3.

[0297] This allows the user P to know what foods are stored in the storage area of ​​the refrigeration compartment 11 of the refrigerator 1 that he or she is actually using. Therefore, the user P can more intuitively know what foods are stored in the refrigeration compartment 11.

[0298] When displaying a plurality of photographed food images FG in one divided area BA, the application execution unit 301 displays the plurality of photographed food images FG side by side.

[0299] This allows a plurality of photographed food images FG to be displayed in an easy-to-view manner for each divided area BA, so that the user P can intuitively and easily grasp what kind of food is stored in the refrigerator compartment 11.

[0300] In the divided area BA, a maximum number of photographed food images FG that can be displayed simultaneously is set.

[0301] This prevents a decrease in visibility when there are many photographed food images FG displayed in one divided area BA. Therefore, the user P can intuitively and easily grasp what kind of food is stored in the refrigerator compartment 11.

[0302] When the number of photographed food images FG to be displayed in one divided area BA exceeds the upper limit number, the application execution unit 301 switchably displays the photographed food images FG within the range of the upper limit number.

[0303] This prevents a decrease in visibility when there are many photographed food images FG displayed in one divided area BA, while allowing the user P to view the photographed food images FG that cannot be displayed.

[0304] The application execution unit 301 accepts memo input from the user P of the terminal device 3 for each photographed food image FG, and displays a memo input icon MIC indicating that a memo has been accepted in association with the photographed food image FG.

[0305] This allows user P to know for which food the memo has been input. Therefore, user P can intuitively know what kind of food is stored in refrigerator compartment 11, and can also know for which food the memo has been input.

[0306] The food management application 311 is an application program that can be installed in the terminal device 3.

[0307] According to this, by installing the food management application 311, a terminal device 3 that does not have the function of displaying the refrigerator compartment interior image RG and the photographed food image FG can be made into a terminal device 3 that can display these images.

[0308] As described above, the food management application 311 causes the terminal processor 300 of the terminal device 3 equipped with the touch panel 32 to function as an application execution unit 301 that displays an image captured by the drawer camera 204 capturing an image of the drawer 16A of the refrigerator 1 having an upper case JCA and a lower case GCA. The application execution unit 301 is capable of displaying an upper case image JCG and a lower case image GCG, and accepts input of a note from the user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG. When the application execution unit 301 accepts input of a note from the user P for the upper case image JCG, it displays a note input icon MIC indicating that the note input has been accepted in association with the upper case image JCG, and when it accepts input of a note from the user P for the lower case image GCG, it displays the note input icon MIC in association with the lower case image GCG.

[0309] The terminal device 3 also includes a touch panel 32 and an application execution unit 301 that displays an image captured by the drawer camera 204. The application execution unit 301 is capable of displaying an upper case image JCG and a lower case image GCG, and accepts input of a note from a user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG. When the application execution unit 301 accepts input of a note from the user P for the upper case image JCG, the application execution unit 301 displays a note input icon MIC indicating that the note input has been accepted in association with the upper case image JCG, and when the application execution unit 301 accepts input of a note from the user P for the lower case image GCG, the application execution unit 301 displays the note input icon MIC in association with the lower case image GCG.

[0310] In addition, the control method of the terminal device 3 is capable of displaying an upper case image JCG and a lower case image GCG, and accepts memo input from a user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG, and when memo input from the user P is accepted for the upper case image JCG, a memo input icon MIC indicating that memo input has been accepted is displayed in association with the upper case image JCG, and when memo input from the user P is accepted for the lower case image GCG, the memo input icon MIC is displayed in association with the lower case image GCG.

[0311] According to this, the food management application 311, the terminal device 3, and the control method of the terminal device 3 display a photographed image of the case CA in the drawer 16A, and the user P can input a memo into the displayed photographed image. This allows the user P to intuitively grasp the foods stored in the drawer 16A of the refrigerator 1, and improves the convenience of the user P.

[0312] The application execution unit 301 displays the upper case image JCG superimposed on the lower case image GCG, and when an operation to move the upper case image JCG is received from the user P, the application execution unit 301 moves the first case image and displays the lower case image GCG.

[0313] This allows the lower case image GCG to be displayed in a manner that moves the actual case CA, allowing the user P to more intuitively grasp the foods stored in the drawer 16A of the refrigerator 1, and improving convenience for the user P.

[0314] When a memo input icon MIC is displayed in association with the upper case image JCG, the application executing unit 301 moves the memo input icon MIC together with the upper case image JCG.

[0315] This allows the association between the upper case image JCG and the memo input icon MIC to be maintained even if the user P moves the upper case image JCG. Therefore, even if the user P moves the upper case image JCG, the user P can properly understand which image the memo input icon MIC relates to, improving the convenience for the user P.

[0316] The application execution unit 301 displays a switch button for switching the captured image to be displayed between the upper case image JCG and the lower case image GCG, and switches the captured image to be displayed between the upper case image JCG and the lower case image GCG in response to an operation on the switch button.

[0317] According to this, the case image CG to be displayed can be easily switched by operating the switching button, thereby improving convenience for the user P.

[0318] The upper case image JCG is an image captured when the upper case JCA is pulled out more than a predetermined threshold with respect to the refrigerator 1. The lower case image GCG is an image captured when the lower case GCA is pulled out more than a predetermined threshold with respect to the refrigerator 1.

[0319] This allows a captured image that shows as much food as possible to be used as the case image CG, allowing the user P to intuitively grasp the food stored in the drawer 16A of the refrigerator 1 and improving convenience for the user P.

[0320] The food management application 311 is an application program that can be installed in the terminal device 3.

[0321] According to this, a terminal device 3 that does not have the function of displaying a case image CG and the function of accepting input of notes for the displayed case image CG can be made into a terminal device 3 that has these functions by installing the food management application 311.

[0322] (Embodiment 2) Next, a description will be given of embodiment 2. In the description of embodiment 2, the same components as those of each part of food control system 1000 of embodiment 1 will be given the same reference numerals and detailed description will be omitted as appropriate.

[0323] [2-1. Configuration] FIG. 26 is a block diagram showing the configuration of the terminal device 3 and the food management server 4 in the second embodiment. As is clear from a comparison between Figures 26 and 7, the server processor 400 in embodiment 2 functions as a food recognition unit 403 in addition to a server communication control unit 401 and an information processing unit 402 by executing a control program stored in the server memory unit 410.

[0324] The food recognition unit 403 performs food recognition processing based on the photographed food image FG indicated by the food image data 4116 included in each of the first update information and the second update information.

[0325] The food recognition process is a process of recognizing food from the photographed food image FG and retrieving an icon representing the same type of food as the recognized food from an icon database. This icon database stores an icon representing each food. Hereinafter, this icon will be referred to as a food icon. The icon database may be stored in the food management server 4, or in an external device with which the food management server 4 can communicate. In the present embodiment, the food icon corresponds to an example of a food image of the present disclosure.

[0326] For example, the food recognition unit 403 recognizes food as follows. The food recognition unit 403 identifies an image that is most similar to the photographed food image FG indicated by the food image data 4116 included in the update information from among the images stored in the food image database. The food image database is a database in which image data for each food is stored. The degree of similarity is determined based on features such as shape and color. The food recognition unit 403 then recognizes that the food corresponding to the identified image is the food indicated by the food image data 4116 included in the update information.

[0327] The food recognition unit 403 may recognize food by AI (Artificial Intelligence). For example, a machine learning process is performed to learn feature quantities such as the color and shape of food from a teacher photographed food image FG, and a trained model is constructed. The trained model is stored in the server storage unit 410. The food recognition unit 403 uses food image data 4116 included in the update information as input data, refers to the trained model, and recognizes the food indicated by the photographed food image FG. Here, the trained model may be stored in a device other than the food management server 4 connected to the global network GN, and the food may be recognized in the food management server 4. In this case, the food recognition unit 403 transmits the food image data 4116 included in the update information to a device that stores the trained model, and acquires the food identification result from the device. The food recognition unit 403 then recognizes the food indicated by the acquired identification result as the food indicated by the photographed food image FG indicated by the food image data 4116 included in the update information.

[0328] When the food recognition unit 403 recognizes a food, it acquires a food icon of the recognized food from the icon database. Then, in the second embodiment, the icon data of the food icon acquired by the food recognition unit 403 is appropriately stored instead of or together with the food image data 4116 held in the refrigerator food management record RR. Also, in the second embodiment, the icon data of the food icon acquired by the food recognition unit 403 is appropriately stored instead of or together with the food image data 4116 held in the drawer food management record HR.

[0329] As a result, in the second embodiment, the application UI 320 can display a food icon instead of the photographed food image FG.

[0330] [2-2. Effects, etc.] According to the second embodiment, the same effects as those of the first embodiment are achieved.

[0331] (Other embodiments) As described above, the above-mentioned embodiment has been described as an example disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to an embodiment in which modifications, substitutions, additions, omissions, etc. are made. In addition, it is also possible to combine the components described in the above-mentioned embodiment to create a new embodiment. Therefore, other embodiments will be exemplified below.

[0332] In the above-described embodiment, the refrigerator 1 is exemplified as a storage facility. However, the storage facility of the present disclosure is not limited to the refrigerator 1, and may be, for example, a warehouse installed in an office or the like. In the above-described embodiment, food is exemplified as an item of the present disclosure, but the item of the present disclosure is not limited to food, and may be other items such as books depending on the storage facility.

[0333] In the above-described embodiment, a configuration has been exemplified in which the position restriction member 205 has the mark 207 at the center in the left-right direction of the main body 201. However, the position of the mark 207 is not limited to the center in the left-right direction of the main body 201. The position of the mark 207 may be other than the center in the left-right direction of the main body 201. In this case, each mark is appropriately recorded on the center position recording paper CY so that the camera unit 2, whose position is adjusted based on the mark 207, can appropriately perform various detections.

[0334] In the above-described embodiment, the refrigerator compartment interior image RG, which is an image of the refrigerator compartment 11 viewed from the front when the left door 11A and the right door 11B are in an open state, has been exemplified as the refrigerator compartment interior image of the present disclosure. However, the refrigerator compartment interior image of the present disclosure is not limited to this refrigerator compartment interior image RG, and may be an image of the refrigerator compartment viewed from the front when a single door is in an open state, or an image of the interior of a refrigerator 1 including a refrigerator compartment and a freezer when the doors are in an open state.

[0335] The imaging range of the drawer camera 204 may be a range that combines one or a plurality of any drawer-type housing compartments depending on the type of the drawer-type housing compartments on the screen displayed by the application UI 320.

[0336] The food control system 1000 may manage other elements such as remaining quantity and expiration date in addition to the storage date and time.

[0337] The main body 201 and the photographing member 202 of the camera unit 2 may be configured as separate bodies.

[0338] The camera unit 2 may be configured to transmit the case image CG to the food management server 4 even when the food is not pulled out by the predetermined threshold or more. In this configuration, the food management server 4 stores the case image CG when the food is not pulled out by the predetermined threshold or more, and provides it to the terminal device 3. The application execution unit 301 then displays this case image CG on the application UI 320 so as to be replaceable with each of the upper case image JCG and the lower case image GCG.

[0339] In the above embodiment, the case where the user P installs the camera unit 2 in the refrigerator 1 has been illustrated, but the camera unit 2 may be shipped in a state where it is already installed in the refrigerator 1.

[0340] A configuration may also be used in which the server processor 400 functions as the food detection unit 225, and the camera unit processor 220 does not function as the food detection unit 225. In this configuration, the camera unit 2 transmits the recording file 231 to the food management server 4.

[0341] In the above-described embodiment, the human presence sensor 212 has been exemplified as a sensor of the present disclosure. However, the sensor of the present disclosure is not limited to the human presence sensor 212, and may be an illuminance sensor, an electrostatic sensor provided on the grip of the refrigerator 1, or the like. In the case of this electrostatic sensor, the refrigerator 1 and the camera unit 2 are capable of communicating with each other, and the refrigerator 1 transmits the detection value of the electrostatic sensor to the camera unit 2.

[0342] The refrigerator food management record RR and the drawer food management record HR may be stored in the terminal device 3, and the food management app 311 may manage various information related to food detected by the camera unit 2. In this configuration, the app execution unit 301 appropriately updates the contents of each record stored in the terminal device 3 in the same manner as the above-mentioned food management server 4. This eliminates the need for the food management system 1000 to include the food management server 4, simplifying the system configuration. Furthermore, in this configuration, the app execution unit 301 may function as the food recognition unit 403.

[0343] For example, the types of compartments formed in the main box 10 of the refrigerator 1 are not limited to the refrigeration compartment 11, the ice-making compartment 12, the fresh-freezing compartment 13, the freezer compartment 14, and the vegetable compartment 15, and there may be fewer compartments or other types of compartments. Also, the number of doors provided at the opening on the front of the refrigeration compartment 11 may be one.

[0344] For example, the number of cases provided in drawers 14A, 15A is not limited to two, and may be greater. In this case, one case corresponds to the first case of the present disclosure, and the case below this one case corresponds to the second case of the present disclosure.

[0345] For example, the camera unit processor 220, the terminal processor 300, and the server processor 400 may be configured by a single processor, or may be configured by multiple processors.

[0346] Each part shown in Figures 6, 7, and 26 is an example, and the specific implementation form is not particularly limited. In other words, it is not necessarily necessary to implement hardware corresponding to each part individually, and it is of course possible to configure the functions of each part by executing a program by one processor. Also, some of the functions realized by software in the above-mentioned embodiment may be hardware, or some of the functions realized by hardware may be software. In addition, the specific detailed configurations of each part of the camera unit 2, the terminal device 3, and the food management server 4 can also be arbitrarily changed within the scope of the present disclosure.

[0347] Furthermore, for example, the operation step units shown in Figures 13, 14, 22, and 25 are divided according to the main processing content in order to facilitate understanding of the operation of each part of the food management system 1000, and the present disclosure is not limited by the manner in which the processing units are divided or their names.

[0348] It should be noted that the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, substitutions, additions, omissions, and the like can be made within the scope of the claims or their equivalents. [Industrial Applicability]

[0349] As described above, the program, terminal device, and terminal device control method according to the present invention can be used to display an image of a case in a drawer-type storage compartment. [Explanation of symbols]

[0350] 1 Refrigerator (storage) 2 Camera unit (photography device) 3 Terminal Equipment 4. Food Management Server 10 Main box 10A Front edge 10B Top surface 11 Refrigerator 11A Left door 11B Right door 11C Door 12 Ice Room (Drawer-type Storage Room) 12A, 13A, 14A, 15A, 16A drawer 13 Fresh Freezing Room (Drawer-type Containment Room) 14 Freezer (Drawer-type Storage Room) 15 Vegetable compartment (drawer-type storage compartment) 32 Touch panel (display) 201 Main unit 202 Photography materials 203 Refrigerator Camera 204 Drawer Camera 205 Position control member 206 Bottom 207 marks 212 Human Sensor (Sensor) 213 Camera 223 Shooting control section 224 Recording Control Unit 225 Food Detection Unit (Item Detection Unit) 231 Recording files 300 Terminal Processors (computers) 301 Application execution unit (display control unit) 311 Food management apps (programs, application programs) 1000 Food Control System BA Division Area (Area) FG, FG1 Food images (food images) GCA Lower Case (2nd Case) GCG lower case image (second case image) JCA Upper Case (1st Case) JCG upper case image (first case image) MIC memo input icon (memo input image) P User RG refrigerator interior image (interior image)

Claims

1. A computer of a terminal device having a display unit, functioning as a display control unit that displays an image captured by a drawer camera that captures an image of a drawer of a refrigerator having a first case and a second case that is a lower case of the first case; The display control unit is A first case image, which is the captured image of the first case, and a second case image, which is the captured image of the second case, can be displayed; accepting input of a memo from a user of the terminal device for each of the first case image and the second case image; When a memo input is received from the user for the first case image, a memo input image indicating that the memo input has been received is displayed in association with the first case image; when a memo input from the user is received for the second case image, the memo input image is displayed in association with the second case image. program.

2. The display control unit displaying the first case image superimposed on the second case image; When an operation to move the first case image is received from the user, the first case image is moved and the second case image is displayed. The program according to claim 1.

3. The display control unit is When the memo input image is displayed in association with the first case image, the memo input image is moved together with the first case image. The program according to claim 2.

4. The display control unit is displaying a switch button for switching the photographed image to be displayed between the first case image and the second case image; switching the captured image to be displayed to the first case image or the second case image in response to an operation on the switching button; The program according to claim 1.

5. The first case image is the captured image when the first case is pulled out from the refrigerator by a predetermined threshold or more, The second case image is the captured image when the second case is pulled out from the refrigerator by a predetermined threshold or more. The program according to any one of claims 1 to 4.

6. The terminal device is an installable application program; The program according to any one of claims 1 to 5.

7. A display unit; a display control unit that displays an image captured by a drawer camera that captures an image of a drawer of a refrigerator having a first case and a second case that is a lower case of the first case, The display control unit is A first case image, which is the captured image of the first case, and a second case image, which is the captured image of the second case, can be displayed; accepting input of a memo from a user for each of the first case image and the second case image; When a memo input is received from the user for the first case image, a memo input image indicating that the memo input has been received is displayed in association with the first case image; when a memo input from the user is received for the second case image, the memo input image is displayed in association with the second case image. Terminal device.

8. 1. A control method for a terminal device that displays an image captured by a drawer camera that captures an image of a drawer of a refrigerator having a first case and a second case that is a lower case of the first case, comprising: A first case image, which is the captured image of the first case, and a second case image, which is the captured image of the second case, can be displayed; accepting input of a memo from a user of the terminal device for each of the first case image and the second case image; When a memo input is received from the user for the first case image, a memo input image indicating that the memo input has been received is displayed in association with the first case image; when a memo input from the user is received for the second case image, the memo input image is displayed in association with the second case image. A method for controlling a terminal device.

Images