Imaging device

Abstract

To provide an imaging device capable of accurately imaging an article taken in and out of a storage.SOLUTION: An imaging device comprises: a body which is installed on an upper face of a storage; and an imaging member which is extended from the body in a forward direction of the storage and has a camera taking an image looking down the storage from an upper front section thereof. The body is brough into contact with a front edge of the storage and has a position regulating member which regulates a position of the imaging member in a longitudinal direction of the storage.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an imaging device. [Background technology]

[0002] Patent Document 1 discloses a technology in which a camera is installed on top of a refrigerator, and when an object passing through the opening of the refrigerator is detected, a photograph is taken, and the food stored in the refrigerator is managed based on the photographed result. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-070476 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides an imaging device that can accurately capture images of items entering and leaving a storage facility. [Means for solving the problem]

[0005] The photographing device according to the present disclosure is a photographing device separate from a refrigerator having a door and a drawer, and includes a main body installed on the top surface of the refrigerator, a photographing member extending from the main body to the front of the refrigerator and equipped with a camera for photographing the area from above the front of the refrigerator downward, and a camera mounted on a lower part of the photographing member. In the closed state The door and In the closed stateand a drawer opening / closing detection means located forward of the drawer and detecting the opening and closing of the drawer, the main body being equipped with a position regulating member that abuts against the front edge of the refrigerator and regulates the position of the photographing member in the front-to-rear direction of the refrigerator, the position regulating member being a position regulating piece that extends downward from the bottom surface of the main body in an installed state and is formed to be shorter than the length to a gasket provided between the refrigerator and the door, the camera being located forward of the door in a closed state in the installed state, and when the drawer opening / closing detection means detects the opening or closing of the drawer, it photographs an area including the drawer in an open state. [Effects of the Invention]

[0006] In the photographing device disclosed herein, the position of the photographing member in the front-to-rear direction of the storage facility is restricted by the position restricting member, so that the camera is positioned to photograph the front of the storage facility from above to below. Therefore, items entering and leaving the storage facility can be accurately included in the photographed results of the camera, and the items entering and leaving the storage facility can be accurately photographed. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 shows the configuration of a food management system according to a first embodiment. [Figure 2] FIG. 1 is a side view of a camera unit installed in a refrigerator according to the first embodiment, viewed from the right side. [Figure 3] FIG. 10 is a perspective view of a position restricting member abutting against the front edge of the main box body according to the first embodiment, as viewed from the left front side; [Figure 4] FIG. 1 is a front view of a refrigerator and a camera unit according to a first embodiment, viewed from the front side. [Figure 5] FIG. 1 is a plan view of a refrigerator and a camera unit according to a first embodiment, viewed from the right side. [Figure 6] FIG. 1 is a block diagram showing a configuration of a camera unit according to a first embodiment. [Figure 7] FIG. 1 is a block diagram showing the configuration of a terminal device and a food management server according to a first embodiment. [Figure 8]FIG. 1 is a diagram showing an example of a refrigerator compartment management database according to the first embodiment. [Figure 9] FIG. 10 is a diagram showing an example of an image of the inside of a refrigerator compartment in the first embodiment. [Figure 10] FIG. 1 shows an example of a withdrawal management database according to the first embodiment. [Figure 11] FIG. 10 shows an example of an adjustment screen according to the first embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a central position recording sheet according to the first embodiment; [Figure 13] 1 is a flowchart showing the operation of the camera unit according to the first embodiment. [Figure 14] Flowchart showing the operation of the food management system in embodiment 1 [Figure 15] FIG. 10 is a diagram for explaining an entry / exit determination image in the first embodiment; [Figure 16] FIG. 10 is a diagram showing an example of a refrigerator compartment home screen in the first embodiment. [Figure 17] FIG. 10 is a diagram showing a procedure for inputting a memo in the first embodiment. [Figure 18] FIG. 10 is a diagram showing an example of an accommodation list screen in the first embodiment. [Figure 19] FIG. 10 is a diagram showing an example of a new addition list screen in the first embodiment. [Figure 20] FIG. 1 is a diagram illustrating a procedure for adding food information according to the first embodiment. [Figure 21] FIG. 10 is a diagram showing an example of a deletion target list screen in the first embodiment. [Figure 22] Flowchart showing the operation of the food management system in embodiment 1 [Figure 23] FIG. 10 is a diagram showing an example of a vegetable compartment home screen in the first embodiment. [Figure 24] FIG. 10 is a diagram showing an example of a food image list screen in the first embodiment. [Figure 25] Flowchart showing the operation of the food management system in embodiment 1 [Figure 26] A block diagram showing the configuration of a terminal device and a food management server according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] (Findings that formed the basis of this disclosure) At the time the inventors conceived the present disclosure, there was a technology for photographing items entering and leaving a storage facility such as a refrigerator. Conventional technologies sometimes use a camera separate from the storage facility to capture images. In such cases, the camera is often installed on the top surface of the storage facility. However, the inventors discovered a problem in that when the camera is installed on the top surface of the storage facility, for example, if the camera is installed toward the rear of the storage facility, there is a risk that items entering and leaving the storage facility may not be accurately captured in the camera's image capture. The subject matter of the present disclosure was created to solve this problem. Therefore, the present disclosure provides an imaging device that can accurately capture images of items entering and leaving a storage facility.

[0009] Hereinafter, embodiments will be described in detail with reference to the drawings. However, in some cases, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or redundant explanation of substantially the same configuration may be omitted. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0010] (Embodiment 1) First, the first embodiment will be described. [1-1.Configuration] [1-1-1. Food Control System Configuration] FIG. 1 is a diagram showing the configuration of a food management system 1000.

[0011] The food management system 1000 is a system that manages the storage date and time of the food stored in the refrigerator 1, etc. The refrigerator 1 corresponds to an example of a storage unit of the present disclosure, and the food corresponds to an example of an article of the present disclosure.

[0012] The food management system 1000 includes a refrigerator 1. The refrigerator 1 includes a main box 10 with an open front. The main box 10 is formed with a refrigerator compartment 11, an ice making compartment 12, a fresh freezing compartment 13, a freezer compartment 14, and a vegetable compartment 15. The opening at the front of the refrigerator compartment 11 is provided with a rotating left door 11A and a right door 11B. The ice making compartment 12, the fresh freezing compartment 13, the freezer compartment 14, and the vegetable compartment 15 are each provided with drawers 12A, 13A, 14A, and 15A for storing food. The drawers 14A and 15A of the present disclosure include two cases CA: an upper case JCA and a lower case GCA. The upper case JCA corresponds to an example of the first case of the present disclosure, and the lower case GCA corresponds to an example of the second case of the present disclosure.

[0013] In the following description, when there is no need to distinguish between the left door 11A and the right door 11B, they will be referred to as "doors" and given the symbol "11C." Also, in the following description, when there is no need to distinguish between the drawers 12A, 13A, 14A, and 15A, they will be referred to as "drawers" and given the symbol "16A."

[0014] Food management system 1000 includes camera unit 2. Camera unit 2 is a device that takes photographs and detects food entering or leaving refrigerator 1 from the photographed images. Camera unit 2 is provided on top surface 10B of main box 10 of refrigerator 1. The camera unit 2 corresponds to an example of the imaging device of the present disclosure.

[0015] The food management system 1000 includes a terminal device 3. The terminal device 3 is, for example, a smartphone or a tablet terminal. An application program for managing food in the refrigerator 1 is installed on the terminal device 3, and the terminal device 3 communicates with the food management server 4 using the functions of this application program. In the following description, this application program will be referred to as the "food management application" and will be given the reference number "311." The food management application 311 corresponds to an example of a program and an application program of the present disclosure.

[0016] In FIG. 1, a user P who is at home is indicated by a solid line, and a user P who has left home H is indicated by a dotted line. When used by user P who is at home, terminal device 3 communicates with food management server 4 via communication device 5 or without via communication device 5, using the function of food management app 311. When terminal device 3 is used by user P who has left home H and cannot establish a communication connection with communication device 5, terminal device 3 communicates with food management server 4 without via communication device 5, using the function of food management app 311.

[0017] The communication device 5 is connected to a global network GN including the Internet, a telephone network, and other communication networks, and communicates with the food management server 4 connected to the global network GN. The communication device 5 is an interface device for connecting the terminal device 3 to the global network GN.

[0018] The food management system 1000 includes a food management server 4. The food management server 4 is a server device that manages the food in the refrigerator 1 and is connected to the global network GN. In each figure, the food management server 4 is represented by a single block, but this does not necessarily mean that the food management server 4 is composed of a single server device. For example, the food management server 4 may be composed of multiple server devices with different processing contents.

[0019] Next, the configurations of the camera unit 2, the terminal device 3, and the food management server 4 will be described.

[0020] [1-1-2. Camera unit configuration] The camera unit 2 will be described with reference to FIGS. Figures 2 to 4 show the X-axis, Y-axis, and Z-axis. The X-axis, Y-axis, and Z-axis are perpendicular to each other. The Z-axis indicates the up-down direction. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the left-right direction and also indicates the width direction of the refrigerator 1 and camera unit 2. The Y-axis indicates the front-to-back direction and also indicates the depth direction of the refrigerator 1 and camera unit 2. The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the front direction. The positive direction of the Z-axis indicates the upward direction.

[0021] Fig. 2 is a side view of the camera unit 2 installed in the refrigerator 1, seen from the right side. Fig. 3 is a perspective view of the position restriction member 205 abutting against the front edge 10A of the main box body 10, seen from the left front side.

[0022] 2 and 3, camera unit 2 includes main body 201 having a box-like appearance and installed on top surface 10B of main box 10. Camera unit 2 further includes imaging member 202 extending forward from main body 201 above the front of main body 201. Imaging member 202 includes refrigerator compartment camera 203 and drawer camera 204. Refrigerator compartment camera 203 and drawer camera 204 are each provided below imaging member 202 and forward of main body 201.

[0023] The main body 201 is provided with a position restricting member 205 that restricts the position of the imaging member 202 in the front-to-rear direction of the refrigerator 1 by abutting against the front edge 10A of the main box body 10. The position restricting member 205 is a position restricting piece that extends downward from the bottom surface 206 of the main body 201 in the installed state. The position restricting member 205 extends in the left-to-right direction of the main body 201 on the front surface of the main body 201, and has a mark 207 in the center of the main body 201 in the left-to-right direction. This mark 207 indicates the center position of the camera unit 2 in the left-to-right direction.

[0024] The length of position regulating member 205 is shorter than the length from top surface 10B of main box body 10 to a packing provided between refrigerator 1 and door 11C in the installed state. More specifically, the length of position regulating member 205 is shorter than the length from top surface 10B of main box body 10 to a door packing provided substantially parallel to top surface 10B on the upper inner surface of door 11C in the installed state.

[0025] Main body 201 is fixed at a position regulated by position regulating member 205 by built-in magnet 208. The means for fixing main body 201 is not limited to magnet 208, and other means such as double-sided tape may also be used.

[0026] FIG. 4 is a front view of refrigerator 1 and camera unit 2 as viewed from the front side. In the refrigerator 1 shown in FIG. 4, left door 11A and right door 11B are in an open state. FIG. 5 is a plan view of refrigerator 1 and camera unit 2 as viewed from the right side. In the refrigerator 1 shown in FIG. 5, drawer 15A is in an open state, i.e., pulled out. In the camera unit 2 shown in FIGS. 4 and 5, mark 207 is located approximately in the center of the refrigerator 1 in the left-right direction, and position restriction member 205 abuts against front edge 10A.

[0027] The camera unit 2 has a refrigerator compartment camera 203 and a drawer camera 204 . The refrigerator compartment camera 203 photographs the food entering and leaving the refrigerator compartment 11 by photographing the area from above the front of the refrigerator 1 downwards.

[0028] For example, the shooting range of refrigerator compartment camera 203 includes range A1 shown in Fig. 4 when viewed from the front. Range A1 is a range that includes door pocket 111 provided in door 11C and the opening on the front side of refrigerator compartment 11 when door 11C is in the open state. Furthermore, the shooting range of refrigerator compartment camera 203 includes range A3 shown in Fig. 5 when viewed from the side. Range A3 is a range that includes, in the front-to-back direction, from door 11C in the open state to the front end of shelf 112, and also includes the opening of refrigerator compartment 11 in the up-down direction.

[0029] Drawer camera 204 photographs the area from the upper front to the lower front of refrigerator 1, thereby photographing food items being put into or taken out of the drawer-type storage compartment.

[0030] For example, the imaging range of drawer camera 204 includes range A2 shown in Fig. 4 when viewed from the front. Range A2 is a range that includes drawer 16A when viewed from the front of refrigerator 1. Furthermore, the imaging range of drawer camera 204 includes range A4 shown in Fig. 5 when viewed from the side. Range A4 is a range that includes drawer 16A when it is pulled out to the furthest position in the front-to-rear direction.

[0031] The camera unit 2 has a distance measuring sensor 209 for the left door, a distance measuring sensor 210 for the right door, and a distance measuring sensor 211 for the drawer.

[0032] The left door distance measuring sensor 209 is a distance measuring sensor for detecting the open / closed state of the left door 11A. The left door distance measuring sensor 209 detects the distance from the left door 11A as the open / closed state of the left door 11A.

[0033] The right door distance measuring sensor 210 is a distance measuring sensor for detecting the open / closed state of the right door 11B. The right door distance measuring sensor 210 detects the distance from the right door 11B as the open / closed state of the right door 11B.

[0034] The drawer distance measuring sensor 211 is a distance measuring sensor for detecting the open / closed state of the drawer 16A. The open state of the drawer 16A refers to the state in which the drawer 16A is drawn out from the corresponding drawer-type storage chamber. The closed state of the drawer 16A refers to the state in which the drawer 16A is stored in the corresponding drawer-type storage chamber. The drawer distance measuring sensor 211 detects the distance from the drawer 16A as the open / closed state of the drawer 16A. In this embodiment, drawer distance measuring sensor 211 detects the open / closed state of each of drawers 14A and 15A.

[0035] The camera unit 2 has a human presence sensor 212. The human presence sensor 212 outputs a detection value to the camera unit control unit 20. The human presence sensor 212 is provided at a predetermined position on the imaging member 202. The human presence sensor 212 corresponds to an example of a "sensor" in the present disclosure.

[0036] Note that the arrangement order and installation positions of each sensor and each camera in Figures 4 and 5 are illustrated for the convenience of explaining the camera unit 2, and these arrangement order and installation positions are not limited to the arrangement order and installation positions shown in Figures 4 and 5.

[0037] FIG. 6 is a block diagram showing the configuration of the camera unit 2. The camera unit 2 includes a camera unit control unit 20, a camera unit communication unit 21, a sensor unit 22, an image capturing unit 23, and a power supply unit 24.

[0038] The camera unit control unit 20 includes a camera unit processor 220, which is a processor such as a CPU (Central Processing Unit) or an MPU (Micro-processing unit), and a camera unit storage unit 230. The camera unit control unit 20 controls each unit of the camera unit 2 by the camera unit processor 220 reading and executing a control program stored in the camera unit storage unit 230. By executing the control program stored in the camera unit storage unit 230, the camera unit processor 220 functions as a camera unit communication control unit 221, a detection value processing unit 222, an imaging control unit 223, a recording control unit 224, and a food detection unit 225.

[0039] The camera unit storage unit 230 includes a memory that stores programs executed by the camera unit processor 220 and data processed by the camera unit processor 220. The camera unit storage unit 230 stores the control programs executed by the camera unit processor 220, a camera unit ID 4112 that is identification information for the camera unit 2, and various other data. The camera unit storage unit 230 includes a non-volatile storage area. The camera unit storage unit 230 may also include a volatile storage area and constitute a work area for the camera unit processor 220.

[0040] The camera unit communication unit 21 is a communication interface equipped with a communication circuit, connector, etc. that conforms to a predetermined communication standard, and communicates with the food management server 4 and the terminal device 3 under the control of the camera unit control unit 20. In this embodiment, the communication standard used by the camera unit communication unit 21 is exemplified as a wireless communication standard, but a wired communication standard may also be used.

[0041] The sensor section 22 includes a left door distance measuring sensor 209, a right door distance measuring sensor 210, a drawer distance measuring sensor 211, and a human sensor 212, and outputs the detection value of each sensor to the camera unit control section 20.

[0042] The photographing section 23 includes a refrigerator compartment camera 203 and a drawer camera 204. The photographing section 23 outputs the photographing results of the refrigerator compartment camera 203 and the drawer camera 204 to the camera unit control section 20. The refrigerator compartment camera 203 and the drawer camera 204 capture moving images. Hereinafter, when the refrigerator compartment camera 203 and the drawer camera 204 are not distinguished from each other, they will be referred to as "cameras" and given the reference number "213".

[0043] Power supply unit 24 includes hardware such as a power supply circuit, and supplies power to each component of camera unit 2. In the present embodiment, power cable 7 of camera unit 2 is connected to commercial AC power supply 8, and power supply unit 24 supplies power to each component of camera unit 2 based on the power supplied from external commercial AC power supply 8.

[0044] The source of power supply to power supply unit 24 is not limited to the external commercial AC power supply 8, but may be the refrigerator 1 or a battery. If the source is the former, power supply unit 24 and refrigerator 1 have hardware such as a port that complies with the USB power supply standard. If the source is the former, a USB cable of camera unit 2 is connected to a USB port on refrigerator 1, and refrigerator 1 supplies power to power supply unit 24 via the USB cable. If the source is the latter, camera unit 2 has a battery.

[0045] As described above, the camera unit control unit 20 functions as the camera unit communication control unit 221, the detection value processing unit 222, the photography control unit 223, the recording control unit 224, and the food detection unit 225. The food detection unit 225 corresponds to an example of an article detection unit.

[0046] The camera unit communication control unit 221 communicates with the food management server 4 via the camera unit communication unit 21.

[0047] The detection value processing unit 222 determines whether the left door 11A, the right door 11B, and the drawers 14A and 15A are in an open or closed state based on the detection values ​​of the sensors output from the sensor unit 22. If the detection value processing unit 222 determines that either the left door 11A or the right door 11B is in an open state, it outputs door open state information indicating that the door 11C is in an open state to the recording control unit 224 and the food detection unit 225. If the detection value processing unit 222 determines that both the left door 11A and the right door 11B are in a closed state, it outputs door closed state information indicating that the door 11C is in a closed state to the recording control unit 224 and the food detection unit 225. If the detection value processing unit 222 determines that either the drawers 14A or 15A is in an open state, it outputs drawer open state information indicating that either the drawers 14A or 15A is in an open state to the recording control unit 224 and the food detection unit 225. Furthermore, when the detection value processing unit 222 determines that both of the drawers 14A and 15A are closed, it outputs to the recording control unit 224 and the food detection unit 225 drawer closed state information indicating that the drawers 14A and 15A are closed.

[0048] Furthermore, the detection value processing unit 222 determines whether or not a person is present around the refrigerator 1 based on the detection value of the human presence sensor 212 output from the sensor unit 22. If the detection value processing unit 222 determines that a person is present around the refrigerator 1, it outputs presence / absence information indicating the presence of a person to the photography control unit 223. If the detection value processing unit 222 determines that no person is present around the refrigerator 1, it outputs presence / absence information indicating the presence of a person to the photography control unit 223.

[0049] When the detection value processing unit 222 outputs presence / absence information indicating that a person is present, the photography control unit 223 causes the refrigerator compartment camera 203 and the drawer camera 204 to start photographing. When the detection value processing unit 222 outputs presence / absence information indicating that a person is not present, the photography control unit 223 causes the refrigerator compartment camera 203 and the drawer camera 204 to end photographing.

[0050] When the detection value processing unit 222 outputs door open state information, the recording control unit 224 starts recording the image capture results of the refrigerator compartment camera 203. When the detection value processing unit 222 outputs door closed state information, the recording control unit 224 ends the started recording of the image capture results of the refrigerator compartment camera 203. When the detection value processing unit 222 outputs drawer open state information, the recording control unit 224 starts recording the image capture results of the drawer camera 204. When the detection value processing unit 222 outputs drawer closed state information, the recording control unit 224 ends the started recording of the image capture results of the drawer camera 204. When the recording control unit 224 ends the recording, it generates a video from the start to the end of recording as one recording file 231 and stores it in the camera unit storage unit 230.

[0051] The food detection unit 225 detects food items that have been put into or taken out of the refrigerator 1 based on the video recording file 231 stored in the camera unit storage unit 230.

[0052] FIG. 7 is a block diagram showing the configuration of the terminal device 3 and the food management server 4.

[0053] [1-1-3. Food Management Server Configuration] First, the configuration of the food management server 4 will be described. The food management server 4 includes a server control unit 40 and a server communication unit 41.

[0054] The server control unit 40 includes a server processor 400, which is a processor such as a CPU or MPU, and a server storage unit 410. The server control unit 40 controls each unit of the food management server 4 by the server processor 400 reading and executing a control program stored in the server storage unit 410. The server processor 400 executes the control program stored in the server storage unit 410, thereby functioning as a server communication control unit 401 and an information processing unit 402.

[0055] The server storage unit 410 has a memory for storing programs executed by the server processor 400 and data processed by the server processor 400. The server storage unit 410 stores the control programs executed by the server processor 400, a refrigerator compartment food management database 411, a drawer food management database 412, and various other data. The server storage unit 410 has a non-volatile storage area. The server storage unit 410 may also have a volatile storage area and constitute a work area for the server processor 400.

[0056] FIG. 8 is a diagram showing an example of one record stored in the refrigerator compartment food management database 411. The refrigerator compartment food management database 411 is a database that stores various information related to the food stored in the refrigerator compartment 11. In the following description, one record stored in the refrigerator food management database 411 is referred to as a "refrigerator food management record" and is given the symbol "RR."

[0057] The refrigerator compartment food management record RR includes an account ID 4111, a camera unit ID 4112, refrigerator model information 4113, refrigerator compartment interior image data 4114, whether the item has been received or not information 4115, food image data 4116, storage date and time information 4117, a divided area number 4118, whether a memo has been entered information 4119, memo information 4130, deletion information 4131, and deletion date and time information 4132.

[0058] The account ID 4111 is identification information that identifies an account assigned to a user P who uses the food management application 311.

[0059] The refrigerator model information 4113 is information indicating the model of the refrigerator 1 used by the user P indicated by the account ID 4111 associated with the same record.

[0060] Refrigerator compartment interior image data 4114 is image data of an image showing the interior of refrigerator compartment 11 of refrigerator 1. Hereinafter, an image showing the interior of refrigerator compartment 11 will be referred to as a "refrigerator compartment interior image" and given the symbol "RG." Refrigerator compartment interior image data 4114 indicates a refrigerator compartment interior image RG corresponding to the model indicated by the refrigerator model information 4113 associated with it in the same record. The refrigerator compartment interior image RG corresponds to an example of an interior image.

[0061] FIG. 9 is a diagram showing an example of the refrigerator compartment interior image RG. The refrigerator compartment interior image RG shown in Fig. 9 includes a left door pocket image RG1, a refrigerator compartment front surface image RG2, and a right door pocket image RG3. The refrigerator compartment interior image RG shown in Fig. 9 has the left door pocket image RG1, the refrigerator compartment front surface image RG2, and the right door pocket image RG3 arranged in this order from left to right in the figure. The left door pocket image RG1 is an image showing the door pocket 111 formed in the left door 11A. The refrigerator compartment front surface image RG2 is an image of the refrigerator compartment 11 viewed from the front when the left door 11A and the right door 11B are in an open state. The right door pocket image RG3 is an image showing the door pocket 111 formed in the right door 11B.

[0062] The image region of the refrigerator compartment interior image RG is divided into a plurality of areas. Each of these areas is called a "divided area" and is assigned the symbol "BA." The refrigerator compartment interior image RG shown in FIG. 9 is divided into 15 divided areas BA. A different number is assigned to each divided area BA. In the following explanation, these numbers are called divided area numbers and are assigned the symbol "4118." Note that FIG. 9 conveniently illustrates which divided area number is assigned to which divided area BA, and the divided area number 4118 is not displayed on the refrigerator compartment interior image RG displayed on the refrigerator compartment home screen GM2 described later.

[0063] Returning to the explanation of FIG. 8, the in-stock / out-stock presence information 4115 is information indicating whether or not there has been new in-stock / out-stock.

[0064] Food image data 4116 is image data of food. In this embodiment, the image represented by food image data 4116 is an image captured by refrigerator compartment camera 203.

[0065] The storage date and time information 4117 is information indicating the storage date and time when the food was stored in the refrigerator 1.

[0066] The refrigerator compartment food management record RR has a division area number 4118 for each food image data 4116 .

[0067] The memo input information 4119 is information indicating whether or not a memo has been input.

[0068] The memo information 4130 is information indicating the input memo.

[0069] The deletion information 4131 indicates whether the associated sub-record is a sub-record to be deleted from the refrigerator food management record RR. A sub-record is a record stored in the refrigerator food management record RR, and includes food image data 4116, storage date and time information 4117, divided area number 4118, memo entry presence / absence information 4119, and memo information 4130.

[0070] The deletion date and time information 4132 indicates the date and time when the corresponding sub-record was designated as a deletion target. When the deletion information 4131 indicates a deletion target, the information processing unit 402 (described later) deletes the sub-record, and the deletion information 4131 and deletion date and time information 4132 associated with this sub-record, from the refrigerator compartment food management record RR at a predetermined timing.

[0071] FIG. 10 is a diagram showing one record stored in the withdrawn food management database 412. As shown in FIG. Drawer food management database 412 is a database that stores various information related to food in the drawer-type storage compartment. In the explanation with reference to Figures 23 to 25, drawer food management database 412 is a database related to vegetable compartment 15. In the following description, one record stored in the withdrawn food management database 412 is referred to as a "drawn food management record" and is denoted by the symbol "HR."

[0072] The drawn food management record HR includes an account ID 4111, a camera unit ID 4112, update date and time information 4121, upper case information 4122, and lower case information 4124.

[0073] The update date and time information 4121 is information indicating the date and time when the contents of the withdrawn food management record HR were updated.

[0074] The upper case information 4122 includes upper case image data 4123, food image data 4116, memo entry presence / absence information 4119, and memo information 4130.

[0075] The upper case image data 4123 is image data of an upper case image JCG, which is an image of the upper case JCA. The upper case image JCG is an image captured by the drawer camera 204. The upper case image JCG corresponds to an example of a first case image of the present disclosure.

[0076] In this embodiment, the image indicated by the food image data 4116 included in the upper case information 4122 is an image cut out from the upper case image JCG.

[0077] The lower case information 4124 includes lower case image data 4125, food image data 4116, memo entry presence / absence information 4119, and memo information 4130.

[0078] The lower case image data 4125 is image data of the lower case image GCG, which is an image of the lower case GCA. The lower case image GCG is an image captured by the drawer camera 204. The lower case image GCG corresponds to an example of a second case image of the present disclosure. Hereinafter, when there is no need to distinguish between the upper case image JCG and the lower case image GCG, they will be referred to as "case images" and marked with "CG."

[0079] In this embodiment, the image indicated by the food image data 4116 included in the lower case information 4124 is an image cut out from the lower case image GCG.

[0080] 7, the server communication unit 41 is a communication interface that includes a communication-related configuration such as a wireless circuit and an antenna that conforms to a predetermined communication standard, and communicates with devices connected to the global network GN in accordance with the predetermined communication standard. In this embodiment, the server communication unit 41 communicates with the camera unit 2 and the terminal device 3.

[0081] As described above, the server control unit 40 functions as the server communication control unit 401 and the information processing unit 402 .

[0082] The server communication control unit 401 communicates with the camera unit 2 and the terminal device 3 via the server communication unit 41.

[0083] The information processing unit 402 performs various information processes based on the information received by the server communication control unit 401 from the camera unit 2 and the terminal device 3 .

[0084] [1-1-4. Terminal Device Configuration] Next, the functional configuration of the terminal device 3 will be described. The terminal device 3 includes a terminal control unit 30, a terminal communication unit 31, and a touch panel 32. The touch panel 32 corresponds to an example of a display unit.

[0085] The terminal control unit 30 includes a terminal processor 300, which is a processor such as a CPU or MPU, and a terminal storage unit 310. The terminal control unit 30 controls each unit of the terminal device 3 by the terminal processor 300 reading and executing a control program stored in the terminal storage unit 310. A food management application 311 is installed in the terminal device 3. The terminal processor 300 functions as an application execution unit 301 by reading and executing the food management application 311. The terminal processor 300 corresponds to an example of a computer of the present disclosure. The application execution unit 301 corresponds to an example of a display control unit of the present disclosure.

[0086] The terminal storage unit 310 has a memory for storing programs executed by the terminal processor 300 and data processed by the terminal processor 300. The terminal storage unit 310 stores the control program executed by the terminal processor 300, the food management application 311, the account ID 4111, and various other data. The terminal storage unit 310 has a non-volatile storage area. The terminal storage unit 310 may also have a volatile storage area and constitute a work area for the terminal processor 300.

[0087] The terminal communication unit 31 is a communication interface having a configuration of a wireless circuit, an antenna, etc. that conforms to a predetermined communication standard, and communicates with devices connected to the global network GN in accordance with the predetermined communication standard. In this disclosure, a wireless communication standard is exemplified as the communication standard used by the terminal communication unit 31.

[0088] The touch panel 32 includes a display panel such as a liquid crystal display panel, and a touch sensor that is provided over the display panel or is integral with the display panel.

[0089] As described above, the terminal control unit 30 functions as the application execution unit 301. The application execution unit 301 transmits and receives various information to and from the food management server 4 via the terminal communication unit 31.

[0090] The application execution unit 301 also displays a user interface related to managing food in the refrigerator 1 on the touch panel 32. In the following description, this user interface will be referred to as the "application UI" and will be denoted by the reference numeral "320." By displaying the application UI 320 on the touch panel 32, the application execution unit 301 provides the user P with various information related to the food in the refrigerator 1, and also receives various inputs related to the food in the refrigerator 1 from the user P.

[0091] [1-2. Operation] Next, the operation of each part of the food management system 1000 will be described.

[0092] [1-2-1. Installing the camera unit] Camera unit 2 is separate from refrigerator 1. Therefore, user P installs camera unit 2 on top surface 10B of refrigerator 1. As described above, position restriction member 205 restricts the position of camera unit 2 in the front-to-rear direction of refrigerator 1, so user P adjusts the position of camera unit 2 in the left-to-right direction of refrigerator 1 when installing camera unit 2. Below, several examples of methods for adjusting the position of the camera unit 2 in the left-right direction of the refrigerator 1 will be given.

[0093] [1-2-1-1. First position adjustment]

[0094] In the first position adjustment, the user P uses the application UI 320. In the first position adjustment, the application execution unit 301 displays the application UI 320 of the adjustment screen GM1 on the touch panel 32.

[0095] FIG. 11 is a diagram showing an example of the adjustment screen GM1. The adjustment screen GM1 has a photography result display area HA1. The photography result display area HA1 is a rectangular area that displays the photography results of either the refrigerator compartment camera 203 or the drawer camera 204. In the photography result display area HA1, a line L1 extending in the vertical direction is provided at approximately the center in the horizontal direction in the drawing.

[0096] The adjustment screen GM1 includes adjustment method information J1, which prompts the user to adjust the position of the refrigerator 1 so that the center in the horizontal direction of the image captured in the image capture result display area HA1 is aligned with a line L1 provided in the image capture result display area HA1.

[0097] User P installs camera unit 2 in refrigerator 1 according to the adjustment method indicated in adjustment method information J1. This enables camera unit 2 to capture food items being taken in and out of refrigerator 1 using each of refrigerator compartment camera 203 and drawer camera 204. Camera unit 2 also enables each distance measuring sensor to detect whether the corresponding door 11C or drawer 16A is open or closed.

[0098] In the first position adjustment, camera unit 2 transmits the photographing results of either refrigerator compartment camera 203 or drawer camera 204 to terminal device 3. In the first position adjustment, camera unit 2 and terminal device 3 communicate via food management server 4 or directly by short-range wireless communication or the like.

[0099] The camera unit communication control unit 221 of the camera unit 2 transmits the image capture results of either the refrigerator compartment camera 203 or the drawer camera 204 to the terminal device 3. Then, the application execution unit 301 displays the received image capture results in the image capture result display area HA1.

[0100] [1-2-1-2. Second position adjustment] In the second position adjustment, user P uses center position recording paper CY.

[0101] FIG. 12 is a diagram showing an example of the central position recording paper CY. The center position recording paper CY is a paper on which a center mark MR2 is recorded, which indicates the center position in the left-right direction of the refrigerator 1. In addition to the center mark MR2, the center position recording paper CY also has recorded thereon a left edge mark MR1, which indicates the left edge position of the refrigerator 1 in the left-right direction, and a right edge mark MR3, which indicates the right edge position of the refrigerator 1 in the left-right direction.

[0102] The central position recording sheet CY is packaged together with the refrigerator 1 and delivered to the user P. The camera unit 2 may or may not be packaged together with the central position recording sheet CY and the refrigerator 1. The length of the central position recording sheet CY packaged together with the refrigerator 1, from the left end mark MR1 to the right end mark MR3, corresponds to the width of the refrigerator 1.

[0103] In the second position adjustment, user P uses center position recording paper CY to determine the center position of refrigerator 1 in the left-right direction. For example, user P attaches center position recording paper CY to refrigerator 1 so that left edge mark MR1 is at the left edge of refrigerator 1 and right edge mark MR3 is at the right edge of refrigerator 1, and uses center mark MR2 as a landmark to determine the center position of refrigerator 1 in the left-right direction. User P then adjusts the position of camera unit 2 so that mark 207 on position restriction member 205 of camera unit 2 is at the determined center position.

[0104] The second position adjustment allows the camera unit 2 to perform various detections, similar to the first position adjustment.

[0105] [1-2-1-3. Third position adjustment] In the third position adjustment, a mark indicating the center position in the left-right direction of refrigerator 1 is attached to a predetermined position on refrigerator 1. In the third position adjustment, mark 207 on position restriction member 205 and a mark attached to refrigerator 1 are used.

[0106] In the third position adjustment, user P adjusts the position of camera unit 2 in the left-right direction of refrigerator 1 so that the position of the mark on refrigerator 1 matches the position of mark 207 on position restriction member 205.

[0107] The third position adjustment allows the camera unit 2 to perform various detections, similar to the first position adjustment.

[0108] Next, the operation of food management system 1000 after camera unit 2 is installed in refrigerator 1 will be described. [1-2-2. Recording-related operations] First, the operations relating to recording will be described. FIG. 13 is a flowchart FA showing the operation of the camera unit 2.

[0109] 13, the sensor section 22 outputs the detection values ​​of each sensor to the camera unit control section 20. Then, in the operation shown in FIG. 13, the detection value processing section 222 performs processing based on the various detection values ​​received from the sensor section 22.

[0110] The detection value processing unit 222 determines whether or not a person is present around the refrigerator 1 based on the detection value of the human sensor 212 (step SA1).

[0111] If the detection value processing unit 222 determines that a person is present around the refrigerator 1 (step SA1: YES), it outputs presence information indicating the presence of a person to the photography control unit 223 (step SA2).

[0112] When the presence / absence information indicating the presence of a person is input from the detection value processing unit 222, the photographing control unit 223 causes the refrigerator compartment camera 203 and the drawer camera 204 to start photographing (step SA3).

[0113] Next, the detection value processing unit 222 determines whether any of the left door 11A, the right door 11B, and the drawers 14A and 15A is in the open state based on the detection values ​​of the various sensors output from the sensor unit 22 (step SA4).

[0114] When the detection value processing unit 222 determines that none of the left door 11A, the right door 11B, and the drawers 14A and 15A are open (step SA4: NO), the detection value processing unit 222 performs the process of step SA16.

[0115] If the detection value processing unit 222 determines that any of the left door 11A, right door 11B, drawers 14A, and 15A has entered the open state (step SA4: YES), it determines whether the door 11C has entered the open state or whether any of the drawers 14A and 15A has entered the open state (step SA5).

[0116] When the detection value processing unit 222 determines that the door 11C is in the open state (step SA5: door), it outputs door open state information to the recording control unit 224 (step SA6).

[0117] Next, when the door open state information is input from the detection value processing unit 222, the recording control unit 224 starts recording the photographing results of the refrigerator compartment camera 203 (step SA7).

[0118] Next, the detection value processing unit 222 determines whether the door 11C is in the closed state (step SA8).

[0119] When the detection value processing unit 222 determines that the door 11C is in the closed state (step SA8: YES), it outputs door closed state information to the recording control unit 224 (step SA9).

[0120] Next, when the door closed state information is input from the detection value processing unit 222, the recording control unit 224 ends recording of the photographing results of the refrigerator compartment camera 203 (step SA10). As a result, one recording file 231 of the refrigerator compartment camera 203 is generated in the camera unit storage unit 230.

[0121] Returning to the explanation of step SA5, when the detection value processing unit 222 determines that either of the drawers 14A and 15A has entered the open state (step SA5: Drawer), it outputs the drawer open state information to the recording control unit 224 (step SA11).

[0122] Next, when the drawer closed state information is input from the detection value processing section 222, the recording control section 224 starts recording the results of photography by the drawer camera 204 (step SA12).

[0123] Next, the detection value processing unit 222 determines whether or not either of the open drawers 14A, 15A has become closed (step SA13).

[0124] When the detection value processing unit 222 determines that either of the open drawers 14A, 15A has been closed (step SA13: YES), it outputs the drawer closed state information to the recording control unit 224 (step SA14).

[0125] Next, when the drawer closed state information is input from the detection value processing unit 222, the recording control unit 224 ends recording of the image capture results of the drawer camera 204 (step SA15). As a result, one recording file 231 of the drawer camera 204 is generated in the camera unit storage unit 230.

[0126] The detection value processing unit 222 determines whether or not there is a person around the refrigerator 1 (step SA16).

[0127] If the detection value processing unit 222 determines that a person is present around the refrigerator 1 (step SA16: YES), it performs the processes from step SA4 onwards again.

[0128] On the other hand, if the detection value processing unit 222 determines that no person is present around the refrigerator 1 (step SA16: NO), it outputs presence / absence information indicating that no person is present to the photography control unit 223 (step SA17).

[0129] When the presence / absence information indicating that no person is present is input from the detection value processing unit 222, the photography control unit 223 causes the refrigerator compartment camera 203 and the drawer camera 204 to end photography (step SA18).

[0130] [1-2-3. Food detection operations] Next, the operation relating to food detection will be described. FIG. 14 is a flowchart showing the operation of the food management system 1000. In FIG. 14, a flowchart FB shows the operation of the camera unit 2, and a flowchart FC shows the operation of the food management server 4.

[0131] The food detection unit 225 of the camera unit 2 determines whether or not the recording file 231 has been stored in the camera unit storage unit 230 (step SB1).

[0132] When food detection unit 225 determines that recording file 231 is stored in camera unit storage unit 230 (step SB1: YES), it determines whether stored recording file 231 is recording file 231 of refrigerator compartment camera 203 or recording file 231 of drawer camera 204 (step SB2). The determination in step SB2 is made based on information output by detection value processing unit 222.

[0133] When food detection unit 225 determines that the video file is video file 231 of refrigerator camera 203 (step SB2: refrigerator camera), food detection unit 225 detects food based on stored video file 231 (step SB3).

[0134] In step SB3, for example, food detection unit 225 detects food from video recording file 231 of refrigerator compartment camera 203 based on feature quantities such as shape and color. Data necessary for food detection is stored in a storage area accessible by food detection unit 225.

[0135] Food detection unit 225 determines whether or not food has been detected from video recording file 231 of refrigerator compartment camera 203 (step SB4).

[0136] If food detection unit 225 determines that no food has been detected from video recording file 231 of refrigerator compartment camera 203 (step SB4: NO), it moves the process to step SB10.

[0137] On the other hand, if food is detected from the video recording file 231 of the refrigerator compartment camera 203 (step SB4: YES), the food detection unit 225 performs an in / out determination process (step SB5).

[0138] The in / out determination process is a process for determining whether the food detected in step SB3 is food that has been put into refrigerator 1 or food that has been taken out of refrigerator 1. In the in / out determination process, an in / out determination image HG is superimposed on the video image shown in the video recording file 231 of the refrigerator compartment camera 203.

[0139] FIG. 15 is a diagram for explaining the warehouse entry / exit determination image HG. The storage / retrieval determination image HG shown in FIG. 15 is superimposed on a moving image of user P storing an egg pack into refrigerator 1.

[0140] The entry / exit determination image HG has a boundary line L2. The boundary line L2 has boundary lines L21, L22, L23, L24, and L25. The boundary line L21 is a line that extends rightward from the left edge of the entry / exit judgment image HG at the top of the image. The boundary line L22 is a line that connects the right edge of the boundary line L21 to the left edge of the boundary line L23. The boundary line L23 is a line that extends rightward from the left-right position of the right edge of the boundary line L21 at approximately the center in the up-down direction of the entry / exit judgment image HG. The boundary line L24 is a line that connects the right edge of the boundary line L23 to the left edge of the boundary line L24. The boundary line L25 is a line that extends rightward from the left-right position of the right edge of the boundary line L23 at the top of the entry / exit judgment image HG to the right edge of the entry / exit judgment image HG.

[0141] The image area of ​​the warehouse entry / exit determination image HG is divided by a boundary line L2 into a concave area HG1, which is a concave area, and a convex area HG2, which is a convex area.

[0142] Food detection unit 225 performs image analysis on the moving image on which the entry / exit determination image HG is superimposed to determine whether the food detected in step SB3 has moved from convex region HG2 to concave region HG1 or from concave region HG1 to convex region HG2. If food detection unit 225 determines that the food detected in step SB3 has moved from convex region HG2 to concave region HG1, it determines that the food detected in step SB3 has been entered into refrigerator 1. On the other hand, if food detection unit 225 determines that the food detected in step SB3 has moved from concave region HG1 to convex region HG2, it determines that the food detected in step SB3 has been removed from refrigerator 1.

[0143] Returning to the explanation of the flowchart FB in FIG. 14, the food detection unit 225 performs a divided area number identification process (step SB6).

[0144] The divided area number identification process is a process of identifying the divided area BA corresponding to the storage area into which the food detected in step SB3 was stored and removed, and identifying the divided area number 4118 assigned to the identified divided area BA.

[0145] For example, food detection unit 225 uses image analysis to identify the storage area in refrigerator compartment 11 into which the food detected in step SB3 was transferred. Furthermore, for example, a mark identifying the storage area is attached to the top end of each shelf 112 and inside each door pocket 111, and food detection unit 225 identifies the storage area based on the positional relationship between this mark and the detected food in the video. Next, food detection unit 225 determines which divided area BA in refrigerator interior image RG the identified storage area corresponds to, and identifies divided area number 4118 assigned to the identified divided area BA. The correspondence between the storage area, divided area BA, and divided area number 4118 is stored as data in camera unit storage unit 230.

[0146] After completing the divided area number identification process, food detection unit 225 extracts food images for each food item detected in step SB3 from video recording file 231 of refrigerator compartment camera 203 (step SB7). Hereinafter, the food images extracted from the photographing results of camera 213 will be referred to as "photographed food images" and will be assigned the symbol "FG." The photographed food image FG corresponds to an example of a food image of the present disclosure.

[0147] Next, the food detection unit 225 generates first update information (step SB8).

[0148] The first update information generated in step SB8 includes the camera unit ID 4112 of the camera unit storage section 230. The first update information generated in step SB8 also includes date and time information indicating the date and time when the video recording file 231 of the refrigerator compartment camera 203 was generated. In addition, the first update information generated in step SB8 includes, for each food detected in step SB3, food image data 4116 indicating the photographed food image FG cut out in step SB7, in-stock / out-stock determination result information indicating the determination result of the in-stock / out-stock determination process, and the divided area number 4118 identified in the divided area number identification process.

[0149] Next, the camera unit communication control unit 221 transmits the first update information generated by the food detection unit 225 to the food management server 4 (step SB9).

[0150] Food detection unit 225 deletes recording file 231 from camera unit storage unit 230 (step SB10).

[0151] Returning to the explanation of step SB2, when the food detection unit 225 determines that the file is the recording file 231 of the drawer camera 204 (step SB2: Drawer Camera), it determines whether at least one of the upper case JCA and the lower case GCA shown in the video image of the recording file 231 of the drawer camera 204 has been pulled out by more than a predetermined threshold (step SB11).

[0152] For example, the food detection unit 225 distinguishes the cases CA in the video based on feature quantities such as shape and color. Then, the food detection unit 225 determines whether the upper case JCA and the lower case GCA have been pulled out from the refrigerator 1 by a predetermined threshold or more through image analysis of the video file 231. Furthermore, for example, upper case JCA and lower case GCA each have a mark at a predetermined position to identify case CA, and food detection unit 225 distinguishes case CA in the moving image by reading this mark from the moving image. Then, food detection unit 225 calculates the movement distance of this mark in the moving image to determine whether the food has been pulled out of refrigerator 1 by more than a predetermined threshold.

[0153] If the food detection unit 225 determines that neither the upper case JCA nor the lower case GCA has been pulled out by the predetermined threshold value or more (step SB11: NO), it performs the process of step SB18.

[0154] On the other hand, if the food detection unit 225 determines that at least one of the upper case JCA and the lower case GCA has been pulled out by the predetermined threshold or more (step SB11: YES), it cuts out a case image CG from the video file 231 (step SB12).

[0155] In step SB12, the food detection unit 225 extracts an upper case image JCG from the video file 231 if only the upper case JCA is pulled out by more than the predetermined threshold value. In step SB12, the food detection unit 225 cuts out the lower case image GCG from the video file 231 if only the lower case GCA is pulled out by a predetermined threshold or more. In step SB12, the food detection unit 225 extracts an upper case image JCG and a lower case image GCG from the video file 231 if the upper case JCA and the lower case GCA are pulled out by a predetermined threshold or more.

[0156] Next, the food detection unit 225 detects food from the cut-out case image CG (step SB13).

[0157] Next, the food detection unit 225 determines whether or not food has been detected from the case image CG (step SB14).

[0158] If the food detection unit 225 determines that no food can be detected (step SB14: NO), it performs the process of step SB16.

[0159] On the other hand, if the food detection unit 225 determines that the food has been detected (step SB14: YES), it cuts out a photographed food image FG for each detected food item from the case image CG (step SB15).

[0160] Next, the food detection unit 225 generates second update information (step SB16).

[0161] If only the upper case image JCG is cut out, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage unit 230, upper case image data 4123 indicating the cut-out upper case image JCG, and food image data 4116 indicating the photographed food image FG cut out from the upper case image JCG. Note that if a negative determination is made in step SB14, the second update information does not include the food image data 4116.

[0162] Furthermore, if only the lower case image GCG is cut out, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage unit 230, lower case image data 4125 indicating the cut-out lower case image GCG, and food image data 4116 indicating the photographed food image FG cut out from the lower case image GCG. Note that if a negative determination is made in step SB14, the second update information does not include the food image data 4116.

[0163] Furthermore, when the upper case image JCG and the lower case image GCG are cut out, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage unit 230, the upper case image data group, and the lower case image data group. The upper case image data group includes upper case image data 4123 indicating the cut-out upper case image JCG and food image data 4116 indicating the photographed food image FG cut out from the upper case image JCG. The lower case image data group includes lower case image data 4125 indicating the cut-out lower case image GCG and food image data 4116 indicating the photographed food image FG cut out from the lower case image GCG. Note that if a negative determination is made in step SB14, the second update information does not include the food image data 4116.

[0164] Next, the camera unit communication control unit 221 transmits the second update information generated by the food detection unit 225 to the food management server 4 (step SB17).

[0165] Next, food detection section 225 deletes recording file 231 from camera unit storage section 230 (step SB18).

[0166] As shown in the flowchart FC, the server communication control unit 401 of the food management server 4 determines whether or not either the first update information or the second update information has been received (step SC1).

[0167] If server communication control unit 401 determines that update information has been received (step SC1: YES), information processing unit 402 updates the database based on the update information received by server communication control unit 401 (step SC2).

[0168] Step SC2 will now be described in detail. If the update information received by the server communication control unit 401 is the first update information, the information processing unit 402 identifies the refrigerator food management record RR that has the camera unit ID 4112 included in the first update information. Then, the information processing unit 402 updates the stock / unstock presence information 4115 of the identified refrigerator food management record RR to information indicating "present."

[0169] Next, if the in / out determination result information included in the first update information indicates in-stock, information processing unit 402 generates a sub-record, associates blank deletion information 4131 and deletion date / time information 4132 with the generated sub-record, and stores the sub-record in the identified refrigerator compartment food management record RR. The sub-record generated here includes food image data 4116 included in the first update information, storage date / time information 4117 indicating the same date and time as the date / time information included in the first update information, divided area number 4118 included in the first update information, blank memo entry presence / absence information 4119, and blank memo information 4130.

[0170] Furthermore, when the inbound / outbound determination result information included in the first update information indicates outbound, the information processing unit 402 identifies a sub-record from the refrigerator compartment food management record RR that includes food image data 4116 indicating a photographed food image FG that closely matches the photographed food image FG indicated by the food image data 4116 included in the first update information. This identification is performed for each food image data 4116 included in the first update information. The information processing unit 402 then updates the deletion information 4131 associated with the identified sub-record to information indicating that the sub-record is to be deleted. The information processing unit 402 also updates the deletion date and time information 4132 associated with the identified sub-record to information indicating the current date and time as the deletion date and time.

[0171] Step SC2 will now be described in detail. If the update information received by the server communication control unit 401 is the second update information, the information processing unit 402 identifies the withdrawn food management record HR having the camera unit ID 4112 included in the second update information.

[0172] If the second update information is for the case where only the upper case image JCG is extracted, the information processing unit 402 updates the upper case image data 4123 and food image data 4116 contained in the upper case information 4122 of the identified drawer food management record HR to the data contained in the second update information. If the second update information is for the case where only the lower case image GCG is cut out, the information processing unit 402 updates the lower case image data 4125 and food image data 4116 contained in the lower case information 4124 of the identified drawer food management record HR to the data contained in the second update information. If the second update information is for the case where the upper case image JCG and the lower case image GCG have been extracted, the information processing unit 402 appropriately updates the upper case image data 4123 and the food image data 4116 contained in the upper case information 4122 of the identified drawer food management record HR to the data contained in the upper case image data group. Also, the information processing unit 402 updates the lower case image data 4125 and the food image data 4116 contained in the lower case information 4124 of the identified drawer food management record HR to the data contained in the lower case image data group.

[0173] Furthermore, the information processing unit 402 updates the update date and time information 4121 of the identified withdrawn food management record HR to information indicating the date and time when the withdrawn food management record HR was updated.

[0174] [1-2-4. Actions related to app UI] Next, the operation related to the application UI 320 will be described. In this embodiment, the application UI 320 displays a screen relating to the refrigerator compartment 11 and a screen relating to the drawer-type storage compartment.

[0175] [1-2-4-1. Refrigerator related screens]

[0176] The screens related to the refrigerator compartment 11 include a refrigerator compartment home screen GM2, a refrigerator compartment list screen GM3, and a food information addition list screen GM4.

[0177] FIG. 16 is a diagram showing an example of the refrigerator compartment home screen GM2.

[0178] The refrigerator compartment home screen GM2 has a food image display area HA2. The food image display area HA2 is an area for displaying photographed food images FG. The food image display area HA2 displays the refrigerator compartment interior image RG, and the photographed food images FG are displayed superimposed on the refrigerator compartment interior image RG. Each photographed food image FG is displayed superimposed on the divided area BA indicated by the divided area number 4118 corresponding to the food image data 4116 in the refrigerator compartment interior image RG.

[0179] The food image display area HA2 in Fig. 16 displays the refrigerator compartment interior image RG shown in Fig. 9. For example, in the refrigerator compartment food management record RR, food image data 4116 representing photographed food image FG1 in Fig. 16 is associated with divided area number 4118 of "4." In this case, photographed food image FG1 is displayed superimposed on divided area BA to which divided area number 4118 of "4" is assigned.

[0180] The default food image display area HA2 displays the refrigerator compartment interior image RG with the left door pocket image RG1 and the right door pocket image RG3 cut off. The food image display area HA2 accepts a scroll operation in the left and right directions in the figure from the user P. The food image display area HA2 changes the display range of the refrigerator compartment interior image RG according to the scroll amount and scroll direction of the accepted scroll operation.

[0181] For example, suppose the scroll direction is right and the scroll amount is such that the left edge of the left door pocket image RG1 is displayed. In this case, the food image display area HA2 displays the refrigerator compartment interior image RG with the right door pocket image RG3 completely cut off. Note that the photographed food image FG and the memo entry icon MIC move in accordance with changes in the display range of the refrigerator compartment interior image RG, in other words, with the movement of the superimposed divided area BA. The memo entry icon MIC will be described later.

[0182] When multiple photographed food images FG are displayed in one divided area BA, food image display area HA2 displays them side by side in the left-right direction of the figure. Food image display area HA2 can display up to five photographed food images FG side by side simultaneously in one divided area BA. Food image display area HA2 accepts scrolling operations from user P when there are six or more photographed food images FG to be displayed in one divided area BA. When one divided area BA accepts scrolling operations, the first divided area BA displays the sixth and subsequent photographed food images FG. When there are six or more photographed food images FG to be displayed in one divided area BA and a scrolling operation in the left-right direction is accepted, food image display area HA2 displays the sixth and subsequent photographed food images FG according to the amount of scrolling while maintaining the maximum number of photographed food images that can be displayed side by side. At this time, the food image display area HA2 hides the first through fifth food images FG that were displayed, or displays the first and subsequent food images FG after the sixth and subsequent food images FG, depending on the amount of scrolling and the number of photographed food images FG to be displayed. Note that the memo input icon MIC moves in accordance with the movement of the associated photographed food images FG. In other words, when an associated photographed food image FG is hidden by a scrolling operation, the memo input icon MIC also becomes hidden.

[0183] Each photographed food image FG displayed in the food image display area HA2 accepts input of a memo. The memo is information that the user P wants to leave for the food in the refrigerator 1.

[0184] FIG. 17 is a diagram showing the procedure for inputting a memo.

[0185] On the refrigerator compartment home screen GM2, the user P performs a predetermined touch operation, such as a long press, on a photographed food image FG for which the user P wants to enter a memo. When the application execution unit 301 receives the predetermined touch operation on one photographed food image FG, it displays a memo edited image MHG in association with the photographed food image FG on which the predetermined touch operation was performed.

[0186] The memo edit image MHG has a memo input area MNA, a memo save button B1, a memo delete button B2, and a food delete button B3.

[0187] The memo input area MNA is an area for inputting a memo. When the memo input area MNA is touched, the application execution unit 301 displays a software keyboard on the touch panel 32. Then, the application execution unit 301 inputs a memo corresponding to the operation on the software keyboard into the memo input area MNA.

[0188] The save memo button B1 is a software button for saving a memo entered in the memo input area MNA. The save memo button B1 cannot be touched if no memo has been entered in the memo input area MNA. When the save memo button B1 is touched, the application execution unit 301 displays the photographed food image FG with the memo entered on the screen related to the refrigerator compartment 11 in a display mode different from the display mode of the photographed food image FG without the memo entered. In this embodiment, the application execution unit 301 displays the frame of the photographed food image FG with the memo entered in a different color, and associates the photographed food image FG with the memo entered with a memo input icon MIC, which is an icon indicating that a memo has been entered. The memo input icon MIC can be touched. When the memo input icon MIC is touched, the entered memo is displayed in the application UI 320. The memo input icon MIC corresponds to an example of a memo input image of the present disclosure.

[0189] The memo erase button B2 is a button for erasing the input memo. When the memo erase button B2 is touched, the memo input to the photographed food image FG is erased.

[0190] The food deletion button B3 is a button for deleting the photographed food image FG from the food image display area HA2.

[0191] Returning to the description of the refrigerator compartment home screen GM2 in Fig. 16, the food image display area HA2 has a list display button B4, which is a touch-operable software button.

[0192] When the list display button B4 is touched, the app UI 320 transitions to a refrigerator compartment list screen GM3.

[0193] The refrigerator compartment list screen GM3 includes a storage list screen GM31, a new addition list screen GM32, and a deletion target list screen GM33.

[0194] The accommodation list screen GM31 will be explained. FIG. 18 is a diagram showing the accommodation list screen GM31.

[0195] The storage list screen GM31 has a stored food information display area HA3. The stored food information display area HA3 displays a list of first food information J21 for each food item stored in the refrigerator compartment 11. The first food information J21 includes a photographed food image FG, storage date and time information 4117, and divided area information J3. The divided area information J3 is a refrigerator compartment interior image RG in which the divided area BA is displayed in color.

[0196] If a memo has been entered into the photographed food image FG of the first food information J21, the frame is displayed in a different color and a memo entry icon MIC is associated with it. The contained food information display area HA3 can be scrolled up and down in the figure. By scrolling the contained food information display area HA3, user P can display the first food information J21 that cannot be displayed in the contained food information display area HA3.

[0197] The accommodation list screen GM31 has a first switching button B5 and a second switching button B6.

[0198] The first switching button B5 is a software button for switching the order of the first food information J21 displayed in the stored food information display area HA3 to the next order. That is, when the first switching button B5 is touched, the stored food information display area HA3 displays the first food information J21 having the photographed food image FG with a note input higher than the first food information J21 having the photographed food image FG without a note input. Displaying higher means displaying in a way that reduces the amount of scrolling required when displaying. Furthermore, when there are multiple pieces of first food information J21 having photographed food images FG with a note input, the stored food information display area HA3 displays the food information J2 with the oldest storage date and time indicated by the storage date and time information 4117 higher. Furthermore, when there are multiple pieces of first food information J21 having photographed food images FG without a note input, the stored food information display area HA3 displays the first food information J21 with the oldest storage date and time indicated by the storage date and time information 4117 higher. The accommodation list screen GM31 on the left side of FIG. 18 shows the accommodation list screen GM31 when the first switching button B5 is touched.

[0199] The second switching button B6 is a software button for switching the order of the first food information J21 displayed in the stored food information display area HA3 to the next order. That is, when the second switching button B6 is touched, the contained food information display area HA3 displays the first food information J21 in each divided area BA in an order that may be predetermined by the user P. In each divided area BA, the contained food information display area HA3 displays the first food information J21 in the same order as when the first switching button B5 is touched. The accommodation list screen GM31 on the right side of FIG. 18 shows the accommodation list screen GM31 when the second switching button B6 is touched.

[0200] The accommodation list screen GM31 has an accommodation list display button B7, a new addition list display button B8, and a deletion target list display button B9.

[0201] The accommodation list display button B7 is a software button for switching the screen displayed by the application UI320 to the accommodation list screen GM31. The new addition list display button B8 is a software button for switching the screen displayed by the application UI320 to the new addition list screen GM32. The deletion target list display button B9 is a software button for switching the screen displayed by the application UI320 to a deletion target list screen GM33.

[0202] FIG. 19 is a diagram showing an example of the new addition list screen GM32. The newly added food list screen GM32 has a newly added food information display area HA4. The newly added food information display area HA4 displays a list of first food information J21 related to foods that have been newly stored in the refrigerator compartment 11. Note that foods that have been newly stored in the refrigerator compartment 11 refer to foods that have been stored within a predetermined period of time (e.g., three days) from the current date and time. The newly added food information display area HA4 displays the first food information J21 of foods that correspond to newly stored foods in order of the most recent storage date and time indicated by the storage date and time information 4117.

[0203] Each of the first food information J21 displayed in the newly added food information display area HA4 accepts a predetermined touch operation on the photographed food image FG. On the newly added food list screen GM32, a first edited image HNG1 is displayed in association with the photographed food image FG that has been touched.

[0204] The first edited image HNG1 has a food delete button B10, an area correction button B11, and a back button B12. The food deletion button B10 is a software button that, when touched, causes the food shown in the photographed food image FG associated with the first edited image HNG1 to be deleted from the refrigerator compartment home screen GM2. The area correction button B11 is a software button that, when touched, corrects the divided area BA indicated by the divided area information J3. The back button B12 is a software button for stopping the display of the first edited image HNG1.

[0205] The newly added food information display area HA4 has an Add Food Information button B13 at the end of the list of first food information J21. The Add Food Information button B13 is a software button for manually adding new first food information J21. The procedure for adding the first food information J21 will be described with reference to FIG.

[0206] FIG. 20 is a diagram illustrating the procedure for adding the first food information J21.

[0207] When the Add Food Information button B13 is touched, the app UI 320 displays the Add Food Information List screen GM4. The Add Food Information List screen GM4 has multiple Add buttons B14, which are software buttons. The Add Food Information List screen GM4 also has a blank image BG and a food name input area SNA for inputting a food name. When the blank image BG is touched, an image appears within the blank image BG that shows the food name input in the food name input area SNA.

[0208] When the Add button B14 is touched on the food information addition list screen GM4, a refrigerator compartment interior image RG is displayed on the app UI 320. Here, the displayed refrigerator compartment interior image RG is an image in which each divided area BA is assigned a divided area number 4118, and each divided area BA can be selected by touching. When a divided area BA is selected on this refrigerator compartment interior image RG, the next first food information J21 is added on the app UI 320.

[0209] That is, the added first food information J21 includes an image of the food corresponding to the touched Add button B14, storage date and time information 4117 indicating the date and time when the first food information J21 was added as the storage date and time, and divided area information J3 indicating the selected divided area BA. Note that the image of the food indicated by the added first food information J21 may also be an icon. In this embodiment, it is assumed that the image superimposed on the refrigerator compartment interior image RG on the refrigerator compartment home screen GM2 is a shadow food image FG. However, when the first food information J21 is added, an icon is displayed on the refrigerator compartment interior image RG together with the photographed food image FG.

[0210] The procedure for adding the first food information J21 may include a step in which the user P inputs the storage date and time. In this case, the added first food information J21 includes storage date and time information 4117 indicating the storage date and time input by the user P.

[0211] Returning to the explanation of FIG. 19, the new addition list screen GM32 has an accommodation list display button B7, a new addition list display button B8, and a deletion target list display button B9.

[0212] FIG. 21 is a diagram showing an example of the deletion target list screen GM33.

[0213] The deletion target list screen GM33 has a deletion target food information display area HA5. The deletion target food information display area HA5 displays a list of second food information J22 for each food to be deleted. Note that foods to be deleted refer to foods to be deleted from the refrigerator compartment home screen GM2 and foods determined to have been removed from the refrigerator compartment 11. The second food information J22 includes photographed food images FG, deletion date and time information 4132, and divided area information J3. The deletion target list screen GM33 displays the second food information J22 in order of most recent deletion date and time.

[0214] Each of the second food information J22 displayed in the deletion target food information display area HA5 receives a predetermined touch operation on the photographed food image FG. On the deletion target list screen GM33, a second edited image HNG2 is displayed in association with the photographed food image FG that has been touched.

[0215] The second edited image HNG2 has a delete cancel button B15 and a back button B16. The delete cancel button B15 is a software button for setting the photographed food image FG associated with the second edited image HNG2 as the image displayed on the refrigerator compartment home screen GM2. The back button B16 is a software button for stopping the display of the second edited image HNG2.

[0216] [1-2-4-2. Food Control System Operation] Next, the operation of food management system 1000 when displaying a screen related to refrigerator compartment 11 will be described.

[0217] 22 is a flowchart showing the operation of the food management system 1000. In Fig. 22, a flowchart FD shows the operation of the terminal device 3, and a flowchart FE shows the operation of the food management server 4.

[0218] The application execution unit 301 of the terminal device 3 determines whether to request the refrigerator compartment food management record RR from the food management server 4 (step SD1).

[0219] For example, when the application executing unit 301 receives a touch operation on the application UI 320 to instruct display of the refrigerator compartment home screen GM2, the application executing unit 301 makes an affirmative determination in step SD1. Also, for example, if a touch operation to instruct updating of the display is performed on the refrigerator compartment home screen GM2, the application executing unit 301 makes an affirmative decision in step SD1.

[0220] If the application execution unit 301 determines that the refrigerator management record RR is to be requested (step SD1: YES), it transmits first record request information requesting the refrigerator management record RR to the food management server 4 (step SD2). The first record request information transmitted in step SD2 includes the account ID 4111 stored in the device storage unit 310.

[0221] As shown in flowchart FF, the server communication control unit 401 of the food management server 4 receives the first record request information via the server communication unit 41 (step SE1).

[0222] Next, the information processing unit 402 identifies the refrigerator food management record RR that includes the account ID 4111 included in the first record request information received by the server communication control unit 401 from the refrigerator compartment food management database 411 (step SE2).

[0223] Then, the server communication control unit 401 transmits the refrigerator food management record RR identified by the information processing unit 402 to the terminal device 3 (step SE3). The information processing unit 402 updates the in-stock / out-stock presence information 4115 of the transmitted refrigerator food management record RR to information indicating "absent."

[0224] Referring to the flowchart FD, when the application execution unit 301 of the terminal device 3 receives the refrigerator food management record RR from the food management server 4 (step SD3), it generates a screen related to the refrigerator 11 based on this refrigerator food management record RR (step SD4).

[0225] Step SD4 will now be described in detail. The application execution unit 301 generates a refrigerator home screen GM2 that displays, in the food image display area HA2, the refrigerator interior image RG indicated by the refrigerator interior image data 4114 included in the refrigerator food management record RR. Furthermore, the application execution unit 301 generates a refrigerator compartment home screen GM2 in which the photographed food image FG indicated by the food management image data 4116 is superimposed on the divided area BA indicated by the divided area number 4118 for each sub-record. Furthermore, if the memo entry information 4119 indicates that a memo exists, the application execution unit 301 generates a refrigerator home screen GM2 that displays a memo entry icon MIC associated with the photographed food image FG for each sub-record.

[0226] Step SD4 will now be described in detail. The application execution unit 301 generates first food information J21 for each piece of food image data 4116. This first food information J21 includes the photographed food image FG indicated by the food image data 4116, storage date and time information 4117 associated with the food image data 4116, and divided area information J3 in which the divided area BA indicated by the divided area number 4118 is colored. The application execution unit 301 then generates a storage list screen GM31 that displays a list of first food information J21 related to sub-records associated with deletion information 4131 that does not indicate a deletion target. The application execution unit 301 also generates a new addition list screen GM32 that displays a list of first food information J21 displayed on the storage list screen GM31, where the storage date and time information 4117 indicates a date and time within a predetermined period from the current time.

[0227] Step SD4 will now be described in detail. The application execution unit 301 generates second food information J22 for each piece of food image data 4116. The second food information J22 includes the photographed food image FG indicated by the food image data 4116, deletion date and time information 4132 associated with the food image data 4116, and divided area information J3 in which the divided area BA indicated by the divided area number 4118 is colored. The application execution unit 301 then generates a deletion target list screen GM33 that displays a list of the generated second food information J22.

[0228] Returning to the explanation of the flowchart FD, the application execution unit 301 displays a screen related to the refrigerator compartment 11 in response to the touch operation (step SD5). If the stock / warehousing presence information 4115 of the refrigerator compartment food management record RR received in step SD3 indicates "present," when the refrigerator compartment home screen GM2 is displayed for the first time in step SD5, the application execution unit 301 displays a pop-up on the refrigerator compartment home screen GM2 indicating that new stock / warehousing has occurred.

[0229] Next, the operation of food management system 1000 when an operation is performed on the screen related to refrigerator compartment 11 will be described.

[0230] When the memo save button B1 is touched, the application execution unit 301 sends information to the food management server 4 to update the sub-record for the photographed food image FG that is the target of the touch operation. This information includes memo entry presence / absence information 4119 indicating the presence of a memo, and memo information 4130 indicating the memo that has been entered. The food management server 4 updates the sub-record appropriately based on the received information.

[0231] When the memo delete button B2 is touched, the application execution unit 301 sends information to the food management server 4 to update the sub-record for the photographed food image FG that is the target of the operation. This information includes memo entry presence / absence information 4119, which indicates that no memo has been entered. The food management server 4 updates the sub-record appropriately based on the received information.

[0232] When the food deletion buttons B3 and B10 are touched, the application execution unit 301 transmits information for updating the sub-record for the photographed food image FG to be operated to the food management server 4. This information includes deletion information 4131 indicating the food to be deleted. The food management server 4 updates the sub-record appropriately based on the received information.

[0233] When the area correction button B11 is touched to correct the divided area BA, the application execution unit 301 transmits information for updating the sub-record for the photographed food image FG to be operated to the food management server 4. This information includes the divided area number 4118 of the corrected divided area BA. The food management server 4 updates the sub-record appropriately based on the received information.

[0234] 20, the application execution unit 301 transmits information for adding a sub-record related to the added first food information J21 to the food management server 4. The food management server 4 updates the sub-record appropriately based on the received information.

[0235] When the cancel deletion button B15 is touched, the application execution unit 301 sends information to the food management server 4 to update the sub-record for the photographed food image FG that is the target of the operation. This information includes deletion information 4131 indicating blank. The food management server 4 updates the sub-record appropriately based on the received information.

[0236] [1-2-4-3. Drawer-type containment room screen] Next, a screen related to the drawer-type storage compartment will be described. In the description using Figures 23 to 25, the vegetable compartment 15 will be used as an example of the drawer-type storage compartment. In this description, the database shown in Figure 10 is a database that stores various information related to the food in the vegetable compartment 15.

[0237] The screens related to the vegetable compartment 15 include a vegetable compartment home screen GM5 and a food image list screen GM6.

[0238] FIG. 23 is a diagram showing an example of the vegetable compartment home screen GM5.

[0239] The crisper home screen GM5 has a case image display area HA6. The case image display area HA6 is an area where the case image CG is displayed. The case image display area HA6 displays the upper case image JCG superimposed on the lower case image GCG. In the default case image display area HA6, the upper case image JCG is superimposed over the entire image area of ​​the lower case image GCG, and only the upper case image JCG is displayed.

[0240] The case image display area HA6 accepts scrolling operations in the up and down directions of the figure. The case image display area HA6 changes the display range of the upper case image JCG depending on the scroll amount and scroll direction of the accepted scrolling operation. For example, if the scroll direction of the accepted scrolling operation is upward, the case image display area HA6 displays a partially cut-off upper case image JCG. When the upper case image JCG moves upward in the case image display area HA6, the lower case image GCG on which the upper case image JCG was superimposed is displayed.

[0241] Each case image CG displayed in the case image display area HA6 accepts the input of memos. The procedure for inputting memos is the same as that for the refrigerator compartment 11, except that the image operated by the user P is either the photographed food image FG or the case image CG. As shown in FIG. 23, the memo input icon MIC associated with the upper case image JCG moves in accordance with the movement of the upper case image JCG.

[0242] The vegetable compartment home screen GM5 has a list display button B17. The list display button B17 is a touch-operable software button.

[0243] When the list display button B17 is touched, the app UI 320 displays a food image list screen GM6.

[0244] The vegetable compartment home screen GM5 may have a switch button for switching the case image CG displayed in the case image display area HA6 to an upper case image JCG or a lower case image GCG. In this configuration, the application execution unit 301 switches the displayed case image CG to an upper case image JCG or a lower case image GCG in response to an operation on the switch button.

[0245] FIG. 24 is a diagram showing an example of the food image list screen GM6.

[0246] The food image list screen GM6 has update date and time information 4121. The food image list screen GM6 in Fig. 24 has update date and time information 4121 that indicates the update date and time of "12 / 1 10:05".

[0247] The food image list screen GM6 displays a list of photographed food images FG for each case. The photographed food image FG displayed in association with the upper case JCA item is an image cropped from the upper case image JCG. The photographed food image FG displayed in association with the lower case GCA item is an image cropped from the lower case image GCG.

[0248] [1-2-4-5. Food Control System Operation] Next, the operation of food management system 1000 when displaying a screen related to vegetable compartment 15 will be described.

[0249] 25 is a flowchart showing the operation of the food management system 1000. In FIG. 11, the flowchart FF shows the operation of the terminal device 3, and the flowchart FJ shows the operation of the food management server 4.

[0250] The application execution unit 301 of the terminal device 3 determines whether or not to request the withdrawal management record HR from the food management server 4 (step SF1).

[0251] For example, when the application executing unit 301 receives a touch operation on the application UI 320 to instruct display of the vegetable compartment home screen GM5, the application executing unit 301 makes a positive determination in step SF1. Also, for example, when a touch operation to instruct updating of the display is performed on the vegetable compartment home screen GM5, the application executing unit 301 makes an affirmative determination in step SF1.

[0252] If the application execution unit 301 determines that the withdrawal management record RR is to be requested (step SF1: YES), it transmits second record request information requesting the withdrawal management record HR to the food management server 4 (step SF2). The second record request information transmitted in step SF2 includes the account ID 4111 stored in the terminal storage unit 310.

[0253] Referring to flowchart FJ, the server communication control unit 401 of the food management server 4 receives the second record request information (step SJ1).

[0254] Next, the information processing unit 402 identifies the withdrawal management record HR that includes the account ID 4111 included in the second record request information received by the server communication control unit 401 from the withdrawn food management database 412 (step SJ2).

[0255] Then, the server communication control unit 401 transmits the withdrawal management record HR identified by the information processing unit 402 to the terminal device 3 (step SJ3).

[0256] Referring to flowchart FJ, when the application execution unit 301 of the terminal device 3 receives the drawn food management record HR from the food management server 4 (step SF3), it generates a screen related to the vegetable compartment 15 (step SF4).

[0257] Step SF4 will now be described in detail. The application execution unit 301 generates a vegetable compartment home screen GM5 having a case image display area HA6 in which the lower case image GCG indicated by the lower case image data 4125 included in the drawer food management record HR is superimposed with the upper case image JCG indicated by the upper case image data 4123 included in the same record. Furthermore, if the memo input presence / absence information 4119 indicates that a memo exists, the application execution unit 301 generates a vegetable drawer home screen GM5 that displays a memo input icon MIC in association with the corresponding case image CG.

[0258] Step SF4 will now be described in detail. The application execution unit 301 generates a food image list screen GM6 on which photographed food images FG indicated by the food image data 4116 are arranged for each case CA.

[0259] Returning to the explanation of the flowchart FF, the application execution unit 301 displays a screen related to the vegetable compartment 15 in response to the touch operation (step SF5).

[0260] Next, the operation of food management system 1000 when an operation is performed on a screen related to vegetable compartment 15 will be described.

[0261] When the memo save button B1 is touched, the application execution unit 301 transmits information to the food management server 4 to update the memo entry presence / absence information 4119 and the memo information 4130. This information includes information indicating whether the operation target is the upper case image JCG or the lower case image GCG, the memo entry presence / absence information 4119 indicating that a memo exists, and the memo information 4130 indicating the memo that has been entered. The food management server 4 appropriately updates the withdrawn food management record HR based on the received information.

[0262] When the memo deletion button B2 is touched, the application execution unit 301 transmits information for updating the memo entry presence / absence information 4119 and the memo information 4130 to the food management server 4. This information includes information indicating whether the memo is being deleted from the upper case image JCG or the lower case image GCG, and the memo entry presence / absence information 4119 indicating that no memo has been entered. The food management server 4 appropriately updates the withdrawal food management record HR based on the received information.

[0263] 23-25, the vegetable compartment 15 is exemplified as a drawer-type storage compartment. However, the screen displays and operations of the food management system 1000 shown in FIGS. 23-25 ​​may also be performed in the same manner for the freezer compartment 14. In this case, the database shown in FIG. 10 is a database that stores various information related to the food in the freezer compartment 14.

[0264] [1-3.Effects] As explained above, camera unit 2 comprises main body 201 installed on top surface 10B of refrigerator 1, and photographing member 202 that extends from main body 201 to the front of refrigerator 1 and is equipped with camera 213 that photographs the area from above and below the front of refrigerator 1. Main body 201 comprises position restricting member 205 that abuts front edge 10A of refrigerator 1 and restricts the position of photographing member 202 in the front-to-rear direction of refrigerator 1.

[0265] In this way, the position of the photographing member 202 in the front-to-rear direction of the refrigerator 1 is restricted by the position restricting member 205, so that the camera 213 is positioned so that it can photograph the area from above to below the front of the refrigerator 1. Therefore, food items being taken in and out of the refrigerator 1 can be accurately included in the photographs taken by the camera 213, and food items being taken in and out of the refrigerator 1 can be accurately photographed.

[0266] The position restricting member 205 is a position restricting piece that extends downward from the bottom surface 206 of the main body 201 in the installed state.

[0267] This allows the position of camera unit 2 to be regulated by utilizing the gap formed between main box body 10 and door 11C. Therefore, camera unit 2 can be installed in an existing refrigerator 1, and food items entering and leaving the refrigerator 1 can be photographed with high accuracy even in an existing refrigerator 1.

[0268] The position restricting member 205 is formed to be shorter than the length to the packing provided between the refrigerator 1 and the door 11C of the refrigerator 1.

[0269] This allows the position of camera unit 2 to be regulated by utilizing the gap formed between main box body 10 and door 11C without impeding the cooling effect of refrigeration compartment 11. Therefore, food entering and leaving refrigerator 1 can be photographed with high accuracy without impeding the cooling effect of refrigeration compartment 11.

[0270] The position restriction member 205 extends in the width direction of the main body 201 .

[0271] This ensures a large contact area with the main box body 10, stabilizing the installation posture of the camera unit 2. As a result, food items being taken in and out of the refrigerator 1 can be photographed with higher accuracy.

[0272] Position restriction member 205 has mark 207 for adjusting the position of camera unit 2 in the width direction of refrigerator 1.

[0273] By using mark 207 as a guide, user P can easily install camera unit 2 on the left and right sides of refrigerator 1. Therefore, user P can easily install camera unit 2 so that food items entering and leaving refrigerator 1 can be photographed with high accuracy.

[0274] The position restricting member 205 has a mark 207 at the center in the width direction of the main body 201 .

[0275] According to this, by using mark 207 as a guide, user P can easily install camera unit 2 in the center of the left-right direction of refrigerator 1. Therefore, user P can easily install camera unit 2 so that food items entering and leaving refrigerator 1 can be photographed with greater accuracy.

[0276] As explained above, the camera unit 2 includes a camera 213 that captures moving images of the refrigerator 1 from above and in front of the refrigerator 1, a shooting control unit 223 that causes the camera 213 to start shooting when the human presence sensor 212 that detects whether or not a person is present around the refrigerator 1 detects that a person is present around the refrigerator 1, and a recording control unit 224 that starts recording the results of the shooting by the camera 213 at a predetermined timing after the shooting control unit 223 causes the camera 213 to start shooting. Furthermore, the control method of camera unit 2 is such that, when human presence sensor 212, which detects whether or not there is a person around refrigerator 1, detects that there is a person around refrigerator 1, camera 213 is made to start capturing images, and at a predetermined timing after camera 213 has started capturing images, recording of the captured images by camera 213 is started.

[0277] According to camera unit 2 and the control method for camera unit 2, when motion sensor 212 detects the presence of a person near refrigerator 1, camera 213 starts capturing images, which increases the likelihood that food items being taken in or out of refrigerator 1 before door 11C or drawer 16A of refrigerator 1 is opened will be included in the image captured by camera 213. This allows food items being taken in or out of refrigerator 1 to be captured with high accuracy.

[0278] After the photographing control unit 223 causes the camera 213 to start photographing, the recording control unit 224 starts recording the photographing results of the camera 213 when the door 11C or the drawer 16A of the refrigerator 1 is opened.

[0279] This allows recording to start at a time when there is a high possibility that food will start being taken in or out of refrigerator 1, thereby preventing unnecessary recording when no food is being taken in or out of refrigerator 1. Furthermore, recording control unit 224 can prevent the data volume of recording file 231 from increasing, thereby preventing an increase in the number of processing targets of food detection unit 225 and an increase in the processing time and processing load of food detection unit 225.

[0280] After starting recording, if the door 11C or the drawer 16A of the refrigerator 1 that was open is closed, the recording control unit 224 stops recording.

[0281] This stops recording when there is a high possibility that food has been taken in or out, thereby preventing unnecessary recording when no food is being taken in or out of refrigerator 1. Furthermore, recording control unit 224 can prevent the data volume of recording file 231 from increasing, thereby preventing an increase in the number of processing targets of food detection unit 225 and an increase in the processing time and processing load of food detection unit 225.

[0282] When the recording control unit 224 stops recording, it generates one recording file 231 containing the results of shooting from when the door 11C or the drawer 16A of the refrigerator 1 is opened to when it is closed.

[0283] According to this, the shooting results from when door 11C or drawer 16A of refrigerator 1 is opened to when it is closed are generated as one recording file 231, so it is possible to quickly generate one recording file 231 while preventing an increase in the data capacity of one recording file 231. Furthermore, since it is possible to quickly generate one recording file 231 while preventing an increase in the data capacity of one recording file 231, food detection unit 225 can quickly start processing related to recording file 231 after food is stored in or taken out of refrigerator 1.

[0284] If the human presence sensor 212 detects that no one is present around the refrigerator 1 after the recording control unit 224 has stopped recording, the imaging control unit 223 causes the camera 213 to end imaging.

[0285] According to this, when the motion sensor 212 detects that there is no person around the refrigerator 1, the camera 213 stops taking pictures, which prevents the camera 213 from taking pictures when no food is being taken in or out of the refrigerator 1. This prevents an increase in power consumption and allows accurate pictures to be taken of food being taken in and out of the refrigerator 1.

[0286] The camera unit 2 includes a food detection unit 225 that detects food items entering or leaving the refrigerator 1 based on the recording results of the recording control unit 224.

[0287] According to this, since the food items being taken in and out of refrigerator 1 can be detected based on the photographed images taken with high accuracy, the food items being taken in and out of refrigerator 1 can be detected with high accuracy.

[0288] As explained above, the food management application 311 causes the terminal processor 300 of the terminal device 3 equipped with the touch panel 32 to function as an application execution unit 301 that displays an image RG of the interior of the refrigerator compartment using the touch panel 32 and displays a photographed food image FG in the divided area BA in the displayed image RG of the interior of the refrigerator compartment 11 that corresponds to the storage area of ​​the food stored in the refrigerator compartment 11. The terminal device 3 also includes a touch panel 32 and an application execution unit 301 that displays an image RG of the interior of the refrigerator compartment using the touch panel 32 and displays a photographed food image FG in a divided area BA in the displayed image RG of the interior of the refrigerator compartment that corresponds to the storage area of ​​the food stored in the refrigerator compartment 11. In addition, the control method of the terminal device 3 displays the refrigerator compartment interior image RG, and displays the photographed food image FG in the divided area BA in the displayed refrigerator compartment interior image RG that corresponds to the storage area of ​​the food stored in the refrigerator compartment 11.

[0289] According to the food management application 311, the terminal device 3, and the control method for the terminal device 3, the photographed food image FG is displayed in the divided area BA corresponding to the storage area of ​​the food stored in the refrigerator in the image showing the inside of the refrigerator compartment 11. This allows the user P to intuitively grasp what kind of food is stored in the refrigerator compartment 11.

[0290] The photographed food image FG is an image taken by the refrigerator compartment camera 203.

[0291] This allows user P to be provided with photographed images of the food actually stored in refrigerator 1, allowing user P to more intuitively understand what kind of food is stored in refrigeration compartment 11.

[0292] The refrigerator compartment interior image RG is an image according to the model of the refrigerator 1 used by the user P of the terminal device 3.

[0293] This allows the user P to be provided with a refrigerator compartment interior image RG that corresponds to the refrigerator compartment 11 of the refrigerator 1 that is actually being used, allowing the user P to more intuitively understand what kind of food is stored in the refrigerator compartment 11.

[0294] The refrigerator compartment interior image RG is divided into a plurality of divided areas BA. The number of divided areas BA included in the refrigerator compartment interior image RG corresponds to the model of the refrigerator 1 used by the user P of the terminal device 3.

[0295] This allows the user P to know what kind of food is stored in the storage area of ​​the refrigeration compartment 11 of the refrigerator 1 that he or she is actually using. Therefore, the user P can more intuitively know what kind of food is stored in the refrigeration compartment 11.

[0296] When displaying a plurality of photographed food images FG in one divided area BA, the application execution unit 301 displays the plurality of photographed food images FG side by side.

[0297] This allows a plurality of photographed food images FG to be displayed in an easy-to-view manner for each divided area BA, allowing the user P to intuitively and easily grasp what kind of food is stored in the refrigerator compartment 11.

[0298] In the divided area BA, a maximum number of photographed food images FG that can be displayed simultaneously is set.

[0299] This prevents a decrease in visibility when there are many photographed food images FG displayed in one divided area BA, and therefore allows the user P to intuitively and easily grasp what kind of food is stored in the refrigerator compartment 11.

[0300] When the number of photographed food images FG to be displayed in one divided area BA exceeds the upper limit, the application execution unit 301 displays photographed food images FG in a switchable manner within the range of the upper limit.

[0301] This prevents a decrease in visibility when there are many photographed food images FG displayed in one divided area BA, while allowing the user P to view the photographed food images FG that cannot be displayed.

[0302] The application execution unit 301 accepts memo input from the user P of the terminal device 3 for each photographed food image FG, and displays a memo input icon MIC indicating that a memo has been accepted in association with the photographed food image FG.

[0303] This allows user P to know for which food the memo has been input. Therefore, user P can intuitively know what kind of food is stored in refrigerator compartment 11, and can also know for which food the memo has been input.

[0304] The food management application 311 is an application program that can be installed on the terminal device 3.

[0305] This allows a terminal device 3 that does not have the function of displaying refrigerator compartment interior images RG and photographed food images FG to become a terminal device 3 that can display these images by installing the food management application 311.

[0306] As described above, the food management application 311 causes the terminal processor 300 of the terminal device 3 equipped with the touch panel 32 to function as the application execution unit 301 that displays images captured by the drawer camera 204 that captures images of the drawer 16A of the refrigerator 1 having an upper case JCA and a lower case GCA. The application execution unit 301 is capable of displaying an upper case image JCG and a lower case image GCG, and accepts memo input from the user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG. When the application execution unit 301 accepts memo input from the user P for the upper case image JCG, it displays a memo input icon MIC indicating that the memo input has been accepted, in association with the upper case image JCG. When the application execution unit 301 accepts memo input from the user P for the lower case image GCG, it displays the memo input icon MIC in association with the lower case image GCG.

[0307] The terminal device 3 also includes a touch panel 32 and an application execution unit 301 that displays images captured by the drawer camera 204. The application execution unit 301 is capable of displaying an upper case image JCG and a lower case image GCG, and accepts memo input from a user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG. When the application execution unit 301 accepts memo input from the user P for the upper case image JCG, it displays a memo input icon MIC indicating that the memo input has been accepted, in association with the upper case image JCG, and when the application execution unit 301 accepts memo input from the user P for the lower case image GCG, it displays the memo input icon MIC in association with the lower case image GCG.

[0308] In addition, the control method of the terminal device 3 is capable of displaying an upper case image JCG and a lower case image GCG, and accepts memo input from a user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG, and when memo input from the user P for the upper case image JCG is accepted, a memo input icon MIC indicating that memo input has been accepted is displayed in association with the upper case image JCG, and when memo input from the user P for the lower case image GCG is accepted, the memo input icon MIC is displayed in association with the lower case image GCG.

[0309] According to this, the food management app 311, the terminal device 3, and the control method of the terminal device 3 display a photographed image of the case CA in the drawer 16A, and the user P can input a memo into this displayed photographed image. This allows the user P to intuitively grasp the food stored in the drawer 16A of the refrigerator 1, and improves the convenience for the user P.

[0310] The application execution unit 301 displays the upper case image JCG superimposed on the lower case image GCG, and when an operation to move the upper case image JCG is received from the user P, the application execution unit 301 moves the first case image and displays the lower case image GCG.

[0311] This allows the lower case image GCG to be displayed in a manner that moves the actual case CA, allowing the user P to more intuitively grasp the foods stored in the drawer 16A of the refrigerator 1 and improving convenience for the user P.

[0312] When the memo input icon MIC is displayed in association with the upper case image JCG, the application executing unit 301 moves the memo input icon MIC together with the upper case image JCG.

[0313] This allows the association between the upper case image JCG and the memo input icon MIC to be maintained even when the user P moves the upper case image JCG. Therefore, even when the user P moves the upper case image JCG, the user P can properly understand which image the memo input icon MIC relates to, improving convenience for the user P.

[0314] The application execution unit 301 displays a switch button for switching the captured image to be displayed between the upper case image JCG and the lower case image GCG, and switches the captured image to be displayed between the upper case image JCG and the lower case image GCG in response to an operation on the switch button.

[0315] This allows the user P to easily switch the case image CG to be displayed by operating the switch button, thereby improving convenience for the user P.

[0316] The upper case image JCG is an image captured when the upper case JCA is pulled out by more than a predetermined threshold relative to the refrigerator 1. The lower case image GCG is an image captured when the lower case GCA is pulled out by more than a predetermined threshold relative to the refrigerator 1.

[0317] This allows the case image CG to be a photographed image that shows as much food as possible, allowing the user P to intuitively understand the food stored in the drawer 16A of the refrigerator 1 and improving convenience for the user P.

[0318] The food management application 311 is an application program that can be installed on the terminal device 3.

[0319] According to this, a terminal device 3 that does not have the function of displaying a case image CG or the function of accepting input of notes for the displayed case image CG can be made into a terminal device 3 that has these functions by installing the food management app 311.

[0320] (Embodiment 2) Next, a description will be given of embodiment 2. In the description of embodiment 2, the same components as those of each part of food control system 1000 of embodiment 1 will be given the same reference numerals, and detailed description will be omitted as appropriate.

[0321] [2-1.Configuration] FIG. 26 is a block diagram showing the configuration of the terminal device 3 and the food management server 4 in the second embodiment. As is clear from a comparison between Figures 26 and 7, the server processor 400 in embodiment 2 functions as a food recognition unit 403 in addition to a server communication control unit 401 and an information processing unit 402 by executing a control program stored in the server memory unit 410.

[0322] The food recognition unit 403 performs food recognition processing based on the photographed food image FG indicated by the food image data 4116 included in each of the first update information and the second update information.

[0323] The food recognition process is a process of recognizing food from the photographed food image FG and retrieving an icon representing the same type of food as the recognized food from an icon database. This icon database stores an icon representing each food. Hereinafter, this icon will be referred to as a food icon. The icon database may be stored in the food management server 4 or in an external device with which the food management server 4 can communicate. In this embodiment, the food icon corresponds to an example of a food image of the present disclosure.

[0324] For example, the food recognition unit 403 recognizes food as follows. The food recognition unit 403 identifies the image that most closely matches the photographed food image FG indicated by the food image data 4116 included in the update information from the images stored in the food image database. The food image database is a database that stores image data for each food item. The degree of match is determined based on feature quantities such as shape and color. The food recognition unit 403 then recognizes that the food corresponding to the identified image is the food indicated by the food image data 4116 included in the update information.

[0325] The food recognition unit 403 may recognize food using AI (artificial intelligence). For example, a machine learning process is performed to learn feature quantities such as food color and shape from a training food image FG, and a trained model is constructed. This trained model is stored in the server storage unit 410. The food recognition unit 403 uses food image data 4116 included in the update information as input data and recognizes the food indicated by the photographed food image FG by referring to the trained model. Here, this trained model may be stored in a device other than the food management server 4 connected to the global network GN, and the food may be recognized by the food management server 4. In this configuration, the food recognition unit 403 transmits the food image data 4116 included in the update information to a device that stores the trained model and obtains the food identification result from this device. The food recognition unit 403 then recognizes the food indicated by the obtained identification result as the food indicated by the photographed food image FG indicated by the food image data 4116 included in the update information.

[0326] When the food recognition unit 403 recognizes a food, it acquires a food icon for the recognized food from the icon database. In the second embodiment, the icon data of the food icon acquired by the food recognition unit 403 is appropriately stored instead of or in addition to the food image data 4116 contained in the refrigerator food management record RR. In the second embodiment, the icon data of the food icon acquired by the food recognition unit 403 is appropriately stored instead of or in addition to the food image data 4116 contained in the drawer food management record HR.

[0327] As a result, in the second embodiment, the app UI 320 can display a food icon instead of the photographed food image FG.

[0328] [2-2. Effects, etc.] According to the second embodiment, the same effects as those of the first embodiment can be achieved.

[0329] (Other embodiments) As described above, the above-described embodiments have been described as examples disclosed in the present application. However, the technology in the present disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above-described embodiments to create new embodiments. Therefore, other embodiments will be exemplified below.

[0330] In the above-described embodiment, a refrigerator 1 is exemplified as a storage facility of the present disclosure. However, the storage facility of the present disclosure is not limited to the refrigerator 1 and may be, for example, a warehouse installed in an office or the like. In the above-described embodiment, food is exemplified as an item of the present disclosure, but the item of the present disclosure is not limited to food and may be other items such as books depending on the storage facility.

[0331] In the above-described embodiment, a configuration has been exemplified in which the position restriction member 205 has the mark 207 in the center in the left-right direction of the main body 201. However, the position of the mark 207 is not limited to the center in the left-right direction of the main body 201. The position of the mark 207 may be other than the center in the left-right direction of the main body 201. In this case, each mark is appropriately recorded on the center position recording paper CY so that the camera unit 2, whose position is adjusted based on the mark 207, can appropriately perform various detections.

[0332] In the above-described embodiment, the refrigerator compartment interior image RG, which is an image of the refrigerator compartment 11 viewed from the front when the left door 11A and the right door 11B are open, has been exemplified as the refrigerator compartment interior image of the present disclosure. However, the refrigerator compartment interior image of the present disclosure is not limited to this refrigerator compartment interior image RG, and may be an image of the refrigerator compartment viewed from the front when a single swinging door is open, or an image of the refrigerator compartment 1 including a refrigerator compartment and a freezer compartment when the doors are open.

[0333] The imaging range of the drawer camera 204 may be a range that combines one or more arbitrary drawer-type storage rooms, depending on the type of the drawer-type storage room on the screen displayed by the application UI 320.

[0334] The food control system 1000 may manage other elements such as remaining quantity and expiration date in addition to the storage date and time.

[0335] The main body 201 and the photographing member 202 of the camera unit 2 may be configured as separate bodies.

[0336] The camera unit 2 may be configured to transmit the case image CG to the food management server 4 even when the amount of food pulled out is not equal to or greater than the predetermined threshold. In this configuration, the food management server 4 stores the case image CG when the amount of food pulled out is not equal to or greater than the predetermined threshold, and provides it to the terminal device 3. The application execution unit 301 then displays this case image CG on the application UI 320 so that it can be substituted for each of the upper case image JCG and the lower case image GCG.

[0337] In the above embodiment, the case where the user P installs the camera unit 2 in the refrigerator 1 has been exemplified, but the camera unit 2 may be shipped in a state where it is installed in the refrigerator 1.

[0338] The server processor 400 may function as the food detection unit 225, and the camera unit processor 220 may not function as the food detection unit 225. In this configuration, the camera unit 2 transmits the recording file 231 to the food management server 4.

[0339] In the above-described embodiment, the human presence sensor 212 is exemplified as a sensor of the present disclosure. However, the sensor of the present disclosure is not limited to the human presence sensor 212, and may be an illuminance sensor or an electrostatic sensor provided on the handle of the refrigerator 1. In the case of this electrostatic sensor, the refrigerator 1 and the camera unit 2 can communicate with each other, and the refrigerator 1 transmits the detection value of the electrostatic sensor to the camera unit 2.

[0340] Alternatively, the terminal device 3 may store the refrigerator food management record RR and the drawer food management record HR, and the food management app 311 may manage various information related to food detected by the camera unit 2. In this configuration, the app execution unit 301 updates the contents of each record stored in the terminal device 3 as needed, similar to the food management server 4 described above. This simplifies the system configuration, eliminating the need for the food management system 1000 to include the food management server 4. In addition, in this configuration, the app execution unit 301 may function as the food recognition unit 403.

[0341] For example, the types of compartments formed in the main box 10 of the refrigerator 1 are not limited to the refrigeration compartment 11, ice-making compartment 12, fresh freezing compartment 13, freezer compartment 14, and vegetable compartment 15, and there may be fewer or more types of compartments. Also, the number of doors provided at the opening on the front of the refrigeration compartment 11 may be one.

[0342] For example, the number of cases provided in drawers 14A and 15A is not limited to two and may be greater than two. In this case, one case corresponds to the first case of the present disclosure, and the case below this case corresponds to the second case of the present disclosure.

[0343] For example, the camera unit processor 220, the terminal processor 300, and the server processor 400 may be configured by a single processor, or may be configured by multiple processors.

[0344] The components shown in Figures 6, 7, and 26 are merely examples, and the specific implementation form is not particularly limited. In other words, it is not necessary to implement hardware corresponding to each component individually; it is of course possible to configure the components so that the functions of each component are realized by a single processor executing a program. Furthermore, some of the functions realized by software in the above-described embodiments may be implemented by hardware, or some of the functions realized by hardware may be implemented by software. In addition, the specific detailed configurations of the camera unit 2, terminal device 3, and other components of the food management server 4 may also be changed as desired without departing from the spirit of the present disclosure.

[0345] Furthermore, for example, the step units of operation shown in Figures 13, 14, 22, and 25 are divided according to the main processing content in order to make it easier to understand the operation of each part of the food management system 1000, and the present disclosure is not limited by the way in which the processing units are divided or their names.

[0346] It should be noted that the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents. [Industrial Applicability]

[0347] As described above, the imaging device and the control method for the imaging device according to the present disclosure can be used to photograph items entering and leaving a storage facility. [Explanation of symbols]

[0348] 1 Refrigerator (storage) 2. Camera unit (photography device) 3 Terminal Devices 4. Food Management Server 10 Main box 10A Front edge 10B Top surface 11 Refrigerator 11A Left door 11B Right door 11C Door 12 Ice Room (Drawer-Type Storage Room) 12A, 13A, 14A, 15A, 16A drawer 13 Fresh Freezing Room (Drawer-Type Storage Room) 14 Freezer (Drawer-Type Storage Room) 15 Vegetable compartment (drawer-type storage compartment) 32 Touch panel (display) 201 Main Unit 202 Photography materials 203 Refrigerator Camera 204 Drawer Camera 205 Position control member 206 bottom 207 marks 212 Human Sensor (Sensor) 213 Camera 223 Imaging control unit 224 Recording control unit 225 Food detection unit (item detection unit) 231 Recording files 300 terminal processors (computers) 301 Application execution unit (display control unit) 311 Food management app (program, application program) 1000 Food Control Systems BA Division Area (Area) FG, FG1 Food Image (Food Image) GCA lower case (second case) GCG lower case image (second case image) JCA upper case (first case) JCG upper case image (first case image) MIC memo input icon (memo input image) P user RG refrigerator compartment interior image (interior image)

Claims

1. A photographing device separate from a refrigerator having a door and a drawer, a main body installed on the top surface of the refrigerator; a photographing member extending from the main body to the front of the refrigerator and including a camera for photographing an area from above the front of the refrigerator to below; a drawer open / close detection means located below the photographing member and forward of the door in a closed state and the drawer in a closed state, for detecting whether the drawer is open or closed; the main body includes a position restricting member that abuts against a front edge of the refrigerator and restricts the position of the photographing member in a front-rear direction of the refrigerator, the position restriction member is a position restriction piece that extends downward from the bottom surface of the main body in an installed state, and is formed to be shorter than a length to a packing provided between the refrigerator and the door, the camera is located in front of the door in a closed state when installed, and captures an image of an area including the drawer in an open state when the drawer open / close detection means detects whether the drawer is open or closed; Filming equipment.

2. The position restriction member extends in the width direction of the main body. The imaging device according to claim 1 .

3. the position restriction member has a mark for adjusting the position of the imaging device in the width direction of the refrigerator, The imaging device according to claim 2 .

4. the position restriction member has the mark at the center in the width direction of the main body; The imaging device according to claim 3 .

Images