Imaging device

Abstract

To provide a food detection device 2 that can accurately detect food taken into and out of a refrigerator 1, by cameras.SOLUTION: The food detection device 2 is attached to an upper part of the refrigerator 1 by a user P, detects the food taken into and out of the refrigerator 1, by a refrigerating chamber camera 21 and a drawer camera 22, and comprises a camera module 220 comprising the refrigerating chamber camera 21 and the drawer camera 22 for photographing the refrigerator 1 from an upper front part of the refrigerator 1, and a power supply unit 230 for supplying power to at least the camera module 220.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a food detection device. [Background technology]

[0002] Patent Document 1 discloses a system that allows a user to check the interior of a refrigerator even from a remote location. The system captures images of the interior of the refrigerator using a camera unit installed in the refrigerator door pocket and transmits the captured images to the user's communication terminal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-111026 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a food detection device that can accurately detect food items entering and leaving a refrigerator using a camera. [Means for solving the problem]

[0005] In this disclosure photograph The device is attached by the user to the top of the refrigerator. Shooting A device for photographing the refrigerator from above the front of the refrigerator. mosquito a camera module having a camera; a sensor module having sensors related to the refrigerator; Camera module and the sensor module a power supply unit for supplying power to the A power cable of the photographing device is connected to a commercial AC power source outside the refrigerator, and the power supply unit supplies power to the camera module and the sensor module based on power supplied from the commercial AC power source via the power cable. .

[0006] The photographing device according to the present disclosure is an photographing device that is installed on top of a refrigerator, and includes a camera module having a camera that photographs the refrigerator from above the front of the refrigerator, a sensor module having sensors related to the refrigerator, and a power supply unit that supplies power to the camera module and the sensor module, and a power cable for the photographing device. but The power supply unit is connected to a commercial AC power source outside the refrigerator and supplies power to the camera module and the sensor module based on power supplied from the commercial AC power source via the power cable. [Effects of the Invention]

[0009] The food detection device disclosed herein can photograph food items being taken in and out of the refrigerator from above, so that the food items being taken in and out of the refrigerator can be accurately included in the photographed results. Therefore, the food items being taken in and out of the refrigerator can be accurately detected by the camera. [Brief explanation of the drawings]

[0010] [Figure 1] Diagram showing the structure of the food control system [Figure 2] Front view of refrigerator and food detection device [Figure 3] Right side view of the refrigerator and food detection device [Figure 4] Block diagram showing the configuration of a food detection device [Figure 5] Block diagram showing the configuration of the terminal device and food management server [Figure 6] An example of a food management database [Figure 7] A diagram showing an example of an installation screen [Figure 8] Flowchart showing the operation of the food detection device [Figure 9] Flowchart showing the operation of the food detection device and the food management server [Figure 10] A diagram showing an example of a food management screen [Figure 11]Flowchart showing the operation of the terminal device and the food management server DETAILED DESCRIPTION OF THE INVENTION

[0011] (Findings that formed the basis of this disclosure) The conventional technology described above is configured to take images from the front of the refrigerator compartment toward the back, so even if other foods are present behind a certain food, the captured image may not include the other foods. Therefore, it is difficult to accurately detect foods being put in and taken out of the refrigerator from the results of images taken by a camera that takes images like the conventional technology described above. Therefore, the present disclosure provides a food detection device that can accurately detect food items being put into and taken out of a refrigerator using a camera.

[0012] Hereinafter, embodiments will be described in detail with reference to the drawings. However, in some cases, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or redundant explanation of substantially the same configuration may be omitted. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0013] (Embodiment 1) First, the first embodiment will be described. [1-1.Configuration] FIG. 1 is a diagram showing the configuration of a food management system 1000.

[0014] The food management system 1000 is a system that manages the remaining amount of food stored in the refrigerator 1, the storage start date, the best-before date or expiration date, and the like.

[0015] Food management system 1000 includes a refrigerator 1. Refrigerator 1 includes a main box 10 with an open front. Main box 10 is formed with a refrigerator compartment 11, an ice-making compartment 12, a fresh-freezing compartment 13, a freezer compartment 14, and a vegetable compartment 15. The opening at the front of refrigerator compartment 11 is provided with a rotating left door 11A and a right door 11B. Ice-making compartment 12, fresh-freezing compartment 13, freezer compartment 14, and vegetable compartment 15 are each provided with drawers 12A, 13A, 14A, and 15A for storing food. In the following description, when there is no need to distinguish between the left door 11A and the right door 11B, they will be referred to as "doors" and collectively designated by the symbol "11C." Also, in the following description, when there is no need to distinguish between the drawers 12A, 13A, 14A, and 15A, they will be referred to as "drawers" and collectively designated by the symbol "16A."

[0016] Food management system 1000 includes food detection device 2. Food detection device 2 is a device that detects food entering and leaving refrigerator 1, and is provided on the top surface of refrigerator 1 as an upper portion of refrigerator 1.

[0017] The configuration of the food detection device 2 will now be described with reference to FIGS. Fig. 2 is a view of the refrigerator 1 and the food detection device 2 as seen from the front side. The refrigerator 1 shown in Fig. 2 has its left door 11A and right door 11B open. Fig. 3 is a view of the refrigerator 1 and the food detection device 2 as seen from the right side. The refrigerator 1 shown in Fig. 3 has its drawer 15A in an open state, i.e., pulled out.

[0018] 2 and 3 illustrate the X-axis, Y-axis, and Z-axis. The X-axis, Y-axis, and Z-axis are perpendicular to each other. The Z-axis indicates the up-down direction. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the left-right direction. The Y-axis indicates the front-back direction. The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the forward direction. The positive direction of the Z-axis indicates the upward direction.

[0019] Food detection device 2 has refrigerator compartment camera 21 and drawer camera 22. Food detection device 2 is installed on the top surface of refrigerator 1 so that refrigerator compartment camera 21 and drawer camera 22 photograph the front surface of refrigerator 1 from above and in front of refrigerator 1.

[0020] The refrigerator compartment camera 21 is a camera for detecting food entering and leaving the refrigerator compartment 11.

[0021] The photographing range of the refrigerating compartment camera 21 is set to a range that allows photographing of food entering and leaving the refrigerating compartment 11 from above the front of the refrigerator 1.

[0022] For example, the photographing range of refrigerator compartment camera 21 includes range A1 shown in Fig. 2 when viewed from the front. Range A1 is a range that includes door pockets 11A1, 11A2, 11A3, and 11A4 provided in left door 11A, door pockets 11B1, 11B2, 11B3, and 11B4 provided in right door 11B, and the opening of refrigerator compartment 11 when left door 11A and right door 11B are in an open position. Furthermore, the photographing range of refrigerator compartment camera 21 includes range A3 shown in Fig. 3 when viewed from the side. Range A3 is a range that includes, in the front-to-back direction, from door 11C in an open position to the front ends of shelves 111, 112, 113, and 114, and also includes the opening of refrigerator compartment 11 in the up-down direction. Shelves 111, 112, 113, and 114 are components provided within refrigerator compartment 11 that divide refrigerator compartment 11 into multiple areas.

[0023] Drawer camera 22 is a camera for detecting food entering or leaving ice making compartment 12, fresh freezing compartment 13, freezer compartment 14, and vegetable compartment 15.

[0024] The imaging range of drawer camera 22 is set to a range that allows imaging of food entering and leaving each of ice making compartment 12, fresh freezing compartment 13, freezer compartment 14, and vegetable compartment 15 from above the front of refrigerator 1.

[0025] For example, the imaging range of drawer camera 22 includes range A2 shown in Fig. 2 when viewed from the front. Range A2 is a range that includes drawers 12A, 13A, 14A, and 15A when viewed from the front of refrigerator 1. Furthermore, the imaging range of drawer camera 22 includes range A4 shown in Fig. 3 when viewed from the side. Range A4 is a range that includes the top surfaces of drawers 12A, 13A, 14A, and 15A when they are pulled out in the front-to-rear direction.

[0026] The food detection device 2 has a left door distance measuring sensor 23, a right door distance measuring sensor 24, and a drawer distance measuring sensor 25. The left door distance measuring sensor 23, the right door distance measuring sensor 24, and the drawer distance measuring sensor 25 correspond to examples of opening and closing sensors. In the following description, when the left door distance measuring sensor 23, the right door distance measuring sensor 24, and the drawer distance measuring sensor 25 are not distinguished, they will be referred to as the "opening and closing distance measuring sensor" and assigned the reference number "28." The opening and closing distance measuring sensor 28 outputs its detection value to the sensor module control unit 211, which will be described later.

[0027] The left door distance measuring sensor 23 is a distance measuring sensor for detecting the open / closed state of the left door 11A. The left door distance measuring sensor 23 detects the distance to the left door 11A as the open / closed state of the left door 11A.

[0028] The right door distance measuring sensor 24 is a distance measuring sensor for detecting the open / closed state of the right door 11B. The right door distance measuring sensor 24 detects the distance to the right door 11B as the open / closed state of the right door 11B.

[0029] Drawer distance measuring sensor 25 is a distance measuring sensor for detecting the open / closed state of each of drawers 12A, 13A, 14A, and 15A. The open state of drawers 12A, 13A, 14A, and 15A means that they are pulled out of the corresponding storage chambers. The closed state of drawers 12A, 13A, 14A, and 15A means that they are stored in the corresponding storage chambers. Drawer distance measuring sensor 25 detects the distance between each of drawers 12A, 13A, 14A, and 15A as the open / closed state of each of drawers 12A, 13A, 14A, and 15A.

[0030] The food detection device 2 has an illuminance sensor 26 and a human presence sensor 27. The illuminance sensor 26 and the human presence sensor 27 output detection values ​​to the sensor module control unit 211.

[0031] The illuminance sensor 26 detects the illuminance of the location where the food detection device 2 is placed, that is, the location where the refrigerator 1 is placed.

[0032] The human presence sensor 27 detects people present around the refrigerator 1. In this embodiment, the human presence sensor 27 is provided on the front surface of the food detection device 2, and detects people present in front of the refrigerator 1 around the refrigerator 1.

[0033] 2, drawer distance measuring sensor 25, drawer camera 22, and refrigerator compartment camera 21 are shown as being arranged in this order in the left-right direction. However, this is an illustration for the convenience of explaining food detection device 2, and they do not necessarily have to be arranged as shown in FIG. 3, drawer distance measuring sensor 25, drawer camera 22, and refrigerator compartment camera 21 are shown as being arranged in this order in the front-to-back direction. However, this is an illustration for food detection device 2, and they do not necessarily have to be arranged as shown in FIG.

[0034] Returning to the explanation of Figure 1, food management system 1000 includes terminal device 3 with touch panel 32. Terminal device 3 is configured, for example, as a smartphone or tablet terminal. An application program for managing food stored in refrigerator 1 is installed on terminal device 3, and the terminal device communicates with food management server 4 using the application's functions. In the following description, the application program for managing food stored in refrigerator 1 is referred to as a "food management application" and is assigned the reference number "323."

[0035] In FIG. 1, a user P who is at home is indicated by a solid line, and a user P who has left home H is indicated by a dotted line. When used by user P who is at home, terminal device 3 communicates with food management server 4 connected to global network GN via communication device 5 or without communication device 5, using the function of food management app 323. When used by user P who has left home H, terminal device 3 communicates with food management server 4 without via communication device 5, using the function of food management app 323.

[0036] The communication device 5 connects to a global network GN, which includes the Internet, a telephone network, and other communication networks, and communicates with the food management server 4 connected to the global network GN. The communication device 5 is an interface device for connecting the terminal device 3 to the global network GN. The communication device 5 has functions such as a modem function, a router function, and a NAT (Network Address Translation) function. The communication device 5 transfers data sent and received between the terminal device 3 and the food management server 4 connected to the global network GN.

[0037] The food management system 1000 includes a food management server 4. The food management server 4 is a server device that manages the food stored in the refrigerator 1 and is connected to the global network GN. Note that in each figure, the food management server 4 is represented by a single block, but this does not necessarily mean that the food management server 4 is composed of a single server device. For example, the food management server 4 may be composed of multiple server devices with different processing contents.

[0038] Next, the configurations of the food detection device 2, the terminal device 3, and the food management server 4 will be described.

[0039] FIG. 4 is a block diagram showing the configuration of the food detection device 2. The food detection device 2 includes a sensor module 210, a camera module 220, and a power supply 230.

[0040] The sensor module 210 includes a sensor module control unit 211 , a sensor unit 212 , and a sensor-side inter-module communication unit 213 .

[0041] The sensor module control unit 211 includes a sensor processor 2111, which is a processor that executes programs such as a CPU or MPU, and a sensor storage unit 2112, and controls each unit of the sensor module 210. The sensor module control unit 211 executes various processes through cooperation between hardware and software, such that the sensor processor 2111 reads out a control program 2113 stored in the sensor storage unit 2112 and executes the process.

[0042] The sensor storage unit 2112 has a storage area for storing programs executed by the sensor processor 2111 and data processed by the sensor processor 2111. The sensor storage unit 2112 stores a control program 2113 executed by the sensor processor 2111, setting data 2114 related to the settings of the sensor module 210, and various other data. The sensor storage unit 2112 has a non-volatile storage area. The sensor storage unit 2112 may also have a volatile storage area and constitute a work area for the sensor processor 2111.

[0043] The sensor unit 212 includes a left door distance measuring sensor 23, a right door distance measuring sensor 24, a drawer distance measuring sensor 25, an illuminance sensor 26, and a human presence sensor 27, and outputs the detection value of each sensor to the sensor module control unit 211. Note that the sensor unit 212 may include sensors other than the above five types of sensors.

[0044] The sensor-side inter-module communication unit 213 includes communication hardware such as a communication circuit and connectors that comply with a predetermined communication standard, and communicates with the camera module 220 under the control of the sensor module control unit 211 .

[0045] The camera module 220 includes a camera module control unit 221 , a photographing unit 222 , a camera module communication unit 223 , and a camera side inter-module communication unit 224 .

[0046] The camera module 220 includes a camera module control unit 221, a camera processor 2211 which is a processor that executes programs such as a CPU or MPU, and a camera storage unit 2212, and controls each unit of the camera module 220. The camera module control unit 221 executes various processes through cooperation of hardware and software, such that the camera processor 2211 reads out a control program 2213 stored in the camera storage unit 2212 and executes the process. By reading out and executing the control program 2213, the camera module control unit 221 functions as a detection value processing unit 221A, an imaging control unit 221B, a video processing unit 221C, and a food detection unit 221D.

[0047] The camera memory unit 2212 has a memory area for storing programs executed by the camera processor 2211 and data processed by the camera processor 2211. The camera memory unit 2212 stores a control program 2213 executed by the camera processor 2211, setting data 2214 related to settings of the camera module 220, a food detection device ID 2215, and various other data. The food detection device ID 2215 will be described later. The camera memory unit 2212 has a non-volatile memory area. The camera memory unit 2212 may also have a volatile memory area and constitute a work area for the camera processor 2211.

[0048] The photographing unit 222 includes a refrigerator compartment camera 21 and a drawer camera 22. The photographing unit 222 outputs the photographing results of each of the refrigerator compartment camera 21 and the drawer camera 22 to the camera module control unit 221. In this embodiment, the refrigerator compartment camera 21 and the drawer camera 22 capture moving images. Therefore, the photographing unit 222 outputs moving image data as the photographing results to the camera module control unit 221.

[0049] The camera module communication unit 223 includes communication hardware such as a communication circuit and a connector that conforms to a predetermined communication standard, and communicates with the food management server 4 under the control of the camera module control unit 221. In this embodiment, the communication standard used by the camera module communication unit 223 is exemplified as a wireless communication standard, but it may also be a wired communication standard.

[0050] The camera-side inter-module communication unit 224 includes communication hardware such as a communication circuit and connector that conforms to a predetermined communication standard, and communicates with the sensor module 210 under the control of the camera module control unit 221 .

[0051] As described above, the camera module control unit 221 functions as the detection value processing unit 221A, the photography control unit 221B, the moving image processing unit 221C, and the food detection unit 221D.

[0052] The detection value processing unit 221A determines whether each of the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A is in an open state or a closed state based on detection values ​​of various sensors output from the sensor module 210. For example, the detection value processing unit 221A determines whether each of the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A is in an open state based on whether the detection value is below a predetermined threshold. Note that this predetermined threshold may be different for each of the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A. If the detection value processing unit 221A determines that any of the doors is in an open state, it outputs open / closed state information indicating the open state to the shooting control unit 221B and the video processing unit 221C. In addition, if the detection value processing unit 221A determines that the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A are all in the closed state, it outputs opening / closing state information indicating that they are in the closed state to the shooting control unit 221B and the video processing unit 221C.

[0053] Furthermore, the detection value processing unit 221A determines whether or not a person is present around the refrigerator 1 based on the detection values ​​of the illuminance sensor 26 and the human presence sensor 27 output from the sensor module 210. For example, the detection value processing unit 221A determines whether or not a person is present around the refrigerator 1 based on whether or not the detection value is below a predetermined threshold. If the detection value processing unit 221A determines that a person is present around the refrigerator 1, it outputs person presence / absence information indicating the presence of a person to the photography control unit 221B. If the detection value processing unit 221A determines that no person is present around the refrigerator 1, it outputs person presence / absence information indicating the absence of a person to the photography control unit 221B.

[0054] The photographing control section 221B controls the photographing section 222 to cause the refrigerator compartment camera 21 and the drawer camera 22 to photograph moving images. Details of the photographing control section 221B will be described with reference to FIG.

[0055] The video processing unit 221C performs various processes on the images captured by the refrigerator compartment camera 21 and the drawer camera 22, and stores the processed images in the camera storage unit 2212. Details of the video processing unit 221C will be described with reference to FIG. 8.

[0056] Food detection unit 221D detects food entering or leaving refrigerator 1 based on the photographing results of refrigerator compartment camera 21 and drawer camera 22. Details of food detection unit 221D will be described with reference to FIG.

[0057] Power supply unit 230 includes hardware such as a power supply circuit, and supplies power to each of sensor module 210 and camera module 220. Power supply unit 230 may be configured to supply power in parallel to each of sensor module 210 and camera module 220, as in the present embodiment, or may be configured to supply power to one of the modules via the other.

[0058] The power supply unit 230 of this embodiment supplies power to each of the sensor module 210 and the camera module 220 based on power supplied from an external commercial AC power supply 7. The commercial AC power supply 7 corresponds to an example of a commercial power supply. A power cable of the food detection device 2 is connected to the commercial AC power supply 7, and the power supply unit 230 receives power from the commercial AC power supply 7 via the power cable, and supplies power to each of the sensor module 210 and the camera module 220 based on the supplied power. The power supply unit 230 includes, as hardware, for example, a circuit for converting AC to DC, a circuit for stepping down, etc.

[0059] The source of power supply to the power supply unit 230 is not limited to the external commercial AC power supply 7, but may be the refrigerator 1 or a battery. When the source is the former, the power supply unit 230 and the refrigerator 1 are equipped with hardware that complies with the USB power supply standard. When the source is the former, a USB cable of the food detection device 2 is connected to a USB port of the refrigerator 1, and the refrigerator 1 supplies power to the power supply unit 230 via the USB cable. When the source is the latter, the food detection device 2 is equipped with a battery.

[0060] FIG. 5 is a block diagram showing the configuration of the terminal device 3 and the food management server 4.

[0061] First, the configuration of the food management server 4 will be described.

[0062] The food management server 4 includes a server control unit 40 and a server communication unit 41.

[0063] The server control unit 40 includes a server processor 410, which is a processor that executes programs such as a CPU or MPU, and a server storage unit 420, and controls each unit of the food management server 4. The server control unit 40 executes various processes through cooperation of hardware and software, such that the server processor 410 reads and executes a control program 421 stored in the server storage unit 420. The server control unit 40 functions as a food recognition unit 4100 by reading and executing the control program 421.

[0064] The server storage unit 420 has a storage area for storing programs executed by the server processor 410 and data processed by the server processor 410. The server storage unit 420 stores a control program 421 executed by the server processor 410, setting data 422 related to the settings of the food management server 4, a food management database 423, and various other data. The server storage unit 420 has a non-volatile storage area. The server storage unit 420 may also have a volatile storage area and constitute a work area for the server processor 410.

[0065] FIG. 6 is a diagram showing an example of the food management database 423. The food management database 423 is a database that stores various information related to the food stored in the refrigerator 1.

[0066] Each record R stored in the food management database 423 includes a user ID 4231, a food detection device ID 2215, food name information 4232, remaining quantity information 4233, storage location information 4234, storage start date information 4235, expiration date information 4236, expiration date notification timing information 4237, new notification timing information 4238, and food image data 4239.

[0067] Note that one record R stored in the food management database 423 may further include one or more other types of information.

[0068] The user ID 4231 is identification information for identifying the user P who uses the food management application 323, and is appropriately assigned to the user P who uses the food management application 323.

[0069] The food detection device ID 2215 is identification information for identifying the food detection device 2. The food detection device ID 2215 is, for example, the manufacturing number of the food detection device 2, the serial number of the food detection device 2, the manufacturing number of the camera module 220, the serial number of the camera module 220, etc.

[0070] The food name information 4232 is information indicating the name of a food. In one record R, one or more pieces of food name information 4232 are associated with a user ID 4231.

[0071] The remaining amount information 4233 is information indicating the remaining amount of food. One record R is associated with the same number of remaining amount information 4233 as the number of food name information 4232 associated with the user ID 4231. The remaining amount of food indicated by the remaining amount information 4233 in FIG. 6 is shown in units corresponding to the name of the food. The remaining amount indicated by the remaining amount information 4233 is updated appropriately by the server control unit 40.

[0072] The storage location information 4234 is information indicating the storage location of the food. In this embodiment, the storage location information 4234 indicates one of the storage compartments of the refrigerator 1 as the storage location. One record R is associated with the same number of storage location information 4234 as the number of food name information 4232 associated with the user ID 4231.

[0073] The storage start date information 4235 is information indicating the storage start date of the food. In one record R, the same number of pieces of storage start date information 4235 as the number of pieces of food name information 4232 associated with the user ID 4231 are associated.

[0074] The expiration date information 4236 is information indicating the expiration date of the food product, such as the best-before date or the expiration date of the food product. In one record R, the same number of expiration date information 4236 as the number of food product name information 4232 associated with the user ID 4231 are associated.

[0075] The expiration date notification timing information 4237 is information indicating the timing at which the expiration date indicated by the corresponding expiration date information 4236 is notified to the user P. As shown in Fig. 6, the timing indicated by the expiration date notification timing information 4237 is the number of days before the expiration date indicated by the expiration date information 4236. In one record R, the same number of pieces of expiration date notification timing information 4237 as the number of pieces of food name information 4232 associated with the user ID 4231 are associated.

[0076] New notification timing information 4238 is information indicating the timing of notifying that new food has been stored in refrigerator 1. As shown in Fig. 6, the timing indicated by new notification timing information 4238 is the number of days from the storage start date indicated by storage start date information 4235.

[0077] The food image data 4239 is image data of a food image FG, which is an image of food. In one record R, the same number of food image data 4239 as the number of food name information 4232 associated with the user ID 4231 are associated.

[0078] 5, the server communication unit 41 includes communication hardware such as communication circuits and connectors that comply with a predetermined communication standard, and communicates with devices connected to the global network GN in accordance with the predetermined communication standard under the control of the server control unit 40. In this embodiment, the server communication unit 41 communicates with the food detection device 2 and the terminal device 3.

[0079] As described above, the server control unit 40 functions as the food recognition unit 4100. The food recognition unit 4100 recognizes the food detected by the food detection device 2. Details of the food recognition unit 4100 will be described with reference to FIG.

[0080] Next, the functional configuration of the terminal device 3 will be described. The terminal device 3 includes a terminal control unit 30, a terminal communication unit 31, and a touch panel 32.

[0081] The terminal control unit 30 includes a terminal processor 310, which is a processor that executes programs such as a CPU or MPU, and a terminal storage unit 320, and controls each unit of the terminal device 3. The terminal control unit 30 executes various processes through cooperation of hardware and software, such that the terminal processor 310 reads out a control program 321 stored in the terminal storage unit 320 and executes the process. A food management application 323 is pre-installed in the terminal device 3. The food management application 323 is a program that is read out from the terminal storage unit 320 by the terminal processor 310 and executed, causing the terminal control unit 30 to function as an application execution unit 3100.

[0082] The terminal storage unit 320 has a storage area for storing programs executed by the terminal processor 310 and data processed by the terminal processor 310. The terminal storage unit 320 stores a control program 321 executed by the terminal processor 310, setting data 322 related to the settings of the terminal device 3, a food management application 323, a user ID 4231, and various other data. The terminal storage unit 320 has a non-volatile storage area. The terminal storage unit 320 may also have a volatile storage area and constitute a work area for the terminal processor 310.

[0083] The terminal communication unit 31 includes communication hardware such as a communication circuit conforming to a predetermined communication standard, and communicates with devices connected to the global network GN according to the predetermined communication standard under the control of the terminal control unit 30. In this embodiment, the terminal communication unit 31 communicates with the food management server 4 according to the predetermined communication standard under the control of the application execution unit 3100. In this embodiment, the communication standard used by the terminal communication unit 31 is exemplified as a wireless communication standard, but may also be a wired communication standard.

[0084] The touch panel 32 includes a display panel such as a liquid crystal display panel and a touch sensor that is overlaid on or integrated with the display panel. The display panel displays various images under the control of the terminal control unit 30. The touch sensor detects touch operations and outputs the detected operations to the terminal control unit 30. The terminal control unit 30 executes processing corresponding to the touch operations based on input from the touch sensor.

[0085] As described above, the terminal control unit 30 functions as the application execution unit 3100. The application execution unit 3100 controls the terminal communication unit 31 to send and receive various information to and from the food management server 4. If a communication connection between the terminal device 3 and the communication device 5 is possible, the application execution unit 3100 sends and receives various information to and from the food management server 4 via the communication device 5; otherwise, the application execution unit 3100 sends and receives various information to and from the food management server 4 without going through the communication device 5. Therefore, even if the user P is not located in or near the home H, the terminal device 3 can send and receive various information to and from the food management server 4 using the functions of the food management application 323.

[0086] The application execution unit 3100 also displays a user interface for managing the food stored in the refrigerator 1 on the touch panel 32. In the following description, this user interface will be referred to as the "food management UI" and will be denoted by the symbol "3200." By displaying the food management UI 3200 on the touch panel 32, the application execution unit 3100 provides the user P with various information related to the food stored in the refrigerator 1 and also receives various inputs related to the food stored in the refrigerator 1 from the user P.

[0087] [1-2. Operation] Next, the operation of each part of the food management system 1000 will be described. In this embodiment, the food detection device 2 is illustrated as being separate from the refrigerator 1. Therefore, the food detection device 2 is attached to the top surface of the refrigerator 1 by the user P.

[0088] First, the operation of each part of the food management system 1000 when the food detection device 2 is attached to the refrigerator 1 will be described.

[0089] When the food detecting device 2 is attached to the refrigerator 1, the application executing unit 3100 causes the touch panel 32 to display the food management UI of the attachment screen TG.

[0090] FIG. 7 is a diagram showing an example of the attachment screen TG. The installation screen TG is provided with a photography result display area HA1. The photography result display area HA1 is a rectangular area that displays the photography results of either the refrigerator compartment camera 21 or the drawer camera 22. The photography result display area HA1 has a line L1 extending in the vertical direction at approximately the center in the horizontal direction in the drawing.

[0091] The installation screen TG includes installation method information J1. The installation method information J1 is information that indicates how to install the food detection device 2 on the top surface of the refrigerator 1. The installation method information J1 shown in Fig. 7 indicates that the refrigerator 1 should be installed so that the center in the left-right direction of the refrigerator 1 shown in the photography result displayed in the photography result display area HA1 is aligned with a line L1 provided in the photography result display area HA1.

[0092] When user P attaches food detection device 2 to the top surface of refrigerator 1 following the installation method indicated in installation method information J1, the shooting range of refrigerator compartment camera 21 and the shooting range of drawer camera 22 are set to the shooting ranges explained with reference to Figures 2 and 3. In other words, food detection device 2 is attached to refrigerator 1 so that food entering and leaving refrigerator 1 can be included in the shooting ranges of refrigerator compartment camera 21 and drawer camera 22, respectively.

[0093] Here, we will explain the operation of each part of the food management system 1000 when the food detection device 2 is attached to the refrigerator 1. In this case, the power supply unit 230 supplies power to at least the camera module 220, and the camera module 220 can transmit the image capture results of either the refrigerator compartment camera 21 or the drawer camera 22 to the terminal device 3. When the food detection device 2 is attached to the refrigerator 1, the camera module 220 and the terminal device 3 may communicate via the food management server 4, or may communicate directly via short-range wireless communication or the like.

[0094] When the application execution unit 3100 of the terminal device 3 receives the instruction to display the attachment screen TG, the application execution unit 3100 transmits request information requesting the photographing result to the food detection device 2 via the terminal communication unit 31. When the camera module control unit 221 of the camera module 220 of the food detection device 2 receives the request information via the camera module communication unit 223, it transmits the photographic results of either the refrigerator compartment camera 21 or the drawer camera 22 to the terminal device 3 via the camera module communication unit 223. Then, the application execution unit 3100 displays the received photographing results in the photographing result display area HA1.

[0095] As described above, the application executing unit 3100 causes the touch panel 32 to display the food management UI 3200 of the installation screen TG, so that the user P can install the food detecting device 2 in the refrigerator 1 simply and appropriately.

[0096] Next, the operation of each part of food management system 1000 related to food management will be described.

[0097] First, the operation of food detection device 2 relating to the photographing by refrigerator compartment camera 21 and drawer camera 22 will be described. FIG. 8 is a flowchart showing the operation of the food detection device 2. 8, the sensor module 210 transmits detection values ​​of the various sensors included in the sensor section 212 to the camera module 220 via the sensor-side inter-module communication section 213, and the camera module 220 receives the detection values ​​via the camera-side inter-module communication section 224. Then, in the operation shown in FIG. 8, the detection value processing section 221A performs processing based on the various detection values ​​received from the sensor module 210.

[0098] The photography control section 221B of the camera module 220 determines whether or not there is a person around the refrigerator 1 (step SA1).

[0099] If the shooting control unit 221B acquires person presence / absence information indicating the presence of a person from the detection value processing unit 221A, it makes a positive judgment in step SA1, and if the shooting control unit 221B acquires person presence / absence information indicating the absence of a person from the detection value processing unit 221A, it makes a negative judgment in step SA1.

[0100] When the photographing control section 221B determines that a person is present around the refrigerator 1 (step SA1: YES), it starts photographing moving images by the refrigerator compartment camera 21 and the drawer camera 22 (step SA3).

[0101] On the other hand, if the photography control unit 221B determines that no person is present around the refrigerator 1 (step SA1: NO), it determines whether any of the left door 11A, right door 11B, and drawers 12A, 13A, 14, and 15A is open (step SA2).

[0102] If the shooting control unit 221B acquires opening / closing state information indicating an open state from the detection value processing unit 221A, it makes a positive judgment in step SA2, and if it acquires opening / closing state information indicating a closed state from the detection value processing unit 221A, it makes a negative judgment in step SA2.

[0103] If the photography control unit 221B determines that none of the left door 11A, the right door 11B, and the drawers 12A, 13A, 14, and 15A are open (step SA2: NO), the photography control unit 221B returns the process to step SA1 and executes the process of step SA1 again.

[0104] If the photography control unit 221B determines that any one of the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A is open (step SA2: YES), it executes the process of step SA3.

[0105] When shooting starts in step SA3, moving image processing section 221C of camera module 220 executes the processes of steps SA4 to SA8 for each frame that constitutes the moving image.

[0106] When the refrigerator compartment camera 21 and the drawer camera 22 start capturing images, the video processing unit 221C acquires the open / closed state information from the detection value processing unit 221A (step SA4).

[0107] Next, the video processing unit 221C reduces the size of each frame of the captured image from each of the refrigerator compartment camera 21 and the drawer camera 22 (step SA5). The reduction here refers to so-called resizing.

[0108] The video processing unit 221C associates the reduced captured image for one frame with the open / close state information acquired in step SA4 and stores them in the camera storage unit 2212 (step SA6). Note that the video processing unit 221C stores the images in chronological order each time step SA6 is performed.

[0109] Next, the photography control section 221B determines whether or not there is a person around the refrigerator 1, similar to step SA1 (step SA7).

[0110] If the photography control unit 221B determines that a person is present around the refrigerator 1 (step SA7: YES), the video processing unit 221C executes the processes from step SA4 onwards again.

[0111] On the other hand, if the photography control unit 221B determines that no person is present around the refrigerator 1 (step SA7: NO), it determines whether the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A are all closed, as in step SA2 (step SA8).

[0112] If the photography control unit 221B determines that the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A are not all closed (step SA8: NO), the moving image processing unit 221C executes the processes from step SA4 onwards again.

[0113] On the other hand, if the photography control unit 221B determines that the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A are all closed (step SA8: YES), photography by the refrigerator compartment camera 21 and the drawer camera 22 is terminated (step SA9).

[0114] Next, the operation related to the management of food based on the photographing results of refrigerator compartment camera 21 and drawer camera 22 will be described. Fig. 9 is a flowchart showing the operations of the food detection device 2 and the food management server 4. In Fig. 9, the flowchart FB shows the operations of the food detection device 2. Also in Fig. 9, the flowchart FC shows the operations of the food management server 4.

[0115] Food detection unit 221D of camera module 220 of food detection device 2 determines whether or not video images captured by refrigerator compartment camera 21 and drawer camera 22 are stored in camera storage unit 2212 (step SB1).

[0116] If the food detection unit 221D determines that the video image is not stored in the camera storage unit 2212 (step SB1: NO), it executes the process of step SB1 again.

[0117] On the other hand, if the food detection unit 221D determines that the video image has been stored in the camera storage unit 2212 (step SB1: YES), it executes a valid frame determination process (step SB2).

[0118] The valid frame determination process determines valid frames from the frames that make up the moving image stored in the camera storage unit 2212, extracts frames determined to be valid frames from the moving image, and discards frames that are not identified as valid frames from the moving image. A valid frame is a frame that shows an image when at least one of the left door 11A, right door 11B, and drawers 12A, 13A, 14A, and 15A is open.

[0119] Here, several methods for determining valid frames in the valid frame determination process will be described. <Method 1> The first method is to make a determination based on open / close state information associated with a frame. In determining valid frames, the food detection unit 221D determines that a frame associated with open / closed state information indicating an open state is a valid frame, and determines that a frame associated with open / closed state information indicating a closed state is not a valid frame.

[0120] <Second Method> The second method is to make a judgment by extracting differences between images. The food detection unit 221D extracts differences between frames constituting the moving image to identify frames that show an image when any of the left door 11A, right door 11B, and drawers 12A, 13A, 14A, and 15A is open. The food detection unit 221D then determines the identified frames as valid frames and determines frames other than the identified frames as ineffective frames. For example, the food detection unit 221D extracts differences between frames in a closed state and determines a frame in which there is a difference in any of the left door 11A, right door 11B, and drawers 12A, 13A, 14A, and 15A as a valid frame. The closed state frame that serves as the basis for difference extraction is, for example, the oldest frame among the frames constituting the moving image.

[0121] <Third Method> The third method is a determination based on the first and second methods. The food detection unit 221D determines a frame associated with open / closed state information indicating the open state as a valid frame. Next, the food detection unit 221D extracts differences from the closed state frame for frames associated with open / closed state information indicating the closed state, and determines a frame with a difference in any of the left door 11A, right door 11B, and drawers 12A, 13A, 14A, and 15A as a valid frame.

[0122] After executing the valid frame determination process, the food detection unit 221D detects food items that have been put into or taken out of the refrigerator 1 from the valid frames (step SB3). In the process of step SB3, if food items are detected from the valid frames, the food type and the storage room to which the detected food items have been put or taken out are also detected. The food detection unit 221D detects the type of food as a broader classification than the food name. For example, if the food that has actually been put into or taken out is milk, the food detection unit 221D detects the type of food as "packaged beverage" in step SB3.

[0123] For example, food detection unit 221D detects food that has been put into or taken out of refrigerator 1 from the effective frame based on feature quantities such as shape and color. Also, for example, food detection unit 221D detects the type of food that has been detected based on feature quantities such as shape and color. Also, for example, food detection unit 221D detects the storage compartment into which the detected food has been put or taken out based on the type of camera and the size of drawer 16A captured in the effective frame. The data required for detection in step SB3 is stored in a storage area that the camera module control unit 221 can acquire.

[0124] The food detection unit 221D determines whether or not food items that have been put into or taken out of the refrigerator 1 can be detected from the valid frames (step SB4).

[0125] If the food detection unit 221D determines that food entering or leaving the refrigerator 1 could not be detected from the valid frames (step SB4: NO), it returns the process to step SB1 and executes the processes from step SB1 onwards again. Note that the food detection unit 221D may discard valid frames between the negative determination in step SB4 and the re-execution of step SB1. This prevents unnecessary videos in which food could not be detected from being continuously stored in the camera storage unit 2212.

[0126] On the other hand, if the food detection unit 221D determines that food has been detected in or out of the refrigerator 1 (step SB4: YES), it performs an in / out determination process (step SB5).

[0127] The in / out determination process is a process for determining whether the food detected in step SB3 is food that has been put into refrigerator 1 or food that has been taken out. For example, in the receiving / retrieving determination process, food detection unit 221D determines the direction of movement in the video of the food image detected in step SB3 based on valid frames. If the direction of movement of the food image is backward, i.e., toward refrigerator 1, food detection unit 221D determines that the detected food is food that has been received from refrigerator 1. Also, if the direction of movement of the food image is forward, i.e., away from refrigerator 1, food detection unit 221D determines that the detected food is food that has been removed from refrigerator 1.

[0128] Next, the food detection unit 221D cuts out a food image FG of the detected food from the valid frame (step SB6). In the following description, the cut-out food image FG will be referred to as a "cut-out food image."

[0129] The food detection unit 221D transmits food management request information, which requests the food management server 4 to manage the food, to the food management server 4 via the camera module communication unit 223 (step SB7).

[0130] The food management request information sent to the food management server 4 includes the food detection device ID 2215 stored in the camera storage unit 2212. The food management request information sent to the food management server 4 also includes date and time information indicating the date and time when the video was captured. In addition, the food management request information sent to the food management server 4 includes, for each food detected in step SB3, image data of the cut-out food image, in-stock / out-stock determination result information indicating the determination result of the in-stock / out-stock determination process, storage location information 4234 indicating the detected in-stock or out-stock destination, and food type information indicating the type of food detected.

[0131] Referring to the flowchart FC, the food recognition unit 4100 of the server control unit 40 of the food management server 4 determines whether or not food management request information has been received via the server communication unit 41 (step SC1).

[0132] If the food recognition unit 4100 determines that the food management request information has not been received (step SC1: NO), it executes the process of step SC1 again.

[0133] On the other hand, if the food recognition unit 4100 determines that food management request information has been received (step SC1: YES), it performs food recognition processing based on the cut-out food image indicated by the image data included in the food management request information (step SC2).

[0134] The food recognition process in this embodiment is a process of recognizing food from a cut-out food image, and acquiring food images FG and food name information 4232 for the recognized food from the food information database.

[0135] The food information database is a database that stores information related to food, and for example, for each food item, stores image data of the food image FG and food name information 4232. The food information database may be stored in the food management server 4, or may be stored in a device with which the food management server 4 can communicate.

[0136] For example, food recognition is performed as follows: The food recognition unit 4100 identifies the food image FG that most closely matches the cut-out food image from the food images FG stored in the food information database. The degree of match is calculated using a predetermined algorithm based on feature quantities such as shape and color. The food recognition unit 4100 then recognizes that the food represented by the identified food image FG is the food represented by the cut-out food image.

[0137] The food recognition unit 4100 may also perform food recognition using AI (artificial intelligence). For example, a machine learning process is performed to learn feature quantities such as food color and shape from training food images FG, and a trained model is constructed. This trained model is stored in the server storage unit 420. The food recognition unit 4100 uses the food image FG included in the food management request information as input data and recognizes the food indicated by the food image FG by referring to the trained model. Here, this trained model may be stored in a device other than the food management server 4 connected to the global network GN, and the food may be recognized in the food management server 4. In this configuration, the food recognition unit 4100 transmits the food image included in the food management request information to the device storing the trained model and obtains the food identification result from this device. The food recognition unit 4100 then recognizes the food indicated by the obtained identification result as the food indicated by the cut-out food image.

[0138] When the food recognition unit 4100 recognizes a food, it acquires a food image FG and food name information 4232 for the recognized food from the food information database.

[0139] Next, the food recognition unit 4100 performs the food recognition process and updates the food management database 423 based on the results of the food recognition process (step SC3).

[0140] Step SC3 will now be described in detail. First, the food recognition unit 4100 identifies, from the food management database 423, a record R having the food detection device ID 2125 included in the food management request information.

[0141] Next, the food recognition unit 4100 determines whether the recognized food is food that has been stored in refrigerator 1 or food that has been removed from refrigerator 1 based on the storage / removal determination result information included in the received food management request information.

[0142] When the food recognition unit 4100 determines that the recognized food is food that has been removed from the refrigerator 1, it updates the information associated with the food name information 4232 of the recognized food in the identified record R. More specifically, the food recognition unit 4100 performs a process to reduce the remaining amount indicated by the remaining amount information 4233. The amount to be reduced is calculated, for example, during the food recognition process, based on the food image FG included in the food management request information. Here, if the updated remaining amount information 4233 indicates zero, the food recognition unit 4100 may blank out the storage start date indicated by the storage start date information 4235 and the expiration date indicated by the expiration date information 4236.

[0143] When the food recognition unit 4100 determines that the recognized food is a food that has been stored in the refrigerator 1, for food that has been stored in the past, it updates the information corresponding to the food name information 4232 of the recognized food in the identified record R. More specifically, the food recognition unit 4100 performs a process to increase the remaining amount indicated by the remaining amount information 4233. The amount to increase is calculated, for example, during the food recognition process, based on a clipped food image included in the food management request information. The food recognition unit 4100 also performs a process to change the date indicated by the date and time information included in the food management request information to the storage start date indicated by the storage start date information 4235. The food recognition unit 4100 also performs a process to acquire an expiration date based on the date indicated by the date and time information included in the food management request information using a predetermined method such as calculation or inquiry, and changes the expiration date indicated by the expiration date information 4236 to the acquired expiration date.

[0144] In addition, if the food recognition unit 4100 determines that the recognized food is food that has been stored in the refrigerator 1, for food that has not been stored in the past, it associates various new information with the user ID 4231 of the identified record R.

[0145] The various information associated with the user ID 4231 of the identified record R includes food name information 4232, remaining quantity information 4233, storage location information 4234, storage start date information 4235, expiration date information 4236, expiration date notification timing information 4237, new notification timing information 4238, and food image data 4239. Here, the food name information 4232 is food name information 4232 acquired during the food recognition process. Furthermore, the remaining amount information 4233 indicates the remaining amount calculated based on the clipped food image included in the food management request information during the food recognition process, for example. The storage location information 4234 is the storage location information 4234 included in the food management request information. The storage start date information 4235 indicates the date indicated by the date and time information included in the food management request information. Furthermore, the expiration date information 4236 indicates the expiration date based on the date indicated by the date and time information included in the food management request information, or indicates blank. Furthermore, deadline notification timing information 4237 and new notification timing information 4238 indicate default timing or are blank. Furthermore, food image data 4239 indicates food image FG acquired during food recognition processing.

[0146] As described above, in the food management system 1000, the food detection device 2 determines valid frames from the video images captured, detects food from the valid frames, and cuts out food images FG of the detected food from the valid frames. Then, in the food management system 1000, the food management server 4 recognizes the food from the cut-out food images. This division of roles allows the food management server 4 to recognize food while minimizing the amount of data sent by the food detection device 2 to the food management server 4. Therefore, in the food management system 1000, it is possible to recognize food that has been put into and taken out of the refrigerator 1 while minimizing an increase in communication volume.

[0147] Next, a description will be given of the operation of the food management system 1000 when the application execution unit 3100 displays the food management screen SKG on the food management UI 3200. The food management screen SKG is a screen related to the management of food stored in the refrigerator 1.

[0148] First, before explaining the operation, the food management screen SKG will be explained. FIG. 10 is a diagram showing an example of the food management screen SKG.

[0149] The food management screen SKG includes an object OB related to each food item. The food management screen SKG shown in Fig. 10 includes an object OB1 related to milk, an object OB2 related to beer, and an object OB3 related to eggs.

[0150] The object OB of one includes a food image FG, remaining quantity information 4233, and storage start date information 4235. The object OB of one also includes expiration date warning information KJ that warns that the expiration date of the food is approaching if the expiration date of the food falls within the timing indicated by expiration date notification timing information 4237. The object OB of one also includes new storage indication information MJ that indicates that the food is newly stored if the storage start date of the food falls within the timing indicated by new notification timing information 4238.

[0151] The object OB1 shown in FIG. 10 includes a food image FG1 showing milk, remaining amount information 4233 showing the remaining amount of milk, storage start date information 4235 showing the storage start date of the milk, and expiration date warning information KJ. Moreover, the object OB2 shown in FIG. 10 includes a food image FG2 showing beer, remaining amount information 4233 showing the remaining amount of beer, storage start date information 4235 showing the storage start date of the beer, and new storage specification information MJ. Moreover, the object OB3 shown in FIG. 10 includes a food image FG3 showing an egg, remaining amount information 4233 showing the remaining amount of eggs, and storage start date information 4235 showing the storage start date of the eggs.

[0152] The food management screen SKG has a storage room switching button B1. The storage room switching button B1 is a software button for switching the type of object OB displayed on the food management screen SKG for each storage room of the refrigerator 1. When the storage room switching button B1 is touched, the food management screen SKG displays a list of the names of the storage rooms of the refrigerator 1. When a storage room is selected from the list by touching, the food management screen SKG displays the object OB for the storage location information 4234 that indicates the selected storage room.

[0153] The food management screen SKG has a delete button B2. The delete button B2 is a software button for deleting an object OB from display on the food management screen SKG. On the food management screen SKG, when one or more objects OB are selected and the delete button B2 is selected by a touch operation, the selected objects OB are hidden. Furthermore, in the food management database 423, various pieces of information related to the food indicated by the object OB that was selected when the delete button B2 was operated are deleted.

[0154] The food management screen SKG has an update button B3 and update date and time information J2 that indicates the most recent update date and time. The update button B3 is a software button for updating the objects OB displayed on the food management screen SKG and the various information held by the objects OB. When the update button B3 is touched, the update date and time indicated in the update date and time information J2 changes to the date and time of the most recent update.

[0155] With reference to FIG. 11, the operation of the food management system 1000 when the application execution unit 3100 displays the food management screen SKG on the food management UI 3200 will be described.

[0156] 11 is a flowchart showing the operations of the terminal device 3 and the food management server 4. In FIG. 11, the flowchart FD shows the operations of the terminal device 3, and the flowchart FE shows the operations of the food management server 4.

[0157] The application execution unit 3100 of the terminal device 3 determines whether to request the record R from the food management server 4 (step SD1).

[0158] For example, when the application execution unit 3100 receives a touch operation instructing the food management U3200 to display the food management screen SKG, the application execution unit 3100 makes a positive determination in step SD1. Also, for example, if the update button B3 on the food management screen SKG is touched, the application executing unit 3100 makes an affirmative determination in step SD1.

[0159] If the application execution unit 3100 determines that record R is to be requested (step SD1: YES), it transmits record request information requesting record R to the food management server 4 via the device communication unit 31 (step SD2). The record request information transmitted in step SD2 includes the user ID 4231 stored in the device storage unit 320.

[0160] Referring to flowchart FE, when the server control unit 40 of the food management server 4 receives record request information via the server communication unit 41 (step SE1), it identifies a record R containing the user ID 4231 included in the received record request information from the food management database 423 (step SE2).

[0161] Then, the server control unit 40 transmits the record R identified by the server communication unit 41 to the terminal device 3 (step SE3).

[0162] Referring to the flowchart FD, when the application execution unit 3100 of the terminal device 3 receives the record R via the terminal communication unit 31 (step SD3), it generates a food management screen SKG (step SD4).

[0163] Here, the processing of step SD4 will be described in detail using an example in which the record R shown in FIG. 6 is received and the food management screen SKG shown in FIG. 10 is generated.

[0164] The application execution unit 3100 determines whether the current date corresponds to the timing indicated by the expiration date notification timing information 4237 associated with the food product name information 4232 indicating milk. In the case of generating the object OB1 in Fig. 10, the application execution unit 3100 determines yes and generates expiration date warning information KJ. Note that if the determination is no, the expiration date warning information KJ is not generated. Next, the application execution unit 3100 determines whether the storage start date indicated by the storage start date information 4235 associated with the food product name information 4232 indicating milk corresponds to the timing indicated by the new notification timing information 4238 associated with the food product name information 4232 indicating milk. In generating the object OB in Fig. 10, the application execution unit 3100 determines that the answer is no and does not generate new storage specification information MJ. Note that if the answer is yes, the application execution unit 3100 generates new storage specification information MJ. The application execution unit 3100 generates an object OB1 that includes the generated expiration date warning information KJ, a food image FG indicated by food image data 4239 corresponding to food name information 4232 indicating milk, and storage start date information 4235 corresponding to food name information 4232 indicating milk. If the application execution unit 3100 has not generated deadline warning information KJ, it does not include the deadline warning information KJ in the object OB1, and if it has generated new accommodation explicit information MJ, it includes the new accommodation explicit information MJ in the object OB1.

[0165] The application execution unit 3100 generates the objects OB2 and OB3 in the same manner as the object OB1, and includes the generated objects OB1, OB2, and OB3 in the food management screen SKG.

[0166] Although the application execution unit 3100 generates an object OB related to an onion, the object OB related to an onion is not included in the food management screen SKG shown in Fig. 10 because the food management screen SKG shown in Fig. 10 is a screen that displays the object OB stored in the refrigerator compartment 11. The object OB related to an onion is included in the food management screen SKG that displays the object OB stored in the vegetable compartment 15.

[0167] Returning to the explanation of the flowchart FD shown in FIG. 11, once the application execution unit 3100 generates the food management screen SKG, it displays the generated food management screen SKG on the food management UI 3200 (step SD5).

[0168] Note that on the food management screen SKG described above, the user P can modify the information displayed for each food item. The information that can be modified is various information associated with the user ID 4231 in the food management database 423. When information is modified on the food management screen SKG, the application execution unit 3100 transmits update request information to the food management server 4, requesting an update of the food management database 423. The food management server 4 then updates the food management database 423 based on the received update request information. For example, when the remaining amount information 4233 of beer is modified from three bottles to two bottles, the application execution unit 3200 transmits update request information to the food management server 4, requesting that the remaining amount information 4233 associated with the beer food name information 4232 beer be updated to indicate two bottles remaining. The food management server 4 then updates the remaining amount indicated in the remaining amount information 4233 from three bottles to two bottles. In this way, since the information can be corrected by the user P, the food management system 1000 can manage food appropriately even if there is a false detection by the food detection device 2 or a false recognition of food by the food management server 4. When the food recognition unit 4100 recognizes food using AI, the food recognition unit 4100 may perform machine learning on the content corrected by the user P and reflect the corrected content in the trained model.

[0169] [1-3. Effects, etc.] As explained above, food detection device 2 is attached to the top of refrigerator 1 by user P, and detects food items entering or leaving refrigerator 1 using refrigerator compartment camera 21 and drawer camera 22. Food detection device 2 is equipped with camera module 220, which has refrigerator compartment camera 21 that takes an image of refrigerator 1 from above the front of refrigerator 1, and drawer camera 22 that takes an image of refrigerator 1 from above the front of refrigerator 1. Food detection device 2 also includes power supply unit 230 that supplies power to at least camera module 220.

[0170] With this configuration, food items being taken in and out of refrigerator 1 can be photographed from above the food items, so that the food items being taken in and out of refrigerator 1 can be accurately included in the photographed results. Therefore, food detection device 2 can accurately detect food items being taken in and out of refrigerator 1 using the camera.

[0171] The power supply unit 230 supplies power to at least the camera module 220 based on power supplied from any one of the commercial AC power supply 7, the refrigerator 1, and a battery provided in the food detection device 2.

[0172] With this configuration, food detection device 2 can take photographs by itself using refrigerator compartment camera 21 and drawer camera 22. Therefore, even if food detection device 2 is configured separately from refrigerator 1, food entering and leaving refrigerator 1 can be detected with high accuracy by the cameras.

[0173] Food detection device 2 is installed on top of refrigerator 1 and detects food entering or leaving refrigerator 1 using refrigerator compartment camera 21 and drawer camera 22. Food detection device 2 includes camera module 220 having refrigerator compartment camera 21 and drawer camera 22 that photograph refrigerator 1 from above the front of refrigerator 1, and power supply unit 230 that supplies power to at least camera module 220. A power cable of food detection device 2 is connected to commercial AC power supply 7. Power supply unit 230 supplies power to at least camera module 210 based on the power supplied via this power cable.

[0174] With this configuration, food detection device 2 can stand alone and take photographs using refrigerator compartment camera 21 and drawer camera 22. Therefore, even if food detection device 2 is configured separately from refrigerator 1, food entering and leaving refrigerator 1 can be accurately detected by the cameras using power supplied from commercial AC power source 7.

[0175] Food detection device 2 is installed on top of refrigerator 1 and detects food entering or leaving refrigerator 1 using refrigerator compartment camera 21 and drawer camera 22. Food detection device 2 includes camera module 220 having refrigerator compartment camera 21 and drawer camera 22 that capture images of refrigerator 1 from above the front of refrigerator 1, and power supply unit 230 that supplies power to at least camera module 220. A USB cable of food detection device 2 is connected to refrigerator 1. Power supply unit 230 supplies power to at least camera module 220 based on the power supplied via the USB cable.

[0176] With this configuration, food detection device 2 can stand alone and take photographs using refrigerator compartment camera 21 and drawer camera 22. Therefore, even if food detection device 2 is configured separately from refrigerator 1, food entering and leaving refrigerator 1 can be accurately detected by the camera using power supplied via the USB standard.

[0177] Food detection device 2 is installed on top of refrigerator 1 and detects food entering or leaving refrigerator 1 using refrigerator compartment camera 21 and drawer camera 22. Food detection device 2 includes camera module 220 having refrigerator compartment camera 21 and drawer camera 22 that photograph refrigerator 1 from above the front of refrigerator 1, power supply unit 230 that supplies power to at least camera module 220, and a battery. Power supply unit 230 supplies power to at least camera module 220 based on the power supplied from the battery.

[0178] With this configuration, food detection device 2 can stand alone and take photographs using refrigerator compartment camera 21 and drawer camera 22. Therefore, even if food detection device 2 is configured separately from refrigerator 1, food entering and leaving refrigerator 1 can be accurately detected by the cameras using power supplied from the built-in battery.

[0179] The food detection device 2 includes a sensor module 210 having an open / close distance measuring sensor 28 that detects the open / closed state of at least one of the door 11C and the drawer 16A of the refrigerator 1. The refrigerator compartment camera 21 and the drawer camera 22 start capturing images when the open / close distance measuring sensor 28 detects the open state, and stop capturing images when the distance measuring sensor detects the closed state.

[0180] This configuration allows food items entering and leaving refrigerator 1 to be included in the photographed results with greater accuracy, while preventing unnecessary photography. Therefore, food items entering and leaving refrigerator 1 can be detected with greater accuracy by the camera while preventing unnecessary photography.

[0181] Sensor module 210 has illuminance sensor 26 that detects the illuminance of the location where refrigerator 1 is installed, and human presence sensor 27 that detects the presence or absence of a person around refrigerator 1. If the detection result of at least one of illuminance sensor 26 and human presence sensor 27 indicates the presence of a person around refrigerator 1, refrigerator compartment camera 21 and drawer camera 22 start capturing images even if opening / closing distance measuring sensor 28 detects the closed state.

[0182] According to this configuration, since photography begins before the user P opens either the door 11C or the drawer 16A of the refrigerator 1, the camera can detect food items being taken in and out of the refrigerator 1 with higher accuracy.

[0183] The refrigerator compartment camera 21 and the drawer camera 22 capture moving images.

[0184] This configuration allows food being taken in and out of refrigerator 1 to be accurately included in the photographed image, regardless of the speed of the food being taken in and out or the angle of the hand taking it in and out. As a result, the camera can detect food being taken in and out of refrigerator 1 with greater accuracy.

[0185] (Embodiment 2) The second embodiment differs from the first embodiment in the number of cameras included in camera module 220. Camera module 220 of the first embodiment is configured to include two cameras: refrigerator compartment camera 21 and drawer camera 22. Camera module 220 of the second embodiment includes one camera. Hereinafter, this camera will be referred to as the refrigerator photography camera.

[0186] In the second embodiment, the food detection device 2 is provided on the top surface of the refrigerator 1 so that the refrigerator photographing camera photographs the front surface of the refrigerator 1 from above and in front of the refrigerator 1. The refrigerator photographing camera captures moving images.

[0187] The photographing range of the refrigerator photographing camera is set to a range that allows photographing of food entering and leaving the refrigerator 1 from above the front of the refrigerator 1.

[0188] For example, the shooting range of the refrigerator photographing camera is a range that includes the shooting range of refrigerator compartment camera 21 and the shooting range of drawer camera 22. In this case, the shooting range of the refrigerator photographing camera includes ranges A1 and A2 shown in Fig. 2 when viewed from the front, and includes ranges A3 and A4 shown in Fig. 3 when viewed from the side.

[0189] In the second embodiment, the food detection device 2, the terminal device 3, and the food management server 4 perform the same operations as those in the first embodiment described above based on the photographing results of the refrigerator photography camera instead of the photographing results of the refrigerator compartment camera 21 and the drawer camera 22.

[0190] According to the second embodiment, the same effects as those of the first embodiment can be achieved.

[0191] (Other embodiments) As described above, the above-described embodiments have been described as examples disclosed in the present application. However, the technology in the present disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above-described embodiments to create new embodiments. Therefore, other embodiments will be exemplified below.

[0192] In the embodiment described above, the imaging range of drawer camera 22 includes ice making compartment 12, fresh freezing compartment 13, freezer compartment 14, and vegetable compartment 15. However, the imaging range of drawer camera 22 is not limited to this range, and may be a range that includes one or a combination of any two or more storage compartments. For example, the imaging range of drawer camera 22 may be a range that includes only freezer compartment 14 and vegetable compartment 1, or may be a range that includes only vegetable compartment 15.

[0193] In the above-described embodiment, the sensor module 210 and the camera module 220 are separate from each other, but they may be integrated into one unit.

[0194] For example, in the above-described embodiment, a distance measuring sensor that measures distance was used as an example of the opening / closing sensor of the present invention, but the opening / closing sensor of the present invention may be any sensor that can detect the open / closed state of the door 11C and the drawer 16A, and may be, for example, a camera.

[0195] For example, in the above-described embodiment, the food detection device 2 is illustrated as a device separate from the refrigerator 1, but the food detection device 2 may be incorporated into the refrigerator 1 and provided in the refrigerator 1.

[0196] For example, in the embodiment described above, record R is stored in food management database 423, and food detected by food detection device 2 is managed by food management server 4. However, record R may be stored in device storage unit 320, and various information related to food detected by food detection device 2 is managed by food management app 323. In this configuration, app execution unit 3100 functions as food recognition unit 4100 to recognize food, and also appropriately updates the contents of record R stored in device storage unit 320, similar to the food management server 4 described above. In this configuration, food management system 1000 does not need to include food management server 4, simplifying the system configuration.

[0197] For example, the types of compartments formed in the main box 10 of the refrigerator 1 are not limited to the refrigeration compartment 11, ice-making compartment 12, fresh freezing compartment 13, freezer compartment 14, and vegetable compartment 15, and there may be fewer or more types of compartments. Also, the number of doors provided at the opening on the front of the refrigeration compartment 11 may be one.

[0198] For example, the functions of the sensor module control unit 211, the camera module control unit 221, the terminal control unit 30, and the server control unit 40 may be realized by multiple processors or semiconductor chips.

[0199] The components shown in Figures 4 and 5 are merely examples, and the specific implementation form is not particularly limited. In other words, it is not necessary to implement hardware corresponding to each component individually; it is of course possible to configure the components so that the functions of each component are realized by a single processor executing a program. Furthermore, some of the functions realized by software in the above-described embodiments may be implemented by hardware, or some of the functions realized by hardware may be implemented by software. In addition, the specific detailed configurations of the other components of the food detection device 2, terminal device 3, and food management server 4 may also be changed as desired without departing from the spirit of the present invention.

[0200] Furthermore, for example, the operational step units shown in Figures 8, 9, and 11 are divided according to the main processing content to facilitate understanding of the operation of each part of the food control system 1000, and the method of dividing the processing units or their names does not limit the present invention. The processing may be divided into more step units depending on the processing content. Furthermore, one step unit may be divided so as to include more processing. Furthermore, the order of the steps may be changed as appropriate within the scope of the present invention.

[0201] It should be noted that the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents. [Industrial Applicability]

[0202] As described above, the food detection device according to the present invention can be used to detect food items entering and leaving a refrigerator. [Explanation of symbols]

[0203] 1 refrigerator 2. Food detection equipment 3 Terminal Devices 4. Food Management Server 7 Commercial AC power supply (commercial power supply) 11 Refrigerator 11A Left door (door) 11B Right door (door) 11C Door 12A, 13A, 14A, 15A, 16A drawer 21 Refrigerator camera (camera) 22 Drawer camera (camera) 23 Distance sensor for left door (opening / closing sensor) 24 Right door distance sensor (opening / closing sensor) 25 Drawer distance sensor (open / close sensor) 26 Illuminance sensor 27 Human Sensor 28 Opening and closing distance sensor (opening and closing sensor) 210 Sensor Module 220 Camera Module 221A Detected value processing section 221B Imaging control unit 221C Video Processing Unit 221D Food detection unit 230 Power supply section 4100 Food Recognition Department

Claims

1. A photographing device that is attached to the top of a refrigerator by a user, a camera module having a camera that photographs the refrigerator from above the front of the refrigerator; a sensor module having sensors related to the refrigerator; a power supply unit that supplies power to the camera module and the sensor module, a power cable of the imaging device is connected to a commercial AC power source outside the refrigerator; the power supply unit supplies power to the camera module and the sensor module based on power supplied from the commercial AC power supply via the power cable. Filming equipment.

2. An imaging device provided on the top of a refrigerator, a camera module having a camera that photographs the refrigerator from above the front of the refrigerator; a sensor module having sensors related to the refrigerator; a power supply unit that supplies power to the camera module and the sensor module, a power cable of the imaging device is connected to a commercial AC power source outside the refrigerator; the power supply unit supplies power to the camera module and the sensor module based on power supplied from the commercial AC power supply via the power cable. Filming equipment.

3. The power supply unit includes a circuit for converting AC to DC.

3. The imaging device according to claim 1.

4. the sensor module includes an open / close sensor that detects the open / close state of at least one of a door and a drawer of the refrigerator, The camera starts capturing an image when the open / close sensor detects an open state, and stops capturing an image when the open / close sensor detects a closed state. The imaging device according to claim 1 .

5. the sensor module includes an illuminance sensor that detects illuminance at a location where the refrigerator is installed, and a human presence sensor that detects a person present around the refrigerator, When a detection result of at least one of the illuminance sensor and the human presence sensor indicates the presence of a person around the refrigerator, the camera starts capturing an image even if the open / close sensor detects a closed state. The imaging device according to claim 4.

6. The camera captures moving images. The imaging device according to claim 1 .

Images