Photographing device
The food detection device on the refrigerator's top captures comprehensive photographic results using a camera positioned above and in front, addressing inaccuracies in existing systems and improving detection precision.
Patent Information
- Application Number
- JP2024161771
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2040-09-03
AI Technical Summary
Existing food detection systems struggle to accurately capture food entering and leaving a refrigerator due to limitations in camera positioning, leading to incomplete photographic results and inaccurate food detection.
A food detection device installed on the top of a refrigerator, equipped with a camera module that photographs from above and in front, and a power supply unit connected to a commercial power source, USB, or battery, to ensure accurate detection of food entering and leaving by capturing comprehensive photographic results.
The device accurately detects food entering and leaving the refrigerator by ensuring all relevant areas are captured in the photographic results, enhancing detection precision and preventing unnecessary photography.
Smart Images

Figure 2024166382000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a food detection device. [Background technology]
[0002] Patent Document 1 discloses a system that allows a user to check the interior of a refrigerator even in a remote location. The system captures images of the interior of the refrigerator using a camera unit installed in the door pocket of the refrigerator and transmits the captured images to a communication terminal of the user. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2015-111026 A Summary of the Invention [Problem to be solved by the invention]
[0004] The present disclosure provides a food detection device that can accurately detect food items entering or leaving a refrigerator using a camera. [Means for solving the problem]
[0005] The food detection device of the present disclosure is a food detection device that is provided on top of a refrigerator and detects food entering or leaving the refrigerator using a camera, and includes a camera module having a camera that photographs the refrigerator from above the front of the refrigerator, and a power supply unit that supplies power to at least the camera module.
[0006] The food detection device of the present disclosure is a food detection device that is mounted on top of a refrigerator and detects food entering or leaving the refrigerator using a camera, and includes a camera module having a camera that photographs the refrigerator from the front above the refrigerator, and a power supply unit that supplies power to at least the camera module, wherein a power cable of the food detection device is connected to a commercial power source, and the power supply unit supplies power to at least the camera module based on the power supplied via the power cable.
[0007] The food detection device of the present disclosure is a food detection device that is mounted on top of a refrigerator and detects food entering or leaving the refrigerator using a camera, and includes a camera module having a camera that takes images of the refrigerator from above the front of the refrigerator, and a power supply unit that supplies power to at least the camera module, wherein a USB cable of the food detection device is connected to the refrigerator, and the power supply unit supplies power to at least the camera module based on the power supplied via the USB cable.
[0008] The food detection device of the present disclosure is a food detection device that is mounted on top of a refrigerator and detects food entering or leaving the refrigerator using a camera, and includes a camera module having a camera that photographs the refrigerator from above the front, a power supply unit that supplies power to at least the camera module, and a battery, and the power supply unit supplies power to at least the camera module based on the power supplied from the battery. Effect of the Invention
[0009] The food detection device disclosed herein can capture images of food entering and leaving the refrigerator from above the food, so that the food entering and leaving the refrigerator can be accurately included in the captured image. Therefore, the food entering and leaving the refrigerator can be accurately detected by the camera. [Brief description of the drawings]
[0010] [Figure 1]Diagram showing the configuration of a food control system [Diagram 2] Front view of refrigerator and food detection device [Diagram 3] Right side view of the refrigerator and food detection device [Figure 4] Block diagram showing the configuration of a food detection device [Diagram 5] Block diagram showing the configuration of a terminal device and a food management server [Figure 6] An example of a food management database [Figure 7] FIG. 13 is a diagram showing an example of an installation screen. [Figure 8] Flowchart showing the operation of the food detection device [Figure 9] Flowchart showing the operation of the food detection device and the food management server [Figure 10] FIG. 13 is a diagram showing an example of a food management screen. [Figure 11] Flowchart showing the operation of the terminal device and the food management server DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] (The knowledge and other information that formed the basis of this disclosure) The above-mentioned conventional technology is configured to take images from the front side of the refrigerator compartment toward the back side, so even if there is another food behind a certain food, the captured image may not include the other food. Therefore, it is difficult to accurately detect foods being put in and taken out of the refrigerator from the results of the image taken by a camera that takes images like the above-mentioned conventional technology. Therefore, the present disclosure provides a food detection device that can accurately detect food items being put into and taken out of a refrigerator using a camera.
[0012] Hereinafter, the embodiments will be described in detail with reference to the drawings. However, in some cases, more detailed explanations than necessary may be omitted. For example, detailed explanations of already well-known matters or duplicate explanations of substantially the same configurations may be omitted. It should be noted that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.
[0013] (Embodiment 1) First, the first embodiment will be described. [1-1. Configuration] FIG. 1 is a diagram showing the configuration of a food management system 1000.
[0014] The food management system 1000 is a system that manages the remaining amount of food stored in the refrigerator 1, the storage start date, the best-before date or expiration date, and the like.
[0015] Food management system 1000 includes a refrigerator 1. Refrigerator 1 includes a main box 10 with an opening at the front. Main box 10 is formed with a refrigerator compartment 11, an ice-making compartment 12, a fresh-freezing compartment 13, a freezer compartment 14, and a vegetable compartment 15. A rotating left door 11A and a right door 11B are provided at the opening at the front of refrigerator compartment 11. Ice-making compartment 12, fresh-freezing compartment 13, freezer compartment 14, and vegetable compartment 15 are each provided with drawers 12A, 13A, 14A, and 15A for storing food. In the following description, when there is no need to distinguish between the left door 11A and the right door 11B, they will be referred to as "doors" and collectively referred to as "11C." Additionally, in the following description, when there is no need to distinguish between the drawers 12A, 13A, 14A, and 15A, they will be referred to as "drawers" and collectively referred to as "16A."
[0016] The food management system 1000 includes a food detection device 2. The food detection device 2 is a device that detects food entering or leaving the refrigerator 1, and is provided on the top surface of the refrigerator 1 as an upper portion of the refrigerator 1.
[0017] Here, the configuration of the food detection device 2 will be described with reference to FIGS. Fig. 2 is a view of the refrigerator 1 and the food detection device 2 as seen from the front side. The refrigerator 1 shown in Fig. 2 has its left door 11A and right door 11B open. Fig. 3 is a view of the refrigerator 1 and the food detection device 2 as seen from the right side. The refrigerator 1 shown in Fig. 3 has its drawer 15A in an open state, i.e., pulled out.
[0018] 2 and 3 show an X-axis, a Y-axis, and a Z-axis. The X-axis, Y-axis, and Z-axis are perpendicular to each other. The Z-axis indicates the up-down direction. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the left-right direction. The Y-axis indicates the front-back direction. The positive direction of the X-axis indicates the rightward direction. The positive direction of the Y-axis indicates the forward direction. The positive direction of the Z-axis indicates the upward direction.
[0019] Food detection device 2 has a refrigerator compartment camera 21 and a drawer camera 22. Food detection device 2 is provided on the top surface of refrigerator 1 so that refrigerator compartment camera 21 and drawer camera 22 capture images of the front surface of refrigerator 1 from above and in the front of refrigerator 1.
[0020] The refrigerator compartment camera 21 is a camera for detecting food entering or leaving the refrigerator compartment 11.
[0021] The photographing range of the refrigeration compartment camera 21 is set to a range in which food entering and leaving the refrigeration compartment 11 can be photographed from above the front of the refrigerator 1.
[0022] For example, the photographing range of the refrigerator compartment camera 21 includes the range A1 shown in FIG. 2 when viewed from the front. The range A1 is a range including the door pockets 11A1, 11A2, 11A3, and 11A4 provided in the left door 11A, the door pockets 11B1, 11B2, 11B3, and 11B4 provided in the right door 11B, and the opening of the refrigerator compartment 11 when the left door 11A and the right door 11B are in an open state. The photographing range of the refrigerator compartment camera 21 includes the range A3 shown in FIG. 3 when viewed from the side. The range A3 is a range including from the door 11C in an open state to the front ends of the shelves 111, 112, 113, and 114 in the front-rear direction, and includes the opening of the refrigerator compartment 11 in the up-down direction. The shelves 111, 112, 113, and 114 are members provided in the refrigerator compartment 11 and divide the refrigerator compartment 11 into a plurality of areas.
[0023] Drawer camera 22 is a camera for detecting food entering or leaving ice-making compartment 12, fresh-freezing compartment 13, freezer compartment 14, and vegetable compartment 15.
[0024] The imaging range of drawer camera 22 is set to a range capable of imaging food entering or leaving ice-making compartment 12, fresh freezing compartment 13, freezer compartment 14, and vegetable compartment 15 from above the front of refrigerator 1.
[0025] For example, the shooting range of drawer camera 22 includes range A2 shown in Fig. 2 when viewed from the front. Range A2 is a range that includes drawers 12A, 13A, 14A, and 15A when viewed from the front of refrigerator 1. Moreover, the shooting range of drawer camera 22 includes range A4 shown in Fig. 3 when viewed from the side. Range A4 is a range that includes the top surfaces of drawers 12A, 13A, 14A, and 15A when they are pulled out in the front-to-rear direction.
[0026] The food detection device 2 has a left door distance measuring sensor 23, a right door distance measuring sensor 24, and a drawer distance measuring sensor 25. The left door distance measuring sensor 23, the right door distance measuring sensor 24, and the drawer distance measuring sensor 25 correspond to examples of opening and closing sensors. In the following description, when the left door distance measuring sensor 23, the right door distance measuring sensor 24, and the drawer distance measuring sensor 25 are not distinguished from each other, they will be referred to as "opening and closing distance measuring sensors" and given the symbol "28". The opening and closing distance measuring sensor 28 outputs detection values to the sensor module control unit 211 described later.
[0027] The left door distance measuring sensor 23 is a distance measuring sensor for detecting the open / closed state of the left door 11A. The left door distance measuring sensor 23 detects the distance to the left door 11A as the open / closed state of the left door 11A.
[0028] The right door distance measuring sensor 24 is a distance measuring sensor for detecting the open / closed state of the right door 11B. The right door distance measuring sensor 24 detects the distance to the right door 11B as the open / closed state of the right door 11B.
[0029] Drawer distance measuring sensor 25 is a distance measuring sensor for detecting the open / closed state of each of drawers 12A, 13A, 14A, 15A. The open state of drawers 12A, 13A, 14A, 15A means that they are drawn out of the corresponding storage chambers. The closed state of drawers 12A, 13A, 14A, 15A means that they are stored in the corresponding storage chambers. Drawer distance measuring sensor 25 detects the distance between each of drawers 12A, 13A, 14A, 15A as the open / closed state of each of drawers 12A, 13A, 14A, 15A.
[0030] The food detection device 2 has an illuminance sensor 26 and a human presence sensor 27. The illuminance sensor 26 and the human presence sensor 27 output detection values to the sensor module control unit 211.
[0031] The illuminance sensor 26 detects the illuminance of the location where the food detection device 2 is placed, i.e., the location where the refrigerator 1 is placed.
[0032] The human presence sensor 27 detects people around the refrigerator 1. In this embodiment, the human presence sensor 27 is provided on the front side of the food detection device 2, and detects people around the refrigerator 1 in front of the refrigerator 1.
[0033] 2, drawer distance measuring sensor 25, drawer camera 22, and refrigerator compartment camera 21 are illustrated as being arranged side by side in this order in the left-right direction. However, this is illustrated for the convenience of explaining food detection device 2, and they do not necessarily have to be arranged as shown in FIG. 3, drawer distance measuring sensor 25, drawer camera 22, and refrigerator compartment camera 21 are illustrated as being arranged in this order in the front-to-rear direction. However, this is an illustration for food detection device 2, and they do not necessarily have to be arranged as in FIG.
[0034] Returning to the explanation of Fig. 1, the food management system 1000 includes a terminal device 3 having a touch panel 32. Terminal device 3 is, for example, a smartphone or a tablet terminal. An application program for managing the foods stored in refrigerator 1 is installed on terminal device 3, and communicates with food management server 4 using the function of the application. In the following description, an application program for managing food stored in refrigerator 1 is referred to as a "food management application" and is given the reference number "323."
[0035] 1, a user P who is at home is indicated by a solid line, and a user P who has left home H is indicated by a dotted line. When used by user P who is at home, terminal device 3 communicates with food management server 4 connected to global network GN via communication device 5 or without communication device 5, using the function of food management app 323. When used by user P who has left home H, terminal device 3 communicates with food management server 4 without via communication device 5, using the function of food management app 323.
[0036] The communication device 5 connects to a global network GN including the Internet, a telephone network, and other communication networks, and communicates with the food management server 4 connected to the global network GN. The communication device 5 is an interface device for connecting the terminal device 3 to the global network GN. The communication device 5 has functions such as a modem-related function, a router function, and a NAT (Network Address Translation) function. The communication device 5 transfers data sent and received between the terminal device 3 and the food management server 4 connected to the global network GN.
[0037] Food management system 1000 includes a food management server 4. Food management server 4 is a server device that manages the food stored in refrigerator 1, and is connected to global network GN. Note that in each figure, food management server 4 is represented by a single block, but this does not necessarily mean that food management server 4 is composed of a single server device. For example, food management server 4 may be composed of multiple server devices with different processing contents.
[0038] Next, the configurations of the food detection device 2, the terminal device 3, and the food management server 4 will be described.
[0039] FIG. 4 is a block diagram showing the configuration of the food detection device 2. The food detection device 2 comprises a sensor module 210, a camera module 220, and a power supply 230.
[0040] The sensor module 210 includes a sensor module control unit 211 , a sensor unit 212 , and a sensor side inter-module communication unit 213 .
[0041] The sensor module control unit 211 includes a sensor processor 2111, which is a processor that executes programs such as a CPU or MPU, and a sensor storage unit 2112, and controls each unit of the sensor module 210. The sensor module control unit 211 executes various processes through cooperation between hardware and software, such that the sensor processor 2111 reads out a control program 2113 stored in the sensor storage unit 2112 and executes processing.
[0042] The sensor storage unit 2112 has a storage area for storing programs executed by the sensor processor 2111 and data processed by the sensor processor 2111. The sensor storage unit 2112 stores a control program 2113 executed by the sensor processor 2111, setting data 2114 related to settings of the sensor module 210, and various other data. The sensor storage unit 2112 has a non-volatile storage area. The sensor storage unit 2112 may also include a volatile storage area and constitute a work area for the sensor processor 2111.
[0043] The sensor unit 212 includes a left door distance measuring sensor 23, a right door distance measuring sensor 24, a drawer distance measuring sensor 25, an illuminance sensor 26, and a human presence sensor 27, and outputs detection values for each sensor to the sensor module control unit 211. Note that the sensor unit 212 may include sensors other than the above five types of sensors.
[0044] The sensor side inter-module communication unit 213 includes communication hardware such as a communication circuit and a connector that conforms to a predetermined communication standard, and communicates with the camera module 220 under the control of the sensor module control unit 211 .
[0045] The camera module 220 includes a camera module control unit 221 , an image capturing unit 222 , a camera module communication unit 223 , and a camera side inter-module communication unit 224 .
[0046] The camera module control unit 221 includes a camera processor 2211 which is a processor that executes programs such as a CPU or MPU, and a camera storage unit 2212, and controls each unit of the camera module 220. The camera module control unit 221 executes various processes by cooperation of hardware and software, such that the camera processor 2211 reads out a control program 2213 stored in the camera storage unit 2212 and executes the process. The camera module control unit 221 reads out and executes the control program 2213, thereby functioning as a detection value processing unit 221A, an imaging control unit 221B, a video processing unit 221C, and a food detection unit 221D.
[0047] The camera storage unit 2212 has a storage area for storing programs executed by the camera processor 2211 and data processed by the camera processor 2211. The camera storage unit 2212 stores a control program 2213 executed by the camera processor 2211, setting data 2214 related to settings of the camera module 220, a food detection device ID 2215, and various other data. The food detection device ID 2215 will be described later. The camera storage unit 2212 has a non-volatile storage area. The camera storage unit 2212 may also include a volatile storage area and constitute a work area for the camera processor 2211.
[0048] The photographing unit 222 includes a refrigerator compartment camera 21 and a drawer camera 22. The photographing unit 222 outputs the photographing results of each of the refrigerator compartment camera 21 and the drawer camera 22 to the camera module control unit 221. In the present embodiment, the refrigerator compartment camera 21 and the drawer camera 22 capture moving images. Therefore, the photographing unit 222 outputs moving image data to the camera module control unit 221 as the photographing result.
[0049] The camera module communication unit 223 includes communication hardware such as a communication circuit and a connector that comply with a predetermined communication standard, and communicates with the food management server 4 under the control of the camera module control unit 221. In this embodiment, the communication standard used by the camera module communication unit 223 is exemplified as a wireless communication standard, but it may also be a wired communication standard.
[0050] The camera side inter-module communication unit 224 includes communication hardware such as a communication circuit and a connector that conforms to a predetermined communication standard, and communicates with the sensor module 210 under the control of the camera module control unit 221 .
[0051] As described above, the camera module control unit 221 functions as the detection value processing unit 221A, the photography control unit 221B, the video processing unit 221C, and the food detection unit 221D.
[0052] The detection value processing unit 221A judges whether each of the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A is in an open state or a closed state based on the detection values of various sensors output from the sensor module 210. For example, the detection value processing unit 221A judges whether each of the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A is in an open state or a closed state based on whether the detection value is below a predetermined threshold value or not. Note that the predetermined threshold value may be different for each of the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A. When the detection value processing unit 221A judges that any of the doors is in an open state, it outputs open / closed state information indicating that the door is in an open state to the shooting control unit 221B and the video processing unit 221C. In addition, when the detection value processing unit 221A determines that the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A are all in a closed state, it outputs opening / closing state information indicating that they are in a closed state to the shooting control unit 221B and the video processing unit 221C.
[0053] Furthermore, the detection value processing unit 221A determines whether or not a person is present around the refrigerator 1 based on the detection values of the illuminance sensor 26 and the human presence sensor 27 output from the sensor module 210. For example, the detection value processing unit 221A determines whether or not a person is present around the refrigerator 1 based on whether or not the detection value is below a predetermined threshold. If the detection value processing unit 221A determines that a person is present around the refrigerator 1, it outputs person presence information indicating that a person is present to the photography control unit 221B. If the detection value processing unit 221A determines that no person is present around the refrigerator 1, it outputs person presence information indicating that no person is present to the photography control unit 221B.
[0054] The photographing control section 221B controls the photographing section 222 to cause the refrigerator compartment camera 21 and the drawer camera 22 to photograph moving images. The photographing control section 221B will be described in detail with reference to FIG.
[0055] The video processing unit 221C performs various processes on the images captured by the refrigerator compartment camera 21 and the drawer camera 22, and stores the processed images in the camera storage unit 2212. The video processing unit 221C will be described in detail with reference to FIG.
[0056] Food detection unit 221D detects food entering or leaving refrigerator 1 based on the results of shooting by refrigerator compartment camera 21 and drawer camera 22. Details of food detection unit 221D will be described with reference to FIG.
[0057] Power supply unit 230 includes hardware such as a circuit for supplying power, and supplies power to each of sensor module 210 and camera module 220. Power supply unit 230 may be configured to supply power in parallel to each of sensor module 210 and camera module 220 as in this embodiment, or may be configured to supply power to one of the modules via the other.
[0058] The power supply unit 230 of this embodiment supplies power to each of the sensor module 210 and the camera module 220 based on power supplied from an external commercial AC power supply 7. The commercial AC power supply 7 corresponds to an example of a commercial power supply. A power cable of the food detection device 2 is connected to the commercial AC power supply 7, and the power supply unit 230 receives power from the commercial AC power supply 7 via the power cable, and supplies power to each of the sensor module 210 and the camera module 220 based on the supplied power. The power supply unit 230 includes, as hardware, for example, a circuit for converting AC to DC, a circuit for stepping down, etc.
[0059] The source of power supply to the power supply unit 230 is not limited to the external commercial AC power supply 7, and may be the refrigerator 1 or a battery. When the source is the former, the power supply unit 230 and the refrigerator 1 are equipped with hardware that complies with the USB power supply standard. When the source is the former, a USB cable of the food detection device 2 is connected to a USB port of the refrigerator 1, and the refrigerator 1 supplies power to the power supply unit 230 via the USB cable. When the source is the latter, the food detection device 2 is equipped with a battery.
[0060] FIG. 5 is a block diagram showing the configuration of the terminal device 3 and the food management server 4.
[0061] First, the configuration of the food management server 4 will be described.
[0062] The food management server 4 includes a server control unit 40 and a server communication unit 41.
[0063] The server control unit 40 includes a server processor 410, which is a processor that executes programs such as a CPU or MPU, and a server memory unit 420, and controls each unit of the food management server 4. The server control unit 40 executes various processes through cooperation between hardware and software, such that the server processor 410 reads a control program 421 stored in the server memory unit 420 and executes the processes. The server control unit 40 functions as a food recognition unit 4100 by reading and executing the control program 421.
[0064] Server storage unit 420 has a storage area for storing programs executed by server processor 410 and data processed by server processor 410. Server storage unit 420 stores control programs 421 executed by server processor 410, setting data 422 related to settings of food management server 4, food management database 423, and various other data. Server storage unit 420 has a non-volatile storage area. Server storage unit 420 may also have a volatile storage area and constitute a work area for server processor 410.
[0065] FIG. 6 is a diagram showing an example of the food management database 423. The food management database 423 is a database that stores various information related to the food stored in the refrigerator 1.
[0066] Each record R stored in the food management database 423 has a user ID 4231, a food detection device ID 2215, food name information 4232, remaining amount information 4233, storage location information 4234, storage start date information 4235, expiration date information 4236, expiration date notification timing information 4237, new notification timing information 4238, and food image data 4239.
[0067] Each record R stored in the food management database 423 may further include one or more other types of information.
[0068] The user ID 4231 is identification information for identifying the user P who uses the food management application 323, and is appropriately assigned to the user P who uses the food management application 323.
[0069] The food detection device ID 2215 is identification information for identifying the food detection device 2. The food detection device ID 2215 is, for example, a manufacturing number of the food detection device 2, a serial number of the food detection device 2, a manufacturing number of the camera module 220, a serial number of the camera module 220, or the like.
[0070] The food name information 4232 is information that indicates the name of a food. In one record R, one or more pieces of food name information 4232 are associated with a user ID 4231.
[0071] The remaining amount information 4233 is information indicating the remaining amount of food. In one record R, the same number of remaining amount information 4233 as the food name information 4232 corresponding to the user ID 4231 are associated. The remaining amount of food indicated by the remaining amount information 4233 in Fig. 6 is indicated in units corresponding to the name of the food. The remaining amount indicated by the remaining amount information 4233 is appropriately updated by the server control unit 40.
[0072] The storage location information 4234 is information indicating a storage location of the food. In this embodiment, the storage location information 4234 indicates any one of the storage compartments of the refrigerator 1 as the storage location. In one record R, the same number of pieces of storage location information 4234 as the number of pieces of food name information 4232 associated with the user ID 4231 are associated.
[0073] The storage start date information 4235 is information that indicates the storage start date of the food. In one record R, the same number of storage start date information 4235 as the food name information 4232 corresponding to the user ID 4231 are associated.
[0074] The expiration date information 4236 is information that indicates the expiration date of a food product, such as the best-before date or the expiration date. In one record R, the same number of expiration date information 4236 as the food product name information 4232 associated with the user ID 4231 are associated.
[0075] The expiration date notification timing information 4237 is information indicating the timing for notifying the user P of the expiration date indicated by the corresponding expiration date information 4236. As shown in Fig. 6, the timing indicated by the expiration date notification timing information 4237 is the number of days before the expiration date indicated by the expiration date information 4236. In one record R, the same number of pieces of expiration date notification timing information 4237 as the number of food name information 4232 associated with the user ID 4231 are associated.
[0076] The new notification timing information 4238 is information indicating the timing of notifying that new food has been stored in the refrigerator 1. As shown in Fig. 6, the timing indicated by the new notification timing information 4238 is a number of days from the storage start date indicated by the storage start date information 4235.
[0077] The food image data 4239 is image data of a food image FG that is an image of food. In one record R, the same number of food image data 4239 as the food name information 4232 associated with the user ID 4231 are associated.
[0078] 5, the server communication unit 41 includes communication hardware such as a communication circuit and a connector that conforms to a predetermined communication standard, and communicates with devices connected to the global network GN according to the predetermined communication standard under the control of the server control unit 40. In this embodiment, the server communication unit 41 communicates with the food detection device 2 and the terminal device 3.
[0079] As described above, the server control unit 40 functions as the food recognition unit 4100. The food recognition unit 4100 recognizes the food detected by the food detection device 2. Details of the food recognition unit 4100 will be described with reference to FIG.
[0080] Next, the functional configuration of the terminal device 3 will be described. The terminal device 3 includes a terminal control unit 30, a terminal communication unit 31, and a touch panel 32.
[0081] The terminal control unit 30 includes a terminal processor 310, which is a processor that executes programs such as a CPU or MPU, and a terminal storage unit 320, and controls each unit of the terminal device 3. The terminal control unit 30 executes various processes through cooperation of hardware and software, such that the terminal processor 310 reads out a control program 321 stored in the terminal storage unit 320 and executes the processes. A food management application 323 is pre-installed in the terminal device 3. The food management application 323 is a program that is read out from the terminal storage unit 320 by the terminal processor 310 and executed, causing the terminal control unit 30 to function as an application execution unit 3100.
[0082] The terminal storage unit 320 has a storage area for storing programs executed by the terminal processor 310 and data processed by the terminal processor 310. The terminal storage unit 320 stores a control program 321 executed by the terminal processor 310, setting data 322 related to the settings of the terminal device 3, a food management application 323, a user ID 4231, and various other data. The terminal storage unit 320 has a non-volatile storage area. The terminal storage unit 320 may also have a volatile storage area and constitute a work area for the terminal processor 310.
[0083] The terminal communication unit 31 includes communication hardware such as a communication circuit conforming to a predetermined communication standard, and communicates with devices connected to the global network GN according to the predetermined communication standard under the control of the terminal control unit 30. In this embodiment, the terminal communication unit 31 communicates with the food management server 4 according to the predetermined communication standard under the control of the application execution unit 3100. In this embodiment, the communication standard used by the terminal communication unit 31 is exemplified as a wireless communication standard, but may be a wired communication standard.
[0084] The touch panel 32 includes a display panel such as a liquid crystal display panel, and a touch sensor that is overlaid on or integrated with the display panel. The display panel displays various images under the control of the terminal control unit 30. The touch sensor detects a touch operation and outputs it to the terminal control unit 30. The terminal control unit 30 executes a process corresponding to the touch operation based on an input from the touch sensor.
[0085] As described above, the terminal control unit 30 functions as the application execution unit 3100. The application execution unit 3100 controls the terminal communication unit 31 to send and receive various information to and from the food management server 4. If a communication connection between the terminal device 3 and the communication device 5 is possible, the application execution unit 3100 sends and receives various information to and from the food management server 4 via the communication device 5, and if not, sends and receives various information to and from the food management server 4 without going through the communication device 5. Therefore, even if the user P is not located inside or near the home H, the terminal device 3 can send and receive various information to and from the food management server 4 by using the functions of the food management application 323.
[0086] The application execution unit 3100 also displays a user interface related to managing the foods stored in the refrigerator 1 on the touch panel 32. In the following description, this user interface is referred to as a "food management UI" and is given the reference number "3200." By displaying the food management UI 3200 on the touch panel 32, the application execution unit 3100 provides the user P with various information related to the foods stored in the refrigerator 1, and also accepts various inputs related to the foods stored in the refrigerator 1 from the user P.
[0087] [1-2. Operation] Next, the operation of each part of the food control system 1000 will be described. In this embodiment, the food detection device 2 is illustrated as being separate from the refrigerator 1. Therefore, the food detection device 2 is attached to the top surface of the refrigerator 1 by the user P.
[0088] First, the operation of each part of the food management system 1000 when the food detection device 2 is attached to the refrigerator 1 will be described.
[0089] When the food detection device 2 is attached to the refrigerator 1, the application execution unit 3100 causes the touch panel 32 to display the food management UI of the attachment screen TG.
[0090] FIG. 7 is a diagram showing an example of the attachment screen TG. The attachment screen TG is provided with a photography result display area HA1. The photography result display area HA1 is a rectangular area that displays the photography results of either the refrigerator compartment camera 21 or the drawer camera 22. The photography result display area HA1 is provided with a line L1 extending in the vertical direction at approximately the center in the horizontal direction in the figure.
[0091] The installation screen TG includes installation method information J1. The installation method information J1 is information indicating how to install the food detection device 2 on the top surface of the refrigerator 1. The installation method information J1 shown in Fig. 7 indicates that the refrigerator 1 should be installed so that the center in the left-right direction of the refrigerator 1 shown in the photography result displayed in the photography result display area HA1 is aligned with a line L1 provided in the photography result display area HA1.
[0092] When user P attaches food detection device 2 to the top surface of refrigerator 1 following the installation method indicated in installation method information J1, the shooting range of refrigerator compartment camera 21 and the shooting range of drawer camera 22 are set to the shooting ranges described with reference to Figures 2 and 3. In other words, food detection device 2 is attached to refrigerator 1 so that food entering or leaving refrigerator 1 can be included in the shooting ranges of refrigerator compartment camera 21 and drawer camera 22.
[0093] Here, we will explain the operation of each part of the food management system 1000 when the food detection device 2 is attached to the refrigerator 1. In this case, the power supply unit 230 supplies power to at least the camera module 220, and the camera module 220 can transmit the image capture results of either the refrigerator compartment camera 21 or the drawer camera 22 to the terminal device 3. When the food detection device 2 is attached to the refrigerator 1, the camera module 220 and the terminal device 3 may communicate with each other via the food management server 4, or may communicate directly by short-range wireless communication or the like.
[0094] When the application execution unit 3100 of the terminal device 3 receives an instruction to display the attachment screen TG, the application execution unit 3100 transmits request information requesting the photographing result to the food detection device 2 via the terminal communication unit 31. When the camera module control unit 221 of the camera module 220 of the food detection device 2 receives request information via the camera module communication unit 223, it transmits the photographing results of either the refrigerator compartment camera 21 or the drawer camera 22 to the terminal device 3 via the camera module communication unit 223. Then, the application executing unit 3100 displays the received photographing result in the photographing result display area HA1.
[0095] As described above, the application executing unit 3100 causes the touch panel 32 to display the food management UI 3200 of the attachment screen TG, so that the user P can attach the food detecting device 2 to the refrigerator 1 simply and appropriately.
[0096] Next, the operation of each part of food management system 1000 involved in the management of food will be described.
[0097] First, the operation of food detection device 2 relating to the photographing by refrigerator compartment camera 21 and drawer camera 22 will be described. FIG. 8 is a flowchart showing the operation of the food detection device 2. 8, the sensor module 210 transmits detection values of various sensors included in the sensor unit 212 to the camera module 220 via the sensor side inter-module communication unit 213, and the camera module 220 receives the detection values via the camera side inter-module communication unit 224. Then, in the operation shown in FIG. 8, the detection value processing unit 221A executes processing based on the various detection values received from the sensor module 210.
[0098] The photography control section 221B of the camera module 220 determines whether or not there is a person around the refrigerator 1 (step SA1).
[0099] If the shooting control unit 221B acquires person presence / absence information indicating the presence of a person from the detection value processing unit 221A, it makes a positive judgment in step SA1, and if the shooting control unit 221B acquires person presence / absence information indicating the absence of a person from the detection value processing unit 221A, it makes a negative judgment in step SA1.
[0100] When it is determined that a person is present around the refrigerator 1 (step SA1: YES), the shooting control section 221B starts shooting moving images by the refrigerator compartment camera 21 and the drawer camera 22 (step SA3).
[0101] On the other hand, if the photography control unit 221B determines that no person is present around the refrigerator 1 (step SA1: NO), it determines whether any of the left door 11A, right door 11B, and drawers 12A, 13A, 14, and 15A are open (step SA2).
[0102] If the shooting control unit 221B acquires opening / closing state information indicating an open state from the detection value processing unit 221A, it makes a positive judgment in step SA2, and if it acquires opening / closing state information indicating a closed state from the detection value processing unit 221A, it makes a negative judgment in step SA2.
[0103] If the photography control unit 221B determines that none of the left door 11A, the right door 11B, and the drawers 12A, 13A, 14, and 15A are open (step SA2: NO), the photography control unit 221B returns the process to step SA1 and executes the process of step SA1 again.
[0104] When it is determined that any one of the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A is open (step SA2: YES), the photography control unit 221B executes the process of step SA3.
[0105] When shooting starts in step SA3, moving image processing unit 221C of camera module 220 executes the processes of steps SA4 to SA8 for each frame constituting the moving image.
[0106] When the refrigerator compartment camera 21 and the drawer camera 22 start capturing images, the video processing unit 221C acquires the open / closed state information from the detection value processing unit 221A (step SA4).
[0107] Next, the video processing unit 221C reduces one frame of the captured images from each of the refrigerator compartment camera 21 and the drawer camera 22 (step SA5). The reduction here refers to so-called resizing.
[0108] The video processing unit 221C associates the reduced captured image for one frame with the open / close state information acquired in step SA4, and stores them in the camera storage unit 2212 (step SA6). Note that the video processing unit 221C stores the images in chronological order every time it performs step SA6.
[0109] Next, the photography control section 221B determines whether or not there is a person around the refrigerator 1, similar to step SA1 (step SA7).
[0110] If the shooting control section 221B determines that a person is present around the refrigerator 1 (step SA7: YES), the video processing section 221C executes the processes from step SA4 onwards again.
[0111] On the other hand, if the photography control unit 221B determines that no person is present around the refrigerator 1 (step SA7: NO), it determines whether or not the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A are all closed, as in step SA2 (step SA8).
[0112] If the photography control unit 221B determines that the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, 15A are not all closed (step SA8: NO), the video processing unit 221C executes the processes from step SA4 onwards again.
[0113] On the other hand, if the photography control unit 221B determines that the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A are all closed (step SA8: YES), photography by the refrigerator compartment camera 21 and the drawer camera 22 is terminated (step SA9).
[0114] Next, operations related to food management based on the images captured by refrigerator compartment camera 21 and drawer camera 22 will be described. Fig. 9 is a flowchart showing the operations of the food detection device 2 and the food management server 4. In Fig. 9, a flowchart FB shows the operations of the food detection device 2. Also, in Fig. 9, a flowchart FC shows the operations of the food management server 4.
[0115] Food detection unit 221D of camera module 220 of food detection device 2 determines whether or not video images captured by refrigerator compartment camera 21 and drawer camera 22 have been stored in camera storage unit 2212 (step SB1).
[0116] If the food detection unit 221D determines that the video image is not stored in the camera storage unit 2212 (step SB1: NO), it executes the process of step SB1 again.
[0117] On the other hand, if the food detection unit 221D determines that a video image has been stored in the camera storage unit 2212 (step SB1: YES), it executes a valid frame determination process (step SB2).
[0118] The valid frame determination process is a process in which valid frames are determined from the frames constituting the moving image stored in the camera storage unit 2212, and frames determined as valid frames are extracted from the moving image, while frames not identified as valid frames are discarded from the moving image. A valid frame is a frame that shows an image when at least one of the left door 11A, right door 11B, and drawers 12A, 13A, 14A, and 15A is in an open state.
[0119] Here, a number of methods for determining whether a frame is valid in the valid frame determination process will be described. <First Method> The first method is a determination based on open / close state information associated with a frame. In determining a valid frame, the food detection unit 221D determines a frame associated with open / close state information indicating an open state as a valid frame, and determines a frame associated with open / close state information indicating a closed state as not a valid frame.
[0120] <Second method> The second method is to make a judgment by extracting differences between images. The food detection unit 221D extracts the difference between frames constituting the moving image, thereby identifying frames that show an image when any of the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A is open. The food detection unit 221D then determines the identified frames as valid frames, and determines frames other than the identified frames as not valid frames. For example, the food detection unit 221D extracts the difference between frames in a closed state, and determines a frame in which there is a difference between the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A as a valid frame. The closed state frame that is the basis for the difference extraction is, for example, the oldest frame among the frames constituting the moving image.
[0121] <Third Method> The third method is a determination based on the first method and the second method. The food detection unit 221D determines a frame associated with open / closed state information indicating an open state as a valid frame. Next, the food detection unit 221D extracts differences from the closed state frames for frames associated with open / closed state information indicating a closed state, and determines a frame with a difference in any of the left door 11A, the right door 11B, and the drawers 12A, 13A, 14A, and 15A as a valid frame.
[0122] When the food detection unit 221D executes the valid frame determination process, it detects food that has been put into or taken out of the refrigerator 1 from the valid frame (step SB3). In the process of step SB3, if food is detected from the valid frame, the type of food and the storage room into or out of the detected food are also detected. The food detection unit 221D detects the type of food as a broader classification than the food name. For example, if the food that has actually been put into or taken out is milk, the food detection unit 221D detects the type of food as "packed beverage" in step SB3.
[0123] For example, the food detection unit 221D detects food that has been put into or taken out of the refrigerator 1 from the effective frame based on feature quantities such as shape, color, etc. Also, for example, the food detection unit 221D detects the type of food that has been detected based on feature quantities such as shape, color, etc. Also, for example, the food detection unit 221D detects the storage room into which the detected food has been put or taken out based on the type of camera, the size of the drawer 16A captured in the effective frame, etc. Data necessary for detection in step SB3 is stored in a storage area accessible by the camera module control unit 221.
[0124] The food detection unit 221D determines whether or not food that has been put into or taken out of the refrigerator 1 can be detected from the valid frames (step SB4).
[0125] If the food detection unit 221D determines that food entering or leaving the refrigerator 1 could not be detected from the valid frames (step SB4: NO), the process returns to step SB1 and executes the processes from step SB1 onwards again. Note that the food detection unit 221D may discard valid frames between the negative determination in step SB4 and the re-execution of step SB1. This makes it possible to prevent unnecessary videos in which no food was detected from being continuously stored in the camera storage unit 2212.
[0126] On the other hand, if the food detection unit 221D determines that food has been detected in or out of the refrigerator 1 (step SB4: YES), it performs an in / out determination process (step SB5).
[0127] The in / out determination process is a process for determining whether the food detected in step SB3 is food that has been put into the refrigerator 1 or food that has been taken out. For example, in the entry / exit determination process, food detection unit 221D determines the direction of movement in the video of the food image detected in step SB3 based on the valid frames. If the direction of movement of the food image is backward, i.e., toward refrigerator 1, food detection unit 221D determines that the detected food is food that has been entered from refrigerator 1. Also, if the direction of movement of the food image is forward, i.e., away from refrigerator 1, food detection unit 221D determines that the detected food is food that has been removed from refrigerator 1.
[0128] Next, the food detection unit 221D cuts out a food image FG of the detected food from the effective frame (step SB6). In the following description, the cut-out food image FG is referred to as a "cut-out food image".
[0129] The food detection unit 221D transmits food management request information, which requests the food management server 4 to manage the food, to the food management server 4 via the camera module communication unit 223 (step SB7).
[0130] The food management request information transmitted to the food management server 4 includes the food detection device ID 2215 stored in the camera storage unit 2212. Moreover, the food management request information transmitted to the food management server 4 includes date and time information indicating the date and time when the video was captured. In addition, the food management request information sent to the food management server 4 includes, for each food detected in step SB3, image data of the cut-out food image, in-stock / out-stock determination result information indicating the determination result of the in-stock / out-stock determination process, storage location information 4234 indicating the detected in-stock or out-stock destination, and food type information indicating the type of food detected.
[0131] Referring to the flowchart FC, the food recognition unit 4100 of the server control unit 40 of the food management server 4 determines whether or not food management request information has been received via the server communication unit 41 (step SC1).
[0132] If the food recognition unit 4100 determines that the food management request information has not been received (step SC1: NO), it executes the process of step SC1 again.
[0133] On the other hand, if the food recognition unit 4100 determines that food management request information has been received (step SC1: YES), it performs food recognition processing based on the cut-out food image indicated by the image data included in the food management request information (step SC2).
[0134] The food recognition process in this embodiment is a process of recognizing food from a cut-out food image, and acquiring a food image FG and food name information 4232 for the recognized food from the food information database.
[0135] The food information database is a database that stores information related to food, and for example, for each food, stores image data of the food image FG and food name information 4232. The food information database may be stored in the food management server 4, or may be stored in a device with which the food management server 4 can communicate.
[0136] For example, food recognition is performed as follows. The food recognition unit 4100 identifies a food image FG that is most similar to the cut-out food image from among the food images FG stored in the food information database. The similarity is calculated by a predetermined algorithm based on feature quantities such as shape and color. The food recognition unit 4100 then recognizes that the food represented by the identified food image FG is the food represented by the cut-out food image.
[0137] The food recognition unit 4100 may recognize food by AI (Artificial Intelligence). For example, a machine learning process is performed to learn feature quantities such as food color and shape from a teacher food image FG, and a trained model is constructed. The trained model is stored in the server storage unit 420. The food recognition unit 4100 uses the food image FG included in the food management request information as input data, refers to the trained model, and recognizes the food indicated by the food image FG. Here, the trained model may be stored in a device other than the food management server 4 connected to the global network GN, and the food may be recognized in the food management server 4. In this case, the food recognition unit 4100 transmits the food image included in the food management request information to a device that stores the trained model, and acquires the identification result of the food from the device. The food recognition unit 4100 then recognizes the food indicated by the acquired identification result as the food indicated by the cut-out food image.
[0138] When the food recognition unit 4100 recognizes a food, it acquires a food image FG and food name information 4232 for the recognized food from the food information database.
[0139] Next, the food recognition unit 4100 performs the food recognition process and updates the food management database 423 based on the results of the food recognition process (step SC3).
[0140] Here, step SC3 will be described in detail. First, the food recognition unit 4100 identifies, from the food management database 423, a record R having the food detection device ID 2125 included in the food management request information.
[0141] Next, the food recognition unit 4100 determines whether the recognized food is a food that has been stored in refrigerator 1 or a food that has been removed from refrigerator 1 based on the storage / receiving determination result information included in the received food management request information.
[0142] When the food recognition unit 4100 determines that the recognized food is food that has been removed from the refrigerator 1, it updates the information corresponding to the food name information 4232 of the recognized food in the identified record R. More specifically, the food recognition unit 4100 performs a process of reducing the remaining amount indicated by the remaining amount information 4233. The amount to be reduced is calculated, for example, during the food recognition process, based on the food image FG included in the food management request information. Here, if the updated remaining amount information 4233 indicates zero, the food recognition unit 4100 may blank out the storage start date indicated by the storage start date information 4235 and the expiration date indicated by the expiration date information 4236.
[0143] When the food recognition unit 4100 determines that the recognized food is a food stored in the refrigerator 1, for food that has been stored in the refrigerator 1 in the past, it updates the information corresponding to the food name information 4232 of the recognized food in the identified record R. More specifically, the food recognition unit 4100 performs a process of increasing the remaining amount indicated by the remaining amount information 4233. The amount to be increased is calculated, for example, during the food recognition process, based on a cut-out food image included in the food management request information. The food recognition unit 4100 also performs a process of changing the date indicated by the date and time information included in the food management request information to the storage start date indicated by the storage start date information 4235. The food recognition unit 4100 also performs a process of acquiring an expiration date based on the date indicated by the date and time information included in the food management request information by a predetermined method such as calculation or inquiry, and changing the expiration date indicated by the expiration date information 4236 to the acquired expiration date.
[0144] In addition, when the food recognition unit 4100 determines that the recognized food is food that has been stored in the refrigerator 1, for food that has not been stored in the past, new various information is associated with the user ID 4231 of the identified record R.
[0145] The various information associated with the user ID 4231 of the identified record R includes food name information 4232, remaining amount information 4233, storage location information 4234, storage start date information 4235, expiration date information 4236, expiration date notification timing information 4237, new notification timing information 4238, and food image data 4239. Here, the food name information 4232 is food name information 4232 acquired during food recognition processing. Furthermore, the remaining amount information 4233 indicates the remaining amount calculated based on the clipped food image included in the food management request information during the food recognition process, for example. Moreover, the storage location information 4234 is the storage location information 4234 included in the food management request information. Furthermore, the storage start date information 4235 indicates the date indicated by the date and time information included in the food management request information. Furthermore, the expiration date information 4236 indicates an expiration date based on the date indicated by the date and time information included in the food management request information, or indicates blank. Furthermore, deadline notification timing information 4237 and new notification timing information 4238 indicate default timing or are blank. Moreover, food image data 4239 indicates a food image FG acquired during food recognition processing.
[0146] As described above, in the food management system 1000, the food detection device 2 determines valid frames from the video images resulting from the shooting, detects food from the valid frames, and cuts out food images FG of the detected food from the valid frames. Then, in the food management system 1000, the food management server 4 recognizes the food from the cut-out food images. This division of roles allows the food management server 4 to recognize food while minimizing the amount of data that the food detection device 2 sends to the food management server 4. Therefore, in the food management system 1000, it is possible to recognize foods that have been put in and taken out of the refrigerator 1 while minimizing an increase in communication volume.
[0147] Next, a description will be given of the operation of the food management system 1000 when the application execution unit 3100 displays the food management screen SKG on the food management UI 3200. The food management screen SKG is a screen related to the management of food stored in the refrigerator 1.
[0148] First, before explaining the operation, the food management screen SKG will be explained. FIG. 10 is a diagram showing an example of the food management screen SKG.
[0149] The food management screen SKG includes an object OB related to each food. The food management screen SKG shown in Fig. 10 includes an object OB1 related to milk, an object OB2 related to beer, and an object OB3 related to eggs.
[0150] One object OB includes a food image FG, remaining amount information 4233, and storage start date information 4235. One object OB also includes expiration date warning information KJ that warns that the expiration date of the food is approaching if the expiration date of the food falls within the timing indicated by expiration date notification timing information 4237. One object OB also includes new storage indication information MJ that indicates that the food is newly stored if the storage start date of the food falls within the timing indicated by new notification timing information 4238.
[0151] An object OB1 shown in FIG. 10 includes a food image FG1 showing milk, remaining amount information 4233 showing the remaining amount of milk, storage start date information 4235 showing the storage start date of the milk, and expiration date warning information KJ. Moreover, the object OB2 shown in FIG. 10 includes a food image FG2 showing beer, remaining amount information 4233 showing the remaining amount of beer, storage start date information 4235 showing the storage start date of the beer, and new storage indication information MJ. Moreover, the object OB3 shown in FIG. 10 includes a food image FG3 showing an egg, remaining amount information 4233 showing the remaining amount of egg, and storage start date information 4235 showing the storage start date of the egg.
[0152] The food management screen SKG has a storage room switching button B1. The storage room switching button B1 is a software button for switching the type of object OB displayed on the food management screen SKG for each storage room of the refrigerator 1. When the storage room switching button B1 is touched, a list of the names of the storage rooms of the refrigerator 1 is displayed on the food management screen SKG. When a storage room is selected from the list by touching, the food management screen SKG displays the object OB for the storage location information 4234 indicating the selected storage room.
[0153] The food management screen SKG has a delete button B2. The delete button B2 is a software button for deleting an object OB from display on the food management screen SKG. On the food management screen SKG, when the delete button B2 is selected by a touch operation with one or more objects OB selected, the selected objects OB are hidden. Also, in the food management database 423, various information related to the food indicated by the object OB selected when the delete button B2 was operated is deleted.
[0154] The food management screen SKG has an update button B3 and update date and time information J2 indicating the most recent update date and time. The update button B3 is a software button for updating the objects OB displayed on the food management screen SKG and various information held by the objects OB. When the update button B3 is touched, the update date and time indicated by the update date and time information J2 changes to the date and time of the most recent update.
[0155] The operation of the food management system 1000 when the application execution unit 3100 displays the food management screen SKG on the food management UI 3200 will be described with reference to FIG.
[0156] Fig. 11 is a flowchart showing the operations of the terminal device 3 and the food management server 4. In Fig. 11, a flowchart FD shows the operations of the terminal device 3, and a flowchart FE shows the operations of the food management server 4.
[0157] The application execution unit 3100 of the terminal device 3 determines whether or not to request the record R from the food management server 4 (step SD1).
[0158] For example, when the application executing unit 3100 receives a touch operation instructing the food management U3200 to display the food management screen SKG, the application executing unit 3100 makes an affirmative determination in step SD1. Also, for example, when the update button B3 on the food management screen SKG is touched, the application executing unit 3100 makes an affirmative determination in step SD1.
[0159] When the application execution unit 3100 determines that record R is to be requested (step SD1: YES), the application execution unit 3100 transmits record request information requesting record R to the food management server 4 via the device communication unit 31 (step SD2). The record request information transmitted in step SD2 includes the user ID 4231 stored in the device storage unit 320.
[0160] Referring to flowchart FE, when the server control unit 40 of the food management server 4 receives record request information via the server communication unit 41 (step SE1), it identifies a record R containing the user ID 4231 included in the received record request information from the food management database 423 (step SE2).
[0161] Then, the server control unit 40 transmits the record R identified by the server communication unit 41 to the terminal device 3 (step SE3).
[0162] Referring to the flowchart FD, when the application execution unit 3100 of the terminal device 3 receives the record R via the terminal communication unit 31 (step SD3), it generates a food management screen SKG (step SD4).
[0163] Here, the process of step SD4 will be described in detail, taking as an example a case where record R shown in FIG. 6 is received and the food management screen SKG shown in FIG. 10 is generated.
[0164] The application execution unit 3100 determines whether the current date corresponds to the timing indicated by the expiration date notification timing information 4237 associated with the food name information 4232 indicating milk. In the case of generating the object OB1 in Fig. 10, the application execution unit 3100 determines the affirmative and generates the expiration date warning information KJ. Note that if the application execution unit 3100 determines the negative, the expiration date warning information KJ is not generated. Next, the application execution unit 3100 determines whether or not the storage start date indicated by the storage start date information 4235 associated with the food name information 4232 indicating milk corresponds to the timing indicated by the new notification timing information 4238 associated with the food name information 4232 indicating milk. In generating the object OB in Fig. 10, the application execution unit 3100 determines that the answer is no and does not generate new storage explicit information MJ. Note that if the application execution unit 3100 determines that the answer is yes, it generates new storage explicit information MJ. The application execution unit 3100 generates an object OB1 including the generated expiration date warning information KJ, the food image FG indicated by the food image data 4239 corresponding to the food name information 4232 indicating milk, and the storage start date information 4235 corresponding to the food name information 4232 indicating milk. If the application execution unit 3100 has not generated the deadline warning information KJ, it does not include the deadline warning information KJ in the object OB1, and if it has generated the new accommodation explicit information MJ, it includes the new accommodation explicit information MJ in the object OB1.
[0165] The application execution unit 3100 generates objects OB2 and OB3 in the same manner as object OB1, and includes the generated objects OB1, OB2, and OB3 in the food management screen SKG.
[0166] Note that the application execution unit 3100 generates an object OB related to onion, but since the food management screen SKG shown in Fig. 10 is a screen that displays objects OB stored in the refrigerator compartment 11, the object OB related to onion is not included in the food management screen SKG shown in Fig. 10. The object OB related to onion is included in the food management screen SKG that displays objects OB stored in the vegetable compartment 15.
[0167] Returning to the explanation of the flowchart FD shown in FIG. 11, when the application execution unit 3100 generates the food management screen SKG, it displays the generated food management screen SKG on the food management UI 3200 (step SD5).
[0168] In addition, in the above-mentioned food management screen SKG, the information displayed for each food can be modified by the user P. The modifiable information is various information associated with the user ID 4231 in the food management database 423. When information is modified in the food management screen SKG, the application execution unit 3100 transmits update request information requesting an update of the food management database 423 to the food management server 4. Then, the food management server 4 updates the food management database 423 based on the received update request information. For example, when the remaining amount information 4233 of beer is modified from 3 bottles to 2 bottles, the application execution unit 3200 transmits update request information to the food management server 4 for updating the remaining amount information 4233 associated with the food name information 4232 of beer to remaining amount information 4233 indicating a remaining amount of 2 bottles. Then, the food management server 4 updates the remaining amount indicated by the remaining amount information 4233 from 3 bottles to 2 bottles. In this way, since the user P can correct the information, the food management system 1000 can manage foods appropriately even if there is a false detection by the food detection device 2 or a false recognition of the food by the food management server 4. When the food recognition unit 4100 recognizes foods using AI, the food recognition unit 4100 may perform machine learning of the content corrected by the user P and reflect the corrected content in the learned model.
[0169] [1-3. Effects, etc.] As described above, food detection device 2 is attached to the top of refrigerator 1 by user P, and detects food entering or leaving refrigerator 1 using refrigerator compartment camera 21 and drawer camera 22. Food detection device 2 includes camera module 220 having refrigerator compartment camera 21 that takes an image of refrigerator 1 from above the front of refrigerator 1, and drawer camera 22 that takes an image of refrigerator 1 from above the front of refrigerator 1. Food detection device 2 also includes power supply unit 230 that supplies power to at least camera module 220.
[0170] According to this configuration, food entering or leaving refrigerator 1 can be photographed from above the food, so that the food entering or leaving refrigerator 1 can be accurately included in the photographed result. Therefore, food detection device 2 can accurately detect food entering or leaving refrigerator 1 by using the camera.
[0171] The power supply unit 230 supplies power to at least the camera module 220 based on power supplied from any one of the commercial AC power supply 7, the refrigerator 1, and a battery provided in the food detection device 2.
[0172] According to this configuration, food detection device 2 can independently take images using refrigerator compartment camera 21 and drawer camera 22. Therefore, even if food detection device 2 is configured separately from refrigerator 1, food entering and leaving refrigerator 1 can be accurately detected by the cameras.
[0173] Furthermore, food detection device 2 is provided on top of refrigerator 1, and detects food entering or leaving refrigerator 1 using refrigerator compartment camera 21 and drawer camera 22. Food detection device 2 includes camera module 220 having refrigerator compartment camera 21 and drawer camera 22 that photograph refrigerator 1 from the front above refrigerator 1, and power supply unit 230 that supplies power to at least camera module 220. A power cable of food detection device 2 is connected to commercial AC power source 7. Power supply unit 230 supplies power to at least camera module 210 based on the power supplied via this power cable.
[0174] According to this configuration, food detection device 2 can stand alone and take images using refrigerator compartment camera 21 and drawer camera 22. Therefore, even if food detection device 2 is configured separately from refrigerator 1, food entering or leaving refrigerator 1 can be accurately detected by the cameras when power is supplied from commercial AC power source 7.
[0175] Food detection device 2 is provided on top of refrigerator 1, and detects food entering or leaving refrigerator 1 using refrigerator compartment camera 21 and drawer camera 22. Food detection device 2 includes camera module 220 having refrigerator compartment camera 21 and drawer camera 22 that photograph refrigerator 1 from above the front of refrigerator 1, and power supply unit 230 that supplies power to at least camera module 220. A USB cable of food detection device 2 is connected to refrigerator 1. Power supply unit 230 supplies power to at least camera module 220 based on power supplied via this USB cable.
[0176] According to this configuration, food detection device 2 can take images by itself using refrigerator compartment camera 21 and drawer camera 22. Therefore, even if food detection device 2 is configured separately from refrigerator 1, food entering or leaving refrigerator 1 can be accurately detected by the cameras by power supply according to the USB standard.
[0177] Food detection device 2 is provided on top of refrigerator 1, and detects food entering or leaving refrigerator 1 using refrigerator compartment camera 21 and drawer camera 22. Food detection device 2 includes camera module 220 having refrigerator compartment camera 21 and drawer camera 22 that photograph refrigerator 1 from above the front of refrigerator 1, power supply unit 230 that supplies power to at least camera module 220, and a battery. Power supply unit 230 supplies power to at least camera module 220 based on power supplied from the battery.
[0178] According to this configuration, food detection device 2 can take pictures by itself using refrigerator compartment camera 21 and drawer camera 22. Therefore, even if food detection device 2 is configured separately from refrigerator 1, food entering or leaving refrigerator 1 can be accurately detected by the cameras using power supplied from the provided battery.
[0179] The food detection device 2 includes a sensor module 210 having an opening / closing distance measuring sensor 28 that detects the open / closed state of at least one of the door 11C and the drawer 16A of the refrigerator 1. The refrigerator compartment camera 21 and the drawer camera 22 start capturing images when the opening / closing distance measuring sensor 28 detects the open state, and stop capturing images when the distance measuring sensor detects the closed state.
[0180] This configuration allows food items entering and leaving refrigerator 1 to be included in the photographed results with greater accuracy, while preventing unnecessary photographing. Therefore, food items entering and leaving refrigerator 1 can be detected with greater accuracy by the camera, while preventing unnecessary photographing.
[0181] The sensor module 210 has an illuminance sensor 26 that detects the illuminance of the place where the refrigerator 1 is installed, and a human presence sensor 27 that detects the presence or absence of a person around the refrigerator 1. If the detection result of at least one of the illuminance sensor 26 and the human presence sensor 27 indicates the presence of a person around the refrigerator 1, the refrigerator compartment camera 21 and the drawer camera 22 start capturing images even if the opening / closing distance measuring sensor 28 detects the closed state.
[0182] According to this configuration, since image capture begins before the user P opens either the door 11C or the drawer 16A of the refrigerator 1, the camera can detect foods being put into or taken out of the refrigerator 1 with greater accuracy.
[0183] The refrigerator compartment camera 21 and the drawer camera 22 capture moving images.
[0184] With this configuration, food entering and leaving refrigerator 1 can be accurately included in the photographed result regardless of the speed of the food entering and leaving refrigerator 1 or the angle of the hand entering and leaving refrigerator 1. Therefore, food entering and leaving refrigerator 1 can be detected with high accuracy by the camera.
[0185] (Embodiment 2) The second embodiment is different from the first embodiment in the number of cameras included in camera module 220. Camera module 220 in the first embodiment is configured to include two cameras, refrigerator compartment camera 21 and drawer camera 22. Camera module 220 in the second embodiment includes one camera. Hereinafter, this camera will be referred to as a refrigerator photography camera.
[0186] In the second embodiment, the food detection device 2 is provided on the top surface of the refrigerator 1 so that the refrigerator photographing camera photographs the front surface of the refrigerator 1 from above and in front of the refrigerator 1. The refrigerator photographing camera captures moving images.
[0187] The photographing range of the refrigerator photographing camera is set to a range that allows photographing of food entering and leaving refrigerator 1 from above the front of refrigerator 1.
[0188] For example, the shooting range of the refrigerator photographing camera is a range that includes the shooting range of refrigerator compartment camera 21 and the shooting range of drawer camera 22. In this case, the shooting range of the refrigerator photographing camera is a range that includes ranges A1 and A2 shown in Fig. 2 when viewed from the front, and includes ranges A3 and A4 shown in Fig. 3 when viewed from the side.
[0189] In embodiment 2, the food detection device 2, terminal device 3, and food management server 4 perform operations similar to those of embodiment 1 described above based on the photographing results of the refrigerator photography camera instead of the photographing results of the refrigerator compartment camera 21 and the drawer camera 22.
[0190] According to the second embodiment, the same effects as those of the first embodiment are achieved.
[0191] (Other embodiments) As described above, the above-mentioned embodiment has been described as an example disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to an embodiment in which modifications, substitutions, additions, omissions, etc. are made. In addition, it is also possible to combine the components described in the above-mentioned embodiment to create a new embodiment. Therefore, other embodiments will be exemplified below.
[0192] In the embodiment described above, the shooting range of drawer camera 22 includes ice making compartment 12, fresh freezing compartment 13, freezer compartment 14, and vegetable compartment 15. However, the shooting range of drawer camera 22 is not limited to this range, and may be a range that includes one or a combination of any two or more storage compartments. For example, the shooting range of drawer camera 22 may be a range that includes only freezer compartment 14 and vegetable compartment 1, or may be a range that includes only vegetable compartment 15.
[0193] In the above embodiment, the sensor module 210 and the camera module 220 are a separate pair, but they may be configured as an integrated unit.
[0194] For example, in the above-described embodiment, a distance measuring sensor that measures distance was used as an example of the opening / closing sensor of the present invention, but the opening / closing sensor of the present invention may be any sensor that is capable of detecting the open / closed state of the door 11C and the drawer 16A, and may be, for example, a camera.
[0195] For example, in the above embodiment, the food detection device 2 is illustrated as a device separate from the refrigerator 1, but the food detection device 2 may be incorporated in the refrigerator 1 and provided in the refrigerator 1.
[0196] For example, in the embodiment described above, record R is stored in food management database 423 and food detected by food detection device 2 is managed by food management server 4, but record R may be stored in device storage unit 320 and food management app 323 may manage various information related to foods detected by food detection device 2. In this configuration, app execution unit 3100 functions as food recognition unit 4100 to perform food recognition and also appropriately updates the contents of record R stored in device storage unit 320 in the same manner as food management server 4 described above. In this configuration, food management system 1000 does not need to include food management server 4, and the system configuration is simplified.
[0197] For example, the types of compartments formed in the main box 10 of the refrigerator 1 are not limited to the refrigeration compartment 11, the ice-making compartment 12, the fresh-freezing compartment 13, the freezer compartment 14, and the vegetable compartment 15, and there may be fewer compartments or other types of compartments. Also, the number of doors provided at the opening on the front of the refrigeration compartment 11 may be one.
[0198] For example, the functions of the sensor module control unit 211, the camera module control unit 221, the terminal control unit 30, and the server control unit 40 may be realized by multiple processors or semiconductor chips.
[0199] Each unit shown in Figures 4 and 5 is an example, and the specific implementation form is not particularly limited. In other words, it is not necessarily necessary to implement hardware corresponding to each unit individually, and it is of course possible to configure the functions of each unit by having one processor execute a program. Also, some of the functions realized by software in the above-mentioned embodiment may be hardware, or some of the functions realized by hardware may be software. In addition, the specific detailed configurations of other units of the food detection device 2, the terminal device 3, and the food management server 4 may also be changed arbitrarily without departing from the spirit of the present invention.
[0200] Also, for example, the step units of the operation shown in Figures 8, 9, and 11 are divided according to the main processing content in order to facilitate understanding of the operation of each part of the food control system 1000, and the present invention is not limited by the manner in which the processing units are divided or the names thereof. The processing units may be further divided into more step units according to the processing content. Furthermore, one step unit may be further divided to include more processing. Furthermore, the order of the steps may be appropriately changed within a range that does not impede the spirit of the present invention.
[0201] It should be noted that the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, substitutions, additions, omissions, and the like can be made within the scope of the claims or their equivalents. [Industrial Applicability]
[0202] As described above, the food detection device according to the present invention can be used to detect food entering and leaving a refrigerator. [Explanation of symbols]
[0203] 1. Refrigerator 2. Food detection equipment 3 Terminal Equipment 4. Food Management Server 7 Commercial AC power supply (commercial power supply) 11 Refrigerator 11A Left door (door) 11B Right door (door) 11C Door 12A, 13A, 14A, 15A, 16A drawer 21 Refrigerator Camera (Camera) 22 Drawer camera (camera) 23 Distance sensor for left door (opening / closing sensor) 24 Right door distance sensor (opening / closing sensor) 25 Distance sensor for drawer (opening / closing sensor) 26 Illuminance sensor 27 Human Sensor 28 Opening / closing distance sensor (opening / closing sensor) 210 Sensor Module 220 Camera Module 221A Detection value processing section 221B Shooting control unit 221C Video processing unit 221D Food Detection Unit 230 Power supply section 4100 Food Recognition Department
Claims
1. A food detection device that is attached to the top of a refrigerator by a user and detects food entering or leaving the refrigerator using a camera, a camera module having the camera for photographing the refrigerator from above the front of the refrigerator; A power supply unit that supplies power to at least the camera module. Food detection equipment.
2. the power supply unit supplies power to at least the camera module based on power supplied from any one of a commercial power source, the refrigerator, and a battery included in the food detection device; The food detection device of claim 1 .
3. A food detection device that is installed on an upper portion of a refrigerator and detects food entering or leaving the refrigerator using a camera, a camera module having the camera for photographing the refrigerator from above the front of the refrigerator; a power supply unit that supplies power to at least the camera module; A power cable of the food detection device is connected to a commercial power source; The power supply unit supplies power to at least the camera module based on power supplied via the power cable. Food detection equipment.
4. A food detection device that is installed on an upper portion of a refrigerator and detects food entering or leaving the refrigerator using a camera, a camera module having the camera for photographing the refrigerator from above the front of the refrigerator; a power supply unit that supplies power to at least the camera module; a USB cable of the food detection device is connected to the refrigerator; The power supply unit supplies power to at least the camera module based on power supplied via the USB cable. Food detection equipment.
5. A food detection device that is installed on an upper portion of a refrigerator and detects food entering or leaving the refrigerator using a camera, a camera module having the camera for photographing the refrigerator from above the front of the refrigerator; a power supply unit for supplying power to at least the camera module; A battery; The power supply unit supplies power to at least the camera module based on the power supplied from the battery. Food detection equipment.
6. a sensor module having an open / close sensor for detecting an open / close state of at least one of the door and the drawer of the refrigerator; The camera starts capturing an image when the open / close sensor detects an open state, and stops capturing an image when the open / close sensor detects a closed state. A food detection device according to any one of claims 1 to 5.
7. the sensor module includes an illuminance sensor that detects illuminance in a place where the refrigerator is installed, and a human presence sensor that detects a person present around the refrigerator, When a detection result of at least one of the illuminance sensor and the human presence sensor indicates the presence of a person around the refrigerator, the camera starts capturing an image even if the opening / closing sensor detects a closed state.
7. The food detection device of claim 6.
8. The camera captures video images. A food detection device according to any one of claims 1 to 7.
Citation Information
Patent Citations
Food material identification method and refrigerator
CN111444880A
Refrigerator
JP2003042626A
Refrigerator
JP2004183987A
Refrigerator
JP2007017020A
Refrigerator
JP2014199173A
