refrigerator

The refrigerator system addresses incomplete food recognition by using sensors and cameras to capture multiple images before and after door closure, ensuring comprehensive food information acquisition through precise timing and image processing.

JP7584035B2Active Publication Date: 2024-11-15PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023081987
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-05-18
Publication Date
2024-11-15
Estimated Expiration
2037-10-06

AI Technical Summary

Technical Problem

Conventional refrigerators face issues in reliably acquiring information on stored foods due to overlapping foods in the imaging direction of internal cameras, leading to incomplete food recognition.

Method used

A refrigerator system with an opening/closing sensor and camera that captures multiple images before and after the door is closed, storing images at specific times to ensure complete capture of food entry and exit, using illuminance sensors to adjust imaging times based on light changes, and processing images to extract food information.

Benefits of technology

This approach allows for more reliable acquisition of food information, including name, expiration date, and quantity, by ensuring comprehensive imaging and analysis of food items entering and leaving the storage chamber.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007584035000001
    Figure 0007584035000001
  • Figure 0007584035000002
    Figure 0007584035000002
  • Figure 0007584035000003
    Figure 0007584035000003
Patent Text Reader

Abstract

To provide a refrigerator which can acquire information of foods stored in a storage room in a more reliable manner.SOLUTION: A refrigerator 1 includes a food information acquisition part 114 which causes a camera 60 to capture images multiple times in a time period from when opening / closing sensors 18a, 18b detect doors 11, 12 being switched from a closed state to an open state to when the opening / closing sensors 18a, 18b detect the doors 11, 12 being switched from the open state to the closed state and stores the captured images in a storage part 130 at a first imaging moment when a first predetermined time elapsed from a moment when the opening sensors 18a, 18b detect the doors 11, 12 being switched from the closed state to the open state.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a refrigerator. [Background technology]

[0002] Conventionally, a refrigerator has been proposed in which an interior camera is installed in the center of a food storage compartment, the interior camera captures images of the interior of the storage compartment when the door of the storage compartment is opened and closed, and an image of when the food was stored and a current image are output side by side from the captured images in chronological order (see, for example, Patent Document 1). In the refrigerator described in Patent Document 1, depending on the arrangement of the food in the storage compartment, the food may overlap in the direction of the image capture by the interior camera, making it impossible to recognize some of the food from the captured image. In response to this, a refrigerator has been proposed that is equipped with a switch that detects the opening and closing of the storage compartment door and a camera whose imaging range is the opening of the storage compartment, so that when the door is opened, an image is captured of food being stored in or removed through the opening, rather than of food stored in the storage compartment (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-57022 A [Patent Document 2] JP 2015-81762 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to provide a refrigerator that can more reliably obtain information about food stored in a storage compartment. [Means for solving the problem]

[0005] A refrigerator according to a first aspect of the present invention includes a storage compartment having an opening, a door provided in the storage compartment, an opening / closing sensor for detecting an open / closed state of the door, a camera for capturing an image of an image capturing range including the opening, and a memory unit, and the camera captures an image multiple times from when the opening / closing sensor detects that the door has switched from a closed state to an open state until the opening / closing sensor detects that the door has switched from an open state to a closed state, and captures an image of the door from when the opening / closing sensor detects that the door has switched from a closed state to an open state. The time when the food to be stored in the storage chamber passes through the opening is assumed. During the period from the first image capturing time point when a first predetermined time has elapsed until the opening / closing sensor detects that the door has been switched from an open state to a closed state, the camera repeatedly captures images and stores a plurality of captured images as leaving-time image candidates in the storage unit together with image capturing times. When the opening / closing sensor detects that the door has been switched from an open state to a closed state, among the plurality of leaving-time image candidates, a candidate image that is closer to the opening / closing state than the opening / closing sensor detects that the door has been switched from an open state to a closed state is selected. The time when the food to be taken out of the storage room passes through the opening is assumed. The food information acquisition unit stores the candidate leaving-store image captured at a second imaging time point before a second predetermined time in the storage unit as a leaving-store image. A refrigerator according to a second aspect of the present invention includes a storage compartment having an opening, a door provided in the storage compartment, an opening / closing sensor for detecting an open / closed state of the door, a camera for capturing an image of an image capturing range including the opening, and a memory unit, and the camera captures an image multiple times from when the opening / closing sensor detects that the door has switched from a closed state to an open state until the opening / closing sensor detects that the door has switched from an open state to a closed state, and captures an image of the door from when the opening / closing sensor detects that the door has switched from a closed state to an open state. The time when the food to be stored in the storage chamber passes through the opening is assumed. The image captured at the first imaging time point when the first predetermined time has elapsed is stored in the storage unit as an entrance image, and the image captured at the first imaging time point when the opening / closing sensor detects that the door has switched from an open state to a closed state is stored in the storage unit. The time when the food to be taken out of the storage room passes through the opening is assumed.The food storage device is equipped with a food information acquisition unit that stores an image captured at a second imaging point in time before a second predetermined time in the memory unit as an image at the time of leaving the storage unit, and acquires information on food being stored in the storage room and food being removed from the storage room based on the image at the time of entering the storage room and the image at the time of leaving the storage room. Effect of the Invention

[0006] According to the present invention, information on food stored in a storage room can be obtained more reliably. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram of a configuration for checking the status of food stored in a refrigerator installed in a house using a terminal device. [Diagram 2] Diagram of the refrigerator's appearance [Diagram 3] An explanatory diagram showing the internal structure of a refrigerator in a cross-sectional view from the right side. [Figure 4] An explanatory diagram showing the internal structure of the refrigerator compartment as a cross-sectional view from the front. [Diagram 5] Control unit configuration diagram [Figure 6] 1 is a flowchart showing the overall processing by the control unit. [Figure 7] Flowchart of imaging process [Figure 8] Flowchart of weight information processing [Figure 9] Flowchart of analysis process of images taken at the time of storage [Figure 10] Flowchart of analysis process of images at the time of leaving the warehouse [Figure 11] Flowchart of weight sensor correction process [Figure 12] First flow chart of food management process in warehouse [Figure 13] Second flow chart of food management process in warehouse [Figure 14] List of food in the warehouse and list of expiration dates [Figure 15] 11 is a flowchart of another embodiment of the imaging process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The first invention comprises a storage chamber having an opening, an opening / closing unit which opens and closes the opening, an opening / closing sensor which detects the open / closed state of the opening / closing unit, a camera which captures an image of an imaging range including the opening, a memory unit, and a food information acquisition unit which captures an image by the camera at a first imaging time point which is set to assume a time point when food to be stored in the storage chamber will pass through the opening between when the opening / closing sensor detects that the opening / closing unit has switched from a closed state to an open state and when the opening / closing sensor detects that the opening / closing unit has switched from an open state to a closed state, and stores the image captured at the first imaging time point in the memory unit as an entry image, and which captures an image by the camera at a second imaging time point which is set to assume a time point when food to be removed from the storage chamber will pass through the opening, and stores the image captured at the second imaging time point in the memory unit as an exit image, and acquires information about the food to be stored in the storage chamber and the food to be removed from the storage chamber based on the entry image and the exit image. According to the first aspect of the present invention, the food information acquisition unit stores in the storage unit an entry image captured at a first imaging time point when it is assumed that the food being entered into the storage unit will pass through the opening of the storage unit, and an exit image captured at a second imaging time point when it is assumed that the food being exited from the storage unit will pass through the opening. The food information acquisition unit can then more reliably acquire information on the food being entered into the storage unit and the food being exited from the storage unit based on the entry image and the exit image that are likely to include image portions of the food.

[0009] A second invention includes an illuminance sensor that detects the illuminance within the storage chamber, and an imaging time point setting unit that sets a first reference illuminance and a second reference illuminance based on the detected illuminance of the illuminance sensor at the time point when the opening / closing sensor detects that the opening / closing section has switched from a closed state to an open state, sets the first imaging time point based on the time point when the detected illuminance of the illuminance sensor becomes equal to or lower than the first reference illuminance, and sets the second imaging time point based on the time point when the detected illuminance of the illuminance sensor becomes equal to or higher than the second reference illuminance after becoming equal to or lower than the first reference illuminance. According to the second aspect of the present invention, the illuminance in the storage chamber changes depending on the presence of an object (food, a user's hand, etc.) in the storage chamber. Therefore, based on the change in illuminance in the storage chamber detected by the illuminance sensor, it is possible to appropriately set a first imaging time point that assumes the time when the food being stored in the storage chamber passes through the opening, and a second imaging time point that assumes the time when the food being removed from the storage chamber passes through the opening.

[0010] The third invention includes an imaging time point setting unit that sets the first imaging time point to a time point when a first predetermined time has elapsed since the opening / closing sensor detected that the opening / closing section has switched from a closed state to an open state, and sets the second imaging time point to a time point that is a second predetermined time before the opening / closing sensor detected that the opening / closing section has switched from an open state to a closed state. According to the third aspect of the present invention, the time from when a user opens the opening / closing part of the storage compartment to when the food is stored and the time from when a user removes the food from the storage compartment to when the opening / closing part is closed can be expected to be relatively short and constant in order to maintain the cold state inside the storage compartment. Therefore, the first and second image capturing times can be set by a simple configuration of setting the first and second predetermined times.

[0011] In a fourth invention, the food information acquisition unit repeatedly captures images using the camera and stores the captured images in the memory unit from the time when the first imaging time has passed until the opening / closing sensor detects that the opening / closing section has switched from an open state to a closed state, and when the opening / closing sensor detects that the opening / closing section has switched from an open state to a closed state, saves the captured image captured at a time point the second predetermined time before the time when the opening / closing sensor detected that the opening / closing section had switched from an open state to a closed state and stored in the memory unit as the exit image. According to the fourth aspect of the present invention, the food information acquisition unit repeatedly captures images using the camera and stores the captured images in the storage unit from the time when the first image capturing time point has passed until the opening / closing sensor detects that the opening / closing part has switched from the open state to the closed state. This allows the image captured at the time when the second predetermined time before the time when the opening / closing sensor detects that the opening / closing part has switched from the open state to the closed state to be stored as the image at the time of leaving the warehouse.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiment.

[0013] [1. How the refrigerator is used] Fig. 1 shows a usage state of a refrigerator of this embodiment. Refrigerator 1 of this embodiment is installed in house H, and has a function of communicating with a management server 520 via a gateway 500 and a communication network 510. Refrigerator 1 also has a function of communicating with a terminal device (smartphone, tablet terminal, etc.) 400. Terminal device 400 has a function of communicating with management server 520 via the gateway 500 and the communication network 510.

[0014] A control unit 100 provided in the refrigerator 1 controls the overall operation of the refrigerator 1, and generates an internal food list 131 showing information about the foods stored in the storage compartments of the refrigerator 1 and sends it to the management server 520. A refrigerator management application executed on the terminal device 400 acquires internal food list 131 data from the management server 520, or from the refrigerator 1, and displays information about the foods stored in the refrigerator 1 based on the internal food list 131 on the display unit. A user P of the refrigerator 1 can check the status of the foods stored in the refrigerator 1 (food names, expiration dates, remaining amount, etc.) by visually checking the display on the terminal device 400.

[0015] [2. Refrigerator configuration] The configuration of refrigerator 1 will be described with reference to Fig. 2 to Fig. 5. Fig. 2 is an explanatory diagram showing the external appearance of refrigerator 1. As shown in Fig. 2, refrigerator 1 has a main box body 2 with an opening on the front, and inside main box body 2, there are formed a refrigeration compartment 10 (corresponding to a storage compartment of the present invention), an ice-making compartment 30, a switching compartment 20 adjacent to ice-making compartment 30 that can change the temperature inside, a freezer compartment 40, and a vegetable compartment 50.

[0016] A rotating right door 11 and a left door 12 (corresponding to the opening and closing part of the present invention) are provided at the opening on the front side of the refrigerator compartment 10. In addition, the switchable compartment 20, the ice-making compartment 30, the freezer compartment 40, and the vegetable compartment 50 are provided with drawers 21, 31, 41, and 51, respectively, for storing food.

[0017] Next, Fig. 3 is an explanatory diagram showing the configuration inside the refrigerator 1 in a cross-sectional view seen from the right side, and Fig. 4 is an explanatory diagram showing the internal configuration of the refrigerating chamber 10 in a cross-sectional view seen from the front. With reference to Figs. 3 and 4, the refrigerating chamber 10 is provided with an upper shelf 13, a middle shelf 14, and a lower shelf 15 which divide the refrigerating chamber 10 into sub-areas. The refrigerating chamber 10 is also provided with weight sensors 13a and 13b which detect the weight of food placed on the upper shelf 13, weight sensors 14a and 14b which detect the weight of food placed on the middle shelf 14, and weight sensors 15a and 15b which detect the weight of food placed on the lower shelf 15. The refrigerating chamber 10 is also provided with lighting units 16a to 16f which illuminate the interior of the refrigerating chamber 10, illuminance sensors 17a to 17c which detect the illuminance inside the refrigerating chamber 10, and a camera 60.

[0018] The camera 60 is disposed near the opening 10a at the top of the refrigerator compartment 10 so that the opening 10a is included in the imaging range, and images the range from the opening 10a to the vicinity of the front ends of the shelves 13, 14, and 15. In addition, a compressor 61, a cooling fan 62, a cooler 63, and a condenser 64, which are auxiliary machines that constitute a refrigeration cycle, are provided. Furthermore, a refrigerator compartment duct 70 and cold air outlets 71 to 73 for circulating cold air are disposed at the rear of the refrigerator compartment 10.

[0019] The refrigerator compartment 10 is provided with an opening / closing sensor 18b for detecting the opening / closing of the right door 11, and an opening / closing sensor 18a for detecting the opening / closing of the left door 12. The opening / closing sensor 18b outputs a closed detection signal when the right door 11 is closed, and outputs an open detection signal when the right door 11 is open. Similarly, the opening / closing sensor 18a outputs a closed detection signal when the left door 12 is closed, and outputs an open detection signal when the left door 12 is open.

[0020] An opening / closing sensor 22 that detects whether the opening 20a is open or closed is provided in the switching chamber 20. The opening / closing sensor 22 outputs a closed detection signal when the drawer 21 is accommodated in the switching chamber 20, and outputs an open detection signal when the drawer 21 is pulled out of the switching chamber 20.

[0021] Similarly, ice making compartment 30 is provided with opening / closing sensor 32 which detects opening / closing of opening 30a by drawer 31. Freezer compartment 40 is provided with opening / closing sensor 42 which detects opening / closing of opening 40a by drawer 41, and vegetable compartment 50 is provided with opening / closing sensor 52 which detects opening / closing of opening 50a by drawer 51. Similar to opening / closing sensor 22, opening / closing sensors 32, 42, 52 output a closed detection signal and an open detection signal.

[0022] Next, Fig. 5 is a configuration diagram of the control unit 100. Referring to Fig. 5, the control unit 100 is an electronic circuit unit configured with a CPU (Central Processing Unit) 110, an image processing unit 120, a storage unit 130, a wireless communication unit 140, a date and time recognition unit 141, an interface circuit (not shown), and the like.

[0023] The CPU 110 executes the control program for the refrigerator 1 stored in the memory unit 130, and thereby functions as a passing object detection unit 111, a storage capacity recognition unit 112, an entry / exit determination unit 113, a food information acquisition unit 114, a food information management unit 115, and an image capture time setting unit 116.

[0024] The control unit 100 is connected to opening and closing sensors 18a, 18b, 22, 32, 42, 52, illuminance sensors 17a, 17b, 17c, weight sensors 13a, 13b, 14a, 14b, 15a, 15b, camera 60, lighting units 16a, 16b, 16c, 16d, 16e, 16f, and refrigeration cycle accessories 61, 62, 63, 64.

[0025] Detection signals from the opening / closing sensors 18a, 18b, 22, 32, 42, 52, the illuminance sensors 17a, 17b, 17c, and the weight sensors 13a, 13b, 14a, 14b, 15a, 15b, etc. are input to the control unit 100. In addition, the operation of the lighting units 16a, 16b, 16c, 16d, 16e, 16f, and the refrigeration cycle auxiliary units 61, 62, 63, 64, etc. are controlled by control signals output from the control unit 100.

[0026] Furthermore, the control signal output from the control unit 100 controls the imaging by the camera 60 via the image processing unit 120. The image processing unit 120 converts the video signal output from the camera 60 into a digital gradation signal to generate a captured image, and stores the captured image data in an image memory 121. The image processing unit 120 processes the captured image stored in the image memory 121 in accordance with the control signal output from the CPU 110.

[0027] The date and time recognition unit 141 recognizes the current date and time (date and time) by measuring a clock signal output from a clock circuit (not shown) provided in the control unit 100. Note that the current date and time may be recognized by acquiring information on the current date and time from the management server 520 through communication with the management server 520.

[0028] In addition to the data of in-fridge food list 131, memory unit 130 also stores data of received food information 132, which records information on foods received in refrigerator compartment 10, and sent food information 133, which records information on foods sent out of refrigerator compartment 10. Memory unit 130 also stores data of sample images 134 of various foods, and a best-before date list 135, which shows general best-before dates for various foods.

[0029] The passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10 based on changes in illuminance inside the refrigerator compartment 10 detected by the illuminance sensors 17a, 17b, and 17c. The storage amount recognition unit 112 recognizes the amount of food stored on the upper shelf 13, middle shelf 14, and lower shelf 15 of the refrigerator compartment 10 by weight based on detection signals from the weight sensors 13a, 13b, 14a, 14b, 15a, and 15b.

[0030] Storage amount recognition unit 112 recognizes the weight of food stored on upper shelf 13 based on detection signals from weight sensors 13a and 13b, and recognizes the weight of food stored on middle shelf 14 based on detection signals from weight sensors 14a and 14b. In addition, storage amount recognition unit 112 detects the weight of food stored on lower shelf 15 based on detection signals from weight sensors 15a and 15b.

[0031] When the passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10, the entry / exit determination unit 113 determines whether food has been entered into or removed from the refrigerator compartment 10 based on the change in the amount (weight) of food placed on each shelf 13, 14, 15 recognized by the storage capacity recognition unit 112.

[0032] The food information acquisition unit 114 captures an image of an object that passes through the opening 10a of the refrigerator compartment 10 using the camera 60. Then, when the entry / exit determination unit 113 determines that a food has been entered into or removed from the refrigerator compartment 10, the food information acquisition unit 114 extracts an image portion of the food from the captured image.

[0033] The food information acquisition unit 114 acquires information such as the name, content, place of origin, and expiration date of the food that has been stored or shipped by recognizing characters included in the image portion of the food or by extracting features of the image portion of the food. Then, the food information acquisition unit 114 records the acquired information in stored food information 132 for foods determined to be stored (stored foods), and in shipped food information 133 for foods determined to be shipped (shipped foods).

[0034] The food information management unit 115 updates the food information recorded in the in-fridge food list 131 based on the weight of the food in the refrigerator compartment 10 recognized by the storage capacity recognition unit 112, the incoming food information 132 and the outgoing food information 133 recorded by the food information acquisition unit 114, etc.

[0035] The image capturing time setting unit 116 sets the time when the camera 60 captures an image from the time when the right door 11 or the left door 12 is switched from a closed state to an open state by the opening / closing sensors 18a, 18b until it is detected that both the right door 11 and the left door 12 have become closed. The image capturing time setting unit 116 sets a first image capturing time assuming the time when food to be put into the refrigerator compartment 10 passes through the opening 10a, and a second image capturing time assuming the time when food removed from the refrigerator compartment 10 passes through the opening 10a.

[0036] [3. Processing by the control unit] Next, the process for managing food stored in the refrigerator compartment 10, which is executed by the control unit 100, will be described with reference to Figs.

[0037] 6 is a flow chart showing an overview of a series of processes executed by the control unit 100. The control unit 100 executes an "imaging process" in step S1, a "weight information process" in step S2, an "image analysis process" in step S3, a "weight sensor correction process" in step S4, and an "in-storage food management process" in step S5. Each process will be described in detail below.

[0038] [3-1. Imaging processing] The execution procedure of the "imaging process" will be described with reference to the flowchart shown in Fig. 7. The "imaging process" is executed by the passing object detection unit 111, the food information acquisition unit 114, and the imaging time point setting unit 116. The food information acquisition unit 114 captures images with the camera 60 at a first imaging time point and a second imaging time point from when it recognizes in step S10 that one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 have switched from a closed state to an open state based on the detection signals of the opening / closing sensors 18a, 18b, until it recognizes in step S15 that both the right door 11 and the left door 12 have become closed.

[0039] When the food information acquisition unit 114 recognizes in step S10 that one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 have been switched from a closed state to an open state, the food information acquisition unit 114 advances the process to step S10. Steps S11 to S14 are processes performed by the passing object detection unit 111 and the imaging time point setting unit 116. In this embodiment, the imaging time point setting unit of the present invention is configured by a combination of the passing object detection unit 111 and the imaging time point setting unit 116.

[0040] In step S11, the passing object detection unit 111 detects the illuminance of each of the upper shelf 13, middle shelf 14, and lower shelf 15 of the refrigerator compartment 10 (the illuminance of each space partitioned by the shelves) using the illuminance sensors 17a, 17b, and 17c. Then, the passing object detection unit 111 sets the detected illuminance of the upper shelf 13 as an upper reference illuminance, the illuminance of the middle shelf 14 as a middle reference illuminance, and the illuminance of the lower shelf 15 as a lower reference illuminance. The passing object detection unit 111 stores this data in the memory unit 130.

[0041] The upper reference illuminance, the middle reference illuminance, and the lower reference illuminance are the illuminances of the shelves 13, 14, 15 when one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 are opened and before an object (a user's hand, food, etc.) passes through the opening 10a of the refrigerator compartment 10. Note that, taking into consideration errors and fluctuations in the detected illuminance, the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance may be set to be slightly lower or slightly higher than the detected illuminances of the illuminance sensors 17a, 17b, 17c.

[0042] Furthermore, the upper reference illuminance, middle reference illuminance, and lower reference illuminance used in step S13 described below may be set to different values ​​from the upper reference illuminance, middle reference illuminance, and lower reference illuminance used in step S14. In this case, the upper reference illuminance, middle reference illuminance, and lower reference illuminance used in step S13 correspond to the first reference illuminance of the present invention, and the upper reference illuminance, middle reference illuminance, and lower reference illuminance used in step S14 correspond to the second reference illuminance of the present invention.

[0043] In the next step S12, the passing object detection unit 111 detects the illuminance of the upper shelf 13, the middle shelf 14, and the lower shelf 15 using the illuminance sensors 17a, 17b, and 17c. Then, in the next step S13, the passing object detection unit 111 determines whether there is a shelf whose detected illuminance is lower than the reference illuminance by a predetermined level or more. Specifically, the passing object detection unit 111 determines whether each of the following conditions (1) to (3) is met.

[0044] (1) The illuminance detected by the illuminance sensor 17a has changed from the "upper reference illuminance" to "upper reference illuminance-α" or lower. (2) The illuminance detected by the illuminance sensor 17b has changed from the "medium reference illuminance" to "medium reference illuminance-α" or lower. (3) The illuminance detected by the illuminance sensor 17c has changed from the "lower reference illuminance" to "lower reference illuminance-α" or lower.

[0045] where α is a constant for setting a threshold determined by experiments, computer simulations, etc., assuming a drop in illuminance that occurs when an object (food, a user's hand, etc.) enters the upper shelf 13, the middle shelf 14, or the lower shelf 15. Note that α may be a common value for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, or may be set to different values ​​individually for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance.

[0046] Passing object detection unit 111 determines that an object has passed through opening 10a when at least one of the above (1) to (3) is met. Here, when at least one of the above (1) to (3) is met, it is assumed that the user has put their hand into refrigeration compartment 10 to put food into refrigeration compartment 10 (storing), to take food out of refrigeration compartment 10 (removing), or to change the position of food in refrigeration compartment 10. In this case, passing object detection unit 111 can determine that an object (the user's hand, food held by the user's hand, etc.) has passed through opening 10a of refrigeration compartment 10.

[0047] Therefore, the imaging time setting unit 116 sets the time when the passing object detection unit 111 determines in step S13 that at least one of the above (1) to (3) is established as the first imaging time, and proceeds to step S20. The first imaging time is set assuming the time when food to be stored in the refrigerator compartment 10 passes through the opening 10a. Note that in step S13, the first imaging time may be set to a time that is a predetermined time after the time when the passing object detection unit 111 determines that at least one of the above (1) to (3) is established.

[0048] On the other hand, when none of the above (1) to (3) is satisfied, the imaging time point setting unit 116 advances the process to step S14. Step S14 is a process by the passing object detection unit 111 and the imaging time point setting unit 116. The passing object detection unit 111 judges whether or not there is a shelf whose detected illuminance has changed from less than or equal to "reference illuminance-β" to more than or equal to "reference illuminance". Specifically, the passing object detection unit 111 judges whether or not each of the following conditions (4) to (6) is satisfied.

[0049] (4) The illuminance detected by the illuminance sensor 17a has changed from below the "upper reference illuminance-β" to above the "upper reference illuminance." (5) The illuminance detected by the illuminance sensor 17b has changed from below the "medium reference illuminance-β" to above the "medium reference illuminance." (6) The illuminance detected by the illuminance sensor 17c has changed from less than or equal to the "lower reference illuminance-β" to more than or equal to the "lower reference illuminance."

[0050] where β is a constant for determining a threshold value determined by experiments, computer simulations, etc., assuming an increase in illuminance that occurs when an object (food, a user's hand, etc.) that has entered the upper shelf 13, the middle shelf 14, or the lower shelf 15 passes through the opening 10a and leaves the refrigerator compartment 10. β may be set to a common value for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, or may be set to different values ​​individually for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance.

[0051] When at least one of the above (4) to (6) is established, passing object detection unit 111 determines that an object (food, a user's hand, etc.) that was present in refrigeration compartment 10 has left refrigeration compartment 10 and passed through opening 10a. Here, when at least one of the above (4) to (6) is established, it is assumed that the user has removed their hand that was in refrigeration compartment 10 to put food into refrigeration compartment 10 (storage), to take food out of refrigeration compartment 10 (removal), or to change the position of food contained in refrigeration compartment 10. In this case, passing object detection unit 111 can determine that an object (a user's hand, food held by a user's hand, etc.) has passed through opening 10a of refrigeration compartment 10.

[0052] Therefore, the imaging time setting unit 116 sets the time when the passing object detection unit 111 determines in step S14 that at least one of the above conditions (4) to (6) is met as the second imaging time, and proceeds to step S22. The second imaging time is set assuming the time when food being removed from the refrigerator compartment 10 passes through the opening 10a. Note that in step S14, the second imaging time may be set to a time that is a predetermined time after the time when the passing object detection unit 111 determines in step S14 that at least one of the above conditions (4) to (6) is met.

[0053] On the other hand, when none of the above (4) to (6) is satisfied, the passing object detection unit 111 advances the process to step S15. In step S15, the passing object detection unit 111 judges whether or not both the right door 11 and the left door 12 of the refrigerator compartment 10 are closed, based on the detection signals of the open / close sensors 18a, 18b. Then, when both the right door 11 and the left door 12 are closed, the passing object detection unit 111 advances the process to step S15 and ends the "imaging process". On the other hand, when at least one of the right door 11 and the left door 12 is open, the passing object detection unit 111 advances the process to step S12.

[0054] Next, steps S20 to S21 and steps S22 to S23 are processes performed by the food information acquisition unit 114. In step S20, the food information acquisition unit 114 captures an image of an area including the opening 10a of the refrigerator compartment 10 with the camera 60 via the image processing unit 120. In the following step S21, the food information acquisition unit 114 stores the image data captured in step S20 in the storage unit 130 as "entry image" data, and proceeds to step S15. Note that the image may be captured multiple times, and the multiple image data may be stored in the storage unit 130 as "entry image" data.

[0055] In step S22, the food information acquisition unit 114 captures an image of an area including the opening 10a of the refrigerator compartment 10 with the camera 60 via the image processing unit 120. In the following step S23, the food information acquisition unit 114 stores the image data captured in step S22 in the storage unit 130 as "image at time of leaving the warehouse" data, and proceeds to step S15. Note that the image may be captured multiple times, and the multiple image data may be stored in the storage unit 130 as "image at time of leaving the warehouse" data.

[0056] By the "imaging process" shown in Figure 7, the passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10 based on the change in illuminance of the upper shelf 13, the middle shelf 14, or the lower shelf 15 detected by the illuminance sensors 17a, 7b, 17c.

[0057] Furthermore, an "entry image", which is an image captured near the opening 10a at the time when the passing object detection unit 111 detects an object that has passed through the opening 10a and entered any one of the shelves 13, 14, 15, is stored in the storage unit 130. Furthermore, an "exit image", which is an image captured near the opening 10a at the time when the passing object detection unit 111 detects an object that has left any one of the shelves 13, 14, 15 and passed through the opening 10a, is stored in the storage unit 130.

[0058] 7, if an object passing through opening 10a is detected multiple times in step 13 before right door 11 and left door 12 are detected as being closed in step S15, multiple "entry images" are captured and stored in steps S20 and S21. Then, "image analysis processing" is performed on each of the "entry images" of foods determined to have been stored by "weight information processing" described below.

[0059] Similarly, when an object passing through opening 10a is detected multiple times in step S14 before it is detected in step S15 of Figure 7 that right door 11 and left door 12 are closed, multiple "images at the time of leaving" are captured and stored in steps S22 and S23. Then, "image analysis processing" is performed on each of the "images at the time of leaving" of the food determined to have been left by "weight information processing" described below.

[0060] [3-2. Weight information processing] Next, the execution procedure of the "weight processing" will be described with reference to the flowchart shown in Fig. 8. The "weight information processing" is executed by the entry / exit determination unit 113 and the food information acquisition unit 114.

[0061] Steps S30 to S33 and S40 in Fig. 8 are processes performed by the incoming / outgoing determination unit 113. In step S30, the incoming / outgoing determination unit 113 recognizes the weight of each shelf 13, 14, 15 by the storage capacity recognition unit 112 and stores it in the memory unit 130. In the following step S31, when the passing object detection unit 111 detects an object passing through the opening 10a of the refrigeration compartment 10 in step S13 or step S14 in Fig. 7 (when an object enters or leaves the refrigeration compartment 10), the incoming / outgoing determination unit 113 advances the process to step S32.

[0062] In step S32, the incoming / outgoing determination unit 113 recognizes the weight of each shelf 13, 14, 15 by the storage capacity recognition unit 112. In the next step S33, the incoming / outgoing determination unit 113 determines whether or not there is a shelf where the weight recognized in step S32 has increased compared to the weight recognized in step S30. Then, if there is a shelf where the weight has increased (in this case, it can be determined that food has been stored in the refrigerator compartment 10), the incoming / outgoing determination unit 113 proceeds to step S34.

[0063] On the other hand, if there is no shelf where the weight has increased, the entry / exit determination unit 113 advances the process from step S33 to step S40, and determines whether there is a shelf where the weight recognized in step S32 has decreased compared to the weight recognized in step S30. Then, if there is a shelf where the weight has decreased (in this case, it can be determined that food has been removed from the refrigerator compartment 10), the entry / exit determination unit 113 advances the process to step S41. If there is no shelf where the weight has increased (in this case, it can be determined that food has not been entered into or removed from the refrigerator compartment 10), the entry / exit determination unit 113 advances the process to step S36 and ends the "weight information processing".

[0064] Steps S34 to S35 and step S41 are processes performed by food information acquisition unit 114. Food information acquisition unit 114 estimates the storage position of the stored food from the left and right weights of each shelf 13, 14, 15 recognized by storage amount recognition unit 112 in step S34.

[0065] In the next step S35, the food information acquisition unit 114 records the weight and storage location information of the stored food in the stored food information 132 (see FIG. 5), and proceeds to step S36, terminating the "weight information processing." The data of the stored food information 132 is temporarily stored in the memory unit 130 in order to add or update information to the stored food list 131.

[0066] In step S41, food information acquisition unit 114 records the weight information of the delivered food in delivered food information 133 (see FIG. 5) and proceeds to step S36, terminating the "weight information processing." The data of delivered food information 133 is temporarily stored in storage unit 130 for adding or updating food information to in-warehouse food list 131.

[0067] [3-3. Image analysis processing] Next, the execution procedure of the "image analysis process" will be described according to the flowcharts shown in Figs. 9 and 10. The "image analysis process" is executed by the food information acquisition unit 114. Fig. 9 shows a process for acquiring information from an "image at the time of entry" for food stored in the refrigerator compartment 10. Fig. 10 shows a process for acquiring information from an "image at the time of exit" for food removed from the refrigerator compartment 10.

[0068] First, the "image analysis process" for the "image at time of storage" will be described with reference to Fig. 9. In step S50 of Fig. 9, food information acquisition unit 114 executes character extraction process for the "image at time of storage" of stored food by image processing unit 120. Here, stored food is food that is determined to have been stored in refrigerator compartment 10 by storage / return determination unit 113 in step S33 of Fig. 8. In addition, the "image at time of storage" of stored food is the "image at time of storage" captured and stored by food information acquisition unit 114 in steps S20 and S21 of Fig. 7 for food that is determined to have been stored in step S33 of Fig. 8.

[0069] In the next step S51, the food information acquisition unit 114 determines whether or not characters have been extracted from the “stored image” by the image processing unit 120. If characters have been extracted, the food information acquisition unit 114 proceeds to step S52, and if characters have not been extracted, the food information acquisition unit 114 proceeds to step S60.

[0070] In step S52, food information acquisition unit 114 analyzes the text information extracted by image processing unit 120. Then, in the following step S53, food information acquisition unit 114 acquires information about the stored food, such as the food name, expiration date, place of origin, and content volume, from the text information. In the next step S54, food information acquisition unit 114 records the acquired information about the stored food in stored food information 132, and ends the "image analysis process."

[0071] Furthermore, in step S60, food information acquisition unit 114 extracts an image portion of the stored food from the "storage-time image" by image processing unit 120. In the following step S61, food information acquisition unit 114 calculates a match rate between the extracted image portion of the stored food and sample images 134 of various foods (see FIG. 5) stored in advance in storage unit 130.

[0072] In the next step S62, food information acquisition unit 114 estimates and acquires the food name of sample image 134 with the highest matching rate as the food name of the stored food, and proceeds to step S54. By the processing of steps S60 to S62, even if characters cannot be extracted from the "image at the time of storage" of the stored food, it is possible to estimate the food name of the stored food and record it in stored food information 132.

[0073] Next, the "image analysis process" for the "image at time of leaving the warehouse" will be described with reference to Fig. 10. In step S70 of Fig. 10, food information acquisition unit 114 executes character extraction process on the "image at time of leaving the warehouse" of the leaving food by image processing unit 120. Here, the leaving food is the food that is determined to have been left from refrigerator compartment 10 by incoming / outgoing determination unit 113 in step S40 of Fig. 8. In addition, the "image at time of leaving the warehouse" of the leaving food is the "image at time of leaving the warehouse" that is captured and stored by food information acquisition unit 114 in steps S22 and S23 of Fig. 7 for the food that is determined to have been left in step S40 of Fig. 8.

[0074] In the next step S71, the food information acquisition unit 114 determines whether or not characters have been extracted from the "image at the time of leaving the warehouse" by the image processing unit 120. If characters have been extracted, the food information acquisition unit 114 proceeds to step S72, and if characters have not been extracted, the food information acquisition unit 114 proceeds to step S80.

[0075] In step S72, food information acquisition unit 114 analyzes the text information extracted by image processing unit 120. Then, in the following step S73, food information acquisition unit 114 acquires information about the shipped food, such as the food name, expiration date, place of origin, and content volume, from the text information. In the next step S74, food information acquisition unit 114 records the acquired information about the shipped food in shipped food information 133, and ends the "image analysis process."

[0076] In step S80, food information acquisition unit 114 extracts an image portion of the delivered food from the "image at time of delivery" using image processing unit 120. In the following step S81, food information acquisition unit 114 calculates a match rate between the extracted image portion of the delivered food and sample images 134 of various foods stored in advance in storage unit 130.

[0077] In the next step S82, food information acquisition unit 114 estimates and acquires the food name of sample image 134 with the highest matching rate as the food name of the shipped food, and proceeds to step S74. By the processing of steps S80 to S82, even if characters cannot be extracted from the "image at time of shipment" of the shipped food, it is possible to estimate the food name of the shipped food and record it in shipped food information 133.

[0078] The image analysis process in Fig. 9 is performed on an image at the time of entry when entry into the refrigerator compartment 10 is determined, and the image analysis process in Fig. 10 is performed on an image at the time of exit when exit from the refrigerator compartment 10 is determined. Therefore, when the right door 11 or the left door 12 is detected by the passing object detection unit 111 instead of food, it is not determined that entry or exit has occurred, and therefore the image analysis process is not performed (the captured image is invalid). This makes it possible to prevent erroneous food information from being obtained from the image portion of the right door 11 or the left door 12.

[0079] [3-4. Weight sensor correction process] The execution procedure of the "weight sensor calibration process" for correcting the detected weights of the weight sensors 13a, 13b, 14a, 14b, 15a, and 15b will be described with reference to the flowchart shown in Fig. 11. The "weight sensor correction process" is executed by the contained amount recognition unit 112. The "weight sensor correction process" is a process for correcting (calibrating) the detection errors of the weight sensors 13a, 13b, 14a, 14b, 15a, and 15b that may occur due to changes over time, etc.

[0080] In step S90 of Figure 11, when the in / out determination unit 113 determines that food has been stored in the refrigerator compartment (YES in step S33 of Figure 8), the storage amount recognition unit 112 proceeds to step S91. In step S91, the storage amount recognition unit 112 determines whether or not information on the content amount of the stored food is recorded in the stored food information 132 (processing of step S54 of Figure 9). Then, when information on the content amount of the stored food (the content amount of the food obtained in step S53 of Figure 9) is recorded in the stored food information 132, the storage amount recognition unit 112 proceeds to step S92.

[0081] In step S92, the storage amount recognition unit 112 calculates the ratio between the weight and content amount of the stored food recorded in the stored food information 132. The storage amount recognition unit 112 then sets the calculated ratio as the correction coefficient for the weight sensor used to detect the weight of the stored food. For example, when the stored food is stored on the upper shelf 13 and its weight is detected by the weight sensors 13a and 13b, the storage amount recognition unit 112 sets the calculated ratio as the correction coefficient for the detected weight of the weight sensors 13a and 13b.

[0082] In the next step S93, storage amount recognition unit 112 stores the correction coefficient data in memory unit 130, and proceeds to step S94 to end the "weight sensor correction process." Thereafter, storage amount recognition unit 112 treats the detected weight multiplied by the correction coefficient by weight sensors 13a, 13b as the detected weight. Similarly, when the stored food is stored on middle shelf 14, storage amount recognition unit 112 sets the correction coefficient for weight sensors 14a, 14b, and when the stored food is stored on lower shelf 15, it sets the correction coefficient for weight sensors 15a, 15b.

[0083] On the other hand, if in step S91 no content volume information is recorded in the stored food information 132, the content volume recognition unit 112 advances the process to step S94 and ends the "weight sensor correction process." In this case, the correction coefficient is not calculated or set.

[0084] [3-5. Food management in the warehouse] 12 and 13, a description will be given of the procedure for executing the "in-fridge food management process" for managing information on food stored in the refrigerator compartment 10. The "in-fridge food management process" is executed by the food information management unit 115.

[0085] The food information management unit 115 manages information on the foods stored in the refrigerator compartment 10 using the stored food list 131 (see FIG. 5). As shown in FIG. 14, the stored food list 131 records the numbers (1, 2, 3, ...) given to the foods stored in the refrigerator compartment 10, and information on each stored food with a number, such as the food name, weight (first time, current, consumed amount), storage / removal date / time (first time stored, latest storage / removal, latest storage), stock (1: present, 0: absent), expiration date, and place of origin. The food information management unit 115 generates and updates the stored food list 131 based on the stored food information 132 and the stored food information 133 recorded by the food information acquisition unit 114.

[0086] In step S100 of Fig. 12, the food information management unit 115 determines whether the entry / exit determination unit 113 has determined that food has been entered into or removed from the refrigerator compartment 10, and proceeds to step S101 when entry or exit is determined. In step S101, the food information management unit 115 determines whether entry has been determined, and proceeds to step S102 when entry has been determined. On the other hand, when entry has not been determined (when exit has been determined), the food information management unit 115 proceeds to step S110.

[0087] In step S102, the food information management unit 115 determines whether or not food has been removed from the refrigerator compartment 10 within a predetermined time. If food has been removed from the refrigerator compartment 10 within the predetermined time, the process proceeds to step S103, and if food has not been removed from the refrigerator compartment 10 within the predetermined time, the process proceeds to step S111. Note that it is possible to recognize from the food in-fridge list 131 that food has been removed from the refrigerator compartment 10 within the predetermined time. The predetermined time is set to, for example, about 10 minutes, assuming a case in which the user uses some of the food removed from the refrigerator compartment 10 and returns it to the refrigerator compartment 10 (re-storing it).

[0088] In step S103, the food information management unit 115 judges whether the coincidence rate between the food previously sent out and the food currently stored is equal to or greater than a predetermined value. Here, the food information management unit 115 compares the information of the previously sent out product recorded in the in-store food list 131 with the information of the currently stored product recorded in the stored food information 132, and calculates the coincidence rate between the previously sent out food and the currently stored food from the coincidence status of the food name, expiration date, place of origin, etc.

[0089] Then, if the matching rate between the food previously shipped and the food currently stocked is equal to or greater than a predetermined value, the food information management unit 115 proceeds to step S104, and if the matching rate between the food previously shipped and the food currently stocked is less than the predetermined value, the food information management unit 115 proceeds to step S112.

[0090] Step S104 is a process performed when the matching rate between the food previously taken out and the food currently stored is equal to or greater than a predetermined value, and it can be determined that the food previously taken out and the food currently stored are the same. In step S104, food information management unit 115 updates the current weight, latest stored date and time, and consumed amount (difference between the weight at the time of first storing and the current weight) of the food taken out before a predetermined time in stored food list 131 based on the information recorded in stored food information 132, and proceeds to step S105 in FIG. 13.

[0091] Step S112 is a process performed when the matching rate between the food that was taken out and the food that was currently stored is less than a predetermined value, and it can be determined that the food that was taken out last time is different from the food that was currently stored. In step S112, food information management unit 115 newly registers the stored food in in-store food list 131 based on the information recorded in stored food information 132, and proceeds to step S105 in FIG. 12.

[0092] Step S111 is a process to be performed when no food is removed from the warehouse within a predetermined time. In step S111, the food information management unit 115 newly registers the received food in the warehouse food list 131 based on the information recorded in the received food information 132, and proceeds to step S105 in FIG.

[0093] Step S110 is a process performed when it is determined that food has been removed from refrigerator compartment 10. In step S110, food information management unit 115 updates the current weight of the food (removed food) to 0 in stored food list 131. Food information management unit 115 also updates the amount consumed of the food in stored food list 131, updates the latest removal time based on the information recorded in removed food information 133, and proceeds to step 105 in FIG. 13.

[0094] 13, the food information management unit 115 determines whether or not there is any food whose current weight is 0 grams as a result of the processing in step S110. If there is any food whose current weight is 0 grams, the food information management unit 115 proceeds to processing in step S105, and if there is no food whose current weight is 0 grams, the food information management unit 115 proceeds to processing in step S120.

[0095] In step S106, the food information management unit 115 updates the information of the relevant food (food determined to have been taken out this time) in the stored food list 131 to 'out of stock' (stock 0), and proceeds to step S107. Also, in step S120, the food information management unit 115 updates the information of the relevant food (food determined to have been received this time) in the stored food list 131 to 'in stock' (stock 1), and proceeds to step S107.

[0096] In step S107, food information management unit 115 determines whether or not expiration date information based on character recognition is recorded in stored food information 132. If expiration date information is recorded in stored food information 132, food information management unit 115 proceeds to step S108, and if expiration date information is not recorded, food information management unit 115 proceeds to step S121.

[0097] In step S108, food information management unit 115 registers the expiration date of the food currently stored in storehouse food list 131 based on the expiration date information recorded in stored food information 132, and proceeds to step S109. Also, in step S121, food information management unit 115 refers to expiration date list 135 stored in memory unit 130 to search for the general expiration date of the relevant food.

[0098] As shown in Fig. 14, the expiration date list 135 is a list that associates food names with the expiration dates of the foods, and the food information management unit 115 searches for the expiration date associated with the food name of the currently stored food. In the following step S122, the food information management unit 115 registers the expiration date in the data of the corresponding food in the in-warehouse food list 131, and proceeds to step S109.

[0099] In step S109, the food information management unit 115 transmits the updated data of the in-storage food list 131 to the management server 520 via the communication network 510 using the wireless communication unit 140, and then ends the "in-storage food management process."

[0100] The management server 520 stores the data of the stored food list 131 received from the refrigerator 1 in a storage device (not shown), and transmits the consumption status of each product recognized from the stored food list 131, the number of days remaining until the expiration date, etc. to the user's terminal device 400. This allows the user to check the status of the products stored in the refrigerator 1 using the terminal device 400 even when away from home, etc.

[0101] [4. Another embodiment of imaging processing] Next, another embodiment of the "imaging process" will be described with reference to the flowchart shown in Fig. 15. The flowchart shown in Fig. 15 is executed by the food information acquisition unit 114 and the imaging time point setting unit 116. The imaging time point setting unit 116 sets the time point when a first predetermined time has elapsed since the right door 11 or the left door 12 of the refrigerator compartment 10 was opened as the first imaging time point, and sets the time point when a second predetermined time has elapsed since the right door 11 and the left door 12 of the refrigerator compartment 10 were closed as the second imaging time point.

[0102] The food information acquisition unit 114 performs imaging using the camera 60 at the first imaging point and the second imaging point from the time when it recognizes in step S130 from the detection signals of the opening / closing sensors 18a, 18b that one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 have switched from a closed state to an open state until it recognizes in step S137 that both the right door 11 and the left door 12 have switched from a closed state to an open state.

[0103] When the food information acquisition unit 114 recognizes in step S130 that one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 have been switched from a closed state to an open state, the food information acquisition unit 114 proceeds to step S131. Steps S131 and S132 are processes performed by the image capture time point setting unit 116.

[0104] In step S131, the imaging time point setting unit 116 starts a timer that counts the first predetermined time until the timer times out. Then, in the following step S132, when the timer times out (when the first imaging time point has arrived), the process proceeds to step S133.

[0105] Steps S133 to S138 are processes performed by the food information acquisition unit 114. In step S133, the food information acquisition unit 114 captures images multiple times using the camera 60, and in the following step S134, stores the captured image data in the storage unit 130 as "entry image" data.

[0106] In the subsequent loop of steps S135 to S137, the food information acquisition unit 114 repeatedly executes the processes of steps S135 and S136 until the opening / closing sensors 18a, 18b detect in step S137 that the right door 11 and the left door 12 are closed. In step S135, the food information acquisition unit 114 captures an image using the camera 60 and records the image capture time in the storage unit 130, and in step S136, stores the captured image data in the storage unit 130 as candidate data for an image at the time of leaving the warehouse. As a result, the captured image data in chronological order is stored in the storage unit 130 in sequence.

[0107] When the opening / closing sensors 18a, 18b detect in step S137 that the right door 11 and the left door 12 are closed, the food information acquisition unit 114 proceeds to step S138. In step S138, the food information acquisition unit 114 selects, from among the candidate leaving-time image data stored in the storage unit 130, image data captured a second predetermined time before the point in time when it is detected in step S137 that both the right door 11 and the left door are closed, and stores the image data in the storage unit 130 as a leaving-time image.

[0108] As another configuration for setting the first and second imaging time points by the imaging time point setting unit 116, the first and second imaging time points may be set based on the detection status of an object by the objective sensor 80 (see Figs. 3 to 5) that detects an object passing through the opening 10a. In this case, the imaging time point setting unit 116 sets the first imaging time point to, for example, a time point when a predetermined time has elapsed from the time point when the objective sensor 80 switches from a state in which an object is not detected to a state in which an object is detected. Furthermore, the imaging time point setting unit 116 subsequently sets the second imaging time point to, for example, a time point a predetermined time before the time point when the objective sensor 80 switches from a state in which an object is detected to a state in which an object is not detected.

[0109] 5. Other embodiments In the above embodiment, the passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10 based on the change in illuminance in the refrigerator compartment 10 detected by the illuminance sensors 17a, 17b, and 17c. As another configuration of the passing object detection unit 111, as shown in Figs. 3 to 5, an objective sensor 80 (such as a reflective or transmissive optical sensor, ultrasonic sensor, or capacitance sensor) that includes the opening 10a in its detection range and directly detects an object passing through the opening 10a may be used. In this case, the detection range of the objective sensor 80 may be set in front of the imaging range of the camera 60 in the direction from the opening 10a toward the inside of the refrigerator compartment 10. This makes it possible to prevent failure to capture an image of the food to be stored due to a delay time from the time an object passing through the opening 10a is detected until the camera 60 captures an image.

[0110] Furthermore, when the opening / closing sensor 18a detects that the left door 12 is open, and when the opening / closing sensor 18b detects that the right door 11 is open, power may be started to be supplied to the camera 60, enabling the camera 60 to capture images. With this configuration, when the left door 12 and the right door are closed, power is cut off to the camera 60, thereby saving electricity. Furthermore, by starting power to the camera 60 when the right door or the left door 12 is opened, it is possible to prevent failure to capture images of food items stored in the storehouse due to a delay in starting the camera 60.

[0111] Furthermore, the food information acquisition unit 114 may be configured to invalidate image data captured by the camera 60 when the opening / closing sensor 18a detects that the left door 12 has switched from an open state to a closed state, or when the opening / closing sensor 18b detects that the right door 11 has switched from an open state to a closed state, within a predetermined time from when the passing object detection unit 111 detects an object passing through the opening 10a. With this configuration, when the left door 12 or the right door 11 is detected by the passing object detection unit 111 and imaged by the camera 60 when the left door 12 or the right door 11 is closed, it is possible to prevent erroneous food information from being extracted from the image of the left door 12 or the right door 11.

[0112] In the above embodiment, the storage amount recognition unit 112 recognizes the amount of food stored on each shelf 13, 14, 15 of the refrigerator compartment 10 based on the weight detected by the weight sensors 13a, 13b, 14a, 14b, 15a, 15b. As another configuration of the storage amount recognition unit 112, the amount of food stored on each shelf 13, 14, 15 of the refrigerator compartment 10 may be recognized based on the illuminance detected by the illuminance sensors 17a, 17b, 17c with the lighting units 16a, 16b, 16c, 16d, 16e, 16f turned on. Alternatively, the amount of food stored on each shelf 13, 14, 15 of the refrigerator compartment 10 may be recognized by extracting an image portion of the food from an image of the inside of the refrigerator compartment 10 captured by the camera 60 or a separately arranged camera.

[0113] In the above embodiment, the image captured by the camera 60 is processed by the image processing unit 120. However, the function of the image processing unit 120 may be provided in the CPU 110, and the CPU 110 may process the captured image.

[0114] In the above embodiment, the data of the sample image 134 and the expiration date list 135 are stored in the memory unit 130, but they may also be stored in the management server 520 and downloaded from the management server 520 to the control unit 100 of the refrigerator 1 when used.

[0115] Also, although sample image 134 is stored in memory unit 130 or management server 520, an image recognition program that has learned the characteristics of each food or stored item in advance, such as color, shape, size, etc., may be implemented in CPU 110, and the image portion of the stored food may be processed by the image recognition program to calculate the match rate and obtain an estimated food name of the stored food. This makes sample image 134 unnecessary, making it possible to reduce the storage capacity stored in memory unit 130 or management server 520 and also shortens the recognition processing time for estimating the food name of the stored food. The food name of the outgoing food may also be estimated in the same manner as above.

[0116] In the above embodiment, the in / out determination unit 113 determines whether food is in or out of the refrigerator compartment 10 of the refrigerator 1, the food information acquisition unit 114 acquires information on the in-store food and the out-store food, and the food information management unit 115 manages information on the food stored in the refrigerator compartment 10. The switchable compartment 20, the freezer compartment 40, and the vegetable compartment 50 are also provided with a passing object detection unit that detects objects passing through the openings 20a, 40a, and 50a, and a storage amount recognition unit that recognizes the amount of stored food, so that the in / out of food can be determined. In addition, a camera that captures an image of an area including the openings 20a, 40a, and 50a can be provided to acquire information on the in-store food and the out-store food from the captured image (image at the time of in-store and image at the time of out-store), so that information on the food stored in the switchable compartment 20, the freezer compartment 40, and the vegetable compartment 50 can be acquired and the food can be managed. [Industrial Applicability]

[0117] As described above, the refrigerator of the present invention can more reliably obtain information about food being entered into and food being released from the storage chamber by capturing images of food being entered into and food being released from the storage chamber at the anticipated time of passing through the storage chamber, and can therefore be applied to managing food stored in the refrigerator. [Explanation of symbols]

[0118] 1. Refrigerator 10 Refrigerator 10a opening 11 Right door 12 Left door 13 Upper shelf 13 Shelves 13a, 13b Weight sensor 14 Middle shelf 14a, 14b Weight sensor 15 Lower shelf 15a, 15b Weight sensor 16a, 16b, 16c, 16d, 16r, 16f Lighting section 17a, 17b, 17c Light sensor 18a, 18b Open / close sensor 20 Switching room 20a opening 22 Open / close sensor 30 Ice Room 30a opening 32 Open / close sensor 32 Steps 32 Open / close sensor 40 Steps 40 Freezer 40a opening 42 Open / close sensor 50 Vegetable compartment 50a opening 52 Open / close sensor 60 Cameras 61 Compressor 62 Cooling fan 63 Cooler 64 Condenser 100 Control Unit 110 CPU 111 Passing object detection unit 112 Capacity Recognition Unit 113 Entry / Exit Judgment Department 114 Food Information Acquisition Department 115 Food Information Management Department 116 Imaging time setting unit 120 Image Processing Unit 121 Image Memory 130 Storage section 131 Food List 132 Stocked Food Information 133 Food shipment information 134 Sample Images 135 Best Before List 140 Wireless Communication Division 141 Date and time recognition section 400 Terminal Equipment 500 Gateway 510 Communication Network 520 Management Server

Claims

1. A refrigerator, A storage chamber having an opening; A door provided in the storage chamber; an opening / closing sensor for detecting an opening / closing state of the door; a camera that captures an image of an imaging range including the opening; A storage unit; The camera captures images multiple times from the time when the opening / closing sensor detects that the door has switched from the closed state to the open state until the opening / closing sensor detects that the door has switched from the open state to the closed state, and captures images from the time when the opening / closing sensor detects that the door has switched from the closed state to the open state until the time when the opening / closing sensor detects that the door has switched from the open state to the closed state after a first image capture time has elapsed, the first image capture time being set assuming that food to be stored in the storage chamber will pass through the opening. a food information acquisition unit that repeatedly captures images using the camera, and stores a plurality of captured images in the storage unit as candidate images at the time of leaving the storage unit together with the capture time, and when the opening / closing sensor detects that the door has switched from an open state to a closed state, stores in the storage unit, as a candidate image at the time of leaving the storage unit, the candidate image at the time of leaving the storage unit, which is captured at a second capturing time point that is a second predetermined time before the time when the opening / closing sensor detects that the door has switched from an open state to a closed state, the second capturing time point being set assuming a time when the food being left from the storage room passes through the opening; A refrigerator equipped with

2. The food information acquisition unit stores the image captured at the first imaging time point in the storage unit as an entry image, and acquires information on the food to be stored in the storage room and the food to be removed from the storage room based on the entry image and the removal image.

2. The refrigerator according to claim 1.

3. A refrigerator, A storage chamber having an opening; A door provided in the storage chamber; an opening / closing sensor for detecting an opening / closing state of the door; a camera that captures an image of an imaging range including the opening; A storage unit; a food information acquisition unit that performs imaging using the camera a plurality of times from when the opening / closing sensor detects that the door has switched from a closed state to an open state until the opening / closing sensor detects that the door has switched from an open state to a closed state, and stores in the storage unit an image captured at a first imaging time point when a first predetermined time has elapsed from when the opening / closing sensor detects that the door has switched from a closed state to an open state, the first predetermined time point being set assuming a time point when food to be stored in the storage chamber will pass through the opening, and stores in the storage unit an image captured at a second imaging time point when a second predetermined time has elapsed from when the opening / closing sensor detects that the door has switched from an open state to a closed state, the second predetermined time point being set assuming a time point when food to be removed from the storage chamber will pass through the opening, the second predetermined time point being set assuming a time point when food to be removed from the storage chamber will pass through the opening, the second predetermined time point being set assuming a time point when food to be removed from the storage chamber will pass through the opening, and acquires information about the food to be stored in the storage chamber and the food to be removed from the storage chamber based on the image captured at the entrance and the image removed. A refrigerator equipped with

Citation Information

Patent Citations

  • Refrigerator

    JP2003042626A

  • Refrigerator

    JP2015081762A

  • Food management system

    JP2016057022A