refrigerator
The refrigerator uses multiple cameras to calculate parallax for precise food size and amount recognition, addressing the complexity of conventional weight sensors and improving food management accuracy.
Patent Information
- Application Number
- JP2023129791
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-08-09
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2037-12-20
AI Technical Summary
Conventional refrigerators struggle to accurately recognize the amount and type of foods like cabbage and radish, which are often divided into several portions, using weight sensors, leading to a complex internal structure and difficulty in individual food detection.
A refrigerator equipped with multiple cameras (left, right, and side cameras) calculates parallax to determine food size and amount, and a food information management unit processes images to recognize food type and quantity.
Accurately recognizes the amount and type of foods using a simple configuration, avoiding the complexity of weight sensors and enhancing food management efficiency.
Smart Images

Figure 0007584036000001 
Figure 0007584036000002 
Figure 0007584036000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a refrigerator. [Background technology]
[0002] Conventionally, there is known a refrigerator that uses a camera to capture images of food stored in the refrigerator compartment, extracts images of individual foods from the captured images, compares the images of the food extracted from images captured at two consecutive points in time to determine whether the food has been stored or taken out, and saves the images (see, for example, Patent Document 1). In the refrigerator described in Patent Document 1, the type and amount of food stored in and removed from the refrigerator compartment is identified based on images of each food extracted from images captured by a camera, by referring to information stored in a food image database. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2016-200380 A Summary of the Invention [Problem to be solved by the invention]
[0004] In the refrigerator described in the above Patent Document 1, the type and amount of food stored in the refrigerator compartment is determined from information stored in a food image database based on an image of the food when it was put in, so the amount of food that can be recognized is the number of pieces or a pre-stored specified amount (length of the major axis). Therefore, it is not possible to recognize the remaining amount of food that is often used in several portions, such as cabbage or radish. In order to determine the amount of food remaining, it is possible to use a weight sensor to detect changes in the weight of the food, but it is difficult to detect the weight of individual food items using a weight sensor, and the internal structure of the refrigerator becomes complex in order to secure space and wiring paths for placing the weight sensor. The present invention has been made in view of the above background, andThe amount and type of food stored in the Simple configuration recognition The object of the present invention is to provide a refrigerator that can [Means for solving the problem]
[0005] In order to solve the above-mentioned problems, the refrigerator of the present invention has a food storage chamber and an opening for the storage chamber. and a nozzle is disposed on the left side of the upper portion of the opening, and the nozzle enters the storage chamber from the left side of the upper portion of the opening. To capture left A camera and a right camera disposed on the right side of an upper portion of the opening and capturing an image of the inside of the storage chamber from the right side of the upper portion of the opening; a side camera disposed on a side wall of the storage chamber and capturing an image of the inside of the storage chamber from the side wall side; a food size recognition unit that recognizes the size of the target food by calculating a parallax between the left camera and the right camera for multiple locations of the target food from images of the same target food captured by the left camera and the right camera and calculating real space positions of the multiple locations based on the parallax between the left camera and the right camera; a food amount recognition unit that recognizes the amount of the target food based on the size of the target food recognized by the food size recognition unit; and a food type recognition unit that recognizes the type of the target food from images of the target food extracted from images captured by the left camera, the right camera, and the side camera. It is something.
[0006] This means, The food size recognition unit recognizes the size of the target food from the images of the same target food captured by the left camera and the right camera, the food amount recognition unit recognizes the amount of the target food, and the food type recognition unit recognizes the type of the target food from the images of the target food captured by the left camera, the right camera, and the side camera. It is possible. Effect of the Invention
[0007] According to the refrigerator of the present invention, The amount and type of food stored in the storage compartment is recognized by a simple configuration using multiple cameras. It is possible. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is an explanatory diagram of a configuration for checking the status of food stored in a refrigerator installed in a house using a terminal device. [Diagram 2] Diagram of the refrigerator's appearance [Diagram 3] An explanatory diagram showing the internal structure of a refrigerator in a cross-sectional view from the right side. [Figure 4] An explanatory diagram showing the internal structure of the refrigerator compartment as a cross-sectional view from the front. [Diagram 5] A diagram explaining how to calculate the distance to an object from images captured by two cameras. [Figure 6] Control unit configuration diagram [Figure 7] 1 is a flowchart showing the overall processing by the control unit. [Figure 8] Flowchart of imaging process [Figure 9] Flowchart of analysis process of images taken at the time of storage [Figure 10] Flowchart of analysis process of images at the time of leaving the warehouse [Figure 11]First flow chart of food management process in warehouse [Figure 12] Second flow chart of food management process in warehouse [Figure 13] List of food in the warehouse and list of expiration dates DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] A refrigerator according to a first aspect of the present invention is a refrigerator comprising: a camera that captures an image of a food storage chamber and an area including an opening of the storage chamber; a food image acquisition unit that acquires, by the camera, an entry-time image which is an image of stored food stored in the storage chamber and an exit-time image which is an image of stored food exited from the storage chamber; and a food information management unit that generates an in-fridge food list in which information of each food stored in the storage chamber and an entry-time / exit date and time are recorded, based on entered food information in which information of the entered food based on the entry-time image and exit food information in which information of the exit food based on the exit-time image are recorded. When the acquisition unit acquires the image at the time of storage, it determines whether food has been removed from the storage chamber within a predetermined time based on the stored food list, and if food has been removed from the storage chamber within the predetermined time and the matching rate between the information of the food previously removed recorded in the stored food list and the information of the food currently stored in the stored food information is equal to or greater than a predetermined value, it determines that the food previously removed and the food currently stored in are the same item, and updates the information of the food previously removed recorded in the stored food list with the information of the food currently stored in the stored food information. According to the first invention, when an image of a food when it is stored is acquired, the food information management unit determines whether the food has been removed from the storage room within a predetermined time based on the stored food list, and if the food has been removed from the storage room within the predetermined time and the coincidence rate between the information of the food previously removed recorded in the stored food list and the information of the food currently stored in the stored food information is equal to or greater than a predetermined value, it determines that the food previously removed and the food currently stored are the same, and updates the information of the food previously removed recorded in the stored food list with the information of the food currently stored in the stored food information. Therefore, it is possible to determine the re-entry of a food removed from the storage room with a simple configuration, and to manage the information of the food stored in the storage room.
[0010] In a second invention, the food information includes the type and weight of the food, and the food information management unit is configured such that when the food previously shipped out and the food currently stored are of the same type of food, and the matching rate between the information of the food previously shipped out and the information of the food currently stored is equal to or greater than the specified value, but the weight of the food currently stored is greater than the weight of the food previously shipped out, the food information management unit determines that the food currently stored is a newly stored food, and does not update the information of the food previously shipped out recorded in the stored food list with the information of the food currently stored recorded in the stored food information. According to the second invention, when the food that was last shipped out and the food that is currently stored are the same type of food, and the matching rate between the information of the food that was last shipped out and the information of the food that is currently stored is equal to or greater than the specified value, but the weight of the food that is currently stored is greater than the weight of the food that was last shipped out, it is determined that the food that is currently stored is a newly stored food, thereby preventing the information of the food that was last shipped out, which is recorded in the food in stock list, from being updated with information of another food that is currently stored, which is recorded in the stored food information.
[0011] In a third invention, when the food information management unit determines, when the food image acquisition unit acquires the image at the time of storage, based on the stored food list, that no food has been removed from the storage room within a predetermined time, or when the rate of match between the information of the previously removed food recorded in the stored food list and the information of the currently stored food recorded in the stored food information is less than the predetermined value, the food information management unit is configured to newly register the currently stored food in the stored food list based on the information of the currently stored food recorded in the stored food information. According to the third invention, when it is determined based on the stored food list that no food has been removed from the storage room within a specified time, or when the rate of match between the information of the previously removed food recorded in the stored food list and the information of the currently stored food recorded in the stored food information is less than a specified value, the currently stored food can be newly registered in the stored food list.
[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiment.
[0013] [1. How the refrigerator is used] Fig. 1 shows a usage state of a refrigerator of this embodiment. Refrigerator 1 of this embodiment is installed in house H, and has a function of communicating with a management server 520 via a gateway 500 and a communication network 510. Refrigerator 1 also has a function of communicating with a terminal device (smartphone, tablet terminal, etc.) 400. Terminal device 400 has a function of communicating with management server 520 via the gateway 500 and the communication network 510.
[0014] A control unit 100 provided in the refrigerator 1 controls the overall operation of the refrigerator 1, and generates an internal food list 131 showing information about the foods stored in the storage compartments of the refrigerator 1 and sends it to the management server 520. A refrigerator management application executed on the terminal device 400 acquires internal food list 131 data from the management server 520, or from the refrigerator 1, and displays information about the foods stored in the refrigerator 1 based on the internal food list 131 on the display unit. A user P of the refrigerator 1 can check the status of the foods stored in the refrigerator 1 (food names, expiration dates, remaining amount, etc.) by visually checking the display on the terminal device 400.
[0015] [2. Refrigerator configuration] The configuration of refrigerator 1 will be described with reference to Fig. 2 to Fig. 6. Fig. 2 is an explanatory diagram showing the external appearance of refrigerator 1. As shown in Fig. 2, refrigerator 1 has a main box body 2 with an opening on the front, and inside main box body 2, there are formed a refrigeration compartment 10 (corresponding to a storage compartment of the present invention), an ice-making compartment 30, a switching compartment 20 adjacent to ice-making compartment 30 that can change the temperature inside, a freezer compartment 40, and a vegetable compartment 50.
[0016] A rotating right door 11 and a left door 12 (corresponding to the opening and closing part of the present invention) are provided at the opening on the front side of the refrigerator compartment 10. In addition, the switchable compartment 20, the ice-making compartment 30, the freezer compartment 40, and the vegetable compartment 50 are provided with drawers 21, 31, 41, and 51, respectively, for storing food.
[0017] Next, Fig. 3 is an explanatory diagram showing the configuration inside refrigerator 1 in a cross-sectional view seen from the right side, and Fig. 4 is an explanatory diagram showing the internal configuration of refrigerator compartment 10 in a cross-sectional view seen from the front. With reference to Figs. 3 and 4, refrigerator compartment 10 is provided with upper shelf 13, middle shelf 14, and lower shelf 15 which divide refrigerator compartment 10 into sub-areas. Also provided in refrigerator compartment 10 are lighting units 16a-16f which illuminate refrigerator compartment 10, illuminance sensors 17a-17c which detect the illuminance inside refrigerator compartment 10, left camera 60a, right camera 60b, and side camera 60c.
[0018] The left camera 60a is disposed on the left side near the top of the opening 10a of the refrigerator compartment 10 so that the opening 10a is included in its imaging range, and images the range from the opening 10a to the front ends of the shelves 13, 14, and 15. The right camera 60b is disposed on the right side near the top of the opening 10a of the refrigerator compartment 10 so that the opening 10a is included in its imaging range, and images the range from the opening 10a to the front ends of the shelves 13, 14, and 15. The side camera 60c is disposed on the left wall of the refrigerator compartment and images the inside of the refrigerator compartment 10 from the left wall side. By providing the side camera 60c, it becomes possible for the side camera 60c to image foods that cannot be imaged by the left camera 60a or the right camera 60b.
[0019] Also provided are compressor 61, cooling fan 62, cooler 63, and condenser 64, which are auxiliary machines that configure the refrigeration cycle. Furthermore, at the rear surface of refrigerator compartment 10, refrigerator compartment duct 70 and cold air outlets 71-73 for circulating cold air are arranged.
[0020] The refrigerator compartment 10 is provided with an opening / closing sensor 18b for detecting the opening / closing of the right door 11, and an opening / closing sensor 18a for detecting the opening / closing of the left door 12. The opening / closing sensor 18b outputs a closed detection signal when the right door 11 is closed, and outputs an open detection signal when the right door 11 is open. Similarly, the opening / closing sensor 18a outputs a closed detection signal when the left door 12 is closed, and outputs an open detection signal when the left door 12 is open.
[0021] An opening / closing sensor 22 that detects whether the opening 20a is open or closed is provided in the switching chamber 20. The opening / closing sensor 22 outputs a closed detection signal when the drawer 21 is accommodated in the switching chamber 20, and outputs an open detection signal when the drawer 21 is pulled out of the switching chamber 20.
[0022] Similarly, ice making compartment 30 is provided with opening / closing sensor 32 which detects opening / closing of opening 30a by drawer 31. Freezer compartment 40 is provided with opening / closing sensor 42 which detects opening / closing of opening 40a by drawer 41, and vegetable compartment 50 is provided with opening / closing sensor 52 which detects opening / closing of opening 50a by drawer 51. Similar to opening / closing sensor 22, opening / closing sensors 32, 42, 52 output a closed detection signal and an open detection signal.
[0023] 5 shows a situation in which the left camera 60a and the right camera 60b capture images of the same object F. In an image Va captured by the left camera 60a, Ga1 indicates the position of an image portion corresponding to a first point P1 of the object F, and Ga2 indicates the position of an image portion corresponding to a second point P2 of the object F. In an image Vb captured by the right camera 60b, Gb1 indicates the position of an image portion corresponding to the first point P1 of the object F, and Gb2 indicates the position of an image portion corresponding to the second point P2 of the object F.
[0024] By calculating the parallax between the left camera 60a and the right camera 60b relative to the first point P1 from the position (coordinates) (xa1, ya1) of Ga1 in the captured image Va and the position (xb1, yb1) of Gb1 in the captured image Vb, the distance from the left camera 60a and the right camera 60b to the first point P1 of the object F and the position of the first point P1 in real space can be calculated using known trigonometric techniques.
[0025] Similarly, by calculating the parallax between the left camera 60a and the right camera 60b with respect to the second point P2 from the position (xa2, ya2) of Ga2 in the captured image Va and the position (xb2, yb2) of Gb2 in the captured image Vb, the distance from the left camera 60a and the right camera 60b to the second point P2 of the object F and the position of the second point P2 in real space can be calculated by a known trigonometric method. In this way, the food size recognition unit 112 can determine the size of the object (external dimensions, volume, etc.) by calculating the positions of multiple points of the object F. Note that the configuration in which the food size recognition unit 112 recognizes the position of the food stored in the refrigerator compartment 10 corresponds to the storage position recognition unit of the present invention.
[0026] Next, Fig. 6 is a configuration diagram of the control unit 100. Referring to Fig. 6, the control unit 100 is an electronic circuit unit configured with a CPU (Central Processing Unit) 110, an image processing unit 120, a storage unit 130, a wireless communication unit 140, a date and time recognition unit 141, an interface circuit (not shown), and the like.
[0027] The CPU 110 executes the control program for the refrigerator 1 stored in the memory unit 130, and thereby functions as a food image acquisition unit 111, a food size recognition unit 112, a food type recognition unit 113, a food quantity recognition unit 114, a food information management unit 115, an imaging time setting unit 116, and a food density information acquisition unit 117.
[0028] The control unit 100 is connected to opening and closing sensors 18a, 18b, 22, 32, 42, 52, illuminance sensors 17a, 17b, 17c, left camera 60a, right camera 60b, side camera 60c, lighting units 16a, 16b, 16c, 16d, 16e, 16f, and refrigeration cycle accessories 61, 62, 63, 64.
[0029] Detection signals from the open / close sensors 18a, 18b, 22, 32, 42, 52 and the illuminance sensors 17a, 17b, 17c, video signals from the cameras 60a, 60b, 60c, etc. are input to the control unit 100. In addition, the operation of the lighting units 16a, 16b, 16c, 16d, 16e, 16f and the refrigeration cycle auxiliary units 61, 62, 63, 64, etc. are controlled by control signals output from the control unit 100.
[0030] Furthermore, the control signal output from the control unit 100 controls the imaging by the cameras 60a, 60b, and 60c via the image processing unit 120. The image processing unit 120 converts the video signals output from the cameras 60a, 60b, and 60c into digital gradation signals to generate captured images, and stores the captured image data in an image memory 121. The image processing unit 120 processes the captured images stored in the image memory 121 in accordance with the control signal output from the CPU 110.
[0031] The date and time recognition unit 141 recognizes the current date and time (date and time) by measuring a clock signal output from a clock circuit (not shown) provided in the control unit 100. Note that the current date and time may be recognized by acquiring information on the current date and time from the management server 520 through communication with the management server 520.
[0032] In addition to the data of in-fridge food list 131, data of received food information 132 which records information on foods received in refrigerator compartment 10, and sent food information 133 which records information on foods sent out of refrigerator compartment 10 are stored in memory unit 130. Furthermore, data of sample images 134 of various foods, a best-before date list 135 which shows general best-before dates of various foods, and a food density table 136 which associates food types with densities as shown in FIG.
[0033] The food image acquisition unit 111 captures images of food entering the refrigerator compartment 10 and food leaving the refrigerator compartment 10 using the left camera 60a, right camera 60b, and side camera 60c, and acquires images of the food.
[0034] The food size recognition unit 112 extracts an image of the same food from two images taken by the left camera 60a and the right camera 60b acquired at the same time by the food image acquisition unit 111, and recognizes the size of the food by calculating the distance and position based on the parallax, as described above, for the extracted image portion of the food.
[0035] The food type recognition unit 113 recognizes the type of food by recognizing characters contained in the food image extracted from the images captured by the left camera 60a, right camera 60b, and side camera 60c acquired by the food image acquisition unit 111, or by comparing the food image with a sample image.
[0036] The food quantity recognition unit 114 recognizes the weight of food stored in and removed from the refrigerator compartment 10 based on the food size recognized by the food size recognition unit 112 and the food type recognized by the food type recognition unit 113.
[0037] The food information management unit 115 records information on foods determined to be stored (stored foods) in stored food information 132, and records information on foods determined to be sent out (sent out foods) in sent out food information 133. Then, the food information management unit 115 updates the food information recorded in the in-warehouse food list 131 based on the stored food information 132 and the sent out food information 133.
[0038] The image capturing time setting unit 116 sets the time when images are captured by the cameras 60a, 60b, 60c from the time when the right door 11 or the left door 12 is switched from a closed state to an open state by the opening / closing sensors 18a, 18b until it is detected that both the right door 11 and the left door 12 have become closed. The image capturing time setting unit 116 sets a first image capturing time assuming the time when food to be put into the refrigerator compartment 10 passes through the opening 10a, and a second image capturing time assuming the time when food removed from the refrigerator compartment 10 passes through the opening 10a.
[0039] Food density information acquisition unit 117 reads the data of food density table 136 stored in storage unit 130, and makes food density table 136 available for reference.
[0040] [3. Processing by the control unit] Next, the process for managing food stored in the refrigerator compartment 10, which is executed by the control unit 100, will be described with reference to the flow charts shown in Figs.
[0041] 7 is a flowchart showing an overview of a series of processes executed by the control unit 100. The control unit 100 executes an "imaging process" in step S1, an "image analysis process" in step S2, and an "in-storage food management process" in step S3. Each process will be described in detail below.
[0042] [3-1. Imaging processing] The procedure for executing the "imaging process" will be described with reference to the flowchart shown in Fig. 8. The "imaging process" is executed by food image acquisition unit 111 and imaging time point setting unit 116. Food image acquisition unit 111 captures images using cameras 60a, 60b, 60c at a first imaging time point and a second imaging time point from when it recognizes in step S10 that one or both of right door 11 and left door 12 of refrigerator compartment 10 have switched from a closed state to an open state based on detection signals from opening / closing sensors 18a, 18b, until it recognizes in step S15 that both right door 11 and left door 12 have become closed.
[0043] When the food image acquisition unit 111 recognizes in step S10 that one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 have been switched from a closed state to an open state, the food image acquisition unit 111 advances the process to step S11. Steps S11 to S14 are processes performed by the imaging time point setting unit 116.
[0044] In step S11, the imaging time point setting unit 116 detects the illuminance of each of the upper shelf 13, middle shelf 14, and lower shelf 15 of the refrigerator compartment 10 (the illuminance of each space partitioned by the shelves) using the illuminance sensors 17a, 17b, and 17c. Then, the imaging time point setting unit 116 sets the detected illuminance of the upper shelf 13 as the upper reference illuminance, the illuminance of the middle shelf 14 as the middle reference illuminance, and the illuminance of the lower shelf 15 as the lower reference illuminance. The imaging time point setting unit 116 stores data of the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance in the storage unit 130.
[0045] The upper reference illuminance, the middle reference illuminance, and the lower reference illuminance are the illuminances of the shelves 13, 14, and 15 when one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 are opened and before an object (a user's hand, food, etc.) passes through the opening 10a of the refrigerator compartment 10. In consideration of errors and fluctuations in the detected illuminance, the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance may be set to be slightly lower or slightly higher than the detected illuminances of the illuminance sensors 17a, 17b, and 17c. In addition, the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S13 described later may be set to different values from the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S14.
[0046] In the next step S12, the imaging time point setting unit 116 detects the illuminance of the upper shelf 13, the middle shelf 14, and the lower shelf 15 using the illuminance sensors 17a, 17b, and 17c. Then, in the next step S13, the imaging time point setting unit 116 determines whether there is a shelf whose detected illuminance is lower than the reference illuminance by a predetermined level or more. Specifically, the imaging time point setting unit 116 determines whether each of the following conditions (1) to (3) is met.
[0047] (1) The illuminance detected by the illuminance sensor 17a has changed from the "upper reference illuminance" to "upper reference illuminance-α" or lower. (2) The illuminance detected by the illuminance sensor 17b has changed from the "medium reference illuminance" to "medium reference illuminance-α" or lower. (3) The illuminance detected by the illuminance sensor 17c has changed from the "lower reference illuminance" to "lower reference illuminance-α" or lower.
[0048] where α is a constant for setting a threshold determined by experiments, computer simulations, etc., assuming a drop in illuminance that occurs when an object (food, a user's hand, etc.) enters the upper shelf 13, the middle shelf 14, or the lower shelf 15. Note that α may be a common value for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, or may be set to different values individually for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance.
[0049] The imaging time point setting unit 116 determines that an object has passed through the opening 10a when at least one of the above (1) to (3) is met. Here, when at least one of the above (1) to (3) is met, it is assumed that the user has put their hand into the refrigerator compartment 10 to put food into the refrigerator compartment 10 (storing), to take food out of the refrigerator compartment 10 (removing), or to change the position of food in the refrigerator compartment 10. In this case, the imaging time point setting unit 116 can determine that an object (the user's hand, food held by the user's hand, etc.) has passed through the opening 10a of the refrigerator compartment 10.
[0050] Therefore, the imaging time point setting unit 116 sets the time point when it is determined in step S13 that at least one of the above (1) to (3) is established as the first imaging time point, and proceeds to step S20. The first imaging time point is set assuming the time point when food to be stored in the refrigerator compartment 10 passes through the opening 10a. Note that in step S13, a time point that is a predetermined time after the time point when it is determined by the imaging time point setting unit 116 that at least one of the above (1) to (3) is established may be set as the first imaging time point.
[0051] On the other hand, when none of the above (1) to (3) is satisfied, the imaging time point setting unit 116 advances the process to step S14. In step S14, the imaging time point setting unit 116 judges whether or not there is a shelf where the detected illuminance has changed from less than or equal to "reference illuminance-β" to more than or equal to "reference illuminance". Specifically, the imaging time point setting unit 116 judges whether or not each of the following conditions (4) to (6) is satisfied.
[0052] (4) The illuminance detected by the illuminance sensor 17a has changed from below the "upper reference illuminance-β" to above the "upper reference illuminance." (5) The illuminance detected by the illuminance sensor 17b has changed from below the "medium reference illuminance-β" to above the "medium reference illuminance." (6) The illuminance detected by the illuminance sensor 17c has changed from less than or equal to the "lower reference illuminance-β" to more than or equal to the "lower reference illuminance."
[0053] where β is a constant for determining a threshold value determined by experiments, computer simulations, etc., assuming an increase in illuminance that occurs when an object (food, a user's hand, etc.) that has entered the upper shelf 13, the middle shelf 14, or the lower shelf 15 passes through the opening 10a and leaves the refrigerator compartment 10. β may be set to a common value for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, or may be set to different values individually for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance.
[0054] When at least one of the above (4) to (6) is met, the imaging time point setting unit 116 determines that an object (food, the user's hand, etc.) that was in the refrigerator compartment 10 has left the refrigerator compartment 10 and passed through the opening 10a. Here, when at least one of the above (4) to (6) is met, it is assumed that the user has removed their hand that was in the refrigerator compartment 10 to put food into the refrigerator compartment 10 (storage), to take food out of the refrigerator compartment 10 (removal), or to change the position of the food contained in the refrigerator compartment 10. In this case, the imaging time point setting unit 116 can determine that an object (the user's hand, food held by the user's hand, etc.) has passed through the opening 10a of the refrigerator compartment 10.
[0055] Therefore, the imaging time point setting unit 116 sets the time point when it is determined in step S14 that at least one of the above conditions (4) to (6) is met as the second imaging time point, and proceeds to step S22. The second imaging time point is set assuming the time point when the food being removed from the refrigerator compartment 10 passes through the opening 10a. Note that in step S14, a time point that is a predetermined time after the time point when it is determined in step S14 that at least one of the above conditions (4) to (6) is met may be set as the second imaging time point.
[0056] On the other hand, when none of the above (4) to (6) is satisfied, the image capture time point setting unit 116 advances the process to step S15. Step S15 is a process by the food image acquisition unit 111, which determines whether or not both the right door 11 and the left door 12 of the refrigerator compartment 10 are closed based on the detection signals of the open / close sensors 18a, 18b. If both the right door 11 and the left door 12 are closed, the food image acquisition unit 111 advances the process to step S15 and ends the "image capture process". On the other hand, if at least one of the right door 11 and the left door 12 is open, the food image acquisition unit 111 advances the process to step S12.
[0057] Next, steps S20 to S21 and steps S22 to S23 are processes performed by food image acquisition unit 111. In step S20, food image acquisition unit 111 captures an image of an area including opening 10a of refrigerator compartment 10 using left camera 60a and right camera 60b via image processing unit 120, and captures an image of the inside of refrigerator compartment 10 from the left side using side camera 60c. In the following step S21, food image acquisition unit 111 stores the image data captured by cameras 60a, 60b, and 60c in step S20 in storage unit 130 as data of "image at time of storage", and proceeds to step S15. Note that imaging may be performed multiple times, and data of the multiple images may be stored in storage unit 130 as data of "image at time of storage".
[0058] Furthermore, in step S22, the food image acquisition unit 111, via the image processing unit 120, images an area including the opening 10a of the refrigerator compartment 10 using the left camera 60a and the right camera 60b, and images the interior of the refrigerator compartment 10 from the left side using the side camera 60c. In the following step S23, the food image acquisition unit 111 stores the image data captured by the cameras 60a, 60b, and 60c in step S22 in the storage unit 130 as "entering and leaving image" data, and proceeds to step S15. Note that the image capture may be performed multiple times, and the multiple image data may be stored in the storage unit 130 as "leaving image" data.
[0059] 8, an "entry image", which is an image captured near opening 10a at the time when food image acquisition unit 111 detects an object that has passed through opening 10a and entered any one of shelves 13, 14, or 15, is stored in storage unit 130. In addition, an "exit image", which is an image captured near opening 10a at the time when food image acquisition unit 111 detects an object that has left any one of shelves 13, 14, or 15 and passed through opening 10a, is stored in storage unit 130.
[0060] In addition, when an object passing through the opening 10a is detected multiple times in step 13 before it is detected in step S15 of Fig. 8 that the right door 11 and the left door 12 are closed, multiple "entry images" are captured and stored in steps S20 and S21. Then, an "image analysis process" to be described later is executed for each "entry image."
[0061] Similarly, when an object passing through the opening 10a is detected multiple times in step S14 before it is detected in step S15 of Fig. 7 that the right door 11 and the left door 12 are closed, multiple "leaving image" are captured and stored in steps S22 and S23. Then, an "image analysis process" to be described later is executed for each "leaving image".
[0062] [3-3. Image analysis processing] Next, the execution procedure of the "image analysis process" will be described according to the flowcharts shown in Figures 9 and 10. The "image analysis process" is executed by the food size recognition unit 112, the food type recognition unit 113, and the food quantity recognition unit 114. Figure 9 shows the process of obtaining information from an "image at the time of entry" for food entered into the refrigerator compartment 10. Also, Figure 10 shows the process of obtaining information from an "image at the time of exit" for food removed from the refrigerator compartment 10.
[0063] First, the "image analysis process" for the "image at time of storage" will be described with reference to Fig. 9. Steps S50 to S53 and steps S60 to S62 in Fig. 9 are processes performed by the food type recognition unit 113. In step S50, the food type recognition unit 113 executes character extraction process for the "image at time of storage" of the stored food by the image processing unit 120. Here, the stored food is the food determined to have been stored in the refrigerator compartment 10 by the food image acquisition unit 111 in step S33 in Fig. 8. In addition, the "image at time of storage" of the stored food is the "image at time of storage" captured and stored by the food image acquisition unit 111 in steps S20 and S21 in Fig. 8 for the food determined to have been stored in step S13 in Fig. 8.
[0064] In the next step S51, the food type recognition unit 113 determines whether or not characters have been extracted from the “storage-time image” by the image processing unit 120. If characters have been extracted, the food type recognition unit 113 proceeds to step S52, and if characters have not been extracted, the food type recognition unit 113 proceeds to step S60.
[0065] In step S52, the food type recognition unit 113 analyzes the text information extracted by the image processing unit 120. Then, in the following step S53, the food type recognition unit 113 obtains information on the stored food, such as the food name (indicating the type of food), expiration date, place of origin, and content volume, from the text information, and proceeds to step S54.
[0066] Furthermore, in step S60, the food type recognition unit 113 extracts an image portion of the stored food from the "storage-time image" by the image processing unit 120. In the following step S61, the food type recognition unit 113 calculates a matching rate between the extracted image portion of the stored food and sample images 134 of various foods stored in advance in the storage unit 130 (see FIG. 6).
[0067] In the next step S62, the food type recognition unit 113 estimates and obtains the food name of the sample image 134 with the highest matching rate as the food name of the stored food, and proceeds to step S54. By the processing of steps S60 to S62, even if characters cannot be extracted from the "image at the time of storage" of the stored food, it is possible to estimate the food name of the stored food and record it in the stored food information 132.
[0068] Step S54 is a process performed by food size recognition unit 112. Food size recognition unit 112 performs calculations using the method described above with reference to Fig. 5 on the image portion of the food extracted from the "stored image" to calculate the outer dimensions and storage position of the stored food.
[0069] The next step S55 is a process performed by food quantity recognition unit 114. Food quantity recognition unit 114 refers to food density table 136 to acquire the density of the food type recognized by food type recognition unit 113. As shown in FIG. 13, food density table 136 is a table that associates food names (food types) with densities, and is read from storage unit 130 by food density information acquisition unit 117. Note that food density information acquisition unit 117 may be configured to acquire data of food density table 136 stored in an external device such as management server 520 via wireless communication unit 140. Food quantity recognition unit 114 calculates the weight of the stored food by multiplying the volume calculated from the outer dimensions of the stored food by the density.
[0070] The next step S56 is processing by the food information management unit 115. The food information management unit 115 records the information of the stored food (type, outer dimensions, volume, weight, etc.) recognized by the food size recognition unit 112, the food type recognition unit 113, and the food quantity recognition unit 114 in "stored food information" 132, and ends the "image analysis process."
[0071] Next, the "image analysis process" for the "image at time of leaving the warehouse" will be described with reference to Fig. 10. Steps S70 to S73 and steps S80 to S82 in Fig. 10 are processes performed by the food type recognition unit 113. In step S70, the food type recognition unit 113 executes character extraction process on the "image at time of leaving the warehouse" of the leaving food by the image processing unit 120. Here, the leaving food refers to the "image at time of leaving the warehouse" that is captured and stored by the food image acquisition unit 111 in steps S22 and S23 in Fig. 8 for the food determined to have been left the warehouse in step S14 in Fig. 8.
[0072] In the next step S71, the food type recognition unit 113 determines whether or not characters have been extracted from the "image at the time of leaving the warehouse" by the image processing unit 120. If characters have been extracted, the food type recognition unit 113 proceeds to step S72, and if characters have not been extracted, the food type recognition unit 113 proceeds to step S80.
[0073] In step S72, the food type recognition unit 113 analyzes the text information extracted by the image processing unit 120. Then, in the following step S73, the food type recognition unit 113 obtains information about the shipped food, such as the food name, expiration date, place of origin, and content volume, from the text information, and proceeds to step S74.
[0074] In step S80, the food type recognition unit 113 extracts an image portion of the output food from the “output image” by the image processing unit 120. In the following step S81, the food type recognition unit 113 calculates a matching rate between the extracted image portion of the output food and sample images 134 of various foods stored in advance in the storage unit 130.
[0075] In the next step S82, the food type recognition unit 113 estimates and obtains the food name of the sample image 134 with the highest matching rate as the food name of the shipped food, and proceeds to step S74. By the processing of steps S80 to S82, even if characters cannot be extracted from the "image at time of shipment" of the shipped food, it is possible to estimate the food name of the shipped food and record it in the shipped food information 133.
[0076] Step S74 is a process performed by food size recognition unit 112. Food size recognition unit 112 performs calculations using the method described above with reference to Fig. 5 on the image portion of the food extracted from the "storage-time image" to calculate the outer dimensions and storage position of the outgoing food.
[0077] The following step S75 is processed by food quantity recognition unit 114. Food quantity recognition unit 114 refers to food density table 136 and obtains the density of the food type recognized by food type recognition unit 113. Food quantity recognition unit 114 multiplies the volume calculated from the outer dimensions of the output food by the density to calculate the weight of the output food.
[0078] Next, step S76 is processing by food information management unit 115. Food information management unit 115 records the information of the stored food (type, outer dimensions, volume, weight, etc.) recognized by food size recognition unit 112, food type recognition unit 113, and food quantity recognition unit 114 in "outgoing food information" 133, and ends the "image analysis process."
[0079] The image analysis process in Fig. 9 is performed on an image at the time of entry when entry into the refrigerator compartment 10 is determined, and the image analysis process in Fig. 10 is performed on an image at the time of exit when exit from the refrigerator compartment 10 is determined. Therefore, when the right door 11 or the left door 12 is detected by the food image acquisition unit 111 instead of food, it is not determined that an item has entered or left the refrigerator compartment 10, and therefore the image analysis process is not performed (the captured image is invalid). This makes it possible to prevent erroneous food information from being acquired from the image portion of the right door 11 or the left door 12.
[0080] [3-4. Food management in the warehouse] 11 and 12, the execution procedure of the "in-fridge food management process" for managing information on food stored in the refrigerator compartment 10 will be described. The "in-fridge food management process" is executed by the food information management unit 115. The food information management unit 115 includes the functions of the inventory quantity management unit and inventory position management unit of the present invention.
[0081] The food information management unit 115 manages information on the foods stored in the refrigerator compartment 10 using the stored food list 131 (see FIG. 5). As shown in FIG. 13, the stored food list 131 records the numbers (1, 2, 3, ...) given to the foods stored in the refrigerator compartment 10, and information on each stored food with a number, such as the food name, weight (first time, current, consumed amount), storage / removal date and time (first time stored, latest storage, latest storage), stock (1: present, 0: absent), expiration date, and place of origin. The food information management unit 115 generates and updates the stored food list 131 shown in FIG. 13 based on the stored food information 132 and the stored food information 133 recorded by the food information management unit 115.
[0082] In step S100 of Fig. 11, the food information management unit 115 determines whether the food image acquisition unit 111 has determined whether a food has been stored in the refrigerator compartment 10 or whether a food has been removed from the refrigerator compartment 10, and when storing or removing has been determined, the process proceeds to step S101. In step S101, the food information management unit 115 determines whether storing has been determined, and when storing has been determined, the process proceeds to step S102. On the other hand, when storing has not been determined (when removing has been determined), the food information management unit 115 proceeds to step S110.
[0083] In step S102, the food information management unit 115 determines whether or not food has been removed from the refrigerator compartment 10 within a predetermined time. If food has been removed from the refrigerator compartment 10 within the predetermined time, the process proceeds to step S103, and if food has not been removed from the refrigerator compartment 10 within the predetermined time, the process proceeds to step S111. Note that it is possible to recognize from the food in-fridge list 131 that food has been removed from the refrigerator compartment 10 within the predetermined time. The predetermined time is set to, for example, about 10 minutes, assuming a case in which the user uses some of the food removed from the refrigerator compartment 10 and returns it to the refrigerator compartment 10 (re-storing it).
[0084] In step S103, the food information management unit 115 judges whether the coincidence rate between the food previously sent out and the food currently stored is equal to or greater than a predetermined value. Here, the food information management unit 115 compares the information of the previously sent out product recorded in the in-store food list 131 with the information of the currently stored product recorded in the stored food information 132, and calculates the coincidence rate between the previously sent out food and the currently stored food from the coincidence status of the food name, expiration date, place of origin, etc.
[0085] Then, if the matching rate between the food previously sent out and the food currently stored is equal to or greater than a predetermined value, the food information management unit 115 proceeds to step S104, and if the matching rate between the food previously sent out and the food currently stored is less than the predetermined value, the food information management unit 115 proceeds to step S112. Here, if the food previously sent out and the food currently stored are the same type of food and the matching rate is equal to or greater than a predetermined value, but the weight calculated by the food quantity recognition unit 114 has increased since the previous time, it may be determined that the food is newly stored.
[0086] Step S104 is a process performed when the matching rate between the food previously taken out and the food currently stored is equal to or greater than a predetermined value, and it can be determined that the food previously taken out and the food currently stored are the same. In step S104, food information management unit 115 updates the current weight, latest stored date and time, and consumed amount (difference between the weight at the time of first storing and the current weight) of the food taken out before a predetermined time in storehouse food list 131 based on the information recorded in stored food information 132, and proceeds to step S105 in FIG. 12.
[0087] Step S112 is a process performed when the matching rate between the food that was taken out and the food that was currently stored is less than a predetermined value, and it can be determined that the food that was taken out last time is different from the food that was currently stored. In step S112, food information management unit 115 newly registers the stored food in in-store food list 131 based on the information recorded in stored food information 132, and proceeds to step S105 in FIG. 12.
[0088] Step S111 is a process to be performed when no food is removed from the warehouse within a predetermined time. In step S111, the food information management unit 115 newly registers the received food in the warehouse food list 131 based on the information recorded in the received food information 132, and proceeds to step S105 in FIG.
[0089] Step S110 is a process performed when it is determined that food has been removed from refrigerator compartment 10. In step S110, food information management unit 115 updates the current weight of the food (removed food) to 0 in stored food list 131. Food information management unit 115 also updates the amount consumed of the food in stored food list 131, updates the latest removal time based on the information recorded in removed food information 133, and proceeds to step 105 in FIG. 12.
[0090] 12, the food information management unit 115 determines whether or not there is any food whose current weight is 0 grams as a result of the processing in step S110. If there is any food whose current weight is 0 grams, the food information management unit 115 proceeds to processing in step S105, and if there is no food whose current weight is 0 grams, the food information management unit 115 proceeds to processing in step S120.
[0091] In step S106, the food information management unit 115 updates the information of the relevant food (food determined to have been taken out this time) in the stored food list 131 to 'out of stock' (stock 0), and proceeds to step S107. Also, in step S120, the food information management unit 115 updates the information of the relevant food (food determined to have been received this time) in the stored food list 131 to 'in stock' (stock 1), and proceeds to step S107.
[0092] In step S107, food information management unit 115 determines whether or not expiration date information based on character recognition is recorded in stored food information 132. If expiration date information is recorded in stored food information 132, food information management unit 115 proceeds to step S108, and if expiration date information is not recorded, food information management unit 115 proceeds to step S121.
[0093] In step S108, food information management unit 115 registers the expiration date of the food currently stored in storehouse food list 131 based on the expiration date information recorded in stored food information 132, and proceeds to step S109. Also, in step S121, food information management unit 115 refers to expiration date list 135 stored in memory unit 130 to search for the general expiration date of the relevant food.
[0094] As shown in Fig. 13, the expiration date list 135 is a list in which food names are associated with the expiration dates of the foods, and the food information management unit 115 searches for the expiration date associated with the food name of the currently stored food. In the following step S122, the food information management unit 115 registers the expiration date in the data of the corresponding food in the in-warehouse food list 131, and proceeds to step S109.
[0095] In step S109, the food information management unit 115 transmits the updated data of the in-storage food list 131 to the management server 520 via the communication network 510 using the wireless communication unit 140, and ends the "in-storage food management process."
[0096] The management server 520 stores the data of the stored food list 131 received from the refrigerator 1 in a storage device (not shown), and transmits the consumption status of each product recognized from the stored food list 131, the number of days remaining until the expiration date, etc. to the user's terminal device 400. This allows the user to check the status of the products stored in the refrigerator 1 using the terminal device 400 even when away from home, etc.
[0097] [4. Other embodiments] In the above embodiment, the left camera 60a, the right camera 60b, and the side camera 60c are provided as the multiple cameras, but the present invention can be applied as long as at least two cameras that capture images of food from different directions are provided. Also, a configuration with three or more cameras may be used.
[0098] In the above embodiment, the food amount recognition unit 114 calculates the weight as the amount of food, but it may also calculate the volume.
[0099] In the above embodiment, the food image acquisition unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10 based on the change in illuminance in the refrigerator compartment 10 detected by the illuminance sensors 17a, 17b, and 17c. As another configuration, an object sensor (such as a reflective or transmissive optical sensor, ultrasonic sensor, or capacitance sensor) that includes the opening 10a in its detection range and directly detects an object passing through the opening 10a may be used. In this case, the detection range of the object sensor may be set in front of the imaging ranges of the left camera 60a and the right camera 60b in the direction from the opening 10a toward the inside of the refrigerator compartment 10. This makes it possible to prevent failure to image the food to be stored due to a delay time from the time when an object passing through the opening 10a is detected until the left camera 60a and the right camera 60b perform imaging.
[0100] Moreover, when the opening / closing sensor 18a detects that the left door 12 is opened, and when the opening / closing sensor 18b detects that the right door 11 is opened, power may be supplied to the cameras 60a, 60b, and 60c to enable imaging by the cameras 60a, 60b, and 60c. With this configuration, when the left door 12 and the right door are closed, power supply to the cameras 60a, 60b, and 60c is cut off to save power. Also, by starting power supply to the cameras 60a, 60b, and 60c at the timing when the right door or the left door 12 is opened, it is possible to prevent failure to image the food being stored due to a delay in starting the cameras 60a, 60b, and 60c.
[0101] The food image acquisition unit 111 may be configured to invalidate the image data captured by the cameras 60a, 60b, and 60c when the opening / closing sensor 18a detects that the left door 12 has switched from an open state to a closed state, or when the opening / closing sensor 18b detects that the right door 11 has switched from an open state to a closed state, within a predetermined time from the time of detecting an object passing through the opening 10a. With this configuration, when the left door 12 or the right door 11 is detected and captured by the cameras 60a, 60b, and 60c when the left door 12 or the right door 11 is closed, it is possible to prevent erroneous food information from being extracted from the image of the left door 12 or the right door 11.
[0102] In the above embodiment, the images captured by the cameras 60a, 60b, and 60c are processed by the image processor 120. However, the functions of the image processor 120 may be provided in the CPU 110, and the CPU 110 may process the captured images.
[0103] In the above embodiment, the data of the sample image 134 and the expiration date list 135 are stored in the memory unit 130, but they may also be stored in the management server 520 and downloaded from the management server 520 to the control unit 100 of the refrigerator 1 when used.
[0104] Also, although sample image 134 is stored in memory unit 130 or management server 520, an image recognition program that has learned the characteristics of each food or stored item in advance, such as color, shape, size, etc., may be implemented in CPU 110, and the image portion of the stored food may be processed by the image recognition program to calculate the match rate and obtain an estimated food name of the stored food. This makes sample image 134 unnecessary, making it possible to reduce the storage capacity stored in memory unit 130 or management server 520 and also shortens the recognition processing time for estimating the food name of the stored food. The food name of the outgoing food may also be estimated in the same manner as above.
[0105] In the above embodiment, for food entering and leaving refrigeration compartment 10 of refrigerator 1, images were acquired by food image acquisition unit 111, the type was recognized by food type recognition unit 113, the size was recognized by food size recognition unit 112, and the weight was recognized by food amount recognition unit 114. For switchable compartment 20, freezer compartment 40, and vegetable compartment 50, the type, size, and weight of food can also be recognized by capturing images of food passing through openings 20a, 40a, 50a with multiple cameras. [Industrial Applicability]
[0106] As described above, the refrigerator according to the present invention can be applied to applications in which the refrigerator determines the re-entry of food that has been removed from the storage compartment with a simple configuration and manages information about the food stored in the storage compartment. [Explanation of symbols]
[0107] 1. Refrigerator 10 Refrigerator 10a opening 11 Right door 12 Left door 16a, 16b, 16c, 16d, 16r, 16f Lighting section 17a, 17b, 17c Light sensor 18a, 18b Open / close sensor 20 Switching room 30 Ice Room 40 Freezer 50 Vegetable compartment 60a Right Camera 60b Left camera 60c Side Camera 61 Compressor 62 Cooling fan 63 Cooler 64 Condenser 100 Control Unit 110 CPU 111 Food Image Acquisition Unit 112 Food size recognition unit 113 Food Type Recognition Division 114 Food amount recognition section 115 Food Information Management Department 116 Imaging time setting unit 117 Food density information acquisition department 120 Image Processing Unit 121 Image Memory 130 Storage section 131 Food List 132 Stocked Food Information 133 Food shipment information 134 Sample Images 135 Best Before List 136 Food Density Table 140 Wireless Communication Division 141 Date and time recognition section 400 Terminal Equipment 500 Gateway 510 Communication Network 520 Management Server
Claims
1. [0023] A food containment chamber; a left camera disposed on the left side of an upper portion of an opening of the storage chamber and configured to capture an image of the inside of the storage chamber from the left side of the upper portion of the opening; a right camera disposed on the right side of an upper portion of the opening and configured to capture an image of the inside of the accommodation chamber from the right side of the upper portion of the opening; A side camera is disposed on a side wall of the storage chamber and captures an image of the inside of the storage chamber from the side wall side; a food size recognition unit that recognizes the size of the target food by calculating a parallax between the left camera and the right camera for a plurality of locations of the target food from images of the same target food captured by the left camera and the right camera, and calculating real space positions of the plurality of locations based on the parallax between the left camera and the right camera for the plurality of locations; a food amount recognition unit that recognizes an amount of the target food based on the size of the target food recognized by the food size recognition unit; a food type recognition unit that recognizes the type of the target food from the image of the target food extracted from the images captured by the left camera, the right camera, and the side camera; A refrigerator equipped with
2. A food density information acquisition unit that acquires information from a food density table that associates food types with densities, the food size recognition unit recognizes the volume of the target food as the size of the target food, The food quantity recognition unit recognizes, as the quantity of the food, a weight obtained by multiplying the volume of the food recognized by the food size recognition unit by a density obtained by applying the type of food determined by the food type recognition unit to the food density table.
2. The refrigerator according to claim 1.
3. An inventory management unit that manages the amount of food recognized by the food amount recognition unit. The refrigerator according to claim 1 or 2.
4. A storage position recognition unit that recognizes the position of the food in the storage chamber based on the parallax; an inventory position management unit that manages the position of the food in the storage chamber recognized by the storage position recognition unit; The refrigerator according to any one of claims 1 to 3.
5. The food type recognition unit recognizes the type of the target food by recognizing characters included in the image of the target food extracted from the images captured by the left camera, the right camera, and the side camera. The refrigerator according to any one of claims 1 to 4.
Citation Information
Patent Citations
Device, method, and recording medium for three- dimensional body recognition including feedback process
JP2001175860A
Automatic recipe creation apparatus and program
JP2010061382A
Storage device
JP2012193873A
Refrigerator and camera device
JP2015111025A
Shed management device and method
JP2015152211A
