Inventory management system
The inventory management system addresses high computational load by using RFID tags and image similarity calculation to efficiently link captured images with food information, enhancing inventory management accuracy and efficiency.
Patent Information
- Application Number
- JP2023079838
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2024-11-27
AI Technical Summary
Existing inventory management systems face high computational load due to the large number of images being processed to identify and correlate food items in circulation, leading to inefficiencies in linking captured images with food information.
An inventory management system that utilizes RFID tags to read IDs on food materials, generates an inventory list, calculates similarity based on camera images and stored food information, and integrates information exceeding a predetermined threshold, reducing the amount of calculation required.
The system effectively links captured images with food information using a reduced computational load, improving accuracy and efficiency in managing food inventory.
Smart Images

Figure 2024164393000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an inventory control system. [Background technology]
[0002] In order to reduce food waste caused by excess stock of ingredients, inventory management systems for ingredients stored in refrigerators and other storage facilities have been proposed. Some inventory management systems use cameras to take pictures of the inside of the storage facility and allow the captured images to be viewed on a smartphone or other device. Furthermore, some inventory management systems manage inventory based on tags such as RFID (Radio Frequency Identifier) attached to ingredients.
[0003] Patent Document 1 is also known as a technology for generating item management information (inventory list) in table format based on captured images of items and read tag information, and for managing inventory based on the information. Patent Document 1 also states that "the image item identification unit 236 executes a similar image search based on the image of item 10 captured by the imaging unit 224 for multiple images in association data in which multiple item names are associated with multiple images. Then, when the similarity of the image with the highest similarity is greater than a predetermined threshold, the image item identification unit 236 acquires the item name associated with the image, and notifies the item management information update unit 232" (paragraph 0036). [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2015-125535 A Summary of the Invention [Problem to be solved by the invention]
[0005] The technology described in Patent Document 1 searches for images similar to an article image captured by a camera, identifies the article name from images with high similarity, and associates them with the article name identified from tag information. The objects searched for at this time are images related to articles in circulation, but since the number of such images is enormous, the load of calculating the similarity becomes high.
[0006] An object of the present invention is to provide an inventory management system that can link captured images with ingredient information stored in an inventory list with a small amount of calculation. [Means for solving the problem]
[0007] In order to solve the above-mentioned problem, the present invention provides an inventory management system that manages inventory of ingredients using a reading unit that reads an ID affixed to an ingredient and a camera that captures a first image including the ingredient, and is characterized by comprising: an inventory list generation unit that generates an inventory list in which ingredient information corresponding to the ID is stored based on the reading result by the reading unit; a similarity calculation unit that calculates a similarity based on the first image captured by the camera and the ingredient information stored in the inventory list; and an integration unit that links the ingredient information whose similarity is equal to or greater than a predetermined threshold to the first image. Effect of the Invention
[0008] According to the present invention, it is possible to provide an inventory management system that can link a captured image with food ingredient information stored in an inventory list with a small amount of calculation. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiment of the invention. [Brief description of the drawings]
[0009] [Figure 1] Bird's-eye view of the storage facility structure [Diagram 2] Functional block diagram of inventory management according to the first embodiment [Diagram 3] An example of an inventory list according to the first embodiment [Figure 4A] An example of an inventory management app screen (when one of similar ingredients is tapped) [Figure 4B] An example of an inventory management app screen (when a similar ingredient is tapped on the other ingredient) [Figure 4C] An example of an inventory management app screen (when one of the same ingredients is tapped) [Figure 4D] An example of an inventory management app screen (when the same ingredient is tapped on another ingredient) [Diagram 5] A flow diagram showing the operation of inventory management according to the first embodiment. [Figure 6] Functional block diagram of inventory management according to the second embodiment [Figure 7] An example of an inventory list according to the second embodiment [Figure 8] Functional block diagram of inventory management according to the third embodiment [Figure 9] A flow diagram showing the operation of inventory management according to the third embodiment. [Figure 10] FIG. 10 is a flow chart showing details of step S120b in FIG. [Figure 11] Flow diagram showing the operation of inventory management according to the fourth embodiment [Figure 12] An example of an inventory list according to the fifth embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The examples are illustrative for explaining the present invention, and are omitted and simplified as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.
[0011] In addition, in the embodiment of the invention, a refrigerator is used as an example of a storage facility, but other storage facilities such as a storage shelf or a stocker may also be used.
[0012] When there are multiple components having the same or similar functions, they may be described by using the same reference numerals with different subscripts, or when there is no need to distinguish between these multiple components, the subscripts may be omitted. EXAMPLES
[0013] The structure of a storehouse according to a first embodiment of the present invention will be described with reference to FIG.
[0014] FIG. 1 is a bird's-eye view showing the structure of a storage unit. 1 is a refrigerator, 10 is a revolving door storage chamber provided in the refrigerator 1, 11 is a left door for sealing the revolving door storage chamber 10, 12 is a right door for sealing the revolving door storage chamber 10, 20 is a first drawer door storage chamber, 21 is a second drawer door storage chamber, 30 is a third drawer door storage chamber, 40 is a fourth drawer door storage chamber, 100 is an inside-fridge antenna, 110 is an outside-fridge antenna, and 120a is a camera. In FIG. 1, the left door 11 and the right door 12 of the revolving door storage chamber are closed, and the first drawer door storage chamber 20 to the fourth drawer door storage chamber 40 are also closed.
[0015] The first pull-out door storage chamber 20 and the second pull-out door storage chamber 21 are installed at the same height, the third pull-out door storage chamber 30 is installed at a lower position than the first pull-out door storage chamber 20 and the second pull-out door storage chamber 21, and the fourth pull-out door storage chamber 40 is installed at a lower position than the third pull-out door storage chamber 30. Also, the revolving door storage chamber 10 is installed at a higher position than the first pull-out door storage chamber 20 and the second pull-out door storage chamber 21. Here, the revolving door storage chamber 10 is a refrigerator chamber, the first pull-out door storage chamber 20 is an ice-making chamber, the second pull-out door storage chamber 21 is a freezer chamber, the third pull-out door storage chamber 30 is a freezer chamber, and the fourth pull-out door storage chamber 40 is a vegetable chamber. The above classification of the storage compartments is merely an example, and the third drawer door storage compartment 30 may be a vegetable compartment and the fourth drawer door storage compartment 40 may be a freezer compartment.
[0016] The internal antenna 100 is installed on the internal ceiling of the revolving door storage chamber 10, and the external antenna 110 and camera 120a are installed on a protrusion installed on the top of the refrigerator 1 so as to protrude forward from the refrigerator 1 even when the left door 11 and right door 12 are open. The internal antenna 100 has a reading area set so as to read an RFID tag present inside the revolving door storage chamber 10, and the external antenna 110 has a reading area set so as to read an RFID tag present outside the revolving door storage chamber 10. The camera 120a is set so as to be able to capture an image of at least the inside of the revolving door storage chamber 10 from the front when food ingredients are stored, and obtains an image (first image) including the food ingredients.
[0017] Next, the inventory management operation according to the first embodiment of the present invention will be described with reference to FIGS.
[0018] FIG. 2 is a functional block diagram regarding inventory management in revolving door storage room 10, FIG. 3 is an example of an inventory list, FIGS. 4A to 4D are examples of inventory management result displays provided to the user, and FIG. 5 is a system operation flow regarding inventory management in revolving door storage room 10.
[0019] In FIG. 2, 200 is a reading unit, 210 is a door opening / closing information generating unit, 220 is an inventory list generating unit, 230 is an ID-ingredient information database, 240 is a similarity calculating unit, 250 is an integrating unit, 260 is a display unit, 300 is a camera image (first image), and 410 is an external image of the ingredient (second image). The reading unit 200 reads the RFID tag attached to the ingredient and includes an internal antenna 100 and an external antenna 110. The door opening / closing information generating unit 210 includes a sensor that detects the opening and closing of a door such as the revolving door storage room 10. The inventory list generating unit 220 generates an inventory list that stores ingredient information corresponding to the ID based on the reading result by the reading unit 200. The ID-ingredient information database 230 is a database in which ingredient information corresponding to each ID attached to ingredients in circulation is recorded. The similarity calculation unit 240 calculates the similarity based on the camera image 300 captured by the camera 120a and the ingredient information (e.g., exterior image 410) stored in the inventory list. The integration unit 250 links ingredient information whose similarity is equal to or exceeds a predetermined threshold to the camera image 300. The display unit 260 displays the camera image 300 taken inside a storage room such as the revolving door storage room 10, the inventory list, and the like, and when an area of a specific ingredient in the camera image 300 is specified (tapped), the ingredient information linked by the integration unit 250 is displayed superimposed on the camera image 300.
[0020] In Fig. 3, 400 is an inventory list, 401a is ingredient information corresponding to the ID, 410 is an external image of the ingredient corresponding to the ID, 420 is the name of the ingredient corresponding to the ID, 430 is the storage location of the ingredient (revolving door storage compartment 10, first drawer door storage compartment 20 to fourth drawer door storage compartment 40), and 440 is the expiration date (or best before date) of the ingredient. In other words, ingredient information 401a consisting of at least these four elements is stored in inventory list 400 for each ingredient stored in refrigerator 1. The external image 410, ingredient name 420, and expiration date 440 are stored by inventory list generating unit 220 by checking the ID read by reading unit 200 against ID-ingredient information database 230. For the sake of simplicity, these four elements are listed here as the elements that make up the inventory list; however, other elements such as the date and time when the food item was first stored in refrigerator 1, as well as the country of origin and producer linked to the RFID tag ID may also be included.
[0021] In FIG. 4A to FIG. 4D, 600 is the app screen, 700 is the user tapping the app screen, 500a to 500c are the contents of the food candidate corresponding to the food position (area) in the camera image 300a or the camera image 300b, and 510a to 510d are the food. In FIG. 4A and FIG. 4B, the camera image 300a is the same, and different food ingredients (food ingredient 500a is XXX natto, food ingredient 500b is XXX tofu) are stored although the packaging is similar. In FIG. 4C and FIG. 4D, the camera image 300b is the same, and multiple food ingredients are stored that are the same food ingredients but have different expiration dates (or best-before dates) (food ingredients 500c and 500d are XXX natto). In FIG. 4A, the user taps food ingredient 500a, in FIG. 4B the user taps food ingredient 500b, in FIG. 4C the user taps food ingredient 500c, and in FIG. 4D the user taps food ingredient 500d.
[0022] In Figs. 4A to 4D, since there are multiple (two) pieces of ingredient information with similarity levels equal to or higher than a predetermined value, the display unit 260 displays each piece of ingredient information as candidates, arranged vertically. Also, in Figs. 4A and 4B, ingredients with high similarity are given priority and displayed at the top. Furthermore, in Figs. 4C and 4D, ingredients with a close expiration date (or best before date) may be given priority and displayed at the top. In this way, since there is no strict one-to-one correspondence between the positions of ingredients in the camera image and the ingredient information, it is possible to manage the inventory of ingredients even when there are multiple similar or identical ingredients.
[0023] The inventory management operation flow according to the first embodiment of the present invention will be described with reference to FIG. 5. First, it is determined whether the door opening / closing information generating unit 210 outputs "open" information (step S100). When the user opens the left door 11 or the right door 12 of the revolving door storage room 10, the door opening / closing information generating unit 210 outputs "open" information, so the reading unit 200 starts reading the RFID tag and outputs at least the tag ID, the reading time, and the name of the antenna that was read (inside antenna 100 or outside antenna 110) (step S110). Also, when the door opening / closing information generating unit 210 outputs "open" information, the camera 120a starts taking pictures and outputs at least a camera image (step S120a).
[0024] Next, it is determined whether the door opening / closing information generating unit 210 outputs "closed" information (step S130). If the "closed" information is not output even after a predetermined time has elapsed, the process returns to step S110 and step S120a, and the reading of the RFID tag and the taking of the image are repeated.
[0025] On the other hand, when the user closes the left door 11 and the right door 12 of the revolving door storage room 10, the door opening / closing information generating unit 210 outputs "closed" information, and the inventory list generating unit 220 performs inventory determination based on the RFID tag reading result output from the reading unit 200 (step S140). Specifically, if the RFID tag with a certain ID is read by the inside antenna until a time later than the outside antenna, it is determined to be "in stock", and conversely, if the outside antenna is read until a time later than the inside antenna, it is determined to be "out stock". Also, if it is only read by the inside antenna, it is determined to be "in stock", and conversely, if it is only read by the outside antenna, it is determined to be "out stock". Furthermore, if the ID is not read by either antenna but exists in the inventory list, it is determined to be "in stock". And, if the ID is read by the inside antenna until a time later than the outside antenna but exists in the inventory list, it is also determined to be "in stock".
[0026] If the result of step S140 is "in stock" or "in stock," the inventory list generation unit 220 checks the ID information held by the RFID tag of the food ingredient against the ID-food ingredient information database 230, and generates the inventory list 400 (step S150). If the result of step S140 is "out stock," the inventory list generation unit 220 deletes the information about the food ingredient from the inventory list 400. This information about the deleted food ingredient may be managed as another list, such as an "out stock list" or a "history list" that contains the history of storage of the food ingredient in the refrigerator 1.
[0027] After that, the similarity calculation unit 240 compares the camera image 300 output in step S120a with the appearance image 410 stored in the inventory list 400 generated in step S150, and calculates the similarity (step S160). Furthermore, the integration unit 250 links the ingredient information 401a whose similarity is equal to or greater than a preset threshold to a pixel area in the camera image 300 in which an ingredient is captured, and outputs the information as the ingredient candidate 500 corresponding to the ingredient position (step S170). Finally, when the user taps a pixel area in the camera image 300 in which an ingredient is captured, the display unit 260 displays the ingredient candidate 500 linked to the area on the app screen (step S180).
[0028] With the above system operation, when different ingredients are stored in a refrigerator that are similar in packaging, if the user taps (specifies) a pixel area in the camera image where the ingredient is captured, candidate ingredient information is displayed, and the user can be provided with more detailed ingredient information based on the camera image (Figs. 4A and 4B). Even when multiple ingredients that are the same but have different expiration dates (or best before dates) are stored, if the user taps (specifies) a pixel area in the camera image where the ingredient is captured, candidate ingredient information is displayed, and the user can be provided with more detailed ingredient information based on the camera image (Figs. 4C and 4D). Furthermore, the number of ingredients stored in the refrigerator 1, that is, the number of ingredient information 401a in the inventory list 400, is at most on the order of 100, which is overwhelmingly smaller than the number of ingredients (products) in circulation in the world. This increases the accuracy of calculating the similarity between the camera image and the appearance image of the ingredient linked to the ingredient information, while reducing the amount of calculation.
[0029] In addition, in the operation of comparing the camera image 300 with the appearance image 410 of the ingredient stored in the inventory list 400, the similarity calculation unit 240 according to the first embodiment of the present invention may preselect ingredient information 401a corresponding to the ingredient stored in the storage room (revolving door storage room 10 in this embodiment) photographed by the camera 120a from the inventory list 400, and compare the ingredient information 401a (appearance image 410) with the camera image 300. In this case, the ingredient information 401a to be compared is reduced compared to the case where no selection is made, which has the effect of improving the accuracy of the comparison result and reducing the amount of calculation.
[0030] In addition, in the first embodiment of the present invention, the internal antenna for reading RFID tags is installed only in the revolving door storage chamber 10, but it may also be installed in the first drawer door storage chamber 20 to the fourth drawer door storage chamber 40 to generate an inventory list of ingredients stored in the first drawer door storage chamber to the fourth drawer door storage chamber. In addition, the camera 120a is installed so as to capture images (camera images) of ingredients stored at least inside the revolving door storage chamber 10, but it may also be installed so as to capture images (camera images) of ingredients stored inside the first drawer door storage chamber 20 to the fourth drawer door storage chamber 40. In this case, there is an advantage that the management range of ingredients can be expanded to the entire refrigerator 1.
[0031] The camera image to be linked to the ingredient information may be either a raw image captured by a camera or an image that has been processed from a raw image captured by a camera. Furthermore, the camera image to be linked to the ingredient information may be a video captured by a camera and processed into an image.
[0032] It is preferable that the door open / close information generating unit 210 detects the door open / closed state by installing a magnetic sensor on the storage compartment side and a magnet on the door side, and determines whether the door is open or closed when the change in the output power of the magnetic sensor exceeds a predetermined threshold. However, the structure is not limited to the magnetic sensor, and may be an angular velocity sensor or acceleration sensor installed on the revolving door, an acceleration sensor installed on the drawer door, or a TOF (Time of Flight) sensor that detects the distance between the drawer door and the storage compartment, and determine whether the door is open or closed when the change in the sensor output exceeds a predetermined threshold. Also, a temperature sensor or humidity sensor may be installed in each storage compartment, and determine whether the door is open or closed when the change in temperature or humidity exceeds a predetermined threshold. EXAMPLES
[0033] The operation of the storehouse according to the second embodiment of the present invention will be described with reference to FIGS. FIG. 6 is a functional block diagram relating to inventory management in the revolving door storage room 10, and FIG. 7 is an example of an inventory list.
[0034] In Fig. 6, 270 is a feature extraction unit that extracts first feature information from camera image 300, and 310 is camera image 300 to which the first feature information has been added. Here, the feature information is information on the type of food ingredient (dairy products, processed meats, paste products, etc.), information on the food ingredient's packaging (PET bottles, cans, bottles, paper cartons, etc.), character recognition information printed on the food ingredient, etc. Furthermore, inventory list generation unit 220 compares with ID-food ingredient information database 230 and generates inventory list 400 including the second feature information as food ingredient information. In Fig. 7, 461 is food ingredient type information, and 462 is food ingredient packaging information, which correspond to second feature information 460.
[0035] In the second embodiment of the present invention, the similarity calculation unit 240 can use the comparison result between the first feature information extracted by the feature extraction unit 270 and the second feature information 460 stored in the inventory list 400 when calculating the similarity between the camera image 300 and the appearance image 410. At this time, the similarity calculation unit 240 can also use the ingredient name 420 stored in the inventory list 400 to calculate the similarity. In the second embodiment of the present invention, the number of elements for calculating the similarity is increased compared to the first embodiment, which has the effect of improving the accuracy of the similarity calculation. EXAMPLES
[0036] The operation of the storehouse according to the third embodiment of the present invention will be described with reference to FIGS. FIG. 8 is a functional block diagram relating to inventory management in revolving door storage room 10, FIG. 9 is a system operation flow relating to inventory management in revolving door storage room 10, and FIG. 10 is an operation flow relating to the internal operation of step S120b in FIG.
[0037] In Fig. 8, 120b is a camera, 320 is a first storage time determined based on a video captured by the camera that an ingredient has been stored, 330 is a camera image in which the first storage time 320 is linked to a pixel area in which a tracked object (ingredient) is captured, 450 is a second storage time determined by the inventory list generation unit 220 as the "storage" of the ingredient based on an inventory determination based on the RFID tag reading result, 280 is an inventory list candidate narrowing down unit, and 410a is an external image of the narrowed down ingredient. Note that the second storage time 450 is also stored as ingredient information in the inventory list in the third embodiment of the present invention.
[0038] An inventory management operation flow according to the third embodiment of the present invention will be described with reference to Figs. 9 and 10. First, it is determined whether the door opening / closing information generating unit 210 outputs "open" information (step S100). When the user opens the left door 11 or the right door 12 of the revolving door storage room 10, the door opening / closing information generating unit 210 outputs "open" information, so the reading unit 200 starts reading the RFID tag and outputs at least the tag ID, the reading time, and the name of the read antenna (inside antenna 100 or outside antenna 110) (step S110). Also, when the door opening / closing information generating unit 210 outputs "open" information, the camera 120b starts taking pictures and outputs at least a camera image (step S120b).
[0039] Next, the internal operation flow of step S120b will be described with reference to FIG. 10. After the camera 120b starts acquiring images, if it recognizes a part of the body corresponding to the user's hand, it recognizes an object present near the part of the body corresponding to the user's hand (steps S200 to S220). Furthermore, it tracks the object in the middle of entering (step S230), and judges whether the distance between the tracked object and the part of the body corresponding to the user's hand exceeds a threshold (step S240). If the distance exceeds the threshold, it is judged that the tracked object has been stored in the revolving door storage room 10, and time information is acquired as the first entry time (step S250). Furthermore, after acquiring a camera image 300 of the revolving door storage room 10 viewed from the front direction, the first entry time is linked to the pixel area in which the object is captured in the camera image 300 (steps S260 to S270). After that, it outputs a camera image 330 in which the first entry time 320 is linked to the pixel area in which the tracked object is captured (step S280).
[0040] 9, the operation flow after step S130 will be described. In step S130, it is determined whether the door open / close information generating unit 210 outputs "closed" information. If the "closed" information is not output even after a predetermined time has elapsed, the process returns to step S110 and step S120b, and the reading of the RFID tag and the capture of the image are repeated.
[0041] On the other hand, when the user closes the left door 11 and the right door 12 of the revolving door storage room 10, the door opening / closing information generating unit 210 outputs "closed" information, and the inventory list generating unit 220 performs an inventory determination based on the RFID tag reading results output from the reading unit 200 (step S140).
[0042] If the determination in step S140 is "entered" or "in stock," the inventory list generation unit 220 compares the ID information in the RFID tag of the target ingredient with the ID-ingredient information database 230, and generates the inventory list 400 (step S150a). In this step S150a, the inventory list generation unit 220 assigns the time when the inventory determination was performed to the ingredient information as a second entry time. If the determination in step S140 is "released," the inventory list generation unit 220 deletes the information about the target ingredient from the inventory list 400.
[0043] Next, the inventory list candidate narrowing unit 280 compares the first arrival time obtained from the camera image 330 with the second arrival time added to the inventory list 400, and selects from the inventory list 400 ingredient information for which the difference in time is equal to or less than a predetermined threshold (step S190).
[0044] After that, the similarity calculation unit 240 compares the camera image 300 output in step S120b with the appearance image 410a corresponding to the ingredient information selected in step S190, and calculates the similarity (step S160a). Furthermore, the integration unit 250 links the ingredient information whose similarity is equal to or exceeds a preset threshold to a pixel area in the camera image 300 in which an ingredient is captured, and outputs the linked information as the ingredient candidate corresponding to the ingredient position (step S170). Finally, when the user taps a pixel area in the camera image 300 in which an ingredient is captured, the display unit 260 displays the ingredient candidate linked to the area on the app screen (step S180).
[0045] In the third embodiment of the present invention, by using the first arrival time based on the video captured by the camera 120b and the second arrival time stored in the inventory list 400, the number of appearance images 410a of ingredients to be compared with the camera image 330 can be reduced compared to the first or second embodiment, thereby improving the accuracy of the similarity calculation and reducing the amount of calculation.
[0046] In this embodiment, camera 120b may be installed in the same position as camera 120a in Fig. 1. Camera 120b is set so as to capture images from the front of revolving door storage chamber 10, and may be installed in any position so as to capture images of food ingredients (camera images) at least as they are stored inside revolving door storage chamber 10, and at the same time, so as to capture images of food ingredients entering or leaving revolving door storage chamber 10. EXAMPLES
[0047] The operation of the storehouse according to the fourth embodiment of the present invention will be described with reference to FIG. 11 is a flow chart showing the system operation regarding inventory management in the revolving door storage room 10. Steps S100 to S180a are the same as those in the third embodiment.
[0048] After step S180a, the similarity calculation unit 240 recognizes the ingredient information selected by the user as the correct answer value (step S300). Next, the inventory list generation unit 220 adds images taken from various angles while tracking the ingredient (step S230 in FIG. 10) to the ID-ingredient information database 230 as ingredient appearance images 410 linked to the ingredient information (step S310). Note that it is desirable to use an image processing method such as contour extraction to cut out only the tracked object from the images acquired during tracking, and delete background information.
[0049] In the fourth embodiment of the present invention, every time a user puts food in or takes food out of the revolving door storage compartment 10, images taken from various angles can be added to the inventory list as appearance images 410 of the food. This makes it possible to improve the accuracy of similarity calculation along with the frequency of use of the refrigerator 1. EXAMPLES
[0050] The operation of the storehouse according to the fifth embodiment of the present invention will be described with reference to FIG. 12 is an example of an inventory list. Reference numeral 470 denotes a user input area, 471 denotes a text information input area constituting user input area 470, and 472 denotes an order registration area constituting user input area 470.
[0051] In the fifth embodiment of the present invention, after the user taps on the camera image, the user selects the desired ingredient information 401c from the content of the ingredient candidates corresponding to the position of the ingredient in the displayed camera image, and further inputs the desired information in the user input area 470. For example, in the text information input area 471, the user inputs information that the user wants to note, such as the occupant information of the ingredient or information about the user that the ingredient contains allergens and therefore should not be eaten by certain users. In addition, in the order registration area 472, registration is made such as notifying the user when the ingredient is used up, automatically adding the ingredient to the order list, and performing automatic ordering.
[0052] In the fifth embodiment of the present invention, it becomes possible for the user to manage information that he / she wishes to note down. Furthermore, it is possible to reduce the occurrence of forgetting to buy ingredients when the ingredients are used up, and it is possible to improve the convenience for the user.
[0053] In the above embodiments, for the sake of simplicity, an RFID tag has been used as an example of a means for generating an inventory list, but other media capable of acquiring an ID, such as a two-dimensional barcode, may be used. Also, while an ID having a serial number, such as an EPC (Electronic Product Code) code, is preferable, an ID having no serial number, such as a JAN (Japanese Article Number) code, may also be used.
[0054] In each embodiment, the camera 120a or 120b is installed at a position where it can photograph the revolving door storage chamber 10 of the refrigerator 1 from the front direction, but it may be installed at any position, such as the ceiling or side inside the revolving door storage chamber 10, the storage chamber side of the left door 11, or the storage chamber side of the right door 12, or a plurality of positions. In each embodiment, the operation flow is described in which the timing of the camera acquiring an image is between the time when the door opening / closing information generating unit 210 outputs "open" (step S100) and the time when the door opening / closing information generating unit 210 outputs "close" (step S130), but when the camera 120 is installed at the ceiling or side inside the revolving door storage chamber 10, the storage chamber side of the left door 11, or the storage chamber side of the right door 12, it may be a predetermined timing after the door opening / closing information generating unit 210 outputs "close" (until the door opening / closing information generating unit 210 next outputs "open"), not limited to before the door opening / closing information generating unit 210 outputs "close" (step S130). In addition, when the camera 120 is installed on the ceiling or side inside the revolving door storage room 10, it is desirable to turn on the interior light or the light adjacent to the camera 120, or both, when the camera 120 captures an image.
[0055] Furthermore, in each embodiment, a structure and an inventory determination method using two antennas, the internal antenna 100 and the external antenna 110, as antennas for reading RFID tags have been described, but only the internal antenna 100 may be used. Also, in each embodiment, an operation flow has been described in which the timing for reading the RFID tag is between the time when the door opening / closing information generating unit 210 outputs "open" (step S100) and the time when the door opening / closing information generating unit 210 outputs "closed" (step S130), but when only the internal antenna 100 is used, the timing is not limited to before the door opening / closing information generating unit 210 outputs "closed" (step S130), and may be a predetermined timing after the output of "closed" (until the door opening / closing information generating unit 210 next outputs "open"). Similarly, the timing at which the inventory list generation unit 220 performs an inventory determination and generates the inventory list 400 (steps S140 to S150) does not have to be immediately after the door opening / closing information generation unit 210 outputs "closed" (step S130), but may be a predetermined timing after the door opening / closing information generation unit 210 outputs "closed" (step S130) and the RFID tag reading operation (step S110) is completed (until the door opening / closing information generation unit 210 next outputs "open").
[0056] Furthermore, if the timing of capturing a video by camera 120 is arbitrary, the similarity may be calculated by comparing the exterior image stored in the past that is closest to the timing of capturing the video with camera image 300. Furthermore, a part of the configuration or operation of one embodiment may be replaced with or added to the configuration or operation of another embodiment, and the installation positions of cameras and antennas may be combined in any form. [Explanation of symbols]
[0057] 1. Refrigerator 10 Revolving Door Storage Room 11 Left door (of revolving door storage room) 12 Right door (of revolving door storage room) 20 First drawer door storage room 21 Second drawer door storage room 30 Third drawer storage room 40 Fourth Drawer Storage Room 100 Interior antenna 110 Outside antenna 120 Camera 200 Reading unit 210 Door opening / closing information generating unit 220 Inventory List Generation Unit 230 ID-Food Information Database 240 Similarity calculation unit 250 Integration Department 260 Display 270 Feature Extraction Unit 280 Stock List Candidate Selection Section 300 camera images 310 First feature information added to camera image 320 First entry time 330 Camera image linked to the first entry time 400 Stock List 401 Ingredient information 410 Exterior image 420 (food) name 430 (Food storage) 440 (food) expiry date (or best before date) 450 Second entry time 460 Second feature information 461 (Food) Type Information 462 (Food) Package Information 470 User Input Area 471 Text information input area 472 Order Registration Area 500 Contents of candidate ingredients corresponding to the positions of ingredients in the camera image 510 Ingredients 600 app screens 700 users
Claims
1. 1. An inventory management system for managing an inventory of ingredients using a reading unit that reads an ID attached to an ingredient and a camera that captures a first image including the ingredient, an inventory list generating unit that generates an inventory list in which ingredient information corresponding to the ID is stored based on the reading result by the reading unit; a similarity calculation unit that calculates a similarity based on the first image captured by the camera and the ingredient information stored in the inventory list; and an integration unit that links the ingredient information, the similarity of which is equal to or greater than a predetermined threshold, to the first image.
2. In claim 1, The first image is an image captured by a camera of the inside of the storage chamber in a state where the foodstuffs are stored, An inventory management system further comprising a display unit that displays the ingredient information linked by the integration unit when an area of the ingredient in the first image is specified.
3. In claim 2, When there are a plurality of pieces of ingredient information whose similarity is equal to or greater than a predetermined threshold, the display unit displays each piece of ingredient information side by side.
4. In claim 1, The inventory list also stores, as the ingredient information, a second image that is an appearance image corresponding to the ID, An inventory management system characterized in that the similarity calculation unit compares the first image with the second image to calculate the similarity.
5. In claim 4, Further comprising a feature extraction unit that extracts first feature information related to a type or a package of the ingredient from the first image; In the inventory list, second characteristic information corresponding to the ID is also stored as the ingredient information, An inventory management system characterized in that the similarity calculation unit calculates the similarity by comparing the first feature information extracted from the first image with the second feature information stored in the inventory list.
6. In claim 1, The time when the camera determines that the food material has been stored based on the captured video is set as a first storage time, The time when the inventory list generating unit determines that the ingredient has been stored based on the reading result by the reading unit is a second storage time. An inventory management system characterized in that the similarity calculation unit calculates the similarity based on the first image and the ingredient information for which the difference between the first arrival time and the second arrival time is less than a predetermined threshold.
Citation Information
Patent Citations
Management system and program
JP2015125535A
